The Validation of the Evaluation Involvement Scale for Use in Multisite Settings

Abstract

Evaluation researchers and practitioners acknowledge that involving stakeholders in the planning and implementation of an evaluation increases buy-in, understanding, and use. With the recent increase in multi-site evaluations of large federal programs, evaluators must think differently about how to encourage meaningful collaboration by stakeholders. To date, there has been no published measure of such involvement, despite recent calls for more systematic, replicable research. The purpose of this study was to validate the Evaluation Involvement Scale for use in multi-site evaluations. Between the fall of 2006 and spring of 2007 data were collected from an electronic survey and phone interviews of evaluators and principal investigators of four National Science Foundation program evaluations. Using Messick’s unitary concept of validity as a framework, theoretical, statistical, and rational evidence is provided to support the use of the Evaluation Involvement Scale to measure stakeholder involvement in multi-site evaluations.

Keywords

participatory evaluation stakeholder involvement evaluation scale validation

Get full access to this article

View all access options for this article.

References

Burke, B. ( 1998). Evaluating for a change: Reflections on participatory methodology. New Directions for Evaluation, 80, 43-56.

Cousins, J.B. ( 2003). Utilization effects of participatory evaluation. In T. Kellaghan & D. I. Stufflebeam (Eds.), International handbook of educational evaluation (pp. 245-266). Boston: Kluwer Academic Publishers.

Cousins, J.B. , & Earl, L.M. ( 1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 14, 397-418.

Cousins, J.B. , Goh, S.C. , Clark, S. , & Lee, L. ( 2004). Integrating evaluative inquiry into the organizational culture: A review and synthesis of the knowledge base. The Canadian Journal of Program Evaluation, 19, 99-141.

Cousins, J.B. , & Whitmore, E. ( 1998). Framing participatory evaluation. New Directions for Evaluation, 80, 5-23.

DeVellis, R.F. ( 2003). Scale development: Theory and applications (2nd ed.). Thousand Oaks, CA: Sage.

Fetterman, D.M. ( 2000). Foundations of empowerment evaluation. Thousand Oaks, CA: Sage.

Gaventa, J. , Creed, V. , & Morrissey, J. ( 1998). Scaling up: Participatory monitoring and evaluation of a federal empowerment program. New Directions for Evaluation , 80, 81-94.

Greene, J. ( 2005). A value-engaged approach for evaluating the Bunche-DaVinci Learning Academy. New Directions for Evaluation, 106, 27-45.

10.

Guba, E.G. , & Lincoln, Y.S. ( 1989). Fourth generation evaluation. Thousand Oaks, CA: Sage.

11.

Henry, G.T. , & Mark, M.M. ( 2003). Toward an agenda for research on evaluation. New Directions for Evaluation, 97, 69-80.

12.

House, E.R. , & Howe, K.R. ( 1999). Values in evaluation and social research. Thousand Oaks, CA: Sage.

13.

Hu, L. , & Bentler, P.M. ( 1999). Cuttoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-5.

14.

King, J. ( 1998). Making sense of participatory evaluation practice. New Directions for Evaluation, 80, 57-67.

15.

Kirkhart, K.E. ( 1994). Seeking multicultural validity: A postcard from the road . Evaluation Practice, 16, 1-12.

16.

Lawrenz, F. , & Huffman, D. ( 2003). How can multi-site evaluations be participatory? American Journal of Evaluation, 24, 471-481.

17.

Mertens, D. ( 1999). Inclusive evaluation: Implications of transformative theory on evaluation. American Journal of Evaluation, 20, 1-14.

18.

Messick, S. ( 1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18, 5-11.

19.

Messick, S. ( 1995a). Standards of validity and the validity of standards in performance assessment. Educational Measurement: Issues and Practice, 14, 5-8.

20.

Messick, S. ( 1995b). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.

21.

O’Sullivan, R. , & O’Sullivan, J. (1998). Evaluation voices: Promoting evaluation from within programs through collaboration. Evaluation and Program Planning, 21, 21-29.

22.

Roberts, J.K. ( 1999). Basic concepts of confirmatory factor analysis. Southwest Educational Research Association Annual Conference. San Antonio, TX.

23.

Stevens, J. ( 1996) Applied multivariate statistics for the social sciences . Mahwah, NJ: Lawrence Erlbaum.

24.

Thompson, B. ( 2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. Washington, DC: American Psychological Association.

25.

Thorndike, R.M. ( 2005). Measurement and evaluation in psychology and education . Upper Saddle River, NJ: Pearson Merrill Prentice Hall.

26.

Toal, S.A. , Greenseid, L. , Lawrenz, F. , King, J.A. , & Johnson, K. ( 2006). Researching evaluation use and influence: Twenty years of empirical study. American Evaluation Association Annual Conference, Portland, Oregon.