This paper reviews empirical research on the use of evaluation from 1986 to 2005 using Cousins and Leithwood’s 1986 framework for categorizing empirical studies of evaluation use conducted since that time. The literature review located 41 empirical studies of evaluation use conducted between 1986 and 2005 that met minimum quality standards. The Cousins and Leithwood framework allowed a comparison over time. After initially grouping these studies according to Cousins and Leithwood’s two categories and twelve characteristics, one additional category and one new characteristic were added to their framework. The new category is stakeholder involvement, and the new characteristic is evaluator competence (under the category of evaluation implementation). Findings point to the importance of stakeholder involvement in facilitating evaluation use and suggest that engagement, interaction, and communication between evaluation clients and evaluators is critical to the meaningful use of evaluations.
Altschuld, J.W., Yoon, J.S., & Cullen, C. ( 1993). The utilization of needs assessment results. Evaluation and Program Planning, 16, 279-285.
3.
Ayers, T.D. ( 1987). Stakeholders as partners in evaluation: A stakeholder-collaborative approach. Evaluation and Program Planning, 10, 263-271.
4.
Bamberger, M. ( 2004). Influential evaluations: Evaluations that improved performance and impacts of development programs. Washington, DC: Operations Evaluation Department, The World Bank.
5.
Barrios, N.B. ( 1986). Utilization of evaluation information: A case study approach investigating factors related to evaluation utilization in a large state agency . Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 47, 1704 (UMI 8616880).
6.
Bober, C.E., & Bartlett, K.R. (2004). The utilization of training program evaluation in corporate universities. Human Resource Development Quarterly, 15, 363-383.
7.
Boyer, J.F., & Langbein, L.I. (1991). Factors influencing the use of health evaluation research in Congress. Evaluation Review, 18, 507-532.
8.
Brown-McGowan, S. ( 1992). Effects of decision maker and context variables on evaluation utilization. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 53, 2261 (UMI 9226505).
9.
Cai, M. ( 1996). An empirical examination of participatory evaluation: Teachers’ perceptions of their involvement and evaluation use. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 57, 1921 (UMI 9629749).
10.
Callahan, C.M., Tomlinson, C.A., Hunsaker, S.L., Bland, L.C., & Moon, T. ( 1995). Instruments and evaluation designs used in gifted programs . The National Research Center on the Gifted and Talented, The University of Virginia. Research Report: RM-95132.
11.
Carpinello, S.E. ( 1989). The effect of power, consequence, and experience on nurse decision-makers’ utilization of evaluation information. State University of New York at Albany. Dissertation Abstracts International: Section B: The Sciences & Engineering, 50, 3395.
12.
Chin, M.C. ( 2003). An investigation into the impact of using poetry and cartoons as alternative representational forms in evaluation reporting. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 64, 394 (UMI 3081434).
13.
Christie, C.A. ( 2007). Reported influence of evaluation data on decision makers’ actions: An empirical examination. American Journal of Evaluation , 28, 8-25.
14.
Combs, W.L.A. ( 1999). The predictive validity of teachers’ attitudes towards inclusive education on the conceptual use of program evaluation information . Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 60, 3208 (UMI 9946401).
15.
Cousins, J.B. ( 1995). Assessing program needs using participatory evaluation: A comparison of high and marginal success cases. In J. B. Cousins & L. M. Earl (Eds.), Participatory evaluation in education: Studies in evaluation use and organizational learning (pp. 55-71). London: Routledge .
16.
Cousins, J.B. ( 1996). Consequences of researcher involvement in participatory evaluation. Studies in Educational Evaluation, 22, 3-27.
17.
Cousins, J.B. ( 2003). Utilization effects of participatory evaluation. In T. Kelleghan, & D. L. Stufflebeam (Eds.), International handbook of educational evaluation (pp. 245-265). Great Britain: Kluwer Academic Publishers.
18.
Cousins, J.B., Goh, S.C., Clark, S., & Lee, L.E. ( 2004). Integrating evaluative inquiry into the organizational culture: A review and synthesis of the knowledge base. Canadian Journal of Program Evaluation, 19, 99-141.
19.
Cousins, J.B., & Leithwood, K.A. (1986). Current empirical research on evaluation utilization. Review of Educational Research , 56, 331-364.
20.
Crotti, J.G. ( 1993). Evaluation utilization: A study of administrators’ perceptions of the uses of the long-range plan evaluation process in Pennsylvania . Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 54, 4315 (UMI 9414267).
21.
Earl, L.M. ( 1995). District-wide evaluation of school improvement: A system partners approach. In J. B. Cousins & L. M. Earl (Eds.), Participatory evaluation in education: Studies in evaluation use and organizational learning (pp. 21-32), London : Routledge.
22.
Eisendrath, A. ( 1988). The use of development project evaluation information: A study of state agencies in India. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 49, 1572 (UMI 8810460).
23.
Forss, K., Cracknell, B., & Samset, K. ( 1994). Can evaluation help an organization to learn? Evaluation Review, 18, 574-591.
24.
Greene, J.C. ( 1987). Stakeholder participation in evaluation design: Is it worth the effort? Evaluation and Program Planning, 10, 379-394.
25.
Greene, J.C. ( 1988). Communication of results and utilization in participatory program evaluation. Evaluation and Program Planning, 11, 341-351.
26.
Guarino, C., Santibañez, L., Daley, G., & Brewer, D. ( 2004, May). A review of the research literature on teacher recruitment and retention. Technical report TR-164-EDU. Santa Monica, CA: RAND.
Haddock, R.E. ( 1998). State legislative program evaluation: An assessment of recent claims of direct utilization in the states. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 60, 881 (UMI 9921400).
29.
Henry, G.T., & Mark, M.M. ( 2003a). Beyond use: Understanding evaluation’s influence on attitudes and actions. American Journal of Evaluation, 24, 293-314.
30.
Henry, G.T., & Mark, M.M. ( 2003b). Toward an agenda for research on evaluation. New Directions for Evaluation, 97, 69-80.
31.
Hofstetter, C.H., & Alkin, M.C. ( 2003). Evaluation use revisited. In T. Kelleghan & D. L. Stufflebeam (Eds.), International handbook of educational evaluation (pp. 197-222). Great Britain: Kluwer Academic Publishers.
32.
Hopstock, P., Young, M., & Zehler, A. ( 1993). Serving different masters: Title VII evaluation practice and policy. Vol. I - Final report. Arlington, VA: Development Associates IncReport: ED/OPP93-32.
33.
Johnson, K.W. ( 1980). Academia and practice. Knowledge: Creation, Diffusion, Utilization, 2, 237-261.
34.
Johnson, R.B. ( 1993). An exploratory conjoint measurement study of selected variables related to innovative educational evaluation participation and instrumental utilization. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 55, 64 (UMI 9416264).
35.
Johnston, W.P., Jr. (1986). A study of the acceptance of management performance evaluation recommendations by federal agencies: Lessons from GAO reports issued in FY 1983. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 48, 2157 (UMI 8725055).
36.
King, J.A. & Pechman, E.M. ( 1984). Pinning a wave to the shore: Conceptualizing evaluation use in school systems. Educational Evaluation and Policy Analysis , 6, 241-451.
37.
Kirkhart, K.E. ( 2000). Reconceptualizing evaluation use: An integrated theory of influence. New Directions for Evaluation, 88, 5-23.
38.
Kirkhart, K.E., Morgan, R.O., & Sincavage, J. ( 1991). Assessing evaluation performance and use: Test-Retest . Evaluation Review, 15(4), 482-502.
39.
Lafleur, C. ( 1995). A participatory approach to district-level program evaluation: The dynamics of internal evaluation. In J. B. Cousins & L. M. Earl (Eds.), Participatory evaluation in education: Studies in evaluation use and organizational learning (pp. 33-54). London: Falmer.
40.
Lee, L.E., & Cousins, J.B. ( 1995). Participation in evaluation of funded school improvement: Effects and supporting conditions. In J. B. Cousins & L. M. Earl (Eds.), Participatory evaluation in education: Studies in evaluation use and organizational learning (pp. 72-85). London: Routledge .
41.
Leviton, L.C., & Hughes, E.F.X. (1981). Research on the utilization of evaluations: A review and synthesis. Evaluation Review , 5, 525-549.
42.
Malen, B., Murphy, M.J., & Geary, S. ( 1988). The role of evaluation information in legislative decision making: A case study of a loose cannon on deck. Theory into Practice, 27, 111-125.
43.
Mark, M.M., Henry, G.T., & Julnes, G. ( 2000). Evaluation: An integrated framework for understanding, guiding, and improving policies and programs. San Francisco : Jossey-Bass, Inc.
44.
Mark, M.M., & Henry, G.T. ( 2004). The mechanisms and outcomes of evaluation influence. Evaluation, 10, 35-57.
45.
Marra, M. ( 2003). Dynamics of evaluation use as organizational knowledge: The case of the World Bank. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 64, 1070 (UMI 3085545).
46.
Marsh, D.D., & Glassick, J.M. (1988). Knowledge utilization in evaluation efforts: The role of recommendations. Knowledge , 9, 323-341.
47.
McCormick, E.R. ( 1997). Factors influencing the use of evaluation results. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 58, 4187 (UMI 9815051).
48.
Newman, D., Brown, R., Rivers, L., & Glock, R. ( 1983). School boards’ and administrators’ use of evaluation information: Influencing factors. Evaluation Review , 7(1), 110-125.
49.
Newman, D.L., Brown, R.D., & Rivers, L. ( 1987). Factors influencing the decision-making process: An examination of the effect of contextual variables. Studies in Educational Evaluation, 13, 199-209.
50.
Nunneley, R.D. ( 2008). The danger of theorizing under the influence: An analysis of the arguments in Henry and Mark (2003) and Mark and Henry (2004). Unpublished manuscript. Minneapolis, MN: University of Minnesota.
Potts, S.A.K. ( 1998). Impact of mixed method designs on knowledge gain, credibility, and utility of program evaluation findings. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 59, 1942 ( UMI 9837695).
53.
Preskill, H., & Caracelli, V. ( 1997). Current and developing conceptions of use: Evaluation use TIG survey results. Evaluation Practice, 18, 209-225.
54.
Rinne, C. ( 1994). The impact of anxiety as a mediating variable on health educators’ utilization of evaluation results. Dissertation Abstracts International: Section B: The Sciences & Engineering, 54, 3554.
55.
Rockwell, S.K., Dickey, E.C., & Jasa, P.J. ( 1990). The personal factor in evaluation use: A case study of a steering committee’s use of a conservation tillage survey. Evaluation and Program Planning, 13, 389-394.
56.
Russon, C., & Koehly, L. ( 1995). Construction of a scale to measure the persuasive impact of qualitative and quantitative evaluation reports. Evaluation and Program Planning, 18(2), 165-177.
57.
Santhiveeran, J. ( 1995). Factors influencing the utilization of evaluation findings in mental health centers: A national survey. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 56, 3311 (UMI 9539597).
58.
Scriven, M. ( 2007). Activist evaluation. Journal of MultiDisciplinary Evaluation, 4(7), i-ii.
59.
Shea, M.P. ( 1991). Program evaluation utilization in Canada and its relationship to evaluation process, evaluator and decision context variables. University of Windsor (Canada). Dissertation Abstracts International: Section B: The Sciences & Engineering, 53, 597.
60.
Shulha, L.M., & Cousins, J.B. ( 1997). Evaluation use: Theory, research, and practice since 1986. Evaluation Practice, 18, 195-208.
61.
Sleezer, C.M. ( 1987). The relationship between types of evaluation reports and support for the training function by corporate managers. Project number twenty-one. Department of Vocational and Technical Education, University of Minnesota.
62.
Sperlazza, J. ( 1995). Involving school professionals in program evaluation in an urban school district. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 56, 3406 (UMI 9601924).
63.
Turnbull, B. ( 1999). The mediating effect of participation efficacy on evaluation use. Evaluation and Program Planning, 22, 131-140.
64.
Weeks, E.C. ( 1979). The managerial use of evaluation findings. In H.C. Schulberg & J.M. Jerrell (Eds.), The evaluator and management (pp. 137-156). Beverly Hills, CA: Sage.
65.
Weiss, C.H., Murphy-Graham, E., & Birkeland, S. ( 2005). An alternate route to policy influence: How evaluations affect DARE. American Journal of Evaluation, 26, 12-30.
