The main function of program evaluations is to describe programs in order to generate judgments of value. To be considered credible, judgments should be both legitimate and justified. The research presented in this article posed the following question: do program evaluation practitioners generate legitimate and justified judgments? A meta-analysis of 40 program evaluation reports was carried out, which found that only 50 percent of the reports generated judgments. While these judgments seemed legitimate, they were rarely justified. However, the elements required to support legitimate and justified judgments were present in the reports in similar proportions, whether the reports generated a judgment or not.
Arens, S.A. (2005) `Understanding Practice through Exemplary Evaluation Cases', paper presented at the 2005 Joint Conference of CES/AEA, Toronto.
2.
Arens, S.A. (2006) `L'étude du raisonnement dans les pratiques évaluatives' , Mesure et évaluation en éducation29(3): 45-56.
3.
Booth, W., G. Colomb and J. Williams (2003) The Craft of Research. Chicago : University of Chicago Press.
4.
Datta, L.E. (2006) `The Practice of Evaluation Challenges and New Directions', in F. Shaw, J. C. Greene and M. M. Mark (eds) The Sage Handbook of Evaluation, pp. 419-38. Thousand Oaks, CA: SAGE.
5.
Davidson, E.J. (2005) Evaluation Methodology Basics: The Nuts and Bolts of Sound Evaluation. Thousand Oaks, CA: SAGE.
6.
Fournier, D.M. (1995) `Establishing Evaluative Conclusions: A Distinction between General and Working Logic', in D. M. Fournier (ed.) Reasoning in Evaluation: Inferential Links and Leaps, pp. 15-31. New Directions for Evaluation, 68. San Francisco: Jossey-Bass.
7.
Fournier, D.M. and N.L. Smith (1993) `Clarifying the Merits of Argument in Evaluation Practice', Evaluation and Program Planning16(4): 315-23.
8.
Guba, E.G. (1972) `The Failure of Educational Evaluation', in C. H. Weiss (ed.) Evaluating Action Programs: Readings in Social Action and Education, pp. 251-66. Boston, MA: Allyn & Bacon.
9.
Habermas, J. (1979) Communication and the Evolution of Society. Boston, MA: Beacon Press.
10.
House, E.R. (1980) `Logic of Evaluative Argument', in E. R. House (ed.) Evaluating with Validity , pp. 67-96. Beverly Hills, CA: SAGE.
11.
House, E.R. (1995) `Putting Things Together Coherently: Logic and Justice', in D. M. Fournier (ed.) Reasoning in Evaluation: Inferential Links and Leaps, pp. 33-48. New Directions for Evaluation, 68. San Francisco: Jossey-Bass .
12.
House, E.R. and K.R. Howe (1999) Values in Evaluation and Social Research. Thousand Oaks, CA: SAGE.
13.
Hurteau, M. and S. Houle (2005) `Identifying a Core Body of Knowledge for Evaluators' , paper presented at the 2005 Joint Conference of CES/AEA, Toronto.
14.
Hurteau, M., G. Lachapelle and S. Houle (2006) `Comprendre les pratiques évaluatives afin de les améliorer: la modélisation du processus spécifique à l'évaluation de programme' , Mesure et évaluation en éducation29(3): 27-44.
15.
Joint Committee on Standards for Educational Evaluation (1994) Program Evaluation Standards: How to Assess Evaluations of Educational Programs. Thousand Oaks, CA: SAGE.
16.
Krippendorf, K. (2004) Content Analysis: An Introduction to its Methodology . Thousand Oaks, CA: SAGE.
17.
McCarthy, T.A. (1973) `A Theory of Communicative Competence', Philosophy of the Social Sciences3: 135-56.
18.
Merriam-Webster Online Dictionary (2004) Available at http://www.MerriamWebsterCollegiate.com
Patton, M. (1997) Utilization-Focused Evaluation: The New Century Text. Thousand Oaks, CA: SAGE.
21.
Redding, P. (1989) `Habermas' Theory of Argumentation', Journal of Value Inquiry23: 15-32.
22.
Rog, D.J. (1995) `Reasoning in Evaluation: Challenges for the Practitioner' , in D. M. Fournier (ed.) Reasoning in Evaluation: Inferential Links and Leaps, pp. 93-100. New Directions for Evaluation, 68. San Francisco: Jossey-Bass .
23.
Rourke, L. and T. Anderson (2004) `Validity in Quantitative Content Analysis', Educational Technology Research and Development52(1): 5-17.
24.
Schwandt, T.A. (2002) Evaluation Practice Reconsidered. New York: Peter Lang.
25.
Schwandt, T.A. (2008) `The Relevance of Practical Knowledge Traditions to Evaluation Practice', in N. L. Smith and P. R. Brandon (eds) Fundamental Issues in Evaluation, pp. 29-40. New York: Guilford Press.
26.
Scriven, M.( 1980) The Logic of Evaluation. Inverness, CA: Edgepress.
27.
Scriven. M. (1990) `The Evaluation of Hardware and Software', Studies in Educational Evaluation16: 3-40.
28.
Scriven, M. (1991a) Evaluation Thesaurus. Newbury Park, CA: SAGE.
29.
Scriven, M. (1991b) `The Science of Valuing', in W. E. Shadish, T. D. Cook and L. C. Leviton (eds) Foundations of Program Evaluation: Theories of Practice, pp. 73-118. Thousand Oaks, CA: SAGE.
30.
Scriven, M. (1993) `Evaluation and Critical Reasoning: Logic's Last Frontier', in R. Talaska (ed.) Critical Reasoning in Contemporary Culture, pp. 353-406. Albany State, NY: University of New York Publications.
31.
Scriven, M. (1995) `The Logic of Evaluation and Evaluation Practice' , in D. M. Fournier (ed.) Reasoning in Evaluation: Inferential Links and Leaps, pp. 49-70. New Directions for Evaluation, 68. San Francisco: Jossey-Bass .
32.
Shadish, W.E., T.D. Cook and L.C. Leviton (1991) Foundations of Program Evaluation: Theories of Practice. Thousand Oaks, CA: SAGE .
Stake, R., C. Migotsky, R. Davis, E.J. Cisneros, G. Depaul, C. Dunbar, R. Farmer, J. Feltovich, E. Johnson, B. Williams, M. Zurita and I. Chaves (1997) `The Evolving Syntheses of Program Value', Evaluation Practice18(2): 89-103.
35.
Stake, R.E. and T.A. Schwandt (2006) `On Discerning Quality in Evaluation', in I. F. Shaw, J. C Greene and M. M Mark (eds) The Sage Handbook of Evaluation, pp. 404-18. Thousand Oaks, CA : SAGE.
Toulemonde, J. (2005) `Appropriation des résultats de l'évaluation: leçons de la pratique en Région Limousin', paper presented at the 2005 Colloque de la société française de l'évaluation, Lille.
38.
Toulmin, S.E. (1964) The Uses of Argument. New York : Cambridge University Press.
39.
Toulmin, S.E., R.D. Rieke and A. Janik (1984) An Introduction to Reasoning. New York: Macmillan.
40.
Trelogan, T.K. (2001) `Arguments and their evaluation', Department of Philosophy, University of Northern Colorado, unpublished manuscript.
41.
Treasury Board of Canada Secretariat (2004 ) Examen de la qualité des évaluations dans les ministères et les organismes. URL (consulted Sept. 2006) : http://www.tbs-sct.gc.ca/eval/pubs/rev-exam_f.asp
42.
Valovirta, V. (2002) `Evaluation Utilization as Argumentation', Evaluation8(1): 60-80.
43.
Wheeler, P., G.D. Haertel and M. Scriven (1992) `Teacher Evaluation Glossary', Kalamazoo, MI: CREATE Project, Evaluation Center, Western Michigan University, unpublished manuscript.
