American Journal of Evaluation

Evaluation practice has grown in leaps and bounds in recent years. In contrast, the most recent survey data suggest that there has been a sharp decline in the number and strength of preservice evaluation training programs in the United States. In an effort to further understand this curious trend, an alternative methodology was used to examine the current state of university-based evaluation training programs in the United States. An online search and curricular document analysis suggest different trends, including a dramatic increase in the number of evaluation training programs in the United States, specifically within schools of education. The importance of using alternative methodologies for understanding the nature of evaluation training programs is discussed.

As is the case with other fields, there is motivation for studying the impact that the body of evaluation theory literature has within and outside the field. The authors used journal articles written by theorists included on the evaluation theory tree by Alkin and Christie (2004; Christie & Alkin, 2008) and published in the Web of Science, an online academic database, as a data source to address the questions: ‘‘What fields of knowledge do evaluation theorists draw upon in their publications?’’ and ‘‘What fields draw upon the published articles of evaluation theorists in their work?’’ The bibliometric analysis of 731 journal articles published by evaluation theorists shows that evaluation is an intellectual field that is strongly informed by psychology and education, as well as a range of other subjects. There are some consistencies in the publishing patterns of the theorists across the three branches of the theory tree (methods, use, and valuing), but multidimensional scaling maps show that each branch also exhibits a distinctive character of its own. References to a random sample of 500 articles from a subset of 9 theorists indicate that these theorists were cited not only in the areas that they themselves cite, but also in areas beyond where they routinely publish.

A simulation study was conducted in an attempt to examine how evaluators modify their evaluation design in response to differing stakeholder groups. In this study, evaluators were provided with a fictitious description of a school-based program. They were then asked to design an evaluation of the program. After the evaluation design decisions were made, evaluators were presented with feedback from three differing stakeholder groups (i.e., decision maker, implementer, recipient) either endorsing or rejecting the evaluation design. Evaluators were then given the opportunity to modify (or not modify) their original design in response to stakeholder feedback. The findings revealed that the more political power or influence stakeholder groups held over evaluation logistical factors (i.e., funding, data access), the more evaluators were willing to modify their design choices to accommodate perceived stakeholder concerns. These design modifications were typically implemented to ensure data access, reduce stakeholder resistance, and increase stakeholder buy-in.

As bibliometric indicators are objective, reliable, and cost-effective measures of peer-reviewed research outputs, they are expected to play an increasingly important role in research assessment/management. Recently, a bibliometric approach was developed and integrated within the evaluation framework of research funded by the National Cancer Institute of Canada (NCIC). This approach helped address the following questions that were difficult to answer objectively using alternative methods such as program documentation review and key informant interviews: (a) Has the NCIC peer-review process selected outstanding Canadian scientists in cancer research? (b) Have the NCIC grants contributed to increasing the scientific performance of supported researchers? (c) How do the NCIC-supported researchers compare to their neighbors supported by the U.S. National Cancer Institute? Using the NCIC evaluation as a case study, this article demonstrates the usefulness of bibliometrics to address key evaluation questions and discusses its integration, along complementary indicators (e.g., peer ratings), in a practice-driven research evaluation continuum.

In this article, we explore the evaluation capacity of today’s nonprofit organizations. We report the findings of a cluster analysis that suggest that when it comes to evaluation, there are three types of nonprofit organizations. The first type of nonprofit organization is one that, by most accounts, is satisfied with their evaluation efforts. Although these organizations report that they struggle with not having as much time as they would like to devote to evaluation, they are fairly satisfied with their levels of evaluation expertise and report having few problems with the implementation of evaluation systems. The second type of organization has some struggles with evaluation. These organizations report having internal support for evaluation from management, the board and staff, and some capacity to implement an evaluation system, yet they struggle with evaluation design issues, data collection, and resources for evaluation. The third type of organization is one that is struggling across the board. These organizations report having substantial implementation challenges, in terms of lacking basic resources (i.e., staff, funding, time), lacking evaluation expertise, and they report having little support for evaluation from funders, the board, management, and staff. We conclude by exploring the implications of these findings.

Legitimate knowledge claims about causation have been a central concern among evaluators and applied researchers for several decades and often have been the subject of heated debates. In recent years these debates have resurfaced with a renewed intensity, due in part to the priority currently being given to randomized experiments by many funders of evaluation studies, such as the Institute for Educational Sciences. In this dialogue, which took place at Western Michigan University in October 2008, two of the field’s leading theorists and methodologists, Thomas D. Cook and Michael Scriven, described their current thinking and views about causation and causal inference in evaluation. They also discussed recent methodological developments for cause-probing investigations that sometimes produce results comparable to those produced by randomized experiments. Both Cook and Scriven prepared clarifying postscripts after reading the edited transcript.