American Journal of Evaluation

The authors present a stakeholder-driven method, the earliest anticipated timeline of impact, which is designed to assess stakeholder expectations for the earliest time frame in which social programs are likely to affect outcomes. The utility of the anticipated timeline of impact is illustrated using an example from an evaluation of a comprehensive community initiative in which such a timeline was developed using the concept-mapping methodology. The benefits of such a timeline, including for planning programs and evaluations, are explored. Some potential problems that might arise when developing a stakeholder-driven timeline of impact are also discussed.

Interactive online diaries are a novel approach for evaluating project implementation and provide real-time communication between evaluation staff members and those implementing a program. Evaluation teams can guide the lines of inquiry and ensure that data are captured as implementation develops. When used together with conventional evaluation strategies, interactive online diaries allow for an in-depth understanding of project implementation, as well as the challenges program staff members confront and the solutions they develop. Interactive online diaries also can help evaluators address challenges such as self-reporting bias, documenting project evolution, and capturing implementers’ ongoing insights as they develop. These insights might otherwise be lost to the evaluation process. The authors describe the development and use of this online approach in the evaluation of a foundation-sponsored program to improve the provision of preventive care in physicians’ offices. The program included 17 practice-based research networks and their participating primary care practices.

Deliberative democracy has attracted increasing attention in political science and has been suggested as a normative ideal for evaluation. This article analyzes to what extent evaluations carried out in a highly government-driven manner can nevertheless contribute to deliberative democracy. This potential is examined by taking the Organisation for Economic Co-operation and Development’s environmental performance reviews as an example of an expert-led evaluative process built on the ideals of representative democracy. The author argues that although they are not participatory, these reviews lay the groundwork for deliberative democracy by “empowering” weaker actors within governments and by improving the factual basis for political debate and decision making. This example suggests that to enhance deliberative democracy, the evaluation process need not be highly inclusive, dialogical, and deliberative but that a broader view is needed, encompassing the indirect impacts of evaluation on power relations and on the knowledge basis on which decision making relies.

Clearly defined and measurable goals are commonly considered prerequisites for effective evaluation. Goal setting, however, presents a paradox to evaluators because it takes place at the interface of rationality and values. The objective of this article is to demonstrate a method for unlocking this paradox by making goal setting a process of evaluating goals, not simply defining them. Goals can be evaluated by asking program stakeholders why their goals are important to them. Systematic inquiry into goals also prepares the ground for setting consensual goals that express what stakeholders really care about. This article describes the method, provides a case illustration, offers guidelines for practice, and discusses the method in the context of the evaluation literature on goals and goal setting.

This article provides a critical review of the quality of 12 recent large federal program evaluations. The review focused on elements of the evaluation design, inclusion of evaluation expertise among those who have oversight of the evaluation, and evaluation dissemination. Overall, the process analyses from these evaluations provide good models for how to assess implementation and provide feedback to grantees. However, other features of these evaluations require improvement. For example, program monitoring often did not include adequate outcome data and few evaluations had solid impact analyses. Efforts to disseminate findings to grantees, government stakeholders, and academic audiences were also weak. The authors provide recommendations to improve the evaluations of large federal programs.

A general inductive approach for analysis of qualitative evaluation data is described. The purposes for using an inductive approach are to (a) condense raw textual data into a brief, summary format; (b) establish clear links between the evaluation or research objectives and the summary findings derived from the raw data; and (c) develop a framework of the underlying structure of experiences or processes that are evident in the raw data. The general inductive approach provides an easily used and systematic set of procedures for analyzing qualitative data that can produce reliable and valid findings. Although the general inductive approach is not as strong as some other analytic strategies for theory or model development, it does provide a simple, straightforward approach for deriving findings in the context of focused evaluation questions. Many evaluators are likely to find using a general inductive approach less complicated than using other approaches to qualitative data analysis.

Human service fields, and more specifically public health, are increasingly requiring evaluations to prove the worth of funded programs. Many public health practitioners, however, lack the required background and skills to conduct useful, appropriate evaluations. In the late 1990s, the Centers for Disease Control and Prevention (CDC) created the Framework for Program Evaluation in Public Health to provide guidance and promote use of evaluation standards by public health professionals. The emphasis of the Framework is utilization-focused evaluation for program improvement or to assess program impact. This article presents the use of the CDC Framework for Program Evaluation in Public Health to create and teach practical evaluation methods to master’s of public health students. The article includes the teaching approach, semester-long syllabus for students, and course evaluation data; suggests supplementary materials; and discusses implementation issues.

Developing evaluation capacity in organizations is a complex and multifaceted task. This article outlines a framework for building evaluation capacity. The framework is based on four strategic methods for teaching evaluation: (a) using logic models for sound program planning, (b) providing one-on-one help, (c) facilitating small-team collaborative evaluations, and (d) conducting large-scale multisite evaluations. The article also reports the results of using the framework successfully with Extension 4-H field educators. 4-H educators who were trained using this method significantly increased their evaluation knowledge and skill and reported feeling more positively about evaluation. In addition, the results show that the 4-H organization as a whole has developed a more positive evaluation culture. This framework for teaching evaluation has potential value for all organizations interested in developing evaluation capacity in regional educators as well as developing a positive culture of evaluation within the organization.