American Journal of Evaluation

Reasons for unintended consequences of program action are explained. Although this topic has been dealt with extensively in the field of planning, that literature provides little guidance for how evaluation methodologies can be improved. Implications for evaluation are provided by drawing the distinction between consequences that are unforeseen and consequences that are unforeseeable and then recommending tactics for dealing with each.

Is it appropriate and useful for evaluators to use findings to make recommendations? If so, under what circumstances? How specific should they be? This article presents a decision-making framework for the appropriateness of recommendations in varying contexts. On the basis of reviews of evaluation theory, selected evaluation reports, and feedback from American Evaluation Association presentations, the authors present a description of contextual variables likely to affect the making of recommendations (e.g., use context, evaluator role) and a typology of potential recommendations. To enhance the utility of the tools presented, the authors summarize the available literature and present key arguments for and against the provision of various types of recommendations in given contexts. This work is designed to be a step toward creating a more systematic process for considering the use of recommendations by providing evaluators with a comprehensive tool for reflection and decision making.

A multistakeholder evaluation procedure is presented to address the many challenges in evaluating the performance of condominial sewer projects in Brazil. Condominial sewerage is a promising appropriate technology that is coproduced by users and public agencies, but little is known about project performance. This article shows that multistakeholder evaluation is more appropriate to the research setting than evaluations based on a single stakeholder’s perspective, is more reliable than the use of expert judgments alone, and provides more information than recommended sewer performance indicators. The author argues that the perspectives of multiple stake-holders should be included in project evaluations. The multistakeholder approach presented is a workable solution to the condominial sewer evaluation problem.

The purpose of the present study was to examine a common practice in some areas of program evaluation, the retrospective pretest, and to present recommendations regarding its use. The authors review literature to emphasize first, that bias is likely in both prospective and retrospective ratings, and second, that under some circumstances, retrospective pretests may introduce greater bias than traditional pretests. The authors examined data from 100 participants who attended a family program at 15 sites. Results supported the hypothesis that items on which parents were asked to endorse socially desirable parenting behaviors resulted in greater discrepancies between prospective and retrospective ratings. Effect sizes calculated from prospective ratings were comparable with published effect sizes from research trials of the same program. The authors conclude that replacing traditional with retrospective pretests does not eliminate bias. They recommend traditional pretests for examination of program effects and retrospective pretests for examination of subjective experiences of program-related change.

This article reviews changes in the accountability landscape that have occurred for foundations in recent years and several precedents for foundation performance assessment. The authors then present a model of portfolio assessment that is used for organizational accountability and learning. This model, which was piloted in 2002 and 2003 for the Minnesota Partnership for Action Against Tobacco (MPAAT) and is in continuing use today, consists of a report card system based on an organizational logic model and the intended intermediary outcomes of MPAAT’s funding. The model is conducted by an external oversight evaluator and results in aggregated performance measures and findings that are publicly disseminated on an annual basis. This approach to portfolio assessment may be especially relevant for smaller foundations, public trusts, community foundations, health care conversions, government agencies, and other organizations that disburse grants in politically charged fields, such as reproductive rights, the environment, and health policy.

Educational accountability is a fundamental right of citizens in a democratic society serving the public interest. The No Child Left Behind Act of 2001 holds states, school districts, public officials, educators, parents, and students accountable through auditable performance standards. At the same time, the lack of discussion about how to decide what educational outcomes should be typifies a climate of control. This article proposes making educational accountability more democratic by constructing democratic accountability within the context of a local school and/or district as a democratic conversation. The foundations of democratic accountability, the meaning of democratic accountability, and how its meaning is constructed (including accountability for what and to whom and how it is implemented) are outlined.

This article describes the Self-Assessment of Program Evaluation Expertise instrument and procedure developed to help participants assess their learning gains in a 3-week evaluation institute. Participants completed the instrument in a pre- and posttest format. To reduce both the threat of embarrassment from individual results and the temptation to inflate self-ratings, participants responded anonymously. Although each participant saw her or his individual results, only aggregated results were reported to the total group. The article reports on the self-assessment results of a group of recognized evaluation elders and participants in six annual evaluation institutes. The findings indicate that the instrument is sensitive to the respondents’ changed perceptions of competency following instruction. Strengths and limitations associated with the procedure are discussed. The procedure is presented as a work in progress and could benefit from adaptation, research, and development. The instrument is available at http://www.wmich.edu/evalctr/ess/selfassess.html.

The use of fieldwork experiences has been widely accepted by teachers of evaluation as a method of preparing future practitioners for challenges encountered in the field. However, several challenges must be addressed to make fieldwork experiences a useful tool in preparing students for their professional careers. The authors present a case study in which a graduate student in an evaluation class approached an agency seeking a fieldwork project. The experience was not positive. Both a student’s and an instructor’s perspectives are offered in understanding the reasons for the negative experience. Suggestions are provided from both perspectives as to how the experience could have been improved. The article is intended as a teaching tool to be provided by instructors to students seeking applied experiences before approaching agencies. This article provides an opportunity for student evaluators to learn from the experiences of another student; likewise, the instructor in this case study provides insights that other instructors of evaluation may find helpful.

What is the relationship between theory and design in evaluation pretending to be “democratic?” When do we feel able to relinquish elements of intellectual control over evaluation? If not in qualitative versus quantitative tendencies, where lie key value divisions in evaluation? To elicit the views of two leading United Kingdom-!based theorists of evaluation, Ray Pawson and Nigel Norris, Saville Kushner penned a brief argument positing the democratic imperative of ceding theoretical control to participants. Pawson, coauthor of Realistic Evaluation, has intellectual preferences for (contextualized) causal reasoning; Norris, a close associate of Barry MacDonald, is a long-standing advocate of democratic evaluation. Although not necessarily diametrically opposed, their approaches to evaluation theorizing are distinct. In his “Aunt Sally” opener, Kushner argues that choosing or not to cede or share theoretical control of an evaluation with participants better characterizes differences between value positions in evaluation than more conventional bipolarities such as qualitative-quantitative or “strong” and “weak” designs. Norris and Pawson move this simple argument further and explore its more complex dimensions.

The Internet and World Wide Web are increasingly used to accelerate progress in a variety of fields. These applications go beyond the traditional technology- or computer-related fields and have expanded to nontechnical fields, which have benefited greatly from the use of the Web as a data-gathering and management support tool. The purpose of this article is to examine the Web’s use by a federal funding agency and its grantees for performance measurement and program monitoring. The analysis is based on the Center for Substance Abuse Treatment’s (CSAT) implementation of a Web-based data entry and reporting system. This system is used both as a mechanism for monitoring CSAT’s grant portfolios and as a tool to manage performance. The authors conclude with a discussion of the issues surrounding the effectiveness and efficiency of using the Web to measure and monitor program performance.