American Journal of Evaluation

Quality Assessment Frameworks (QAFs) are standardized templates that include pre-defined criteria, open questions, or a combination to assess the quality of evaluations. They support organizations to limit risks associated with poor-quality evaluation. Qualitative research focused on four QAFs shows that five dimensions of quality are assessed within the frameworks that include pre-defined criteria. These are aligned with the criteria that audiences of evaluation identify as important to assessing quality: substantive findings, robust and appropriate methodology, accessibility, inclusive processes and analysis of wider systems. Consequently, these QAFs are likely to support critical assessment of the main dimensions of quality that are relevant to audiences of evaluation. Audiences also note synergies between these five dimensions of quality and appropriate contextualization as crucial to assessing quality. Consequently, the structure and categorization of quality within QAFs which only include pre-defined criteria risks limiting reviewers’ ability to reflect on these synergies and contextualize assessments.

“Integrative evaluation” is an approach with two main phases: identification of plausible rival hypotheses and integration of rival hypotheses. The first phase may correspond to traditional adversary evaluation, whereas the second phase, that is not included in adversary evaluation, requires integrative thinking which can be applied when they are compatible and complementary. Integrative evaluation may facilitate “evaluative thinking” and contribute to deliberative or integrative democracy.

The importance of considering wider contexts when evaluating the success or failure of programs has been increasingly acknowledged with the shift towards culturally responsive evaluation. But one of the important advantages of contextual approaches has been mostly overlooked—that they can provide more “realist” evaluations for why programs fail or succeed. The careful identification of causal mechanisms involved in program delivery is important for avoiding spurious conclusions about the effectiveness of programs. Drawing on findings from a mixed-methods study conducted in Western Kenya among the Luo to evaluate the impacts of a HIV prevention program involving voluntary male medical circumcision (VMMC), it is shown that the VMMC program was one of several variables that contributed to the desired outcome, being not so much the cause but a catalyst for accelerating the desired behavioral change that the surrounding context was already amenable to and contributing to, even before the program was introduced. The need for context evaluations is particularly obvious when programs are part of broader campaigns involving scale-up from one context to another.

Implementation fidelity is the degree of compliance with which the core elements of program or intervention practices are used as intended. The scientific literature reveals gaps in defining and assessing implementation fidelity in early intervention: lack of common definitions and conceptual framework as well as their lack of application. Through a critical review of the scientific literature, this article aims to identify information that can be used to develop a common language and guidelines for assessing implementation fidelity. An analysis of 46 theoretical and empirical papers about early intervention implementation, published between 1998 and 2018, identified four conceptual frameworks, in addition to that of Dane and Schneider. Following analysis of the conceptual frameworks, a four-component conceptualization of implementation fidelity (adherence, dosage, quality and participant responsiveness) is proposed.

Site visits are common in evaluation plans but there is a dearth of guidance about how to conduct them. This paper revisits site visit standards published by Michael Patton in 2017 and proposes a framework for evaluative site visits. We retrospectively examined documents from a series of site visits for examples of Patton's standards. Through this process, we identified additional standards and organized them into four categories and fourteen standards that can guide evaluation site visits: team competencies and knowledge (interpersonal competence, cultural humility, evaluation competence, methodological competence, subject matter knowledge, site specific knowledge), planning and coordination (project design, resources, data management), engagement (team engagement, sponsor engagement, site engagement), and confounding factors (neutrality, credibility). In the paper, we provide definitions and examples from the case of meeting, and missing, the standards. We encourage others to apply the framework in their contexts and continue the discussion around evaluative site visits.

This paper describes a framework for educating future evaluators and users of evaluation through community-engaged, experiential learning courses and offers practical guidance about how such a class can be structured. This approach is illustrated via a reflective case narrative describing how an introductory, undergraduate class at a mid-size, public university in the northwest partnered with a community agency. In the class, students learned and practiced evaluation principles in the context of a Parents as Teachers home visiting program, actively engaged in course assignments designed to support the program's evaluation needs, and presented meta-evaluative findings and recommendations for future evaluation work to the community partner to conclude the semester. This community-engaged approach to teaching evaluation anchors student learning in an applied context, promotes social engagement, and enables students to contribute to knowledge about effective human action, as outlined in the American Evaluation Association's Mission.

Evaluation has been expanding as an important discipline in Latin America, yet there are significant challenges regarding capacity development through educational and training initiatives in the region. This paper first analyzes the evolution and state of the field in terms of the teaching of, and training in evaluation in Latin America, with a special focus on young and emerging evaluators. The paper then draws on published literature, interviews with evaluation experts, and the results of a survey with VOPE leaders and a survey focused on young and emerging evaluators to illustrate Latin American evaluators’ challenges and needs in terms of capacity building and training in evaluation. Lastly, this paper provides some examples of innovative activities and strategies that are being put in place by different actors to address training and teaching needs of Latin American evaluators.

This article explores the utility of mindfulness in the field of evaluation. Mindfulness is a translation of the ancient Indian word, Sati, which means awareness, attention, and remembering. While definitions vary, a practical definition of mindfulness is present-moment awareness in an open and nonjudgmental manner. Mindfulness-based interventions have been employed by a wide variety of professions. Although it has received limited attention in the writings of evaluators, we argue that mindfulness can improve the practice of evaluation and support the development of the professional practice and interpersonal domains of American Evaluation Association (AEA) evaluator competencies. We review several mindfulness-based practices and how they can be used by evaluators in their work. Thus, we posit that far from being an esoteric concept, mindfulness practices can serve the pragmatic end of improving our discipline. We also discuss the limits of mindfulness and propose recommendations for future efforts.