American Journal of Evaluation

The present study explores how Black and White youth respond to measures of subjective well-being within the context of critical race theory (CRT). Three levels of measurement invariance (i.e., configural, metric and scalar) were examined for indicators of subjective well-being. We hypothesized that there would be limited measurement invariance across groups based on the premise established by CRT that youth of color experience and perceive life differently than their White peers, which was supported. The findings of this study demonstrate that the measures work as expected within groups, but there is a considerable lack of invariance across groups. This study also provides some evidence that racial/ethnic differences cannot be taken for granted when assessing SWB in youth.  

This study systematically reviewed evidence regarding health program and service evaluations in Indigenous contexts. Following the PRISMA guidelines and combining terms for ‘Indigenous populations’ and ‘health programs and services’. Eight principles emerged: Principle 1: Adopting Indigenous led or co-led approaches is vital to balance power relationships by prioritizing self-determination, Principle 2: Evaluation team should include local Indigenous community members, Principle 3: Indigenous community knowledge and practice should be foundational, Principle 4: Evaluations must be responsive and flexible to meet the needs of the local community, Principle 5: Evaluations should respect and adhere to local Indigenous protocols, culture, wisdom and language, Principle 6: Evaluations should emphasize reciprocity, shared learnings and capacity building, Principle 7: It is important to build strong relationships and trust between and within researcher teams, evaluators and communities, and Principle 8: The evaluation team must acknowledge community capacity and resources by investing in time and relationships.

Evaluative criteria define a “high quality” or “successful” evaluand and provide the basis for judgment of merit and worth, yet they are often assumed and implicit in the evaluation process. This article presents an empirically supported model that describes and integrates two aspects of criteria: domain and source. Domain identifies the focus or substance of a criterion, while source describes the individual, group, or document from which it is drawn. Developed from a synthesis of evaluation literature and empirical analysis of evaluation reports, the model defines 11 criteria domains and 10 sources and reveals the relationships among them. In this integrated model, the two dimensions can be used together as a thinking tool to guide evaluators in specifying criteria, in empirical research on the valuing process, and as a conceptual framework and language for theorists prescribing criteria selection.

Observations are widely used in research and evaluation to characterize teaching and learning activities. Because conducting observations is typically resource intensive, it is important that inferences from observation data are made confidently. While attention focuses on interrater reliability, the reliability of a single-class measure over the course of a semester receives less attention. We examined the use and limitations of observation for evaluating teaching practices, and how many observations are needed during a typical course to make confident inferences about teaching practices. We conducted two studies based on generalizability theory to calculate reliabilities given class-to-class variation in teaching over a semester. Eleven observations of class periods over the length of a semester were needed to achieve a reliable measure, many more than the one to four class periods typically observed in the literature. Findings suggest practitioners may need to devote more resources than anticipated to achieve reliable measures and comparisons.

Large-scale education interventions aimed at diminishing disparities and generating equitable learning outcomes are often complex, involving multiple components and intended impacts. Evaluating implementation of complex interventions is challenging because of the interactive and emergent nature of intervention components. Methods that build from systems science have proven useful for addressing evaluation challenges in the complex intervention space. Complexity science shares some terminology with systems science, but the primary aims and methods of complexity science are different from those of systems science. In this paper we describe some of the language and ideas used in complexity science. We offer a set of priorities for evaluation of complex interventions based on language and ideas used in complexity science and methodologies aligned with the priorities.

Photovoice is an empowering, qualitative, community-based participatory research method that engages participants to take photographs and reflect on their meanings. We employed a novel application of a Photovoice-inspired methodology to evaluate perinatal home visiting services. This article describes the feasibility and utility of remote implementation of a Photovoice-inspired project in one state’s federally funded Maternal, Infant, and Early Childhood Home Visiting Initiative. The initial project aimed to identify aspects of parenting roles/identities, goals, and home visitor–client relationship that can be leveraged to improve program engagement and retention. Training materials were sent to home visitors who acted as liaisons with parents. Reflective interviews about this photo-elicitation method conducted with parents, home visitors, and the evaluators following the project revealed that informational materials were beneficial to understanding the process. Home visitors were noted as gatekeepers and key motivators. Trainings, flexibility, timely communication, and logistics should be considered when remotely implementing Photovoice.

The profession of evaluation continues to grow, generating more demand for evaluation services than can be fulfilled by the supply of well-trained evaluation practitioners. In this brief forum article, we discuss the current state of evaluator education in the United States and the ways in which university-based programs support the development of evaluation practitioners and scholars. We then discuss the opportunities and challenges university-based programs face, focusing on program design, the challenges of being an evaluation-focused faculty member, and the specific needs of evaluation graduate students. We conclude by providing a research agenda for evaluation scholars interested in advancing the knowledge base about university-based evaluator education.

This method note presents Q methodology as a useful tool for evaluators to add to their practice toolbox. Q methodology, which involves both quantitative and qualitative techniques, can help researchers and evaluators systematically understand subjectivity and the communicability of opinions and perspectives. We first provide an overview of Q methodology, followed by a brief summary of how evaluators are using Q, and an explanation of the steps for implementing Q methodology. Either by itself or with other methods, the potential uses of Q methodology in evaluation are diverse. For practical demonstration, we describe how Q methodology was used in a recent evaluation in the UK to understand stakeholder subjectivity within the program. We then reflect upon the pros and cons of using Q in program evaluation, concluding that it constitutes a worthwhile tool for evaluating complex programs.