Abstract
Since becoming evaluators, we have observed how the field of evaluation has grown and changed. Major areas of development we have witnessed include increased attention to evaluation capacity-building initiatives, diversity, equity, and inclusion efforts, as well as demands for more adaptive evaluative strategies and techniques for improving the quality of evaluation planning and resulting evidence. Many of these areas of development in evaluation practice are in response to increased national and global complexity and uncertainty. Although the field has evolved in response to these challenges, we recognize that there is always room for improvement. We anticipate ongoing complexity and uncertainty as contemporary political, social, economic, and environmental shifts take place in our world. As such, we desire to push the field toward a more inclusive, adaptive, restorative, and effective evaluation praxis.
This desire led us to assemble evaluation scholarship for this first issue of volume 44 in the form of five articles, a commentary, and a section on experimental methodology, including three articles. Separately, the articles in this issue extend the field of evaluation's development in the areas of evaluation capacity building (ECB), responsive and equity-oriented efforts, vigilant evaluation practice, and effective methodology. Collectively, the articles address the growing complexity of our world, providing insights and techniques to build better practices and methods for a brighter future.
In the first article, Gregory Phillips II, Dylan Felt, Esrea Perez-Bill, Megan M. Ruprecht, Erik Elías Glenn, Peter Lindeman, and Robin Lin Miller propose an evaluation orientation that is responsive to the LGBTQ+ community's interests and needs. They abbreviate into LGBTQ+ individuals who identify as lesbian, gay, bisexual, transgender, queer, intersex, and Two-Spirit, inclusively along with other sexual and gender minorities; and they consider the intersectionality of these identity traits with those who are “also Black, Indigenous, and People of Color (BIPOC), those who are disabled, and those who are working-class, poor, and otherwise economically disadvantaged, among other intersecting forms of oppression.” Informed by a review of the current literature on evaluations with LGBTQ+ populations and other inclusive evaluation frameworks, the authors suggest how the field can further transform LGBTQ+ evaluation, in the following areas for priority action: (1) field standards, guidelines, scholarship, and pedagogy; (2) evaluator reflexivity and subjectivity; (3) demographic data collection and interpretation; (4) community engagement; (5) liberation and becoming an accomplice; and (6) remembering that you are not alone. To take immediate action, the authors highly recommend reaching out to the American Evaluation Association's LGBT Issues Topical Interest Group.
In the second article, Isabelle Bourgeois, Sebastian Thomas Lemire, Leslie Ann Fierro, Ann Marie Castleman, and Minji Cho describe the evolution and current state of ECB theory and practice. Their review of the literature offers common strategies and novel techniques for applying ECB. Additionally, they discuss the extent to which these techniques led participants to apply their evaluation skills and organizations to change their evaluation practices. Their work is particularly insightful as it thoughtfully connects organizational culture and evaluation culture and lays the foundation for future areas of scholarship on ECB.
Authors Ayesha S. Boyce, Tiffany L.S. Tovey, Onyinyechukwu Onwuka, J. R. Moller, Tyler Clark, and Aundrea Smith observe that evaluators have increased attention on culture, diversity, inclusion, and equity (DEI); yet, how these issues are conceptualized and measured remains a challenge in evaluation practice. How are evaluators currently defining and measuring DEI in their evaluation practice? Through examination of NSF-funded projects, the investigation sheds light on the specific difficulties evaluators and PIs have when measuring DEI. A key contribution of this piece for the future of evaluation, from our perspective, is the clarification provided on how diversity, equity, and inclusion ought to be defined in evaluation practice.
Next, although complexity and uncertainty are prominent issues faced by the field of evaluation, often overlooked is the role of “vigilance” in evaluation practice to tackle these twin issues. In the fourth article in this issue, Rob Goble, Edward R. Carr, and Jon Anderson consider the role of vigilance in the practice of monitoring, evaluation, and learning (MEL). The authors contextualize the discussion by highlighting how vigilance in MEL can contribute to adaptive management in international development. They delineate two types of vigilance and offer guidance on how pilot projects are useful to gain real-world experience learning, building, and applying vigilance.
We expect that the fifth article in this issue will challenge many American Journal of Evaluation (AJE) readers; and for this reason, we have invited comment to accompany it. Jennifer A.H. Billman introduces “ontologically integrative” evaluation with a framework that suggests ontological competencies and concepts. Overall, the competencies and concepts are intended to serve as a bridge between western and non-western evaluation theory and practice. The framework encourages the interrogation of ontological assumptions during each phase of the evaluation (i.e., data collection and reporting). The framework may assist evaluators in moving from cultural competence to ontological competence, which can contribute to future advancements in reflective evaluation practice.
Long-time development evaluation scholar and practitioner, Robert Picciotto comments on Billman's framework. He argues that the framework fails to address a host of philosophical issues and therefore fails to “frame evaluation in reality” as the article's title proclaims. Picciotto builds his argument by stating that Billman's framework does not sufficiently address the neoliberal forces that influence contemporary evaluation practice and challenges the adoption of an ontologically integrative framework. We value the perspectives of both Billman and Picciotto and encourage AJE readers to read their arguments closely and decide for themselves the extent to which they agree with the points raised by these two AJE contributors!
The final portion of this From the Co-Editors Note foreshadows the topics that motivate this issue's Experimental Methodology Section, and that From the Section Editor Note elaborates further. In brief, although a somewhat narrow topic, experimental evaluation methodologists commonly face concerns over statistical “power,” the “minimum detectable effect,” and the sample sizes needed to generate useful evidence from evaluations. The trio of articles in this issue's Experimental Methodology Section offers concrete guidance for how better to design experiments when study participants are nested within groups (Cox & Kelcey; Hedberg) or when funders face tradeoffs between nationally representative impact estimates versus those from a selected group of study participants (Bell et al.). We are hopeful that these articles’ insights will go to immediate use in the planning and execution of experiments.
In sum, we are pleased to have been able to curate this issue for AJE's readers. The diversity of perspectives and quality of research and thinking reflected bode well for our evolving field. We contend that the field of evaluation will probably never reach a “steady state” in its theory, methods, and practice; there will always be room for improvement with the need to build theory, methods, and practice to lead to a collective brighter future. The problems that evaluators examine are wicked, existing in a complex and dynamic world; as a result, the challenges that evaluators face are thorny, demanding responsive methods and practices. The collection of articles in AJE 44(1) intends to provide useful guidance via the applied examples of how to formalize the issues to aid in decision-making and evaluation practice. We hope these insights and examples illustrate evolving scholarship and practice and compel thought and debate that will in turn improve the work we do in service of a better world.
