Abstract
Empowerment evaluation is the name given by David Fetterman in his American Evaluation Association (AEA) 1993 Presidential Address to an evaluation theory and practice intended “… to foster improvement and self-determination.” Almost 20 years after the first edition of Fetterman, Kaftarian, and Wandersman’s (1996) text on empowerment evaluation, the second edition—Empowerment Evaluation: Knowledge and Tools for Self-Assessment, Evaluation Capacity Building, and Accountability—is seen by the authors as “a transformation, literally decades beyond our first voyage” (2015, p. 16).
My review of the second edition offers a conversation on context, contents, and comments: What are the past and present contexts of empowerment evaluation? What are the contents of this second edition? What are the values, worth, and merits of this edition to prospective readers?
Context, 1993
In 1993, Bill Clinton was the president of the United States, we were embroiled in the Middle East, evaluation funding had declined since a (financial) high point in the 1970s and early 1980s, and prior to Fetterman’s election as AEA president, our presidents were Michael Quinn Patton (1988), Ross Conner (1989), Yvonna Lincoln (1990), Lee Sechrest (1991), and David Cordray (1992). Qualitative and mixed methods were widely used; approaches emphasizing utilization, social justice, participation, and collaboration were launched in theory and gaining much ground in practice. The values of randomized experimental designs were ascendant in some areas, particularly in programs supported by the U.S. Office of Education. We were anticipating changes in the coming millennium, including growth in international cooperation among evaluation associations, greater attention to cultural competence, and the potential for evaluation of democratizing technologies such as the Internet.
Fetterman’s voice thus was joined with quite a few others who underscored the significance of deliberative democratic, social justice, and transformative approaches. But what a voice! This man wasn’t talking only about evaluators involving stakeholders. Fetterman was about self-evaluations, building participant/community capacity with control of all aspects of an evaluation in the hands of the stakeholders. Such empowerment outweighed, he believed, concerns about objectivity, impartiality, evaluator independence, and trustworthy empirical evidence for funders and decision-makers. Empowerment evaluation was to be of the people, for the people, and very much by the people. Our role as evaluators was to be a critical friend, a coach, and a source of technical assistance when it was requested by the self-evaluating communities and organizations. No wonder that in 1993 “conversations and arguments spilled out into the hallways” (2015, p. xi).
Context, 2015
We have been debating these claims ever since. Is empowerment evaluation a cult? A sham? Among the pseudoevaluations? Universally useful? Or, as Chelimsky saw our field, was empowerment evaluation less appropriate for the accountability function and more appropriate for knowledge, development, and improvement purposes? Fetterman’s introductory chapter invites readers to reexamine critiques by Scriven, Patton, Sechrest, Stufflebeam, Donaldson, and others as well as a comprehensive review of empowerment evaluation by Miller and Campbell (2006). Each critique was responded to by Fetterman, not a man to be intimidated, in a dialogue that over the decades seems marked by growing mutual understanding, preciseness, and respect.
By now, empowerment evaluation and its leaders have received AEA’s highest accolades, such as the Myrdal Evaluation Practice Award (awarded to Fetterman in 1995 and Kaftarian in 1996), the Outstanding Publication Award (Wandersman in 2008), and the Lazarsfeld Evaluation Theory Award (Fetterman in 2000) as well as other recognition. Yet, there are still disagreements on the status and value of empowerment evaluation among some distinguished theorists, in contrast to the enthusiasm about the established worth of empowerment evaluation among many others. Thus, the context of the 2015 edition is the continuation of clarifications, refinements, and examples, even if the conversation has now moved far from hallway near riots toward appreciative acceptance.
Contents
The 2015 edition provides over 370 pages, including an appreciative foreword by 2015 AEA President Stewart Donaldson, a preface by the editors giving a history of their journey, and five sections, with 16 chapters in all. Part I, with two chapters by Fetterman, establishes the history of empowerment evaluation and its present breadth of adoptions. The second chapter offers a succinct 22-page presentation of theories, principles, concepts, and implementation steps. These feature the 10 principles which are crosswalked to all the case instances in the following chapters. Part II, Scope and Breadth, begins with descriptions of the value of empowerment evaluation, as seen by two influential foundation personnel (Janice Yost of the Health Foundation of Central Massachusetts and Laura Leviton of the Robert Wood Johnson Foundation). The section continues with stories of capacity building in an Aymara women artisans’ organization in Peru and in teachers as evaluators in the public schools of Melbourne, Australia. In the United States, corporate applications are detailed in Fetterman’s chapter on empowerment evaluation in the US$15 million Digital Village project supported by Hewlett-Packard and by Pamela Imm and her colleagues in a SAMSHA Service to Science project.
Part III adds tools in the shape of checklists and self-assessment forms. Wandersman (Chapter 9) describes his 10-step Getting to Outcomes (GTO) framework, Ivan Haskell and Aidyn Iachini (Chapter 10) discuss using such checklists in a “No Excuses” capacity building project for school social workers, and Regina Day Langhout and Jesica Siham Fernández (Chapter 11) present an empowerment evaluation carried out by fourth- and fifth-grade students, again applying the Wandersman 10-principle checklist. Continuing the spotlight on building evaluation capacity, Yolanda Suarez-Balcazar, Tina Taylor-Ritzler, and Gloria Morales-Curtin (Chapter 12) used the GTO checklist in an organizational transformation initiative, and Andrea Lamont, Annie Wright, Wandersman, and Debra Hamm (Chapter 13) adapted the Quality Implementation Process and Tools in implementing a quality program at large scale. Fetterman and his associates wrap up this almost 200-page section with a recap of a 10-year tobacco prevention initiative, focusing on evaluation capacity and use of the tools. All chapters in this section seem instances of what Miller and Campbell (2006) categorized as structured guidance, and what Stufflebeam and Scriven might recognize as key evaluation checklists. The defining feature is that these are self-evaluations that incorporate reflective, guided critical thinking. The checklists are made available to the community evaluation teams to select (or not), apply, and adapt to the questions they are exploring.
Part IV (Research and Reflections) considers GTO, particularly evidence of results of empowerment evaluation and evaluation capacity building in practice. The chapter by Matthew Chinman and colleagues describes six quasi-experimental studies in what could be considered an empowerment evaluation style of meta-evaluation (Chinman, of RAND, and Wandersman have been program codevelopers and coauthors of many articles on GTO). In Part V (Conclusion), Fetterman, Wandersman, and Kaftarian “reflect on emergent themes and next steps.”
Thus, the contents of this book. Turning now to my comments, I consider what the second edition indicates about empowerment evaluation in 2015 and beyond.
Comments
At first reading, the structural similarities between the 1996 and 2015 editions might suggest that owners of the 1996 edition do not need to buy the 2015 edition. For example,
The same overall format is used: a theory, principles, philosophy section by Fetterman begins both books. This is followed by a papal blessing kind of section, with highly respected evaluation leaders shining the spotlight on the excellent features and wide uses of empowerment evaluation. In 1996, Henry Levin wrote on the application to the Accelerated Schools project while Ricardo Millett described enthusiastically the use of empowerment evaluation in the W. K. Kellogg Foundation framework. In 2015, as mentioned, the writers are Yost and Leviton. Next appear case instances of applications in many settings, including federal, state, and local levels and in fields such as health initiatives, HIV prevention, and African American community development. Both books then consider tools and their applications, such as (in 1996) the Plan Quality Index. Lastly, Fetterman writes the onward chapter, “Conclusion: Reflections on Emergent Themes and Next Steps.”
At second reading, however, my conclusion is: Read this 2015 edition, particularly for readers new to evaluation who want, in one place, a compendium of what empowerment evaluation is about, a statement of its principles, a set of case examples in diverse settings, and an understanding of where Fetterman, Kaftarian, and Wandersman are at now in their thinking. Equally, read this new book particularly for evaluators who have not kept up with Fetterman’s many articles, the great debates with Stufflebeam, Patton, and Scriven, and the books published by Fetterman and his colleagues between 1996 and 2015. That is, evaluators who want to debate, consider, discuss, reject, adapt, and/or adopt empowerment evaluation, as intended by Fetterman, ought to be talking about what he is saying now, not the 1996 launch.
Notable Changes
There are, as I said, notable changes. These include, for example,
Explicit, detailed positioning of empowerment evaluation as one among many approaches to be used solo in specific circumstances, adapted in others, and integrated with different principles and methods in others. For instance, Chapter 10 (“‘No Excuses’: Using Empowerment Evaluation to Build Evaluation Capacity and Measure School Social Worker Effectiveness,” by Haskell and Iachini) integrates a progress and outcome-oriented measurement system with empowerment evaluation. Chapter 14 (“Empowerment Evaluation and Evaluation Capacity Building in a 10-Year Tobacco Prevention Initiative,” by Fetterman and colleagues) blends evaluation capacity building with a RAND-led external, quantitative evaluation in a particularly detailed, dramatic narrative. Instances at large, middle, and small scales, and in health, education, technology, and governance areas, demonstrate the applicability of the principles. The largest scale instance is Chapter 7 (“Hewlett-Packard’s $15 Million Digital Village: A Place-Based Empowerment Evaluation Initiative,” by Fetterman, which also has been reported in a separate book and several articles). The smallest may be Chapter 11 (“Empowerment Evaluation Conducted by Fourth- and Fifth-Grade students,” by Langhout and Fernandez), which involves a university-initiated program, Change4Good, that was evaluated by the students in one school over 2 years. Evaluation capacity building has moved front and center as a purpose of empowerment evaluation. Capacity building now is featured in almost every chapter as well as having been added to the book’s title. This topic covers over 50 pages, with 27 appearances in the 2015 index, compared to 25 pages and one index listing in 1996. Theory and principles are explicit, explicit, and did I mention explicit? Fetterman has been criticized for obscurity, among other concerns, with regard to theory. Those charges should be put to rest after this book. He lays out the 10 central principles, their meaning, and their applications. See, for instance, p. 48, beginning with:
“Improvement: Empowerment evaluation is designed to help people … build on their successes and re-evaluate areas meriting attention.” “Community ownership: Empowerment evaluation values and facilitates community control; use and sustainability are dependent on a sense of ownership.” These tables are followed by extensive, clear, up-to-date discussions of the theory, explication of nuances, theory of change or operations, and illustrations of applications. In my opinion, one can disagree with this vision of the evaluator’s role and prefer other definitions of what evaluation is about, but the disagreements should be linked to reasons for preferring one and perhaps even disparaging the other. Ditto, with regard to getting from theory to practice in empowerment evaluation. Several critiques have faulted Fetterman for being unclear on how the principles are realized in applications, step by step. Not in this book. Each chapter giving a case instance explicitly connects the 10 principles of empowerment evaluation to the processes, decision-making, participation, utilization, and other aspects of the evaluations reported. For example, in Chapter 13 (“An Empowerment Evaluation Approach to Implementing With Quality at Scale”), Lamont and her colleagues report on implementing the 1Two1 program, involving a computer for every student in an entire school district—all 39 schools and centers, all 12,000 students and 1,000 teachers. Table 13.1 enumerates each principle, providing a thumbnail definition and a description of how it was represented in the evaluation. This is followed by Table 13.2, which details the Quality Implementation Tool action steps, Table 13.3, which summarizes the phased-in process of implementation support, 10 pages of analytic text, and an appendix that provides checklist guides for all six implementation action steps. Appreciating the small stuff: The book itself practices what it preaches in small as well as large ways. For instance, the chapter headings provide the affiliations for all coauthors, most of whom are community leaders or from participating agencies. The cover shows the same African continent on the globe as the 1996 edition. However, the 2015 edition, with the earth surrounded by diverse hands, brings to my mind the Hawaiian concept of malama aina and the Jewish concept of tikkun olam: protect, care for, save the world. A priority on dialogue: Empowerment evaluation, like many evaluation theories and practices, has been a work in progress. In 1996, Fetterman saw several priorities for unfinished business. These were, to paraphrase, (1) the useful evaluation framework must now be refined and elaborated based on practice in the field. (2) Reconciling and/or reporting disparate views need additional attention. (3) The environments in which empowerment evaluation is practiced often mitigate against standards of high quality, issues which need solutions. (4) Finally, “… the tension between improvement and accountability needs to be addressed … and ways found to resolve it more thoroughly” (1996, p. 382). In the 2015 edition, Fetterman revisits these earlier next steps, considering them largely resolved. Ahead, he sees first expansion and building, particularly with multisite work and place-based evaluations, and he notes, “Empowerment evaluation can benefit from an ongoing dialogue with similar stakeholder involvement approaches …. The next challenge, however, is to begin to think about how best to apply these approaches in various combinations within the same large-scale place-based initiatives …” (2015, p. 342–343).
Yes, but …
I agree in general with the last chapter in this new edition, but would like to see, in the third edition or sooner, what is for me a “Yes, but …” that is not unique to empowerment evaluation. Our field needs robust meta-evaluations (that is, evaluations of evaluations) of specific evaluations in practice to learn how well the tensions between self-evaluated improvement and accountability are actually resolved. In 1996, for example, bias was discussed in three places; in 2015, bias does not appear in the index (although it is discussed briefly in appendix 16A). Fetterman indicates the bias issue is largely resolved due to participant diversity and self-interest in thoroughness, honesty, and candor. More specifically, “Empowerment evaluation addresses many of these concerns by being transparent, bringing bias to the surface, and generating meaningful data to inform decision making” (p. 344). To me, this doesn’t do sufficient justice to the issues.
We need, as a field, to invest in getting evidence of the extent to which the theories are realized in practice, with what benefits—not only as proclaimed by the theorists, who generally find their babies pretty cute, but also as seen empirically by vigilant third parties, independent critical friends, who will demand evidence in addition to testimonials. I did not find such meta-evaluations among the case instances although some could disagree with my emphasis on independent third parties as distinguishing meta-evaluation from valuable critical thinking and self-reflection. As stated, the one chapter constituting Part IV describes a detailed, perhaps unique, empirical test of results and outcomes coconducted and coauthored by the RAND Corporation with Wandersman of six quite different empowerment evaluations using GTO. But page 316 notes that Getting to Outcomes and GTO are trademarks registered jointly by RAND and Wandersman’s home port, the University of South Carolina. Wandersman is the coauthor of four of the seven references on the experiments. Perhaps this is a gray area where some could consider such a decade of collaborative field testing as a meta-evaluation and others would not.
Further, we need to look systematically at direct and indirect costs. Laudably, the cases do include some reports of evaluand resistance, which were said to be overcome by making data collection easier and by helpful results as well as discussions of extra time and resources reported in some cases. An adequate report of costs relative to benefits is still in absentia, however.
Granted, these chapters are summaries of longer, larger materials. Perhaps other books and articles on these questions by Fetterman, Kaftarian, and Wandersman present data to satisfy even the most demanding meta-evaluators. Overall, however, I am struck less by what isn’t here than by what is here. Splendidly, the results of more than 25 years of dedicated hard work and diligent thought are being shared with us. My recommendation remains, unequivocally and appreciatively: This book belongs on our shelves. Get it, read it, and, as appropriate, use it.