Abstract
The field and practice of evaluation seem to be in a growth phase as of late. Data suggest that the formal and informal systems that prepare evaluators for services have been expanding, as evidenced by a rise in the number of advanced degrees offered that specialize in evaluation, growth in the number of national and international professional evaluation societies and affiliated membership, and substantial demand for evaluation services (LaVelle & Donaldson, 2015). Concurrent with this growth, more universities are offering certificates of advanced study in evaluation than ever before, and both universities and other groups (e.g., American Evaluation Association [AEA], The Evaluators Institute, Minnesota Evaluation Studies Institute, among others) offer short trainings related to evaluation skills, practice, and theory. The demand for evaluation knowledge and skills at the individual level and for the capacity to make quality evaluation and its uses routine at the organizational level create an exciting opportunity for teachers of evaluation to ply their craft in a range of settings. An ongoing challenge for teachers of evaluation, however, is the relative lack of compiled and vetted resources to help deliver high-quality, engaging, and memorable learning experiences. The second edition of Building Evaluation Capacity: Activities for Teaching and Training, by Preskill and Russ-Eft (2016), is a welcome resource in the domain of evaluation teaching and training.
Overview of the Book’s Structure and Content
The book presents 89 activities in sufficient detail for users of the resource to adapt them to meet their own contextual needs and learning objectives. The book is sectioned into familiar topical areas: Overview of Evaluation; Politics and Ethics of Evaluation Practice; Culture and Evaluation; Focusing the Evaluation; Evaluation Models, Approaches, and Designs; Issues of Validity and Sampling; Collecting Evaluation Data; Analyzing Evaluation Data; Communicating and Reporting Evaluation Processes and Findings; Managing the Evaluation; Building and Sustaining Support for the Evaluation; and Reflections on Learning. Each section begins with a brief overview of its content, followed by the main points covered in each of the activities, and a list of additional resources. The section then transitions into detailed instructions for the activities, which include instructional objectives, minimum and maximum number of participants, time estimates, materials needed, classification of the instructional method (e.g., small group work, debate, case scenario), and procedures. The number of activities per topical section ranges from 4 (Managing the Evaluation; Building and Sustaining Support for the Evaluation) to 16 (Collecting Evaluation Data).
For authors creating revised editions of activity-specific handbooks, I imagine it might be a challenge to resist the temptation to simply add more activities to the work and call it a day. In this case, however, it seems that the authors successfully avoided this temptation, and they have diligently revised and restructured the activities. The result is a resource that is both familiar and new. Those familiar with the first edition of the book (Preskill & Russ-Eft, 2005) will recognize many activities in the second edition, although most of the “classic” activities have been treated to some revision and updating. Indeed, according to the authors, only 12 activities in the second edition did not receive any modification or update, and 6 activities were removed entirely (p. xiii). In addition, the authors solicited activity ideas from colleagues via AEA's EVALTALK listserv, which resulted in 24 new activities being included in the new edition. The new and revised activities fit well with the classic activities from the first edition, and I would have found it difficult to identity new activities from old were it not for “source” footnotes that are provided for many of the activities, giving credit to the contributor. To me, the source footnotes give the book more of a sense of community ownership, assigning the authors a more curatorial role, which helps reinforce the idea that building evaluation capacity is everyone’s responsibility and (hopefully) a source of fun.
Many of the activities are fun, but they are also informative, challenging, and anxiety reducing, each in its own turn. For example, novice evaluators are often struck by the difficulty in helping stakeholders move their implicit criteria about an evaluand into a description that could lead to understanding, descriptive outcomes, measurement, and analysis. Queue up the popular chocolate chip cookie exercise (Activity 4, p. 17), wherein participants discuss in a small group what makes a chocolate chip cookie “good.” After lively discussion about what criteria to use (e.g., firmness, caloric value, smell, chocolate to dough ratio) and how to measure performance for each criterion (e.g., a 5-point scale of hard to soft), participants are invited to taste a number of different cookies (comedic instructors can make comments about data sampling at their own risk) to see for themselves how the cookies perform. Participants then summarize their findings on each criterion and make two recommendations: one for which cookie performed best on their selected criteria and one for which one they liked best. This can lead to fruitful discussions about imprecision in measurement, how to negotiate perspectives within a stakeholder group, and how to negotiate perspectives between stakeholder groups (e.g., a group of consumers vs. a group of cookie manufacturers). Overall, this activity helps to translate evaluation from a nebulous concept into a practical tool that most human beings use subconsciously throughout every day. By beginning with nonthreatening cookies, the learners can scaffold to more serious topics such as the programs that seek an evaluator’s aid. I often conclude this very enjoyable activity with a semi-rhetorical question: When is a cookie not a cookie? The answer is, of course, when a cookie is a proxy for the health, well-being, and functioning of another human being.
In creating this resource, the authors draw from a range of experiences in teaching and practicing evaluation and in facilitating adult learning. This expertise and attention to the needs of both instructor and learner is evident in both the large themes and small details of the book. For example, they present a matrix of activities (p. xviii) that succinctly displays the topic of each activity, cross-listed with primary instructional method, time estimate, and whether or not background information is necessary for the activity to be successful. I have found this table very helpful in planning class sessions and workshops alike, especially when instructional time is at a premium. This table has also been helpful in guiding me to rotate intentionally between different instructional styles, so that my learners stay engaged, and so I don’t become reliant on one or two instructional techniques. The matrix is followed by a very succinct glossary of instructional methods.
Critical Reflections
As a teacher and scholar of evaluation, I have enjoyed teaching evaluation concepts and skills to hundreds of fantastic learners, including working professionals, graduate students, and, yes, even undergraduate students. It is a pleasure to finally have a book that helps me share my enthusiasm about evaluation with my learners using field-tested methods. As with any resource, however, there are some suggestions for consumers and users of this book to consider.
First, I believe the book was designed with utility in mind and positioned as a go-to resource for implementing evaluation education activities in a range of contexts for a diverse set of learners. However admirable that goal, the utility of the book is currently compromised because one can only reproduce the activity handouts by either photocopying pages directly from the book (which doesn’t look very nice on the copiers I’ve used) or recreating them with word processing software. This is doable, of course, if one has the time and dedication to reproduce the handouts or someone who can help with the task (e.g., an assistant, student worker, supportive partner, understanding children). I hope that future editions of this text will include access to a repository of editable handouts—either CD based or online—that can be lightly modified based on individual instructional needs.
A second consideration is that Building Evaluation Capacity really is what it sets out to be: a handbook of activities for teaching evaluation and building evaluation capacity. The book is not a primer on adult learning theory or instructional methods. However, I found myself wanting more in-depth information, so that I could make better informed decisions about which technique(s) could be useful with diverse learners in different contexts. I hope that future editions of this text will include a more in-depth treatment of adult learning theories or at least an annotated bibliography. For more information on adult learning theory, and both advantages and limitations of each of the instructional techniques, I recommend seeking out additional resources such as Klatt’s Ultimate Training Workshop Handbook (1999).
A third consideration might go beyond the intended scope of this particular book, but given my current role as a university-based teacher of evaluation, I would be remiss if I didn’t mention it: Research on evaluation receives no attention in this book. It is possible that this is intentional, given the focus on building evaluation capacity in organizations as a matter of professional practice (p. xv), but I feel this is an opportunity missed for both the authors and the profession. Surely, there are activities that teachers of evaluation use to help their students explore “what works” in evaluation practice, to help students and practitioners conceptualize evaluation as an organizational intervention (thus leading to researchable questions about outcomes), or to help them develop questions about the evaluation profession that can be answered with good evidence. If conducting strong research on evaluation is challenging, then including it in a resource such as this is a good place to start normalizing its presence and importance for evidence-based practice.
An enhancement that would be helpful to add in future editions would be a classification of the activities as basic, intermediate, or advanced, perhaps also including information on ways that the activities might be modified based on the participants’ skill level. For example, the chocolate chip cookie activity (Activity 4) is a favorite of mine for introducing the basics of evaluation logic to my students. In my experience, the activity can be made more advanced if the instructor includes one cookie into the mix that does not have chocolate chips (e.g., red velvet with cream cheese flavor chips). This can lead to discussions about what happens when a program being evaluated (represented in the exercise by chocolate chip cookies) cannot be evaluated using the same criteria as the other programs or discussions about operational definitions (e.g., of “chocolate chip,” “cookie,” “chocolate chip cookie”). I have found that after the groaning “aha” realization dies down, the students look at their cookies/programs with a more critical eye and begin to wrestle with a common challenge in evaluation practice: How do you discuss reasonable outcomes with the client when the program characteristics are not aligned with the desired outcomes? My sense is that many of the activities described in Building Evaluation Capacity could be modified in creative ways to make them either more or less challenging to match a diversity of audience expertise levels.
In spite of these considerations and wished-for further enhancements, Building Evaluation Capacity: Activities for Teaching and Training has a permanent home on my professional bookshelf. It’s the resource I use to show new teachers of evaluation, research methods, and organizational development how much fun teaching and training can be for everyone involved. It’s also my go-to book when a novice evaluator or community partner reaches out to me with an issue and my initial reaction is the mental question: “You want me to explain what?”