Abstract
Though several excellent literature reviews and research syntheses have been conducted, and thoughtful frameworks and models have been proposed, I believe it is time for the evaluation field to tackle the “hard stuff” of evaluation capacity building (ECB). This entails engaging staff in ECB activities, building the evaluation capacity of leaders, focusing on learning transfer, and evaluating ECB efforts. In this brief article, I describe these four challenges and pose questions for both practitioners and researchers to consider and act upon.
Keywords
Within the last decade, interest in, and efforts to build evaluation capacity of non-evaluators, has grown exponentially. The evaluation field has developed models and frameworks, synthesized the research literature, and has engaged many in conversations around evaluation capacity building (ECB) practice. As a result, we have developed a fairly robust knowledge base and common set of understandings about what constitutes effective ECB. And, perhaps not surprisingly, there is a good deal of agreement about the construct, goals, objectives, contextual variables, challenges, and opportunities for building evaluation capacity within organizations.
I believe it is now time for evaluators to engage in the “hard stuff” of translating our collective wisdom into better and deeper practice. In particular, we need to focus on ensuring that our ECB efforts make a difference—that they reach the right people, that they are designed and implemented for learning transfer, and that ECB activities are evaluated for their effects, influence, and impact. To complement my colleagues’ excellent research and thinking that is reflected in their articles, papers, and in this forum, my hope is that we begin to consider the following practical challenges and questions, both through our research and practice.
Challenge #1: Engaging Foundation and Nonprofit Staff in ECB Efforts
While their primary responsibilities often lie elsewhere, it is critically important that program staff, in particular, understand evaluative thinking, and that they have some level of evaluation knowledge and skills. However, we have learned that these staff (a) are continually pressed for time, (b) often get mixed messages about the importance of engaging in evaluation, (c) seldom see others using evaluation results, and (d) rarely use evaluation results for their own programmatic or strategic decision making. Furthermore, as Labin, Duffy, Meyers, Wandersman, and Lesene (2012, p. 320) report, staff turnover is a significant issue for building internal evaluation capacity, which further mitigates the “stickiness” of ECB efforts. These observations suggest the following research and practice questions the field might explore:
What specific activities, practices, and behaviors contribute to building a culture of inquiry?
What does it take to sustain evaluative thinking and practice?
How can we more effectively build the evaluation capacity of program officers and nonprofit staff—what would this look like? What knowledge, skills, and attitudes do they really need?
How can we more effectively embed evaluation within nonprofit organizations that tend to have high rates of turnover?
What specific techniques, strategies, and practices will enhance the use of new evaluation knowledge and skills? Or, is this even impossible, without changing organizational culture?
Challenge #2: Building Organization Leaders’ Evaluation Capacity
I do not think we have paid enough attention to the role senior leaders play in organizations, and how they influence, shape, and sustain an evaluation and learning culture. It is not enough to train, educate, and engage staff in ECB activities; leaders must also learn to think evaluatively and support meaningful evaluation practice within the organization. As Labin et al.’s (2012) research has shown, “Leadership was the least frequently targeted organizational factor and the least frequently reported organizational outcome” (p. 321). Yet leaders have the power and responsibility for supporting evaluative thinking and practice. They can do this by: (a) understanding how strategy and evaluation are interconnected (or should be), (b) providing adequate resources for evaluation, (c) being active consumers of evaluation information, (d) building their Board’s understanding of and support for evaluation, and (e) using evaluation as a means for ongoing organizational learning. These observations suggest the following research and practice questions the field might explore:
What do leaders and board members believe about the value of evaluation? What are their conceptions of evaluation and how have these been formed? How and what data do they use for decision making? Have we learned anything new about what inhibits organization leaders from using evaluation findings?
How would organization leaders and board members like to learn about evaluation practice? What would it take to build and sustain their support? At what depth should they learn about evaluation?
Does ECB for organizational leaders look different than that for program staff?
What strategies and tools are most effective for building leaders’ evaluation capacity?
Challenge #3: Focusing on Learning Transfer—The Missing Link
Even when evaluators engaged in building evaluation capacity design and implement well-thought, practical, and relevant evaluation activities and experiences, we know little about what happens to ECB participants once their involvement ends (especially for short term engagements such as workshops or being engaged in an evaluation process). Yet, if we really want evaluative thinking and practice to become embedded in individual and organizational practices, then we must also be acutely attuned to and focused on developing tools, processes, and understandings about how new knowledge and skills are transferred to the everyday work of program staff and leaders. This involves considering the readiness of participants, their motivations and expectations, organizational conditions, opportunities they may or may not have to use their new evaluation knowledge and skills, and the extent to which leaders encourage, coach, support, and resource their evaluation activities (see Preskill & Boyle, 2008). These observations suggest the following research and practice questions the field might explore:
What does transfer of learning of ECB knowledge, skills, and attitudes look like? How much transfer is good enough?
What are the key factors that support or inhibit participants’ ability and willingness to transfer their evaluation knowledge, skills, and attitudes? Under what conditions is transfer most likely to occur?
Which ECB strategies are more likely to lead to effective transfer of learning (e.g., workshops, written materials, coaching, and technical assistance)?
What kinds of evaluation related tools, activities, or processes are most effective with organizational leaders and board members?
To what extent is it more effective to engage leaders in ECB with other leaders, or with their program staff (or other)?
Challenge #4: Evaluating ECB Activities—The Elephant in the Room
While one would think that evaluators would be interested in evaluating the effects and impact of their ECB work, Labin and her colleagues (2012) found that only “about half the cases reported some evaluation of their ECB efforts…” (p. 317). In truth, this is not particularly unexpected. Many evaluation capacity projects are scoped as one-off workshops, or involve a set number of hours for technical assistance, or rely on written or website materials, and rarely involve a contract for evaluating the results of these efforts (though training workshops often include a post-training survey). As a result, the field has little evidence of the effectiveness or overall impact of ECB, and from a design perspective, we know little about how to choose among the various ECB strategies (e.g., for whom, how much, how deep, when, how long)? These observations suggest the following research and practice questions the field might explore:
To what extent has research provided the field with a sufficient set of outcomes and indicators to use for this purpose? What else is needed; what is missing?
Should we be asking if ECB is worth it? If yes, then for whom? At what cost?
What are evaluators’ obligations for evaluating ECB efforts?
Who should conduct evaluations of ECB efforts?
What approaches and methods would be most meaningful and feasible for judging the effectiveness and impact of ECB activities?
While it is heartening to see so much interest in building the evaluation capacity of a wide range of individuals, I believe we have much more to learn about how to make ECB efforts effective and long lasting for those most in need. I am also increasingly wondering what it is we want people to know about evaluation, and what is it we want them to actually do with their evaluation knowledge and skills. Have we sufficiently considered who is best suited for ECB efforts? And, more importantly, have we adequately addressed the issue of organizational readiness for supporting and sustaining ECB efforts? I am grateful to all of those who have developed frameworks and models, and have conducted rigorous syntheses of the literature. But, I believe it is now time for the hard stuff—to ask the more complex and challenging questions about ECB. By doing so, we will continue to enrich the field’s research and practice, as well as the organizations with whom we work.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.