Abstract
Enhancing Evaluation Use: Insights from Internal Evaluation Units, edited by Marlène Läubli Loud and John Mayne, falls within a whole category of contemporary contributions in evaluation that also include moving from one-off evaluations to streams of evaluative information (Rist & Stame, 2006), mainstreaming evaluation, evaluation capacity building (Cousins & Bourgeois, 2014), making evaluation policies, creating organizational learning out of evaluation (Preskill, 1994), and integrating evaluation into organizational cultures. Altogether, we can call this trend the organizationalization of evaluation.
This trend grows out of a contextual understanding of evaluation and, more specifically, an attention to the effects of organizational structures, cultures and processes upon how evaluation is shaped and used. Organizationalization of evaluation means making evaluation systematic, mainstreamed, and managerially relevant. However, organization theory reminds us that organizations have a dark side: power, bureaucratization, formality, standardization, and institutional self-preservation. Good evaluators must take into account the organizational factors that shape systematic evaluation, and they must engage with the question of how to find a place for good evaluation in a complex world defined by organizations.
The particular contribution of this book can be characterized in terms of its perspective, its themes, and its promise. The perspective is defined by the views and experiences of those who work inside (or for) the bodies that commission evaluations—typically evaluation units, oversight services, and so on. The editors argue that we need these accounts to “demonstrate their initiative and innovative thinking in rising to the challenges they face in sometimes complex settings” (p. vii). Let me add that, as a reader, I feel reassured that the editors and authors collectively are experienced and competent. They are the type of people who work in interesting places, who write and teach, and who are awarded by their evaluation associations. I do not hesitate to describe them as insightful, but they are not independent or fully objective in their analyses of those organizations—nor do they claim to be. In fact, they are reporting on their own evaluative work and how it is received in their organizations: sometimes with less than total enthusiasm or even with resistance and therefore with less than perfect results. There is inherent ambiguity concerning loyalty and attachment here. Some authors talk about “our evaluation,” others are a bit more detached. I recommend reading “about the editors” and “about the contributors” along with each chapter to fully understand the character and duration of the author’s relation to the organization in focus, and whether the two are still connected. The present role and affiliation of the author may also serve as an interpretive key to understanding the author’s perspective.
The themes addressed by the authors include the independence of evaluation, the competencies of evaluation commissioners, the institutionalization of evaluation, the structural and procedural steps that enhance evaluation and its use, and the organizational culture necessary in this regard. These issues, states editor Mayne on p. 1, have often been described by outside observers, experts, and academics with a built-in bias in favor of “bad news,” while articles on successful evaluation in organizations are much harder to find. I read this as a promise that the book will deliver good news about the organizationalization of evaluation. However, most of the chapters testify to difficult, long-term endeavors in organizations, and, according to most authors, after many years, the organization in focus is, at best, between three and four on a five-point scale of evaluation maturity. The perfect evaluation culture is like the horizon. We never get there. Optimism? Hmm.
In addition to an introduction and a concluding chapter, the book includes substantial chapters that typically summarize the experiences of the author from one particular organizational setting or from a career across organizations. Empirical examples are drawn from authorities in New Zealand, Switzerland, Scotland, and Canada, and from three large international organizations (the European Commission, World Health Organization, and the International Labour Organization). Some of the chapters exemplify innovative approaches to evaluation policies and to ways of engaging with stakeholders in interpreting and using evaluative information, describing what happened, why, and with which difficulties. Other chapters move quickly forward to prescriptive bullet points.
The chapters that work best for me are those that clearly and directly suggest a conceptual theme before moving to the report from the organization in focus or to broad recommendations based on personal experiences. I shall highlight but three of the conceptual insights I found most stimulating.
In Chapter 2, “Evaluator, Evaluand, Evaluation Commissioner,” Bastian de Laat breaks down the conventional dichotomy between internal and external evaluation. Instead, he talks about “tricky triangles” including all three partners in the chapter title, thereby revealing in particular the possible tensions between commissioners and evaluators. He describes five organizational configurations, that is, five versions of the “tricky triangle,” each of which provides its own set of possible tensions. His analysis is truly organizational in spirit and paves the way for a revision of any conventional equation of “external” with “independent” evaluation. Organizationally speaking, there is no such thing as not being dependent on anything. I found de Laat’s tricky triangles conceptually stimulating and found myself elaborating his idea further. What if we have a semiautonomous project in one of the three corners of the triangle? How many interesting permutations can we think of?
I was also enthusiastic when I read Erica Wimbush’s chapter about attempts to enhance utilization of evaluation in Scotland. The author applies the knowledge-to-action literature, moving from classical paradigms of utilization and uptake to partnerships, systems thinking, and coproduction. While she acknowledges that different stakeholder perspectives can give rise to conflicts over the quality of evaluative information, she also sees the value of competing models of knowledge development. She uses both successful and less successful initiatives as examples and explains them convincingly. Her theoretically informed analysis is, at the same time, attentive to the importance of organizational and political factors, including policy windows.
One thing I learned from Nancy L. Porteous and Steve Montague in Chapter 6 is how they created and used an organizational level logic model (for the Public Health Agency of Canada). While the model’s overall strategic outcomes are, after all, not clearly measurable, and while it surely uses wonderful graphic tricks to appear convincing, the authors do a good job in explaining how this kind of model can be used to help employees understand how their activities fit into a larger meaningful picture. A charming observation was that while senior-level managers were discussing the merit of an organizational logic model, program staff across the agency were already using the draft model (pp. 131–132). The example shows the important role of informal organization, sometimes operating in evaluative work long before formal organization makes up its mind.
The final chapter, by editor Marléne Läubli Loud, articulates the take-home messages from all of the chapters. It looks at what can be done to secure independence and quality in evaluations, including, for example, introducing report mechanisms to identify pressures or obstructions during the evaluation process, and keeping the evaluation unit independent from line management, placing it instead in a direct reporting relationship with high authority in the organization. Complicating these recommendations, as Läubli Loud wisely acknowledges, is the recognition that all other organizational functions also want staff, funding, and close relations with executive management (p. 245). Organization-wide commitment for improving use should be secured through policies and broad involvement.
Overall, this is a good book that is neatly organized. The book can be used in teaching because the examples provide realistic flesh and blood to supplement new students’ learning of dry evaluation theory. Pedagogical discussion questions follow each chapter. Near the end of the book, I have a sense that some great dilemmas deserve further articulation and elaboration, theoretically and practically. First, are organizations really able to say goodbye to 19th-century determinism as they say hello to 21st-century systems approaches (a shift mentioned by Penny Hawkins in Chapter 3)? Am I right in seeing the tools and initiatives in the chapters spread along a continuum, one which has formal structural and procedural instruments, policies, and rules to enhance evaluation use on one end, and exemplary, collaborative and partly informal initiatives on the other? It is not easy to say: Why not both? Imposed use of evaluation through procedures that are—yes, let us say it like it is—bureaucratic and old school is not likely to be consistent with the convictions of the hearts and minds that characterize a genuine, enthusiastic evaluation culture. We are with editor Loud when she mentions (already in chapter 4) that sometimes evaluation is institutionalized in formal terms, but there is still no genuine evaluation culture. While the underlying message seems to be that culture is needed too, extra steps could have been taken in the book to explore the potential theoretical tensions between the formal and the informal aspects of organizationalized evaluation, between procedure and belief, between policy and practice. We do have good organizational theory that says that when people are forced to do things, or are paid to do so, they are not only much less enthusiastic than when they do things out of conviction, but they also engage in all sorts of unofficial behaviors that are meaningful from other perspectives, not just the official organizational one. It may not be easy to reconcile the two ends of the continuum, including both official policy and unofficial norms.
Second, how far should an internal evaluation unit go to make sure that formative evaluation takes place? Will it then eventually be involved in evaluating how well the evaluation unit has helped induce changes in programs along the way that have led to good summative results? Can the organizational integration of evaluation policy and organizational policy become too close? If a radical revision of basic organizational assumptions is needed every 25 years, will an evaluation unit living up to the book’s prescriptions be in position to initiate it?
Third, several authors suggest that evaluation units report directly to strategic top management. If independence from line management is important, is independence of the evaluation unit then also important when the evaluation unit needs to be evaluated? Is the resulting organizational fragmentation conducive to developing a cooperative spirit within organizations, which is needed to solve today’s complex problems? And are organizational fragmentation, checks and balances, and similar organizational mechanisms a good recipe in different cultures around the globe in which international organizations operate?
Finally, my deepest concern—perhaps outside the scope of this book, but let me share it—is whether confidence in evaluation has reached a new level with the organizationalization of evaluation. What I mean here is perhaps best exemplified in the changing role of evaluability assessment. I thought the classical meaning of evaluability assessment was to sift situations in which evaluation made sense from situations where it did not. In other words, sometimes evaluation is not the best answer. With increasing organizationalization come standards, procedures, and mandatory evaluation, all of which make the above distinction less relevant. Evaluability assessment is not mentioned often in this book, and when it is, its role is to ensure that projects early on live up to standards of evaluability. It is the project that needs to comply with evaluability criteria, not evaluation that asks whether the situation is evaluable.
So I have a concern about all the resistance and reluctance toward evaluation that is talked about in this book, the lack of enthusiasm, the forgetfulness about findings, the ad hoc approaches rather than the systematic ones, the inadequate dissemination of findings, the inadequate knowledge management, and the immaturity of evaluation cultures. I wonder whether there is, after all, just one good thing to say about these barriers to good evaluation? It is difficult to know, because enemies of evaluation have no face and no voice, and thus we do not know their views and their values. If we judge them by the standards of an evaluation culture, they also have no culture. It is their hesitation, inadequacy, immaturity, and deficits that define them. Their view is not the perspective of this book, and they have little voice in the chapters.
Could there be any good reason why some stakeholders have problems with the programs, views, mentalities, and interests that evaluation enhances? Could it be that resistance to evaluation is sometimes indicative of value conflicts?
Much to the credit of this book, it does not deliver on the promise of telling only positive stories about organizationalization of evaluation. In fact, it also reports on problems and tensions. The question is how we interpret these problems and tensions.
Could it be that some staff members are uncomfortable with being evaluated not on the basis of what they have achieved but rather on the basis of whether their hearts and minds are found to be consistent with an evaluation culture that is, ultimately, hard to define and in fact never fully accomplished? If an evaluation culture transports modernity and maturity in organizations, what else is transported with it?
To answer these questions, we would probably have to move to specific histories of specific programs in specific contexts, which perhaps result in specific value conflicts. From the perspective of organizational systems, procedures, and cultures that enhance evaluation, we may not be able to tackle these questions. This is, at the same time, a fascinating aspect of the organizationalization of evaluation. It is also an aspect that deserves our attention and critical afterthought.