Book Review: The Future of Evaluation: Global Trends,New Challenges,Shared Perspectives

Abstract

The Future of Evaluation: Global Trends, New Challenges, Shared Perspectives, edited by Reinhard Stockmann and Wolfgang Meyer, joins and contributes to an important commitment of any profession that aspires to have a vibrant and relevant future, namely, to think about, analyze, speculate on, prepare for, and even plan for that future. Articles on the future of evaluation in AJE (including Evaluation Practice, the predecessor to AJE) span nearly three decades. The first that we can find was publication in Evaluation Practice of an American Evaluation Association presidential keynote on “The Future and Evaluation” (Patton, 1988). Wye and Sonnichsen (1992) analyzed and speculated on the future of program evaluation in the federal government. A special issue of Evaluation Practice was put together by Editor Midge Smith (1994) entitled “Review of the Past, Preview of the Future.” Mel Mark and Midge Smith coedited a follow-up special issue of AJE in 2001 that featured soothsaying by many prominent evaluation scholars. Other notable gazes into evaluation’s future include Donaldson’s (2013) edited volume on The Future of Evaluation in Society: A Tribute to Michael Scriven, reviewed in AJE (Mabry, 2016).

What stands out in this latest book on evaluation’s future is the decidedly global perspective that runs through the volume, including contributions from evaluators in Africa, Asia, Australasia, North and South America, and Europe. Some prominent trends are not discussed: the global evaluation network of aboriginal and indigenous evaluators and data visualization trends being two prominent examples. Nor did these evaluation futurists foresee the explicitly anti-intellectual, antiempirical, antidata, misinformation, big lie, and fundamentally antievaluation phenomenon represented by the presidential election of Donald Trump.

Review

The Future of Evaluation is an edited volume based on a 2012 international conference held on the occasion of the 10-year jubilee of the Center for Evaluation at Saarland University, Germany. The book has 24 chapters organized into six parts that focus on general topic areas such as the role of evaluation in society, global trends in evaluation as a profession, new challenges for evaluation, and perspectives on evaluation’s future, among others.

There isn’t space to review all 24 chapters, so we will provide an overview of the book while spotlighting some insights that stood out to us. (The table of contents, with sample chapters, is on the publisher’s web page, provided above.) We have written this review together to offer the perspectives of an experienced evaluator (Michael has 45 years evaluation experience) and a relative newcomer (Charmagne has been a full-time evaluator for 3 years). In what follows, we distinguish what we wrote together from our separate comments.

Evaluation Credibility

In the opening chapter, the editors note that evaluation is experiencing a “global boom and professionalization” and offer five indicators for tracking the development of evaluation:

national public demand for evaluation;

supply of evaluation as a specific service on an evaluation market;

institutionalization of evaluation as a profession;

offer of professional university-based study courses and trainings (capacity-building); and

dissemination of evaluation findings (number of evaluation studies and publications). (p. 12)

In the closing chapter, they conclude that these trends are propelling the future of evaluation. They differentiate how these trends are unfolding in North America, Europe, and the “global South” (a way of referring to developing countries). That sets up the book’s overall conclusions:

A decline of credibility would surely lead to a decline of utilization…[while] the demand for high-quality evaluations will also increase and only academic research institutes will be able to cover this. Finally, there is a trend toward popularization: evaluations will become more open, transparent and participative by using new media for consultation processes and data mining. This may lead to new forms of evaluation use and a new wave of concept development….

To sum up: the global trends for the future of evaluation are still positive, even if many pitfalls can be identified. While evaluation is steadily on the increase, this continuously produces new challenges for the integration of evaluation as a scientific, practical and politically useful endeavor. Today, the shared perspective of being one global evaluation community dominates and many different ways of doing evaluations are accepted. The tasks for the future will be more scientific research on evaluation and improved utilization in public policy. This will be a dance on the volcano—as it ever has been. (p. 357)

Michael: The editors express concern about the “risk of de-scientification of evaluation” (p. 356), which turns out to mean evaluation training not linked to and certified by universities, citing workshops sponsored by EvalPartners as an example of de-scientification. This is a classically self-serving conclusion by academics celebrating an evaluation center operating in a university. My view is that the support for evaluation in universities has proved to be tenuous, fickle, unreliable, and dependent on a small number of academic leaders (often one) whose departure typically leads to the demise of the short-lived evaluation center or program. With strong professional associations, decreasing dependence on universities for evaluation training, which universities haven’t been very good at anyway, would be a positive sign, at least in my view. Deacademifying should not be confused with de-scientification. But this confusion is the basis for their concern that “in the long term, this may cause a decline in the quality in evaluation and decrease its credibility, at least in the eyes of the broader public” (p. 356).

Charmagne: Credibility and validity are indeed concerns in evaluation but not because of de-scientification as the authors in this volume suggest. I suppose that de-scientification may be the concern of some but likely not those for whom evaluation is really intended to be used. Rather, I think the authors overlook cultural relevance and responsiveness as intricately connected to evaluation credibility and use. Indeed, I found the conclusion to be somewhat paternalistic. The only direct discussion of culturally responsive evaluation, a major trend in evaluation in the United States, is offered by Donna Mertens in Chapter 16. Mertens describes how discussions of credibility need to consider social justice and ask the question, “how do voices of marginalized communities enhance our understandings of credibility?” To me, this is an important trend in evaluation that does not seem to be captured in the conclusions drawn by the editors.

Constitutive Effects of Evaluation

Dahler-Larsen (2012) is author of the insightful book, The Evaluation Society (for the AJE review of that book, see Julnes, 2015). Dahler-Larsen’s chapter in this volume, on the “Changing Role of Evaluation in a Changing Society,” discusses trends in evaluation thoughtfully, including an aspect not given much attention: constitutive effects.

An evaluation regime has constitutive effects when people under evaluation respond to it in such a way that a new reality is created, for example because a good score on evaluation criteria becomes a goal in itself…The basic idea in constitutive effects is that sometimes it is difficult to achieve clarity and consensus about what should be measured, how and why, but once the measurement system is in place, people react to it, and some react to these reactions, and a new reality is created. There is no guarantee that such effects are in line with “intentions,” “goals” or “improvements,” but they are part of reality anyway…

[T]the likelihood of constitutive effects has increased as a result of more emphasis on routine-based, mandatory evaluation machines in the broader context of the audit society…Perhaps non-utilization is no longer the main problem. Perhaps we should direct our attention toward complex, unintended and constitutive effects of evaluation systems. (pp. 34–35)

Michael: Using a systems perspective to examine and understand the multiple, emergent consequences and effects of evaluation will mean moving beyond treating issues like nonuse, misuse, overuse, inappropriate use, symbolic use, and constitutive effects as discrete and separate phenomena. Instead, we should examine how such diverse uses and effects occur together and interact as part of the power dynamics that permeate evaluation systems. An evaluation isn’t just used, ignored, or misused. All or parts of an evaluation can be used, ignored, and/or misused simultaneously by different stakeholders.

Charmagne: I appreciate Dahler-Larson drawing attention to constitutive effects and agree that it is worth looking at how constitutive effects of evaluation systems influence the systems in which they are working. But I think there is a broader social justice issue not addressed here— that is, who has the power to make decisions about what gets evaluated, when and how. This connects to the concept of “popularization,” discussed in an earlier part of his chapter in what he sees as a positive trend of “bringing evaluation to the people.” That said, I would have liked to see him take the discussion one step further—to address whether evaluation (or evaluators) should be more active in promoting evaluation use toward equity rather than just hoping for positive constitutive effects to take place. A related question concerns what responsibilities evaluators may have to attempt to mitigate negative constitutive effects, when and if they emerge.

Sustainable Development

The role of evaluation in sustainable development must be part of any serious analysis of the future of evaluation. Andre Martinuzzi and Wolfgang Meyer provide such an analysis in Chapter 6, “Evaluating Sustainable Development in a Global Society.” They explain the origin of the term sustainable development in deliberations of the United Nations and then turn to the challenges for evaluation.

It is not so easy to find a shared understanding and interpretation of sustainable development. Even when it comes to the academic viewpoint, differences in understanding and in handling the concept of sustainable development are commonplace. There is only one consensus among sustainability researchers—that of the broad variety and different connotations and definitions of sustainable development. On one hand, this vague and diffused understanding must be seen as a weakness…Sustainable development is not a clear code of practice—this makes it difficult to monitor and evaluate…

On the other hand, the openness of the concept can also be seen as its specific strength. While many different aspects can be subsumed, people with various political, religious, scientific and other backgrounds and beliefs are able to unite under the flag of sustainability. In a variegated and heterogeneous world, an integrating concept cannot be simple, clear and straightforward. The concept of sustainable development has proved its strength as an integrating power for more than three decades now, since the mid-1970s, and almost no fundamental critics exist. This brings about some consequences for the monitoring and evaluation of sustainability: while it will be obviously very difficult to develop globally shared criteria on what sustainability is and what it is not, it seems to be possible to assess the ability of sustainability to bring about global integration. This can be a vantage point for monitoring and evaluation. (p. 83)

Charmagne: As a newcomer to evaluation and to motherhood, I am glad to see sustainable development receiving such prominent placement among the priorities of the future of evaluation. Indeed, sustainability is a critical issue for the future of our civilization not just for the future of evaluation. I would have liked to see some inclusion of the work and voices of indigenous evaluators, who are, in many ways, leading the charge on this issue globally (e.g., Anderson et al., 2012; Wehipeihana, McKegg, Thompson, & Pipi, 2016).

Evaluation Embedded in the Political Culture of a Society

The editors synthesize lessons based on thematic analysis of the contributors’ chapters. One such lesson concerns embedding evaluation in political culture.

Evaluation as a service for decision-makers must be embedded into the political culture of a society. Without the demand of deciders, no evaluations would be commissioned and, even if some evaluations would be done on behalf of other people, these evaluations would be senseless because of the lack of utilization…

[T]he utilization of evaluation needs a specific political culture. First, the society and its political elites must be willing to learn from its own mistakes in order to increase the effects of interventions. However, the history of mankind reveals this as a very rare attribute of political leaders .…(p. 329)

Charmagne: I wish the authors would look at whether there might actually be demand from those implementing sustainable development projects, programs and initiatives if our world were more learning-focused. The top-down way in which decisions about development are still made reinforces the notion that evaluation will never happen unless those in power commission it. But what about evaluative thinking? What about learning? I would have liked to see more consideration of an embedded approach to evaluation that makes it relevant and useful without being dependent on the whims of leaders.

Cyber Society and Evaluation

Frans Leeuw, one of evaluation’s pioneering thought leaders, peers into the future in a chapter provocatively entitled “Cyber Society and ‘Rulejungling’: New Challenges for Evaluation in the 21st Century.” His chapter includes consideration of “big data” in the Future of Evaluation. He examines the effects of the internet (“web society”) and transnational governance, both of which are dimensions of globalization. “Rulejungling” refers to the phenomenon of rulemaking becoming less formal, more private, and more competitive in complex, dynamic, and transnational systems; situationally specific contractual relationships replace universal legislative mandates. For example, “the digitization of society has not been steered by central governments. On the contrary, had the internet been under the guidance of governments, it would probably not have been as successful as it is now” (p. 267). He posits that these developments “set an agenda for evaluation studies” because, in part, they involve different theories of change and may require innovative approaches to evidence.

Michael: These emergent developments pose interesting challenges for the evaluation profession in the future, both from a methodological perspective and from a utilization perspective. Who will be the commissioners and users of evaluations of digitization, rulejungling, and transnational governance, and how will evaluators develop the conceptual and methodological capacity to carry out such evaluations?

Charmagne: Agreed that the evaluation world may not be thinking in these terms, but I think this chapter lends a useful perspective on how evaluators might be able to be proactive rather than reactive. It makes me think of airport security in the United States—always seemingly one step behind the latest threat. There was a shoe bomber, so now everyone has to remove their shoes. There was a liquids threat, so no liquids over 3 oz. became the rule. I’d love to see evaluation be proactive, anticipating needs, and developing strategies and approaches that are ahead of their time rather than waiting to see what occurs and then reacting.

Global Perspective and Blue Marble Evaluation

The book is especially forward-looking and futuristic in examining the implications for evaluation of globalization and taking a truly global perspective. Pablo Rodriguez-Bilella and Maria Alejandra Lucero contribute an insightful and important chapter on “Evaluation As a Global Phenomenon: The Development of Transnational Networks.” Reinhard Stockmann, in his chapter, “The Future of Evaluation in Modern Societies,” exemplifies the global perspective.

[T]he globalization of evaluation will intensify in the future because global issues (for example climate change, financial crisis, migration) will become increasingly significant. Such issues ignore national boundaries and make joint action for solving these problems essential. This will mean new tasks for evaluation. The boundaries imposed by national evaluation cultures need to be overcome, the way for transnational joint evaluations must be cleared and the functions of evaluation need to be redefined…

[T]he economic rise and increasing political significance of emerging countries (particularly China, India, Brazil and South Africa) are contributing to the complexity of international politics and extending and complicating transnational relationships. A new world order with a multipolar power structure is coming about, with some emerging countries introducing into the international discussion different concepts of the importance of human rights, democratic participation, policy-making and development aid, and so on. These processes will affect evaluation and also the different functions of evaluation. The demands made on evaluation could change, and the western, primarily US, dominance in evaluation research could begin to falter. (p. 48)

Michael: I resonate with the call for a global systems perspective in evaluation’s future (Patton, 2016a, 2016b). We will need Blue Marble Evaluators—evaluators who have the knowledge, aptitude, competence, and perspective to engage at a global level, with the Blue Marble referencing the perspective of the earth when viewed from space, where no nation-state boundaries can be seen.

Charmagne: It is clear to me that the ability to think beyond borders—be they national, disciplinary, or issue focused—is going to be critical for the future of evaluation. Young people today do not think in silos—they naturally see connections. To remain relevant and useful in an increasingly connected world, evaluators must do the same. The book appropriately emphasizes this need and trend.

Conclusion

Evaluators are basically historians. We gather and analyze data and write reports on what has happened in programs. But any recommendations we offer inevitably make us futurists, for recommendations intrinsically presume some future patterns. For that reason, as well as the importance of working to realize a vision of evaluation that contributes to a sustainable and equitable future (Parsons, 2014), “the art and science of futuring is fast becoming a necessary skill, where we read signals, see trends and ruthlessly test our own assumptions” (Mims, 2017, p. 1). That is surely part of evaluators’ essential competencies in the future.

References

Anderson

Chase

Johnson

Mekiana

Ruerup

Kerr

(2012). It is only new because it has been missing for so long: Indigenous evaluation capacity building. American Journal of Evaluation, 33, 566–582.

Dahler-Larsen

(2012). The evaluation society. Palo Alto, CA: Stanford University Press.

Donaldson

S. I.

(Ed.). (2013). The future of evaluation in society: A tribute to Michael Scriven. Charlotte, NC: Information Age.

Julnes

(2015). Managing evaluation theories, practices, and communities: Reflections on Dahler-Larsen’s The Evaluation Society . American Journal of Evaluation, 36, 584–600.

Mabry

(2016). Orbiting the big dog. American Journal of Evaluation, 37, 139–147.

Mims

(2017). Think like a futurist to be prepared for the totally unexpected. Wall Street Journal, January 2. Retrieved from http://www.wsj.com/articles/think-like-a-futurist-to-be-prepared-for-the-totally-unexpected-1483272006?tesla=y

Parsons

(2014) Visionary evaluation for a sustainable, equitable future. Presidential theme of the annual conference of the American Evaluation Association, October 13–19, Denver, CO.

Patton

M. Q.

(1988). The future and evaluation. Evaluation Practice, 9, 90–93.

Patton

M. Q.

(2016a). A transcultural global systems perspective: In search of blue marble evaluators. Canadian Journal of Program Evaluation, 30, 374–398.

10.

Patton

M. Q.

(2016b). Blue marble evaluators. American Evaluation Association, opening and closing plenary sessions, Chicago, November 12 and 14.

11.

Smith

M. F.

(1994). Evaluation: Review of the past, preview of the future. Evaluation Practice, 15, 215–227.

12.

Wehipeihana

McKegg

Thompson

Pipi

(2016). Cultural responsiveness through developmental evaluation. In Patton

M. Q.

McKegg

Wehipeihana

(Eds.), Developmental evaluation exemplars: Principles in practice. New York, NY: Guilford.

13.

Wye

Sonnichsen

(1992). Another look at the future of program evaluation in the federal government: Five views. Evaluation Practice, 13, 185–195.