Abstract
Peter Dahler-Larsen is a professor in the Department of Political Science at the University of Copenhagen in Denmark. He leads the Centre for Results, Effects, Measurement, and Evaluation (CREME) and has written numerous articles and books on issues relevant to the evaluation discipline, including The Evaluation Society (2011) and an encyclopedia entry on evaluation quality (2010). He is a past president of the European Evaluation Society, and he built and directed the Master’s in Evaluation program at the University of Southern Denmark before moving to the University of Copenhagen. He is an engaging speaker and a prolific thinker and writer who brings a sociologist’s perspective (Julnes, 2015) to evaluation and society.
In Quality: From Plato to Performance (2019), Dahler-Larsen takes a deep dive into the concept of quality, from its earliest references to how it is valued in today’s sphere of public decision making. He argues that if we do not tend to the ways that quality “helps construct, form, and shape the reality it claims to describe” (p. 116), then efforts to improve our existence could become self-defeating systems mired in inexactness, alienation, counterproductivity, and destructiveness. He urges readers to be more aware of the universe surrounding quality inscriptions, which he defines as “forms of quality documentation” (p. 13). Throughout the book, he provides new language and insights to frame these concerns. The prescription for those of us in the evaluation discipline, no matter how we participate, is to consider and weave these understandings into our practice in order to more thoughtfully define and measure each evaluand’s quality.
There is a lot of ground between Plato and performance, and Dahler-Larsen covers it in seven dense chapters. He sets the stage in Chapter 1 by providing definitions for specific terms used in the book and then provides a historical perspective on the concept of quality in Chapter 2. In Chapter 3, he lays out nine different perspectives on quality. Next, in Chapter 4, he offers a review of those who instigate quality mechanisms and the measurement systems that ensue. In Chapters 5 and 6, he describes three different models of quality creation and inscription and articulates some concerns with quality inscriptions as they relate to the public sphere. Finally, in Chapter 7, he lays out his ideas for where to go next.
The chapters have varying degrees of accessibility and potential usefulness to the field of evaluation. Given the multitude of complex ideas in the book, I do not discuss them all in this review. Instead, I describe three concepts that were highlights for me and then reflect more generally on the value of the book to the profession.
Highlights
Those who traffic in the dialogue of evaluation are constantly talking about quality in some way or another. For instance, we can take the general logic of evaluation (Scriven, 2007) as a regulative ideal (Guba, 1990; Rosenberg, 2005). Despite other activity we may engage in and call evaluation, the search for criteria, the identification of standards that guide measurement, and the development of a judgment are its core functions. Those judgments are how we operationalize quality. In short, our job as evaluators is to explicitly identify the value chain and make judgments of quality. Having new language and concepts injected into that conversation is one of the great values of this book. The following three highlights are examples of what this book offers evaluators.
Qualitization
One rather complicated concept Dahler-Larsen introduces is that of qualitization. He defines it as an act “that makes quality relevant in a given instance, by mobilizing, connecting, and setting in motion” (p. 20) the many other concepts circling quality. These concepts include the measure used to assess quality, or the quality inscription, as well as the definition of the thing to be judged—for us, the evaluand; for him, the quality object.
Qualitization could be that moment when someone in charge of resources decides to tell their important stakeholders how well they are managing those resources. This sets in motion a process wherein definitions are developed (e.g., for criteria, standards, measurements, and judgments), data are collected and analyzed, and reports are written. A whole host of processes, tools, and resources are brought to bear on this quality situation, as Dahler-Larsen calls it, and an evaluator may or may not be engaged at any moment. Qualitization can be launched by any player in a complex process.
Dahler-Larsen suggests that three models provide ways of understanding how qualitization is set in motion. He covers these extensively in Chapter 5. They include a deliberative model based on democratic ideals of participation and voice driving the process of defining quality, resonant of deliberative democratic evaluation (House & Howe, 2000); a metrological model focused on precision of measurement of said quality, much like objectives-based evaluation; and a configurative model that, by its nature, entails multiple agents, mechanisms, and processes coming together to “configure” some qualitization or quality situation.
Evaluators often enter into situations where some act of qualitization has occurred. Their job is to put that quality notion into action and to understand the many constitutive effects that come with such an act. For instance, “teaching to the test” is a potential constitutive effect of academic testing and an evaluator tasked with evaluating such an academic test will need to consider disentangling one from the other. Although Dahler-Larsen’s models could provide evaluators with useful ways of understanding how quality expectations have developed, the very idea that a qualitization moment has occurred, requiring the need for an evaluator, seems to be an important concept to consider as we enter any evaluation situation.
Perspectives on Quality
A second major contribution to the evaluation discourse is embedded in the nine perspectives on quality Dahler-Larsen puts forth in Chapter 3. With these perspectives, he sets the stage for us to think about the focus of our evaluation work. These nine perspectives are not mutually exclusive, and they gather their clarity of definition from the context within which they are applied. They range from quality as fulfilling a policy, to quality as utility, to quality as standards.
I find particular value in the perspective of quality as practice. Here, Dahler-Larsen builds on earlier descriptions of evaluation practice as encompassing a hermeneutic circle of practice and reflection where the interaction is always a learning moment that creates a more holistic and useful evaluation—where improvement happens because evaluative thinking has been implemented (Dahler-Larsen, 2010; Schwandt, 1990, 1997, 2003). This happens differently in almost every context and is only “perfected” (a regulative ideal) by being engaged in the dailiness of evaluation.
Dahler-Larsen places quality as practice as the last of his nine perspectives and acknowledges that some might see all of his other perspectives as embedded within it. That is, one could see all other perspectives of quality in the application of good quality practice, making it a super concept of sorts. This resonates with my own interpretation of quality evaluation practice: In most every evaluation, an evaluator will consider questions related to goals, utility, user needs, expert advice, standards, organizational expectations, impact, and excellence, and focus more or less on each based on the needs of the individual evaluation. Dahler-Larsen’s quality perspectives emanate from different value stances, and an evaluator must identify which values are central so they can decide which quality perspective will guide the evaluation. In this way, quality evaluation practice might then be judged by whether it has attempted to address all of the other eight perspectives.
The Configurative Model
Quality configurations, such as the deliberative and metrological approaches, have important contributions in democratic governance and government processes; however, the configurative model is likely problematic for both. Here, Dahler-Larsen plants his flag, providing what I see as his core reason for writing the book. He wants us to think about how quality systems are developed and put into place, to ensure they are still useful and valid, and not overrun by self-defining spirals of “rubber stamping.”
The configurative model he describes has wide-reaching constitutive effects. These effects have the strong potential to displace good deliberations and measurement, assigning responsibility to those in the model’s universe and alienating or marginalizing those anywhere beyond the original qualitization instigator. Left unchecked, the confluence of constitutive effects can sidestep the important components of good governance at the heart of res publica as we know it; it sets the stage for letting quality machineries outstrip us of our participation in defining what is truly good for the development of our existence. This may sound profoundly overstated, over simplified, or both, but this is the foundational thesis at the heart of Quality: From Plato to Performance. I think it is a useful conceptualization to include in our dialogue.
Reflections
I am very appreciative of this book and the concepts Dahler-Larsen puts forth. It serves as another strong example of the deep and important thinking he contributes to the public discourse in evaluation. His writing, however, is not for everyone. He notes in Chapter 1 that he uses language in ways that might not be common but are, nonetheless, clear. He apologizes for his “unconventional terminology” as he states that his purpose is to create a sort of “alienation effect” that is intended to create distance from the everyday thinking that he says is necessary for the analysis (p. 12). This distancing could be quite off-putting for some, but it is not impenetrable; it just requires work on the part of the reader to enter his cadence and use of the English language.
As suggested in my brief discussion of Chapter 3 above, much of evaluation theory can be interpreted within Dahler-Larsen’s nine quality perspectives. However, he does surprisingly little to connect these perspectives to the deep evaluation literature on many of the related concepts. He does refer to Eisner (1976) when discussing quality as expert judgment, and he brings in some of Schwandt’s (1997, 2003, 2018) writing around quality as practice, but he misses other opportunities that surprised me. For instance, he entirely neglects utilization-focused evaluation (Patton, 2008) in the quality-as-utility perspective, leaning more toward products as “useful” to consumers. In his discussion of quality as fulfillment of the needs, expectations, or wishes of users, he does not note important work on needs assessment (e.g., Altschuld & Watkins, 2014; Scriven & Roth, 1990; Witkin, 1994), instead centering more on the context of manufacturing and product sales. When he describes quality as standards, he focuses more on a deficit model (e.g., manufacturing “minimum” standards) rather than aspirational standards as in the Program Evaluation Standards (Yarbrough et al., 2011). Finally, in his discussion of quality as impact and quality as accomplishment of official political goals perspectives, there is no mention of outcomes-oriented evaluation (Wandersman et al., 2000) or consumer-oriented evaluation (Scriven, 1991). With this unfinished integration of extant evaluation literature, Dahler-Larsen does, in effect, provide a torch to be picked up and advanced by others who view quality as a multifaceted construct central to evaluation practice and theory.
Few scholars have been thinking about and studying the concept of quality, especially involving evaluation, as long as Dahler-Larsen has. He combines a sociologist’s perspective (Julnes, 2015) with an evaluator’s lens, providing useful ways to think about quality as a system rather than as a constellation of characteristics. As the world becomes more interconnected, and as we use this interconnectedness to improve our existence, understanding and challenging the systematization and commodification of quality will become more necessary. As evaluators, we need to ensure our central place in defining quality is not lost in the ever-growing breadth of qualitization’s constitutive effects. With this book, Dahler-Larsen challenges us to step into the breach and be reflective, critical, and practical evaluators, especially in the face of large or preexisting systems of quality indicators.
