Abstract
In the original paper, it was argued that while there is an array of methods and methodologies available, their use is delimited by the culture of accountability that prevails in public sector institutions, a fact that is particularly problematic given the complexity and diversity of evaluation contexts today. This short rejoinder, to responses made by J.G. Carmen and L.-E. Datta to The Case for Participatory Evaluation in an Era of Accountability (both, this issue), argues further that given the prevalence and widespread use of evaluation “technologies” as formal mechanisms for decision making, monitoring, reporting, and controlling the activities and expenditures within public sector institutions, we have a responsibility to consciously and conscientiously explore some of the taken-for-granted assumptions that guide evaluation practice. Three main points are advanced: 1) evaluation as institutionalized practice, 2) evaluation as influencing and shaping public perception, and 3) evaluation as a reflection of certain values.
This agenda of accountability goes to the heart of the relationship between governments and their citizens.
The dialogue section of the American Journal of Evaluation encourages the exploration of diverse views as a way to further discussion and promote ongoing interaction on important aspects of evaluation theory and practice. With this goal in mind, two responses were invited to my original article entitled The Case for Participatory Evaluation in an Era of Accountability: Evaluation in an Era of Accountability: Unexpected Opportunities by J. G. Carmen (Carman, 2013) and Paradox Lost, Paradox Regained: Comments on the Case for Participatory Evaluation by L.-E. Datta (Datta, 2013). While both responses provided alternative perspectives and ways of framing the participatory experience in today’s climate of accountability, I did not feel that the responses quite captured the intent, nor the spirit in which my original article was written.
Carmen’s article focused on the multiple opportunities for learning created by the federal government’s emphasis on accountability and performance measurement, arguing that evaluators have a significant role to play in enabling organizations to become more strategic and better versed in the various approaches to evaluation. According to Carmen, “the era of accountability, while imperfect and flawed, has actually created opportunities for evaluators and the field of evaluation to flourish. If designed and approached in the right way, these opportunities could also be participatory and meaningful” (Carmen, p. 262). Datta’s article, for its part, was primarily dedicated to refuting a characterization of my position, namely “that participatory evaluation should rule in national evaluations in the culture of accountability” [which amounts to a form of] methodological tyranny” (Datta, p. 257). While I did not advance this position in my article, this straw man nevertheless provided Datta with sufficient momentum in which to base her responses to my article.
In this short rejoinder, I reiterate the main points raised in my original article, and argue the need to critically examine the location and role of evaluation within public sector institutions, particularly given the plurality and increasing diversity of people both within our borders and beyond. I argue that evaluation provides a form of legitimacy to government practices, what is really a form of accountability to the public, and as such has a major role to play in shaping and influencing the scope and the parameters of public debate.
The intent of my original article was to illustrate that while methodologically diverse approaches are now accepted as credible within federal public sector contexts, evaluations that play an instrumental role and are designed to satisfy technocratic requirements of accountability are nonetheless given preference. While I recognize that in practice there is tremendous blending and mixing of methods and approaches within participatory or collaborative genres (Cousins & Chouinard, 2012), the juxtaposition of technocratic and participatory approaches was used as a heuristic devise to highlight the tensions and illustrate the key challenges in introducing what may be perceived as alternative evaluation approaches in today’s climate of accountability. My argument is that while there is an array of methods and methodologies available, their use is delimited by the culture of accountability that prevails in public sector institutions, particularly problematic given the complexity and diversity of evaluation contexts today.
Given the prevalence and widespread adoption and use of evaluation “technologies” as a formal mechanism for decision making, monitoring, improvement, reporting, and controlling the activities and expenditures within public sector institutions, we have a responsibility to consciously and conscientiously explore some of the taken-for-granted assumptions that guide evaluation practice. If we acknowledge the political nature of our research methods, then our goal is to make visible the many and varied social, political, organizational, and cultural forces that guide our social inquiry methods and practices. There are three main points that I wish to advance: (1) evaluation as institutionalized practice, (2) evaluation as influencing and shaping public perception, and (3) evaluation as a reflection of certain values.
Evaluation as Institutionalized Practice
As a well-established and much sought-after practice in public sector institutions, evaluation has become an institution in our society (Dahler-Larsen, 2012). In fact, as a key feature of new public management (NPM), evaluation has become a growth industry (Leeuw, 2002), with purposes ranging from the instrumental use of findings for decision making, conceptual use to satisfy educative or learning demands, and symbolic uses related to political persuasion or for legitimation purposes. Whether in education, health care, social services, or finance, evaluation is now considered a key part of the fabric of a well-functioning public institution. As such, organizations, including funding agencies, are producing extensive, detailed reporting of performance and program activities, and cost expenditures, becoming what Dahler-Larsen (2012) refers to metaphorically as “evaluation machines.” As a formalized mechanism of organizational and state accountability and control, evaluation now plays a legitimizing function within government, situating it normatively as good governance practice. As a result, we now see the widespread adoption and use of evaluation “technologies” as formal mechanisms for decision making, monitoring, standard-setting, improvement, reporting, and controlling of program activities and expenditures within public sector institutions. As an institutionalized and formalized activity, evaluation processes and practices have come to assume a mythological status in our society (Meyer & Rowan, 1977), providing an organizational legitimacy that is taken for granted, despite its significant epistemological, methodological, and democratic consequences (Dahler-Larsen, 2012; Leeuw & Furubo, 2008).
Evaluation Influences and Shapes Public Perception
Evaluation is considered a form of “assisted sense making” (Mark, Henry & Julnes, 2000), where judgments of value are defined, analyzed, and ascribed based on what are often predefined sets of measures of quality and indicators of goodness. As evaluators, the questions that we ask and the methodologies that we use in our work ultimately determine the value of public goods such as programs and policies. In education, for example, the current trend diverts attention away from questions about what matters in schools, and distracts us from engaging in conversations that extend beyond the measurable to include contestable educational issues involving epistemological and normative questions about educational purposes and goals. In our quest to make teachers and schools more accountable, we seem to have reduced the complexity of teaching to issues of measurable efficiency and effectiveness, to a misplaced focus on what is a quantifiable, objective criteria of educational worth and value. The language and the words that we use shape what we see (Berger & Luckman, 1966), preventing us from being aware of the values, norms, and predispositions that are embedded not only in our vocabulary but in our practice as well. The distinction between what is measured and what is socially desired thus becomes blurred.
Evaluation Reflects Certain Values
Evaluation takes place in the politicized landscape of federal politics, in an arena where the distribution of public goods is discussed and debated, and where decisions are ultimately made about which programs will receive continued funding, which policies will be maintained, and which will not. Thus, beneath the seemingly value-neutral language of measures and the rhetoric of accountability reside fundamental questions about goals and values (Stein, 2001). Consider, for example, the very powerful business ethos behind NPM, with concomitant political, economic, and managerial discourses focusing evaluation on matters of accountability, performance measurement, efficiency, effectiveness, value-for-money, and results (Clarke, 2006; Tolofari, 2005). Given the limited, technological focus on these predominantly quantitative and outcome-based measures of program success, is there space for conversation and debate about how we define the measures and methods that we use to evaluate our social programs? Is there room for conversations within this technocratic space that not only include, but represent the diversity of voices and perspectives within our society?
While there can be little doubt that evaluation has attained a central place in the current context of public sector governance, and that this opportunity does indeed provide tremendous potential for innovation and growth in the field, what is less certain is what direction these opportunities will take in the future. Given that evaluation is an intrinsic component of government policy and decision making, what I have referred to as an institutionalized practice with broad social, political, economic, and cultural consequences, the influence of evaluation extends beyond the program and policy-making community. Evaluation shapes management practices; for instance, as teachers and administrators teach to the test. Thus, as key players in public sector organizations, evaluators are essentially creating institutional structures (Dahler-Larsen, 2012), the scaffolding of which one ultimately hopes will lead to a more vibrant and transparent opening for democratic dialogue and debate.