Abstract
After more than 40 years, Saville Kushner has announced his “swan song from the field of evaluation.” He exits stage left, both dramatically and politically, with a book ostensibly about evaluation research methods that is really a treatise on, and manifesto about, the roles and responsibilities evaluators have for invigorating democracy. He has devoted a distinguished academic career to studying, engaging with, and reflecting on evaluation’s actual, potential, and largely unrealized contributions to democratic society. In previous writings, he has concluded that the future of democracy, perhaps even of civilization, depends on getting right the relationship between expertise and democracy. Evaluation, being a form of expertise, and evaluators, manifesting, expressing, and acting on that expertise, must be concerned with this tempestuous, highly politicized, and therefore difficult relationship between expertise and democracy. Evaluators must take up, he exhorts, the challenge to get it right. Yet, he concludes and laments that “the democratic space for public and evaluative critique and personal exploration is diminishing.” He writes, therefore, with a sense of urgency and in the hope of “passing the torch” of this concern to a new generation, one that he hopes will take up this “same preoccupation with expertise and democracy” (Kushner, 2016, pp. 77–83).
When a community’s elder summons the energy and passion to speak of things seen—of what has been, what is now, and what still might be—it’s worth a listen, or, in this case, a read. But before offering a review, celebration, and critique of his swan song ruminations on the futures of both evaluation and democracy, let me further set the context for Kushner’s benedictory exhortation to help you fully appreciate his new and final book, his evaluation research retrospective as civilization prospective. Spoiler alert: He also offers sage advice on managing the complexities of judgment in the field.
Kushner’s first major book, Personalizing Evaluation (2000), has become an important classic in the evaluation literature. His new book, Evaluative Research Methods: Managing the Complexities of Judgment in the Field (2017), builds on, updates, and expands the earlier work. So a joint review seems both timely and pertinent for understanding the unique perspective of this scholar.
Personalizing Evaluation
Personalizing Evaluation is about the evaluator’s responsibility to take seriously and communicate authentically the perspectives of those we encounter during our evaluation inquiries. Kushner brought forward, deepened, and updated Barry MacDonald’s democratic evaluation model (Kushner, 2005b). He made the case for evaluation as a form of personal expression and political action, with a special obligation to be critics of those in power. He advocated using qualitative methods, especially in-depth case studies, to place at the center of evaluation the experiences of people in programs. The experiences and perceptions of the people in programs, the supposed beneficiaries, is where, for Kushner, we find the intersection of Politics (big P—Policy) and politics (small p—people). He showed how case studies can capture the perspectives of real people—children and teachers and parents—and the realities of their lives in program settings as they experience those realities. Thus, we meet and get to know Ann, Lucy, Peter, Chloe, Lyn, and Michael as they participate in innovative music education programs that Kushner has evaluated. They become real to us and we understand the programs through their experiences and perspectives. And central to the message of personalizing evaluation, we learn about the programs through these people, rather than learning about the people through the programs. He advocates that the perspective of participants be given primacy, that evaluators approach programs through the experience of individuals rather than through the rhetoric of program sponsors and managers. He wrote: I want to emphasize what we can learn about programs from Lucy and Ann. This does not mean ignoring the rights of program managers and sponsors with access to evaluation…. I don’t think there is a serious risk of evaluators losing touch with their contractual obligations to report on programs and to support program management and improvement; I don’t think there is a danger that evaluators will ever lose their preoccupation with program effects. There is always a risk, however, that evaluators lose contact with people; and a danger that in our concern to report on programs and their effects we lose sight of the pluralism of programs. So my arguments will robustly assert the need to address “the person” in the program (Kushner, 2000, pp. 9–10).
In the new book, he reiterates and deepens this central message: Notwithstanding our urgency to understand how people fit into our programs and interventions, the true imperative is to understand how our interventions fit into the lives of people. This is the core democratic task of any project, program, or policy evaluation, and it remains as a significant methodological and writing challenge. (Kushner, 2017, p. 260) Where each social and educational program can be seen as a reaffirmation of the broad social contract (that is, a re-confirmation of the bases of power, authority, social structure, etc.), each program evaluation is an opportunity to review its assumptions and consequences…. All programs expose democracy and its failings; each program evaluation is an assessment of the effectiveness of democracy in tackling issues in the distribution of wealth and power and social goods. Within the terms of the evaluation agreement, taking this level of analysis into some account, that is, renewing part of the social contract, is to act more authentically; to set aside the opportunity is to act more inauthentically, that is, to accept the fictions. (Kushner, 2000, pp. 32–33)
Personalizing Saville Kushner
Kushner’s style of writing is highly personal. Eleanor Chelimsky, on the new book’s back cover, captures well its elements: “Part history, part memoir, part technical guide, and part passionate advocacy.” His books are exemplars of reflective practice, sharing missteps, and failures as much as successes. Becker and Renger (2017) recently published a set of guidelines for writing reflective case narratives. Kushner’s writing exemplifies their call for reflective case narratives that provide “credibility as a resource for knowledge transfer, sharing of ideas, and keeping an eye on the dynamic and fluid process of evaluation” (Becker & Renger, 2017, p. 148).
He addresses the reader directly: “I have firmly located this book in democratic enquiry, and I have invited you to position yourself with me” (Kushner, 2017, p. xii). And later, “I would like to invite you to see yourself as a player in this highly consequential game of democratic enquiry. I’m pleased to have you as a companion” (p. 10). Still later, “What I will try to do now is to induct you into a way of thinking about evaluation and evaluative research—in fact, a way of thinking about its current demise” (p. 20).
When he wrote Personalizing Evaluation, Kushner was Associate Dean and Professor of Applied Research at the University of the West of England, a public university located near the city of Bristol, where he was engaged with the Centre for Research in Education and Democracy. Beyond educational evaluation, he has extensive experience working with the police service on policing culture and national police training. He served as President of the United Kingdom Evaluation Society. He wrote his new book while on the faculty of the University of Auckland where he also served as Chair of the Oversight Board for New Zealand Aid. Since he is insistent on the importance of representing participants in their own words, let me follow that standard by quoting him about himself for a 2015 Australian Evaluation Society International Conference: I am a long-standing advocate of Democratic Evaluation and one of the early pioneers of evaluation case study. I am a methodologist by training. I am more ready to talk about evaluation experience and practice and their complexities than about evaluation theory or policy or its models and exhortations. I am less concerned with the “soft data” challenge of results measurement, more so in the “hard data” challenge of finding program quality. (Kushner, 2015)
Democratic Evaluation Versus Deliberative Democratic Evaluation
Kushner wants to be clear about the difference between two discrete concepts, democratic evaluation and deliberative democratic evaluation. The approach I promote here is derived from democratic evaluation (MacDonald, 1976) and is distinct from deliberative democratic evaluation (House & Howe, 1999) in that democratic procedures are built into the action from its earliest stages, in access and design negotiations. Deliberative democratic evaluation focuses more on transactions subsequent to evaluation reporting. One radical implication of the approach is that in terms of theory that explains and guides educational practices, the wheel has to be reinvented with each study (not that we cannot learn across studies). This does not deny replicability or a role for formal theory, but it does require us to reconceptualize both in relation to democratic rights and inquiry. (Kushner, 2005a, p. 581)
International Development Evaluation
He also has extensive knowledge of and experience in Latin America. He co-edited a volume of New Directions for Evaluation (Kushner & Rotondo, 2012) that focused on how evaluation in Latin America has been part of discourse about the need to reduce disparities and inequalities. Participatory and emancipatory forms of evaluation have strong roots in Latin America, and, as the editors state in their introduction (p. 1): “…the possibility of public knowledge feeding into citizen agency represents a recent entry into democratic possibilities. Evaluation, always and everywhere a significant element in the contest for democratic control, has a special significance on this continent.”
Between 2005 and 2007, Kushner served as the United Nations Children's Fund regional officer for monitoring and evaluation in the Latin America/Caribbean Region. He wrote about that experience in an opinion column for The Guardian newspaper: I recently served as a United Nations (UN) regional officer monitoring progress towards the millennium development goals (MDGs). I learned to lose any sentimental attachment to the idea of “halving extreme poverty by 2015” and “eradicating hunger.” First, though, let me be clear: the world is a better place for MDGs, which stand as a unique moral commitment and make uncomfortably transparent the failures of advanced industrialism in redistributing wealth. As minimalist as they are, they are stretching political and economic capacities and holding governments properly to account. But let us not be seduced by their simplicity and populist appeal. Let us, rather, be realistic and place their ambitions in the contexts of their own limitations and the wider challenge…. The goals themselves were negotiated among and are “owned” by international political elites, not by the citizens and communities whose needs they serve—indeed, there is no goal for community empowerment. The MDGs focus attention and scrutiny on the poor and those who aid them. They avoid any scrutiny of the rich and do not address the broader political contexts of aid and trade. (Kushner, 2008) “Where we ask development evaluation to measure the results and impact of an intervention, we ask too little; we should be requiring generalization at this level of local insight.” “Where we ask evaluation to rule on the worthwhileness of a program, we ask far too much, for its substance, its interactions, what is made of it and what is made out of it are far too complex to be summed up.” “We often talk about striking a balance between evaluation for accountability and evaluation for learning in international development. This is hopelessly simplistic. Aid agencies, acting on behalf of the political community, should be accountable for their learning and they should learn from their accountabilities. There is no dichotomy.” “Just as hopelessly simplistic is to imagine we can sum up the quality of an intervention by assessing its results and impact…. [Evaluation] is too important a process of learning about how the organization fits into the lived context of development to be left with the relatively trivial task of measuring results—which projects and programs should do for themselves, in any event.” The 17 SDGs—with “a wide and numerous array of targets and indicators”—risk “spawning an industrial explosion of evaluation consultancy, which would further intensify the politicalization of data and distract evaluative practices away from democratic ideals.” “One thing we know from evaluation history is that the greater the pressure to demonstrate results the greater the tendency towards the engineering of results—and the greater the pressure for program manager and evaluator to enter into conspiracy, for they both have financial interests in good outcomes” (Kushner, 2017, pp. 262–264).
Provocateur
As the preceding quotations illustrate, Kushner writes as a provocateur. His chapter titles are at once intriguing, inviting, and provocative, as is his writing. Here are a few samples: Chapter 1. What is evaluation research and how are you positioned within it? Chapter 4. Theorizing, contingency, and evaluative cognition: Conceptual tools to help you read the book. Chapter 5. The “self” and the “person”: Democratic individualism. Chapter 7. Observation-based interviewing, triangulation and the redistribution of power. Chapter 8. On bullfighting, the fragile self, and an introduction to evaluative case study. Chapter 10. Writing democratically, and finding your own voice. Chapter 11. Quieting reform: When evaluative research can be dangerous. Chapter 14. Radical manifestoes for evaluative research. Chapter 15. Democracy, theory, and logic in evaluative research: Beware of rationalism.
On Evaluative Thinking
Kushner is deeply concerned about how evaluators think, or more often, fail to think, in following routine procedures and nondemocratic processes. More than a decade ago, he introduced the premise that this new book espouses at length: “We should be experts not in theory and methodology but in theorizing and in methodological thinking” (Kushner, 2005a, p. 581; emphasis in the original). Here, then, he makes explicit evaluators’ roles and responsibilities in the relationship between expertise and democracy. Our craft expertise is grounded in methodological knowledge and skills, but the exercise of that craft expertise is neither routine nor mechanical. Each new evaluation requires methodological thinking in the same sense that a fine artisan crafts a unique and customized product based on a creative, contextually sensitive thought process. Neither the creative artisan nor the thinking evaluator is engaged in factory assembly line work. In a similar vein, theorizing is a collaborative and interactive sensemaking process, not a matter of assembling boxes and arrows in some sequence to produce a theory.
He is especially attentive to and insightful about how evaluators render judgments, and how rendering judgments affects everything we do. The 2017 book’s subtitle, Managing the Complexities of Judgment in the Field, foreshadows his insistence that evaluators do more than merely judge the merit, worth, and significance of a program or policy according to the criteria of those who create, fund, and implement those programs and policies. Rather, his very definition of evaluative research and enquiry is “a process for arriving at judgments about public value, in such a way that reveals the nature of the public” (Kushner, 2017, p. 8). The importance of this definition stems from his conclusion that our contemporary era manifests “the demise of public sectors and public valuing” and raises questions about the future of democracy (p. 11). This leads him to call on evaluators to think beyond the effectiveness of a particular project, to use whatever specifics are found to shed light on and examine larger issues of citizen–state relationships, consumer–corporation interactions, and evaluator–funder–user power dynamics.
Evaluative Research Methods addresses the big picture issues of evaluation’s role in society and the history of the profession that got us to where we are today. That historical and contextual analysis sets the stage for examining fundamental and essential practices and methods like interviewing, case studies, fieldwork, report writing, and utilization. Kushner skillfully and insightfully shows and explains how these larger contextual and historical developments have led to conceptual and methodological approaches that serve those in power and maintain the status quo by the very ways in which evaluative inquiries are undertaken and the inevitably insipid findings and bureaucratic uses that ensue.
On Rights and Responsibilities
Kushner’s democratic evaluation approach places particular emphasis on people’s rights to information and their rights to have their stories told, and their perspectives represented, in their own terms. Evaluators fail to fulfill this function for methodological, epistemological, theoretical, professional, and political reasons, especially because of “our co-option into hierarchical agendas and priorities as we serve those who pay our bills” (Kushner, 2017, p. xii). He makes a fundamental and emphatic distinction between program participants’ rights and responsibilities versus evaluators’ rights and responsibilities: “Those who are evaluated have rights and obligations. You as the evaluator or the evaluator researcher have only obligations” (p. 196). He characterizes citizens as “rights-holders” in contrast to evaluators as “duty-bearers” (p. 202).
When I encountered this argument, I wasn’t sure if I disagreed or didn’t understand, so I emailed Saville and we exchanged perspectives. Here, then, I welcome you, with Saville’s concurrence, into our exchange, two old roosters crowing, cock-a-doodle-doo, not at the brightness of dawn, but at the engulfing dimness of the encroaching dusk.
I see things in UN “rights-based” terms, in the public realm, and in relation to social needs and social justice addressed by policy: we are either rights-bearers or duty-bearers. Our role is defined by the information rights of stakeholders. Our duty is to ensure that their information rights are met.
So, under the Democratic Evaluation rubric the evaluation agenda is made up of diverse stakeholder concerns. The evaluator’s role is to give expression to that diversity. The commissioning of an evaluation is the concretization of stakeholder rights to have their concerns addressed. We define “stakeholder” in terms of holding a substantive interest in the program, and we assume that substantive interests will diverge across constituencies in a program. Democratic evaluators, to remain impartial and independent so as to authentically represent that diversity, have no substantive interest, nor can they exclusively represent the interest of any single constituency. Hence, they have no rights to assert, only obligations to fulfil.
This principle applies methodologically as well. The democratic evaluator has no rights to insist on method or to insist on access. Speaking democratically, people have the right to be represented in terms that make sense of their practices, otherwise, how can they be held accountable? So even methodology is subject to the principle that the evaluator should strive to meet people’s rights for their practices to be rendered in a way that allows them proper expression. That’s not the end of it, of course. There may be an imperative to represent experience in ways not necessarily to the comfort of the practitioner (professional inspection is a case in point). But this merely intensifies the obligation on evaluators to negotiate their way.
Of course, I realize that this is specific to democratic evaluation, though, in ethics terms, I would mount an argument that this generalizes. A recommendation represents the evaluator’s right to own interpretation. But this conceals numerous political traps. Even where an evaluation agenda grows out of the concerns of a single value system (say, program managers), this cannot justify the suppression of dissenting or alternative views.
But perhaps more than anything else is the fact that evaluators move on. We do not live the consequences of our evaluations. Hence, we have no rights to determine futures. We have an obligation to those who will live those futures, to create the intellectual conditions under which they will make the best out of limited resources and diverse values. What say ye??
Ironically, it strikes me as both disrespectful and disingenuous to pretend, from a superior position of power, to be suspending one’s right to rights under the guise of conducting an evaluation. And it is pretense. Evaluators’ rights don’t disappear. They can be exercised at any moment. That pretense is what most disturbs me, that an authentic relationship becomes gaming and inauthentic under the notion that we deal with status and power differences by simply suspending our rights while maintaining our responsibilities.
“Rights-free” reeks of value-free. We’ve dealt with the pretense of being values-free by being explicit about values. Likewise, I suggest a more viable stance for democratic evaluators is to be explicit about rights, how they are exercised, and the mutuality of rights.
On the practical side, there are important reasons why the evaluator’s rights should not only remain intact, but indeed on full display. I have interviewed people who were hostile and threatening. My rights to safety and respect were not suspended, and had they been, I would’ve put myself at risk. I ran a research center and had female interviewers sexually harassed. It was their assertion of their rights, and my enforcing those rights, that protected them.
I find that there is a kind of liberal guilt-condescension in the pretense of giving up rights. That’s what rankles. I would be especially concerned about advising new and inexperienced evaluators to suspend their rights rather than to enter into fieldwork and interviews with the full sense of who they are and how to show respect to the people whose stories they are collecting and behaviors they are observing. But you don’t do that by pretending you have given up your rights. Mutual respect seems to me to be the desired stance rather than one side having rights and the other having given those up.
Being without rights doesn’t make sense to me philosophically, methodologically, professionally, or practically. Your insistence on this point, in my reading, creates a weak foundation for the democratic evaluation stance you advocate throughout. Portraying the evaluator as a servant of democracy but, somehow, not a full participant in that very democracy, is paradoxical and, in the end, unconvincing to me.
I believe the preponderance of evidence is that we are more effective and influential by modeling desired behaviors rather than suspending those behaviors in the hopes that they somehow encourage others to adopt the very behaviors we just suspended. So, my friend, we are on quite opposite sides on this issue.
I see EVALTALK has a discussion about how far we have the “right” to edit data. I say none. One person asserts evaluator rights to arbitrate on what counts as “equitable” outcomes—I disagree. Another would hold that the evaluator has the right (is expected) to rule on where merit lies—I default to the contrary (but boundaries can be blurred). I am probably enough of an extremist to question the commitment of [some distinguished colleagues] to identifying what is “good” in education. Of course, I engage in something less than the Trappist practices I imply—but, then, we are talking of leanings, directions of travel, basic principles of procedure.
Nor do I think of this in terms of value-free enquiry. To invite confidence in impartiality, to draw back from expressing any interest in program outcomes, is a radical value position—especially in a world where there is no shortage of players with partial views. Do I expect the evaluator to have partial values in respect of the work of the program being evaluated? Not at all—other than privately and secretly. Do I expect the (democratic) evaluator to legitimate “goodness” and its absence? No—other than to express the views of stakeholders. That is for other observers. I repeat my favorite mantra of Peter Berger’s: “maturity [professional mastery] is the capacity to endure distance from the object of one’s passions.”
Are we still batting the same ball, here? As I say, do use any of these e-mails, Michael—I’m grateful for the engagement.
And with that, we both pass the issue on to you, the reader.
Ongoing, Ever Deeper, Reflective Practice
Kushner is explicit about what his book is not. He offers “no support for an exclusively results-based or outcomes-focused approach to evaluation,” no guidance on Theories of Change approaches, logic models, methodological models, or quantitative-qualitative-mixed methods debates. “Solving problems is not the role of evaluation I advocate in this book…. There is no support here for the evaluator’s role in ‘changing the world’” (Kushner, 2017, pp. 32–34).
But if you are open to having your ideas about evaluation dissected and examined, and the roles, responsibilities, rights, obligations, methods, theories, and practices of evaluators laid bare and challenged, this is the book for you. Kushner invites you to “think creatively” about who you are and what you are about as an evaluator. In extending this invitation for critical reflection, he explains: “Most of this book, therefore, is about method—not just method in the sense of ‘applied technique’, but also in the sense of mindful action” (Kushner, 2017, p. 32). He invites you to ponder what it means to “think methodologically” and “what you must do to call yourself an evaluator or an evaluator researcher.” In so doing, he promises “to initiate you into action that is morally, ethically, and politically justified” (p. 33). He addresses each of us individually but also invites us to reflect together critically, as a profession, on who we are and what we are about.
It is quite a swan song, if that it be, for he has also started a blog, where he addresses reactions to the book and shares his latest thoughts on the future of evaluation. So it appears that the book may not be the last we’ll hear from him. We need more of his provocative observations, like this one: “The modern democratic state is itself a flaky innovation, still uncertain of its permanence” (Kushner, 2017, p. 269). Democratic evaluation is, likewise, a flaky innovation, still uncertain of its permanence. So, stay awhile, Saville, and nurture both democratic evaluation and, thereby, democracy, for your book makes an eloquent case for the need for, and ultimate interdependence, of both.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.