Abstract
I am as committed to the use of evaluation results as any person alive.
Carol Weiss brought passion to evaluation use. She also brought attention to it, publishing the first article on evaluation utilization in 1967. And she published my first writing on use in her edited book on Using Social Research in Public Policy Making (1977). We subsequently debated at the American Evaluation Association annual conference in 1988 and it was in one such exchange that I provoked the quote above. I made the mistake of categorizing Carol’s position as “academic.” She took exception to that narrow characterization and set me straight:
It’s true that I teach in a university, but I am as committed to the use of evaluation results as any person alive … . I am also committed to improving the methodological quality of evaluations, both quantitatively and qualitatively, so that they have something valid to say. In the putative paradigms that contrast judgment versus service, outcomes versus process, truth versus action, and quantitative versus qualitative, I am hopelessly at sea. I am interested in service and don’t eschew judgment; I study process and outcomes; I use quantitative and qualitative methods; I believe that truth helps to foster action … . Tempting as it is to divide all manner of human behavior into dualistic categories (good/bad, yin/yang, rational/political, academic/caring, etc.), a liberating slogan for the 1990s might be: “Challenge all typologies!”
Thus was I enlightened, which was altogether fitting, since she was also first to identify enlightenment as an important type of evaluation use. Carol taught me not to exaggerate differences, thereby creating rigid oppositions that might attract attention, but ultimately failed to enlighten much. And she did so with humor and grace, trademarks of her style.
Mike [has] elevated [our] differences into a “paradigms debate” … . I don’t agree with the positions he ascribes to me, and, even in the abstract, I don’t see the differences as being as stark as he claims they are. But perhaps I shouldn’t object too loudly. Mike’s formulation gives our interchanges a stature that, if we are canny, we can parlay into a textbook for the next generation of evaluators, an international conference, or a multibuck research grant to collect empirical data on the competing claims … . Since Mike and I have respect and affection for each other, we can conduct this exchange with good humor and in a mutual search for understanding.
Thus did the great paradigms debate on evaluation use evaporate. But perhaps the most enduring enlightenment Carol gifted me with was the insight that seeking and finding common ground was at least as illuminating as identifying differences. While I was painstakingly differentiating our positions in our debate exchanges, often straining at gnats, she would smile and bring us back to the basic ideas and values we shared.
Here are some things that I hope Patton and I can agree about. Let us all strive to improve the quality of our evaluations. First, let us try to ask the right questions—the key questions—the pregnant questions—questions that have important implications for the future of programming. Then let us use designs that are appropriate to the questions we raise, not forgetting that qualitative evaluation requires careful design as well. Let us work to hone our methodological skills, our working-with-people skills, our understanding-of-program-life skills, our capacity to understand and interpret what we find. From the very beginning, let us work to encourage the evaluative cast of mind among policy and program planners, so that they are skeptical of the received wisdom and the latest fads and ready to examine the assumptions on which programs are based. Finally, let us begin the evaluation with dissemination in mind and be sure we continually diffuse the evidence we assemble to all the people who have a say about program futures. But … let us not expect that good skill and goodwill are going to rid this fallible world of self-interest, apathy, search for agency aggrandizement and political advantage, or random events … I still believe in the longer run. Now it is perfectly true, as John Maynard Keynes has said, that in the long run we are all dead. I don’t mean that long a run. But in the space of two, three, five, or eight years, many of the results of social research come into currency and overtake formerly taken-for-granted assumptions.
Carol’s many contributions have come into evaluation currency and overtaken formerly taken-for-granted assumptions. Indeed, her insights that were groundbreaking and controversial at the time are now deeply enmeshed in evaluation thought and practice such that those newer to the profession might think it has ever been thus. I assure you it has not. Space being limited, let me narrow it to my top 10 countdown list of her enduring contributions.
10. Mixed methods. She was using them before they were called that because mixing methods strengthened findings. And she refused to engage in the quantitative–qualitative debate calling it “pretty silly,” an elegantly phrased scholarly judgment.
9. At a time when social science asserted value-free objectivity, she articulated that evaluation is fundamentally, inevitably, and ubiquitously a political activity.
8. She brought into evaluation sociological knowledge about how organizations work (one dimension of our commonality being that we both suffered the affliction of being sociologists by training). Exemplars of her sociological wisdom: that knowledge alone rarely changes behavior, despite this being perhaps the most common and simplistic theory of change marketed to naïve and gullible program funders; that decisions in organizations emerge and gradually coalesce as so-called decision makers muddle about, satisfice, compromise, and negotiate; and, eventually, if all goes well, which is far from a sure thing, knowledge creeps and accretes; that sometimes NOT using evaluation findings is the right thing to do, given other contextual factors and considerations; and that users apply both “truth tests” and “utility tests.”
7. She valued and helped build our professional community. She was a founding member of the Evaluation Research Society (ERS) and wrote the first general evaluation textbook in 1972.
6. She observed early on that evaluation was a profession that would attract women. At the initial gathering that launched ERS she recalled: “The thing I remember most … was that there were more women than men … , the first time in my life I had been at a professional meeting where the women outnumbered the men” (p. 477). In the second edition of her classic 1997 textbook, the evaluator became a “she.”
5. Her astute observations and wily wit have given us many provocative metaphors and creative turns of phrase that enliven evaluation dialogue: Evaluators wishing for a Fairy Godmother who would make decision makers pay attention to evaluation findings. Reflecting on the common “no effects” findings of large-scale experimental designs in the Great Society years: “We had signed on as evaluators with the intent of contributing to the improvement of social programming, but we seemed to end up giving aid and comfort to the barbarians.” Cautioning about asserting that we can generate lessons learned and validate best practices: “I just think we should have a little more humility about what we promise our sponsors and clients.” She included a cartoon in the second edition of Evaluation Roots showing two toga-clad Romans conversing, one saying: “I am so past enlightenment.” And went on to wish she could, herself, find that “blissful state.”
4. Carol made concrete and meaningful Kurt Lewin’s assertion that “nothing [is] as practical as good theory.” She pioneered bringing “theory of change” into program and evaluation design.
3. She turned our attention from the stranglehold of closed-system project mentality to evaluate new units of analysis like communities undergoing comprehensive change initiatives. These demanded innovative evaluation approaches, a harbinger of systems, and complexity thinking now ascendant in evaluation.
2. She articulated that evaluative thinking, not just methods and findings, is the core contribution of the profession. She urged nurturing an evaluative cast of mind among policy and program planners. That phrase, in our 1988 debate, stayed with me and stays with me still as epitomizing what we now understand to be process use (beyond findings use).
1. Her commitment to use inspired the study of use and inspires still. She was ever and always “as committed to the use of evaluation results as any person alive”—the ultimate summative evaluation.
I will miss the collegial relationship and ever-fresh insights she brought to our work and her reminders of the deep and important values we share as members of this profession we call EVALUATION, a profession she did so much to create and nurture.