Abstract
Carol and I first met in 1974. The occasion was a meeting at Harvard hosted by Clark Abt to talk about the timeliness and feasibility of launching an evaluation society. Marcia Guttentag introduced us, and I remember congratulating Carol on her great book, Evaluation Research. Even today, with so much important new work behind us, that book retains its special voice, its realism and fairness, its focus on crucial issues of practice (along with those of theory), and its remarkable alertness—almost unique at the time—to evaluation’s systemic nature. As she wrote 40 years ago: To make research work, when it is coping with the complexities of real people in real programs run by real organizations, takes skill … and some guts. The evaluator has to know a good deal about the formulation of the research question, study design, sampling, measurement, analysis and interpretation. He has to know what is in the research methodology texts, and then he has to learn how to apply that knowledge in a setting that is often inhospitable to important features of his knowledge. If he persists in his textbook stance, he runs the risk of doing work irrelevant to the needs of the agency, antagonizing the program personnel with whom he works, and seeing his study results go unused—if indeed the work is ever completed. So he sometimes has to find alternative ways of conducting his study, while at the same time he stands ready to defend to the death those elements of the study that cannot be compromised. (Weiss, 1972).
I think it is fair to say that this book of hers had an enormous influence on the evaluation field and on evaluators generally. She taught us, among other things (and with great attention to the “details” of practice), that evaluation is a deeply inclusive process, that all evaluation methods have at least some problems, and that conflicts between program practitioners and evaluators spring eternal. I know these and other ideas affected my own thinking and that of other evaluators as well.
Carol and I stayed in touch after that initial meeting in 1974 (which eventually produced the Evaluation Research Society in 1976) and I asked her to be a keynote speaker at a symposium I had developed, which focused on the sustainability of federal programs at the local level. I will never forget how courageously—but unaggressively—she confronted some federal officials at the symposium, who had been making rather lofty comments about state and local research skills. What she said was: It’s vital to recognize that federal programs can be and are used to serve two different and distinct purposes: not only to aid local agencies to adopt more effective strategies of service, but also to change their goals and priorities. Federal programs may be marked under the banner of R&D, and heralded as ‘knowledge products,’ but along with the attempt to put better knowledge into practice, there is often the covert political aim of altering the objectives to which local agencies address their effort. It seems to me that this undermines both reform and science. (Chelimsky, 1978. p. 57).
In my judgment, Carol’s 1978 view of the donor’s hidden agenda—an agenda which grant receivers always perceive instantaneously—is still highly relevant to the issues of sustainability today, another of her gifts to us.
In 1980, when I went to the Government Accountability Office, I asked Carol to become a visiting scholar at GAO’s Program Evaluation and Methodology Division and a permanent member of our Advisory Board. Her reviews of our work were always pithy and persuasive to staff but often painfully acute. She examined what we were doing, understood why we were doing it (along with Dick Light and Peter Rossi, she saw the value of the synthesis method—a method bitterly disputed at GAO—for answering that perpetual congressional question: “what does research tell us about such-and-such a problem?”), she warned us about potential rival hypotheses that could explain our findings, cautioned us against either understatement or what she called “misplaced specificity,” and applauded us for freely acknowledging limits in our data. She caught all our unnecessary equivocations and hammered us for them, but she was nonetheless happy with the (necessary) equivocations in our paper on estimating the size of the illegal alien population. She wrote “The review of existing estimates of illegal in-migration and population is a beautiful piece of synthesis. It explains the assumptions of the several studies and their methods of estimation with admirable clarity, and it explains the sources and likely direction of error. I know it must have been tempting to come up with your own estimate of the population after all that work, but given the data limitations you describe, I think your conclusions are sound” (personal communication, August 8, 1983).
We had our differences. I put much more emphasis on deliberate external obstructions to use than did Carol, worried more about how to foresee and neutralize them, and used the design phase as an “infinitely expandable resource” (her words) to predict the political pitfalls ahead. But we agreed on many other things, in particular, the importance of realistic (vs. idealistic) perceptions and the duty of the evaluator to be honest and to try to increase honesty in the policy-making process. This was not so much for virtue’s sake, perhaps, but because of the grievous and dangerous mistargeting of government resources that dishonesty engenders. Just recently, at the end of July, she e-mailed me that she shared the concern that Saville Kushner, Bob Stake, and I had evoked at the American Evaluation Association's annual meeting about the “single narrative,” that is, the deliberate suppression of evidence which now increasingly occurs in the formulation of policy questions that evaluators are expected to answer. She wrote that she wanted to join any effort we might be planning to bring the problem to wider public attention.
Carol Weiss leaves a major legacy in evaluation. In addition to her substantive work and teaching, I think that her pioneering spirit and enlightenment mode of reasoning opened many doors for many people. She loved evaluation and acted as a mentor to evaluators generally, not just her own students. For me, she was a trusted advisor and a longtime friend whose intellectual curiosity, political sensitivity, and rounded methodological scholarship made every conversation with her a joy. Hers was a unique combination of wisdom, worldliness, and wit. How we will miss her!