Abstract
Carol Weiss’s fine sense of humor was, at times, mordant. She would not then object to my declaring that I have become, on her account, more practiced at writing obituaries than I would prefer to be.
Here, I honor Carol not by reminiscence. Nor do I praise Carol here by reminding readers of her astonishing (annoyingly high?) productivity in evaluation research and her contributions to sociology and applied social research in the United States and other countries. Various papers and books of hers have been translated into eight languages, to judge from entries in Worldcat.org. She received awards from the American Evaluation Association and the Policy Studies Organization, among others, for the work. I will not cover her influence on criminology because others have done well in this; see http://ww.asc41.com/obituaries_home.html. And I will not comment on her deliciously skeptical spirit, except indirectly in what follows. Instead, I would like to honor Carol by extending a few of her ideas, ones that might not be recognized or exploited sufficiently.
First, consider Weiss’s (2002) “What to Do until the Random Assigner Comes.” Carol’s invited presentation on the topic was given at a Harvard conference on the topic of randomized controlled trials in education. It became the last chapter in a volume born of the conference. Carol got the last word, though she had not been an energetic advocate of such trials. One of the merits of Carol’s chapter was the production of two exceedingly complex diagrams and the prose to go with them. One portrayed how a comprehensive community-based program might eventually have positive effects on the health and productivity of adolescents. Presumptively casual arrows were numerous. Their number, 50 or so, would confound and frustrate the conscientious path analyst or the abiding causal modeler in any of the statistical disciplines. It was nonetheless an explicit representation of a reality as Carol saw it: How the right ingredients, introduced at the right times and in the right ways, could result in the right outcomes.
The second diagram embodied Carol’s more creative and skeptical spirit. She laid out how the comprehensive community program might fail, or indeed result in negative effects, such as more violent youth. It was not a happy picture, nor is it an easily comprehended one. Again, our path analytic colleagues would have gone nuts. Nonetheless, it was explicit which is to the good. Better, it reminded readers that the prospect of failure is real. Most important, it suggested how one might arrange one’s thinking about how failure might occur. Imperfect, but provocative.
Contemporary “logic models,” theories of change,” “program theory,” and so on do not consider the second kind of diagram at all. To judge from Weiss’s (2013) “Rooting for Evaluation,” we often do not even get the first one laid out well. Indeed, the provincial experimentalist might merely abide by formal statistical tests of a null hypothesis that yield dependable estimates of effect size for the program and a confidence interval for the estimate. But statistically, many experimenters understand that the results of such a test often tell us nothing about what to do when we cannot reject the null hypothesis because they were not designed to do so.
More generally, the social, behavioral, and education sciences lack orderly approaches to anticipating failure and planning for it. Weiss (2013) opined that “program theory was particularly suitable to cases in which randomized experiments are inappropriate” (p. 135). In this, we would have disagreed. But we did not have a chance to do that. Moreover, it is partly on Carol’s account, and on account of others who include her mentors, and colleagues such as Lois-ellin Datta, and on account of antecedents such as Pressman and Wildavsky that new work is being undertaken to confront the prospect of failure with a bit more temerity (aka guts) in the context of experiments as well as in nonrandomized evaluations of the impact of programs. See, for instance, Boruch and Ruby (in press) and the references therein. As the foregoing suggests, Weiss was no slouch in creating tantalizing titles. Weiss, Murphy-Graham, Erin, Anthony, and Gandhi (2008) “Fairy Godmother-and Her Warts” is illustrative. The article’s contents reflect her abiding interest in understanding when and how results of dependable evaluations are used and are not.
The Fairy Godmother (TFG) was described as being in the service of the presumably optimistic evaluator in her paper. TFG is the government agency that takes dependable evaluative evidence seriously. In TFG piece, Weiss and her colleagues focused on Drug Abuse Resistance Education (D.A.R.E.) which, in the United States, has been a fashionable and popular approach to drug use prevention over a 20-year period (Petrosino, 2005). Taking D.A.R.E. seriously in the face of considerable popular support, learning about and reporting on its negligible effects, have been no easy matter to judge from Petrosino’s, Weiss’s, and others’ discussions with D.A.R.E. folks.
More important, Weiss et al (2008) got deep into the weeds to try to understand how federal reviews of evidence on such programs and how regulations about funding of such programs at the school level might be made to depend, or not, on the evidence in related sectors. The bottom line for Weiss and her colleagues, in this instance, is that the imposed use of evidence has some merit. But the devil lies in the details and the prospect of people’s paying attention is not high.
If not disgruntled, Weiss (2013) was not entirely gruntled with some of her fans’ interpreting her work to declare that enlightenment was the most one could expect from good evaluative evidence, given the other factors that influence the use of results of such evidence. Nonetheless, she was the only essayist in Alkin (2013) to register a sense of bemusement and to provide readers with pictorial amusement on this account. See the reproduction of the New Yorker cartoon depicting two monks, one of whom is saying to the other: “I am so past enlightenment” (p. 139). Carol described the origin of some of her work in evaluation and on the use of evidence in Weiss (2013). One might consider the pedigree as being older, and at least as prestigious and direct on historical grounds.
That pedigree might be dated back to John Graunt’s (1662/1939) Natural and Political Observations Upon the Bills of Mortality in which he compiled the first ever tract on statistical evidence for England. In a closing chapter, Graunt posed the question: “…to what purpose tends all this laborious bustling and groping……to know the number of people, fighting men, teeming women, what years are fruitful…?” (p. 96).
In responding to his own question, Graunt said: “To this I might answer in general by saying that those who cannot apprehend the reason are unfit to trouble themselves to ask…” Further, he declared “…it is much pleasure in deducing so many abstract and unexpected inference…,” and finally “…the foundation of this honest and harmless policy (notably his efforts and his tract) is to understand the land and the hands of the territory…for good, certain, and easy government” (pp. 98–100).
Carol Hirschon Weiss was a remarkable colleague and kindred spirit. She will be sorely missed.