American Journal of Evaluation

A theory-driven approach to evaluation (TDE) emphasizes the development and empirical testing of conceptual models to understand the processes and mechanisms through which programs achieve their intended goals. However, most reported applications of TDE are limited to large-scale experimental/quasi-experimental program evaluation designs. Very few (limited) examples of the relevance of TDE to nonexperimental program evaluation designs exist in literature. Using the method of structural equation modeling to analyze data from the Interns for Indiana (IfI) program, this study demonstrates how evaluation practitioners can test logical and sequential relationships among tiers of outcomes of nonexperimental programs, especially programs with limited datasets. The study also describes how the empirical feedback can be used to understand program dynamics and improve program implementation and evaluation.

Evidence-based policy-making and other recent reforms in public steering emphasize the role systematic evidence can play in improving decision making and public policies. Increasing deficits heighten the pressure on public authorities to legitimate public spending and to find savings. Existing studies show that the influence of research-based information on decision making is shaped by several factors, but they typically do not distinguish between different types of information. Our contribution aims to compare the influence of efficiency analysis to information about performance effectiveness. We do so by looking at 10 cases in which public policies are being revised at the federal level in Switzerland, and do so by tracing the entire policy reform process. This qualitative analysis sheds light on which actors use efficiency information, how and under which conditions, and highlights the contribution of efficiency analysis for evidence-based policy-making.

The authors present a case study examining the potential for policies to be “evidence-based.” To what extent is it possible to say that a decision to implement a complex social intervention is warranted on the basis of available empirical data? The case chosen is whether there is sufficient evidence to justify banning smoking in cars carrying children. The numerous assumptions underpinning such legislation are elicited, the weight and validity of evidence for each is appraised, and a mixed picture emerges. Certain propositions seem well supported; others are not yet proven and possibly unknowable. The authors argue that this is the standard predicament of evidence-based policy. Evidence does not come in finite chunks offering certainty and security to policy decisions. Rather, evidence-based policy is an accumulative process in which the data pursue but never quite capture unfolding policy problems. The whole point is the steady conversion of “unknowns” to “knowns.”

In this paper we describe an evaluation training program sponsored by the Robert Wood Johnson Foundation and led by Duquesne University and OMG Center for Collaborative Learning designed to meet the challenge of developing a cadre of diverse evaluation professionals, specifically those from traditionally underrepresented or underserved communities, who have come to the field with little formal academic training in evaluation. Our interview reveals that an essential strength of this work is the collaborative relationship between an academic institution and a non-profit firm. Three additional discussion points emerged that are important learnings from this work. First, post-degree practicum training models must consider not only the training of participants, but also organizational transformation. Second, there is a genuine tension between expanding the program and the seemingly necessary responsive and intimate nature of educational programs intended to develop communities of practice, and a third and related point, are considerations of program sustainability.