The Roles Foundations Are Playing in the Evidence-Based Policy Movement

Collaborative Funding

Foundations promoting evidence-based policymaking may work with each other and public grant makers to create a pool of funds to support specific activities or grantees. This approach contrasts with a more typical scenario in which grant recipients cobble together support from a mix of funding sources. Collaborative funding thereby reduces the cost of fundraising for nonprofit organizations that receive grants, allowing them to focus on their missions. It also permits investments that exceed the capacity of individual funders—though it may risk steering resources toward a small set of initiatives at the expense of other worthy efforts.

Collaborative funding arose in part because of a push from the public sector. Several evidence-based, competitive federal grant-making programs launched during the Obama administration, including Investing in Innovation (i3) and the Social Innovation Fund (SIF), required a private sector match. SIF required that federal funds be matched by three times as much private sector funding. Federal money was awarded to intermediary funders who had to match the awards dollar for dollar. That pool of funding then had to be matched dollar for dollar once again when awarded to the nonprofit “subgrantees,” who implemented and evaluated their innovative social programs. For example, the Edna McConnell Clark Foundation (EMCF) received three $10 million SIF grants as an intermediary, matched that with $30 million of EMCF money, and recruited fourteen other “coinvestors,” who contributed an additional $56 million to a pool that was eventually awarded to twelve subgrantees, who then raised the final $4 million that the SIF required (Haskins and Margolis 2014).

The strategy of pooling public and private funding to advance the evaluation of innovative social programs is not new. MDRC (a social policy research firm) and the Ford Foundation created an early example—the Work Welfare Demonstration—in the 1980s to evaluate work-based welfare programs in eight states. Ford committed $3.6 million (half the estimated budget) in a challenge grant. States used those funds in several ways to generate a match from federal funds meant to reimburse them for the administrative costs of launching innovative welfare programs, including the costs of studying the effects of the innovations (Gueron and Rolston 2013, 596).

Although public-private collaborative funding is not new, and although it has advantages for both grantees and funders, it remains uncommon. Funders’ interests often differ, and collaboration requires some loss of autonomy and control—especially in public-private partnerships where procurement rules require the public side to control funding decisions. But the Work Welfare Demonstration, i3, and SIF show that creative matching strategies can limit the downsides of collaboration for private funders and grantees while preserving its benefits.

Supporting Intermediaries

As recently as 15 years ago, it was common for foundations to fund public agencies to advance reforms (Reckhow 2015). But partly because of the perceived failure of prominent initiatives such as the Chicago Annenberg Challenge or the more recent Zuckerberg award to the Newark Public Schools, this direct approach is in decline (Russakoff 2015). Foundations are now likely to support intermediary organizations that advance the market for certain ideas or approaches by promoting the ideas and connecting the parties needed to make the market functional.

The William T. Grant Foundation followed this strategy from 2005 to 2014 with a series of modest grants to the Coalition for Evidence-Based Policy, notably to support the work of its founder, Jon Baron. (Baron is now a vice president at the Laura and John Arnold Foundation and an author in this volume.) The coalition was a consistent and early voice for using well-designed evaluations to assess the effectiveness of programs and policies. Its influence came partly from Baron’s willingness to dig into the details of public regulations and legislation. He often drafted suggested language and helped federal agency staff with advice or reviews (Coalition for Evidence-Based Policy 2015).

More recently, Results for America (RFA)—funded by Bloomberg Philanthropies and the Edna McConnell Clark, Hewlett, Laura and John Arnold, George Kaiser Family, and Annie E. Casey foundations—has emerged as a leading advocate for using evidence of effectiveness to guide public funding. While the coalition is nonpartisan, RFA is assiduously bipartisan in its approach and public positions. For example, the former heads of the Office of Management and Budget (OMB) in the Obama and Bush administrations coedited RFA’s call to action Moneyball for Government, and high-profile centrist Republicans and Democrats coauthor its blog posts and position papers.

Building the Infrastructure

Executing evidence-based policy requires an infrastructure that gives policy-makers and practitioners accurate information that they can use to understand current problems and the results of attempts to solve them.

Foundations have focused on several weaknesses in this infrastructure, including a lack of reliable measures for important practices and outcomes, gaps in the data and shortcomings among the tools for designing and evaluating programs and policies, and a lack of accessible information about studies conducted to date.

K–12 education shows how an improved infrastructure can change the understanding of a problem. The last two reauthorizations of the federal Elementary and Secondary Education Act (No Child Left Behind in 2002 and the Every Student Succeeds Act in 2015) emphasized the school as the unit of reform, with the idea that we can make progress by improving low-performing schools. Once longitudinal, student-level data became available for multiple states, however, a classroom effect became evident to foundations interested in education reform. That is, within individual schools, similar students may perform very differently on standardized tests. Further, data linking student school performance to the labor market showed that achievement tests do a poor job of predicting success in employment. Students’ noncognitive skills such as persistence appeared to better predict later success. Taken together, these two findings are shifting the interests of policy-makers and researchers to the effects of individual teachers rather than schools and to skills that are not captured by tests of educational achievement. Yet the new interest in teachers and noncognitive skills is not yet manifest in policy, partly because of gaps in the infrastructure.

One of those gaps is a lack of good measures of teaching and noncognitive skills. Policy-makers need measures that can be widely deployed to support accurate inferences, and foundations have grown interested in helping to create such tools.

For example, in 2007 the William T. Grant and Spencer foundations realized that we lacked measures that could be widely used by policy-makers to understand what goes on among teachers, youth workers, and young people in schools and youth organizations. They committed $1 million apiece, issued a request for proposals for teams to develop such measures, and made several grants. Shortly thereafter, Tom Kane of the Gates Foundation met with me and my counterpart at the Spencer Foundation and described Gates’s plan to commit $45 million to develop ways to measure effective teaching (Cantrell and Kane 2013). More recently, a number of organizations—the Einhorn Family Charitable Trust, and the Gates, Hewlett, Mozilla, Overdeck Family, Raikes, S. D. Bechtel, Jr., Spencer, John Templeton, and Wallace foundations—formed the Funders’ Collaborative for Innovative Measurement to make progress on measuring students’ noncognitive skills (Wujcik 2015).

Funding does not guarantee that policy will change, because it may be impractical to implement what is learned. Work on how to measure teaching practices clarified that the number of ratings, the timing of the ratings, and the raters themselves influenced the teacher’s score. To produce accurate scores requires an average of three to four ratings from different dates and raters. But schools and principals rarely have the time or resources to measure teaching well (Bell et al. 2012). And assessments of teaching and students’ noncognitive skills become less accurate when they are used to make high-stakes decisions about performance (Duckworth and Yeager 2015). Policy-makers have thus been cautious about deploying such assessments.

Foundations are also working to make individual-level longitudinal data more widely available for designing and assessing reforms. Thirty years ago, to understand the composition of the welfare caseload over time and to design and assess the effects of welfare reforms, the early welfare reform studies used records of welfare payments and earnings that had been collected for administrative program purposes (Gueron and Rolston 2013). Because it is cheaper to use already available administrative data for planning and evaluation than to collect new data, foundations such as the Laura and John Arnold Foundation (LJAF) have successfully encouraged the use of administrative data in low-cost evaluations of the effects of reforms.

But administrative records present a number of pitfalls. For example, records on employment and earnings, public assistance, school performance, health, child welfare, housing, and criminal justice are separately collected and rarely integrated. Also, states and local jurisdictions often define policy-relevant information differently (e.g., what constitutes high school dropout and graduation levels) and they do not track people once they leave the jurisdiction or state. And because these are records of individuals, privacy must be protected.

LJAF, Gates, and other foundations are working with federal, state, and local agencies to better integrate administrative data and make them more available to policy-makers and evaluators by creating large, integrated, publicly available data files for problem analysis, program planning, and evaluation. One example is the Stanford Education Data Archive (SEDA), supported by the Spencer, William T. Grant, Overdeck, and Russell Sage foundations. SEDA contains data on educational conditions, contexts, and outcomes in schools and school districts across the United States. It includes measures of academic achievement, achievement gaps, school and neighborhood racial and socioeconomic composition, school and neighborhood racial and socioeconomic segregation patterns, and other features of school systems (Stanford Center for Education Policy Analysis 2016).

Another issue with the infrastructure is a lack of tools and statistical techniques for answering important policy questions. In the vernacular of the evidence-based movement, policy-makers want to go beyond learning “what works” to understand “for whom, under what conditions, and why.” Policy-makers are particularly frustrated when the research community cannot give them clear information about why and under what conditions a reform will succeed or fail. These questions become paramount when results vary across different studies of the same or similar programs (Raudenbush and Bloom 2015).

In response, the William T. Grant Foundation funded a decade-long collaboration between Stephen Raudenbush of the University of Chicago and Howard Bloom of MDRC. They and their colleagues conducted basic methodological work, developed software and analysis techniques, and conducted extensive training at meetings of researchers. The researchers started by working to understand how to appropriately design assessments of the effects of reforms delivered to groups of people (e.g., whole school or whole classroom reforms). They then moved on to problems with measurement in such settings, and to estimating the effect of a short-term change on longer-term outcomes (e.g., if an intervention improves graduation and subsequent earnings, was the effect on earnings caused by the effect on graduation or something else?). In 2014, the Spencer Foundation extended the collaboration between Raudenbush and Bloom. They are now working with methodologists from other universities and research firms to understand why programs’ effects so often vary across locations and participants. (See Bloom [2005]; Bloom et al. [2016]; Martinez and Raudenbush [2008]; Raudenbush and Bloom [2015]; Raudenbush, Martinez, and Spybrook [2007]; and Weiss, Bloom, and Brock [2014] as good examples of the many publications that this collaboration has produced.)

State and local policy-makers also need accessible and clear information about prior results. To meet this need, the U.S. Department of Education set up its What Works Clearinghouse, and a number of foundation-supported clearinghouses have also emerged. They include the Results First Clearinghouse Database funded by the Pew Charitable Trusts and the MacArthur Foundation, and Social Programs That Work, which was begun by the Coalition for Evidence-Based Policy and is now part of LJAF’s Evidence-Based Policy Initiative (Pew-MacArthur Results First Initiative 2015). ² All the clearinghouses are limited by the information in the evaluation reports they summarize. This reduces their relevance for policy-makers and practitioners. For example, few evaluation reports provide information on the cost of an innovation, the resources needed to put it in place, the contexts where it is likely to succeed, and the funding streams that will support it.

Moving toward Relevance and Continuous Improvement

In the interviews I conducted, the most common theme was that researchers and their funders need to do a better job meeting the needs of practitioners and policy-makers. Several interviewees felt that reforms are developed without enough understanding of the systems they are meant to improve. One common critique is that funders have focused too much on the supply side of the market and not enough on the client or demand side. Unless we build practitioner and policy-maker demand, most interviewees felt, evidence-based policy could fade away.

This belief has led to great interest in collaborations among practitioners, policy-makers, and researchers to solve practical problems. The form of such arrangements varies. In some cases, foundations support research alliances patterned after the University of Chicago Consortium on School Research. Most of these are based at elite private universities, focus on large urban school districts, and are supported by a mix of foundation and public funds. In a second strategy, foundations support research-practice partnerships driven by the needs of the practitioners. This approach has grown rapidly, leading to the establishment of a National Network of Education Research-Practice Partnerships (NNERPP) with more than twenty members, funded by the Annie E. Casey, Laura and John Arnold, Spencer, Wallace, and William T. Grant foundations. NNERPP “aims to develop, support, and connect partnerships between agencies and research institutions in order to improve the relationships between research, policy, and practice” (Kinder Institute for Urban Research 2016). A third strategy is to embed evaluation in a state or local jurisdiction to help policy-makers understand the nature of persistent problems and assess locally crafted solutions. Examples include The Lab @ DC and the Rhode Island Innovative Policy Lab, created to advance evidence-based policy in their respective jurisdictions. The LJAF was instrumental in launching both initiatives (Laura and John Arnold Foundation 2016).

Interviewees also expressed frustration with the cost and timing of the phased trial research model, borrowed from medical research, where reforms or innovations must successfully pass through a series of evaluations before they are “taken to scale.” That cost and time might be tolerated if the approach produced an appreciable number of successful, durable, and cost-effective reforms. But so far it has not, leading most of the interviewees to contemplate continuous improvement approaches, in which researchers and practitioners work together to understand a problem and then rapidly test possible solutions. (Though it increases speed and lowers costs, this approach often sacrifices some of the certainty that outcomes are changing because of changes in policy or practice.) The Carnegie Foundation for the Advancement of Teaching, led by methodologist Anthony S. Bryk, is the most visible proponent of this approach in education (Bryk et al. 2015). Bryk and others draw heavily on the work of the Institute for Healthcare Improvement (IHI), which pioneered continuous improvement in health care.

Looking Forward

The evidence-based policy movement is facing a transition. Until the last few years, most public and private funders focused on conducting and implementing well-done evaluations to identify programs and policies that could consistently produce positive results at an acceptable cost. That goal has proved elusive, and funders now tilt toward helping state and local policy-makers and practitioners make continuous improvements using local data. This shift will not be a panacea, given how little we know about the resources, personnel, and practices needed to implement it. But it is understandable given the recent track record of evidence-based policy.

Foundations are facing a number of challenges: a federal administration that is unlikely to allocate grant funding based on evidence of effectiveness, a cascade of mixed or null findings from Obama-era efforts, a restive practitioner community that has not seen strong benefits from rigorous evaluations, and a pervasive belief that evidence-based policy must become less costly and more nimble. It may be worth recalling the situation that produced the welfare reform studies that are often described as an iconic example of a successful evidence-based policymaking, and the nature of those studies.

The Work Welfare Demonstration was launched shortly after President Reagan’s election. Like the current administration, the Reagan administration was distinguished by a belief that it knew how to solve social problems and did not need research to point the way. And like today, state- and local-level policy-makers and senior practitioners were under increasing social and fiscal pressures to use their resources wisely.

These policy-makers became convinced that it was possible to learn from evaluations that tested changes before implementing them widely. The authority to create these reforms was codified in law, and the funding for their evaluation came through existing, predictable, formula-funding streams, rather than competitive grants. Several of my interview subjects knew this history and argued that we need to get back to practitioner-designed solutions. They also said that the current federal environment is shifting their focus to state and local jurisdictions, where they hope to find willing partners.

This version of history supports many of the directions foundations are taking. But it is important to recall some of the limits of the 1980s Work Welfare case as a guide for current practice. Most of the state-level tests were of simple, easily implemented reforms that did not exist in common practice. In contrast, many current reforms—such as professional development for teachers, teacher evaluation systems, intensive case management and counseling for at-risk youth, and high-quality preschool programs—are difficult to implement well. And while the 1980s reforms were modest in design (and in results), they were distinctly different from the welfare system at the time. Today, some version of a reform’s services often exists in the status quo. For both these reasons, it may be harder now than before to produce discernible, replicable effects.

Like many of today’s reforms, later welfare reforms in the 1990s and beyond became more ambitious and complicated, and their evaluations produced more varied results than the consistently positive findings in the initial studies. To better understand those results, the newer studies demanded data on implementation, outcomes such as family income that were not available in administrative records, and the services received by members of the intervention and comparison groups. Collecting and analyzing such additional data added cost and time that was justified because policy-makers had more ambitious goals for the studies than a simple thumbs up or down. But in the face of slow progress and the erosion of formula funding for evaluations, policy-makers and practitioners became less enthusiastic—a cycle that appears to be repeating itself.

All these factors justify foundations’ current emphasis on meeting the needs of practitioners, making evaluation less costly by increasing access to administrative records and other tools, supporting new arrangements between practitioners and researchers, promoting continuous improvement, and favoring local solutions and local data. Yet even as foundations support those trends, they must temper expectations that the movement will produce breakthrough solutions to persistent social problems. Foundations must also remain committed to gathering reliable information on the cost and effects of promising innovations. Most likely, the progress we make will be incremental and will demand the same thoughtful, data-driven, and continuous approach that foundations are suggesting for others.