Abstract
The nation’s lawmakers have frequently voiced the basic principle that important policy decisions should be evidence based. In this commentary, the authors describe the approach the U.S. Department of Education has taken in its Increasing Educational Productivity project. The authors argue that the department’s actual practice in this instance has fallen short of the rhetorical embrace of evidence-based decision making, and they explain the potential harm done when leaders do not heed the importance of grounding policy in high-quality research.
Advocating the idea of improving the productivity and efficiency of America’s public schools, the U.S. Department of Education and Secretary of Education Arne Duncan have repeatedly promoted a so-called new normal for public schools. “It’s time to stop treating the problem of educational productivity as a grinding, eat-your-broccoli exercise. It’s time to start treating it as an opportunity for innovation and accelerating progress” (Duncan, 2010; see also Stratman-Krusemark, 2011). The premise of the new normal idea is that tight budgets are, for the next several years at least, going to remain in place and are therefore going to demand that schools “do more with less.” As part of the Department of Education’s effort, it unveiled on its website a series of materials and supporting documents explaining how public school districts can stretch their dwindling dollars, live within the new normal, and become more productive and efficient (U.S. Department of Education, 2011).
In this commentary, we call upon the U.S. Department of Education to embrace fully the goal of shaping policy through knowledge gained from high-quality research, and we offer suggestions on how this can be accomplished as regards productivity research. At a time when states and districts are cutting budgets and need help to avoid indiscriminate programmatic cuts, the department and the secretary of education have reason to focus on educational productivity. But we contend that the materials provided on the department’s website as guiding resources present poorly supported policy advisement. The materials listed and recommendations expressed within those materials repeatedly fail to provide substantive analyses of the cost-effectiveness or efficiency of public schools, of practices within public schools, of broader policies pertaining to public schools, or of resource allocation strategies. 1 Instead, the sources listed on the site’s “resources” page are speculative think-tank reports and related documents that do not include or even cite the types of analyses that would need to be conducted to arrive at their conclusions and policy recommendations. 2
These omissions are particularly troubling because high-quality research in this area is available and, if used, would provide the sort of policy guidance the department is ostensibly seeking. Accordingly, this commentary concludes with a call for the department to create a national consortium to accomplish two goals: (a) to summarize what is known about educational productivity, efficiency, and cost-effectiveness and to translate the academic research for policy makers, with a focus on the implications for practice, and (b) to develop and pursue a research agenda that will provide information concerning promising practices as well as popular but untested new approaches.
Sound Productivity Analyses Are Feasible
Education productivity has in the past been explored through two branches of academic research. The first involves studies of the production and cost efficiency of schools and school systems. The second looks specifically at the cost-effectiveness of educational models, strategies, or broad-based reforms.
In the first branch—efficiency analysis—researchers attempt to use statistical models that capture the complexity of the real world to identify the outcome levels that can be produced with certain students, given certain levels of spending, or alternatively to determine the spending levels that would be needed to produce a certain level of outcomes with certain students. Some schools and districts do better than expected and others do worse. The ultimate goal is to figure out why. Researchers have addressed the conceptual basis and empirical methods for evaluating the technical efficiency of production and cost efficiency in education or government services more generally (see Bessent & Bessent, 1980; Duncombe, Miner, & Ruggiero, 1997; Grosskopf, Hayes, Taylor, & Weber, 2001; Levin, 1974).
In the second branch—cost-effectiveness analysis—researchers identify the various costs of implementing alternative programs, models, or strategies intended to improve the same outcome measure. These researchers then compare those cost estimates to the differences in outcomes achieved, under highly controlled conditions. The ultimate goal is to determine which strategy most improves the outcome measures in question for a given cost. That is, the researchers ask which of these strategies is most cost-effective (see Levin, 1983; Levin & McEwan, 2001).
These established bodies of scholarly research were, however, largely ignored by the Department of Education in its Increasing Educational Productivity project. We have no inside knowledge about how or why this research was passed over. Ideology and expediency may certainly have played a role. It is also true that most of the best research on educational productivity, efficiency, or cost-effectiveness is found in scholarly journals, which are not particularly accessible to the average lay reader or the average policy maker. But the capacity certainly exists within the department to access this work or those familiar with the work.
It is true as well that less nuanced presentations of research are more likely to be used by lawmakers (Hennigan, Flay, & Cook, 1980), whereas the best peer-reviewed, academic research on educational productivity and on cost-effectiveness is methodologically meticulous and does not provide simple answers or one-size-fits-all solutions to complicated problems. A parallel literature produced by advocacy organizations, on the other hand, does tend toward such uncomplicated answers yet is rarely backed by high-quality evidentiary support.
But the inaccessibility of the scholarly literature is also linked to an important strength: The research is carefully conducted and vetted by peers with considerable expertise in research methods and other relevant scholarship. It considers alternative explanations and approaches, and it is cautious in drawing implications from complex but incomplete and imperfect models. In the end, good research conducted using exacting methods leading to appropriately qualified policy recommendations can be made accessible and can indeed be used to inform policy. Policy makers need not trade rigor for accessibility.
High-Quality Research Can Guide Policy
As noted above, the U.S. Department of Education posted a set of resources in the spring of 2011 that it identified as key readings in educational productivity. Yet none of the resources listed on the site actually include, or even provide citations to, rigorous empirical research along either branch of inquiry identified above. When productivity analyses are foregone or poorly carried out, the result can be the promotion of policies with little if any evidentiary support. The result can also be the promotion of a large-scale implementation of unproven strategies equally likely to do harm as to do good. 3 Further, this experimentation is most likely to occur on our nation’s most vulnerable children, who attend schools and districts systematically labeled as failing and in need of “reform,” however designed.
This danger is well illustrated by a common thread in the materials posted on the Department of Education website: that policies centered on leveraging existing resources toward the goal of improving teacher quality are necessarily more cost-effective than policies centered on increasing teacher quantity (see Petrilli & Roza, 2011; Miles, 2010). That is, improving teacher quality will, according to these materials, cost less and produce better outcomes than reducing class size. Of course, “improving teacher quality,” like “improving student outcomes,” is hardly a policy proposal. Rather, it is a policy objective that policy makers and practitioners have long struggled to achieve. So the first step toward determining cost-effectiveness regarding improving teacher quality is to identify the specific, testable policy options intended to improve teacher quality. For instance, one common assertion is that paying teachers based on years of experience and degree levels is inefficient and that if teachers were instead paid additional salary only on the basis of student performance (generally, a growth in test scores), efficiency would improve.
This assertion has not been directly tested, but it is testable. It would be an interesting, although difficult, experiment to compare two options: (a) a group of schools where teachers are compensated only on measures and ratings of effectiveness and (b) a comparison group of schools with all else equal (students, total resources, etc.) but where teachers are compensated according to traditional salary schedules. This would be a bold and complicated endeavor, but the existing research base does not support the assertion that performance-based pay will produce positive outcomes, much less whether it would produce positive outcomes at comparable cost. Existing studies on performance pay show little or no short-term effect, and even if studies were to find small effects that are statistically significant, they may not be economically significant (Glazerman & Seifullah, 2010; Marsh et al., 2011; Springer et al., 2010).
If, however, researchers were to carry on this experiment long enough, they might be able to evaluate whether these alternative salary structures have different long-term effects on outcomes such as recruitment and retention of teachers and in turn on school quality. Then, if such outcomes emerged, the next step would be to determine whether putting a specific amount of additional funding into (a) performance-based pay or (b) class size reduction (or something else) would be more cost-effective.
A related policy proposal arises when teachers are laid off due to budget cuts. In one of the reports that the department recommends, Petrilli and Roza (2011) suggest that productivity will increase if teachers are laid off according to measures of teaching “effectiveness” as opposed to seniority. Yet, aside from a handful of recent hypothetical simulation studies, little evidence exists regarding the actual effectiveness gains that might result from layoff policies using some measure of effectiveness, and little evidence has been compiled regarding potential short-run and long-run costs of such policies. Further, no high-quality evidence exists regarding how either of these strategies compares with or potentially interacts with changes to, for example, class sizes. These are empirical questions that are complex and that require long-term analysis framed either in terms of cost-effectiveness or relative efficiency.
We want to be clear here that we are neither opposing innovation nor supporting the status quo. Yes, we are calling attention to the importance of careful productivity analyses and to the reality that these analyses take time to carry out. But schooling practices and policies can and should exist within different categories. When they are experimental and unproven, they should exist in pilot form. If they are shown through those pilot studies to have beneficial effects, irrespective of cost, they can safely be scaled up with the knowledge that children will not be harmed. Finally, if they are shown through productivity analyses to be efficient or cost-effective, policy makers would have a sound basis for pushing widespread adoption.
Accordingly, it is reasonable for policy makers to move ahead in adopting a policy based on positive outcomes (effects) alone and then to evaluate costs and compare cost-effectiveness against other options over time. Our concern is that the Department of Education is pushing widespread adoption of specific reform agenda items in a way that is far outpacing even effectiveness research, let alone productivity research, on those items. Meanwhile, the department has continued to disregard those areas that have been more thoroughly researched. We see little justification for pushing widespread policy adoption without support from either effectiveness research or productivity research.
Further, the political rhetoric around the immediacy of reform focuses on so-called failing schools, and failure is identified through performance metrics heavily influenced by student demographics. Simply put, we have ethical concerns with imposing unproven and sometimes unstudied policies on schools in low-income communities of color. And this is what we see happening.
Proposal for a National Consortium
As one way to accomplish the U.S. Department of Education’s stated goal of bringing high-quality productivity information to and for the nation’s schools, we recommend that it seek guidance from leading published scholars in the areas of understanding and measuring education costs, productivity, and efficiency. By collaborating with experts and relying on existing high-quality research, the department can develop an agenda focused on a balance of five key factors: (a) improving empirical methods and related data; (b) evaluating major education reform models, programs, and strategies; (c) disseminating the results of those evaluations; (d) expanding and improving stakeholder understanding of cost-effectiveness, cost–benefit, and relative-efficiency analyses; and (e) supporting the training of future scholars in these methods.
This could be done through various institutions and approaches, but we suggest a consortium effort directed by the department, with a clear mechanism set in place at the outset for the use and application of the consortium’s work product. This could build on work the department has done in the past—supporting relevant research and producing publicly accessible volumes of high-quality research. 4 Alternatively, an independent review panel might be established by the National Research Council, which also has in the past convened panels on topics concerning school finance and has supported research and produced research syntheses. 5 But regardless of the mechanism, these new efforts should focus on taking the next step of translating and communicating high-quality research for school districts, state policy makers, and other end users.
We also suggest that the consortium researchers be selected based on a track record of relevant peer-reviewed publications and national and international leadership roles, including membership on editorial boards of relevant, major peer-reviewed academic journals. Although we acknowledge that credible experts and credible research exist that are not constrained by these requirements, all indications are that the department has thus far relied overwhelmingly on work that is not peer reviewed, most of which is neither credible nor rigorous.
The scholars forming a national consortium on the topic of educational productivity should be asked to summarize what is known from academic research and to translate that research for policy consumption. The efforts should focus on the implications of the research for practice and the operations of schools and districts. The scholars should also be asked to develop short- and long-term agendas for cost-effectiveness analysis, cost–benefit analysis, and relative-efficiency analysis.
These productivity inquiries should be approached systematically and rigorously, with no unrealistic expectations that facile solutions will miraculously emerge. In doing so, the Department of Education would embrace fully the goal of shaping policy through knowledge gained from high-quality research.