Abstract
Benefit–cost analysis is an important part of regulatory decision-making, yet there are questions as to how often and how well it is performed. Here we examine 28 Regulatory Impact Assessments performed by the federal government on education regulations since 2006. We find many Regulatory Impact Assessments estimated costs, albeit using informal methods, but most failed to adequately report benefits. Also, most studies did not estimate net present value or clearly report methodological assumptions. In reviewing the relatively high quality studies we identified a number of discrepancies from best practice. Most importantly, few Regulatory Impact Assessments attempted a social benefit–cost analysis: Most examined “administrative burdens” from compliance with legislation. This alternative focus on administrative burdens has significant implications for economic evaluation in practice.
Introduction
Today’s era is considered “the golden age of evidence-based policy” due to record growth and utilization of rigorous research to address social problems (Haskins, 2016). This movement is intended to develop higher standards and to efficiently allocate resources toward building and replicating programs and practices that work. A critical component inherent in this is examining efficiency through economic evaluations.
Over prior decades, a series of executive orders have been issued to establish a regulatory planning and review process to make federal policy more efficient, that is, in favor of policies where the benefits justify the costs (since Executive Order No. 12,291, 1981). Generally, government utilizes regulation to intervene when market failure exists to ensure that the economy continues to progress according to the values established by society (Nas, 1996). This Executive Order built upon that framework by requiring economic evaluation of the costs and benefits of any significant federal regulation having an effect on the economy of US$100 million or more, an adverse material effect, or an inconsistency with another agency or action (Executive Order No. 12,866, 1993). President Obama reaffirmed this order in 2011 to ensure that the regulatory system protects “public health, welfare, safety, and our environment while promoting economic growth, innovation, competitiveness, and job creation” (Executive Order No. 13,563, 2011). The most recent Executive Order on Regulatory Reform (January 26, 2017) proposes a “two-for-one” regulatory exchange process: Each new regulation can only be adopted if two existing regulations are removed (Peacock, 2016).
Generally, the federal role in education is to guide discussion of the policy agenda, to provide supplemental support for educational programming at the K–12 level, and to provide policies and support for postsecondary education institutions and students. In order to continue to build our use of evidence in policy-making within education and to ensure that our policies are efficient, it is important that we examine the economic evaluations conducted of federal education regulations to continue to improve the quality and relevance of this evidence base.
The primary method for economic evaluation is benefit–cost analysis (BCA; see Boardman, Greenberg, Vining, & Weimer, 2011; Institute of Medicine, The National Research Council, 2014; Karoly, 2012; Levin, McEwan, Belfield, Bowden, & Shand, 2018). As described in detail by Vining and Weimer (2010), this method can be adapted to evaluate social policy interventions, including those in education. Straightforwardly, policy makers and education professionals should decide in favor of policies that have greater benefits than costs. As a tool that guides policy, the quality of BCA becomes crucial: Costs and benefits should be estimated with precision to the extent possible; if elements that are not easily quantifiable are included in the analysis, transparent statements of their assumptions should be made available; and benefits should be mapped into costs in a clear and explicit way. Hence, good quality analysis should yield an economic metric that can guide investment of public (and private) funds. Of course, BCA is only one tool that is available to the policy maker to aid in decision-making; ultimately, the decision maker must make a reasoned determination, recognizing society’s preferences and goals as well as existing practices.
Fundamentally, BCA is justified insofar as it improves the quality of decision-making (R. Posner, 2000; Revesz & Livermore, 2008). If the results of BCAs help to improve policy, these analyses should be performed. In order to improve decision-making, however, BCAs need to be performed to a high methodological standard (see Farrow & Zerbe, 2013).
In this article, we critically evaluate prior BCA of education regulations at the federal level. We begin by documenting the many practical challenges in performing BCA and reviewing the quality of BCAs in other policy fields. We then evaluate the quality of education policy BCAs as performed by the federal government. First, we collate findings from a checklist appraisal of each BCA. Next, we review a subset of (relatively) high-quality BCAs. This appraisal and review illustrates the many ways in which federal attempts at benefit–cost evaluation differ from social BCA. We conclude with discussion of the implications for policy when BCAs fall short of accepted methodological standards.
BCA From Policy to Theory to Practice
BCA plays an important role in determining the efficiency of federal policies. Executive branch agencies promulgate regulations to implement laws enacted by the Congress. Under Executive Order No. 12,866 (1993), each agency must prepare a unified regulatory agenda containing the regulatory plans of the most important significant regulatory actions that it expects to issue in that fiscal year. These plans are then forwarded it to the Office of Information and Regulatory Affairs (OIRA).
Together with the relevant agency, OIRA plays an integral role in reviewing these regulatory plans and ensuring they are not in conflict with other policies. 1 Once OIRA notifies the agency that the review is completed and that there are no further considerations, the regulatory action then becomes a final rule and is published in the Federal Registry. Major final rules must go through Congress to be approved. One important component of this review is the application of BCA; this form of analysis helps policy makers come to a “reasoned determination” as to whether to implement policy.
The basic theory of BCA is well-established. There are several excellent textbook treatments of BCA, its principles, and theoretical foundations (Adler & Posner, 2006; Boardman et al., 2011; Farrow & Zerbe, 2013). Recently, however, it is the practical application of BCA that has received more scrutiny (for an overview, see Belfield, 2015; for how to read BCAs, see Dudley et al., 2017). This scrutiny has highlighted two main concerns—the lack of BCAs and the quality of the BCAs that are performed.
A number of studies have drawn attention to the limited application of BCA. In a recent review, Ellig (2016, p. 4) concludes that “regulatory agencies often adopt regulations without knowing whether a given regulation will really solve a significant problem, whether a more effective alternative solution exists, or whether a more targeted solution could achieve the same result at lower cost.” So, there are not enough BCAs to determine whether the most efficient policy has been implemented and, even when attempted, BCAs are often incomplete. Checklist studies have counted the ways in which BCA is not fully performed. Hahn and Dudley (2007) reviewed 78 regulations and found that only 65% considered costs, 22% considered benefits, and 12% considered benefits minus costs. This work is summarized in (Table 1). This lack of BCA is not just a concern across federal departments. Even as other agencies conform to different standards, their economic evaluations have also been found to be incomplete. For example, in a review of practices at the state level, Schwartz (2010) found BCAs to be very infrequently performed by state legislatures. Also, there has been extensive inquiry into the inadequacy and inconsistency of regulatory impact assessments (RIAs) across the European Union and the United Kingdom (Dunlop & Radaelli, 2016; Fritsch, Kamkhaji, & Radaelli, 2017).
Checklist Evaluation of 28 Federal RIAs.
Note. RIAs from 2006 to 2016 are listed in Appendix Table A1. RIA = regulatory impact assessment.
The second concern is that the BCAs that are performed—even the incomplete ones—are not of high quality. On the one hand, there are many challenges in estimating costs. These include gaining access to proprietary data, identifying resources across agencies, misspecifying business-as-usual, preimplementation or noncompliance behaviors, technological change, and program infidelity (see Harrington, Morgenstern, & Nelson, 2000).
On the other hand, there are also many challenges in estimating benefits. Uncertainty in the estimation of benefits can come from several sources. First, it is unclear how, and the extent to which, institutions comply with regulations. The compliance strategy institutions adopt dictate both the cost structure and the benefits generated by the regulatory action. Second, once institutions have adopted the new regulation, its causal impact on the target population (such as students and institutions) is usually hard to identify. Some policies are not easily amenable to BCA (e.g., if they mostly involve transfers or have strong equity implications that cannot be precisely modeled or necessitate normative approaches). Third, even if the causal estimates are identified, monetization of the benefits (and costs) is subject to the derivation of shadow prices for nonmarket goods (Ellig & MacLaughlin, 2012), which are sensitive to the assumptions used to recover them. Sunstein (2014a, Chapter 5) proposes a number of ways to respond; nevertheless, these challenges must usually be overcome in each BCA.
Moreover, the lack of BCAs makes it harder to perform high-quality BCAs. With few studies, there are even fewer methodological inquiries (e.g., for sensitivity testing or shadow pricing). Also, although there are guidance manuals (e.g., Office of Management and Budget Circular A-4), there are little precedence to help evaluators harmonize with, or compare against, their analysis.
Here, we examine the extent to which these concerns—particularly regarding the quality of BCA—are valid for economic evaluation of education policies and programs. There are two main reasons why it is important to evaluate the practice of BCA in education policy at the federal level.
One reason is that the current quantity of BCA in education is insufficient. Although there are a few high-quality BCAs within education, these focus almost exclusively on the returns to one reform—preschool (Garcia, Heckman, Leaf, & Prados, 2017; Heckman, Moon, Pinto, Savelyev, & Yavitz, 2010; Nores, Belfield, Barnett, & Schweinhart, 2006). Moreover, these BCAs are of small-scale programs that do not enroll many children and have budgets of less than US$10 million; and, typically, they are retrospective in that the preschool programs have already been implemented. By contrast, federal regulations affect multiple cohorts of students across the United States with economic effects of billions of dollars; typically, these regulations require a benefit–cost appraisal that is prospective and so might influence whether and how the policy is implemented. Thus, even within a policy realm where there are too few BCAs generally, education policy receives relatively little attention.
A second reason is that there are specific challenges in undertaking BCAs of educational policies. One challenge is that many educational policies involve large-scale transfers. E. A. Posner (2003) has discussed this aspect at length: Transfers are not amenable to BCA (they almost certainly fail a benefit–cost test because transfers are not benefits); and evaluators have been reluctant to apply cost-effectiveness analysis as an alternative (despite its inclusion in Executive Order No. 12,866, 1993). A second challenge arises because education policies have features that make them, according to Sunstein (2014a), “Hard Cases”: They have significant distributional and equity consequences and/or their benefits are heavily loaded on specific individuals (students). Also, as Vining and Weimer (2010) note, social BCAs require significant sensitivity testing; this leaves open the possibility that an efficiency ruling cannot be clearly determined. Finally, the field of educational research does not place a strong emphasis on BCA. Indeed, in its most recent guidance document on evidence to strengthen education investments, the U.S. Department of Education (2016) does not consider the application of cost analysis or BCA. Thus, even as BCAs are challenging across many policy domains (e.g., environmental and health policy), there are a number of specific challenges for BCAs in education policy.
Method for Evaluating BCAs
Our evaluation focuses on federal education policies for which a BCA is expected. 2 Our evidence is taken from OIRA reports on all 28 education regulations over the most recent decade from 2006 to 2015 that were deemed economically significant and so for which some form of RIA was performed. A regulatory action is defined to be economically significant if the regulation is expected to have an annual effect on the economy of US$100+ million; may create a serious inconsistency or interfere with an action taken or planned by another agency; alters the budgetary impact of entitlements, grants, user fees, or loan programs or the rights and obligations of recipients; or raises novel legal or policy issues. A summary list of the 28 RIAs is given in Table A1 of Appendix A.
Under Executive Order No. 12,866 (1993), the issuing agency must provide an assessment of the potential costs and benefits of all economically significant regulatory actions. The assessment of these potential costs and benefits must be, to the extent feasible, quantified. In addition, these costs and benefits should be compared to potentially effective and reasonably feasible alternatives of the planned regulation. Thus, each RIA should include some attempt at BCA.
We use two approaches to evaluating these education RIAs. The first is a checklist method of all 28 RIAs. Evaluators have criticized checklists and scorecards on several grounds: the use of unclear terms for scoring; ad hoc weighting of scores across items; and the vagueness of using the same scorecard for accountability, communications, and process improvement purposes (Fritsch & Kamkhaji, 2016; Radaelli and Fritsch, 2012). However, this method has been used successfully by Hahn and Dudley (2007) and Shapiro and Morrall (2016). Also, our checklist is adapted to the specifics of educational interventions and looks at the basic requirements for a satisfactory BCA. Thus, it provides only a rudimentary understanding of why RIAs take the form they do and whether that is actually the most appropriate form of evaluation. Therefore, our second approach to evaluating these RIAs is by direct review. We examine in detail the seven most complete BCAs to assess their validity as an economic evaluation of the proposed policy.
Checklist Approach
There is great variation in the way assessment of benefits and costs is done across regulatory actions. Therefore, we try to capture in our data the depth of analysis each planned regulation uses in their BCA. The information is classified and gathered as follows.
Costs
The cost analysis performed in each regulatory report is coded into four levels. These levels represent progressively more extensive cost analyses. First, cost analyses are classified as “stated” if there is any reference in the report to resource use other than budget allocations and if there is some description of how the resources required correspond to the policy being implemented. Second, cost analyses are classified as “ingredients” if there is a list of resource ingredients needed or description of them with detail. Third, cost analyses are coded as “agency” if there is any discussion of costs incurred by other agencies. Finally, cost analyses are coded as “dollar value calculated” if there is a dollar value assigned to cost. This last code is the most extensive type of cost analysis (although we note that regulations can and do calculate costs without considering ingredients or costs to other agencies).
Benefits
Similarly, benefit analyses are also classified into four levels that reflect the depth of the analysis. Analogous to the costs classification, benefits are classified as stated if there is any reference in the report to monetized benefits of the proposed policy. Next, benefit analyses are coded as “described” if there is a description of how the benefits are estimated. Third, benefit analyses are coded agency if there is any discussion of benefits from multiple perspectives. Finally, benefit analyses are classified as “calculated” if there is a dollar value assigned to the benefits.
BCA
In order to capture how benefits are weighed against costs, we attempted to assign two different measures. The first coding is “net present value” (NPV) and refers to analyses where an NPV (benefit minus cost) dollar value is reported. The second coding is “B/C” and is for analyses which plausibly assert that the benefits exceeded the costs (but do not provide a numerical estimate).
Additional information
In addition to the information collected above, one other coding was derived from the OIRA documentation. This coding captured the methodological transparency of the analysis. Studies were coded according to whether the report clarifies assumptions used in terms of interest rates, inflation, discount rates, and time frame. Information about these assumptions is necessary for researchers to adjudicate on the quality of each BCA and to compare results from these BCAs with other analyses.
For each of the 28 RIAs, the information was coded separately by two persons. There were very few discrepancies; these were reconciled in discussions.
Text Review
Given the results from our checklist, we reviewed the RIAs to determine what efforts were made to perform an economic evaluation congruent with Executive Order No. 12,866 (1993). For the text review, we selected the seven highest quality RIAs for detailed interpretation and assessment. (These RIAs are shaded in Table A1 of Appendix A). The review allowed us to assess in detail what type of analysis was performed, how the analysis was structured, and the most salient modeling assumptions and shadow price valuations. These seven specimens are used to assess the quality and features of BCAs—conditional on BCA actually having been performed.
Evaluating Educational BCAs: Summary Findings
Summary findings on the quality of educational BCAs at the federal level are derived from the checklist analysis. These checklist results are given in Table 1. Most of the RIAs perform some form of cost analysis. Almost all (93%) make a statement about social costs, that is, costs beyond simple budgetary allocations. One quarter (25%) make an attempt to estimate costs by distinguishing the quantity of inputs and their prices. Almost one half (54%) investigate cost implications for different government agencies or levels. In addition, approximately three quarters (71%) of the RIAs is a dollar value of costs reported.
The RIA benefit analyses are significantly weaker. Only four fifths (82%) state the benefits that are expected from the legislation and only one third (29%) describe these benefits in some detail. Half (57%) of the RIAs do look at different benefits across government. Notably, only one study (4%) calculated a dollar value of benefits.
Together, these frequencies mean that no RIA calculated a dollar value for the NPV or was able to assert how benefits might exceed costs. Also, we find that only one third (36%) of RIAs adequately documented key methodological assumptions.
For the seven highest quality BCAs, the checklist results are given in Table 2. For these studies, almost all performed a full cost analysis: They estimated resource use using a version of the ingredients method, separated out costs by agency, and reported a dollar value for costs. With respect to benefits, most of the studies stated the benefits, described them, and apportioned them across agency; however, none calculated the dollar value of benefits. Therefore, even across these high-quality studies, no NPV estimate was reported. Also, less than half of these studies clearly reported the assumptions used.
Checklist Evaluation of Seven High-Quality Federal Regulatory Impact Assessment.
As an additional exercise, we reported the checklist results for groups of regulations by year and budget allocation. The results are reported in Tables B1 and B2 of Appendix B. These results show that the quality of BCAs is not improving over time, although there is some indication that higher cost regulations receive a more intensive application of BCA.
Overall, these findings show that most educational BCAs performed by the federal government are incomplete. These findings are similar to reviews of economic evaluations in other sectors (e.g., from Hahn & Tetlock, 2008; Ellig, 2016; and Shapiro & Morrall, 2016). However, we emphasize two points. First, although we contend there are significant informational and policy gains from performing complete BCAs, complete analyses may not be essential for justifying policy decisions. As noted above, policy decisions may be obvious after a basic review of costs and benefits; or policy decisions may be justifiable based on a “reasoned determination,” taking account of the fact that “some costs and benefits are hard to quantify” (Executive Order No. 12,866, 1993). Second, in no way should we conclude that these policies have a negative NPV, rather the magnitude of the NPV—and possibly its sign—is uncertain.
Assessing the Quality of Educational BCAs
We base our assessment of the quality of educational RIAs on a thorough review of seven of the highest quality BCAs. These RIAs included sufficient information for us to assess the quality of BCA when a reasonable attempt at analysis has been made. We find that the practice of BCA differs considerably from, and in some important ways does not meet, the methodologies and protocols as prescribed in textbooks (Boardman et al., 2011). These differences extend across estimation of both costs and benefits as well as across how a BCA is structured. However, the most important distinction is that these RIAs do not perform social BCA as it is commonly understood: Their actual analyses are far more constricted and narrow.
Estimating Costs
The first concern is that some costs are not accounted for in federal BCAs. Notably, some RIAs only considered the resources required to perform new tasks and did not consider the resources required to be in a position to perform these new tasks. Crucially, in order to perform new tasks, workers must be trained and new managerial and organizational procedures need to be developed. The resources required for training, management, and organization were not taken into account. For example, regulations to limit the eligibility length for direct subsidized student loans have cost consequences for colleges. The RIA only includes the “reporting and financial aid counselling activity” to conform to this rule; changes colleges must make to program offerings (as student enrollment patterns change) are not included.
Second, although most regulations did account for the time burden on professional staff, they used very low estimates of time costs. That is, they typically assigned a value in the range of US$25–US$30 per hour. This is a low value for a professional occupation: Bureau of Labor Statistics (2015) data show estimates of education administrators time in postsecondary education at US$49 per hour. It is also an underestimate in that there was no information on whether employer costs of compensation or overheads were included. It is therefore likely to be an underestimate of the opportunity cost of professional staff, that is, their value in alternative roles. Also, studies rarely reported any resource use by the professional staff such as office space or computer use. 3
Review of the RIAs generated additional concerns. An implicit assumption was that all inputs are variable and neither fixed costs nor economies of scale were salient. Hence, it would be possible to either reduce inputs and save the full expenditure (input times input price) or increase inputs without facing an upward-sloping supply of inputs (increasing input prices). In practice, it would seem plausible to assume that there are some fixed costs and that marginal costs would exceed average costs. A final concern is that, for all regulations, in few of the RIAs were the sourcing and methods for calculating costs clear. In particular, few RIAs specify whether the accounting, survey, or full ingredients method was used even as results will likely differ depending on which method was used (Levin et al., 2018, Chapter 4).
Estimating Benefits
As shown in the checklist analysis, few RIAs reported benefits in sufficient detail. Even in the highest quality BCAs, benefits usually were only stated and were not explored further.
As well as precluding calculation of NPV, this omission is especially important for educational BCAs. For educational BCAs, distributional issues are salient; in some cases, they are central or integral to the policy. (We recognize that distributional issues are important in many BCAs: On these issues for environmental BCAs, see Robinson, Hammitt, & Zeckhauser, 2016). Regulatory actions often require resource commitments from—and convey benefits to—various groups such as students, state education agencies, local education agencies, or the federal government. It is important to establish whether each group receives benefits that outweigh the costs and that each group is treated equitably. These distributional considerations were rarely reported. For example, Regulation Identifier Number (RIN) 1840-AD01, a regulation of the College Assistance Migrant Program, states that there is no need to discuss the changes to the regulations…because the changes to regulations for these programs were minor. The most significant changes…address who can be considered an immediate family member of a migrant individual in order to be eligible for program services. (High School Equivalency Program and College Assistance Migrant Program, The Federal TRIO Programs, and Gaining Early Awareness and Readiness for Undergraduate Program, 2010)
Analyzing Costs and Benefits Together
We identify three areas where the joint analysis of costs and benefits should be improved so as to conform more closely to recommended standards and thresholds (Zerbe, Davis, Garland, & Scott, 2013).
Few BCAs considered alternative policy options. Under Executive Order No. 12,866 (1993), costs and benefits of the proposed regulation are to be compared to costs and benefits of alternative, feasible regulations. Only a few of the RIAs included any consideration of alternatives and those RIAs typically referred to an alternative of “no implementation” (rather than the next best alternative).
Also, no sensitivity analyses of the estimated costs, benefits, or NPVs are reported. 5 Sensitivity analysis is especially important for prospective BCAs where decision makers need to know the possible downside risks. This is particularly interesting for some regulatory actions where projections of costs over time are made.
Finally, the RIAs typically fail to apply the proportionality principle. That is, they fail to allocate the most attention to the most important factors in their BCA. 6 Application of the proportionality principle is especially important, given the limited resources federal staff has to perform BCAs.
Administrative Burden BCA
On review, we find that the educational RIAs performed by the federal government are a particular type of BCA. Based on how the RIAs interpret costs, these BCAs do not calculate costs of implementing a policy. Instead, they calculated costs of documenting compliance with regulations (in accordance with the Paperwork Reduction Act of 1995). Instead of evaluating a policy as a social investment and applying social BCA, the RIAs were actually evaluating how to implement a policy or a change in policy. We refer to these as “administrative burden” BCAs. They differ in three distinct ways from social BCA.
First, these administrative burden BCAs use a very particular approach to calculate costs. As one example, we refer to RIN 1840-AD02 (Program Integrity Issues, 2010) that relates to institutional eligibility under the Higher Education Act of 1965. The RIA does not attempt to calculate the costs and benefits of the new number of eligible institutions. Instead, the RIA is directed toward estimating the administrative burden of changing eligibility status. The administrative burden increase for this regulation is estimated at US$126 million over 5 years. This burden is calculated as a product of the hours worked and the hourly wage rate, but it is not derived from a formal costing exercise using the ingredients method. Instead, the costs are estimated using what we refer to as a “conveyor belt” approach. For each student, the regulation necessitates a change in status or eligibility; this change is estimated to take x minutes of personnel time. Therefore, if the college has 100 students, then the cost is 100x multiplied by the wage rate of personnel (if 10,000 students, the cost is 10,000x times the wage rate).
This conveyor belt approach is incomplete and potentially misleading. First, it does not account for other costs such as overheads, management, or facilities. Instead, the conveyor belt approach focuses on marginal costs of performing an administrative task; it assumes administrative changes can be placed on top of existing reporting structures and do not require additional capital, managerial personnel, or computer systems. Second, it does not account for the scale of operations: It assumes that there are no fixed costs to changing administrative burdens and no resources are required for training personnel to meet new administrative requirements. Instead, the full ingredients approach requires identification of the inputs and their prices to implement a regulatory change at a college of a prescribed scale, that is, to estimate a cost function with fixed and variable costs.
Second, the administrative burden approach significantly restricts what can appropriately be considered as a benefit. With an administrative burden approach, the benefits are from compliance with a regulation and it is difficult to describe and calculate these benefits. It may be tempting to consider regulations as unnecessarily onerous and so having zero benefits: Any reduction in compliance costs should therefore have a positive NPV. Nevertheless, it is still important to specify the benefits of reducing administrative burdens. For example, the stated benefits of RIN 1840-AD02 (Program Integrity Issues, 2010) are that these administrative changes will lead to more accurate determination of status as a college student eligible for federal support and “greater transparency for borrowers” (p. 66971). Neither of these benefits is quantified; no attempt is made even to bound the value of these benefits so that they might be compared to the costs. (There are also other vague benefits, including “increased clarity about incentive compensation for employees at institutions of higher education.”) Administrative regulations typically require colleges to provide information; an economic evaluation would consider whether the administrative burden is justified in terms of the information obtained. The value of this information depends on how much the new information changes the expected value of a given policy; this value is often difficult to estimate (see the discussion in Boardman et al., 2011, Chapter 10).
The distinction between a social BCA and an administrative burden BCA is most clearly illustrated with RIN 1840-AD15 (Program Integrity: Gainful Employment, 2014), a regulation to change eligibility for Pell grants. This regulatory change would cost US$126 million to implement, but it would yield savings of US$4.3 billion in Pell grants not taken over the subsequent decade. Yet these “savings” are only benefits if we adopt an administrative burden perspective (in some BCAs, they would be considered as transfers). From a social perspective, the reduction in Pell grants is not a benefit unless all those Pell grants produced zero human capital for the recipients (and even under that very unlikely scenario, policy makers might still be concerned about the regressive effect of reducing Pell grants).
The third and final observation about administrative burden BCAs relates to their correspondence to new regulatory directives. As noted in the Introduction section, the January 2017 Executive Order proposes a “two-for-one” regulatory exchange process. (Before this Executive Order, the U.S. approach to reviewing existing regulations was a more formal and structured “look-back’ approach [Sunstein, 2014b].) This order accords directly with BCAs that are essentially about administrative burdens: A new regulation with an administrative burden of $X million can only be implemented if two existing regulations with administrative burdens of $2X million are rescinded. (Similar approaches have been adopted in other countries, e.g., Canada, the United Kingdom, and Australia.) Given the lack of consideration of benefits in the “two-for-one” regulatory exchange process, the desirability of regulations is yet harder to define, when considering either social or administrative scopes. Leaving aside the desirability, efficiency, and practicality of this type of Executive Order, it prioritizes administrative burden BCA.
Summary and Conclusions
Methods and theories for performing BCA are clearly described in textbooks (Boardman et al., 2011; Zerbe et al., 2013). And, the need for BCA is clear: As summarized by Dudley et al. (2017, p. 201), “Regulatory impact analysis can be an invaluable method for transparently evaluating contentious policy choices before they are put in effect.” However, the practice of BCA is important.
Our findings for regulatory impact assessments of education policies at the federal level demonstrate that—in the infrequent instances in which it is undertaken—practice falls short of methodological standards in a number of ways. Cost estimates appear to be underestimated and lack transparency with respect to method and assumptions. Benefit estimates are very infrequent and analyses often lack sensitivity testing, proportionality, and a reasonable counterfactual. Overall, these findings on costs, benefits, and economic metrics echo those in other reviews of regulatory review standards (e.g., Hahn & Tetlock, 2008; Shapiro & Morrall, 2012, 2016). Most notably, these RIAs are not attempts at social BCA but instead are evaluations of administrative burdens; this focus unavoidably affects the structure of economic evaluation and impairs our ability to use BCA as a tool in policy-making.
We share the concern of Gordon (2016) that BCA should guide decision-making, rather than being an instrument of justification of an already determined policy. In addition, in order for BCA to provide said guidance, it must be of high quality if it is to be informative in the decision-making process. There are a number of remedies beyond simply exhorting analysts to perform more rigorous analyses. Certainly, more time and funding should be provided, so that analysts can undertake more rigorous study. A full application of the ingredients method and shadow pricing of benefits requires a similar level of research resources as an impact evaluation. More training for analysts may be desirable, as well as a greater emphasis on harmonizing studies for comparative purposes.
In addition, we suggest the following recommendations that could help strengthen the BCAs we have reviewed. These recommendations correspond to the general tips listed by Dudley et al. (2017).
Benefits
Several RIAs include a description of possible benefits to students, but none attempt to estimate the number of students receiving these benefits or the dollar value of these benefits. We encourage the government agencies to assign a number, along with a measure of its precision or a confidence interval. This estimate of the number of beneficiaries of a new regulation could be varied in the sensitivity analysis to verify the importance of the assumption on the overall results.
Performing a rigorous estimation of benefits may not be plausible for government agencies, given limited time and other resource constraints. Therefore, we suggest that government agencies refer to external rigorous benefit calculations, such as those provided by the Washington State Institute for Public Policy. The match between prior estimates and the RIA under consideration need not be exact as this can be varied in sensitivity analyses.
Costs
Most RIAs only refer to costs incurred by the funding agency or by other government agencies. This is what we refer to as administrative burden BCA. Only a few RIAs consider costs to students and education institutions. We encourage government agencies to consider multiple perspectives and all associated costs of regulations.
Benefit–Cost
A very important step missing for all RIAs analyzed is that none of them match the costs with the benefits. This may be in part due to the lack of quantified benefits within the RIAs reviewed. We strongly suggest that whenever benefits are quantifiable, government agencies should include in the BCAs a comparison of the benefits to the costs in a simple subtraction (B – C) or a ratio (B:C). Ultimately, the objective of quantifying and valuing costs and benefits is to provide guidance in the decision-making process. Weighing costs against benefits, whether it is a subtraction or a ratio, is the most important and useful result of a BCA to policy makers, and it should be estimated whenever possible.
At a more basic level, RIAs need to be more explicit in three respects. One is the alternative policy options that are under consideration: Explaining why specific options were chosen is necessary but often these choices are presumed. A second is that the theory of action is often unclear: Education policies are complex and may bring about change in many different ways. (These 2 items match with the primary tips Dudley et al. [2017] suggest to readers of RIAs.) Finally, the third basic clarification relates to which type of analysis—social BCA or administrative burden BCA—is being undertaken. The choice between them has fundamental consequences for economic evaluation. In particular, we believe social BCAs should be prioritized over administrative BCAs. Currently, the latter are being undertaken, perhaps giving the impression that they are equivalent to the former. They are not. If administrative BCAs are an important analysis needed for government agencies, these should be done as a part of the broader social BCA.
Notwithstanding these deficiencies, we believe these economic evaluations are helpful in substantiating education policy decisions. Even when RIAs fail to follow best practices or face methodological challenges, they may still be an improvement over the alternative of no information. More importantly, we emphasize that a lack of evidence does not imply that these regulations are inefficient. Hundreds of studies have established the economic value of education in terms of higher earnings, as well as in terms of private and social benefits, for those with more education (Autor, 2014; Barrow & Malamud, 2015; Belfield & Levin, 2007). Therefore, we expect these regulations are efficient in the sense that the benefits of the regulations exceed the costs. Nevertheless, it is still important to substantiate this expectation with high-quality BCAs.
Footnotes
Appendix A
Twenty-Eight Federal Regulatory Impact Assessments.
| Year | Order | Department | Regulation | Description |
|---|---|---|---|---|
| 2009 | 1810-AB04 | OESE | State Fiscal Stabilization Fund Program—Notice of Proposed Requirements, Definitions, and Approval Criteria | Formula grants to states to help stabilize state and local budgets in order to minimize and avoid reductions in education and other essential services, in exchange for a state’s commitment to advance essential education reform in key areas |
| 2009 | 1810-AB06 | OESE | School Improvement Grants—Requirements under American Recovery and Reinvestment Act of 2009; Title I of the Elementary and Secondary Education Act of 1965 | Funds for school improvement, school improvement grants are used to improve student achievement in Title I schools identified for improvement, corrective action, or restructuring so as to enable those schools to make adequate yearly progress and exit improvement status |
| 2009 | 1810-AB07 | OESE | Race to the Top Fund—Notice of Proposed Priorities, Requirements, Definitions, and Selection Criteria | Encourages and rewards states that are creating the conditions for education innovation and reform; achieving significant improvement in student outcomes, closing achievement gaps, improving high school graduation rates, and ensuring student preparation for success in college and careers; and implementing ambitious plans in four core education reform areas |
| 2010 | 1810-AB08 | OESE | Teacher Incentive Fund—Priorities, Requirements, Definitions, and Selection Criteria | Supports projects that develop and implement Performance Based Compensation Systems (PBCSs) for teachers, principals, and other personnel in order to increase educator effectiveness and student achievement, measured in significant part by student growth in high-need schools |
| 2012 | 1810-AB12 | OESE | Teacher Incentive Fund | Support projects that develop/implement PBCSs for teachers, principals, and other personnel to increase educator effectiveness and student achievement, measured in significant part by student growth, in high-need schools |
| 2012 | 1810-AB15 | OESE | Race to the Top—Early Learning Challenge Phase 2 | Improves the quality of early learning and development and close the achievement gap for children with high needs. |
| 2013 | 1810-AB17 | OESE | Race to the Top—District | Supports bold, locally directed improvements in learning and teaching that will directly improve student achievement and educator effectiveness |
| 2013 | 1810-AB18 | OESE | Race to the Top—Early Learning Challenge | Improves the quality of early learning and development and close the educational gaps for children with high needs |
| 2006 | 1840-AC86 | OPE | Student Assistance General Provisions and Federal Student Aid Programs—Academic Competitiveness and National Science and Mathematics Access to Retain Talent Grant Programs | These interim final regulations implement certain provisions of the Higher Education Reconciliation Act (HERA) of 2005 |
| 2006 | 1840-AC87 | OPE | Institutional Eligibility Under the Higher Education Act of 1965, as Amended; Student Assistance General Provisions and Federal Student Aid Programsa | Reflect the provisions of the HERA that affect students, borrowers, postsecondary educational institutions, lenders, and other program participants in the Federal student aid programs authorized under Title IV of the Higher Education Act (HEA) |
| 2007 | 1840-AC89 | OPE | Federal Perkins Loan Program, Federal Family Education Loan (FFEL) Program, and William D. Ford Federal Direct Loan (DL) Program | Amending these regulations to strengthen and improve the administration of the loan programs authorized under Title IV of Higher Education Act of 1965 |
| 2008 | 1840-AC93 | OPE | Title IV of the Higher Education Act of 1965, as Amended (Teacher Education Assistance for College and Higher Education [TEACH] Grant Program) | Nonneed-based grant program for students who are enrolled in an eligible program and who agree to teach in a high-need field, at a low-income elementary or secondary school for at least 4 years within 8 years of completing the program for which the TEACH Grant was awarded |
| 2008 | 1840-AC94 | OPE | Federal Perkins Loan, FFEL, and William D. Ford Federal DL Programsa | The Secretary amends the Federal Perkins Loan (Perkins Loan) Program, FFEL Program, and William D. Ford Federal DL Program regulations to implement provisions of the College Cost Reduction and Access Act of 2008 |
| 2009 | 1840-AC96 | OPE | Student Assistance General Provisions; TEACH Grant, Federal Pell Grant, and Academic Competitiveness Grant (ACG), and National Science and Mathematics Access to Retain Talent Grant Programs | Amends the regulations for the ACG and National Science and Mathematics Access to Retain Talent Grant (National SMART Grant) Programs. These interim final regulations are needed to implement provisions of the HEA of 1965, as amended by the Ensuring Continued Access to Student Loans Act of 2008 and the Higher Education Opportunity Act of 2008 |
| 2009 | 1840-AC99 | OPE | General and Non-loan Programmatic Issues | Amends the regulations for Institutional Eligibility Under HEA 1965, the Student Assistance General Provisions, the Federal Work-Study Programs, the TEACH Education Grant Program, the Federal Pell Grant Program, and the Leveraging Educational Assistance Partnership Program |
| 2010 | 1840-AD01 | OPE | Federal TRIO Programs, Gaining Early Awareness and Readiness for Undergraduate Program, and High School Equivalency and College Assistance Migrant Programs | Helps migrant and seasonal farmworkers and their immediate family members obtain a general equivalency diploma; College assistance migrant programs assist students to complete their first academic year of college and continue in postsecondary education |
| 2010 | 1840-AD02 | OPE | Institutional Eligibility Under the Higher Education Act of 1965; Student Assistance General Provisions | Improving integrity in programs authorized under title IV of HEA 1965 by amending the regulations for Institutional Eligibility, the Secretary’s Recognition of Accrediting Agencies, the Secretary’s Recognition Procedures for State Agencies, the Student Assistance General Provisions, FFEL Program, the TEACH Grant Program, the Federal Pell Grant Program, and the AGC and National Smart Grant Programs |
| 2010 | 1840-AD04 | OPE | Program Integrity: Gainful Employment | Amend the Student Assistance General Provisions to establish measures for determining whether certain postsecondary educational programs lead to gainful employment in recognized occupations, and the conditions under which these educational programs remain eligible for the student financial assistance programs authorized under title IV of HEA 1965 |
| 2012 | 1840-AD05 | OPE | Federal Perkins Loan Program, FFEL Program, and William D. Ford Federal DL Program | Income-Contingent Repayment plans in the DL program, incorporate recent statutory changes to the IBR plan in the DL and FFEL programs, and streamline and add clarity to the total and permanent disability discharge process for borrowers in loan programs under title IV of HEA 1965 |
| 2011 | 1840-AD06 | OPE | Program Integrity: Gainful Employment-Measures | Amend the Student Assistance General Provisions to establish measures for determining whether certain postsecondary educational programs lead to gainful employment in recognized occupations, and the conditions under which these educational programs remain eligible for the student financial assistance programs authorized under title IV of HEA 1965 |
| 2013 | 1840-AD11 | OPE | Federal Pell Grant Program | Amends four sections of the Federal Pell Grant Program regulations to make them consistent with recent changes in the law that prohibit a student from receiving two consecutive Pell Grants in a single award year |
| 2013 | 1840-AD12 | OPE | Transitioning from the FFEL Program to the DL Program and Loan Rehabilitation Under the FFEL, DL, and Perkins Loan Programs | Amends the FFEL and DL program regulations to reflect changes made to HEA 1965 by the Student Aid and Fiscal Responsibility Act included in the Health Care and Education Reconciliation Act of 2010; incorporate statutory changes to interest rates and other recent statutory changes in the DL Program regulations; update, strengthen, and clarify various areas of the Student Assistance General Provisions, Perkins Loan, FFEL, and DL program regulations; and provide for greater consistency in the regulations governing the title IV, HEA student loan programs |
| 2013 | 1840-AD13 | OPE | William D. Ford Federal DL Program | Amendments to ensure a borrower may not receive Direct Subsidized Loans for more than 150% of the published length of the educational program in which the borrower is enrolled |
| 2014 | 1840-AD15 | OPE | Gainful Employment | Establish measures for determining whether certain postsecondary educational programs prepare students for gainful employment in a recognized occupation, and the conditions under which these educational programs remain eligible under the Federal Student Aid programs authorized under title IV of the HEA |
| 2015 | 1840-AD18 | OPE | Revised Pay as You Earn (REPAYE) | The REPAYE plan is modeled on the existing Pay as You Earn repayment plan and will be available to all DL student borrowers regardless of when the borrower took out the loans |
| 2010 | 1855-AA06 | OII | Investing in Innovation—Priorities, Requirements, Definitions, and Selection Criteria | Supports local education agencies and nonprofit organizations to provide competitive grants to applicants with a record of improving student achievement and attainment |
| 2013 | 1855-AA09 | OII | Investing in Innovation | Establish priorities, requirements, definitions, and selection criteria that will enable effective grant making, resulting in the selection of high-quality applicants who propose to implement activities that are most likely to have a significant national impact on educational reform and improvement |
| 2015 | 1855-AA12 | OII | Charter Schools Grants to SEAs | Priorities, requirements, definitions, and selection criteria for charter school programs grants to SEAs |
| 2011 | 1894-AA01 | OII | Race to the Top Fund Phase 3 | Requirements for Phase 3 of the Race to the Top program. In this phase, the department intends to make awards to states that were finalists but did not receive funding under the Race to the Top Fund Phase 2 competition held in fiscal year 2010 |
Note. All orders ruled “consistent with change” except a denotes “consistent without change.” OESE = Office of Elementary and Secondary Education; OPE = Office of Postsecondary Education; OII = Office of Innovation and Improvement; SEA = state education agency.
Appendix B
Checklist Evaluation of 28 Federal Regulatory Impact Assessments by Budgetary Allocation.
| Checklist Criteria | Percentage Performing Activity | |
|---|---|---|
| Lower Budgets | Higher Budgets | |
| Cost | ||
| Statement of costs beyond budgetary statement (stated) | 86 | 100 |
| Estimated costs using ingredients method (ingredients) | 14 | 36 |
| Estimated costs separately by agency (agency) | 43 | 64 |
| Dollar value of costs reported (calculated) | 71 | 71 |
| Benefits | ||
| Statement of monetized benefits (stated) | 86 | 79 |
| Description of benefits estimation (described) | 14 | 43 |
| Estimated benefits separately by agency (agency) | 50 | 64 |
| Dollar value of benefits reported (calculated) | 0 | 7 |
| Net present value (NPV; benefits minus costs) | ||
| Dollar value calculated (NPV) | 0 | 0 |
| Stated as positive (B > C) | 100 | 100 |
| Methodology | ||
| Assumptions described (year, inflation, discount rate) | 21 | 50 |
Acknowledgments
The authors would like to thank Nora Gordon, Doug Harris, and the participants of the Association for Education Finance and Policy and the Society for Benefit-Cost Analysis conferences for helpful discussions and/or comments on earlier versions of this article.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.