Functioning or Dysfunctioning? The Effects of Performance-Based Funding

Overview of PBF

When states commenced the adoption of PBF policies, their primary goal was to find a way to distribute public resources more efficiently among a growing number of public higher education institutions (Bogue & Johnson, 2010). This early form of PBF emphasized input or process metrics, such as student enrollment. However, as state policymakers are continuously searching for strategies to promote the accountability and productivity of higher education institutions (Bogue & Johnson, 2010; Hillman et al., 2014; Ness et al., 2015), several states shifted to a newer form of PBF. The goal of doing so was to influence institutions’ behavior in ways that align with state policy priorities using financial incentives as the instrument. This newer PBF approach, which is rooted in outcomes-based funding, highlights student success and degree completion, while embracing some form of weights for underrepresented students (Hearn, 2015; Snyder & Fox, 2016). An early form of PBF differed from outcomes-based funding in that, in the early model, the associated funding was characterized as being additional to base state funding. Further, the proportion of state appropriations was relatively small: 1% to 5% of state appropriation. However, the newer PBF system directly links state funding to performance metrics, and the portion of funding is as high as 90% of state appropriation (Dougherty et al., 2016).

Although PBF’s characteristics (metrics, weights, and reward methods) differ considerably across states, at least 30 states have implemented some form of PBF. Adoption years varied among states, and some states have discontinued, readopted, or attempted to reform the policy (Dougherty & Natow, 2015). To reflect institutional differences in performance indicators, for example, some states formulated different measures for 2- and 4-year institutions. In addition, states, such as Tennessee, weigh some indicators differently than others, based on institutional missions. Others, including Missouri and Pennsylvania, apply different formulas in accordance with an institution’s mission and student body (Dougherty et al., 2016). Notwithstanding these differences, policymakers commonly provide institutions with financial incentives by allocating funding and information to promote state goals and institutional accomplishments, and build institutional capacity (Reddy et al., 2014).

As PBF formulas evolved, 16 of the 28 states that implemented outcomes-based funding during the 2015 and 2016 academic year incorporated equity measures for selected populations (Holly & Fulton, 2017; Kelderman, 2019). These equity metrics differ among states with regard to which groups are prioritized (e.g., minority, low-income, first-generation, adult, or underprepared students). There are also differences in types of equity measures used (e.g., direct vs. bonus and input vs. progress vs. outcome), whether such measures are required or optional, and how measures are weighted (Cielinski & Pham, 2017). The three states considered in this study incorporated some form of equity metrics with the establishment of the newer PBF system (i.e., outcomes-based funding), albeit in different forms. Tennessee adds extra weight for low-income students and adults, whereas Ohio awards additional points for low-income and minority students, among others, and Indiana provides greater weight for low-income students.

Empirical Research on PBF

A wealth of studies have examined the effect of PBF on student outcomes. Findings from studies that used descriptive analysis indicated that, after states adopted PBF, the number of students who received college degrees increased in Tennessee (Callahan et al., 2017; Johnson & Yanagiura, 2016), Ohio (Carey, 2014), and Indiana (Callahan et al., 2017). But unlike the descriptive studies, the results from most quasi-experimental studies indicate that PBF does not induce college degree production. For example, Shin (2010) found that PBF was not significantly associated with graduation rates (or federal research funding levels). Sanford and Hunter (2011) found that PBF did not have any effect on retention or 6-year graduation rates. Rutherford and Rabovsky (2014) concluded that states with PBF did not outperform their peers in improving student graduation, persistence, and degree attainment, and that PBF was, in fact, associated with lower performance over time. In a similar context, Hillman et al. (2014) found that PBF in Pennsylvania did not systematically increase degree productivity in the state, and Hillman, Fryar, and Crespín-Trujillo (2018) found that PBF, on average, did not increase baccalaureate degree completion in Tennessee and Ohio. Furthermore, Ward and Ost (2021) found that the policies did not improve key academic outcomes, including baccalaureate degree production, institutional spending, and student enrollment in Tennessee and Ohio.

As multiple researchers have reported null effects of PBF on student outcomes, scholars have attempted to determine whether impediments to implementation of the policy exist. In terms of the structure of PBF, factors such as the low funding amounts at stake, phase-in of the model, and the use of 3-year average indicators as metrics may contribute to low variations within and across states (Dougherty et al., 2016). In addition, institutions subject to PBF may confront obstacles, such as inappropriateness of performance measures, lack of knowledge or buy-in of faculty members, and structural problems of institutions (Dougherty et al., 2016). In particular, institutions may struggle with the student body composition. On one hand, institutions with open admission policies tend to have more students who are academically less prepared, of low income, or attend part time, which may inhibit improving graduation rates. Conversely, although selective institutions’ student bodies may include academically and financially well-prepared students, those institutions may suffer from a ceiling effect, in that they already have high levels on targeted student outcomes and do not have much room for improvement (Dougherty et al., 2016).

However, even in cases where researchers found a positive association between PBF and student outcomes, it is inconclusive whether these results yielded unintended consequences. The most commonly identified unintended effects of PBF include restricting admissions of students less likely to graduate and reducing academic requirements for enrolled students (Dougherty et al., 2016). Previous research (Gándara & Rutherford, 2020; Umbricht et al., 2017) found that PBF was associated with declining admission rates and increased selectivity. In a similar context, Kelchen and Stedrak (2016) found that universities subject to PBF were less likely to receive Pell Grant revenue, indicating that PBF may induce institutions to favor high-income students. On the other hand, Gándara and Rutherford (2020) found that the percentage of students who received a Pell Grant increased in institutions under PBF.

Several researchers (Favero & Rutherford, 2020; Hagood, 2019) examined the different effects of PBF across institutions and reported that the policy is likely to reward high-resource institutions. Likewise, Hillman and Corral (2017) found that minority-serving institutions in PBF states are less likely to receive state funding. Kelderman (2019) also reported that PBF with a reward structure for graduates in Science, Technology, Engineering, and Mathematics (STEM) fields may work against liberal arts colleges. Furthermore, PBF schemes prioritizing graduation of in-state students may work against institutions enrolling a large proportion of out-of-state students, due to their locations.

Although prior research has provided insightful and instructive information, previous studies are limited in several ways. First, studies that used descriptive data are limited because these studies failed to control for other factors that may contribute to student outcomes and some studies did not allow sufficient time to observe policy effects after outcomes-based funding implementation (Callahan et al., 2017; Dougherty et al., 2016). Even prior studies that employed rigorous methods (e.g., difference-in-differences) may not provide a full picture regarding differential policy effects over time because the method allows observation of only a single, aggregated average treatment effect for all posttreatment periods (Furquim et al., 2019). Furthermore, many studies had problems finding a comparable control group with pretreatment trajectories on outcomes similar to the treatment group, a fundamental assumption to successfully employ the difference-in-differences estimator. Prior studies also have inferential limitations in that they did not cluster standard errors appropriately or did not do so at all; further, they did not pay close attention to making a more accurate inference with the small number of clusters that are often encountered in state policy analysis. We remedy these deficiencies by employing multiple estimators (difference-in-differences and event study) for PBF programs, especially for relatively newly adopted outcomes-based funding programs. By employing an event study analysis, in addition to a difference-in-differences approach, we provide a more complete empirical picture by observing the annual treatment effect for each individual time period and varying patterns in the treatment effect over time. We also incorporate a coarsened exact matching method to reduce the imbalance between treated and control groups (Blackwell et al., 2009; King et al., 2011) and calculate precise standard errors by clustering at the level where the variation of interest happens (i.e., at the state level) and dealing with the small number of treated clusters.

Data Sources

We primarily utilized data from the IPEDS but supplemented them with data from the Bureau of Labor Statistics (BLS). The IPEDS provides information on institutional characteristics, institutional prices, admissions, enrollment, student financial aid, student persistence, student success, and institutional resources for higher education. For this study, we extracted the individual university’s characteristics and outcome variables from the IPEDS data and used county unemployment variables from the BLS data. Through this process, we built an institution-level panel data ³ for 2005 to 2018. In terms of the choice of the data period, we incorporated at least three time periods before the policy implementation (Tennessee in 2010; Ohio and Indiana in 2009) to observe the trend line clearly (Furquim et al., 2019). With respect to the posttreatment period, we incorporated the most recently available data to ensure sufficient time to observe variations in the policy effect over time following the outcomes-based funding enactment (Callahan et al., 2017; Dougherty et al., 2016).

We examined the effects of PBF in three states, Tennessee, Ohio, and Indiana, that have established PBF programs over time. We used as a comparison group other states that have never enacted PBF. Although PBF may have important effects on 2-year colleges, the focus of our study is 4-year universities. This is largely because PBF policies are applied differently according to institutional sectors (2- or 4-year institutions) in some states, which affects our construction of the comparison group. Also, the goals and behaviors of 2- and 4-year institutions often differ in important ways, which affects the choice of outcome variables examined. Tribal colleges and special-focus universities (using Carnegie classification) were excluded (693 institutions as of 2005). Special-focus universities include faith-related institutions, medical schools and centers, other health profession schools, engineering schools, other technology-related schools, business and management schools, music and design schools, and law schools.

Variables

The key independent (“treatment”) variable in this study is the implementation of PBF. Based on previous literature (Dougherty & Natow, 2015; National Conference of State Legislatures, 2015; Snyder & Boelscher, 2018; Snyder & Fox, 2016), we identified whether states have adopted PBF policies, when the policies were implemented, and the current status of PBF across states. Table 1 summarizes the comparison group selection criteria and locations for each comparison group (explained in detail in the “Robustness Checks” section).

OPEN IN VIEWER

Table 1

Summary of Comparison Group Selection Criteria and Locations

		Geographical matches		CEM match
		Group 1	Group 2	Group 3
		Neighboring states	All states	Full match
Selection criteria		Institutions in regional states that have never adopted PBF	Institutions across the nation that have never adopted PBF	Institutions from Group 2 that have been matched (in any year before the treatment) to institutions in each state
States included	Tennessee	Alabama, Delaware, Maryland, South Carolina, Texas, West Virginia	Alabama, Alaska, California, Connecticut, Delaware, Idaho, Iowa, Maryland, Nebraska, New Hampshire, New Jersey, Rhode Island, South Carolina, Texas, Vermont, Washington, West Virginia, Wisconsin, Wyoming	Alabama, California, Iowa, Maryland, Nebraska, Rhode Island, Texas, Washington, West Virginia, Wisconsin
	Ohio	Iowa, Nebraska, Wisconsin		Alabama, Alaska, California, Connecticut, Maryland, Nebraska, New Jersey, Rhode Island, South Carolina, Texas, Washington, West Virginia, Wisconsin
	Indiana	Iowa, Nebraska, Wisconsin		Alabama, Alaska, California, Connecticut, Idaho, Maryland, Nebraska, New Hampshire, New Jersey, Rhode Island, South Carolina, Texas, Washington, West Virginia, Wisconsin

Note. CEM = coarsened exact matching; PBF = performance-based funding.

Outcome variables are the total number of baccalaureate degrees conferred during each academic year (as measured by the logged value), admission rates (measured as the share of total applicants who were admitted), the 25th and 75th percentiles of ACT scores, and the percentage of minority and low-income students. Considering that this study focuses on the effects of PBF on degree completion, admission practices, and the enrollment of underrepresented students, the present analysis does not encompass some of the other outcomes often included in PBF policies (e.g., graduation rates, retention, or STEM-degree production). Whereas we measured degree completion as the log of number of bachelor’s degrees, we did not utilize graduation rates, partly because this measure appears to be easily manipulated by institutions (Hearn, 2015; Hillman et al., 2014). As the graduation rate is calculated as the percentage of first-time, full-time students attending the same institution and graduating within 6 years, some students (e.g., part-time, transfer students) are not included. Furthermore, institutions may be tempted to accept students who are more likely to graduate, even at the expense of reducing the total number of incoming students, to easily achieve PBF performance indicators (Hillman et al., 2014). In fact, many states focus more on student counts rather than rates to expand access for underrepresented students (Hearn, 2015). Reflecting upon these issues, we used the total number of bachelor’s degrees conferred as an outcome variable for student degree completion. In addition, although there are factors such as intermediate outcomes (e.g., first-year retention rates) or degree attainment within specific major subjects (e.g., STEM-degree production), we focused our attention on the ultimate goal of increasing degree attainment for all students.

The percentage of minority students was measured by the total enrollment percentages for Black, Hispanic, and American Indian or Alaska Native students. The percentage of low-income students was measured by the percentage of full-time, first-time undergraduates awarded federal grants. Although most researchers use the number of students receiving Pell Grants as a proxy variable for low-income students, this variable is only available from the 2007 and 2008 academic year in the IPEDS and is not available across the full analysis period of this study (2005–2018). Thus, we use federal grants as a proxy variable for low-income students to be comparable by year. Considering that federal grants, including Pell Grants, Supplemental Education Opportunity Grants, Department of Veteran Affairs Grants, and the Leveraging Educational Assistance Partnership Program, are primarily needs based, it is a reasonable proxy variable for low-income students (Umbricht et al., 2017).

Based on prior research (Hillman et al., 2014, 2018; Umbricht et al., 2017) and our conceptual framework, we included control variables that could affect the dependent variables. As principal–agent theory and the ecology of games perspective suggest, institutional agents (players) may behave differently based on their objectives, constraints, preferences, and resources. To proxy for such behavior, we controlled for institutional characteristics, including FTE enrollment, tuition and fees, and total educational and general expenditures per FTE. Moreover, given that the economic conditions of the state may affect student and institutional behavior and therefore educational outcomes, we included the county unemployment rate as a proxy for state economic situations. To adjust for inflation and make data comparable over time, we converted all finance data into 2018 dollars using the consumer price index.

Analytic Strategy

We employed several analytic strategies. First, we utilized a canonical difference-in-differences strategy to estimate the average treatment effect and promote comparability with previous studies. Difference-in-differences is a quasi-experimental research design that allows for a comparison of the average change over time between the treated and the control groups when using observational data. This approach has become increasingly popular to analyze the effect of policies on educational outcomes (Flores, 2010; Garces, 2012). This may be because the method has a relatively simple, intuitive research design for evaluating policies, and it is also an effective analytic tool for drawing causal inference (Furquim et al., 2019). The difference-in-differences model employed is formalized as follows:

y i t = α + β 1 ( treat i ) + β 2 ( post t ) + β 3 ( treat i × post t ) + γ i + η t + θ X i t + ε i t ,

(1)

where y_it is the outcome variable for institution i in year t, treat_i is a dummy indicator for PBF take-up, post_t indicates the years after PBF was enacted, and the interaction between the two treat_i × post_t is the key variable of interest, and an estimate of its parameter β₃ is the average treatment effect of PBF. γ_i represents institution fixed effects that capture any unobserved time-invariant effects of institutions, whereas η_t represents year fixed effects capturing unobserved time trends. θX _it represents a vector of controls, including FTE enrollment, tuition and fees, total educational and general expenditures per FTE, and the county unemployment rate. ε_it is the error term, and standard errors are clustered at the state level. We report confidence intervals obtained from the wild cluster bootstrap, which is discussed in detail in the “Robustness Checks” section.

Second, we employed the DDD method by adding an additional layer of heterogeneous treatments to equation (1) (Andrews et al., 2010). As the ecology of games perspective implies, different institutions may respond to PBF differently, and the treatment effects of PBF on outcome variables may vary according to distinctive institutional characteristics. For example, some institutions with higher resources (e.g., tuition or endowment) may be less financially dependent on other external income, such as PBF, especially when they find that PBF represents only a small amount at stake. It is also possible that institutions with higher resources are actually more responsive to PBF if they are more financially sensitive or if they may respond effectively with their existing capabilities and resources. On the other hand, institutions with higher dependence on state funding may be relatively more sensitive to PBF than their counterparts.

To check these hypotheses, we used three different subgroups, including those with high/low tuition levels, high/low endowments, and high/low state revenue share. We define high and low compared with the median level of each variable at baseline (Ward & Ost, 2021). For Tennessee, the median value of tuition level is $6,068; that of endowment is $2,452; and that of state revenue share is 35%, whereas for Ohio and Indiana, the median value of tuition level is $5,668; that of endowment is $2,857; and that of state revenue share is 36%. We estimate a DDD model of the following form:

y i t = α + β 1 ( treat i ) + β 2 ( post t ) + β 3 ( treat i × post t ) + δ 1 ( IC i t ) + δ 2 ( treat i × IC i t ) + δ 3 ( post t × IC i t ) + δ 4 ( treat i × post t × IC i t ) + γ i + η t + θ X i t + ε i t .

(2)

Compared with the difference-in-differences in equation (1), the DDD in equation (2) adds IC _it , an indicator that assumes a value of one if institutional characteristics (i.e., tuition, endowment, and state share) are above the median. The coefficient of interest, the DDD estimate, is δ₄ representing the difference in the treatment effect for the above median institutions relative to their below median peers. By estimating equation (2), we test whether the effects of PBF on outcome variables vary across institutional characteristics.

Finally, and importantly, we employed an event study approach to mitigate limitations of the difference-in-differences method. Although the difference-in-differences method provides estimates of the average treatment effect during the posttreatment period, the treatment effects may differ over time. A single, aggregated treatment effect yielded via the difference-in-differences approach does not provide information on differential policy effects over time. In this sense, an event study model may shed additional light on whether the policy effects are immediate, lagging, increasing, or decreasing (Furquim et al., 2019). Applying an event study model is particularly relevant in higher education policy analysis because, as the ecology of games perspective suggests, higher education policy games are not static but may vary over time as the players (i.e., institutions) change their goals, strategies, and tactics. We formalize the event study model employed as follows:

y i t = α + ∑ j = − m q β j treat i t ( t = k + j ) + γ i + η t + θ X i t + ε i t .

(3)

The notable difference of this equation from equation (1) is β j , which represents a treatment effect for each jth lead or lag of the treatment. When the treatment happens at time k, the summation term ∑ includes m leads and q lags of the treatment. We estimated all models with and without controls to examine whether adding covariates affects the statistical significance, magnitude, or sign of the baseline model estimate (Furquim et al., 2019). We find that the results with and without controls are not substantially different. For brevity, we only present the results from the models with control variables (results without controls are available upon request).

Robustness Checks

We conducted extensive robustness checks, including clustered and bootstrapped standard errors, parallel trends assumption checks, state-specific linear time trends, a falsification test using a placebo of treatment timing, and the use of multiple comparison groups. We took great care in calculating the standard errors for the difference-in-differences estimates from two important perspectives: clustering at the appropriate unit level and dealing with the small number of treated clusters. Although several previous PBF studies provide estimates with standard errors clustered at the institution level, clustering at the state level is a correct method in that the PBF treatment is assigned at the state level; thus, a source of variation comes from differences across states (Abadie et al., 2017). Furthermore, a growing area of literature is concerned with inferences in clustered difference-in-differences, particularly when the data contain only a few states (Biewen & Schwerter, 2022; Brewer et al., 2018; Roodman et al., 2019). Reflecting on these important issues, we utilized conventional inference methods, including robust standard errors (clustered at the state level) and multiway robust inference (clustered by state and year). We also employed various alternative clustering techniques, including ordinary wild bootstrap, wild cluster bootstrap (clustered at the state level), and multiway wild cluster bootstrap (clustered by state and year). For the wild cluster bootstrap, we computed the restricted and unrestricted bootstraps, and also applied weights to account for the small number of treated clusters.

It is generally noted that cluster-robust standard errors are underestimated when the number of clusters is small (Cameron et al., 2008) and that wild bootstrapping can produce standard errors that are less biased (Brewer et al., 2018; Furquim et al., 2019). Following examples from the econometric literature (Cameron et al., 2008; MacKinnon & Webb, 2018; Roodman et al., 2019), we focus our discussion of the difference-in-differences results on estimates produced using the wild cluster restricted variant (i.e., the wild cluster bootstrap with the null imposed).

Central to the successful implementation of the difference-in-differences method is meeting the parallel trends assumption, where the trajectories of the treatment and control groups would not be different in the absence of the policy intervention. Although it is not possible to formally test this assumption (Furquim et al., 2019; Holland, 1986; Rubin, 2005), we used indirect methods that are often employed, whereby we visually examined the pre/post trends via graphical methods. In addition, we conducted statistical analysis to examine whether there were differences in pre/post trends by including time dummy interactions before the treatment and checking the balance across the treated and control groups.

Although the difference-in-differences model used herein includes institutional and year fixed effects, it is plausible that time-based trends may not be identical across states during the observation period, and those variations may differently affect the behavior and performance of institutions. To account for such unobserved time-varying trends, we examine whether our estimates change when we incorporated state-specific linear time trends. Estimates produced with and without state-specific linear time trends should not be systematically different if the difference-in-differences model assumption holds (Ward & Ost, 2021).

We also conducted a series of falsification tests to ascertain whether the treatment group was on a different path than the control group before the policy was initiated. One way to check this is to conduct placebo tests. We did so by simulating that the policy treatment occurred not in the actual implementation year but 2 years before and 2 years after actual adoption. For example, in the case of Tennessee, we conducted placebo tests for the years 2008 and 2012, instead of 2010, which is the actual policy adoption year. In econometrics, an existing practice is that if a PBF policy has a causal relationship with an outcome variable, statistically significant effects should be observed in the original model specifications but not in placebo models. Other studies argue that failing to reject the null hypothesis does not in itself necessarily invalidate causal findings (Hartman & Hidalgo, 2018). Discerning the underlying treatment mechanism is the key point; for example, in the present study, if there are either anticipatory effects or delayed effects, statistically significant placebo effects may be detected. In fact, we may detect larger and more precise difference-in-differences estimates for the placebo difference-in-differences when the placebo time period is some years after the time periods since the event study conducted finds delayed effects.

To test the support for the assumptions underlying difference-in-differences analysis, comparison groups should be constructed in a way that the trend for the comparison group provides a compelling counterfactual for the trend in the treatment group. Following the example of previous research (Hillman et al., 2014, 2018), we utilize multiple comparison groups using a local matching strategy that identifies geographically adjacent states and a focal matching strategy that identifies statistically adjacent states by matching techniques. Table 1 details the full set of comparison states under each group for each state.

In terms of the local matching strategy, the first control group consists of public 4-year universities in neighboring states that have never adopted PBF. The states compared with Tennessee included Southern Region Education Board states not subject to PBF during this analysis period. We compared Ohio and Indiana with states in the Midwestern Higher Education Compact not subject to PBF. Interstate policy diffusion likely occurs among geographically proximate states, whereby policy adoption by a state influences the policy adoption by other neighboring states (Berry & Berry, 1990; Tandberg & Hillman, 2014). Moreover, in the policy practices of PBF, final funding amounts in some states (e.g., Tennessee) are decided by also considering factors in their geographic region (THEC, 2013), meaning that policy decisions may be interdependent among states in their region. The second control group expands geographical boundaries to the nation by including counterfactual states that have never adopted PBF. In terms of inclusion in the comparison group, we chose to focus on states that have never adopted PBF and exclude states that have had PBF but discontinued the same, through the reasoning that institutions may have been affected by PBF once they experienced it. ⁴

With regard to the focal matching strategy, we used a coarsened exact matching approach (Blackwell et al., 2009; Hillman et al., 2014; King et al., 2011). This procedure allowed us to select a more comparable control group that displays patterns that are more similar to the treatment group before the treatment year. Coarsened exact matching has several advantages over other existing matching methods, such as propensity score matching. The matching technique diminishes the imbalance between treated and control groups, lessens the degree of model dependence, and is easily automated and computationally efficient (Blackwell et al., 2009). To implement this method, we first coarsened the data via automated coarsening and then matched the treatment and control groups utilizing the covariates of this study. Matched observations in a stratum were retained, and unmatched observations were eliminated. Following the strategy outlined in Hillman et al. (2014), we identified two matching groups. We constructed the first-year match sample for institutions that have been matched in 2005 only, and we created the full match sample for institutions that were matched at least 1 year prior to the policy intervention (Tennessee in any year prior to 2010; Ohio and Indiana in any year prior to 2009). The results from the two matched samples are not systematically different, and for brevity, we discuss the results from the full match sample. For Tennessee, 40 institutions were matched (9 in the treatment group and 31 in the control group), and for Ohio, 74 institutions were matched (22 in the treatment group and 52 in the control group). For Indiana, 64 institutions were matched (13 in the treatment group and 51 in the control group).

Limitations

One of the primary challenges of a difference-in-differences approach involves the possible endogeneity of the treatment itself, where there are unobserved determinants that may affect the treatment and the outcomes simultaneously (Besley & Case, 2000). Possibly, the economic, social, and political factors in a state may drive changes in the policy, and those confounders may also determine educational outcomes, such as the number of college graduates. When we diagnosed the existence of endogeneity using observable covariates, our results indicate that the treatment group tends to be larger institutions, and the school size is also associated with our outcome measures. It appears that the endogeneity bias may not be completely ruled out in our analysis.

Another concern addressed in econometric studies is whether a difference-in-differences approach isolates a specific behavioral parameter (Blundell & MaCurdy, 1999; Heckman, 2000). The effect of the presence of concurrent policies may not be the same for all institutions within a state. However, it is also possible that some forms of accountability policies, even if they are not PBF, are implemented in neighboring states, and the policies may also lead to different behavioral responses of the institutions in the states.

In addition, it appears that not all assumptions related to the difference-in-differences estimators may hold in our data. In terms of a parallel trends assumption, the treated and control institutions may not be identical prior to the treatment in the observational data, and we cannot rule out the imbalance between the treated and the control groups. In particular, as will be shown in our visual inspection of the data (Panel B of Figure 1), there was some possibility of the existence of pre-event trends in Ohio’s admission rates (explained in the “Results” section). To compensate for this problem, we examined various alternative model specifications, including a coarsened exact matching method that reduces the imbalance between treated and control groups. We also tested for parallel pretrends using event study models by observing the trends of the coefficients during the pretreatment period. It is nevertheless important to be aware that the results may be subject to bias due to threats to internal validity.

OPEN IN VIEWER

Figure 1.

Trends in group averages of outcome variables prepolicy and postpolicy adoption. Panel A: bachelor’s degree. Panel B: admission rates. Panel C: 25% ACT score. Panel D: 75% ACT score. Panel E: minority students. Panel F: students with federal grants.

Finally, researchers should be cautious when attempting to generalize the results of this study across the nation. We focus on three individual states (Tennessee, Ohio, and Indiana) in this study. Therefore, the results are mostly concerned with the effects of PBF in these states, and the policy effects of these states might differ from those of other states. Thus, generalizing the PBF effects of specific states nationwide may not be feasible.

Descriptive Statistics

Table 2 provides descriptive statistics of the outcome and control variables for the treatment and control groups in Tennessee, Ohio, and Indiana. For the control groups, Table 2 only includes institutions in neighboring states not subject to PBF for each treatment state. To compare the treatment and control states, Table 2 presents values for the years prior to the PBF change. For the outcome variables, Ohio and Indiana had similar tendencies regarding the relationship between the treatment and the control groups, but Tennessee’s results differed. For example, the treatment group in Tennessee produced more bachelor’s degrees than did the state’s control group, whereas the treatment groups in Ohio and Indiana produced fewer bachelor’s degrees than did the states’ control groups. Similarly, the treatment group in Tennessee had higher admission rates and average 25th and 75th percentile ACT scores than the control group, whereas the treatment groups in Ohio and Indiana had lower rates than their respective control groups. In addition, Tennessee had lower percentages of minority students and of students receiving federal grants compared with the state’s control group, whereas the percentages of those students were higher in Ohio and Indiana compared with those of their neighboring states not subject to PBF.

OPEN IN VIEWER

Table 2

Descriptive Statistics for Variables Used in Analyses (Prior to PBF)

Variable	Tennessee		Ohio		Indiana
	Treatment	Control	Treatment	Control	Treatment	Control
Outcome
Bachelor’s degree	1,899.62 (1,125.44)	1,547.45 (1,699.05)	1,380.92 (2,011.01)	1,799.40 (1,569.52)	1,673.78 (2,025.58)	1,799.40 (1,569.52)
	[940.6, 3,827.4]	[0.4, 8,743]	[0, 8,468]	[0, 6,317.25]	[12.5, 6,080.25]	[0, 6,317.25]
Admission rate	0.74 (0.12)	0.69 (0.16)	0.74 (0.15)	0.83 (0.08)	0.82 (0.07)	0.83 (0.08)
	[0.56, 0.88]	[0.13, 0.99]	[0.40, 0.89]	[0.67, 0.94]	[0.72, 0.97]	[0.67, 0.94]
25% ACT score	19.42 (1.84)	18.34 (3.29)	19.96 (3.36)	20.74 (1.87)	18.21 (2.04)	20.74 (1.87)
	[16, 23.2]	[5, 26]	[15, 24.5]	[17.5, 26]	[16, 22.75]	[17.5, 26]
75% ACT score	24.33 (2.08)	23.24 (3.15)	24.83 (3.55)	25.35 (1.82)	23.35 (2.26)	25.34 (1.82)
	[20, 28]	[13, 30]	[18, 29]	[22.5, 30]	[21, 28.25]	[22.5, 30]
Minority students	22.78 (21.98)	34.08 (27.61)	11.66 (16.74)	5.86 (3.51)	11.48 (9.00)	5.86 (3.51)
	[5, 75.2]	[2, 95.2]	[2, 91.5]	[2, 17.75]	[5, 35.5]	[2, 17.75]
Students with federal grants	35.84 (10.01)	36.5 (17.44)	37.48 (15.97)	24.36 (9.05)	29.17 (7.81)	24.36 (9.05)
	[18.4, 50.2]	[0, 76.8]	[8.25, 82.25]	[10.75, 43]	[14.75, 45.75]	[10.75, 43]
Control
FTE enrollment	12,672.98 (7,260.87)	9,413.08 (9,395.04)	9,087.15 (11,027.98)	11,089.37 (9,284.72)	11,715.17 (11,758.4)	11,089.37 (9,284.72)
	[6,162.8, 27,246.4]	[939, 47,418.6]	[612.75, 47,909]	[1,615.5, 38,427.75]	[1,713.25, 37,847.5]	[1,615.5, 38,427.75]
Tuition and fees	6,346.22 (461.61)	6,722.15 (2,361.68)	7,963.66 (3,318.16)	6,771.37 (1,179.49)	7,057.47 (1,285.78)	6,771.37 (1,179.49)
	[5,970.30, 7,259.36]	[0, 14,572.57]	[4,176.10, 20,600.38]	[3,687.39, 8,700.93]	[4,798.00, 9,382.28]	[3,687.39, 8,700.93]
Expenses per FTE	22,548.69 (10,634.65)	22,188.54 (12,726.82)	18,145.62 (8,428.94)	20,826.34 (12,662.24)	19,729.78 (9,039.61)	20,826.34 (12,662.24)
	[14,427.44, 48,454.84]	[8,760.57, 90,736.42]	[10,550.36, 41,513.77]	[10,622.44, 63,191.93]	[12,973.15, 44,354.25]	[10,622.44, 63,191.93]
Unemployment rate	6 (0.99)	5.54 (1.28)	6.08 (0.81)	3.97 (0.89)	5.48 (0.72)	3.97 (0.89)
	[4.92, 8.2]	[3.48, 10.46]	[5.08, 8.05]	[2.45, 5.73]	[4.33, 6.63]	[2.45, 5.73]
Institutions	9	95	36	32	15	32

Note. For brevity, descriptive statistics appear for one control group (each state’s neighboring states not subject to PBF). Means, standard deviations (in parentheses), and minimum and maximum values (in square brackets) are shown. FTE = full-time equivalent; ACT = American College Testing; PBF = performance-based funding.

To display the trends of the six outcomes of interest across the study period (2005–2018), Figure 1 presents the change in mean outcomes over time of the six dependent variables (Panels A–F) for our three treatment groups (Tennessee, Ohio, and Indiana) and two comparison groups (neighboring and nationwide states) for each treatment group. Vertical lines indicate the timing of PBF implementation in each state (Tennessee in 2010; Ohio and Indiana in 2009), allowing a comparison of trends before and after the policy. In Panel A of Figure 1, the graphs indicate that the number of bachelor’s degrees steadily increased during the study period in Tennessee and was trending similarly to other control groups before the enactment of the PBF. The trend line of bachelor’s degrees in Ohio shows small fluctuations around the time of the policy intervention, which is somewhat different from the trends of other control groups.

In Panel B of Figure 1, the trajectory of admission rates displays the highest level of variation among all outcome variables for both the treatment and the comparison groups. In particular, the trends in Ohio and Indiana reveal that admission rates experienced sharp declines at the beginning of the policy intervention and began to increase later. The average 25th percentile of ACT scores and the average 75th percentile of ACT scores show little variation over time in the data (Panels C and D). The trends for the percentage of minority students indicate general increases over time in Ohio and Indiana, whereas it relatively leveled off in Tennessee (Panel E). The percentage of students receiving federal grants in Ohio and Indiana tended to increase significantly at the beginning of policy implementation. Thereafter, Ohio showed a slight decrease 2 years after the policy was implemented, and Indiana showed a tendency to plateau from 3 years after policy implementation (Panel F).

Taken together, these descriptive plots provide preliminary evidence regarding the effect of PBF on the outcome measures, albeit tentatively. In terms of the parallel trends assumption, diagnostic results from these graphs indicate that trends in the admission rates in Ohio and control groups of the state do not appear to be parallel prior to the policy intervention (second graph in Panel B), suggesting that the parallel trends assumption may not hold for this particular outcome.

Difference-in-Differences and DDD

Table 3 reveals the difference-in-differences estimates for the six outcomes of interest for the three case states: Tennessee, Ohio, and Indiana. The three columns for each state, in turn, compare each state’s public 4-year universities with (a) the neighboring states that had not adopted PBF during this study’s analysis period; (b) all nationwide states that did not adopt PBF; and (c) matched institutions via the coarsened exact matching technique. Given that the wild cluster bootstrap produces only p values and confidence intervals, and given that it does not produce standard errors, we present confidence intervals in the tables.

OPEN IN VIEWER

Table 3

Difference-in-Differences Estimates of the Effects of PBF

	Tennessee			Ohio			Indiana
	(1)Neighboring states	(2)Nationwide	(3)Matching	(1)Neighboring states	(2)Nationwide	(3)Matching	(1)Neighboring states	(2)Nationwide	(3)Matching
Bachelor’s degree (ln)	−0.044	−0.089	0.022	0.226	−0.158	0.030	0.344 †	0.172	0.325*
	[−1.848, 1.938]	[−4.603, 4.966]	[−0.246, 0.326]	[−0.949, 1.397]	[−1.258, 1.101]	[−0.118, 0.265]	[−0.675, −0.565]	[−2.422, 2.761]	[0.191, 0.739]
Admission rate	0.028	0.010	0.018	−0.051	−0.054	−0.035	−0.062*	−0.091	−0.074
	[−0.476, 0.382]	[−0.784, 1.053]	[−0.193, 0.195]	[−0.327, 0.288]	[−0.558, 0.484]	[−0.333, 0.430]	[−0.104, −0.081]	[−0.507, 0.414]	[−0.213, 0.052]
25% ACT score	−0.330	−0.186	0.069	0.924	0.738	0.372	0.738	0.331	0.422
	[−6.036, 4.99]	[−6.728, 7.463]	[−0.976, 1.357]	[−5.182, 4.997]	[−2.001, 3.521]	[−6.569, 6.58]	[−1.596, 1.329]	[−2.222, 3.002]	[−1.11, 1.654]
75% ACT score	−0.168	−0.030	0.328	0.466 †	0.161	−0.359	0.853 †	0.400	0.513
	[−3.591, 4.621]	[−5.375, 6.603]	[−1.224, 3.382]	[−0.807, 1.842]	[−2.999, 3.053]	[−7.487, 4.685]	[−0.062, 0.353]	[−2.335, 3.717]	[−2.043, 2.854]
Minority enrollment	−1.805	−3.300	−3.262	−0.354	−1.610	−0.354	0.717	−0.956	−1.082
	[−34.8, 34.73]	[−75.37, 63.71]	[−15.89, 8.664]	[−4.523, 6.215]	[−8.157, 4.134]	[−4.417, 2.992]	[−2.609, 9.806]	[−22.23, 20.56]	[−6.183, 4.238]
Students with federal grants	2.515	1.490	0.206	2.380	0.857	0.587	4.727 †	3.058	1.701
	[−27.14, 25.87]	[−62.93, 58.98]	[−6.072, 5.895]	[−27.03, 12.82]	[−6.953, 8.113]	[−9.802, 6.039]	[−24.61, 19.63]	[−16.9, 23.18]	[−6.528, 13.75]

Note. Columns (1), (2), and (3) represent each state’s comparison groups (neighboring states, nationwide, and matching states, respectively). Institution fixed effects and year fixed effects, in addition to covariates, were added to all model specifications. Confidence intervals from the wild cluster bootstrap are presented in square brackets. ACT = American College Testing; PBF = performance-based funding.

***

p < .001. **p < .01. *p < .05. ^†p < .1.

As we tested a number of hypotheses (54 comparisons for three treatment states, three comparison states, and six outcome variables), some coefficients may be statistically significant by chance (Ward & Ost, 2021). We have thus focused more on general patterns of treatment effects rather than explaining statistically significant coefficients individually.

Overall, no evidence exists that PBF policies in Tennessee and Ohio are associated with key outcome variables in this study, whereas PBF in Indiana shows a few statistically significant results in one specification (Indiana vs. neighboring states). For example, when Indiana is compared with neighboring states, the PBF policy in Indiana is positively associated with the number of bachelor’s degrees. After the policy, Indiana institutions produced an average of 34.4% more bachelor’s degrees than neighboring states did (representing approximately 624 students). On the other hand, Indiana PBF institutions lowered their admission rates and raised the average 75th percentile of ACT scores compared with their counterparts in neighboring states. Indiana PBF institutions have a 4.7% higher rate of students with federal grants compared with public institutions not subject to PBF in neighboring states. Despite several statistical significances in the difference-in-differences estimates discussed above, results must be interpreted with caution as they are not robust across multiple comparison groups.

Table 4 presents the regression results from the DDD model, indicating how universities in three states (Tennessee, Ohio, and Indiana) may respond differently to the three characteristics (tuition levels, endowments, and state revenue share). We tested DDD based on the model with neighboring states, which is our preferred model. This is largely because the imbalance between the treated and the neighboring states appears to be the lowest relative to other comparison groups.

OPEN IN VIEWER

Table 4

DDD Estimates of the Effects of PBF

	Tuition			Endowment			State share
	TN	OH	IN	TN	OH	IN	TN	OH	IN
Bachelor’s degree (ln)	0.157	−0.307	−0.804	0.146	−0.088	0.360	0.129	0.285 †	0.458
	[−5.459, 6.413]	[−0.694, 1.23]	[−2.522, 1.243]	[−2.768, 2.547]	[−0.468, 0.121]	[−1.295, 1.827]	[−4.055, 3.035]	[−0.313, 0.828]	[−1.138, 1.676]
(treat × post)	−0.129	0.332**	0.907**	−0.152	0.172*	0.112 †	−0.092	0.061	0.166 †
Admission rate	−0.004	−0.004	0.093	−0.134	−0.066	0.055	−0.037	0.093	0.074
	[−0.973, 0.721]	[−1.67, −1.44]	[−1.352, 1.452]	[−0.611, 0.449]	[−1.069, 1.727]	[−0.022, 0.164]	[−0.856, 1.099]	[−3.741, 2.507]	[−0.497, 0.630]
(treat × post)	0.032	−0.048	−0.147**	0.141*	−0.004	−0.097**	0.044	−0.060 †	−0.088**
25% ACT score	−0.242	1.775	0.144	0.610	−0.779	0.199	0.390	−1.443	−0.059
	[−13.66, 17.25]	[−36.43, 99.57]	[−4.953, 13.07]	[−8.857, 10.99]	[−14.55, 7.211]	[−7.008, 7.882]	[−16.73, 17.78]	[−16.6, 9.53]	[−1.333, 2.643]
(treat × post)	−0.235	−0.754 †	0.596 †	−0.827 †	1.748*	0.649	−0.482	0.978*	0.768*
75% ACT score	−0.604	0.957	0.028	0.708	−0.878	0.340	0.281	−0.667	−0.580 †
	[−6.253, 8.392]	[−10.65, −6.02]	[−12.4, 13.93]	[−7.225, 10.99]	[−13.46, 10.08]	[−2.111, 2.956]	[−14.76, 17.05]	[−8.39, 15.92]	[−1.466, 1.489]
(treat × post)	0.148	−0.438	0.816**	−0.672*	1.250*	0.608*	−0.280	0.446*	1.105**
Minority enrollment	0.389	−0.005	2.072	−6.960	−0.463	0.969	1.458	0.504*	−0.090
	[−50.85, 42.18]	[−26.13, 22.72]	[−26.12, 32.86]	[−49.73, 32.68]	[−16.57, 5.918]	[−7.301, 13.36]	[−16.73, 18.46]	[−1.41, −0.63]	[−10.8, 14.17]
(treat × post)	−1.897	−0.972	−0.986	3.054	−0.207	0.258	−2.323	−0.355	0.723
Students with federal grants	−0.637	−2.101	−2.136	−7.067	−1.293	0.407	−1.361	0.321	4.175*
	[−107.8, 123.9]	[−35.25, 55.28]	[−45.59, 69.65]	[−90.2, 77]	[−50.1, 41.81]	[−39.16, 34.03]	[−46.52, 44.12]	[−10.96, 0.828]	[−4.17, −1.49]
(treat × post)	2.779	4.400	6.358	7.915**	4.232	4.591 †	2.912	2.468	3.122 †

Note. The DDD model uses neighboring states not subject to PBF as the control group. The DDD estimates in Table 4 indicate results for the higher level group in each of the three subgroups: institutions with higher tuition levels, institutions with larger endowments, and institutions with higher state shares. The parameter attached to treat × post represents the DD estimate for the below median treated observations during the post periods. Institution fixed effects and year fixed effects, in addition to covariates, were added to all model specifications. Confidence intervals from the wild cluster bootstrap are presented in square brackets. ACT = American College Testing; DDD = difference-in-difference-in-differences; PBF = performance-based funding; TN = Tennessee; OH = Ohio; IN = Indiana.

***

p < .001. **p < .01. *p < .05. ^†p < .1.

For tuition levels and endowments, we found no evidence that institutions responded differently in all three states. On the other hand, for state revenue share, institutions in Ohio and Indiana responded differently in several cases. For instance, in Ohio public institutions with higher state revenue shares, the DDD estimate of the effect of PBF on the number of bachelor’s degrees is statistically significant at the 10% level. Moreover, in Ohio public institutions with higher state revenue shares, the DDD estimate of PBF on the percentage of minority students increased by 0.5%. On the other hand, in Indiana public institutions with higher state revenue shares, the average 75th percentile of ACT scores was lower by 0.58 points after the policy. With respect to the DDD estimate of the effect of PBF on the percentage of students with federal grants, 4.2% more students with federal grants were enrolled at Indiana PBF institutions with higher state revenue shares.

In an attempt to provide further details about the institutions, we also included the coefficients and statistical significances for the treat × post term in the DDD table (Table 4). In Tennessee, the terms were all zero for tuition and state share but nonzero for endowment in some cases (e.g., admission rates, average 75th percentile of ACT scores, and percentage of students receiving federal grants). In Ohio and Indiana, in some cases, the treat × post terms were nonzero for all of tuition, endowment, and state share, which means that PBF may have some effect on the below median schools.

Event Study Analysis

For the purpose of visualizing the event study results, Figure 2 displays the point estimates and 95% confidence intervals for the treatment effect over time. The graphs are shown in the order of Tennessee, Ohio, and Indiana for each of the six outcome variables (Panels A–F). In terms of the pretreatment patterns, in Panel B of Figure 2, the second graph exhibits a downward pattern of the pretreatment coefficients for admission rates in Ohio. This may signal that pretrends may threaten the internal validity of the results, and the parallel trends assumption may not hold for this particular outcome. In terms of the posttreatment patterns, the graph for minority students in Ohio (second graph in Panel E of Figure 2) clearly shows a declining pattern in the proportion of minority students after the policy’s adoption.

OPEN IN VIEWER

Figure 2.

Event study analysis of effects of PBF on outcomes. Panel A: bachelor’s degree (TN, OH, IN). Panel B: admission rates (TN, OH, IN). Panel C: 25% ACT score (TN, OH, IN). Panel D: 75% ACT score (TN, OH, IN). Panel E: minority students (TN, OH, IN). Panel F: students with federal grants (TN, OH, IN).

Supplemental Tables A1 and A2, available in the online version of this article, provide the results from the event study model specifications. Online Supplemental Table A1 presents results on the number of bachelor’s degrees and the enrollment of underrepresented students (minorities and low-income students). Online Supplemental Table A2 displays the results for admission practices, including admission rates and ACT scores. We report robust standard errors clustered at the state level, but the results are very similar using multiway robust standard errors clustered by state and year instead.

In online Supplemental Table A1, we use observations from several years’ pretreatment and posttreatment to examine the variations in the treatment effect over time. The year prior to the policy implementation is omitted, as it serves as the reference year (2009 for Tennessee and 2008 for Ohio and Indiana).

In terms of bachelor’s degree production in Tennessee, no emerging patterns exist for the treatment effect coefficients for the eight post-PBF periods, although one coefficient is statistically significant (first year after the policy’s adoption). On the other hand, clear patterns are observed in Ohio and Indiana. Bachelor’s degree production in Ohio began to increase in the fourth year after the policy was adopted, and bachelor’s degree production in Indiana began to increase in the third year after the policy’s adoption. These lagged effects of PBF on bachelor’s degree production in Ohio and Indiana may suggest that the policy’s effect may take some time to appear.

For the effects of PBF on the proportion of minority students, a declining pattern emerges in Ohio, with negative coefficients in the seventh year of the policy’s implementation. On the other hand, for the effects of PBF on the proportion of low-income students (as measured by students with federal grants), the results indicate that PBF institutions in all three states enrolled more low-income students starting from the beginning of the PBF policy. Although in Ohio, the effect appears only in the first 3 years and disappears after this period.

Online Supplemental Table A2 indicates that in terms of admission rates, Indiana PBF institutions significantly lowered the admission rates and raised the ACT score standard (25% ACT score and 75% ACT score) starting from the first year of the PBF. This suggests that admission practices in Indiana were altered shortly after the policy was implemented.

Discussion

The findings of this study highlight that PBF policies have differential effects on the key outcomes of interest by institutional characteristics, by policy implementation period, and across states. First, in terms of the first research question, the difference-in-differences results indicated some of the strongest detected effects of PBF in Indiana institutions subject to PBF. These institutions increased the number of bachelor’s degrees earned while simultaneously lowering their admission rates. This finding aligns with the results from prior research (Gándara & Rutherford, 2020; Umbricht et al., 2017), suggesting that Indiana institutions change their behaviors by increasing their selectivity and admitting students who are more likely to graduate. Considering that Indiana only uses PBF for a relatively small share of institutional funding compared with other states (e.g., Tennessee, Ohio), the strongest response to PBF in Indiana appears counterintuitive. Although some reasonably argue that increasing the stakes (e.g., the proportion of state appropriations) is one of the most significant factors for improving the effectiveness of the PBF policy, others caution that there may be many other intangible components, including prosocial values (perceived social impact of one’s work; Moynihan et al., 2012; Umbricht et al., 2017). As the ecology of games perspective suggests, Indiana institutions may also respond differently to the PBF game, taking into account different goals, preferences, and payoffs in their different political and socioeconomic structures. It must be noted that using only the number of graduates as a key performance indicator of the PBF formula, without a more sophisticated PBF design, may have unintended consequences.

Second, the findings from DDD, which answer the second research question, demonstrate that the effects of PBF on the outcomes differ by institutional characteristics. Depending on their state revenue shares, institutions responded differently to the PBF policy in Ohio and Indiana. Overall, 4-year public institutions with higher state revenue shares in Ohio were more responsive to PBF, especially in terms of minority enrollment, whereas 4-year public institutions with higher state revenue shares in Indiana enrolled more low-income students than their counterparts. The results suggest that the way in which institutions with specific attributes respond to PBF involves an intricate mechanism that varies by state, partly because policy environments and specific program designs, such as the funding amount, introduction methods, and indicators, are different for each state.

In addition to the results from difference-in-differences and DDD, important patterns emerged in the event study findings, which relates to our third research question. The event study results indicated that Indiana institutions were quick to change their admission practices in response to PBF. Conversely, bachelor’s degree achievement began to increase in the 3 to 4 years after the policy was implemented in Indiana and Ohio, respectively. This suggests that although institutions subject to PBF yielded more bachelor’s degrees, it might take some time to observe the policy effects. Prior studies that employed only a difference-in-differences approach generally reported that PBF was not successful in increasing degree completion, as the policy effect was not captured with a single, aggregated treatment coefficient in the method they employed. However, when we used an event study analysis in addition to the difference-in-differences technique, we found evidence that PBF may have a delayed effect on bachelor’s degree production. Nevertheless, one caveat needs to be noted when one is interpreting these long-term effects of PBF, largely because patterns of the policy effect were not the same across all three states. Tennessee’s result differed from those of Ohio and Indiana, showing no long-term effect. It is thus important to recognize that the effects of PBF may differ by state rather than generalizing them to all nationwide states.

The association between PBF and the proportion of underrepresented students enrolled is less straightforward. It appears that minority enrollment significantly decreased over time in Ohio, whereas low-income student enrollment increased over time in all three states, although the effect disappears later in Ohio. Gándara and Rutherford (2020) also found that 4-year institutions subject to PBF had higher percentages of students who received federal grants. Sequentially, the researchers argued that those states that gave greater weight to low-income students may observe the effect of alleviating the negative consequences of PBF on those students. It is nevertheless interesting to note that the same logic does not apply to minority student enrollment. Although Ohio’s program awards additional points for minority students, among others, the results from this study reveal that institutions subject to PBF in Ohio actually enrolled fewer minority students over time. It appears that incorporating equity metrics into PBF formulas and adding extra weight for specific groups of underrepresented students does not necessarily exert PBF’s intended effects in practice. Jones et al. (2017) maintained that the weighting for at-risk students may serve only to symbolize equity or to obtain political support. As principal–agent theory suggests, rather than merely increasing admission opportunities for minority students, higher education institutions could try other efficient ways of attaining better scores under a PBF scheme. In other cases, even if colleges and universities strive to improve their racial diversity, they may lack the capacity or resources to do so.

Implications for Practice

The results of this study have practical implications for the attainment of the policy goals of PBF, as well as propositions to better serve underrepresented students in higher education. First, it is important to fully garner the interest and understanding of key stakeholders, including faculty members and administrative officials (Dougherty et al., 2016; Miao, 2012). Given that a single principal or agent cannot improve student outcomes (Hillman, 2016), every possible agent related to the complicated policy process should be actively involved in each stage of planning, implementation, and evaluation.

Second, PBF needs to be sophisticatedly designed and continuously monitored to assess whether the policy has the intended effect. For example, performance indicators may need to reflect the new environmental changes in higher education, such as artificial intelligence and robotics, to strengthen accountability for higher education finances, as well as to secure the competitiveness of higher education institutions. Furthermore, to ensure the quality of higher education, indicators related to student learning and experiences should be considered. Such indicators should accurately measure student learning and competencies, as well as enable institutional comparisons (Dougherty & Natow, 2015; Miller, 2016). In addition, it is advisable to divide performance metrics into several types. The PBF in Oregon, for instance, consists of various types of metrics, such as common metrics (generally applied to all institutions) and self-regulating metrics (mandated to academic leaders and institutional administrators), thus allowing institutions to better reflect their missions and student characteristics.

Third, states should develop the optimal way of embracing equity measures in their PBF systems. For institutions to better serve traditionally underserved (minority and low-income) students, it is necessary to sufficiently weight carefully constructed equity measures (Cielinski & Pham, 2017). In a similar context, a needs-based funding model that supports institutions with higher financial needs is likely to be more efficient and effective compared with a merit-based funding model that compensates for institutions that are already performing well. The primary obstacle with the low number of students graduating from college is likely that institutions serving low-income and underrepresented students have insufficient resources (Hillman, 2016; Jones et al., 2017). Undoubtedly, additional support and resources are required to serve underrepresented students in higher education.

Finally, the importance of the reliable valid data cannot be overestimated. For the purpose of measuring student outcomes precisely, all related data should be collected in a way that takes the contexts and circumstances of PBF in each state into consideration (Rorison et al., 2016). Texas, for example, collects and utilizes data for employment outcomes by tracking the earnings of students in the state’s technical college system (Selingo & Van Der Werf, 2016). State policymakers may also need to build research capacity at the institutional level so that institutions may keep track of their own progress through updated data, as well as proactively utilize the results.

Implications for Future Work

We suggest several fruitful areas for future research. Chiefly, empirical studies regarding the effects of changes in PBF on student outcomes should be continued. Although Tennessee improved its policy in 2015 to give greater weight to marginalized student populations (Callahan et al., 2017), the effects of this type of policy change require thorough analysis, and the policy feedback needs to be provided based on credible, convincing empirical evidence. An examination of additional outcome variables that are not investigated in this study, such as the acquisition of degrees with specific major subjects (e.g., STEM-degree production), which several states began to embrace in their PBF formulas, could be a worthwhile subject for future research. The extension of this study’s analysis model is another promising area for future research. Although PBF has no direct effect on certain outcomes, it might indirectly affect outcomes through other factors. Possibly, the effects of PBF may be mediated by other factors, such as university expenditure. In addition, although this study included FTE enrollment and tuition as control variables, it is plausible that institutions respond to PBF by increasing or decreasing FTE enrollment or changing tuition. More thorough research is needed to capture how the effects of PBF are mediated and how PBF systems may work to affect certain outcomes.

More nuanced research is also needed to disentangle how institutional features may affect their ability to improve performance metrics. As is well documented in qualitative studies (e.g., Dougherty et al., 2016), student body composition (e.g., underprepared students, low-income students, part-time status) may operate as an obstacle for institutions to respond effectively to PBF policies. More sophisticated quantitative research is also needed to tease out how this mechanism functions, and to provide practical information on ways to address these concerns in state policy designs. Furthermore, considering that institutions may respond to PBF by changing academic and student service programs, in-depth studies are necessary to address these institutional intermediate processes based on students’ actual experiences. For example, several qualitative studies (e.g., Dougherty et al., 2016; Natow et al., 2014) have delineated the unintended consequences of PBF on undermining academic standards including the pressure to inflate grade or lower academic demands to facilitate degree completions. As a quantitative study utilizing publicly available data, the present research has limitations in deriving empirical evidence in this regard. Given that the National Survey of Student Engagement (NSSE) data provides variables related to institutional changes, such as interactions with faculty and institutional support, if the overall reliability and validity of college student surveys such as NSSE are improved (Porter, 2011) and more universities participate in NSSE data collection in the future, the sophisticated systematic analysis of NSSE data in conjunction with PBF could capture some institutional intermediate outcomes and yield meaningful results.

Although the present analysis did not distinguish PBF states with at-risk bonuses from those without at-risk bonuses, future research could investigate how different types of at-risk premiums in PBF formulas affect at-risk students’ outcomes differently. While a growing number of states are adopting provisions that incentivize at-risk students in their PBF formulas, considerable variations exist across states (Kelchen, 2018). Institutions subject to at-risk provisions likely differ from institutions not subject to the same provisions. In this context, exploring the differential effect of at-risk components deserves further investigation.

Finally, it is important to probe how the context of the time period being studied might affect the external validity of the results. Each of the PBF policy changes in this study occurred during the onset of the Great Recession where the labor market was quite poor and college enrollment was increasing. Furthermore, state budget cuts were also hitting higher education and federal financial aid concurrently. The present study controls for unemployment rates but this might not fully capture the effect of the Great Recession on the generalizability of the estimates. As more states initiate PBF policies, especially in a time of economic stability, further empirical studies are needed to determine whether PBF policies have differential effects according to the time period being studied. It is plausible that the same policy might have different effects if it were implemented during a time of general economic expansion. Such examinations will contribute to the general understanding of PBF moving forward.