Performance Funding Policy Effects on Community College Outcomes: Are Short-Term Certificates on the Rise?

Abstract

Objective: Performance funding (PF) policies allocate a portion of state funding to colleges based on student outcomes. This study is the first to account for policy type and design differences, and explores the impact of performance funding on three levels of credential completions: short-term certificates, medium-term certificates, and associate’s degrees. Method: We create a panel dataset of 751 two-year colleges from years 1990 to 2013 using data from the Integrated Postsecondary Education Data System. We conduct a series of analyses using difference-in-differences with the inclusion of college- and state-level control variables. Results: We find that, on average, performance funding produces no significant changes in completions of any of the three credentials. Policy types characterized by a greater proportion of funding tied to the base budget, mission differentiation in performance metrics, inclusion of underrepresented student metrics, and longer periods of operating years produce an increase in short-term certificates, no significant change in medium-term certificates, and a decrease in associate’s degrees. Contributions: This study’s findings suggest that because awarding more short-term certificates is a relatively quick and cost-effective way to capture performance funds, colleges might be engaging in a path of least resistance by churning out short-term certificates and redirecting focus away from associate’s degrees, which is concerning given that short-term certificates generally offer limited labor market benefits compared to medium-term certificates and associate’s degrees. Our results also underscore the importance of policy designs in explaining differential impacts on credential completion.

Keywords

state policy higher education finance accountability performance funding certificates difference-in-differences

Performance funding (PF) policies for higher education tie a proportion of state appropriations to student retention and completion metrics (Burke, 2002). In the community college sector, the most frequently incentivized outcomes are earned associate’s degrees and certificates—which are defined as postsecondary credentials requiring less than 2 years of study. Previous research casts doubt on the effectiveness of PF on increasing associate’s degree attainment (Tandberg, Hillman, & Barakat, 2014), while recent evidence suggests that these policies generate increased certificate awards (Hillman, Tandberg, & Fryar, 2015). In all but three state performance funding formulas,¹ each associate’s degree secures the same amount of funding as each certificate, although it is quicker and less costly for a college to graduate students from certificate programs. There exists much disciplinary variation with regard to economic advantages of certificate attainment—Graduates experience wage gains in some fields but not others (Grubb, 2002a; Liu, Belfield, & Trimble, 2014). On average, short-term certificates that take less than a year to complete provide limited benefits in terms of wage gains and chances of employment, compared with a high school diploma (Dadgar & Trimble, 2015). Colleges may be responding to performance funding by creating new certificate programs or routing students into these programs, regardless of labor market value. If performance funding incentivizes colleges to engage in certificate production, this is a potentially problematic policy side effect, also known as an unintended consequence.

This study examines whether there is a relationship between operating a community college pay for performance mechanism and certificate completions. Our research extends the preliminary single-state research (i.e., Hillman et al., 2015) on certificate output to a national sample of states. Findings will inform state policymaker decisions to redesign performance funding to better incentivize degree completion as well as reduce potential unintended consequences. Community colleges are highly dependent on state funding, and fears about funding cuts may prompt colleges to engage in practices incongruent with one of their core missions—to prepare students for jobs and increase their marketability in the labor force. Specifically, our study poses the following research questions:

Research Question 1: Does the existence of a state performance funding policy for community colleges increase the number of short-term and medium-term certificates awarded?

Research Question 2: Are community colleges shifting priorities away from associate’s degrees to certificates based on performance funding incentives?

Research Question 3: Does variation in policy design explain any effects on credential completions?

Literature Review

Certificates and Labor Market Returns

Short-term certificates are defined as postsecondary credentials requiring less than 1 year of study, and medium-term certificates as those taking between 1 and 2 years (Carnevale, Rose, & Hanson, 2012). Long-term certificates, a third category, require 2 to 4 years of instructional time. Between 2000 and 2010, certificate production increased by 151% nationally (Dadgar & Trimble, 2015). Certificates make up 25% of all sub-baccalaureate credentials awarded, up from 16% a decade ago, while associate’s degrees have grown more modestly (Dadgar & Trimble, 2015). Over 1 million certificates were awarded in 2010, up from 300,000 in 1994, and as of 2012, certificates were the second most common postsecondary award after BA degrees (Carnevale et al., 2012). Most certificates are in vocational fields, while others consist of liberal education in which students can transfer these general credits to a further degree. Certificates are more concentrated among students of color. Yet, many of these underrepresented students must first take remedial coursework before earning college credit and are more likely than traditional college-aged students to face competing professional and familial demands (Carnevale et al., 2012).

Published studies on pre- and postcollege earnings of graduates who complete certificates tend to use student-level data and report quarterly or annual earnings as differences (in dollar amounts or percentage gains), which takes into account differences due to number of hours worked (Belfield & Bailey, 2011). Earlier studies found variation among disciplines and between males and females, although, on average, there were negligible wage returns on short-term certificates when compared with taking college credit without earning a credential (Grubb, 2002a, 2002b). With respect to associate’s degrees, a review of studies published from 1995 to 2009 concluded that almost all studies found positive earnings gains from associate’s degrees, with an average of 13% for males and 22% for females (Belfield & Bailey, 2011).

Using data from Washington state, researchers found that short-term certificates on average had a negative earnings effect for women (−2.8%) and no effect for men (Dadgar & Trimble, 2015). Disaggregated data revealed a considerable 22% wage gain for men who obtained a short-term certificate in protective services. Longer term certificates were beneficial in nursing and allied health fields. Compared with earning no credential, female associate’s degree recipients earned 6.3% more and male associate’s degree recipients earned 2.1% more. In Michigan, Bahr et al. (2015) found that short-term certificates (defined in the study as credentials requiring less than one semester) did not produce wage gains for either females or males. By contrast, females did earn 14% more for completing a long-term certificate while males saw no effect. Associate’s degrees generated the greatest increase in earnings: 39% for females and 13% for males (Bahr et al., 2015). Using data from Kentucky, Jepsen, Kenneth, and Coomes (2014) concluded that earnings gains from associate’s degrees were about 56% for women (US$2,363 a quarter) and 24% for men (US$1,484 a quarter). Gains from longer term certificates were 45% for women (US$1,914 a quarter) and 21% for men (US$1,265 a quarter). Much smaller returns were observed for short-term certificate earners: 7% for women and 5% for men, which amounted to only US$300 per quarter (Jepsen et al., 2014). Finally, a study of North Carolina found that both women and men who held short-term certificates earned significantly lower earnings per quarter than students who completed credits but no credential (US$347 less for women, US$279 less for men; Liu et al., 2014). Consistent with other studies, females benefited from longer term certificates but males did not. Associate’s degrees resulted in wage gains of 61% for females and 25% for males.

To summarize, average wage gains from short-term certificates appear to be negative to null, or minimally positive. Long-term certificates were generally more advantageous for women than men, while associate’s degrees produced the most payoffs, on average. Labor market benefits were concentrated in nursing and health care fields, and protective services especially stood out as beneficial for males. The studies reviewed underscore the timeliness and importance of our study. Performance funding policies as they currently stand do not award differential funding based on the varying market values of community college credentials, such as by the discipline and level of the credential. This study examines whether the policy, aimed at helping students earn a postsecondary credential and ultimately secure gainful employment, leads to disproportionate growth in short-term certificates, which may in fact be counterintuitive to the policy’s aims.

Performance Funding

The last decade has seen a resurgence in performance funding policies for public higher education. These policies use a formula to tie the allocation of a portion of state appropriations to institutions based on outcomes rather than enrollments (Burke, 2002; Dougherty & Reddy, 2013). The goal of these policies is to improve student retention and completion, and colleges are presumably incentivized by financial rewards to achieve these goals. A number of states adopted performance funding policies in the 1990s, but many abandoned their programs, often due to state fiscal stress (Burke, 2002; Dougherty, Natow, & Vega, 2012). In the last 15 years, prompted by the national college completion agenda and the public accountability movement, performance funding is regaining traction. As of January 2016, 30 states are developing or implementing a policy (National Conference of State Legislatures, 2015; Snyder & Fox, 2016). Performance funding also enjoys the support of influential organizations including the Bill & Melinda Gates Foundation, the Lumina Foundation, and Complete College America.

The earlier generation of performance funding has been coined PF 1.0, where funding was a bonus amount above the base state budget. More contemporary policies, PF 2.0, embed funding within the base formula (shave off the base budget and designate this amount as performance-based funds) and typically tie a higher percentage to outcomes (Dougherty & Reddy, 2013). The underlying theory of action is that by placing more funds at stake, institutions will be motivated to revamp internal operations, such as student support services and teaching practices that facilitate graduation (Dougherty et al., 2013). In recent years, more states have adopted PF 2.0, although evidence points to a policy learning effect in which states are delaying policy adoption until after consequences and impacts can be better understood in neighboring states (Li, 2017a). Frequently incentivized metrics include year-to-year retention rates, graduation rates, total degrees awarded, and degrees per full-time equivalent (FTE) enrollment (Li, 2014), and the majority of states allocate 5% to 10% of total state support based on performance (National Conference of State Legislatures, 2015).

A national study on performance funding impacts demonstrated minimal effect on associate’s degree completions, but the authors did not model certificates (Tandberg et al., 2014). In a study using Washington state data, the policy was found to produce increases in short-term certificates, decreases in long-term certificates, and no changes in retention rates or associate’s degrees (Hillman et al., 2015). The authors speculated that Washington community colleges were awarding more short-term certificates simply to capture funding or avoid funding losses. Yet, a 2010 internal study by the Washington State Board concluded that the disproportional growth in short-term certificates was in part due to better data tracking by colleges and that growth was concentrated in disciplines leading to higher wage jobs (Li, 2017b).

When designing policies, policymakers may not have anticipated a potential disproportional growth in short-term certificates. Even in performance formulas where associate’s degrees secure more funds, the number of short-term certificates can still be more rapidly increased given their shorter completion time. Consequently, colleges may be tempted to churn out certificates because it is the path of least resistance. Strategies to do so include creating new certificate programs, adding an embedded certificate along the way to earning an associate’s degree, automatically awarding certificates once students reach a certain number of credit hours, and routing existing or recruiting new students into these programs (Ness, Deupree, & Gándara, 2015). Faced with a greater selection of academic programs, students might choose to pursue a certificate over a lengthier associate’s degree, or discontinue enrollment once earning an embedded certificate even though the initial intent was to complete an associate’s degree.

Conceptual Framework

To explore the association between perofrmance funding policies at 2-year institutions and the attainment of credentials by level (i.e., short term, medium term, associate’s degree), we turn to resource dependence theory (RDT), new public management (NPM) theory, and principal–agent theory, and apply the idea of anticipatory changes. These theories are complementary—They can all frame the interrelationship between states and institutions and propose expected behaviors due to such relationships. RDT posits that organizations depend on contingencies in the external environment (Pfeffer & Salancik, 1978). Framed in RDT terms, states and institutions operate as agents in symbiotic interdependence. The output of states, performance-based funding, is an input for institutions. The extent to which an institution complies with the state’s attempts to control the institution varies based on the institution’s dependence on these external resources; an institution will only adjust to the degree that conditions of scarcity and uncertainty involve an element of critical organizational independence. For instance, a community college that receives 20% of its base state funding based on outcomes and 60% based on enrollments may perceive its dependency on state funding as critical and will respond urgently to these financial incentives. In contrast, a research university with access to alternative revenues such as out-of-state tuition payers, federal grants, philanthropy, and auxiliary operations is less likely to respond to external demands.

The NPM theoretical framework for public organizations emphasizes performance metrics and pay for performance (Frey, Homberg, & Osterloh, 2013), which relates to principal–agent theory, whereby the principal (the state) employs agents (colleges within the state) to whom they delegate authority, and agents are expected to exert efforts to achieve the principal’s goals (Kivistö, 2008). According to Frey et al. (2013), “High-powered monetary incentives are assumed to align the interests of the agent and the principal. Following the theoretical price effect, the higher the price the more effort is exerted” (p. 951).

In addition, we incorporate the notion of anticipatory changes discussed by Husig and Mann (2010), who proposed that leaders of higher education institutions propel changes “as soon as new ideas can be implemented or are responsive to trends that are assumed or expected to emerge” (p. 182). In contrast, “followers or adopters realize reactive changes that directly respond to developments in the environment of the organization” (Husig & Mann, 2010, p. 182). Performance funding is a new idea in the sense that it is new to the institutions affected by the policy (or the latest version of the policy is new), which is relevant to the idea of anticipatory changes. That is, if a state adopts the policy in one year (such as via legislative statute or system-level approval), institutions will potentially be responsive to the policy in that same year or in the year following adoption, even if performance-based appropriations are not allocated immediately.² Performance funding policies take several years to develop, design, and formalize, and often involve multiple policymakers, researchers, external consultants, institutional leaders, and/or higher education commissions. Actions taken during the years leading up to a funding bill represent a political signal to institutions, who are made aware of assumed or anticipated adoption of performance funding and begin preparing for future consequences of the policy. Thus, we conceptualize policy change as occurring as early as the adoption year.

Next, the unintended consequences of performance funding can be framed using concepts from NPM and RDT. Specifically, performance management can lead to negative consequences such as vanity—“the use of measures to make the organization look good”—and inanity—“adopting metrics without considering consequences for human behavior” (Newcomer & Caudle, 2011, p. 111). Pfeffer and Salancik (1978) proposed that organizations are coalitions “altering their purposes and domains to accommodate new interests, sloughing off parts of themselves to avoid some interests” (p. 24). To manage external pressure, organizations may find it necessary to engage in activities far removed from their core purpose. Applied to community colleges, when faced with the possibility of severe funding cuts, they may slough off a portion of their mission—to help students graduate with associate’s degrees and facilitate transfers to 4-year institutions—and instead shift priorities to increasing certificates.

Conceptualizing and empirically accounting for policy variation in performance funding are also important—a topic that researchers and policymakers have struggled with in the past. In 2014, HCM Strategists, an advocacy organization, developed a policy classification system organized around a series of characteristics that represent policy sophistication and utilization of best practices, which was updated in 2016 (Snyder, 2014; Snyder & Fox, 2016). Policy types range from I to IV, with IV being the most advanced. HCM Strategists has explored performance funding quite extensively and been involved with consultation and advocacy regarding the policy (e.g., HCM Strategists, 2011, 2012), and was the only source we could find that has systematically utilized a combination of multiple policy features to develop a typology.

In Table 1, we display the major characteristics differentiating each policy type. As seen, Type II and above take away at least some portion of funding from the base budget, which colleges must earn back by meeting performance goals (PF 2.0), as opposed to Type I, which allows colleges to secure bonus funds on top of the base. Type I and Type II policies are considered to offer low levels of performance funds at less than 5% of the total budget, while Type III policies allocate a moderate level up to 25% and Type IV represents stronger, more consequential policies that allocate substantial levels of funding at 25% or greater.

Table 1.

Performance Funding Policy Typology.

	Type I	Type II	Type III	Type IV
State completion goals	Yes/no	Yes/no	Yes	Yes
Base funding (only or in addition to bonus funding)	No	Yes	Yes	Yes
Proportion of funding tied	<5%	<5%	5%-24.9%	≥25%
Funding level	Low	Low	Moderate	Substantial
Both 2-year and 4-year sectors	No	Yes/no	Yes	Yes
All colleges within sector	Yes/no	Yes	Yes	Yes
Mission differentiation metrics	No	Yes/no	Yes	Yes
Completion metrics	Yes/no	Yes	Yes	Yes
Underrepresented student metrics	Yes/no	Yes/no	Yes	Yes
Sustained for 2 or more consecutive fiscal years	No	No	No	Yes

Source. Adapted from Snyder (2014) and Snyder and Fox (2016).

Note. Yes/no indicates a policy can be in this category whether it meets the requirement or not. Snyder and Fox identify Type I as having no completion metrics, but we code states with completion metrics and only bonus funding as Type I (all states in our sample have completion metrics).

Applying our conceptual framework of RDT, NPM, and principal–agent theory, we hypothesize that colleges with greater reliance on state funding, characterized by base funding and higher proportions of funding, are more likely to change behaviors in response to monetary incentives. Larger incentives and consequences should engender more pronounced policy responses.

Moreover, we anticipate that performance funding policies that have been sustained for more than 2 fiscal years (Type IV), representing more prolonged interdependency between the college and state, will have greater effects on credential completion. The longevity of policies not only has implications in the form of concrete funding allocations but also indicates future sustainability of the policy, which may cause colleges to pay more attention to delivering the outcomes desired.

Another interesting feature in performance funding policies is the use of underrepresented student metrics (e.g., extra funding or premiums for students of color, Pell grant recipients, students of age 25 and older, and first-generation students who achieve performance goals), for which Types I and II may or may not incorporate, in contrast to the definite inclusion of such metrics in Types III and IV. Adding extra funding for underrepresented students is intended to avoid disincentivizing colleges from enrolling students who require more resources to be retained. By accounting for these added investments in the form of funding premiums, states encourage colleges to continue serving these students. We propose that the existence of underrepresented student metrics avoids the performance management pitfall of inanity (adopting metrics without consideration of consequences). It demonstrates greater consideration of higher education demographics in the state, and may improve institutional support of perofrmance funding, particularly at colleges that would have otherwise been disadvantaged. In essence, tailored student metrics, representative of the more sophisticated policy types (Types III and IV), is hypothesized to increase the policy’s effect on outcomes.

Data

Policy Treatment and Sample

We built a state-level panel dataset from 1990 to 2013, during which the waves of performance funding policy adoptions for community colleges took place. We operationalized the policy treatment as the existence of a state performance funding policy for community colleges inclusive of the adoption year. The performance funding states in our dataset all allocated appropriations based on completion metrics measured as total credentials, credentials per FTE, and/or graduation rate. These policies contained metrics that reward associate’s degrees and certificates separately, or utilized a single metric capturing all credentials.³

Among scholarly literature and policy reports, conflicting information exists on the exact years during which a state operated performance funding and the characteristics of the policy. We relied on the following primary sources of data in efforts to address discrepancies: Burke (2002), Dougherty and Natow (2015), National Conference of State Legislatures (2015), Snyder (2014), and Snyder and Fox (2016) . In addition, we consulted other sources, including articles, funding commission reports, and websites in attempts to confirm policy years and improve accuracy (Burke & Minassians, 2003; Crowder & Janosik, 2001; Davies, 2014; Friedel, Thornton, D’Amico, & Katsinas, 2013; Harbour, 2002; Jones, 2013; Kansas Board of Regents, 2013; Li, 2017a; Ohio Department of Higher Education, 2012; Preis, 2012; Rabovsky, 2014; SRI International, 2012; Texas Association of Community Colleges, 2017).

For each college located in a state that operated a performance funding policy, we coded a dummy variable as 1 during the adoption year and each year thereafter. Given our conceptual framework of anticipatory changes, we considered the policy to have potential impacts starting in the year that legislation was passed or, in some instances, when the policy was approved at the system level. That is, a policy adopted or approved in 2005 was coded as operating in 2005. Granted, a state does not always start allocating performance-based funds right away, and in some cases, actual funding allocations are delayed for several years.

Operating years starting with the adoption year and policy types for performance funding states are listed in Table 2 and visually displayed in Figure 1. As seen from this figure, some states adopted and then discontinued policies during the period analyzed. The policy dummy that we created is equal to 1 for these states only when a policy is in place. For each state–year with a policy, we also created a dummy variable representing each of the four policy types. Policies that operated in years before the typologies were developed had characteristics representative of Type I policies, featuring low amounts of funding and bonus funding. The exceptions were South Carolina, which more closely resembled Type II in its effort to use base funding (Burke, 2002).⁴

Table 2.

State PF for Community Colleges.

State	Years of policy adoption and operation
Arkansas	1995-1997	2011-present
Colorado^a	1994-2004	2011-present
Florida	1996-2008
Hawaii	2011-present
Idaho	2000-2005
Illinois	1998-2002	2011-present
Indiana^b	2007-present
Kansas^c	1999-2008	2013-present
Kentucky	1994-1998
Louisiana^d	2010-present
Massachusetts	2013-present
Michigan	2012-present
Minnesota	1994-1998	2013-present
Missouri	1993-2002	2013-present
Montana	2013-present
Nevada	2013-present
New Jersey	1999-2003
New Mexico^e	2003-present
North Carolina	1998-2008	2012-present
North Dakota	2013-present
Ohio	2009-present
Oklahoma	1997-present
South Carolina	1996-2002
Texas	2013-present
Utah	2013-present
Virginia^f	2005-present
Washington	2007-present
Wisconsin	2013-present
Wyoming	2012-present

Source. Burke (2002), Dougherty and Natow (2015), NCSL (2015), and Snyder and Fox (2016).

Note. NCSL = National Conference of State Legislatures; PF = performance funding.

Colorado had a policy change in 1999, although the policy continuously operated from 1994 to 2004 and remained Type I. Policy was Type I in 2011. We code it until the end of our dataset (2013), but it was updated to Type III for 2016 (Snyder and Fox).

Indiana went from Types I to III in 2009.

Kansas had a policy change in 2002, although the policy continuously operated from 1999 to 2008 and remained Type I.

Louisiana is PF 2.0 (Dougherty and Natow), but we code as Type I according to Snyder and Fox.

New Mexico is PF 1.0 from 2003 to 2011 and PF 2.0 in 2012-present (Dougherty and Natow), but we code as Type I according to Snyder and Fox.

Virginia had a policy change in 2011, although the policy continuously operated from 2005 to present and remained Type I.

Figure 1.

Performance funding policy timeline.

We then added college-level data to our dataset of state policy years. Colleges in our sample were defined in the Integrated Postsecondary Education Data System (IPEDS) as having a public control, an associate’s college basic Carnegie Classification in 2010, and associate’s degree as the highest degree awarded. Our sample included all states except for Tennessee, since Tennessee introduced performance funding in 1979 (Burke, 2002). Given that Tennessee has had the policy during all years of our sample, its inclusion does not allow the estimation of policy impacts directly. Furthermore, given its early introduction, estimating the impact for each operation year is difficult because no policy has been around as long as Tennessee’s. While it is not impossible to include the state in our analysis, we choose to omit it to avoid complications in our lagged policy effect analyses. Our total sample consisted of 751 colleges. The unit of analysis was college–year for an analytical sample size of N = 16,953.

Outcome Variables

We modeled three outcomes of interest: certificates awarded in programs taking less than 1 year (short term), certificates awarded in programs taking 1 to 2 years (medium term), and associate’s degrees awarded. We deliberately excluded certificates that required more than 2 years of study to more precisely evaluate our second research question of whether certificates were being prioritized at the expense of associate’s degrees. In terms of student demand, we did not view certificates of more than 2 years to be in direct competition with associate’s degrees because of the certificates’ extended length of study. IPEDS data also showed that many colleges did not award long-term certificates or had missing data.

We used a log transformation on credential/degree counts to help bring outliers closer to the average of the distribution and create a more symmetric distribution. Degrees per FTE is another way to measure degree output, yet we chose not to use this outcome because it creates difficulties in determining whether increases are due to degrees attained as opposed to enrollment changes.

Control Variables

To better isolate the policy effect of performance funding, we included a series of time-varying college-level variables from IPEDS demonstrated by previous research to impact 2-year retention and/or completion. We included the proportion of African American/Black, Hispanic/Latino, Asian American and Pacific Islander, and Native American students enrolled (with proportion of White students omitted due to collinearity), because there are disparate completion rates across racial backgrounds (Dietrich & Lichtenberger, 2015; Feldman, 1993; Porchea, Allen, Robbins, & Phelps, 2010), with studies finding that Hispanic and African American students are less likely to complete associate’s degrees (Bailey, Jenkins, & Leinbach, 2005). Previous research finds that a higher enrollment of female students and full-time students is positively associated with retention and/or completion (Bailey, Calcagno, Jenkins, Kienzi, & Leinbach, 2005; Feldman, 1993; Fike & Fike, 2008; Porchea et al., 2010).⁵ Thus, we controlled for the proportion of female students and the proportion of part-time students. Although less research exists at the 2-year level on how faculty impact student outcomes (Goldrick-Rab, 2010), to capture some measure of faculty availability to students, we included the number of faculty at each college. Consistent with prior studies, we also added in-state tuition and fees, Consumer Price Index (CPI)–adjusted, which relates to affordability and whether students can stay continuously enrolled (Dowd & Coury, 2006; Porchea et al., 2010). Last, we used enrollment size (total fall enrollment) to control for changes in credential completions driven by enrollment. To maintain the linear relationship with the log-transformed dependent variables, we logged enrollment and faculty counts.

In addition, at the state level, we included higher education appropriations per student at all public institutions and personal income per capita, both CPI-adjusted. These variables may affect student demand for community college participation and institutional capacity to improve outcomes and have been included in previous studies of degree outcomes or performance funding effects (Cellini, 2009; Crookston & Hooks, 2012; Tandberg et al., 2014). Appropriations data were collected from State Higher Education Executive Officers (SHEEO), and income data were collected from the Bureau of Economic Analysis. Summary statistics are displayed in Table 3.

Table 3.

Summary Statistics.

Variable	PF states			Non-PF states
Variable	n	M	SD	n	M	SD
College level
Short-term certificates	11,079	156.85	288.14	6,920	116.83	269.21
Medium-term certificates	11,079	102.92	128.62	6,920	115.27	160.34
Associate’s degrees	11,079	528.8	609.97	6,920	542.09	454.47
Fall enrollment	11,069	6,104.35	6,572.84	6,917	7,656.32	6,557.70
2-year tuition and fees	11,034	US$2,336.78	US$1,497.97	6,727	US$1,949.83	US$1,540.00
Number of faculty	10,970	109.92	96.92	6,877	123.94	88.42
Black	11,028	11.58%	13.07%	6,859	13.44%	17.47%
Hispanic	10,988	7.15%	11.90%	6,776	12.73%	15.70%
Asian	10,931	2.96%	8.29%	6,761	5.26%	7.40%
Native American	10,946	1.41%	3.60%	6,702	1.09%	1.97%
part-time	11,030	58.65%	11.78%	6,836	58.31%	16.12%
female	11,030	59.71%	6.25%	6,838	58.31%	6.22%
State level
All higher education appropriations per student	11,079	US$4,929.82	US$1,357.99	6,920	US$4,585.99	US$1,263.01
Personal income per capita	11,079	US$39,382.47	US$5,842.89	6,920	US$40,846.56	US$6,999.85

Note. PF = performance funding.

Method

As described, our dataset consisted of colleges that were affected by a state performance funding policy and colleges that were not, as well as yearly credential completions at community colleges pre- and post-policy. If performance funding did in fact increase credentials, colleges subject to the policy incentive, the treatment group, would be expected to produce significantly more credentials after the policy was put in place, compared with before. However, it is impossible to observe the counterfactual—what would have happened with credential completions at colleges subject to the policy had they not been (Rubin, 1974). To test the policy effect, we compared changes in the outcomes across treatment and control groups before and after the policy. This comparison, arguably, provides an unbiased estimate of the policy’s effect because the control group of colleges were never subject to the policy, yet were exposed to the same unobserved yearly state-level time trends.

To estimate this policy effect of state performance funding on community college outcomes, we incorporated a difference-in-differences (DD) strategy. DD is a quasi-experimental design in a regression framework that estimates the aggregate policy treatment effect by calculating two differences. The first is the difference between the pre- and postlevels of the outcome, for both treatment and comparison groups. The second is the difference between these two differences or the DD (Angrist & Pischke, 2009; Lechner, 2010).

Formally, we estimated different versions of the following model for each of our outcome variables:

D e g_{i s t} = α_{s} + τ_{t} + δ P F_{s t} + β_{1} X_{ist} + β_{2} X_{st} + ε_{i s t}

where Deg_ist is the logged degree count (i.e., one of short-term certificates, medium-term certificates, and associate’s degrees) for community college i in state s in year t. α_s and τ_t are state- and year-fixed effects, respectively. The fixed effects are included to control for time-invariant state characteristics that might affect degree counts as well as national trends in degree completions. PF_st is an indicator for whether there is a performance funding policy in state s during year t. δ is the parameter of interest and, under certain assumptions, provides an estimate for the effect of the policy on community college credentials. X_ist and X_st are vectors of college- and state-level control variables.

There may be some debate on how meaningful δ is in capturing the policy effect. δ, under certain assumptions, measures the average policy effect across all years of operation. For example, if the policy takes a few years to take effect, δ would capture an average of the null effects of the initial years of the policy as well as its true impact. δ would then be lower than the true measure of the policy’s impact. Consequently, it is useful to examine lagged effects of the policy and allow the effect of the policy to change as time passes from the initial adoption of performance funding. We make a minor modification to Equation 1 to consider the possibility of these more nuanced policy impacts in Equation 2:

D e g_{i s t} = α_{s} + τ_{t} + δ (O P_Y e a r) + β_{1} X_{ist} + β_{2} X_{st} + ε_{i s t}

Compared with Equation 1, the only difference in Equation 2 is the addition of the variable Op Year, which is a vector of dummy indicators for each year of operation. Specifically, we created 17 dummy variables to capture all possible years of operation. That is, the first Op Year indicator is equal to 1 in the first year of operation for a state and 0 during all other years. There is a separate dummy variable for all possible years of operation. For states with multiple implementations of PF, we treated each spell as a distinct policy. For example, if a state operated a policy from 1990 to 1995 as well as 2000 to 2005, we coded the first Op Year indicator to be equal to 1 in both 1990 and 2000, the first year of each spell.

Next, to account for policy heterogeneity, we modified Equations 1 and 2 by adding a new policy variable, displayed in Equations 3 and 4, that allows for differential effects across policy types. PF Type is a vector of indicators for each of the four policy types (I-IV).⁶ For instance, the dummy variable for policy Type I is equal to 1 when a Type I policy is in effect for a given state:

D e g_{i s t} = α_{s} + τ_{t} + δ P F_T y p e_{s t} + β_{1} X_{ist} + β_{2} X_{st} + ε_{i s t}

D e g_{i s t} = α_{s} + τ_{t} + δ (O P_Y e a r) x (P F_T y p e_{s t}) + β_{1} X_{ist} + β_{2} X_{st} + {εf}_{i s t}

In our analysis, the DD approach takes advantage of natural experiments that occur due to states adopting PF policies in different years. This staggering of adoptions naturally places community colleges within states into treatment (PF) and control (non-PF) groups. Rather than comparing community colleges within a state before and after the policy adoption, the DD technique compares this first difference (FD) with a similar difference over the same time period for a state that observed no change in policy (Angrist & Pischke, 2009). The problem with the single differencing approach is that any estimated policy effect could be due to a combination of the real policy and trends in the outcome over time. Furthermore, a cross-sectional study comparing a state with a policy in place and another without is also susceptible to finding a biased effect due to unobserved differences that affect states’ outcomes. The double difference in the DD strategy should, under certain assumptions, remove the effects of time trends and unobserved state characteristics to ultimately produce an unbiased estimate of the policy effect.

The DD approach, however, relies on one untestable assumption: the assumption of common or parallel trends. The DD strategy will produce an unbiased estimate of the policy effect if the control states, on average, provide a valid counterfactual for what would have happened in the treatment states had those states not adopted a PF policy. One common way of checking this assumption in the presence of multiple treatment states and start dates is to run the same model presented in Equation 1, but to include time dummy variables of the years preceding the introduction of the policy. This tests whether there are significant departures in prepolicy trends between treatment and control states in the model. While this does not precisely check the parallel trends assumption, it does provide evidence that the treatment and control states followed similar trends prior to the passage of the policy, suggesting that it is likely that these trends would have continued to be similar after the policy was in place, absent the policy (Pischke, 2005). Specifically, say k is the initial time t that the indicator PF_st turns on, we would then include D_s,_k−1, D_s,_k−2, and D_s,_k−3 (dummy indicators for the 3 years prior to the passage of PF) in the model, using a methodology that has been previously applied (Autor, 2003). Testing the joint significance of the lead policy variables in our model revealed that we cannot reject that there are no differences in pretreatment trends between treatment and control states.⁷ In other words, we find evidence supporting the common trends hypothesis given that pretreatment trends between treatment and control states are not significantly different.

Another common issue with DD estimation is serial correlation (Bertrand, Duflo, & Mullainathan, 2004). Serial correlation in the error terms has been shown to downwardly bias standard error estimates. In other words, serial correlation can lead to findings of significance even when there are none. Serial correlation is an issue when estimating DD models that use more than just two time periods, pre- and posttreatment, as we do in the present study. Following previous methodological recommendations, we clustered all standard errors at the state level to correct for this serial correlation (Bertrand et al., 2004). We arrived at this choice after trying clustering at several different levels: college, year, and state–year groups. The standard errors clustered at the state level provide the most conservative parameter variance estimates and, therefore, we chose to report these results. Furthermore, it has been recommended that in DD models, one should cluster on state rather than state–year cells given that there is more likely to be independence across states rather than state–years (Cameron & Miller, 2015).

Results

Average and Lagged Impacts of Performance Funding

Table 4 presents the parameter estimates from the model formalized in Equation 1. Each column displays the results for the models of each of the three outcome variables of interest: short-term certificates, medium-term certificates, and associate’s degrees. On average, none of the three outcomes analyzed showed any significant changes in response to performance funding—Had colleges not been subject to the policy, they would have produced the same number of credentials.

Table 4.

Difference-in-Differences Estimates for Average Policy Effect.

	Log short-term certificates	Log medium-term certificates	Log associate’s degrees
PF policy	0.010	0.036	−0.025
	(0.182)	(0.074)	(0.018)
% Black	−0.174	0.073	−0.572***
% Black	(0.504)	(0.356)	(0.076)
% Hispanic	−0.318	−1.114**	−0.230**
% Hispanic	(0.303)	(0.446)	(0.110)
% Asian	−0.548	−1.119*	−0.729***
% Asian	(0.653)	(0.576)	(0.199)
% Native American	0.422	−2.068**	−0.375
% Native American	(1.388)	(0.864)	(0.286)
% female	−3.338***	−1.160	0.811**
% female	(0.871)	(0.986)	(0.329)
% part-time	1.418*	0.997	−1.187***
% part-time	(0.819)	(0.842)	(0.207)
2-year tuition (US$1,000)	−0.099	−0.137***	−0.021
2-year tuition (US$1,000)	(0.098)	(0.049)	(0.013)
Log fall enrollment	−0.052	−0.155	0.794***
Log fall enrollment	(0.221)	(0.213)	(0.085)
Log No. faculty	0.792***	0.926***	0.231***
Log No. faculty	(0.226)	(0.189)	(0.073)
Appropriations per enroll (US$1,000)	−0.059	−0.117*	−0.009
Appropriations per enroll (US$1,000)	(0.112)	(0.064)	(0.022)
Income per capita (US$1,000)	0.003	−0.020	−0.016
Income per capita (US$1,000)	(0.042)	(0.023)	(0.012)
N	16,953	16,953	16,953
Adjusted R²	0.510	0.342	0.808

Note. Clustered (by state) standard errors are presented in parentheses. State- and year-fixed effects are included in all models. PF = performance funding.

p < .05. **p < .01. ***p < .001.

While these reported analyses suggest no significant average policy effects, it is possible that the model is missing effects by assuming that there is a homogeneous effect of the policy for each year the policy has operated. The null policy effects found above might be attributed to lagged responses to the policy that take place in the years following the initial adoption of performance funding. Table 5 presents the parameter estimates from the model formalized in Equation 2, which allows for the policy to have differential effects up to 17 years following initial adoption. We present estimated policy impacts for the first 5 years of operation. We choose not to present results beyond the fifth year given that only one third of policy states continued performance funding beyond the fifth year. Estimates for all 17 operating years are available in Table A1 in the appendix, but we caution the reader in interpreting effects beyond the fifth year because it seems unlikely that the effects would generalize to all states’ policies.⁸ We do not present parameter estimates for control variables in Table 5 and subsequent tables due to estimates being mostly unchanged from those reported in Table 4.

Table 5.

Difference-in-Differences Estimates for Lagged Policy Effects.

	Log short-term certificates	Log medium-termcertificates	Log associate’s degrees
1st operating year	0.010	−0.070	−0.018
1st operating year	(0.138)	(0.062)	(0.024)
2nd operating year	−0.056	−0.072	−0.055**
2nd operating year	(0.170)	(0.062)	(0.025)
3rd operating year	0.056	0.030	−0.031
3rd operating year	(0.179)	(0.062)	(0.020)
4th operating year	0.133	0.018	−0.034*
4th operating year	(0.223)	(0.058)	(0.020)
5th operating year	0.060	−0.038	−0.028
5th operating year	(0.216)	(0.092)	(0.022)
N	16,953	16,953	16,953
Adjusted R²	0.509	0.343	0.808

Note. Clustered (by state) standard errors are presented in parentheses. State- and year-fixed effects and state/college controls are included in all models.

p < .05. **p < .01. ***p < .001.

Consistent with the first analysis of average effects across all years, our second model reported in Table 5 shows null effects of performance funding on both short-term certificates and medium-term certificates during the first 5 years of the policy. However, we find a significant decrease in associate’s degrees during the second and fourth years of operation.

To visually display our main findings, we took the average college in the average state and posed the question: Based on our estimated models, how would credential completions change if a performance funding policy were introduced in 2014? We simulated counterfactuals and display the patterns in Figure 2, providing a visualization of our regression estimates. The left-hand column shows the findings under the results of Equation 1 and the right-hand column shows the results under Equation 2. Plotted in each subfigure is the expected cumulative first difference (FD) in degree counts (not logged counts) between two counterfactual scenarios: (a) if a performance funding policy were adopted in 2014 and in place through 2018 and (b) if a PF policy were never adopted during this same time span. We stimulated counterfactuals up to 5 years. As mentioned previously, results beyond 5 years may be less generalizable.

Figure 2.

First difference of policy effects.

When assuming there is only an average policy effect, we would expect no changes in the outcomes, which is illustrated in the left column of Figure 2. As seen, the confidence intervals overlap zero. Noted in the description of the corresponding findings from Table 4, on average, short- and medium-term certificates as well as associate’s degrees do not change in response to performance funding. Therefore, we conclude that performance funding produces no average effects on 2-year credential completions.

Allowing the policy effects to differ for each year of operation leads to similar findings, displayed in the right column of Figure 2. It appears that while short-term certificates stay stable and then see a small increase, these changes are not significant. Medium-term certificates appear to experience a slight initial decline and then remain steady, but again, these results are insignificant. Regarding associate’s degrees, while there is a visual decline after performance funding is introduced, these declines are also not statistically significant during the first 5 years and could statistically be null changes. In summary, our model estimates and the visual displays of these estimates illustrate that when we do not consider policy features, performance funding has no impact on its intended goal of graduating more students at community colleges.

Impacts of Performance Funding by Policy Typology

We next investigate our third research question of whether there exists heterogeneity in performance funding policy effects due to differences in policy design captured by the policy typologies (e.g., strength of policy measured by proportion of funding tied, base vs. bonus funding, sector coverage, metrics differentiation, number of fiscal years in place). Table 6 presents the parameter estimates from the model described in Equation 3.

Table 6.

Difference-in-Differences Estimates for Average Policy Type Effects.

	Log short-term certificates	Log medium-term certificates	Log associate’s degrees
PF Type I	−0.006	0.038	−0.019
PF Type I	(0.211)	(0.087)	(0.017)
PF Type II	−0.059	0.027	0.028
PF Type II	(0.232)	(0.092)	(0.047)
PF Type III	0.536*	0.242	−0.084
PF Type III	(0.292)	(0.266)	(0.068)
PF Type IV	0.315**	−0.014	−0.195***
PF Type IV	(0.138)	(0.081)	(0.039)
N	16,953	16,953	16,953
Adjusted R²	0.510	0.342	0.808

Note. Clustered (by state) standard errors are presented in parentheses. State- and year-fixed effects and state/college controls are included in all models. PF = performance funding.

p < .05. **p < .01. ***p < .001.

We find that short-term certificates rise significantly, on average, under performance funding policies of Types III and IV (71% and 37% increases, respectively, because [exp(β = .536) – 1 = 0.709] and [exp(β = .315) – 1 = 0.370]). No effects of performance funding are present for medium-term certificates. However, associate’s degrees show a significant decrease of 17.7% under policies of Type IV [exp(β = -0.195) – 1 = -0.177].

Table 7 shows selected parameter estimates from the model shown in Equation 4. There are four panels in the table and each displays the parameter estimates for the first five operating years of each distinctive policy type. Parameter estimates for all 17 operating years for each policy type are available in Tables A2 and A3 in the appendix.

Table 7.

Difference-in-Differences Estimates for Lagged Policy Effects by Policy Type.

	Log short-term certificates	Log medium-term certificates	Log associate’s degrees
PF Type I
1st operating year	0.002	−0.088	−0.025
1st operating year	(0.172)	(0.075)	(0.022)
2nd operating year	−0.069	−0.101	−0.037*
2nd operating year	(0.193)	(0.067)	(0.022)
3rd operating year	0.054	0.029	−0.008
3rd operating year	(0.206)	(0.075)	(0.015)
4th operating year	0.100	0.014	−0.027
4th operating year	(0.257)	(0.068)	(0.022)
5th operating year	0.041	−0.024	−0.018
5th operating year	(0.249)	(0.105)	(0.020)
PF Type II
1st operating year	−0.084	−0.080	0.045
1st operating year	(0.232)	(0.101)	(0.053)
2nd operating year	−0.411	−0.005	−0.108
2nd operating year	(0.524)	(0.134)	(0.104)
3rd operating year	−0.379	0.117	−0.103
3rd operating year	(0.574)	(0.071)	(0.075)
4th operating year	0.406**	0.128*	0.052**
4th operating year	(0.155)	(0.069)	(0.022)
5th operating year	0.111	0.149*	0.068
5th operating year	(0.143)	(0.077)	(0.052)
PF Type III
1st operating year	0.246	0.258	0.070
1st operating year	(0.544)	(0.352)	(0.063)
2nd operating year	0.961***	0.496***	−0.152***
2nd operating year	(0.161)	(0.095)	(0.030)
3rd operating year	0.599***	0.245	−0.062
3rd operating year	(0.162)	(0.156)	(0.037)
4th operating year	0.541**	−0.270*	−0.303***
4th operating year	(0.268)	(0.145)	(0.074)
5th operating year	0.538**	−0.460***	−0.432***
5th operating year	(0.267)	(0.147)	(0.085)
PF Type IV
1st operating year	0.483***	0.117	−0.177***
1st operating year	(0.141)	(0.092)	(0.046)
2nd operating year	0.247	0.232**	−0.288***
2nd operating year	(0.154)	(0.093)	(0.044)
3rd operating year	0.320**	−0.131	−0.227***
3rd operating year	(0.128)	(0.087)	(0.044)
4th operating year	0.318**	−0.016	−0.133***
4th operating year	(0.139)	(0.084)	(0.039)
5th operating year	0.194	−0.329***	−0.169***
5th operating year	(0.174)	(0.097)	(0.032)
N	16,953	16,953	16,953
Adjusted R²	0.509	0.343	0.808

Note. Clustered (by state) standard errors are presented in parentheses. State- and year-fixed effects and state/college controls are included in all models. PF = performance funding.

p < .05. **p < .01. ***p < .001.

PF Type I policies do not appear to impact short- or medium-term certificates but lead to a decrease in associate’s degrees in the second year of operation. All outcomes appear to rise in the fourth year of operation under PF Type II. Medium-term certificates show a further increase in the fifth year under PF Type II.

Under PF Types III and IV, we find that the number of short-term certificates rises in numerous years, with an effect ranging from 72% (β = .541) to 161% (β = .961) each year during the first 5 years (except the adoption year) of a Type III policy, and 37% to 62% each year (except the second year) during the first 4 years of a Type IV policy. The mean of short-term certificates in our sample across performance funding and non-performance funding states is 141, so even a 37% increase is equivalent to a significant growth of approximately 52 additional certificates awarded each year. Short-term certificates increase in the second year following policy implementation for policy types with stronger features. A higher proportion of base funding tied to outcomes (between 5% and 100% with most states falling within 5% and 25%) most likely prompts urgency and commitment from colleges to capture more funding by increasing student completions of the most quickly attainable credential. As hypothesized, colleges do graduate more students from programs requiring less than 1 year of study in response to the more salient performance funding policies.

For Type III policies, medium-term certificates see a rise of 64% in the second year of operation and a decline of 24% in the fourth operating year and a decline of 37% in the fifth operating year. The average college awards 108 medium-term certificates each year. The increase in the second year would be equivalent to 69 more certificates, and the declines in the fourth operating year and fifth operating year would be equivalent to between 26 and 40 fewer certificates. As for Type IV policies, which differ from Type III by being funded at 25% or more of base funds for at least 2 fiscal years, medium-term certificates increase by 26% in the second operating year, yet in the fifth year, decline by 28%. These patterns do not appear to represent systematic college responses to higher strength performance funding types. However, decreases in medium-term certificates in the latter years coincide with increases in short-term certificates, suggesting that colleges either do not prioritize completion among students in medium-term certificate programs or might be recruiting additional students to short-term programs who would otherwise enroll in medium-term programs.

With respect to associate’s degrees, Type III and Type IV policies produce declines during the first 5 years of policy operation. For Type III policies, which are characterized by funding levels ranging from 5% to 24.9% of base funding, associate’s degrees decline by 14% in the second year and drop further by 26% and 35% in the fourth operating year and fifth operating year, respectively. For Type IV policies, colleges produce between 16% and 25% fewer associate’s degrees each year starting with the first year and consistently follow this declining pattern across all 5 years analyzed. With a mean of 534 students graduating with associate’s degrees each year, a college subject to a Type IV policy would graduate between 86 and 133 fewer students, a considerable decline.

As a whole, our results suggest that performance funding policies of Type I, with low proportions of funding (less than 5%) and bonus funds only, are generally not associated with changes in completions at 2-year colleges. Policies in the Type II category, characterized again by low proportions of funding (less than 5%) albeit on base funds, appear to have positive effects on short-term certificates, medium-term certificates, and associate’s degrees in the fourth year of operation. In addition, our results suggest that for performance funding policies that tie more base funding to outcomes, are more sophisticated in differentiating by mission and accounting for underrepresented students, and/or are in place for longer periods of time (PF Types III and IV), colleges do in fact react in ways that aim to capture the most immediate funding rewards. Specifically, colleges graduate more students with short-term certificates and simultaneously graduate fewer students with associate’s degrees, and, in some years, fewer medium-term certificates as well. Because more resources and more time are needed to graduate more students with associate’s degrees, colleges are likely shifting focus to the lower order and more efficient target of short-term certificates.

Robustness Checks

We next conducted several robustness checks by testing whether alternative comparison groups would yield different results. The first control/comparison group we analyzed against our treatment group consisted of states with higher education governance structures organized as coordinating or planning boards, excluding states with governing boards. States with coordinating or planning boards may be more similar on unobservable characteristics compared with consolidated governing boards, and thus, we separated the sample to analyze differences in outcomes within each of the two samples.⁹ Results for the average policy effect of performance funding using the sample of only coordinating and planning board states produced substantively identical conclusions—There was no effect on any of the three outcomes (reported in Table A4 of the appendix).

In a second robustness check, we limited our sample to only states that had ever adopted a performance funding policy, given that states that adopted a policy at any time point are more similar than states that never adopted the policy. Again, results were consistent with our main analyses (see Table A5 in the appendix). Furthermore, given that some researchers may note that earlier performance funding policies were weaker, we checked to see whether removing these earlier policies would change our results. We set a cut date of 1996, when the first PF 2.0 policy (base funding) began operating, and focus only on those policies that were started in that year or later. Our results were unchanged by focusing only on this sample—We still find insignificant average effects (see Table A6 in the appendix). Last, we chose to include Tennessee, which we previously excluded because it began performance funding in 1979, before the start of our dataset. We find that including Tennessee in our analytical sample does not change any of the conclusions made. Results from the analysis including Tennessee are displayed in Table A7 of the appendix.

Limitations

One limitation of our study is that our performance funding policy typology does not separate the individual features characterizing each policy type. Based on the way HCM Strategists crafted the policy typology, our analysis cannot decipher whether it is the base funding feature in Type IV policies that is generating increases in short-term certificates, the greater than 25% of funding tied, or the sustainment of the policy for 2 or more consecutive years. As the typology bundles all the features together and some features overlap across types, we are not able to isolate single features to develop more specific policy recommendations. Nevertheless, we find that policy typology does indeed matter, and future research is recommended to explore the impact of specific design features in performance funding policies on student outcomes.

Another limitation of our study is that we do not disaggregate by discipline. We cannot determine how the aggregate increase in short-term certificates is distributed across fields of varying employability. Future research would be well served to examine policy effects by discipline—to analyze how much performance funding increases marketable graduates through certificates. From a policy-making perspective, changing the incentive structure to account for marketability and future earnings by field could be advantageous. Our study is also limited in that we do not disaggregate by gender. As covered in the literature review, discipline and gender both contribute to differential wage outcomes. To better tease out more nuanced performance funding effects, if they do exist, future research ought to examine 2-year credential completions among different student subgroups.

Discussion

In this study, we address our research questions on whether performance funding policies impact completions at community colleges, how three levels of credentials are differentially affected, and whether the strength of policies affects completions. Specifically, we find that, on average, short-term certificates, medium-term certificates, and associate’s degrees do not change significantly after the introduction of performance funding. When allowing the direction and size of effects to differ across the first to 17th operating years, our analyses show that policies produce a positive effect on medium-term certificates in select later years but a negative effect in other years, and a negative effect on associate’s degrees in a few select, isolated years. When we account for policy strength and sophistication, however, interesting policy impacts emerge. Policies that are more likely to produce changes are those that have distributed funding for more than 2 fiscal years, tie a higher proportion of base state funding to performance outcomes, differentiate performance metrics by college mission, cover 2- and 4-year sectors, and/or incorporate metrics that account for different populations served (e.g., students of color and low-income students). Within the first 5 years of a policy having been adopted, the more consequential policies of Types III and IV do in fact incentivize colleges to produce more short-term certificates, fewer medium-term certificates over time (after initial increases), and fewer associate’s degrees.

Our findings suggest that, consistent with our hypothesis, colleges subject to more salient performance funding policies are responding in ways that cultivate greater financial rewards and may attempt to minimize investment in less profitable outcome goals. Colleges were hypothesized to have more capacity and more motivation to prioritize short-term certificate production. Indeed, our evidence suggests this approach. In addition, we observe a declining number of associate’s degrees, and although the results may not be causal, they signify that higher order completion goals are likely being supplanted by lower order goals. Yet, these responses to the policy are only evident when colleges are at risk of losing or gaining a considerable amount of funding. For colleges that would see funding changes that encompassed less than 5% of their state appropriations, there appeared to be fewer changes in credentials at any of the three levels.

Previously referenced studies show that short-term certificates in most disciplines generate zero or negative labor market value, possibly due to their concentration in less employable fields (Dadgar & Trimble, 2015; Grubb, 2002b; Jepsen et al., 2014; Liu et al., 2014). Our study finds a significant impact of PF on such certificates; the policy has measurable policy side effects that steer more students into less marketable programs. We propose that our national findings are consistent with findings in Washington state, which also found an increase in short-term certificates (Hillman et al., 2015). More research is needed to uncover actions happening at the campus level that leads to greater production of short-term certificates. College advisors may be tempted to route more students from longer to shorter programs, recruit additional students directly into short-term programs, or more quickly graduate this immediate flow of students.

We further explore our findings on medium-term certificates. On average, colleges award the same number of medium-term certificates regardless of whether they are subject to performance funding. However, when we analyze each policy type and operating year separately, we find that stronger policies in Type III and Type IV categories are associated with more medium-term certificates in Year 2 but fewer medium-term certificates in Years 4 and 5. These later declines are accompanied by a growth in short-term certificates, which suggest that colleges are routing students from medium- to short-term certificate programs. This is again a concern given the more beneficial labor market values for longer term versus shorter term certificates (Dadgar & Trimble, 2015). To better target the different policy outcomes within certificate levels, policymakers might want to separately measure and fund short- versus medium-term certificates. For example, the funding formula might award X dollars for each short-term certificate and Y dollars for each medium-term certificate (perhaps doubling the amount), which would illuminate the presence of any disparate effects.

Moreover, our results demonstrate that policy designs are influential in catalyzing college actions. Performance funding policies that tie a higher proportion of funding to outcomes and are in place for prolonged periods are especially likely to incentivize colleges to engage in internal practices that lead to securing more funding. Policymakers should consider the salience of policies if they expect colleges to change behaviors that lead to the intended changes in student outcomes. This requires allocating from the base budget, tying a significant proportion of funding to outcomes, and most likely requires metrics that accommodate the different missions of each college and different student demographics served.

With regard to associate’s degrees, our results suggest that no significant changes are present, on average, although accounting for policy type reveals significant declines among policy Types III and IV. Declining associate’s degrees are worrisome because not only does PF fail to accomplish its goal of increasing degree completions, it reduces completions. This is particularly concerning because associate’s degrees provide the highest likelihood of employment, more hours worked, and substantial earnings increases, especially for women (Dadgar & Trimble, 2015; Jepsen et al., 2014).

Our findings support previous research that found no average impact of performance funding on associate’s degrees nationally (Tandberg et al., 2014). It is possible that analyzing all states in aggregate masks the differential effects of the policy, even though our study seeks to accommodate policy design differences.¹⁰ Performance funding is also situated in different state contexts, with some states offering more political or institutional support for the policy, which should be further explored.

Conclusion

In conclusion, our study advances the literature on performance funding by examining a critical sector of higher education. The 2-year sector is educating more and more students, and provides an entry point into higher education. Performance funding ultimately aims to improve student completion of credentials, and many states offer equivalent funding allocations to short-term certificates, medium-term certificates, and associate’s degrees. An increase in short-term certificates in response to performance funding suggests detrimental consequences of more students graduating with credentials that offer limited labor market benefits. Indeed, we find that performance funding results in short-term certificate production. In addition, the policy has negative effects on associate’s degrees, which have been found to provide greater economic benefits for graduates. Despite this evidence that performance funding may not work as intended, states continue to operate the policy and more states are adopting new policies, raising additional questions about the ultimate value and long-term consequences of the policy. Growing and sustained interest in performance funding as a public accountability tool to improve institutional effectiveness and efficiency will continue to offer scholars and policymakers an opportunity to analyze potential impacts and consider policy redesigns to better improve student graduation outcomes.

Footnotes

Appendix

Table A7.

Difference-in-Differences Estimates for Average Policy Effect (Tennessee Included).

	Log short-term certificates	Log medium-term certificates	Log associate’s degrees
PF policy	0.000	0.033	−0.025
PF policy	(0.182)	(0.075)	(0.018)
% Black	−0.160	0.113	−0.575***
% Black	(0.486)	(0.343)	(0.074)
% Hispanic	−0.341	−1.121**	−0.230**
% Hispanic	(0.304)	(0.447)	(0.109)
% Asian	−0.601	−1.135*	−0.727***
% Asian	(0.671)	(0.582)	(0.199)
% Native American	0.402	−2.053**	−0.383
% Native American	(1.383)	(0.855)	(0.286)
% female	−3.243***	−1.132	0.822**
% female	(0.861)	(0.969)	(0.324)
% part-time	1.375*	1.057	−1.184***
% part-time	(0.810)	(0.837)	(0.204)
2-year tuition (US$1,000)	−0.100	−0.136***	−0.020
2-year tuition (US$1,000)	(0.098)	(0.048)	(0.013)
Log fall enrollment	−0.050	−0.159	0.792***
Log fall enrollment	(0.221)	(0.213)	(0.085)
Log No. faculty	0.801***	0.933***	0.233***
Log No. faculty	(0.225)	(0.189)	(0.072)
Appropriations per enroll (US$1,000)	−0.063	−0.117*	−0.008
Appropriations per enroll (US$1,000)	(0.110)	(0.064)	(0.021)
Income per capita (US$1,000)	0.005	−0.021	−0.016
Income per capita (US$1,000)	(0.042)	(0.023)	(0.012)
n	17,238	17,238	17,238
Adjusted R²	0.507	0.344	0.808

Note. Clustered (by state) standard errors are presented in parentheses. State- and year-fixed effects are included in all models. PF = performance funding.

p < .05. **p < .01. ***p < .001.

Acknowledgements

We wish to thank Chris Adolph for his feedback on earlier drafts of this manuscript, our discussant Will Doyle for his comments at the 2015 conference of the Association for the Study of Higher Education, and Margaret Sebastian for her editorial assistance.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Amy Y. Li

Notes

Author Biographies

Amy Y. Li is an assistant professor of Higher Education in the Department of Leadership, Policy, and Development at the University of Northern Colorado. Her research focuses on higher education finance and public policy, including performance funding, student loan debt, state policy adoption, and college promise programs.

Alec I. Kennedy is a PhD candidate at the University of Washington’s Evans School of Public Policy and Governance. His research interests include the economics of education and public policy analysis. His work involves the evaluation of education policies at the federal, state, and local level.

References

Angrist

J. D.

Pischke

J.-S.

(2009). Mostly harmless econometrics: An empiricist’s companion. Princeton, NJ: Princeton University Press.

Autor

D. H.

(2003). Outsourcing at will: The contribution of unjust dismissal doctrine to the growth of employment outsourcing. Journal of Labor Economics, 21, 1-42.

Bahr

P. R.

Dynarski

Jacob

Kreisman

Sosa

Weiderspan

(2015). Labor market returns to community college awards: Evidence from Michigan. New York, NY: Center for Analysis of Postsecondary Education and Employment. Retrieved from http://capseecenter.org/wp-content/uploads/2015/03/labor-market-returns-michigan.pdf

Bailey

Calcagno

J. C.

Jenkins

Kienzi

Leinbach

(2005). Community college student success: What institutional characteristics make a difference? (CCRC Working Paper No. 3). New York, NY: Community College Research Center, Teachers College, Columbia University. Retrieved from http://ccrc.tc.columbia.edu/publications/community-college-success-institutional-characteristics.html

Bailey

Jenkins

Leinbach

(2005). What we know about community college low-income and minority student outcomes (Descriptive statistics from national surveys). New York, NY: Community College Research Center, Teachers College, Columbia University. Retrieved from http://ccrc.tc.columbia.edu/publications/low-income-minority-student-outcomes.html

Belfield

C. R.

Bailey

(2011). The benefits of attending community college: A review of the evidence. Community College Review, 39, 46-68. doi:10.1177/0091552110395575

Bertrand

Duflo

Mullainathan

(2004). How much should we trust differences-in-differences estimates? The Quarterly Journal of Economics, 119, 249-275. doi:10.1162/003355304772839588

Burke

J. C.

(2002). Funding public colleges and universities for performance: Popularity, problems, and prospects. Albany, NY: Rockefeller Institute Press.

Burke

J. C.

Minassians

H. P.

(2003). Performance reporting: “Real” accountability or accountability “lite”—Seventh annual survey 2003. Albany, NY: The Nelson A. Rockefeller Institute of Government.

10.

Cameron

A. C.

Miller

D. L.

(2015). A practitioner’s guide to cluster-robust inference. Journal of Human Resources, 50, 317-372.

11.

Carnevale

A. P.

Rose

S. J.

Hanson

A. R.

(2012). Certificates: Gateway to gainful employment and college degrees (Executive summary). Washington, DC: Georgetown University Center on Education and the Workforce. Retrieved from cew.georgetown.edu/certificates

12.

Cellini

S. R.

(2009). Crowded colleges and college crowd-out: The impact of public subsidies on the two-year college market. American Economic Journal: Economic Policy, 1, 1-30. doi:10.1257/pol.1.2.1

13.

Colorado Commission on Higher Education. (2015). 2016-17 Colorado higher education funding allocation model. Denver: Colorado Department of Higher Education. Retrieved from http://highered.colorado.gov

14.

Crookston

Hooks

(2012). Community colleges, budget cuts, and jobs: The impact of community colleges on employment growth in rural U.S. counties, 1976-2004. Sociology of Education, 85, 350-372. doi:10.1177/0038040712441376

15.

Crowder

M. V.

Janosik

S. M.

(2001). Performance funding in Virginia higher education. Virginia Issues and Answers, 7(2). Retrieved from http://www.via.vt.edu/spring01/article4.pdf

16.

Dadgar

Trimble

M. J.

(2015). Labor market returns to sub-baccalaureate credentials: How much does a community college degree or certificate pay? Educational Evaluation and Policy Analysis, 37, 399-418. doi:10.3102/0162373714553814

17.

Davies

(2014). State “shared responsibility” policies for improved outcomes: Lessons learned. Washington, DC: HCM Strategists. Retrieved from http://hcmstrategists.com/wp-content/uploads/2014/04/HCM-State-Shared-Responsibility-RADD-2.0.pdf

18.

Dietrich

C. C.

Lichtenberger

E. J.

(2015). Using propensity score matching to test the community college penalty assumption. The Review of Higher Education, 38, 193-219. doi:10.1353/rhe.2015.0013

19.

Dougherty

K. J.

Jones

Lahr

Natow

Pheatt

Reddy

(2013). Envisioning performance funding impacts: The espoused theories of action for state higher education performance funding in three states (CCRC Working Paper No. 63). New York, NY: Community College Research Center, Teachers College, Columbia University. Retrieved from http://ccrc.tc.columbia.edu/publications/envisioning-performance-funding-impacts.html

20.

Dougherty

K. J.

Natow

. (2015). The politics of performance funding for higher education: Origins, discontinuations, and transformations. Baltimore, MD: Johns Hopkins University Press.

21.

Dougherty

K. J.

Natow

Vega

(2012). Popular but unstable: Explaining why state performance funding systems in the United States often do not persist. Teachers College Record, 114, 1-41.

22.

Dougherty

K. J.

Reddy

(2013). Performance funding for higher education: What are the mechanisms? What are the impacts? ASHE Higher Education Report, 39(2), 1-134.

23.

Dowd

Coury

(2006). The effect of loans on the persistence and attainment of community college students. Research in Higher Education, 47, 33-62.

24.

Feldman

M. J.

(1993). Factors associated with one-year retention in a community college. Research in Higher Education, 34, 503-512.

25.

Fike

D. S.

Fike

(2008). Predictors of first-year student retention in the community college. Community College Review, 36, 68-88. doi:10.1177/0091552108320222

26.

Frey

B. S.

Homberg

Osterloh

(2013). Organizational control systems and pay-for-performance in the public service. Organization Studies, 34, 949-972. doi:10.1177/0170840613483655

27.

Friedel

J. N.

Thornton

Z. M.

D’Amico

M. M.

Katsinas

S. G.

(2013). Performance-based funding: The national landscape. Tuscaloosa: The University of Alabama Education Policy Center.

28.

Goldrick-Rab

(2010). Challenges and opportunities for improving community college student success. Review of Educational Research, 80, 437-469. doi:10.3102/0034654310370163

29.

Grubb

W. N.

(2002a). Learning and earning in the middle, part I: National studies of pre-baccalaureate education. Economics of Education Review, 21, 299-321. doi:10.1016/s0272-7757(01)00042-5

30.

Grubb

W. N.

(2002b). Learning and earning in the middle, part II: State and local studies of pre-baccalaureate education. Economics of Education Review, 21, 401-414. doi:10.1016/S0272-7757(01)00043-7

31.

Harbour

C. P.

(2002). The legislative evolution of performance funding in the North Carolina Community College System. Community College Review, 29, 28-49. doi:10.1177/009155210202900402

32.

HCM Strategists. (2011). Completion based funding in Ohio. Washington, DC: Author. Retrieved from http://strategylabs.luminafoundation.org/wp-content/uploads/2013/10/completion_based_funding_ohio_final-copy.pdf

33.

HCM Strategists. (2012). Performance funding in Indiana: An analysis of lessons from the research and other state models. Washington, DC: Author. Retrieved from http://postsecondary.gatesfoundation.org/report/performance-funding-in-indiana-an-analysis-of-lessons-from-the-resaerch-and-other-state-models/

34.

Hillman

N. W.

Tandberg

D. A.

Fryar

A. H.

(2015). Evaluating the impacts of “new” performance funding in higher education. Educational Evaluation and Policy Analysis, 37, 501-519. doi:10.3102/0162373714560224

35.

Husig

Mann

H.-G.

(2010). The role of promoters in effecting innovation in higher education institutions. Innovation: Organization & Management, 12, 180-191. doi:10.5172/impp.12.2.180

36.

Jenkins

Wachen

Moore

Shulock

(2012). Washington state student achievement initiative policy study: Final report. New York, NY: CCRC-IHELP. Retrieved from http://ccrc.tc.columbia.edu/media/k2/attachments/student-achievement-initiative-final-report.pdf

37.

Jepsen

Kenneth

Coomes

(2014). The labor-market returns to community college degrees, diplomas, and certificates. Journal of Labor Economics, 32, 95-121. doi:10.1086/671809

38.

Jones

D. P.

(2013). Outcomes-based funding: The wave of implementation. Washington, DC: National Center for Higher Education Management Systems. Retrieved from completecollege.org/pdfs/Outcomes-Based-Funding-Report-Final.pdf

39.

Kansas Board of Regents. (2013). Act on performance agreements: Summary and recommendations. Topeka. Retrieved from https://www.kansasregents.org/resources/PDF/2807-AY2014toAY2016PerfAgreementsFinaldocument.pdf

40.

Kivistö

(2008). An assessment of agency theory as a framework for the government–university relationship. Journal of Higher Education Policy and Management, 30, 339-350. doi:10.1080/13600800802383018

41.

Lechner

(2010). The estimation of causal effects by difference-in-difference methods. Foundations and Trends in Econometrics, 4, 165-224. doi:10.1561/0800000014

42.

A. Y.

(2014). Performance funding in the states: An increasingly ubiquitous public policy for higher education. Higher Education in Review, 11, 1-29. Retrieved from https://sites.psu.edu/higheredinreview/2014/02/14/performance-funding-in-the-states-an-increasingly-ubiquitous-public-policy-for-higher-education/

43.

A. Y.

(2017a). Covet thy neighbor or “reverse policy diffusion”? State adoption of performance funding 2.0. Research in Higher Education, 58, 746-771. doi:10.1007/s11162-016-9444-9

44.

A. Y.

(2017b). The point of the point: Washington’s student achievement initiative through the looking glass of a community college. Community College Journal of Research and Practice, 41, 183-202. doi:10.1080/10668926.2016.1179601

45.

Liu

V. Y. T.

Belfield

C. R.

Trimble

M. J.

(2014). The medium-term labor market returns to community college awards: Evidence from North Carolina. Economics of Education Review, 44, 42-55. doi:10.1016/j.econedurev.2014.10.009

46.

McGuinness

A. C.

(2002). The authority of state boards of postsecondary education. Denver, CO: Education Commission of the States.

47.

National Conference of State Legislatures. (2015). Performance-based funding for higher education. Retrieved from http://www.ncsl.org/research/education/performance-funding.aspx

48.

Ness

E. C.

Deupree

M. M.

Gándara

(2015). Campus responses to outcomes-based funding in Tennessee: Robust, aligned, and contested. Nashville: Tennessee Higher Education Commission. Retrieved from http://www.tn.gov/thec/topic/funding-formula-resources

49.

Newcomer

Caudle

(2011). Public performance management systems: Embedding practices for improved success. Public Performance & Management Review, 35, 108-132. doi:10.2753/PMR1530-9576350106

50.

Ohio Department of Higher Education. (2012). Recommendations of the Ohio Higher Education Funding Commission. Columbus: Author. Retrieved from https://www.ohiohighered.org/press/new-performance-based-model-higher-education-ohio

51.

Pfeffer

Salancik

G. R.

(1978). The external control of organizations: A resource dependence perspective. New York, NY: Harper & Row.

52.

Pischke

J.-S.

(2005). Empirical methods in applied economics (University Lecture). London School of Economics. Retrieved from http://econ.lse.ac.uk/staff/spischke/ec524/evaluation3.pdf

53.

Porchea

S. F.

Allen

Robbins

Phelps

R. P.

(2010). Predictors of long-term enrollment and degree outcomes for community colleges. The Journal of Higher Education, 81, 680-708.

54.

Preis

(2012). Funding higher education in Missouri: Model proposal. Missouri Joint Committee on Education. Retrieved from www.senate.mo.gov/jced

55.

Rabovsky

T. M.

(2014). Using data to manage for performance at public universities. Public Administration Review, 74, 260-272.

56.

Rubin

D. B.

(1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688-701. doi:10.1037/h0037350

57.

Snyder

(2014). Driving better outcomes: Typology and principles to inform outcomes-based funding models. Washington, DC: HCM Strategists. Retrieved from http://hcmstrategists.com/drivingoutcomes/wp-content/themes/hcm/pdf/DrivingOutcomes.pdf

58.

Snyder

Fox

(2016). Driving better outcomes: Fiscal year 2016 state status & typology update. Washington, DC: HCM Strategists. Retrieved from http://hcmstrategists.com/drivingoutcomes/

59.

SRI International. (2012). States’ methods of funding higher education. Menlo Park, CA. Retrieved from http://www.sri.com/sites/default/files/brochures/revised-sri_report_states_methods_of_funding_higher_education.pdf

60.

Tandberg

D. A.

Hillman

N. W.

Barakat

(2014). State higher education performance funding for community colleges: Diverse effects and policy implications. Teachers College Record, 116, 1-31.

61.

Texas Association of Community Colleges. (2017). Student success points. Retrieved from http://www.tacc.org/pages/data-and-info/student-success-points

62.

Texas Higher Education Coordinating Board. (2012). Formula funding recommendations 2012. Retrieved from http://www.thecb.state.tx.us/reports/PDF/2591.PDF