Evaluating the Impacts of “New” Performance Funding in Higher Education

Abstract

In 2007, Washington adopted the Student Achievement Initiative, a statewide performance accountability system designed to improve retention rates and degree productivity among community colleges. Using difference-in-differences analysis, we found that the policy change has had little immediate effect on retention rates or associate’s degree productivity. However, community colleges produced more short-term certificates after the policy reform. These results are robust across many alternative comparison groups. Considering that certificates yield less value in the labor market than associate’s degrees but are easier for colleges to produce, we discuss the unintended consequences of rewarding colleges based on the number of credentials they produce.

Keywords

higher education state policy performance funding

One of the most popular state policy trends in higher education has been the re-emergence of performance funding. Many states adopted this strategy during the New Accountability movement of the 1990s, but a lack of financial and political support in the early 2000s resulted in many states abandoning these initiatives (Dougherty & Reddy, 2013). Performance funding policies regained popularity in the mid-2000s, due in large part to efforts of many influential organizations, especially the Bill and Melinda Gates Foundation, Lumina Foundation, National Governors Association, Complete College America, and National Conference of State Legislatures. These organizations led a national effort to draw attention to issues related to college completion and often advocated for policies that incentivized completion by linking state appropriations to institutional outputs (e.g., number of degrees awarded, student retention, transfer rates) instead of funding colleges according to the number of students enrolled (McKeown-Moak, 2013).

Concerned that the United States is falling behind other countries in terms of degree completion and that colleges are not producing enough graduates to keep pace with changes in the labor market, state policymakers view performance funding as a way to align colleges with broader state policy goals. By funding colleges according to their outputs, rather than inputs, state policymakers believe that these incentives will motivate colleges to increase degree productivity. While there is no question that advocates of performance funding policies have been very successful in promoting its adoption and implementation throughout the states, several questions remain about whether this is an effective strategy for improving degree completions.

A growing body of evidence suggests that these policies have done little to improve educational outcomes and that the underlying theory of action behind performance funding is not well articulated or understood (Dougherty et al., 2013). The evaluative research on performance funding systems is often conducted on the macro- or micro levels. The macroapproach uses states as the unit of analysis and examines whether performance funding states outperform non-performance funding states. This approach is an important starting point for understanding performance funding as it documents which states adopted the policy and for what years (Tandberg & Hillman, 2014; Tandberg, Hillman, & Barakat, 2014). While the macroapproach provides a national view of performance funding impacts and trends, that level of aggregation can overlook the rich policy context that exists within each individual performance funding state. The microapproach addresses this limitation by examining how performance funding impacts colleges within a specific performance funding state. This approach provides greater policy context around a particular state’s performance funding model, such as what has been done with case studies of Pennsylvania and Tennessee (Hillman, Tandberg, & Gross, 2014; Sanford & Hunter, 2011).

Using the microapproach, the current study evaluates the effectiveness of Washington’s performance funding model. As described in the next section, the state adopted performance funding for its community colleges in 2007, making Washington one of the nation’s most stable and longest operating performance funding programs currently in place. There are lessons to be learned from states that have sustained performance funding for multiple years, especially as performance funding’s impacts may not manifest until several years after the policy is in place. Likewise, Washington’s policy reform has been widely recognized as one of the most carefully designed performance accountability systems in the United States, serving as a model for the most recent wave of new performance funding reforms (Dougherty & Reddy, 2013). In addition, given the diffusion of performance funding policies across the states and the political capital that goes into these reforms, it behooves state policymakers to look closely at the evidence to assess whether performance funding states have actually generated greater educational outputs because of this policy reform. Evidence of performance funding’s efficacy is scarce, short of the few studies cited above, so our case study of Washington allows us to ask to what extent has performance funding impacted degree productivity?

Policy Context

Performance funding programs grew out of an effort in the 1990s to reorient the state oversight and accountability environment for public colleges and universities. This New Accountability movement would make “performance” a priority, where colleges would be expected to set performance goals, measure results, and be accountable for reaching certain outcomes. In exchange, they would receive a certain amount of funding that either is a bonus to their base level of appropriations or is built into base funding. While the “funding” portion of “performance funding” policies often receives the widest attention, these policy reforms are more accurately characterized as performance accountability systems.

Performance funding reforms involve more than just linking funds to performance: State officials and campus leaders must collaborate to arrive at performance goals, they must create measures and collect the necessary data, and colleges must report their progress prior to receiving the funds. Collecting and reporting performance data alone (regardless of the amount of money at stake) could impact institutional decision making and, ultimately, performance. In evaluating performance accountability efforts, it can be difficult to disentangle which specific feature of the larger performance accountability package is responsible for any observed effect. A college may improve its performance, but it is unclear whether this was due to the financial incentives, the compliance reporting, the establishment of short- and long-term performance goals, or any combination of these factors. Because of this, it is best to view performance funding within its broader accountability context; indeed, “performance funding” is part of an accountability treatment package that includes data utilization, strategic planning, and financial incentives to increase institutional performance. Throughout this study, we refer to the commonly used shorthand, “performance funding,” though it is important to see this policy as part of any state’s broader performance accountability system.

Early proponents of performance funding believed that it would shift institutional culture away from the rudimentary reporting of inputs and accounting for expenditures to one that is more results-oriented (Burke, 2005; McGuinness, 1995; McLendon, 2003b; Volkwein, 2007; Volkwein & Tandberg, 2008). By linking a portion of state funding for higher education “directly and tightly to the performance of public campuses on individual indicators,” colleges would have the incentive to meet the state’s performance goals (Burke & Minassians, 2003, p. 3). Tennessee adopted the nation’s first program in the late 1970s, but in 1985, only two states had followed their lead (McLendon, Hearn, & Deaton, 2006). Although 23 states had performance funding systems in place at the turn of the century, these programs began to lose popularity with a significant number of states giving up their programs due to economic and political reasons (Dougherty, Natow, & Vega, 2012).

Recently, a number of large-scale initiatives backed by some of the largest foundations and higher education policy non-profits and institutes in the country have advocated for the development of “new” state higher education performance funding programs. In response, a significant number of states have adopted (or re-adopted) their programs, including a number of innovations designed to correct some of the weaknesses of previous iterations. This renaissance has been dubbed Performance Funding 2.0 (Dougherty & Reddy, 2011; Rabovsky, 2012).

In Washington, state performance funding has a relatively new history. In 2007, Washington’s State Board for Community and Technical Colleges (SBCTC) initiated its most comprehensive performance accountability reform, the Student Achievement Initiative (SAI). The SBCTC and others have argued that two innovations separate the SAI from previous failed efforts at state performance funding for higher education. The first is that the SAI includes intermediate measures and therefore rewards colleges for students’ progression toward completion and not just completion alone. Perhaps one of the reasons why earlier performance funding efforts failed is because they prioritized long-term goals and did not reward colleges for intermediate measures such as student retention or transfer rates. The second is that the SAI gives colleges data on students’ pathways and progress toward completion. This approach allows college leaders to identify where students are struggling and hopefully identify what changes might need to be made to improve their progress (Jenkins & Shulock, 2013). These innovations have caused a number of researchers, commentators, and advocates to hold the SAI as a model for Performance Funding 2.0 (e.g., Altstadt, 2012; Dougherty & Reddy, 2011; Harnisch, 2011; Miao, 2012; Offenstein & Shulock, 2010).

To increase institutional accountability and improve institutional performance, SAI funds colleges according to performance on a number of metrics, outlined in Table 1 (Jenkins, Wachen, Moore, & Shulock, 2012). These metrics include a number of “milestones,” which are further subdivided into “achievement points.” Milestones track students’ progress through their educational career at their college, from pre-college programs (e.g., developmental education and adult basic education) through the completion of credentials and training programs. Achievement points are then the specific measures related to each milestone. For example, when students earn 15 credit hours or return to school the following year, the college will achieve their “retention” milestone and receive its performance funds. As many short certificate programs can be completed in 15 credit hours, colleges can earn achievement points by expanding these programs. Similarly, when students earn a 1-year certificate, associate’s degree, or apprenticeship program, the college will achieve a “completion” milestone and receive its performance funds. To encourage short-term and long-term success on these performance goals, the state rewards colleges for increasing their total achievement points from 1 year to the next. Likewise, advocates argue that the inclusion of both short-term and long-term performance goals, and the associated data collection efforts, provides institutions with “data to guide colleges in identifying barriers to student progression and taking steps to remove them in order to increase student completion rates” (Jenkins & Shulock, 2013, p. 5), which may improve student outcomes.

Table 1

Student Achievement Initiative Performance Goals

Milestone	Achievement points
1. Building toward college-level skills	Increase in skill level based on standardized test
1. Building toward college-level skills	Passing a pre-college writing or math course
2. First-year retention and progress	Earning 15 quarter credits of college-level coursework
2. First-year retention and progress	Earning 30 quarter credits of college-level coursework
3. Completing college-level math	Passing a math course required for either technical or academic associate degree
4. Completions	Completing a degree, occupational certificate, or apprenticeship training

The actual amount of money colleges receive is based on their performance relative to prior performance, the amount of money available to fund the program, and the total point gains achieved by the system as a whole. The performance funding received by the colleges is a bonus on top of the general appropriation; however, the performance funds the colleges receive in any given year are later added to their base budgets moving forward. Because of state budgetary constraints, performance funds have constituted less than 1% of the system’s total operating budget (Jenkins et al., 2012). However, while the SAI performance funds made up only 1% of the system’s total operating budget, the amount made up of the individual institutions’ operating budgets naturally varied. In addition, it is important to note that the initiative was phased in, with 2007 serving as a learning year when institutions began implementation and data tracking. Because campus began implementing SAI in 2007, we (like previous evaluators) have chosen 2007 as the start year for the policy intervention (e.g., Belfield, 2012; Jenkins, Ellwein, & Boswell, 2009; Jenkins & Shulock, 2013; Jenkins et al., 2012).¹

Literature Review

While previous studies on performance funding have focused on the origins of the policies and best practices (Banta, Rudolph, Van Dyke, & Fisher, 1996; Bogue & Johnson, 2010; Burke & Minassians, 2001; Dougherty, Natow, Hare, Jones, & Vega, 2011; McLendon et al., 2006), a growing number of studies have examined the relationship between performance funding and institutional practices and student outcomes. In their review of existing research and state documents and reports, Dougherty and Reddy (2013) found evidence that performance funding is associated with changes in campus planning efforts and administrative strategies that may improve academic and student support services. Similarly, Rabovsky (2012) found a small positive impact of performance funding on university spending, with a modest increase in spending on instruction. These examples suggest that institutions may adjust their policies and practices in response to the new policies, regardless of the amount of performance funds available from the state.

Left unanswered is whether institutional responses to performance funding also yield positive changes in degree completions and the like. A number of scholars have completed individual state case studies to assess the impact of specific performance accountability programs. For example, using Tennessee as a case study, Sanford and Hunter (2011) estimated how changes in the state’s existing performance funding program impacted retention and graduation rates, concluding that the policy changes had no systematic effect. Hillman et al. (2014) similarly conducted an in-depth analysis of Pennsylvania’s long running performance funding program. The authors utilized a difference-in-differences research design with multiple comparison groups to analyze the impact of Pennsylvania’s performance funding program on degree completions per full-time equivalent (FTE) student. The authors found modest impacts when compared against colleges in neighboring states; however, these impacts disappeared when matched against similar colleges from other states. They concluded that the policy had not systematically increased degree completions within the state.

Other investigators have conducted macro (i.e., state-level) studies. Shin and Milton (2004) included multiple states in their analysis and concluded that performance funding had no systematic impact on graduation rates. However, the Shin and Milton study did not offer an identification strategy for threats to internal validity of the model, so the causal relationship should be interpreted with caution. Most recently, Tandberg and Hillman (2014) and Tandberg et al. (2014) conducted two national state-level studies utilizing difference-in-differences research designs to assess the impact of the implementation of performance funding on degree completions in the 4-year and 2-year college sectors. On average, they found that performance funding had little to no impact on associate or baccalaureate degree completions on average; however, after accounting for the number of years a program has been in place, they found a small yet statistically significant positive impact on baccalaureate completions and a negative impact on associate’s degree completions (Tandberg & Hillman, 2014).

Ours is not the first study to assess Washington’s performance funding program. Jenkins et al. (2012) and Belfield (2012) conducted a series of studies utilizing a mix of qualitative and quantitative assessments to determine the impact of the SAI on institutions and student outcomes. Jenkins et al.’s (2012) study argues that the perception among college administrators and others is that while the SAI is seen as play a role in motivating colleges to improve student success, it has not been as effective as initially expected, in part because of the relatively small amount of funding attached to the program. Similarly, Belfield’s study, which is descriptive and not causal, found that colleges increased their achievement point total by 31% between 2007 and 2011, yet student momentum (or student progression) did not change much after the implementation of SAI (Belfield, 2012). The qualitative insights are helpful for understanding the nuance of implementing SAI, while the only other quantitative study on the policy suggests that the policy has had weak to modest effects. These studies provide important contextual information regarding the implementation of the SAI, yet our study is the first to evaluate potential causal relationships between the SAI and two of the state’s ultimate policy goals (improving retention rates and increasing completion levels).

While the evidence of performance funding suggests that the policy has had little to no effect on degree completion, it is possible that it has positively impacted intermediate outcomes such as student retention rates. This has yet to be studied in the literature. Likewise, while previous analyses of Washington’s performance funding program have provided important insights into the design and implementation of SAI, left unanswered is what would have happened had Washington never implemented the SAI. Washington’s performance funding program represents one of the longest running programs, making it a “proof of concept” for the model advocated by proponents of Performance Funding 2.0. Furthermore, and as indicated earlier, the SAI has been heralded as a model for other states’ performance funding and accountability systems (e.g., Jenkins & Shulock, 2013). Therefore, with this study, we seek to evaluate both intermediate outcomes and completions and to utilize a research design which allows us to more closely approximate causal relationships of a critical state postsecondary policy innovation.

Conceptual Framework

Performance funding policies often mirror a basic principal–agent relationship, in which state policymakers, as principals, seek better ways to specify the contract with their agents, the public colleges in the state. In most conversations that lead to the adoption of these policies, the logic is very clear, compelling, and consistent across states. Policymakers view higher education as a relatively expensive investment, and one that affects many of their constituents. In addition, many recent national efforts to encourage more states to adopt and strengthen performance funding policies have heightened the attention to student outcomes and college expenditures.

The logic of performance funding policies is quite compelling. Historically, revenues from state funding have been tied to enrollment, much like tuition and fee revenues. Under these systems, colleges are “rewarded” only for enrolling students, and as such, college leaders have no (direct financial) incentives for pursuing the goals that policymakers consider to be most important, usually undergraduate degree production linked to educating the state’s future workforce. The absence of these incentives is believed to allow leaders to use taxpayer dollars on functions that promote the interests of the institution (e.g., rankings, salaries, etc.) over the interests of the state, a basic moral hazard argument. When policymakers seek to adopt or strengthen performance funding policies, many college leaders are resistant, arguing that incentive-based policies fail to consider the many pressures on public higher education and the consistent issues faced by less selective institutions, who point to incoming students with skill deficiencies.

These interactions highlight the inevitable conflict that stems from information asymmetry and goal complexity, two important concepts that are regularly acknowledged but rarely addressed in the recent work on performance funding effectiveness. In the political sphere, some of the concerns raised by college leaders may be viewed as “excuses” or efforts to engage in a “smoke and mirrors” discussion. However, theoretical work on performance-based accountability policies offers insights that may allow us to move beyond the questions over if performance-based accountability policies are effective to when and why performance-based policies are effective.

Many of the more recent performance-based policies in higher education (those often described as 2.0) have sought to move away from a one-size-fits-all approach to include a range of performance indicators that value many different types of student success. These changes were motivated, in part, by concerns from institutional leaders who argued that performance-based accountability policies with very limited views on what constitutes “performance” ignore the realities faced by the complex, diverse goals inherent in most public organizations (Chun & Rainey, 2005; Radin, 2006), particularly postsecondary institutions. By offering incentives for a wider range of outcomes, the new policies seek to increase the likelihood of policy effectiveness, as they introduce multiple paths for colleges to demonstrate growth. With very little literature that explores the influence of performance accountability policies on an expanded range of indicators, we have a substantial gap in our knowledge of the effectiveness of these new strategies, which serves as the motivation for this study.

Data and Research Design

We constructed a panel data set for the years 2002 through 2012 using data from the U.S. Department of Education’s Integrated Postsecondary Education Data System (IPEDS). Our goal is to compare community college performance trends after the introduction of SAI. But as it is impossible to observe what would have happened in Washington had the policy never been adopted, we rely on trends from colleges in other states as our counterfactual. These comparison groups are discussed in more detail in the “Robustness Checks” section, where we explain how we constructed these counterfactuals. Our analysis includes community and technical colleges in Washington (n = 31) in addition to community and technical colleges from 12 western states (n = 176) that are members of the Western Interstate Commission for Higher Education (WICHE).² From this regional group, we also compare Washington colleges against colleges from states that share a border but never adopted performance funding for their community or technical colleges (n = 18). The sample also includes community college and technical colleges from 21 non-performance funding state, regardless of region (n = 64).³

Variables

IPEDS provides institution-level records on college pricing and tuition, admissions, completions, enrollment, graduation rates, student financial aid, and finances. We use three outcome variables in our analysis: retention rates, certificates awarded (short term and total), and associate’s degrees awarded. Retention rates at community and technical colleges are measured by the percentage of first-time certificate-seeking or degree-seeking students from the previous fall who returned to the same institution (or completed their program of study) by the following fall. We use retention rates for both part-time and full-time students separately. The number of certificates awarded includes both short (i.e., less than 1 year) and long (i.e., more than 1 year) certificate programs, and the number of associate’s degrees represents either vocational or academic programs. In both cases, the total number of certificates or degrees awarded each year represents the aggregate number of unique graduates from any major or program of study, regardless of whether they enrolled part-time or full-time.

As discussed previously, these outcomes are what the state uses in its performance funding model, so we expect to see an increase in these outcomes after the policy was implemented in 2007. If these increases are not observed, then it suggests that the state’s performance funding efforts are not effective in terms of influencing these outcomes. As discussed in the research design section, we use a difference-in-differences technique to assess these impacts. Because literature finds that retention and degree completion are conditioned by a college’s size, racial and socioeconomic profile, percent of students enrolled part-time, tuition level, and financial capacity, we control for these variables in our analysis (Bahr, 2013; Dowd, 2003; Mullin, 2012; Seidman, 2005; Tebbs & Turner, 2005; Titus, 2006). We also suspect colleges that are more dependent on state appropriations (as a share of their total operating revenues), and those that focus more heavily on certificate programs (certificates as a share of total degrees awarded) may be affected differently than ones that are less dependent. Finally, because community colleges are responsive to local economic conditions, we control for the unemployment rate and size of the labor force for each county in which a community college is located (Betts & McFarland, 1995; Hillman & Orians, 2013).

Research Design

To assess the impact Washington’s performance funding policy had on degree productivity and retention rates, we implement an ordinary least squares regression using the difference-in-differences estimation strategy. Under this quasi-experimental design, the adoption of performance funding is viewed as a “natural experiment” as it introduces into our model a plausible source of exogenous variation. The following equation describes our model:

y_{i t} = α + β_{1} (t r e a t) + β_{2} (p o s t) + β_{3} (t r e a t \times p o s t) + γ_{i} + η_{t} + c o n t r o l s_{i t} + ε_{i t} .

Here, y is the outcome, α is the intercept, the variable treat is a dummy for Washington community colleges affected by the policy, and the variable post accounts for the years after the policy was introduced. The interaction between treat and post, represented by the parameter β₃, measures the impact performance funding had on the treated institutions. The next variable in the equation (γ_i) represents the institution-level fixed effects and η_t represents year fixed effects. By adding these two additional controls, we account for unobserved factors at the institution level that are relatively time-invariant (e.g., quality, organizational culture, mission, etc.), while we also account for events that affect all community colleges in a particular year (e.g., federal policy changes, national economic shocks such as the Great Recession, etc.). These fixed effects help us account for omitted variable bias, thus improving our model’s internal validity, so that we can arrive more closely at drawing causal inference between the policy and the observed outcomes.

In addition to these fixed effects, we add controls that vary for each institution (a) in each year (t). These controls account for the size of each college (FTE enrollment), racial diversity (percent of undergraduates who are White), low-income student enrollment (average federal grants per FTE), enrollment intensity of students (percent part-time), tuition levels and state grant aid per FTE, and the share of each college’s operating revenues accounted for by state appropriations. Finally, ϵ_it represents the error term for each institution in each year. Importantly, the model’s error could be serially correlated over time, resulting in inefficient parameter estimates. To account for this, we first conducted a Wooldridge test and found the presence of autocorrelation in our model (Drukker, 2003). To adjust the standard errors for autocorrelation and panel heteroskedasticity, we implemented our regressions using the Prais–Winsten estimation technique.

Robustness Checks

It is impossible to observe a true counterfactual for Washington’s performance funding: We do not know what would have happened in the state had it not adopted the policy. Because of this, quasi-experimental techniques often rely on comparison groups that were never subject to a similar treatment, to serve as the counterfactual. Under this framework, the choice of comparison groups becomes an important feature of the research design and if not chosen carefully can yield biased results. To address this concern, we use the “local and focal” matching strategy advocated by Shadish, Cook, and Campbell (2002). This technique uses geographical neighbors to make comparisons between the treatment and comparison group (i.e., local matching). It also uses statistically derived comparison groups that follow similar trends as the treatment group during the pre-treatment period (i.e., focal matching). In both cases, we only selected comparison institutions from states that did not operate performance funding during the analysis period (i.e., they were not subject to similar treatments).

For our local matches, we compare Washington against three geographic comparison groups, moving from the least to most local: all states, Western states, and neighboring states. The focal matches do not consider geography; rather, it identifies community colleges from anywhere in the country. We used propensity score matching to generate this group, where our goal was to identify colleges that had similar retention and completion outcomes during the pre-treatment years. While propensity score matching is often used as an estimation strategy, we use it here as a matching strategy. We use the propensity scores to identify our “focal” matching group, and we do not use the statistical technique to estimate treatment effects as is often done with this strategy.

Having multiple comparison groups helps us assess the robustness of our findings, while also ruling out alternative explanations that could be driving our results. If we find significant results in only one comparison group, for example, then this yields less conclusive evidence than results that are significant across all comparison groups. Multiple comparisons offer us multiple hypotheses and multiple chances to rule out plausible alternative explanations, making the multiple comparison group technique a particularly useful strategy in quasi-experimental designs (Meyer, 1995; Shadish & Cook, 1999).

We used one additional strategy to rule out alternative explanations for our results. Following the techniques employed in other difference-in-differences models (Garces, 2013; Lavy, 2010), we manipulated the treatment years to assess whether results from the false years duplicated the results from the actual implementation year (e.g., using 2006 instead of 2007). If we found the same effects in these “false” years as we found in the correct year, then it would be difficult to attribute the effects to the policy itself and not simply a trend that was occurring regardless of the year. We found no evidence of this occurring in our alternative models, which helps us rule out this plausible alternative explanation driving our results. In the interest of space, we do not include results from these tests in this article; however, they are available upon request.⁴

Limitations

Despite our efforts at improving our model’s internal validity, this study is not without limitations. First, it is possible that other events that took place near the same time of adopting performance funding could artificially drive our results. To rule out the possibility of confounding events, we searched for news stories for other policy changes that occurred around 2007 and found one key event that could be driving our results. The Boeing Corporation opened a large job training program at Edmonds Community College (ECC), which offers 12-week long certificate courses. Between 2007 and 2012, the number of certificates produced by ECC increased fivefold, which could be artificially driving our results. It is possible that other colleges opened similar job training programs, but we found no other programs of a similar scale as the Boeing example. Second, the context around Washington’s performance accountability system may not be directly generalizable to the experiences of other states’ performance funding efforts. While our goal is not to generalize results to “all” performance funding states, we believe that in-depth case studies of states operating and sustaining performance funding for the longest period of time provide a “proof of concept” for determining whether this policy is achieving its stated goals. Third, it is impossible in our study to isolate which design component (e.g., funding incentives, data collection efforts, data utilization, public accountability, etc.), so we describe SAI as an accountability “treatment package.” Further research could attempt to disentangle these effects. Finally, we examine only two of the state’s four performance measures (retention and degree completion). It is possible that the policy effects are different for the other outcomes not explored in this article (increasing college-level skills and completing college-level math). These alternative performance goals could be explored in follow-up studies, but we do not have data to examine them in this analysis.

Key Findings

Table 2 provides descriptive statistics of the variables used in each regression model. This table compares Washington colleges against their local and focal comparison groups. In general, Washington colleges have higher (lower) retention rates than neighboring (WICHE) states, yet they produce more certificates per year than any other comparison group. When standardizing certificate production by the number of students enrolled (per 100 FTE), we see that Washington colleges outperform colleges from WICHE and neighbor states. A similar pattern is found with associate’s degree production, where Washington colleges produce more degrees (in aggregate and per 100 FTE) than colleges in WICHE and neighboring states. This is one reason why the matched comparison group is so helpful for this analysis: It provides a comparison group that is more similar to the other groups in terms of retention rates as well as certificate and associate degree production.

Table 2

Mean (SD) of Variables Used in Analysis

	WA	WICHE	Neighbor	PS match
Retention rate—Part-time^a	43.4 (7.5)	37.1 (10.4)	36.6 (11.9)	40.5 (13.3)
Retention rate—Full-time^a	59.6 (6.8)	60.6 (10.9)	52.0 (12.2)	58.4 (9.1)
Short certificates awarded	271.3 (301.1)	208.9 (400.5)	42.1 (76.8)	163.9 (291.6)
Short certificates per 100 FTE	7.8 (7.7)	3.3 (5.5)	2.3 (6.2)	5.6 (8.8)
Long certificates awarded	135.1 (114.4)	149.8 (239.3)	71.5 (92.2)	120.0 (146.8)
Long certificates per 100 FTE	4.1 (4.4)	2.5 (3.7)	3.1 (5.4)	3.8 (4.5)
Associate degrees awarded	679.9 (349.6)	644.5 (519.9)	469.0 (479.1)	575.2 (433.6)
Associate degrees per 100 FTE	17.2 (5.1)	12.7 (10.0)	13.2 (5.3)	16.5 (5.8)
FTE enrollment	3,900.8 (1,641.3)	5,948.3 (4,536.5)	3,587.0 (3,761.7)	3,744.6 (2,960.4)
% enrolled part-time	50.9 (10.7)	66.1 (12.5)	59.0 (13.3)	55.5 (15.2)
% of degrees that are certificates	36.8 (21.0)	30.8 (17.5)	27.8 (17.3)	31.1 (22.2)
% White	64.6 (11.1)	47.8 (24.7)	68.1 (17.0)	64.1 (24.1)
% of revenues from state appropriations	82.0 (14.9)	75.9 (28.0)	51.4 (25.1)	66.7 (25.4)
Tuition and fees	3,126.1 (480.4)	1,609.6 (1,379.8)	3,259.8 (637.4)	3,340.5 (1,583.0)
Federal grant aid per FTE	1,180.6 (939.4)	1,005.2 (1,344.5)	1,742.5 (2,277.3)	1,151.1 (1,430.9)
State grant aid per FTE	526.9 (378.8)	218.6 (572.4)	279.9 (1,029.5)	158.3 (539.7)
County unemployment rate	7.2 (2.2)	7.5 (3.6)	7.9 (2.6)	6.4 (2.8)
Size of county labor force (millions)	0.4 (0.4)	1.1 (1.5)	0.1 (0.1)	0.3 (0.7)

Observations	341	1,934	198	704
Institutions	31	176	18	64

Note. WA = Washington; WICHE = Western Interstate Commission for Higher Education; PS = propensity score; FTE = full-time equivalent.

Retention rates are only available from 2004 forward.

Trends in Outcomes

Figure 1 disaggregates performance trends over time and across comparison groups, where we can see how comparison group averages compare with Washington over time. Here, we see that part-time retention rates (top-left panel) and full-time retention rates (top-right panel) for Washington community colleges were around 45% and 60%, respectively, in the years leading up to SAI. There was a small immediate initial dip after the policy, but retention rates remained at these general rates in the years before and after the policy. The middle two panels of Figure 1 display the (logged) average number of certificates produced each year, separated by “short” and “long” certificates, where we see a spike in short certificates for Washington in the years following SAI. On average, colleges from comparison states also increased their certificate production over time, with Washington having the largest growth in short certificates and less growth (or even flat) trends in long certificates. The bottom panel of Figure 1 displays associate’s degree production over time, where the (logged) average number of degrees (bottom-left panel) stays relatively flat in all states and fluctuating between 6.1 and 6.6 over time for Washington. When shifting to the ratio of degrees per 100 FTE (bottom-right panel), we see the average ratio rose, declined, and then rose again during this period. Each figure provides helpful context for understanding the average patterns occurring in Washington and in comparison group states before and after SAI; next, the regression results will help us identify any systematic patterns in these outcomes after the introduction of SAI.

Figure 1.

Performance trends for various comparison groups, before and after SAI.

Regression Results

Tables 3 through 6 display results from the regression analyses, where each table provides results for each separate outcome variable. Each of these models control for the institutional characteristics (e.g., enrollment size, percent White, tuition, share of revenue from state, certificate’s share, and federal and state aid per FTE) as well as state and year fixed effects.

Table 3

Regression Estimates for SAI’s Impact on Part-Time and Full-Time Retention Rates

	Part-time			Full-time
	WICHE	Neighbors	Matched	WICHE	Neighbors	Matched
a. Main model
Treat × Post	−0.49 (1.93)	−4.07 (2.43)	−2.22 (1.44)	−0.73 (1.65)	−1.66 (2.37)	−0.78 (1.16)
b. Year interactions
Treat × Post × 2007	−2.78*** (0.82)	−5.87* (2.52)	−3.61*** (1.05)	−1.08 (0.86)	0.07 (0.94)	−2.15** (0.69)
Treat × Post × 2008	−1.18 (1.47)	−8.68** (3.08)	−2.85 (2.08)	−2.98** (1.03)	−8.35*** (1.85)	−2.27* (1.06)
Treat × Post × 2009	6.62*** (1.55)	−4.60 (3.04)	−2.07 (1.90)	6.80*** (1.26)	−8.85*** (2.07)	1.42 (1.24)
Treat × Post × 2010	1.06 (1.61)	−1.48 (2.99)	−0.71 (1.85)	−2.12 (1.25)	−1.99 (1.72)	0.39 (1.35)
Treat × Post × 2011	−0.48 (1.59)	−1.68 (3.32)	−0.96 (1.97)	−1.89 (1.24)	0.85 (2.17)	0.83 (1.52)
Treat × Post × 2012	−1.01 (1.56)	0.36 (3.45)	1.41 (2.10)	−1.96 (1.21)	4.65** (1.79)	1.76 (1.58)

Observations	1,854	441	855	1,854	441	855
Institutions	206	49	95	206	49	95
Control variables	Yes	Yes	Yes	Yes	Yes	Yes
Year fixed effects	Yes	Yes	Yes	Yes	Yes	Yes
Institution fixed effects	Yes	Yes	Yes	Yes	Yes	Yes
Main model, R²	.47	.35	.47	.61	.50	.50
Year interactions model, R²	.48	.36	.47	.62	.52	.50

Note. Panel-corrected standard errors in parentheses. SAI = Student Achievement Initiative; WICHE = Western Interstate Commission for Higher Education.

p < .05. **p < .01. ***p < .001.

Table 4

Regression Estimates for SAI’s Impact on Short Certificates Awarded

	Short-term certificates (logged)			Short-term certificates (per 100 FTE)
	WICHE	Neighbors	Matched	WICHE	Neighbors	Matched
a. Main model
Treat × Post	0.33 (0.17)	−0.29 (0.27)	0.12 (0.17)	2.40** (0.91)	0.17 (0.84)	1.52* (0.65)
b. Year interactions
Treat × Post × 2007	0.28 (0.18)	0.15 (0.27)	0.10 (0.13)	0.86* (0.36)	−0.26 (0.65)	0.56 (0.33)
Treat × Post × 2008	−0.03 (0.22)	−0.55 (0.33)	−0.43** (0.15)	0.72 (0.47)	−1.80 (1.16)	0.39 (0.42)
Treat × Post × 2009	0.27 (0.24)	−0.31 (0.33)	0.08 (0.16)	1.58** (0.53)	−0.94 (1.09)	0.87* (0.42)
Treat × Post × 2010	0.48* (0.23)	−0.31 (0.30)	0.26 (0.16)	3.12*** (0.51)	−0.03 (0.82)	1.73*** (0.43)
Treat × Post × 2011	0.45* (0.23)	−0.80* (0.31)	0.36* (0.16)	5.06*** (0.49)	1.13 (0.85)	3.58*** (0.45)
Treat × Post × 2012	0.47* (0.23)	−0.91** (0.29)	0.28 (0.17)	5.82*** (0.48)	2.42** (0.93)	4.15*** (0.46)
Observations	2,266	539	1,045	2,266	539	1,045
Institutions	206	49	95	206	49	95
Control variables	Yes	Yes	Yes	Yes	Yes	Yes
Year fixed effects	Yes	Yes	Yes	Yes	Yes	Yes
Institution fixed effects	Yes	Yes	Yes	Yes	Yes	Yes
Main model, R²	.80	.76	.84	.71	.66	.84
Year interactions model, R²	.80	.77	.84	.72	.69	.85

Note. Panel-corrected standard errors in parentheses; SAI = Student Achievement Initiative; FTE = full-time equivalent; WICHE = Western Interstate Commission for Higher Education.

p < .05. **p < .01. ***p < .001.

Table 5

Regression Estimates for SAI’s Impact on Long Certificates Awarded

	Certificates (logged)			Certificates (per 100 FTE)
	WICHE	Neighbors	Matched	WICHE	Neighbors	Matched
a. Main model
Treat × Post	−0.41** (0.15)	−0.79** (0.27)	−0.60*** (0.17)	−1.14** (0.37)	−1.06 (0.55)	−0.96 (0.41)
b. Year interactions
Treat × Post × 2007	−0.27 (0.16)	−0.41 (0.30)	−0.51** (0.19)	−0.75 (0.40)	−0.19 (0.63)	−0.69 (0.49)
Treat × Post × 2008	−0.47* (0.20)	−0.99* (0.39)	−0.76** (0.24)	−1.44** (0.48)	−1.71 (0.90)	−1.44 (0.59)
Treat × Post × 2009	−0.67** (0.22)	−1.29*** (0.36)	−0.74** (0.26)	−2.01*** (0.53)	−2.32** (0.86)	−1.33* (0.63)
Treat × Post × 2010	−0.53* (0.22)	−0.95** (0.35)	−0.55* (0.25)	−1.58** (0.51)	−1.20 (0.74)	−1.02 (0.62)
Treat × Post × 2011	−0.45* (0.23)	−0.93** (0.35)	−0.55* (0.25)	−1.08* (0.51)	−1.38 (0.72)	−0.94 (0.60)
Treat × Post × 2012	−0.52* (0.23)	−1.02** (0.35)	−0.74** (0.25)	−1.28* (0.52)	−1.34* (0.68)	−1.18* (0.59)
Observations	2,266	539	1,045	2,266	539	1,045
Institutions	206	49	95	206	49	95
Control variables	Yes	Yes	Yes	Yes	Yes	Yes
Year fixed effects	Yes	Yes	Yes	Yes	Yes	Yes
Institution fixed effects	Yes	Yes	Yes	Yes	Yes	Yes
Main model, R²	.67	.63	.76	.65	.79	.78
Year interactions model, R²	.67	.64	.75	.65	.79	.78

Note. Panel-corrected standard errors in parentheses. SAI = Student Achievement Initiative; FTE = full-time equivalent; WICHE = Western Interstate Commission for Higher Education.

p < .05. **p < .01. ***p < .001.

Table 6

Regression Estimates for SAI’s Impact on Associate’s Degrees Awarded

	Associate’s degrees (logged)			Associate’s degrees (per 100 FTE)
	WICHE	Neighbors	Matched	WICHE	Neighbors	Matched
a. Main model
Treat × Post	0.06 (0.06)	0.08 (0.06)	0.08 (0.07)	−0.13 (0.69)	0.15 (0.51)	−0.01 (0.73)
b. Year interactions
Treat × Post × 2007	0.01 (0.07)	0.07 (0.05)	0.01 (0.07)	−0.98* (0.43)	0.07 (0.64)	−0.95 (0.51)
Treat × Post × 2008	0.04 (0.08)	0.08 (0.07)	0.01 (0.09)	−0.89 (0.54)	0.57 (0.86)	−1.09 (0.64)
Treat × Post × 2009	0.04 (0.08)	0.19** (0.07)	0.08 (0.09)	−0.98 (0.59)	0.31 (0.86)	−0.37 (0.65)
Treat × Post × 2010	0.09 (0.09)	0.16* (0.07)	0.12 (0.09)	−0.56 (0.59)	−0.04 (0.71)	0.02 (0.61)
Treat × Post × 2011	0.15 (0.08)	0.07 (0.06)	0.19* (0.09)	1.18* (0.57)	0.25 (0.80)	1.63** (0.62)
Treat × Post × 2012	0.15 (0.08)	0.00 (0.05)	0.19* (0.09)	2.29*** (0.57)	0.06 (0.74)	2.47*** (0.63)
Observations	2,266	539	1,045	2,266	539	1,045
Institutions	206	49	95	206	49	95
Control variables	Yes	Yes	Yes	Yes	Yes	Yes
Year fixed effects	Yes	Yes	Yes	Yes	Yes	Yes
Institution fixed effects	Yes	Yes	Yes	Yes	Yes	Yes
Main model, R²	.95	.93	.93	.84	.80	.75
Year interactions model, R²	.95	.93	.93	.85	.80	.75

Note. Panel-corrected standard errors in parentheses. SAI = Student Achievement Initiative; FTE = full-time equivalent; WICHE = Western Interstate Commission for Higher Education.

p < .05. **p < .01. ***p < .001.

Starting with the impact SAI had on retention rates (Table 3), we see that there are no significant differences in the main model (Row “a”) where we compare the average treatment effect across the three comparison groups and the results are the same for both part-time and full-time retention rates. When compared against WICHE states, neighbor states, and even the matched states, the analysis reveals no statistically significant patterns for the average treatment effect. When interacting the average treatment for each year (Row “b”), we begin to see mixed patterns across the various comparison groups for both part-time and full-time retention rates. For most years and comparisons groups, the SAI had no significant effect on retention rates; however, there are a number of instances early in the implementation where retention rates decreased and a few instances where retention rates at Washington colleges improved after SAI. These mixed patterns indicate a degree of volatility that occurs on this outcome, where SAI appears to have had no average effects on retention rates, and mostly negative (though sometimes positive) effects on retention rates when disaggregated by year.

Tables 4 and 5 examine short and long certificate productivity, respectively. Taking Table 4 first, we examine SAI’s effect on programs that can be completed most immediately, often in less than 1 academic year. We observe effects in several cases, beginning with the 1.52 to 2.40 average increase in short-term certificates per 100 FTE that occurred after SAI (Row “a”). Interestingly, these effects did not occur immediately, as Table 4 shows by the few significant year interactions in Row “b.” Systematic effects did not begin to emerge until after about 2 years of implementation, and they were often positive, suggesting that SAI encouraged colleges to produce more short-term certificates. Moving to Table 5, we examine SAI’s impact on long certificate production to see if it follows similar patterns as the short-term subset. Unlike the previous model, we find negative average treatment effects for the main model (Row “a”) and negative effects for almost all models that are interacted by year (Row “b”). It appears as though SAI may have induced counteracting effects, where it induces the growth of short certificates and a reduction of long certificates.

Table 6 provides results for associate’s degree production, where we follow the same analytic strategy as the certificates model by offering two alternative outcome measures: total degrees awarded (logged) and degrees per 100 FTE. In the “associate’s degrees (logged)” models, we find no average treatment effect (Row “a”) and, like the certificates models, we observe effects after several years of implementation. In the “degrees per 100 FTE” model, there is again no average treatment effect but we see a few instances (2011 and 2012) where SAI impacted degree production in positive and significant ways. During the first several years, this outcome was negative and often insignificant, but that trend began to reverse toward the later years of our analysis.

Across the four tables, we find that SAI had mixed impacts on campus retention rates and these effects lessened over time. There were no average treatment effects on retention and, to the extent effects occurred, they emerged after interacting with the academic year. Interestingly, this is the general pattern we found across all “higher order” models, where the average treatment effects for retention and associate’s degree programs were often insignificant and only the year interactions yield significant results. Only the certificates models generated significant average treatment effects, and they generated the greatest number of instances where we observe significant year interactions. Here, we conclude that SAI increased the number of short-term certificates produced in the state and reduced long-term certificate production. In terms of inducing systematic improvements in productivity, we only find evidence of positive effects for short certificates and either weak or negative impacts on other (higher order) outcomes.

Discussion

Performance funding reforms are increasingly popular policy strategies in state higher education policy, as state lawmakers and advocacy groups grow concerned about college affordability, productivity, and degree completion (McLendon & Cohen-Vogel, 2008; McLendon et al., 2006). Many of these reform efforts draw from the New Accountability movement, where accountability strategies such as financial incentives and performance reporting are expected to induce colleges to change behaviors. To get colleges to improve retention rates, or certificate and associate’s degree production, states such as Washington are changing the way they finance colleges and they are requiring colleges to set and report performance measures to be eligible for continued state support. But are these efforts improving educational outcomes and resulting in greater institutional performance?

Evidence from this study helps answer this question while contributing to the growing body of research on the effects of performance accountability systems on educational outcomes. Proponents in Washington believe that by building from lower order (e.g., retention) to higher order (e.g., certificate and degree production) goals, colleges will have a more straightforward pathway for improving performance. This is in line with what performance funding advocates believe are the necessary conditions for an effective accountability system: Performance goals have clear and linear pathways that include rewards for both short-term and long-term milestones (McKeown-Moak, 2013; Shulock, 2011). Washington followed this logic by designing a set of simple and easily measured performance goals that are designed to build upon one another. If colleges improve their lower order goals (e.g., retention rates), then they should also improve their higher order goals (e.g., associate’s degree completion).

In our analysis, we found that the performance of Washington community and technical colleges was, on average, often not distinguishable from the performance of colleges in other states that were never subject to similar accountability policies. Despite the state’s effort to create an incentive structure to improve retention and associate’s degree production, we found no evidence that these outcomes occurred (on average) among Washington community colleges. Instead, the only outcome that systematically distinguished Washington community colleges from others was the “lowest” lower order performance goals: the production of short-term certificate programs. After implementing SAI, colleges produced far more short certificates and fewer long certificates. We will first discuss why these results likely occurred, followed by a discussion of their implications.

One explanation for these results may be that retaining a student from year to year (and keeping them enrolled through the completion of an associate’s degree) is a more complicated process than SAI designers may have thought. Even “lower order” goals such as retaining students from 1 year to the next can be difficult to achieve, especially among students who work full-time (or are searching for work) while they are enrolled (Perna, 2010). We suspect that few colleges have this internal capacity to make substantial improvements in their retention efforts (beyond what they are already doing), which could be why the results are so mixed across the retention models. This does not mean that colleges cannot improve retention; rather, we suggest that improving retention requires resources and capacity that colleges simply may not currently have.

If retention is too high of a “lower order” goal for colleges to significantly impact, then the next lowest goal is to increase certificate productivity. Short-term certificates can be produced within 1 academic year, and colleges do not need SBCTC approval to offer new programs, so these could be one of the easiest ways a college can gain achievement points. Even easier would be for a college to retroactively award short-term certificates to students who earned enough credits but never received a credential and to ensure that currently enrolled students receive short-term certificates, even if they are working toward a different long-term certificate or degree.

SAI encouraged colleges to do just this, as stated in an SBCTC (2010) report that indicated, “Colleges have become more assiduous in ensuring that students who complete short certificates actually receive them . . .” (p. 10). Similarly, the report indicated that colleges began “modularizing” their long-term certificates, which would allow students in those programs to stop out for work or other reasons with a short-term credential and then reengage later to work toward the long-term certificate. This modularization may be leading to more students dropping or stopping out of the long-term certificate programs, as they can now exit early with a credential. We suspect that these are the primary reasons why short-term certificates grew and long-term certificates declined after SAI. Likewise, these responses are consistent with our theory, where colleges are expected to pursue and achieve the lowest order goals prior to higher order ones. The growth in short-term certificates, therefore, is not a surprising result.

Shifting attention to the highest order goal of improving associate’s degree productivity, impacts appear only after several years of the policy. As associate’s degrees are longer programs that require more credits than certificate programs, it is not surprising to see these delayed effects. However, we do not know for sure why this delay would occur. It is possible that, like with short-term certificates, colleges have gotten better at reporting degree completions or that they are retroactively awarding these credentials. It is also possible that colleges have changed their academic and administrative practices in ways that support students through to degree completion. This is the intended goal of SAI, and we provide limited evidence that this is occurring.

Another explanation why performance goals were largely not met (or were slow to see results) may be because the steps necessary for improving these outcomes are quite complex. There is a large body of literature on this topic, where researchers are finding that a range of factors including student engagement levels, academic support services, campus climate, student satisfaction, financial aid, and a host of additional factors affect students’ persistence to degree attainment (Kuh, Kinzie, Schuh, & Whitt, 2010; Pascarella & Terenzini, 2005; Seidman, 2005). Improving retention rates may sound like a straightforward and immediately attainable performance goal, but it can take a significant amount of campus resources including professional development and additional programming efforts to build the capacity for this change. Without this capacity, it is difficult to also improve longer term goals such as associate’s degree completion. Considering the financial incentives embedded within SAI were relatively small, we suspect the policy did not build the capacity necessary to accomplish these higher order goals. Not addressing these capacity constraints is indicative of an incomplete theory of action that links performance goals with the resources necessary to achieve these goals (Dougherty et al., 2013).

Further research is necessary to understand the mechanics through which these outcomes occur. For example, are colleges changing their academic offerings, providing more academic support, or otherwise engaging students in new ways to see them through to degree completion? SAI does not provide specific details or guidance on how colleges should support students, so further research is necessary to understand the factors behind these impacts. Considering the popularity of Washington’s performance funding model (often pointed to as a model of success), we are surprised the impacts on associate’s degree productivity are so modest.

Regardless of why these results occurred, we are concerned that SAI seems to have encouraged colleges to produce more short-term certificates. This is problematic for two reasons. First, these terminal credentials may not count toward future academic degree programs, so certificate holders may be less likely to pursue (or will be slowed down in their pursuit of) further education beyond these credentials (Roksa, 2006). Second, these credentials often yield poor labor market outcomes where the wage increase for certificate holders is typically lower than an associate’s degree and often have returns that are indistinguishable from workers with only a high school diploma (Belfield, Liu, & Trimble, 2014; Dadgar & Weiss, 2012; Jepsen, Troske, & Coomes, 2014; Marcotte, Bailey, Borkoski, & Kienzl, 2005; Mattoon, 2009). These studies also found a few programs that yield large returns, but these are exceptions to the rule. In fact, the short-term certificate which saw the largest increase in completions was the nursing assistant certificate, with an 84% increase. The most recent median wage for completers with this certificate was US$17,000 per year (Prince, 2013; SBCTC, 2010).

Although there are relatively few studies that distinguish between short- and long-term certificates, the research is consistent in finding that short-term certificates, on average, yield few (if any) benefits over a high school diploma. Long-term certificates, unlike short-term certificates, often lead to increased wages in certain occupations. In light of this, the growth in short-term credentials may not result in positive long-term labor market or educational outcomes for those who completed their programs. As these programs disproportionately serve older, lower income, and minority students, further research should focus on the equity implications of SAI (e.g., Did the policy track disadvantaged groups into low-wage professions?). Some of these concerns about short-term certificates may be alleviated with the state’s most recent changes to the state’s policy, which were revised in December 2012, and which now only awards achievement points to those short-term certificates which lead to middle- and higher wage jobs and/or lead to a more advanced credential (Prince, 2013), but these equity concerns are something researchers and policymakers should be monitoring closely (SBCTC, 2012).

Further research is needed to fully understand the causes and consequences of Washington’s SAI. While results from this analysis inform us about one state’s experience with performance accountability systems for higher education, only over time and through more national and statewide analyses can we begin to assess the full impacts of these reform efforts. This study builds upon this growing body of research, which has found that performance accountability reforms like Washington’s have resulted in limited to no improvements in the majority of educational outcomes measured in this study. Washington has invested significant amounts of time and financial, as well as political, resources into a performance system designed to avoid the pitfalls of the “old” performance funding systems. While the design of this new system is indeed different from past models and follows many of the best practices being promoted by advocates, the results are consistent with much of the previous research findings. Tying state financing to college “performance” is expected to create an incentive for colleges to produce more degree recipients and retain students at higher rates, but evidence from the Washington experiment suggests that these results do not occur in systematic ways.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

Authors

NICHOLAS W. HILLMAN is an assistant professor in the Department of Educational Leadership and Policy Analysis at the University of Wisconsin-Madison. He studies state and federal higher education finance.

DAVID A. TANDBERG is currently assistant professor of higher education and an associate director of the Center for Postsecondary Success at Florida State University. His research interests center on state higher education policy and politics.

ALISA H. FRYAR is an associate professor of political science at the University of Oklahoma. Her research interests include higher education policy, public management, and minority politics and policy.

References

Altstadt

(2012). Tying funding to community college outcomes: Models, tools, and recommendations for states (Jobs for the Future). Retrieved from http://www.jff.org/sites/default/files/publications/TyingFunding2CommColleges-042312.pdf

Bahr

P. R.

(2013). The deconstructive approach to understanding community college students’ pathways and outcomes. Community College Review, 41, 137–153.

Banta

T. W.

Rudolph

L. B.

Van Dyke

Fisher

H. S.

(1996). Performance funding comes of age in Tennessee. The Journal of Higher Education, 67, 23–45.

Belfield

C. R.

(2012). Washington state student achievement initiative: Achievement points analysis for academic years 2007–2011. Retrieved from http://ccrc.tc.columbia.edu/media/k2/attachments/sai-achievement-point-analysis.pdf

Belfield

C. R.

Liu

Y. T.

Trimble

M. J.

(2014, March). The medium-term labor market returns to community college awards: Evidence from North Carolina (Working paper). New York, NY: Center for Analysis of Postsecondary Education & Employment. Retrieved from http://capseecenter.org/wp-content/uploads/downloads/2014/03/medium-term-labor-market-return-to-community-college-awards.pdf

Betts

J. R.

McFarland

L. L.

(1995). Safe port in a storm: The impact of labor market conditions on community college enrollments. Journal of Human Resources, 30, 741–765.

Bogue

E. G.

Johnson

B. D.

(2010). Performance incentives and public college accountability in the United States: A quarter century policy audit. Higher Education Management and Policy, 22(2), 9–30.

Burke

J. C.

(2005). The many faces of accountability. In Burke

(Ed.), Achieving accountability in higher education: Balancing public, academic and market demands (pp. 1–24). San Francisco, CA: Jossey-Bass.

Burke

J. C.

Minassians

(2001). Linking state resources to campus results: From fad to trend—The fifth annual survey. Retrieved from http://files.eric.ed.gov/fulltext/ED456781.pdf

10.

Burke

J. C.

Minassians

H. P.

(2003). Performance reporting: “Real” accountability or accountability “lite”—The seventh annual survey. Albany, NY: The Rockefeller Institute.

11.

Chun

Y. H.

Rainey

H. G.

(2005). Goal ambiguity and organizational performance in US federal agencies. Journal of Public Administration Research and Theory, 15, 529–557.

12.

Dadgar

Weiss

M. J.

(2012, June). Labor market returns to sub-baccalaureate credentials: How much does a community college degree or certificate pay? (CCRC Working Paper No. 45). New York, NY: Community College Research Center, Columbia University.

13.

Dougherty

K. J.

Jones

Lahr

Natow

Pheatt

Reddy

(2013, July). Envisioning performance funding impacts: The espoused theories of action for state higher education performance funding in three states (Working Paper No. 63). New York, NY: Community College Research Center.

14.

Dougherty

K. J.

Natow

Hare

Jones

Vega

(2011). The politics of performance funding in eight states: Origins, demise, and change. New York, NY: Community College Research Center.

15.

Dougherty

K. J.

Natow

R. S.

Vega

B. E.

(2012). Popular but unstable: Explaining why state performance funding systems in the United States often do not persist. Teachers College Record, 114(3), 1–42.

16.

Dougherty

K. J.

Reddy

V. T.

(2011). The impacts of state performance funding systems on higher education institutions: Research literature review and policy recommendations. Retrieved from http://ccrc.tc.columbia.edu/media/k2/attachments/impacts-state-funding-higher-education.pdf

17.

Dougherty

K. J.

Reddy

V. T.

(2013). Performance funding for higher education: What are the mechanisms what are the impacts? ASHE Higher Education Report, 39(2). Retrieved from http://onlinelibrary.wiley.com/doi/10.1002/aehe.v39.2/issuetoc

18.

Dowd

A. C.

(2003). From access to outcome equity: Revitalizing the democratic mission of the community college. The ANNALS of the American Academy of Political and Social Science, 586, 92–119.

19.

Drukker

D. M.

(2003). Testing for serial correlation in linear panel-data models. Stata Journal, 3, 168–177.

20.

Garces

L. M.

(2013). Understanding the impact of affirmative action bans in different graduate fields of study. American Educational Research Journal, 50, 251–284.

21.

Harnisch

T. L.

(2011). Performance-based funding (PBF): A re-emerging strategy in public higher education financing. Washington, DC: American Association of State Colleges and Universities.

22.

Hillman

Orians

(2013). Community colleges and labor market conditions: How does enrollment demand change relative to local unemployment rates? Research in Higher Education, 54, 765–780.

23.

Hillman

Tandberg

Gross

(2014). Performance funding in higher education: Do financial incentives impact college completions? Journal of Higher Education, 85, 826–857.

24.

Jenkins

P. D.

Ellwein

Boswell

(2009). Formative evaluation of the student achievement initiative “learning year” (Report to the Washington State Board for Community and Technical Colleges and College Spark Washington). New York, NY: Community College Research Center.

25.

Jenkins

P. D.

Shulock

(2013, March). Metrics, dollars, and systems change: Learning from Washington state’s student achievement initiative to design effective postsecondary performance funding policies (A State Policy Brief). New York, NY: Community College Research Center, Columbia University. Retrieved from http://ccrc.tc.columbia.edu/media/k2/attachments/student-achievement-initiative-policy-brief.pdf

26.

Jenkins

P. D.

Wachen

Moore

Shulock

(2012). Washington state student achievement initiative policy study: Final report. New York, NY: Community College Research Center, Columbia University. Retrieved from http://ccrc.tc.columbia.edu/media/k2/attachments/student-achievement-initiative-final-report.pdf

27.

Jepsen

Troske

Coomes

(2014). The labor-market returns to community college degrees, diplomas, and certificates. Journal of Labor Economics, 32, 95–121.

28.

Kuh

Kinzie

Schuh

Whitt

(2010). Student success in college: Creating conditions that matter (2nd ed.). San Francisco, CA: Jossey-Bass.

29.

Lavy

(2010). Effects of free choice among public schools. The Review of Economic Studies, 77, 1164–1191.

30.

Marcotte

D. E.

Bailey

Borkoski

Kienzl

G. S.

(2005). The returns of a community college education: Evidence from the National Education Longitudinal Survey. Educational Evaluation and Policy Analysis, 27, 157–175.

31.

Mattoon

D. S.

(2009). Short-term certificates: Case studies of three California community colleges (Doctoral dissertation, University of the Pacific). Available from ProQuest Dissertations and Theses database. (UMI No. 3356559)

32.

McGuinness

A. C.

(1995). Prospects for state-higher education relations: A decade of grinding tensions. New Directions for Institutional Research, 1995(85), 33–45.

33.

McKeown-Moak

M. P.

(2013). The “new” performance funding in higher education. Educational Considerations, 40(2), 3–12.

34.

McLendon

M. K.

(2003a). The politics of higher education: Toward an expanded research agenda. Educational Policy, 17, 165–191.

35.

McLendon

M. K.

(2003b). State governance reform of higher education: Patterns, trends, and theories of the public policy process. In Smart

J. C.

(Ed.), Higher education: Handbook of theory and research (pp. 57–143). Dordrecht, The Netherlands: Springer.

36.

McLendon

M. K.

Cohen-Vogel

(2008). Understanding education policy change in the American States: Lessons from political science. In Cooper

Cibulka

Fusarelli

(Eds.), Handbook of education politics and policy (pp. 30–51). New York, NY: Routledge.

37.

McLendon

M. K.

Hearn

Deaton

(2006). Called to account: Analyzing the origins and spread of state performance-accountability policies for higher education. Educational Evaluation and Policy Analysis, 28, 1–24.

38.

Meyer

B. D.

(1995). Natural and quasi-experiments in economics. Journal of Business & Economic Statistics, 13, 151–161.

39.

Miao

(2012). Performance-based funding of higher education: A detailed look at best practices in 6 states. Washington, DC: Center for American Progress. Retrieved from http://cdn.americanprogress.org/wp-content/uploads/issues/2012/08/pdf/performance_funding.pdf

40.

Mullin

C. M.

(2012). Student success institutional and individual perspectives. Community College Review, 40, 126–144.

41.

Offenstein

Shulock

(2010). Taking the next step: The promise of intermediate measures for meeting postsecondary completion goals (Jobs for the Future). Retrieved from http://www.jff.org/sites/default/files/publications/ATD_TakingtheNextStep_092810.pdf

42.

Pascarella

E. T.

Terenzini

P. T.

(2005). How college affects students: A third decade of research (1st ed.). San Francisco, CA: Jossey-Bass.

43.

Perna

(2010). Understanding the working college student: New research and its implications for policy and practice ( Perna

, Ed.). Sterling, VA: Stylus. Retrieved from https://sty.presswarehouse.com/books/BookDetail.aspx?productID=217367

44.

Prince

(2013). Student achievement recommendation for including certificates. Olympia, WA: State Board for Community and Technical Colleges.

45.

Rabovsky

T. M.

(2012). Accountability in higher education: Exploring impacts on state budgets and institutional spending patterns. Journal of Public Administration Research and Theory, 22, 675–700.

46.

Radin

(2006). Challenging the performance movement: Accountability, complexity, and democratic values. Washington, DC: Georgetown University Press.

47.

Roksa

(2006). Does the vocational focus of community colleges hinder students’ educational attainment? The Review of Higher Education, 29, 499–526.

48.

Sanford

Hunter

J. M.

(2011). Impact of performance-funding on retention and graduation rates. Education Policy Analysis Archives, 19(33), 1–30.

49.

Seidman

(2005). College student retention: Formula for student success. Westport, CT: American Council on Education.

50.

Shadish

W. R.

Cook

T. D.

(1999). Comment-design rules: More steps toward a complete theory of quasi-experimentation. Statistical Science, 14, 294–300.

51.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.

52.

Shin

J. C.

Milton

(2004). The effects of performance budgeting and funding programs on graduation rate in public four-year colleges and universities. Education Policy Analysis Archives, 12(22), 1–26.

53.

Shulock

(2011). Concerns about performance-based funding and ways that states are addressing the concerns. Sacramento, CA: Institute for Higher Education Leadership and Policy. Retrieved from http://www.csus.edu/ihelp/PDFs/B_performance%20funding_05-11.pdf

54.

State Board for Community and Technical Colleges. (2010). Growth in short-term certificates at Washington’s community and technical colleges (Research Report No. 10-3). Olympia, WA: Author.

55.

State Board for Community and Technical Colleges. (2012). Revised student achievement framework 2012. Olympia, WA: Author. Retrieved from http://www.sbctc.ctc.edu/college/_e-student-achievement-2012.aspx

56.

Tandberg

Hillman

(2014). State higher education performance funding: Data, outcomes and policy implications. Journal of Education Finance, 39, 222–243.

57.

Tandberg

Hillman

Barakat

(2014). State higher education performance funding for community colleges: Diverse effects and policy implications. Teachers College Record, 116(12). Retrieved from http://www.tcrecord.org/Issue.asp?volyear=2014&number=12&volume=116

58.

Tebbs

Turner

(2005). Low-income students a caution about using data on Pell grant recipients. Change: The Magazine of Higher Learning, 37(4), 34–43.

59.

Titus

M. A.

(2006). Understanding college degree completion of students with low socioeconomic status: The influence of the institutional financial context. Research in Higher Education, 47, 371–398.

60.

Volkwein

J. F.

(2007). Assessing institutional effectiveness and connecting the pieces of a fragmented university. In Burke

(Ed.), Fixing the fragmented university (pp. 145–180). Boston, MA: Anker.

61.

Volkwein

J. F.

Tandberg

D. A.

(2008). Measuring up: Examining the connections among state structural characteristics, regulatory practices, and performance. Research in Higher Education, 49, 180–197.