Abstract
This article exploits a natural experiment to estimate the effects of need-based aid policies on first-year college persistence rates. In fall 2006, Ohio abruptly adopted a new state financial aid policy that was significantly more generous than the previous plan. Using student-level data and very narrowly defined sets of students, I estimate a difference-in-differences model to identify the program effects. Students who benefited from the program received awards about US$800 higher than they would have received under the prior program. These students’ drop-out rates fell by 2% as a result of the program. The new program also increased the likelihood that students attend 4-year campuses and increased their first-year grade point averages. The program may not have been cost-effective given the combination of its generosity and inability to target the marginal students who would be most sensitive to financial aid.
Keywords
College affordability is one oft-cited reason why some students may start college but not complete it (e.g., St. John, Cabrera, Nora, & Asker, 2000), and recent federal and state financial aid policies (e.g., Federal Academic Competiveness Grants) have aimed at improving retention through improving college affordability. In recent years, scholars in economics, sociology, and education have begun to focus on the impact of need-based aid programs on college retention (Bettinger, 2004; Castleman & Long, 2013; Goldrick-Rab, Harris, Kelchen, & Benson, 2013). 1
Need-based aid has received substantial interest from educators, policymakers, and academics. Most of the research focuses on the effects of Pell Grants on initial enrollment in and choice among colleges. The general consensus is that financial aid and the corresponding college costs affect students’ enrollment decisions (e.g., Ehrenberg & Sherman, 1984; Kane, 1999; Leslie & Brinkman, 1987). In terms of measuring the effects of aid on college outcomes, the literature generally finds that student aid improves graduation and retention rates (e.g., Bettinger, 2004; Castleman & Long, 2013; Dynarski, 2008; Goldrick-Rab et al., 2013).
This new wave of research has emphasized identifying causal links between need-based aid and student outcomes. Establishing such a link is inherently difficult. Need-based aid programs are means-tested programs. As such, need-based aid recipients come from poorer families and simple comparisons between recipients and non-recipients (e.g., Wei, Horn, & Carroll, 2002) may not show the effects of need-based aid programs. Similarly, comparisons based on variation in the size of students’ grants are flawed as college choice and the size of students’ need-based aid grants are directly connected. Students who attend more expensive (and often higher quality) schools are eligible for larger grant awards than students at other colleges or universities. Even in the absence of need-based aid grants, more academically qualified students are more likely to attend more expensive schools. To measure accurately the impact of aid on retention, researchers must exploit variation in need-based grants that is independent of college choice.
In this article, I exploit a unique natural experiment to measure the effects of need-based aid policies on student outcomes. The natural experiment that I focus on occurred in the 2006–2007 academic school year in Ohio. 2 In this year, Ohio embarked on one of the nation’s most ambitious, need-based financial aid policies, the Ohio College Opportunity Grant (OCOG), abandoning the previous policy, the Ohio Instructional Grant (OIG). The changeover to the OCOG program was abrupt, and the changes in financial aid packages were potentially large. A student who started college in 2006 could receive a financial aid award that was as much as 60% higher than an identical student who started college in 2005. Similarly, some students observed their aid packages decline as compared with what they would have been if they entered college a year earlier.
To identify the effects of the change in financial aid policies on students’ outcomes, I use student-level data for nearly 86,000 students who filed a Free Application for Federal Student Aid (FAFSA) in either 2005 or 2006 and attended college for the first time in either the 2005–2006 or 2006–2007 school years. I focus only on students who were dependent students by FAFSA guidelines. My data come from the Ohio Board of Regents (OBR), the agency that regulates higher education in Ohio. Because OBR collects comprehensive data on college enrollment in all Ohio’s public 2- and 4-year colleges, I can track students within and across schools. I can distinguish between students who withdraw from school and students who transfer to other Ohio schools. The data also include student demographics, college entrance ACT exams, and other student descriptors from the surveys conducted on the ACT exam and the FAFSA. The FAFSA data are the exact data used by the state and federal governments to determine students’ need-based grant eligibility. The outcomes I focus on include choice of institution at enrollment, persistence through the first year, transfer behavior during the first year, total hours attempted during the first year, and grade point average (GPA).
Empirically, I use a difference-in-differences methodology to identify the effects of the program. Specifically, I identify students who entered college in different years yet appear similar across financial characteristics. Although these students have nearly identical financial characteristics, their financial aid awards may differ because of their exposure to the new financial aid program. In the simplest specification, I identify three sets of students in each of the two cohorts I compare. The first group in each cohort is the policy “winners.” Although similar, “winners” in 2006 received higher financial aid awards than their counterpart “winners” from 2005. The second group in each cohort is the policy “losers.” The awards for policy “losers” in 2006 were less than the awards given to similar students in 2005. 3 The final group of students in each cohort were unaffected by the program. I first measure the changes within these groups of students across cohorts. I then compare these changes across various types of students to measure the effect. To gain more precision, I can control more narrowly for students’ financial characteristics and match students across years who have nearly identical financial characteristics in a tight geographic area.
I find evidence that student outcomes are sensitive to the change in financial aid programs that they receive. Students who received more aid after the policy change were less likely to drop out of college or transfer from their original institution after 1 year than they would have been under the old regime. The average impact of the program was a 2% reduction in drop-out rates among the students who received higher awards as a result of the program. Students who benefited from the program were also more likely to attend a 4-year campus and had higher GPAs after 1 year.
This article is organized as follows: The “Background” section provides an overview of previous literature on the effects of need-based aid and discusses both the OIG and OCOG grant programs. Section “Data and Empirical Strategies” describes the data in greater detail and presents the empirical strategies. Section “Empirical Results” presents the results, and “Discussion” section discusses the policy implications, including a cost–benefit analysis of the program. The final section concludes.
Background
Previous Research on Need-Based College Grants
Researchers have long focused on the effects of financial aid on student outcomes. Early studies of the Pell Grant generally failed to find significant positive effects on initial enrollment (e.g., Hansen, 1983; Kane, 1996; Manski & Wise, 1983). Reviews of the early literature generally conclude that Pell Grants did not improve enrollment rates among low-income students and minorities but likely affected which colleges students choose to attend (Ehrenberg & Sherman, 1984; Kane, 1999; Leslie & Brinkman, 1987).
Part of the difficulty in the early research was in trying to separate the effects of financial aid from confounding factors. More recent research has attempted to use “natural experiments” such as discontinuities in aid awards or changes in programs to identify the causal effects of financial aid programs. For example, Kane (2003) uses discontinuities in the Cal Grant award program to estimate the impact of the program on college decisions. He finds that Cal Grants increased college attendance by 3 to 4 percentage points. He also concludes that the program affected college choice. Seftor and Turner (2002) focus on how need-based aid affects older, nontraditional students. They find that Pell Grants affect enrollment decisions of non-traditional students. Dynarski (2003) uses discontinuities arising from the removal of the Social Security Administration’s Survivor Benefit Program. By comparing students who were eligible in the last year of the program and students who would have been eligible had the program continued, Dynarski finds sizable effects on both access and completion.
The literature measuring the impact of aid on student outcomes has developed in recent years. Dynarski (2004), for example, shows that large, state-run merit programs also increase completion rates, and Bettinger (2004) uses discontinuities in the Pell formulae caused by small differences in family size and the number of kids in college. He finds that Pell Grants increase students’ completion rates during their first year in college. Castleman and Long (2013) use a regression-discontinuity strategy to identify the impacts of Florida’s need-based grants. They find that college attendance rates increased by 3.2 percentage points, persistence rates increased by 4.3 percentage points, and ultimately, the probability of having a degree within 6 years increased by 4.6 percentage points. Goldrick-Rab et al. (2013) use a randomized experiment in Wisconsin to measure the impacts of need-based awards on students’ outcomes. They find that an increase in US$1,000 of aid caused a 3.5 percentage point increase in the retention after the first year for students in non-selective colleges, while finding no impact for students in selective colleges.
The most similar paper to the present study is Bert (2013). Bert’s dissertation focuses on subsequent changes in OCOG. With the economic downturn in 2008, it became apparent (for reasons I discuss in the “Discussion” section) that OCOG was unsustainable. OCOG and the Pell awards dramatically changed in response to the recession. 4 Bert’s results reinforce the finding that students’ outcomes improve (or worsen) as their financial aid awards increase (or decrease).
The present study builds on the growing consensus in the prior evidence. It demonstrates the external validity of evidence to date. The present study also presents some observations about the cost-effectiveness of Ohio’s financial aid program.
Ohio’s Need-Based College Grants
Through the 2005–2006 school year, Ohio offered need-based financial aid solely based on family size and income. The award program was called the OIG. Its awards ranged from US$174 to US$2,190.
In late-2005, there was a growing perception that the state was not providing sufficient aid for low-income families. Enrollment and completion rates for low-income families continued to lag behind other states, and the gap between low-income and high-income enrollment and completion rates was increasing. For example, drop-out rates after the first year of college were more than 50% higher for students who were Pell eligible (Bettinger, 2004). In addition, the National Center for Public Policy and Higher Education (2004) had given Ohio an “F” grade on student affordability in its 2004 report. This perception that Ohio’s higher educational institutions were unaffordable led to Ohio replacing OIG with a much more generous program.
In the 2006–2007 school year, Ohio introduced a new, ambitious, need-based financial aid program called the OCOG. For students initially enrolling in the 2006–2007 school year, OCOG awards ranged between US$300 and US$2,496. Although the range of the rewards was similar to the prior program, the distribution was not. Together with the Pell Grant, students’ need-based aid award increased by 10%, and some students saw increases in aid as high as 60% relative to what similar students would have received in the preceding year.
One of the key differences in the OIG and OCOG programs was the eligibility criteria. Under OIG, the state only used income and family size to determine eligibility. By contrast, OCOG used students’ estimated family contributions (EFCs) from the FAFSA as the basis for rewarding financial aid. EFC relies on family size and a broader definition of income, but it also incorporates details on family assets and the number of children in college. As a result of this shift, students from families with low incomes but high assets or alternative sources of income received less state aid, whereas poor families with no alternative sources of income were able to receive more aid than before. Figure 1 shows these differences graphically. I plot the average change in state aid (award under OCOG − award under OIG) for students in 23 groups defined by US$100 increments of EFC. Students with an EFC below US$1,500 received over US$500 more in aid under OCOG, and some students received over US$1,000 more. As this figure shows, however, the OIG to OCOG change had quite different implications for students across the EFC distribution.

Change in individual state aid award under OCOG.
Students were grandfathered into the program, so if a student entered college under the OIG program, they would remain in the OIG program throughout their college career. Similarly, a student who entered college under OCOG remained in OCOG throughout their college career. The OIG schedule between the 2005–2006 and 2006–2007 school years was unchanged, so I can identify how large an entering student’s financial aid package would have been in 2006–2007 had the program not changed.
Data and Empirical Strategies
Data
The data for this project come from the OBR as part of a collaborative partnership between OBR and a set of higher education researchers. At the time when the project was initiated, the author was appointed as a Special Assistant to the Chancellor of the OBR. In this role, it became apparent that research on the effects of OCOG would be important for assessing future financial aid policy. Early versions of these analyses were shared with the Chancellor and his staff.
OBR is a strong research partner, in that it has significant data on Ohio public schools. OBR gathers admissions, transcripts, and graduation data for students at Ohio’s public institutions of higher education. OBR has collaborative arrangements enabling them to link students’ college data with FAFSA and ACT and SAT records. The ACT exam is the most commonly used exam in Ohio, and the accompanying survey includes student-reported data on high school performance. The FAFSA data include information about the finances of both students and their families. The data also include students’ EFCs, which institutions use in computing students’ Pell Grant awards and OCOG awards. The FAFSA data also include family size and income, which the state of Ohio uses to compute students’ OIG awards. I do not observe any supplementary institutional aid, either need- or merit-based.
One limitation of the data is that they only include students attending Ohio public universities. Students from Ohio that attend universities in other states and students that attend private schools in Ohio are excluded from the sample. A majority of the students in the sample are “dependent” by FAFSA standards (i.e., below the age of 24, unmarried, and without children), and so I focus solely on dependent students.
I focus on incoming first year students in the 2005–2006 and the 2006–2007 school years. Students in the 2005–2006 cohort were eligible for OIG. Their financial aid was determined completely by their parents’ income and family size. The students in the 2006–2007 cohort were eligible for the OCOG. Their financial aid awards were determined by students’ EFCs. Students must submit a FAFSA to receive federal aid such as Pell Grants, and only a handful of non-FAFSA filers received state aid. I therefore limit this discussion to students who filed a FAFSA. Because the OIG schedule was unchanged between 2005 and 2006, I can use income and family size data to estimate what the OIG awards would have been for entering students in 2006–2007 even though they received aid under the new regime. I can similarly measure what students’ OCOG awards would have been for students entering in 2005 even though they received their aid awards under OIG.
Table A1 in the appendix shows some descriptive statistics for the sample and the overall population of students in Ohio. It demonstrates how the financial aid regimes changed the aid awards across the 2005 and 2006 cohorts. In the 2005, about 21% of students at main university campuses received state aid whereas 23% received state aid in 2006. Students’ Pell Grant awards were fairly similar (within US$150), yet there were large differences in state aid received. State financial awards to students receiving any aid increased by almost US$700 at university main campuses, representing a 52% increase in aid. At other campuses, state aid awards increased by 54%.
In Table 1, I show the basic elements of the identification strategy. I identify three distinct groups of students who, as groups, would have had different experiences with the financial aid regimes for the 2005–2006 and 2006–2007 school years depending on when they entered college. To identify these groups, I estimate the aid that they would have received under either the OIG or OCOG schedules. In the final three columns, I present t statistics comparing the types of students from 1 year to the next within each group.
Descriptive Statistics by Winner/Loser Status
Note. Data are for first-time college freshman entering Ohio public colleges and universities in fall 2005 and 2006 who filed a FAFSA. FAFSA = Free Application for Federal Student Aid; OIG = Ohio Instructional Grant; OCOG = Ohio College Opportunity Grant; w/I = within; AGI = adjusted gross income.
, **, *** indicate significance over 90%, 95%, and 99% confidence intervals, respectively.
The first group consists of people for whom the change in policy worsened or would have worsened their college aid package. If individuals did or would have received less aid under the new regime, I label them a “loser” under the policy change. About 4.1% of all FAFSA-filing students had worse aid packages under the new system. All of these policy “losers” would have received aid under OIG; however, only 29% of them received aid under OCOG. In other words, a policy “loser” type of student would have definitely received a financial aid award if he or she started school in 2005, but if he or she started school in 2006, the likelihood of receiving an award dropped dramatically. Moreover, as Table 1 shows, this group received about US$550 less aid as a result of the shift in the aid program. Their state aid went from US$807 to US$250. Pell Grant aid decreased by almost US$70 from the 2005 to the 2006 cohort of policy “losers”; these students were about 8 percentage points less likely to qualify for a Pell Grant in the later cohort. These families have lower incomes than the families of students whose aid was unaffected by the policy change, but they have a smaller family size than both other groups. They also have higher predicted assets 5 than families that benefited from the shift in financial aid regimes.
The second group includes people for whom the program would not have or did not lead to any change in their financial aid. About 68.8% of all FAFSA filers were in this second group. All of these “status quo” students received no state aid, and their average Pell Grant awards are small (less than US$100). From the 2005–2006 to 2006–2007 cohorts, parental incomes increased, predicted assets decreased slightly, and family size remained constant.
The students in the final group were the “winners” in the policy change, those who had or would have had better college aid packages as a result of the policy shift. About 27.0% of all FAFSA-filing students were in this group. On average, these students’ state financial aid packages increased by more than US$800 going from US$1,225 to US$2,030 across the two cohorts, corresponding to a 66% increase in state aid. Pell Grant aid increased slightly although given that the Pell schedules did not change from year to year, this increase is likely indicative of how the cohorts changed in their underlying composition. Family size remained constant among this group across cohorts, while parental income and predicted assets both fell slightly. “Winners” and “losers” differ along several dimensions. Winners were generally poorer than other students, and they have fewer assets and larger family size.
The next few rows show some basic demographics on the sample. The changes in demographics between the cohorts are statistically similar in each of the three groups for gender, race, age and the likelihood that the student lives on campus. One interesting note is the racial comparison across the three groups. About 40% of the winners in the policy shift were non-White, whereas about 21% of the losers in the policy shift were non-White and 13% of the “status quo” were non-White. There were declines in the proportion of students taking the ACT exam across all of the groups. In addition, test scores increased in all three groups. The increase in test scores comes either from improved student preparation or is the result of some less prepared students not taking the ACT exams.
The final rows in Table 1 show a few college outcomes for these three groups across cohorts. Among the “losers,” students in the later cohort completed fewer semester hours during their fall term. The “losers” also observe their drop-out rates rise, although the increase is not statistically significant. Among the “status quo” and “winners,” drop-out and transfer rates declined. Hours completed also increased for “status quo” students. The key to knowing the impact of the policy will be seeing whether the increases for winners are statistically different than the changes for the other groups. I turn to this in the next section.
Empirical Strategy
I use a difference-in-differences approach across successive cohorts to estimate the impact of the program. Specifically, I estimate
where winner, loser, and post-reform are indicators for the type of student or the time cohort in question. Xit includes characteristics of student i in cohort t. In some specifications, I also include zip code fixed effects, so that my estimates compare similar students within narrow geographic areas.
The key identifying assumptions in the difference-in-differences approach are that there are fixed, time-invariant differences across groups and that the treatment is the only factor altering these differences over time. If the groups of individuals have different trends or trajectories, it would violate these identifying assumptions. Ideally, I would like to observe the types of students who would have been winners and losers under the new policy for many years before the change in policy. Unfortunately, I only have data for 2 years—the year before and after the reform.
Another weakness of the difference-in-differences strategy is that I only observe the average effect of the program on each group, but the treatment effects could be quite heterogeneous within groups. For example, among policy losers, close to 25% of students received US$250 less in total aid under the new policy than they would have received under the old regime. By contrast, about 5% of losers received US$1,974 less than what they would have received. If the effects of the program vary by the amount of money the students lose or gain, then the average effects from the difference-in-differences parameters may hide important heterogeneity in the elasticity of college outcomes to college aid.
One strategy that might help us control for some of this heterogeneity is to define the groups of “winners” and “losers” more narrowly. With my data, I can compute the exact EFC that each family would have had. Rather than control for winners or losers as a homogeneous group, I can control for US$100 increments in students’ EFC amounts. 6 I can include a fixed effect for each of these groups (replacing α i in Equation 1) and look for differences in outcomes across years within these groups. In other words, rather than dividing both cohorts into three types of students (i.e., winners, losers, and status-quo students), I can further divide the groups into more homogeneous groupings (e.g., policy winners who had an EFC between US$100 and US$200). By including these fixed effects, I can estimate the effect of the program by comparing students across years with almost identical financial characteristics.
My primary focus is on 2 first-year outcomes—dropout and transfer. Dropout is defined as withdrawing from higher education altogether after students’ first year of college. Given that I only observe students in the public colleges in Ohio, students could be conceivably transferred to a private or out-of-state college. Most college dropouts occur in the first year of college making it a focus of much of the literature on college persistence (e.g., Herzog, 2005; Jamelske, 2009). Many students transfer after their first year as well. To an institution, these transfer students will appear to be dropouts. Transfers have the impact of slowing students’ progress through college (e.g., Arum & Roksa, 2010). Students often lose credit hours. Although ultimately, some transfer behavior could improve student outcomes, I do not have the data to detect these future benefits. As such I use it as a marker of persistence within the same institution and examine how transfer responds to the change in regimes. I focus only on the first year for two reasons. First, I only have data for the year before and the first year of the OCOG program. Second, midway through the second year of OCOG, the Great Recession began, and it became clear that dramatic changes were coming to OCOG. The recession or the expectation of the dramatic changes in OCOG (analyzed in Bert, 2013) may have changed students’ behavior. In the time upon which I focus, federal aid and the Ohio economy were stable.
One of the challenges in measuring the effects of financial aid policies on persistence is that the policies may also have affected who attends Ohio colleges. For example, in Table 1, the number of “losers” attending college dropped by about 625 students—nearly a 27% decline among “losers” once the policy was enacted. In 2005–2006, these students had OIG awards that provided some aid. Similar students who entered college a year later had less favorable terms. Similarly, the number of “winners” enrolled in 2006–2007 was about 650 students higher than those enrolled in the previous year.
The problem with measuring the effects of the policies across these cohorts is that OCOG may have introduced new “winning” students to college. I do not know if these new students were a random sample of other winners, if these students were less prepared than other winners, or if the increase in the number of winners attending college was even the result of OCOG. This knowledge is necessary to know whether the estimated effects are biased upward or downward.
To see this, consider the estimated effects from the difference-in-differences parameters in Equation 1. These represent the average change in persistence for the cohort of winners; however, suppose that the program had the effect of increasing the percentage of winners in the sample by η%. If this is the case, then the estimated difference-in-differences parameter γ will be equal to
where τ is the difference in persistence rates between the newly attending students and the status-quo students and π is the change in persistence rates as a result of OCOG for winners who would have attended college even in the absence of the program.
If the newly attending students are similar to the overall population of winners, then τ = π = γ and my estimates are not biased. However, if these new “winners” are less prepared than the existing winners, then τ is likely lower than π and the estimated difference-in-differences parameter is likely understated. Table 1 shows only small changes in the overall number of students (6% increase in the number of winners), suggesting that η is likely small. I discuss the likely value of τ below.
Empirical Results
Difference-in-Differences Estimates
Table 2 shows my baseline difference-in-differences estimates of OCOG’s effects. I focus on how the generosity of financial aid changes as a result of the OCOG. In column 1, I just present the simple difference-in-differences estimates without any covariates. My standard errors correct for heterogeneity across individuals in this column and throughout the table.
OLS Estimates of Impact of OCOG on Total Aid
Note. Robust standard errors are provided in parentheses. Covariates include gender, race, age, whether students took the ACT, ACT score, and whether students lived on campus. Data are for first-time college freshman entering Ohio public colleges and universities in fall 2005 and 2006 who filed a FAFSA. OCOG = Ohio College Opportunity Grant; FAFSA = Free Application for Federal Student Aid; EFC = estimated family contribution; FE = fixed effects; OLS = ordinary least squares.
, **, *** indicate significance over 90%, 95%, and 99% confidence intervals, respectively.
The key rows in Table 2 are the first two that report the difference-in-differences effects. Echoing the results from Table 1, losers’ financial aid awards dropped by about US$630, while winners’ awards increased by about US$860. The results suggest that the program dramatically changed financial aid awards, and the estimated effects are statistically significant. The other coefficients are in line with those shown in Table 1. Winners received on average US$4,680 more than the status-quo students in both state and federal aid combined (“winner main effect”). Policy losers received about US$2,027 more than status-quo students in overall aid.
In column 2, I add controls for basic covariates, including age, race, gender, whether the student lived on campus, whether the student took the ACT exam, and the student’s ACT score. The difference-in-differences estimates of the treatment effects change very little from column 1. In column 3, I include fixed effects for the zip codes of students’ permanent home address. Again, the estimates change very little.
In columns 4 and 5, I also include fixed effects for the EFC categories described in the previous section. This strategy compares the financial aid awards across years of students living in the same zip code and having similar financial backgrounds (defined very narrowly by EFC categorical dummy variables). The estimated changes in financial aid as a result of the program are very similar to the estimates in the other columns. Policy losers’ financial aid awards fell by about US$630 with the introduction of the OCOG award while policy winners’ financial aid awards increased by about US$750. These results clearly demonstrate that the policy dramatically changed financial aid awards.
In Table 3, I now turn to the effects of the program on students’ college outcomes. The first columns of Table 3 focus on the likelihood that students withdrew from college during their first year. The next columns focus on whether students transferred to another campus or dropped out. The “main” effects show that winners and losers were more likely to drop out or transfer as compared with the “status quo” students. These effects are significant in my basic regressions but, at least for policy winners, the “main” effects become small and insignificant once I include fixed effects for students’ financial backgrounds. Policy losers appear to have been 4 to 5 percentage points more likely to drop out and about 3 percentage points more likely to either transfer or drop out.
OLS Estimates of Dropout and Transfer
Note. Robust standard errors are provided in parentheses. Covariates include gender, race, age, whether students took the ACT, ACT score, and whether students lived on campus. Data are for first-time college freshman entering Ohio public colleges and universities in fall 2005 and 2006 who filed a FAFSA. FAFSA = Free Application for Federal Student Aid; EFC = estimated family contribution; FE = fixed effects; OLS = ordinary least squares.
, **, *** indicate significance over 90%, 95%, and 99% confidence intervals, respectively.
My estimated difference-in-differences effects differ for both winners and losers. Policy winners appear to have had lower drop-out rates than the status-quo students. This estimated effect is around 2 percentage points for both drop-out and transfer behaviors. The estimate is stable and consistent even in my most restrictive of specifications. Students who entered school in the 2006–2007 school year and were beneficiaries of the change in financial aid policy regimes were less likely to drop out than similar students who started college the previous year. However, policy losers’ drop-out and transfer rates appear to have been unmoved by the change in the financial aid policy. The point estimate is remarkably close to 0 and statistically insignificant.
The changes in aid were almost equal and opposite for winners and losers, yet the estimated effects are very different. One possibility is that these groups of individuals (i.e., winners and losers) have different elasticities of demand for education. It might also be that I just lack statistical precision in estimating the effects for losers. The group of policy losers is much smaller than the group of policy winners. The standard errors are more than twice the size of the corresponding estimates for winners, and the confidence intervals are sufficiently large that I cannot reject that the estimates are the same across winners and losers. Also, it could be that the change in the composition of the loser population from 1 year to another “undid” the potentially negative effects of aid. Table 1 shows that policy losers in the later time cohort had slightly lower ACT scores, which might have reduced dropout if anything. Even so, the lack of precision may impede any ability to identify impacts, unless the net impacts of aid and cohort composition were at least 3 percentage points in magnitude.
The estimated difference-in-differences estimate is the average change in outcomes across a range of changes in financial aid. The result is a reduced-form result and does not fully capture the heterogeneity that might exist between students. The prior literature has attempted to identify an effect size per dollar of financial aid implicitly assuming that the impacts are somewhat linear. I find limited support for linearity. Whereas the impact of the shift in financial aid regimes had almost equivalent albeit opposite impacts on the size of the financial aid awards for winners and losers, the impact on outcomes was largely centralized among winners. Moreover, if I measure the program impact separately for each EFC group, I find little evidence of linearity. The sizes of the estimated coefficients within each bin have little relationship to the underlying EFCs and the corresponding levels of financial aid. The simple Wald estimate comparing average changes in outcomes with average changes in financial aid amounts suggests that drop-out rates fall by about 2 percentage points per US$1,000 of financial aid, 7 but given the lack of evidence for linearity, the Wald estimate does not capture the inherent non-linearities, heterogeneity, or even the marginal impacts.
Discussion
Robustness
The difference-in-differences estimates are valid if the differences in the trajectories of the different types of students (i.e., winners, losers, and status-quo students) are constant over time except through the influence of the program. However, the program led to differences in the composition of students across years. The “winners” in the new regime had low incomes, and the increased financial aid might have increased the likelihood that these students enrolled. As observed in Table 1, there was an increase in the population of “winners” over this period, and the average income of the “winning” population declined.
One specification check would be to see if mean student characteristics changed differentially across years. To test this, I compare how student academic characteristics changed over time across groups. I report these estimates in Table 4. The difference-in-differences estimates show the relative change in composite, math, and English ACT scores across both winners and losers. In both cases, there are no significant differences across student achievement over time. Almost all of the point estimates are negative, suggesting that there was a slight decline in both winners’ and losers’ test scores over time. The lack of significance may indicate that the difference-in-differences strategy is identifying similar students.
ACT Comparisons in Difference-in-Differences
Note. Robust standard errors are provided in parentheses. Covariates include gender, race, age, and whether students lived on campus. Data are for first-time college freshman entering Ohio public colleges and universities in fall 2005 and 2006 who filed a FAFSA and took the ACT. FAFSA = Free Application for Federal Student Aid; EFC = estimated family contribution; FE = fixed effects.
Table 4 also sheds light on possible biases in my results. As observed in Table 1, the overall number of “winners” increased suggesting that the program could have led to an increase in college attendance among potential winners. In Equation 1, my difference-in-differences estimates are likely biased downward if new “winners” are less prepared than the existing winners. Table 1 shows that winners’ incomes dropped over time and Table 4 shows that the average ACT score declined as well (although this change was not statistically significant). The decline in income and test scores may suggest that less prepared students did enroll as a result of the financial aid change and suggest that the estimated difference-in-differences estimates are an understatement of the true effect. Going back to Equation 2, τ, the impact of the financial aid program on students who attended college as a result of the financial aid program is likely lower than π, the impact of the financial aid program on students who would have attended college even in the absence of the program. Hence, the estimated difference-in-differences estimate likely understates the impact of the financial aid program on students already enrolled in college.
Another robustness check is to see whether students with lower EFCs had different trends in drop-out behavior than other students. If they were less likely to drop out over time for reasons other than aid, then I would observe the results in this article. To test this, I divided the “status quo” group into low-EFC and high-EFC subgroups. While even the low-EFC individuals in the “status quo” groups have higher EFCs than the “winners” or the “losers,” the contrast between the EFC groups within the “status quo” might shed light on whether there was a trend in lower-income students to persist at greater levels. I find no such trend. The trend among low-EFC and high-EFC students in the “status quo” group is statistically similar between the years. 8
Possible Mechanisms
There are some other outcomes that might shed light on the potential mechanisms that might explain the effect. For example, in Table 5, I examine the effects of the program on the likelihood that students attend the main university campus or a university branch campus. My difference-in-differences estimates suggest no change in the probability of enrolling at a 4-year university main campus; however, I find that there was a 1.5% to 2% increase in the likelihood that “winners” started college at a university main campus or a branch campus. Previously, these students would have attended a 2-year college. It could be that the increase in academic rigor that comes with attending a 4-year campus may have increased the likelihood that students persist. This finding may be supportive of Castleman and Long’s (2013) work where effects were strongest in the 4-year college. The work by Goldrick-Rab et al. (2013) finds impacts at non-selective 4-year colleges but not selective colleges.
Difference-in-Differences Estimates for Other Outcomes
Note. Robust standard errors are provided in parentheses. Covariates include gender, race, age, whether students took the ACT, ACT score, and whether students lived on campus. Data are for first-time college freshman entering Ohio public colleges and universities in fall 2005 and 2006 who filed a FAFSA. FAFSA = Free Application for Federal Student Aid; EFC = estimated family contribution; FE = fixed effects.
, **, *** indicate significance over 90%, 95%, and 99% confidence intervals, respectively.
I also examine whether the program led to changes in the number of credit hours attempted. The program did not alter hours for winners, while losers reduced the number of hours that they took in their first semester. I also find that GPAs increased for winners by about 0.04 GPA points. The estimate is significant for winners, and while it is similar in magnitude for losers, it is not significant. This increase is small, accounting for about 3.6% of standard deviation. The increase in GPA may suggest additional engagement in school. I do not observe hours that students work, but I know from prior studies that financial aid reduces the likelihood that students work (Stinebrickner & Stinebrickner, 2008). Prior literature suggests that working has negative impacts on student performance (Stinebrickner & Stinebrickner, 2003). While I cannot test this mechanism, the working–financial aid relationship found in other literature may explain my finding here. Goldrick-Rab et al. (2013) find that retention results are strongest for students with a predisposition to work.
Cost–Benefit Analysis
Is the program worth it? I can do a simple cost–benefit analysis by comparing the costs of the program with the overall benefits. Given that I only have outcomes for the first year, I have to make some assumptions on the possible outcomes in subsequent years to measure the overall cost–benefit. In the best case scenario, the decrease in drop-out rates after the first year persists until students eventually earn their college degrees. In this case, the increase in degree receipt would be 2 percentage points. In the worst case scenario, the decrease in drop-out rates only lasts for 1 year after which time students leave school having completed only 1 additional year of college.
To compute the internal rate of return, I calculate the following present discounted value in Equation 3:
where α denotes the proportion of students who stay in school as a result of the program; Nw and NL denote the total number of winners and losers, respectively; y(t) represents individuals’ earnings, which is a function of the years attended college (C); Aw and AL denote the aid expense or savings for winners and losers, respectively; δ w and δ L denote the fraction of winners and losers who remain in school after the first year; and r represents the discount rate. I assume that individuals work for 40 years after college and that college only takes 4 years to complete.
The first term in Equation 3 is the increase in lifetime earnings for students who now, as a result of the new aid regime, complete college rather than dropping out. 9 The second term captures the additional cost in financial aid associated with the policy. This additional aid is paid to all winners who remain in school. The third term captures the opportunity cost that winners face in staying in school. This only applies to those who stay in college as a result of the program. The final term captures the cost savings for losers as their financial aid falls as a result of the program. Given that I did not observe statistically significant changes in enrollment for losers, I assume that their enrollment patterns were unchanged by the policy.
I use the previous tables for most of my estimates. There were 11,095 winners and 1,701 losers. Winners’ awards increased by US$806 per year, whereas losers’ awards decreased by US$557 per year (see Table 1). The overall impact on drop-out rates (α) was about 2 percentage points. About 65% of winners and 69% of losers persisted after the first year. I use income measures from the College Board (2007).
For the best case scenario, I assume that the increase in earnings is the difference between the earnings reported for a college degree (US$50,900) and that of having some college but no degree (US$37,100). I assume that the foregone earnings in any given year are only a function of the number of years of college that a student had completed at that time, and I assume a 10% return to each year of education.
In the best case scenario, the 2 percentage point decrease in drop-out rates leads to a 2 percentage point increase in degree completion. In this case, I find that the internal rate of return to the aid program is 5.1%. If I allow the retention rate to decay over time, the internal rate of return can be as high as 5.5%.
In the worst case scenario, the 2 percentage point decrease in drop-out rates only lasts 1 year. In this case, the opportunity cost of attending college is only 1 year, but the benefit from lifetime earnings is smaller representing only 1 additional year of schooling. In addition, the cost of maintaining the program changes as winners (and losers) who would have remained in college in the absence of the program still have a change in their awards. In this case, the internal rate of return is −0.01%. If I allow the retention rate to decay over time for each group, the internal rate of return is just 0.03%.
The return in the best case scenario is similar to other educational programs. As a comparison, the rate of return for Project STAR’s class size experiment was between 5% and 10% (Schanzenbach, 2007), and Dynarski (2008) shows that the internal rate of return for merit-based aid programs typically located in the South (e.g., Georgia Hope) is 9%. In the worst case scenario, however, the return does not suggest cost efficiency.
The program could increase its efficacy if it targeted students whose outcomes are likely affected by the new financial aid policy. The group of “winners” represents almost 25% of all students who filed FAFSAs. Most winners were not affected by the program, and their retention rates increased by just 2%. If policymakers could more effectively target the subsidies to the students who were on the margin of dropping out, it would reduce the overall costs of the program by decreasing the expenditures on winners who would have stayed in college, regardless of the increase in aid. In more tangible terms, the state has to increase its expenditure by roughly US$860 for 11,095 students. Of those 11,095 students, roughly 220 change their behavior and stay in college for an additional year. The rest either drop out like they planned or stay in college as they planned. The entire stream of benefits is realized by these 220 students. Improving the targeting would essentially reduce the total number (11,095) that receives additional aid.
Of course, as in any cost–benefit analysis, there are caveats to the analysis. For example, I have not computed any non-wage benefits to college. I have assumed that college enrollment does not increase in response to the program. I have also not accounted for increase in either tuition or the generosity of the program. Still, the range of possible internal rates of return provides a benchmark from which it is possible to start evaluating the overall effectiveness of the program.
Conclusion
The results suggest that financial aid does affect students’ decisions to withdraw from college. Using the change in policy regimes in Ohio, I show that the change in financial aid regimes led to a 2 percentage point increase in college persistence rates during students’ first year in college. Students were more likely to attend a 4-year campus in response to the program and students’ GPAs increased by a small but significant amount. The results contribute to the growing consensus that demonstrates that financial aid programs influence both college choice and college persistence.
If the effects on college persistence extend beyond the first year through graduation, then the program appears to have a positive return of about 5%. If, however, the effects do not extend beyond the first year, the program would not appear to be cost-effective. Regardless of the duration of the effects, improvements in program targeting could increase the cost-effectiveness of the program.
Footnotes
Appendix
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Notes
Author
ERIC BETTINGER is an associate professor at the Stanford Graduate School of Education. His research focuses on the economics of higher education.
