A Little Can Go a Long Way: The Impact of Advertising Services on Program Take-Up

Abstract

The success of policy interventions is frequently stymied by the inability to induce take-up in target populations. In this article, I show that local advertising in combination with small financial lotteries increases the likelihood that low-income students apply for and receive state aid. I isolate causal impacts by estimating the change in completed aid applications in high schools where the advertising program was canceled due to the expiration of private funding compared with high schools that never participated in the advertising program. Using this differences-in-differences framework, I find that state aid applications declined by approximately 3% to 4% (or roughly four to six applications per high school). Furthermore, postsecondary enrollment in 4-year public colleges declined by about one-half to one percentage point in impacted high schools. These results suggest that small incentives may be a cost-effective means of promoting program take-up for marginal students.

Keywords

at-risk students economics of education educational policy higher education postsecondary education quasi-experimental analysis

Introduction

The success of policy interventions is frequently stymied by the inability to induce take-up in target populations (Hernanz, Malherbet, & Pellizzari, 2004). Well-targeted advertising services are an understudied method to increase program participation, particularly in the field of education. Government agencies frequently use advertising campaigns to promote perceived social goods, such as reducing drug use, increasing enrollment in health insurance, or increasing military enlistment. Identifying the impacts of these commercial campaigns, such as television or online advertisements, is statistically challenging in practice, as product purchases are intermittent, and individuals are constantly exposed to competing claims (Lewis & Rao, 2015). Yet advertising can take on many distinct forms that are more immediate and potentially more effective to the consumer than impersonal commercials for products. In well-targeted contexts—where the “purchase” dates are fixed, individuals are likely to have significant exposure to the advertising, and the messaging comes from a potentially trusted source—we may observe larger effects. High school students, who face a fixed deadline for financial aid submission and attend school most days, provide one such context.

This article analyzes an advertising campaign aimed at increasing enrollment rates of low-income students in a state-based financial aid program, the Cal Grant. The Cal Grant application requires students to complete two documents. The first is the Free Application for Federal Student Aid (FAFSA), which is the primary means of distributing federal aid to low-income students. Research has found that many eligible postsecondary students do not receive federal aid due to the complexity of the FAFSA application process (Bettinger, Long, Oreopoulos, & Sanbonmatsu, 2012; Dynarski, Scott-Clayton, & Wiederspan, 2013; Dynarski & Scott-Clayton, 2006). The second required document is a relatively minor barrier—a single-page grade point average (GPA) verification form completed by a high school guidance counselor, but which required student approval at that time.

I examine the role of two forms of advertising that were used to increase Cal Grant applications. Each year, hundreds of California high schools offer workshops, known as Cash for College (CFC), which provide students with information about the postsecondary system and direct support with completing their financial aid forms. The first method to increase CFC participation was the dissemination of promotional materials, such as posters, flyers, and informational folders, throughout the high school. In addition to students viewing the same promotional materials every day, these materials were also used in communications with family members and school staff to encourage these individuals to advocate for student participation in the workshop. The second method was financial lotteries, where schools would advertise that one attending student, drawn at random, would receive a US$1,000 scholarship to attend college. Both methods occurred in tandem, and their impact is estimated simultaneously.

I estimate the effects of these methods by calculating the change in the number of completed Cal Grant applications in 2014–2015, when the financial lottery program was canceled and the funding for promotional materials was significantly delayed. Simply put, the CFC workshops continued to operate as before, but CFC organizers were unable to effectively advertise or offer financial incentives for attendance. Using a differences-in-differences framework that compares high schools with CFC workshops to control schools that never held a CFC workshop, I find that completed Cal Grant applications declined by about four to six applications per school, off a baseline of roughly 140 students. These results suggest a 3% to 4% decline in completed applications per high school.

I also find that the decline in completed applications resulted in fewer students receiving Cal Grant awards in the subsequent year. This drop in financial aid usage occurred predominately in 4-year public colleges, with no observed impacts on the community college sector. Using a separate data source that contains annual high school counts of first-time freshmen attending the California State University (CSU) and University of California (UC) systems, I find that postsecondary attendance declined by roughly 0.7 to 1.0 percentage points in treated schools that lost advertising support. My estimates also show that the loss of the US$1,000 scholarship and supporting publicity led the state to decrease financial aid payments by roughly US$42,000 per high school. Although this is a decrease in state expenditures, it makes little sense to conceive of this decline as savings when the goal of the program is to support needy students. If declines in Cal Grant completion are driven in part by incomplete FAFSA applications, rather than just missing GPA verification forms, then California likely lost enough federal aid from decreased Pell Grant utilization to offset any advertising expenditures they would have incurred.

That these small inducements could affect individual behavior makes little sense in a typical human capital framework. Given the typical workshop size, the expected value of the financial lottery was roughly US$5 to US$10, whereas the state aid program offers 4 years of full tuition to any in-state public college and generous subsidies to private institutions, potentially worth tens of thousands of dollars. Yet recent research has shown that individuals often behave myopically, either avoiding or procrastinating on important tasks in a variety of consequential contexts (Ainslie, 1975; Baicker, Congdon, & Mullainathan, 2012; Levitt, List, Neckermann, & Sadoff, 2013; Schouwenburg, 1995). Economists have increasingly turned to the broad notion of “inattention,” which argues that individuals generally make decisions based on a few broad guideposts, rather than efficiently weighing all potential costs and benefits as a perfectly rational consumer (Gabaix, 2017). Inattention has been shown to be stronger in situations where individuals can only expect to receive rewards after a long delay, which is particularly relevant for 12th-grade students preparing for the transition to college. Given that inattention is both theoretically and empirically linked to worse decision-making, this suggests that tools that can draw students’ attention to important tasks, even if only briefly, may be welfare enhancing.

This paper suggests that advertising campaigns with small financial lotteries can draw students’ attention to important tasks and may be a cost-effective means of promoting program take-up for marginal students. Although previous work has shown that financial incentives, such as merit aid programs (Bettinger, Gurantz, Kawano, & Sacerdote, 2016; Dynarski, 2008), government subsidies or negative income taxes (Eissa & Hoynes, 2006; Michalopoulos, Robins, & Card, 2005), or paying for grades (Angrist, Lang, & Oreopoulos, 2009; Leuven, Oosterbeek, & van der Klaauw, 2010), can produce program take-up and educational benefits, these programs all rely on distributing rewards to all participants. As a result, these programs can be expensive on a per capita basis, as potentially large reduced form effects come at the cost of subsidizing individuals whose behavior is unchanged by the incentive. In contrast, incentives such as financial lotteries are relatively small payments that leverage behavioral responses to perceived large awards to incentivize individual behavior.

Literature Review

Cross-country studies of welfare take-up in Organisation for Economic Co-Operation and Development (OECD) countries suggest that only 40% to 80% of eligible individuals typically participate in available programs (Hernanz et al., 2004). Individuals are more likely to take up a program when they are provided direct application assistance (Aizer, 2003; Bettinger et al., 2012), when they have access to stronger social networks (Bertrand, Luttmer, & Mullainathan, 2000; Figlio, Hamersma, & Roth, 2015), or when their default is shifted to an opt-out delivery, rather than opt-in (Bergman & Rogers, 2016; Madrian & Shea, 2001).

Research often explains low take-up as a function of three types of barriers: administrative, stigma, and informational (Currie, 1996; Hernanz et al., 2004). Administrative barriers constitute tasks that require effort and are generally considered to be more important than stigma or informational barriers in explaining low take-up (Currie, 1996). For example, Daponte, Sanders, and Taylor (1999) and McGarry (1996) show that low take-up of the Food Stamp and Supplemental Security Income programs may have some informational aspects, but that the recipients’ estimated benefit is more directly related to the likelihood of applying, suggesting that the cost–benefit administrative effort of completing the application drives a significant portion of the results.

Research on the higher education system has frequently pointed to the FAFSA as an administrative barrier to the take-up of financial aid, thus unnecessarily lowering college attendance and completion rates for students from low-income families (Dynarski et al., 2013; Dynarski & Scott-Clayton, 2006). Given this reality, what can be done to increase FAFSA completion among low-income populations? Low-cost, purely informational interventions in higher education tend to produce few results (Bergman, Denning, & Manoli, 2017; Booij, Leuven, & Oosterbeek, 2012), suggesting more proactive measures are needed. Stronger results have been found when providing students with personalized information or concrete activities that focus individual attention and diminish effort costs (Bettinger et al., 2012; Castleman & Page, 2016; Castleman, Page, & Schooley, 2014; Hoxby & Turner, 2013).

The role of advertisements in program participation is less well understood. Although some studies have found advertisements to be effective in inducing military enlistment (Carroll, Rao, Lee, Shapiro, & Bayus, 1985) or commercial purchases (Johnson & Lewis, 2016; Johnson, Lewis, & Nubbemeyer, 2017; Lewis & Reiley, 2014; Sahni, 2015), there are a number of challenges in identifying causal impacts, even in large randomized control trials (Lewis & Rao, 2015). Primarily, there can be significant variance in individual behaviors, such as commercial purchases, that produce large standard errors. Yet advertising can also occur at a local level, significantly increasing their effectiveness. Individuals might not participate in free but important activities that are geographically proximate due to inertia or the desire to avoid onerous tasks and may be more easily swayed by increased attention and peer pressure. This article provides causal evidence of the impacts of precisely this type of localized advertisement campaign on program participation.

Background

The California Student Aid Commission (CSAC) administers the Cal Grant program, which is the largest merit- and need-based aid program in the nation. California residents who meet basic eligibility requirements apply for the Cal Grant by submitting two documents.¹ The first document is the FAFSA, which is used for federal grant and loan applications. Students who submit the FAFSA do not need to specify that they are applying for the Cal Grant, as CSAC is authorized to access federal FAFSA records for all students who list a domestic California address or have listed at least one California postsecondary institution on their FAFSA form. The second document is a GPA verification form that typically requires student authorization but is submitted to CSAC directly by the administration of the corresponding high school or college. The application deadline for high school graduates wishing to attend a 4-year public or private institution is March 2, which corresponds to the FAFSA application deadline for California.

High school graduates are eligible for a Cal Grant if they are from a “middle-income” family and have an unadjusted high school GPA of 3.0 or higher, or from a “low-income” family and have a minimum 2.0 unadjusted GPA. Income cutoffs vary by application year, dependency status, and family size. “Low-income” dependents have family income from below US$30,000 to US$55,000, middle-income dependents have a family income from US$70,000 to US$100,000, and independent students have personal income of below US$30,000. (Exact thresholds vary by family size and are available in the Online Appendix Table 1.) There are two distinct Cal Grant awards, although both offer at least 3 years of full-time tuition and fees to any in-state public 4-year college and a large subsidy of almost US$10,000 per year to private institutions.²

CSAC encourages students to complete Cal Grant applications by directly supporting outreach programs at high schools, colleges, government, and nonprofit organizations. Instead of directly administering these workshops, the state provides funding through Cal-SOAP (The California Student Opportunity and Access Program), which are independent, regional consortiums across the state that form intersegmental partnerships to promote college access. In addition, direct funding is provided to Regional Coordinating Organizations (RCO) throughout the state to provide “Cash for College” workshops; in some regions, the Cal-SOAP and the RCO are the same agency, whereas in others, they differ but work in tandem. CFC workshops began in 2002 before being established by legislation into state statute in 2007. CFC workshops are given at roughly 600 locations in January and February of each year, typically at high school or college campuses but also at nonprofits and other agencies. The CFC workshops are the state’s primary outreach program to encourage FAFSA and Cal Grant completion and have also been supported for a number of years through the federal College Access Challenge Grant Program (CACGP).

Workshop attendance may have been negatively affected in the 2014–2015 school year as a result of two important changes to program delivery. To encourage attendance at CFC workshops, the state has relied on philanthropic funding to support a small financial incentive aimed at boosting program participation. Students who attended CFC workshops and submitted completed financial aid forms were entered into a pool, with one student randomly drawn to receive a US$1,000 scholarship. The actual drawing occurred at a later date after all applications were assessed for completeness. This US$1,000 scholarship was often featured prominently on posters inviting students to attend the workshops. (Sample posters found online are available in the Online Appendix 2.) The expiration of private funding led this incentive to be eliminated in the 2014–2015 school year. The second issue was a delay in federal CACGP outreach funding that was deemed by CSAC to “significantly limit outreach materials available to promote and support workshops.”³ These outreach materials included both promotional and workshop materials, and consisted of items such as flyers, posters, brochures, and folders used to disseminate information about various Cal Grant initiatives, as well as other small giveaways, such as T-shirts.

The loss and delay of these external funding sources had no impact on other funding streams that supported student outreach. State funding for the CFC workshops was unchanged between 2013–2014 and 2014–2015, and total Cal-SOAP funding increased by 3% over the same time period. As a result, the CFC workshops operated in 2014–2015 just as they did in prior years, but with diminished advertising support that was designed to encourage workshop attendance.

CSAC’s internal data suggest that the loss of these advertising activities contributed to a decline in workshop attendance. In a report to the U.S. Department of Education as part of the Challenge Grant program, CSAC noted that

. . . in January and February of 2014 CFC . . . received exit surveys from 42,200 students . . . [in] 2015 CFC . . . received exit surveys from 34,750 students. While the number of offered workshops was maintained from the prior year, 2015 marks the first year in the history of the program recording any decrease in attendance.

CSAC also surveyed workshop organizers, which found that 51% of organizers believed that the loss of the scholarship resulted in fewer students attending the CFC workshops. In addition, 83% said that printed promotional and workshop materials were an “important” part of promoting attendance and that their loss led to fewer students being reached (25% of respondents) and less information about financial aid being made available to students (44% of respondents).⁴ Of course, fewer attending students does not necessarily translate into worse educational outcomes. It is possible that students who skipped the workshops were those likely to complete their financial aid forms even in the absence of these inducements and, therefore, had little motivation to attend regardless of these financial and nonfinancial incentives.

Data

Data from this project primarily comes from individual-level records on all California legal residents who applied for the Cal Grant by submitting FAFSA and GPA verification forms. I focus on the most direct metric for student application behavior, which is the count of fully executed Cal Grant applications. Although the loss of financial incentives may have other impacts, the Cal Grant application metric is of key importance to the state—Did Cal Grant–eligible students lose out on the award due to a lack of application assistance, driven by the loss of advertising used to draw in needy students? In practice, I also choose to focus on completed Cal Grant applications as the GPA verification forms are completed by high school counselors and include a unique high school record number that allows for school-level aggregation; in contrast, the FAFSA only began to ask for the high school a student attended in 2011–2012 through a text field that can be hard to classify and is subject to missing values.

I classify high schools as utilizing the advertising services if they participated in CFC workshops in both of the years prior to the elimination of funding: 2012–2013 and 2013–2014. I have 4 years of data—from 2010–2011 through 2013–2014—that lists CFC participation, and there is some variation with regard to which years CFC schools offered workshops. Classifying participants based on the last two years identifies 70% of the schools that ever served as CFC sites; more than half of the other 30% of CFC sites only hosted a CFC workshop once in the 4 years of my data, often many years prior to the advertising loss. As I later show, results are robust to alternate definitions of CFC participation.

I first aggregate counts of completed Cal Grant applications at the school-by-year level for 2009–2010 through 2014–2015, keeping dependent and independent students who met the requisite income and GPA eligibility thresholds. I create a balanced panel by first restricting the data to only those schools that had at least one Cal Grant submission in all 6 years and then further restricting to schools that averaged at least 10 Cal Grant applications across those 6 years. This restriction removes 178 schools, of which only four had ever hosted a CFC workshop. Inspection reveals that most of these schools were continuation high schools, alternative schools, independent learning institutions, virtual academies, and the like.

Table 1 provides summary statistics for my final sample, which consists of 1,112 high schools, of which 389 are classified as CFC sites. Schools with CFC workshops have significantly higher rates of students participating in the National School Lunch Program (NSLP), which is consistent with the aim of the CFC program to serve high-need schools. These high schools are also more likely to be located in urban centers or rural locales, rather than in the suburbs, and have significantly lower average scores on the SAT. These high schools also submit 30% more completed applications, which is likely due to a combination of having more eligible, low-income students and the support offered by CFC workshops. Figure 1 shows the number of applications for each school in my sample using data from 2013–2014 and previous years. The left panel shows the total number of completed applications, binned by every five applications. There is significant positive skew in this outcome variable, as many more small schools than large schools are represented in the sample. The right panel shows a logged transformation of total applications, which is more normally distributed (although with some negative skew). In the analysis that follows, I estimate effects on both total and log applications.

Table 1

Summary Statistics, 2013–2014

	CFC sites	Non-CFC sites
California Student Aid Commission
Completed applications	122.0	94.1
Average income (US$)	28,670	30,644
Student GPA	3.04	3.12
Dependent students (%)	92.0	91.9
Common Core of Data
Total enrollment	1,577	1,550
Percentage NSLP (%)	61.9	48.5
Asian (%)	10.8	10.6
Black (%)	6.9	6.5
Hispanic (%)	54.5	46.0
White (%)	23.6	32.2
Locale: City (%)	47.6	44.7
Locale: Suburb (%)	28.0	41.2
Local: Town/rural (%)	24.4	14.1
California Department of Education
SAT: Average verbal	462	493
SAT: Average math	474	505
SAT: Average writing	457	489
Number of graduates	321	326
N	389	723

Note. All values taken from 2013–2014, the last year of advertising availability. CFC = Cash for College; NSLP = National School Lunch Program; GPA = grade point average.

Figure 1.

Total completed Cal Grant applications

Finally, I examine potential impacts on subsequent postsecondary attendance using two distinct but complementary data sources. The first data source is Cal Grant utilization among eligible students, where I track individual-level award payments to specific institutions (e.g., community college, CSU). Although these data are available for all Cal Grant applicants, in practice, only 56% of eligible students in my sample actually used a Cal Grant award in the subsequent year.⁵ Thus, these data help me determine whether declines in completed applications actually translate into lower levels of award utilization, as negatively affected students may have been those unlikely to utilize the award regardless. I have only one year of Cal Grant payments for the 2014–2015 cohort, and so all analysis is restricted to the first-year postapplication.

To supplement this analysis on postsecondary attendance, I use a second data source consisting of publicly available high-school level counts of first-time freshmen in both the CSU and UC systems.⁶ These data include all students within a high school, not just low-income Cal Grant applicants, and so provide an opportunity to examine whether advertising loss indeed resulted in lower attendance levels. Although these data are only available for 4-year public institutions, I later show that the primary impacts of the advertising loss are exactly in this sector, rather than in community colleges or private institutions. Not every high school is represented, as the CSU system only provides data if a high school had five students enroll within the system in a given year and the UC system provides data if a high school had at least five applicants and three enrollees. Due to missing data and small issues with matching CSU data to CSAC records, these results rely on a slightly smaller sample of 1,059 schools, of which 385 are treatment, although I show that the main effects in this subsample match those in the full sample and are likely to be representative of the general population. I also used a balanced panel of schools that report CSU and UC enrollment in all 6 years, which includes 951 schools, of which 355 are treatment schools.

Method

My primary strategy for estimating the impact of lottery incentives on Cal Grant completion uses the following equation:

\begin{array}{l} Y_{j t} = β_{0} + β_{1} * T r e a t m e n t_{j} + β_{2} * y e a r_{j} + \\ β_{3} * C F C_{j} * y e a r_{j} + θ_{j} + X_{j t} + ε_{j t} . \end{array}

Y_jt is our primary outcome, either the total or logged counts of applications at the school (j) by year (t) level. In this equation, I estimate linear trends in application completion separately for non-CFC (β₂) and CFC schools (β₃) with school fixed effects (θ_j) that allow the intercept to vary across schools. Treatment_j is defined as one for all CFC sites in 2014–2015, the year the advertising campaign was affected. Thus, I use the slope for non-CFC schools to predict the counterfactual outcome in 2014–2015 for treated schools, and the resulting deviation from this prediction is estimated as our treatment effect (β₁). X_jt is a vector of time-varying observable characteristics that I use primarily to test the robustness of my estimation strategy. This vector comes from two sources: (a) Common Core of Data (CCD) school-level files that include characteristics such as school size, ethnic makeup, and free and reduced-price lunch participation and (b) California Department of Education (CDE) school-level files on total graduates and academic performance on the SAT. I cluster standard errors at the school level and generally find these to be more conservative than either unadjusted or robust standard errors.

One could question the necessity of allowing the slopes for CFC and non-CFC sites to differ. Shown in the following, models that rely on aggregate counts of completed applications exhibit a slight divergence in pretreatment trends over time, with CFC sites growing at a very small but slightly faster rate. In contrast, models based on logged counts exhibit similar slopes. Nonetheless, I favor presenting the results using Equation (1) and allowing for different slopes, rather than requiring strict equality, given that the total applications metric is easier to interpret than logged models. I include both metrics in my main results but predominately discuss the total application results, while noting that the total and log regressions provide similar results (after allowing slopes to vary) and at no point lead to substantively different conclusions.

Results

Impacts on Completed Cal Grant Applications

Figure 2 provides a graphical description of trends in applications for CFC and non-CFC sites, with pretreatment trends shown for both total applications (Figure 2, top panel) and log applications (Figure 2, bottom panel). In both figures, we see evidence of relatively similar pretreatment trends in applications, with a flattening or slight decline in applications in 2014–2015 when CFC scholarship funding was cut.

Figure 2.

Total and log Cal Grant applications, by presence of CFC workshop

Although the figures generally suggest parallel trends, I estimate this statistically using Equation (2). These results derive from a model that estimates a linear time trend in total or log applications for non-CFC sites, with year-specific treatment dummies that measure deviations from this trend at each time point for treated CFC sites (the year prior to treatment is 2014, and is the omitted year in this model):

\begin{array}{l} Y_{j t} = β_{0} + β_{1} * y e a r_{j} + β_{2} * C F C_2010 + \\ β_{3} * C F C_2011 + β_{4} * C F C_2012 + \\ β_{5} * C F C_2013 + β_{6} * C F C_2015 + \\ θ_{j} + ε_{j t} . \end{array}

The estimates for β₂ to β₆ are plotted in Figures 3 (log applications) and 4 (total applications). Figure 3 finds no evidence of large deviations in pretreatment log applications but a large and significant decline in 2014–2015, confirming the equivalency of pretreatment trends in application behavior prior to the policy change in 2014. Figure 4 provides a less convincing picture for total applications, with evidence of a slight upward trend in CFC sites relative to non-CFC sites in the pretreatment period. As discussed in the “Method” section, allowing for differences in trends between the two groups, as described by Equation (1), effectively handles this slight divergence in the preperiod and allows me to focus on changes in total applications, which is generally easier for the reader to interpret.

Figure 3.

Year-specific treatment effects on log applications

Figure 4.

Year-specific treatment effects on total applications

Table 2 builds on these initial results to guide the parametric estimation of the impacts of lost advertising services on total applications (top panel) and log applications (bottom panel). Column 1 estimates impacts based on Equation (1) above and finds that there were approximately six fewer completed applications per school. As the regression parameters predict approximately 138 applications per CFC school in 2014–2015, this constitutes a decline in expected applications of slightly more than 4% as a result of the advertising loss.⁷

Table 2

Impact of Advertising Loss on Completed Cal Grant Applications

	(1)	(2)	(3)	(4)	(5)
	Base model	Alternate models
Total applications
Treatment (CFC = 1 and Year = 2015)	−6.13** (1.81)	−4.74** (1.83)	−6.55** (1.97)	−5.11* (1.99)	−4.87** (1.67)
Year	5.01** (0.32)	4.26** (0.37)	5.82** (0.36)	4.95** (0.43)	5.12** (0.31)
Year × CFC	1.55* (0.61)	1.61* (0.78)	1.31^† (0.67)	1.46^† (0.86)	1.12^† (0.62)
Log applications
Treatment (CFC = 1 and Year = 2015)	−0.080** (0.018)	−0.065** (0.019)	−0.075** (0.018)	−0.065** (0.019)	−0.065** (0.019)
Year	0.058** (0.004)	0.043** (0.005)	0.059** (0.003)	0.047** (0.004)	0.058** (0.004)
Year × CFC	0.001 (0.006)	0.008 (0.007)	0.002 (0.006)	0.009 (0.007)	−0.006 (0.006)
First year	2010	2011	2010	2011	2010
Minimum applications	10	10	25	25	10
School-specific slopes	N	N	N	N	N
Treatment years	2 years	2 years	2 years	2 years	3 years
N	6,672	5,560	5,778	4,815	6,672

Note. All regressions include high school fixed effects and estimate treatment impacts with a dummy variable for workshop schools in 2014–2015. Standard errors are clustered at the school level. CFC = Cash for College.

†

p < .1. *p < .05. **p < .01.

Results are robust using alternate years of study or definitions of CFC sites: eliminating 2010 applicants to focus on more recent application years (column 2), eliminating smaller schools with fewer than 25 applications per year (column 3), or combining these two restrictions (column 4). I also alter my definition of a treatment site to be more restrictive, requiring sites to have offered scholarships in each of the last 3 years instead of two (column 5). Results are essentially unchanged, ranging from declines of 4.7 to 6.1 applications in the treatment year. Results that include time-varying school-level covariates produce similar results, ranging from 3.9 to 6.6 fewer applications (Online Appendix Table 2). Column 6 of the Online Appendix Table 2 also includes an alternate specification that allows for school-specific slopes to a similar effect. The bottom panel of Table 2 shows results for log applications, which suggest a decline of 0.065 to 0.08 log points across specifications. These results broadly mimic those found when examining total applications.

To attribute these impacts to the policy change, it is necessary that there are no large, concurrent changes to the composition of these schools in the treatment year that might bias my results. I cannot test for compositional changes using student characteristics from the Cal Grant application forms, as we do not observe the full population of students but rather an endogenous sample of those who apply. Thus, any deviations in student characteristics may be related to heterogeneous impacts of the policy on specific subgroups. Instead, I use school-level data from both the CCD and the CDE and run the same model as Equation (1) above but use time-varying, school-level covariates as the dependent variable. These results are shown in Table 3, where I examine school size, the percentage of students using the National Student Lunch Program, ethnic and sex composition, average SAT scores, and the total number of high school graduates. In general, the coefficients are not statistically distinct from zero. Although there is a small increase in the total school size in the year of the policy change, there is no difference in the total number of high school graduates who would actually be eligible for the program.

Table 3

Covariate Balance in Differences-in-Differences Model

	CCD
	School Size (N)	School composition (%)
	School Size (N)	Female	FRPL	Asian	Hispanic	Black	White
Treatment (CFC = 1 and Year = 2015)	26.038*(11.592)	−0.003(0.003)	−0.007(0.006)	0.001(0.001)	−0.007**(0.003)	0.001(0.001)	0.004(0.002)
N	6,651	5,539	5,502	6,640	6,640	6,640	6,640
	CDE
	Average SAT			Total Graduates
	Verbal	Math	Writing	Total Graduates
Treatment (CFC = 1 and Year = 2015)	1.658^†(0.930)	−0.117(0.991)	−0.347(0.919)	0.906(2.746)
N	6,239	6,239	6,239	6,662

Note. All regressions are estimated using Equation (1), which allows separate linear slopes for treatment and control schools, includes high school fixed effects, and estimates treatment impacts with a dummy variable for workshop schools in 2014–2015; this is the same specification as Table 2, column 1. Sample sizes vary across years due to missing values for the associated covariate. Standard errors are clustered at the school level. CCD = Common Core of Data; CFC = Cash for College; CDE = California Department of Education; FRPL = free and reduced price lunch.

†

p < .1. *p < .05 **p < .01.

In Table 4, I investigate heterogeneous impacts of the program based on student GPA and income. For simplicity, I divide the sample into four distinct groups: (a) dependents who are classified by CSAC as middle-income and have a GPA of above 3.0, (b) dependents who are low-income and have a GPA of above 3.0, (c) dependents who are low-income and have a GPA of below 3.0, and (d) independents. I select these four groups not only to broadly examine GPA and income differences but also as there are minor differences in their eligibility for the Cal Grant (discussed at length in footnote 3). I find that there is no statistically significant change in total or log applications from middle-income, high-GPA students. These students are likely to be the most resourced in the sample, who we might expect to have appropriate support to take full advantage of the college application process. The largest drop in total applications occurs among low-income, high-GPA dependents, although log changes between the three remaining groups are indistinguishable from one another.

Table 4

Impact of Scholarship Loss on Completed Cal Grant Applications

	(1)	(2)	(3)	(4)
Dependency status	Dependent	Dependent	Dependent	Independent
Income level	Middle-income	Low-income	Low-income	All
Grade point average	≥3.0	≥3.0	≥2.0	≥2.0
Total applications
Treatment (CFC = 1 and Year = 2015)	−0.71	−3.20**	−1.98*	−0.24
Treatment (CFC = 1 and Year = 2015)	(0.46)	(0.68)	(0.82)	(0.25)
Log applications
Treatment (CFC = 1 and Year = 2015)	−0.034	−0.102**	−0.092**	−0.099*
Treatment (CFC = 1 and Year = 2015)	(0.027)	(0.025)	(0.028)	(0.040)

Note. All regressions are estimated using Equation (1), which allows separate linear slopes for treatment and control schools, includes high school fixed effects, and estimates treatment impacts with a dummy variable for workshop schools in 2014–2015; this is the same specification as Table 2, column 1. Each regression uses 6,672 school-year observations, with missing logged values set to zero. Standard errors are clustered at the school level. CFC = Cash for College.

†

p < .1. *p < .05. **p < .01.

Impacts on Postsecondary Attendance

Decreases in completed applications may be inconsequential if the affected student was one who was unlikely to attend college even in the presence of the award, but I find that declines in applications translate into lower levels of financial aid utilization and postsecondary enrollment. I first examine outcomes by using data on Cal Grant utilization, which shows that first-year payments decrease by roughly five students (Table 5, column 1); corresponding graphical results for logged Cal Grant utilization, based on Equation (2), are shown in Figure 5. This result suggests that about 80% of students who did not complete the application would have used their award while attending college the subsequent year. The largest declines are in awards toward public 4-year institutions (columns 3 and 4), with no impact on community college attendance (column 2); graphical results are shown in the Online Appendix Figures 1 through 3. Column 5 shows no change in award usage at private, nonprofit institutions. There appears to be a small positive impact at for-profits (column 6), although this is likely due to functional form issues; CSAC eliminated the use of the Cal Grant at most for-profits in 2012–2013, and so very few students attend these institutions, and the year-to-year results are fairly noisy and susceptible to small changes.

Table 5

Impact of Advertising Loss on Cal Grant Usage

	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)
	All students	Community college	CSU	UC	PrivateNonprofit	PrivateFor-profit	Total payments	Pell grant eligible
Total payments
Treatment (CFC = 1 and Year = 2015)	−4.96**	−0.09	−3.46**	−2.01**	−0.19	0.80**	−41,822**	−4.99**
Treatment (CFC = 1 and Year = 2015)	(1.18)	(0.68)	(0.60)	(0.38)	(0.18)	(0.11)	(5969)	(1.71)
Log payments
Treatment (CFC = 1 and Year = 2015)	−0.090**	−0.037	−0.145**	−0.151**	−0.005	0.287**	−0.149**	−0.075**
Treatment (CFC = 1 and Year = 2015)	(0.021)	(0.030)	(0.028)	(0.035)	(0.051)	(0.086)	(0.028)	(0.019)

Note. All regressions are estimated using Equation (1), which allows separate linear slopes for treatment and control schools, includes high school fixed effects, and estimates treatment impacts with a dummy variable for workshop schools in 2014–2015; this is the same specification as Table 2, column 1. Each regression uses 6,672 school-year observations, with missing logged values set to zero. Standard errors are clustered at the school level. CSU = California State University; UC = University of California; CFC = Cash for College.

†

p < .1. *p < .05. **p < .01.

Figure 5.

Year-specific treatment effects on log Cal Grant payments

As a result of these declines, the total amount disbursed decreased at the school-level by roughly US$42,000 (Table 5, column 7), or about US$16.3 million across the state. As CSAC annually disburses more than US$2 billion in total awards through its various programs and services, this drop in services may have been difficult to detect in practice, particularly given other idiosyncratic changes in attendance patterns unrelated to this relatively small initiative. Although I cannot attribute this drop solely to the financial lottery, given that schools each offered US$1,000 in scholarship, this translates into an astounding 42-to-1 decline in financial aid disbursed for each dollar saved on lottery costs.

In the last column of Table 5, I document another potential consequence of these declines—the loss of federal financial aid that California may have received as a result of incomplete applications. Using data on expected family contribution (EFC), the results show a decline of five applications per school in the number of students that appeared to be Pell eligible. The loss of federal aid has consequences to both individual students, as Pell grants have been shown to increase student persistence (Bettinger, 2004), and the state, which is losing resources that could be put to use providing additional services. The minimum Pell Grant award for 2014–2015 was US$587, with a maximum of US$5,730, suggesting that almost any decline in FAFSA applications would be a net loss for the state. The resulting federal aid would have recovered the costs of the US$1,000 scholarship expenditure under minimal assumptions. To be clear, these impacts rely on a fairly strong assumption that declines in applications were primarily due to FAFSA completion. If the advertising loss caused fewer students to be aware of and turn in GPA verification forms, but had no impact on FAFSA completion, then these results would be overstated.

In Table 6, I turn to CSU and UC data on enrolled first-time freshmen, disaggregated by high school. As described in the “Data” section, the first row of results uses 6,163 of the available 6,672 observations, as some schools may be missing CSU or UC data, whereas the second row uses a balanced panel of 5,706 observations (951 schools) that have CSU and UC data in each year between 2010 and 2015. Figure 6 provides a graphical depiction of total postsecondary attendance in CFC and non-CFC sites, using the balanced panel of 951 schools. Columns 1 and 2 of Table 6 reproduce the main results and confirm that these data restrictions do not alter our general conclusion that advertising loss led to approximately six fewer completed Cal Grant applications per school. Figure 7 shows no difference in pretreatment trends of postsecondary attendance between CFC and non-CFC sites using results derived from Equation (2), providing evidence that we can interpret these results as causal.

Table 6

Impact of Advertising Loss on Postsecondary Attendance

		(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)
		Total applications	Log applications	Total enrollment (N)			Total enrollment (%)	School-level percentage (%)
	N	Total applications	Log applications	CSU	UC	All	Total enrollment (%)	Rate CSU	Rate UC	Rate all
Treatment (all observations)	6,163	−6.23**	−0.071**	−2.96**	−1.91**	−4.87**	−0.006	−0.004	−0.003^†	−0.007*
Treatment (all observations)		(1.88)	(0.018)	(0.74)	(0.50)	(0.99)	(0.007)	(0.003)	(0.002)	(0.004)
Treatment (balanced panel)	5,706	−6.41**	−0.065**	−3.14**	−2.04**	−5.18**	−0.004	−0.006*	−0.004*	−0.010**
Treatment (balanced panel)		(1.96)	(0.018)	(0.77)	(0.52)	(1.03)	(0.007)	(0.003)	(0.002)	(0.003)

Note. All regressions are estimated using Equation (1), which allows separate linear slopes for treatment and control schools, includes high school fixed effects, and estimates treatment impacts with a dummy variable for workshop schools in 2014–2015; this is the same specification as Table 2, column 1. Missing logged values set to zero. Standard errors are clustered at the school level. CSU = California State University; UC = University of California.

†

p < .1. *p < .05. **p < .01.

Figure 6.

Average number of students attending 4-year public colleges, by presence of CFC workshop

Figure 7.

Year-specific treatment effects on log 4-year public postsecondary attendance

Table 6 confirms that declines in completed Cal Grant applications led to significant declines in the number of high school seniors attending 4-year public institutions the following year. There were approximately five fewer CSU and UC attendees due to the advertising loss. In columns 6 through 9, I calculate this statistic another way, by dividing the decline in completed applications by the total 12th-grade CCD enrollment to estimate high-school percentage changes in completed applications and postsecondary attendance.⁸ These results suggest that, at the school-level, total Cal Grant applications declined about 0.4 to 0.6 percentage points (column 6) as a result of the advertising loss. Furthermore, CFC high schools experienced somewhere between a 0.7 and 1.0 percentage point decline in CSU and UC attendance. Predicted attendance in the 4-year public system in 2015 for CFC schools was approximately 22% (8% at UC and 14% at CSU), indicating close to a 3% to 5% decline in overall 4-year public college attendance. Without additional, but unavailable, data, we cannot determine whether these students delayed their 4-year enrollment, chose to enroll in a community college, or attended no postsecondary institution at all.

Conclusion

Although obvious, programs can only be effective if they can reach the students they intend to help. Getting information to targeted populations and convincing them to engage in application processes can be expensive and time-consuming under the best set of circumstances. This article provides evidence that institutions can promote participation through advertising campaigns combined with small financial lotteries. Observed declines in subsequent postsecondary attendance also support previous findings that application assistance can be particularly beneficial to less-resourced students as they transition from high school to college (Bettinger et al., 2012; Carrell & Sacerdote, 2017).

To interpret these difference-in-difference results as causal estimates, we must be reasonably sure that there were no other simultaneous changes that would have negatively affected schools with CFC sites. I have shown a number of quantitative results that support this conclusion, such as similarities in pretreatment trends along a number of outcomes and no concurrent changes in school composition or academic performance. Although it is generally impossible to eliminate all possible unobserved confounders, there are a number of additional features that support the loss of advertising as driving the negative results. First, CFC sites are a disparate group of schools hand-selected by the independent Regional Coordinating Offices and Cal-SOAP consortia, spanning the entire state. Thus, any systematic shocks, particularly those that are geographically driven, would have limited ability to negatively affect CFC sites as a whole. Second, as reported above, the advertising funding was philanthropic, but the primary CFC workshop funding came separately from the state budget. As this budget slightly increased in the treatment year, it is unlikely that workshop attendance would have declined through any mechanism other than the advertising. This is all the more striking given the nationwide push to increase FAFSA completion, where unobserved shocks, such as alternative support services for lower-income students, are likely to bias application estimates upward. I find that FAFSA applications consistently increase over time in both CFC and non-CFC sites, with the only observed decline in applications coming from CFC exactly in the final treatment period.

Two CSAC reports noted that workshop exit surveys declined by roughly 16% to 18%, although I observe drops in completed applications that are only one fifth to one third as large. This shows that most of the decline in workshop attendance was among students who would have completed their applications even in the absence of the advertising program. In this context, where students have the option to complete these applications in their own time, advertising outreach is likely to be less effective than it might be in other circumstances. Yet for the marginal student in this study, the impact of aid on postsecondary attendance is extremely large. My estimates broadly suggest a six-person drop in completed applications and a five-person drop in postsecondary attendance, indicating an 85% decline in postsecondary attendance for the marginal student. Although this result is much larger than observed in other financial aid studies, the financial impact in this study is also extremely large relative to other work. Losing both the Pell Grant and Cal Grant can result in a loss of aid for the marginal student that ranges from US$10,000 per year, given a maximum Pell award and annual CSU tuition that are both more than US$5,000, to close to US$20,000 per year for UC-bound students. Using the drop in applications as an instrument for college attendance would lead to large but extremely noisy estimates, but the overall picture is one in which large financial aid packages can have sizable impacts on whether students attend 4-year public colleges.

Preliminary cost–benefit calculations raise questions about how to value the program. Advertising loss led to a decline in California’s expenditures on financial aid, yet the true aim of the program is to support exactly those students most in need. A key but unanswerable question at this time is whether students who were induced not to attend would have been successful in college. If we assume that affected students are those who would have been unable to graduate or gain skills from their college experience, then the results presented here might constitute actual savings. The true impacts might be mixed, as some students induced to attend could perform poorly and potentially leave with higher debt, whereas research on financial aid suggests that some students would have been helped toward graduation. If the additional US$42,000 per school induced even one more student to complete their degree, that cost would be smaller or comparable with other papers that measure the impacts of financial aid on postsecondary completion (e.g., Barr, in press; Bettinger et al., 2016; Goldrick-Rab, Kelchen, Harris, & Benson, 2016).

Supplemental Material

DS_10.3102_0162373718774630 – Supplemental material for A Little Can Go a Long Way: The Impact of Advertising Services on Program Take-Up

Supplemental material, DS_10.3102_0162373718774630 for A Little Can Go a Long Way: The Impact of Advertising Services on Program Take-Up by Oded Gurantz in Educational Evaluation and Policy Analysis

Footnotes

Acknowledgements

I thank the staff of the California Student Aid Commission for providing the data access and institutional knowledge that made this project possible. I graciously acknowledge the Spencer Foundation and the Smith Richardson Foundation for financial support and acknowledge support by Grant R305B090016 from the U.S. Department of Education, Institute of Education Sciences.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Financial support for this study was provided by the Spencer Foundation and the Smith Richardson Foundation, and Grant R305B090016 from the U.S. Department of Education, Institute of Education Sciences.

Notes

Author

ODED GURANTZ is an Institute of Education Sciences fellow at the Stanford Graduate School of Education. His research focuses on gaps in college enrollment and completion between students from historically underserved groups and their more privileged peers, with an aim to examine programs and policies that are both effective and efficient in ameliorating educational disparities.

References

Ainslie

(1975). Specious reward: A behavioral theory of impulsiveness and impulse control. Psychological Bulletin, 82, 463–496.

Aizer

(2003). Low take-up in Medicaid: Does outreach matter and for whom? The American Economic Review, 93, 238–241.

Angrist

J. D.

Lang

Oreopoulos

(2009). Incentives and services for college achievement: Evidence from a randomized trial. American Economic Journal: Applied Economics, 1, 136–163.

Baicker

Congdon

W. J.

Mullainathan

(2012). Health insurance coverage and take-up—Lessons from behavioral economics. The Milbank Quarterly, 90, 107–134.

Barr

(in press). Fighting for education: The effect of the post-9/11 GI bill on degree attainment. Journal of Labor Economics.

Bergman

Denning

J. T.

Manoli

(2017). Is information enough? Evidence from a tax credit information experiment with 1,000,000 students (IZA Working Paper No. 10997). Bonn, Germany: IZA Institute of Labor Economics.

Bergman

Rogers

(2016). The impact of defaults on technology adoption, and its underappreciation by policymakers (Working Paper 17-021). Cambridge, MA: Harvard Kennedy School.

Bertrand

Luttmer

E. F. P.

Mullainathan

(2000). Network effects and welfare cultures. The Quarterly Journal of Economics, 115, 1019–1055. doi:10.1162/003355300554971

Bettinger

E. P.

(2004). How financial aid affects persistence. In Hoxby

C. M.

(Ed.), College choices: The economics of where to go, when to go, and how to pay for it (pp. 207-238). Chicago, IL: The University of Chicago Press.

10.

Bettinger

E. P.

Gurantz

Kawano

Sacerdote

B. I.

(2016). Long-term effects of financial aid: Evidence from the Cal Grants (NBER Working Paper No. 22347). Cambridge, MA: National Bureau of Economic Research.

11.

Bettinger

E. P.

Long

B. T.

Oreopoulos

Sanbonmatsu

(2012). The role of simplification and information in college decisions: Results from the H&R block FAFSA experiment. The Quarterly Journal of Economics, 127, 1205–1242.

12.

Booij

A. S.

Leuven

Oosterbeek

(2012). The role of information in the take-up of student loans. Economics of Education Review, 31, 33–44. doi:10.1016/j.econedurev.2011.08.009

13.

Carrell

Sacerdote

(2017). Why do college-going interventions work? American Economic Journal: Applied Economics, 9, 124–151. doi:10.1257/app.20150530

14.

Carroll

V. P.

Rao

A. G.

Lee

H. L.

Shapiro

Bayus

B. L.

(1985). The navy enlistment marketing experiment. Marketing Science, 4, 352–374.

15.

Castleman

B. L.

Page

L. C.

(2016). Freshman year financial aid nudges: An experiment to increase FAFSA renewal and college persistence. The Journal of Human Resources, 51, 389–415. doi:10.3368/jhr.51.2.0614-6458R

16.

Castleman

B. L.

Page

L. C.

Schooley

(2014). The forgotten summer: Mitigating summer attrition among college-intending low-income high school graduates. Journal of Policy Analysis and Management, 32, 320–344.

17.

Currie

(1996). The take-up of social benefits. In Auerbach

A. J.

Card

Quigley

J. M.

(Eds.), Public policy and the income distribution (pp. 80–148). New York, NY: Russell Sage Foundation.

18.

Daponte

B. O.

Sanders

Taylor

(1999). Why do low-income households not use food stamps? Evidence from an experiment. The Journal of Human Resources, 34, 612–628. doi:10.2307/146382

19.

Dynarski

S. M.

(2008). Building the stock of college-educated labor. The Journal of Human Resources, 43, 576–610.

20.

Dynarski

S. M.

Scott-Clayton

J. E.

(2006). The cost of complexity in federal student aid: Lessons from optimal tax theory and behavioral economics. National Tax Journal, 59, 319–356.

21.

Dynarski

S. M.

Scott-Clayton

J. E.

Wiederspan

(2013). Simplifying tax incentives and aid for college: Progress and prospects. Tax Policy and the Economy, 27, 161–202.

22.

Eissa

Hoynes

H. W.

(2006). Behavioral responses to taxes: Lessons from the EITC and labor supply. Tax Policy and the Economy, 20, 73–110.

23.

Figlio

D. N.

Hamersma

Roth

(2015). Information shocks and the take-up of social programs. Journal of Policy Analysis and Management, 34, 781–804. doi:10.1002/pam.21855

24.

Gabaix

(2017). Behavioral inattention. In Bernheim

DellaVigna

Laibson

(Eds.), Handbook of Behavioral Economics. New York, NY: Elsevier.

25.

Goldrick-Rab

Kelchen

Harris

D. N.

Benson

(2016). Reducing income inequality in educational attainment: Experimental evidence on the impact of financial aid on college completion. American Journal of Sociology, 121, 1762–1817. doi:10.1086/685442

26.

Hernanz

Malherbet

Pellizzari

(2004). Take-up of welfare benefits in OECD countries (OECD Social, Employment and Migration Working Papers 17). Paris, France: OECD Publishing.

27.

Hoxby

C. M.

Turner

S. E.

(2013). Expanding college opportunities for high-achieving, low income students. Stanford, CA: Stanford Institute for Economic Policy Research.

28.

Johnson

G. A.

Lewis

R. A.

Jr. (2016). Location, location, location: Repetition and proximity increase advertising effectiveness. Retrieved from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2268215

29.

Johnson

G. A.

Lewis

R. A.

Nubbemeyer

E. I.

(2017). Ghost ads: Improving the economics of measuring online ad effectiveness. Journal of Marketing Research, 54, 867–884.

30.

Leuven

Oosterbeek

van der Klaauw

(2010). The effect of financial rewards on students’ achievement: Evidence from a randomized experiment. Journal of the European Economic Association, 8, 1243–1265. doi:10.1111/j.1542-4774.2010.tb00554.x

31.

Levitt

S. D.

List

J. A.

Neckermann

Sadoff

(2013). The behavioralist goes to school: Leveraging behavioral economics to improve educational performance. American Economic Journal: Economic Policy, 8, 183–219.

32.

Lewis

R. A.

Rao

J. M.

(2015). The unfavorable economics of measuring the returns to advertising. The Quarterly Journal of Economics, 130, 1941–1973.

33.

Lewis

R. A.

Reiley

D. H.

(2014). Online ads and offline sales: Measuring the effect of retail advertising via a controlled experiment on Yahoo! Quantitative Marketing and Economics, 12, 235–266. doi:10.1007/s11129-014-9146-6

34.

Madrian

B. C.

Shea

D. F.

(2001). The power of suggestion: Inertia in 401(k) participation and savings behavior. The Quarterly Journal of Economics, 116, 1149–1187. doi:10.1162/003355301753265543

35.

McGarry

(1996). Factors determining participation of the elderly in supplemental security income. The Journal of Human Resources, 31, 331–358. doi:10.2307/146066

36.

Michalopoulos

Robins

P. K.

Card

(2005). When financial work incentives pay for themselves: Evidence from a randomized social experiment for welfare recipients. Journal of Public Economics, 89, 5–29.

37.

Sahni

N. S.

(2015). Effect of temporal spacing between advertising exposures: Evidence from online field experiments. Quantitative Marketing and Economics, 13, 203–247. doi:10.1007/s11129-015-9159-9

38.

Schouwenburg

H. C.

(1995). Academic procrastination: Theoretical notions, measurement, and research. In Ferrari

J. R.

Johnson

J. L.

McCown

W. G.

(Eds.), Procrastination and task avoidance: Theory, research, and treatment (pp. 71-96). New York, NY: Plenum.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.20 MB