The Effects of Student Coaching

Abstract

College graduation rates often lag behind college attendance rates. One theory as to why students do not complete college is that they lack key information about how to be successful or fail to act on the information that they have. We present evidence from a randomized experiment which tests the effectiveness of individualized student coaching. Over the course of two separate school years, InsideTrack, a student coaching service, provided coaching to students attending public, private, and proprietary universities. Most of the participating students were nontraditional college students enrolled in degree programs. The participating universities and InsideTrack randomly assigned students to be coached. The coach contacted students regularly to develop a clear vision of their goals, to guide them in connecting their daily activities to their long-term goals, and to support them in building skills, including time management, self-advocacy, and study skills. Students who were randomly assigned to a coach were more likely to persist during the treatment period and were more likely to be attending the university 1 year after the coaching had ended. Coaching also proved a more cost-effective method of achieving retention and completion gains when compared with previously studied interventions such as increased financial aid.

Keywords

higher education mentoring retention randomized trial

Introduction

While college attendance rates have risen dramatically over the past four decades, college completion has not kept pace. For example, while the percentage of 23-year-old high school graduates with some college experience increased by 31% between 1970 and 1999, degree attainment by this age increased by only 4%. Over this time period, completion rates among college participants fell by more than 25% (Turner, 2004). Whereas the United States previously led the world in the percentage of the population having bachelor’s degrees, it has now lost that leadership. Over the last three decades, cohort-based bachelor’s attainment rates have increased by 2 to 3 percentage points across cohorts in the United States, while other OECD countries such as the United Kingdom and France have seen 10 to 15 percentage point increases in degree attainment (OECD, 2007).

These concerns about educational attainment have led to increased scrutiny of college completion and movements to hold universities accountable for graduation rates. Foundations and policymakers have increased their focus on improving persistence and graduation rates. For example, President Obama has mentioned college completion in his State of the Union addresses, most notably in 2009 when he said,

This country needs and values the talents of every American. That is why we will provide the support necessary for you to complete college and meet a new goal: by 2020, America will once again have the highest proportion of college graduates in the world. (Obama, 2009)

This focus on completion rates is not new; universities have long been concerned with low completion rates and have actively searched for strategies to increase college persistence and completion. One such effort, which is the focus of our article, has been the use of mentors and coaches to facilitate student persistence and completion.

Our article focuses on coaching, a form of college mentoring. InsideTrack is a for-profit provider of coaching services. InsideTrack matches students to potential coaches, and these coaches regularly contact their students to provide help and support as the students start their college careers and continue through their first year in school. In coaches’ interactions with students, they work to help students prioritize their studies, plan how to be successful, and identify and overcome barriers to students’ academic success. Specifically, the coaches focus significant time assessing the student’s life outside of school, which InsideTrack believes is the leading influencer of student persistence and completion. Topics such as personal time commitments (work scheduling), primary care-giving responsibilities, and financial obligations are common during a student-coach interaction.

Over the past decade, InsideTrack has provided student coaching at a variety of public, private, and proprietary colleges. The company’s model focuses on partnering with universities to deliver its mentoring program. InsideTrack provides the required people, processes, and technologies. The company claims that the economies of scale the company realizes from serving multiple institutions enable it to make investments that are typically out of reach for individual colleges and universities.

Our data come from InsideTrack. We requested data from InsideTrack for the 2003–2004 and 2007–2008 school years. We chose these years because the 2007 cohorts were the most recent groups for whom we observe 24-month retention rates and the 2004 cohorts align well with nationally representative data sets.

InsideTrack wanted to convince the participating universities of its effectiveness. So to eliminate bias, InsideTrack used randomization in 17 cohorts to determine with which students they worked.¹ It is on these 17 cohorts that we will focus our study. Within institutions, InsideTrack randomly divided eligible students into two balanced groups. These pseudo-lotteries enable us to compare the set of students who received coaching to those who did not and to create unbiased estimates of the impact of the services.

We find that retention and completion rates were greater in the coached group. This held true for every length of time following enrollment. After 6 months, students in the coached group were 5.2 percentage points more likely to still be enrolled than students in the noncoached group (63.2% vs. 58.0%). At the end of 12 months, the effect was 5.3 percentage points. The effects persisted for at least 1 more year after the coaching had concluded. After 18 months, there was a 4.3 percentage point increase in college retention, and after 24 months, there was still a 3.4 percentage point treatment effect from the coaching. These differences are all statistically significant over a 99% confidence interval. Moreover, these results do not change when we control for a variety of student characteristics. For the three cohorts for which we have degree completion data, we find that graduation rates increased by four percentage points. All of these estimated effects represent the intention to treat, and given that not all students selected for the treatment actually participated in the treatment, estimates of the effect of the treatment on the treated are higher.

Background on Student Coaching

College Retention Studies

College retention has long been the focus of research in education, sociology, and economics, and the relationship between student and institutional characteristics and college graduation rates has been a frequent topic in academic literature (e.g., Gansemer-Topf & Schuh, 2006; Tinto, 1975, 1998). In the past few decades, numerous empirical and theoretical studies have attempted to accurately isolate the most influential obstacles and identify potential interventions. The literature has identified several barriers which could potentially reduce graduation rates.²

One identified barrier to postsecondary success is lack of access to appropriate information. The need for student guidance in college has been well documented.³ Research has found that many community college students have little knowledge of course requirements and are unsure if their courses will meet requirement needs (Goldrick-Rab, 2010). Deil-Amen and Rosenbaum (2003) noted that explicitly structured advising is advantageous to students with less social know-how (first generation college students and those from lower socioeconomic backgrounds). They found that such students often do not know that they need help, do not take the initiative to seek it out, or do not know what questions to ask.

A related line of study comes from the emerging research in behavioral economics. Recent studies have focused on the complexity of processes that students face and the information they use to make decisions (e.g., Bettinger, Long, Oreopoulos, & Sanbonmatsu, 2012). Students often need a “nudge” (Thaler & Sunstein, 2009) to complete complex tasks. In higher education, it is often assumed that course requirements provide that nudge or that students are sufficiently self-motivated and do not need external stimuli. College graduation rates indicate that that assumption might not be true; students may benefit from structured “nudges” to complete necessary tasks.

A second barrier to postsecondary success is students’ academic preparation and performance. Academic preparation has long been acknowledged as a contributing factor to college retention (e.g., Adelman & Gonzalez, 2006). Studies of college remediation (e.g., Bettinger & Long, 2009; Calcagno & Long, 2008) have attempted to identify whether academic remediation can improve students’ college outcomes.

Similarly, other interventions have focused on improving the efficacy of students’ nonacademic school skills, such as time management and study skills. For example, Zeidenberg, Jenkins, and Calcagno (2007) found that enrollment in a student success course at Florida community colleges corresponded to an increase in persistence rates of 8 percentage points. Other studies (e.g., Kern, Fagley, & Miller, 1998; Robbins et al., 2004) have shown a positive association between productive study habits and cumulative GPA and college persistence.

A final obstacle to graduation that is related to college mentoring is students’ lack of integration into the university community. Tinto (1975) articulated a theory of retention that suggests that feelings of academic or social separation lead students to drop out. Researchers have attempted to identify ways to increase students’ feelings of integration (e.g., Bloom & Sommo, 2005) in an attempt to increase college retention rates.

In addition to interventions that aim to address one particular obstacle, there are a number of interventions which attempt to address several barriers and influence students in multiple dimensions. Learning communities, comprehensive programs that enroll a cohort of undergraduate students in a common set of courses and provide academic and advising support, are a well-researched example of such an intervention.⁴

College mentorship, the focus of this study, is another intervention that addresses the problem of college attrition through multiple dimensions; it has elements of academic preparation, information gathering, and social integration. College mentors can have multiple goals: to help a student academically prepare for their courses, to counsel students on how to acquire better study skills, or to provide advice on how to identify additional academic resources at their respective institutions.

Such support may be increasingly necessary, as traditional college counseling programs may be overextended in their efforts to provide support for all students. A study of counselors at community colleges found that counselors report high student-to-counselor ratios; 55% of schools have counselor to student ratios between 1 per 1,500 and 1 per 3,500 (Gallagher, 2010).

Studies of educational interventions that have attempted to use college counseling as a means for improving college outcomes provide an important context for the current investigation. There have been several such studies in the past decade. However, there are two complications that make evaluating these interventions difficult. First, treatments identified as “counseling” or “advising” vary greatly. Some are strictly academic, while others focus on study skills and social needs. Second, the most rigorous evaluations of counseling interventions to date have generally introduced multiple treatments, such as financial awards and social supports. The counseling component has typically been ancillary to the mechanism of interest.

There are a number of high quality studies that have looked at the impact of enhanced counseling layered with financial incentives. As part of MDRC’s Opening Doors demonstrations, Scrivener and Weiss (2009) and Brock and Richburg-Hayes (2006) studied such interventions and found some effects on academic outcomes such as credit accumulation. Angrist, Lang, and Oreopoulos (2009) also examined the effects of financial incentives and support services on academic achievement and persistence. The authors found that students who were in a group receiving a combination of financial incentives and support services benefited the most. There was no impact on grades for the advising only group and the students who received only a fellowship only showed a small increase in grades. Importantly, these results were driven only by significant effects on female students; male students showed no increases in retention or academic success.⁵

These studies suggest that advising can be an effective strategy for improving college retention by addressing common barriers to success. However, the effect of trained one-on-one counselors on retention has not been studied by itself; most rigorous studies have included other interventions in addition to enhanced counseling.

Background on InsideTrack

The motivating principle at InsideTrack is that student coaching can lead to engagement, learning, retention, and an increased probability of completing a degree. InsideTrack began offering services in the 2000–2001 school year and has coached more than 250,000 students nationally. The company first tested its coaching program by offering “free academic strategy sessions” to students at Stanford University and the University of California, Berkeley. Building on the success of these initial coaching curricula, the company partnered with universities to provide coaching to their incoming students. InsideTrack is now the largest provider of student coaching in the country, employing hundreds of coaches who work with thousands of students nationwide. Because InsideTrack has worked with a variety of private, public, and proprietary institutions, lessons from InsideTrack may be more generalizable than studies of a particular institution.

Part of InsideTrack’s business model included clearly demonstrating its success to its partner universities. InsideTrack offered new clients the option of running an experiment at their school. The universities gave a list of potential students to InsideTrack. Each school determined the criteria for inclusion and the size of the sample according to their own priorities. While most schools assigned a representative sample of new entrants, there was some heterogeneity in the assignment systems. Some schools focused on full-time students; others assigned part-time students. Some assigned upperclassmen; others assigned new entrants. One school assigned athletes.

Part of the agreement between the school and InsideTrack included a procedure for quasi-random assignment. Representatives from InsideTrack randomly divided the students into two groups. They then “rebalanced” the groups, moving students from one group to the other to ensure that the groups were balanced on observable characteristics.⁶ After balancing the groups, the partner organization chose which of the two groups would receive counseling and coaching services with a coin flip.⁷ These groupings allowed universities to monitor and evaluate ex-post the efficacy of InsideTrack.⁸

Students in the treatment group were then randomly assigned by InsideTrack to a “coach.” The goal of the college coach was to encourage persistence and completion by helping students find ways to overcome both academic and “real-life” barriers and to identify strategies for success by helping students use resources and advocate for themselves. The company hopes that coaches provide informed, empathetic support separate from students’ academic and personal lives.

Like any successful intervention, InsideTrack’s coaches’ effectiveness is due to a complex formula of factors. However, this “secret sauce” seems to include four primary ingredients: people, methodology, supporting systems, and technology.

InsideTrack is very particular in which coaches they hire. The application and interview processes are rigorous, and InsideTrack hires only a small fraction of applicants. Over the past 11 years, the company has created a large library of tools and resources and coaches are trained to work with these proprietary methodologies and programs to help students navigate college decisions. Coaches receive extensive feedback on the content and tone of their calls (all calls are recorded) and there are many institutionalized support systems in place for training and professional development.⁹

Each coach contacts his or her students via phone, email, text messages, and social networking sites and initially presents himself or herself as a representative of both InsideTrack and the partner institution. Coaches generally work with students over two semesters.

InsideTrack estimates that 20% of the content of their calls is institution-specific and 80% is more general. However, they note that it is the institution-specific 20% that provides the initial “hook.” In some cases, coaches have access to course syllabi, transcripts, and additional information on students’ performance and participation in specific courses. InsideTrack uses this additional information in a set of predictive algorithms that assess each student’s status for the purpose of reaching out to them on the right issues at the right times. Because of this background knowledge, conversations between coaches and students are both individualized and focused on success in school. The InsideTrack management credits these uses of technology as a major part of their expansion.

Students have the option to participate or not when contacted by the coach. All of the students, regardless of whether they opted to participate in the coaching, are included in our analysis. InsideTrack separates student-coach interactions into two broad categories: contacts and meetings. Contacts are brief interactions (typically less than 5 min) that do not cover any specific topic in-depth. Meetings are conversations of at least 5 min in which several topics are covered and next steps are established. About 98% of students in the treatment group receive at least one contact from the coach. About 94% of these students had meetings. About 77% of students receive at least 5 contacts.

Data and Empirical Methodology

Data

To evaluate InsideTrack’s program, we requested directly from InsideTrack the academic records for all of the students who had been randomly assigned to coaching or control groups in the 2003–2004 and 2007–2008 school years. During those 2 years, InsideTrack measured the performance of 13,555 students across eight different postsecondary institutions, including 2- and 4-year schools and public, private not-for-profit, and proprietary colleges.¹⁰ The students were randomly assigned in 17 lotteries—5 occurring in the 2003–2004 school year and 12 in the 2007–2008 school year. Across these 17 cohorts, InsideTrack randomly assigned 8,049 students to receive services. The other 5,506 students did not receive InsideTrack coaching services. All other services (i.e., support from academic counselors, access to tutoring on campus) remained the same for both groups of students.

In InsideTrack’s contracts with participating institutions, schools agreed to provide InsideTrack with data on students’ retention over the duration of the coaching. InsideTrack passed these data to us after anonymizing the data files. There is a conflict of interest here in that InsideTrack has provided the data directly to us. We have no reason to believe that they manipulated or altered the data for the purpose of this study. As we show below, our checks on the data show no anomalies or other cause for alarm.

In Table 1, we report basic descriptive statistics for the control group and the differences (with their corresponding standard errors) for the treatment group. In terms of descriptive characteristics, the profile of students is weighted more toward nontraditional college students. For example, the average age of students is about 31. Only about 25% of students are below the age of 23. Unlike higher education throughout the United States, the sample of students is slightly more male (51%) than female.

Table 1

Descriptive Statistics and Balance Across Lotteries

	Control group mean	Difference for treatment group	Sample size	Number of lotteries with this variable
Female	0.488	0.009 (0.009)	12,525	15
Missing gender	0.091	0.001 (0.002)	13,555	17
Age	30.5	0.123 (0.209)	9,569	8
Missing age	0.294	0.0001 (0.0010)	13,555	17
SAT	886.3	11.01 (16.19)	1,857	4
Missing SAT	0.827	0.001 (0.002)	13,555	17
Living on campus	0.581	.005 (0.017)	1,955	4
HS GPA	2.84	0.008 (0.044)	1,373	2
Missing HS GPA	0.875	0.0002 (0.0002)	13,555	17
Pell grant recipient	0.265	0.003 (0.031)	805	2
Missing pell	0.927	0.000 (0.000)	13,555	17

Note. Standard errors appear in parentheses.

As the fourth column of Table 1 illustrates, the data are somewhat uneven across sites. The most commonly reported variable across sites was gender, which we observed in 15 of the lotteries. Age (8 lotteries), SAT score (4 lotteries), and on-campus living status (4 lotteries) are the next most commonly reported variables.

Random assignment should ensure that our treatment groups are balanced and comparable. As we explained, InsideTrack quasi-randomly divided lists of students provided by the partner schools into two groups. InsideTrack had the same data we have when they did the lottery, so in many cases, the balancing occurred on just one or two student characteristics. Once the lists were divided, the schools chose (using a coin flip) which group received coaching and which group received the control (no additional services) treatment. As mentioned above, InsideTrack balanced the two groups to be similar. The balancing is similar to the strategy which might be used with block randomization where the blocks were generally age and gender (e.g., above age 50 and male, ages 18–19 and female, and so on).

While one might expect some small discrepancies, we should largely observe that there are no significant differences between the control and treatment groups. As shown in Table 1, this is the case. In the sample taken as a whole, there were no significant differences between the coached group and the noncoached group on any of the observable characteristics (gender, age, SAT scores, or on- or off-campus residence). Similarly, these variables were missing in comparable proportions of the coached and noncoached groups; there were no significant differences in the information available for the two groups. Because of our sample sizes, we have sufficient power to identify even small differences in the groups. Hence, our failure to find differences is an affirmation of the randomization.

To further demonstrate the balance of the treatment and control groups, we can also examine the balance of student characteristics by lottery. Table 2 does exactly this. In most cases, we know little about the overall sample; the lotteries differed on the number of observable characteristics recorded (ranging from 1 to 14). For each lottery, we tested the difference between the control and treatment groups. The effectiveness of the randomization holds when examining each lottery individually; of the 73 characteristics compared over the 17 lotteries, only 1 revealed a significant difference between the coached and noncoached groups at the 90% confidence level. Had we used a 95% confidence interval, we would have found no differences in any of the lotteries. Given that InsideTrack used a design that is very similar to block randomization, the precise balance across groups should be expected.

Table 2

Significant Differences in Covariates By Lottery

Lottery	Number of characteristics	Number with significant difference (90%)	% receiving treatment	N in treatment (control)
1. (n = 1,583)	2	0	62.8	994 (589)
2. (n = 1,629)	2	0	67.5	1,099 (530)
3. (n = 1,546)	2	0	54.1	836 (710)
4. (n = 1,552)	2	0	51.4	797 (755)
5. (n = 1,588)	2	0	59.4	944 (644)
6. (n = 552)	3	0	79.9	441 (111)
7. (n = 586)	3	0	84.3	494 (92)
8. (n = 593)	3	0	79.8	473 (120)
9. (n = 974)	9	0	49.8	485 (489)
10. (n = 326)	6	0	49.7	162 (164)
11. (n = 479)	6	0	49.9	239 (240)
12. (n = 400)	2	0	50.0	200 (200)
13. (n = 300)	1	0	50.0	150 (150)
14. (n = 600)	1	0	50.0	300 (300)
15. (n = 221)	3	1	63.3	140 (81)
16. (n = 176)	14	0	39.8	70 (106)
17. (n = 450)	12	0	50.0	225 (225)

Finally, Appendix Figures A1 to A3 graph kernel density estimates of the age distributions, SAT scores, and high school grade point averages of both the treatment and control groups. For each variable, the distributions for control and treatment groups are similar. These similarities validate the randomization, making it possible to identify the effects solely through comparing coached and noncoached groups within each lottery.¹¹

Partner universities also provided data to InsideTrack on student persistence after 6, 12, 18, and 24 months. In some cases, partner institutions provided additional information on students’ degree completion. We only track persistence at the partner colleges and cannot follow students who transfer to another school. Given that many policies are focused on retention at the institutional level, tracking persistence at one school is important for public policies and institutional success.

Empirical Strategy

Because the proposed treatment was administered using randomization, simple comparisons of participants in the treatment and control groups can identify the relative effects of the interventions. We estimate the “intent-to-treat” (ITT) effect using Equation 1:

y_{i j} = δ + β \times {COACH}_{i} + α_{j} \times {Lottery}_{j} + b X_{i} + ε_{i j},

where y is an outcome for individual i who participated in lottery j. COACH represents whether the individual was randomized into the treatment coaching group. We also include lottery fixed effects, and a vector (X) of additional controls such as gender, age, high school GPA, and school type. The outcome of interest is college persistence, measured in 6-month increments from the start of the treatment. Our standard errors control for heteroskedasticity. As we mentioned above, many of our variables are available for one cohort, but not another. In these cases, we include a dummy variable for each variable indicating whether it is missing or not (e.g., a variable for gender missing, a variable for age missing) while substituting either the mean (for continuous variables) or a value of zero (for binary variables) for the variable itself.

Empirical Results

In Table 3, we report our baseline results. Each column focuses on retention, as reported to InsideTrack by the colleges. We look at retention in 6-month increments. In Panel A, we report the baseline differences between coached and uncoached students without any controls except for the lottery fixed effects. In Panel B, we add controls for gender, age, ACT score, high school GPA, degree program, living on campus, PellGrant receipt, prior remediation experience, SAT score, and controls for missing values of covariates. The sample size changes across because of data availability from the individual schools.¹²

Table 3

OLS Estimates of Baseline Treatment Effects on Persistence Over Time

	6-month retention	12-month retention	18-month retention	24-month retention	Completed degree
Control mean	.580	.435	.286	.242	.312
Baseline model
Treatment effect	.052*** (.008)	.053*** (.008)	.043*** (.009)	.034** (.008)	.040* (.024)
Lottery controls	Yes	Yes	Yes	Yes	Yes
n	13,552	13,553	11,149	11,153	1,346
Baseline with covariates
Treatment effect	.051*** (.008)	.052*** (.008)	.042*** (.009)	.033** (.008)	.040* (.024)
Lottery controls	Yes	Yes	Yes	Yes	Yes
n	13,552	13,553	11,149	11,153	1,346

Note. When included, covariates include age, gender, ACT score, high school GPA, SAT score, on- or off-campus residence, receipt of a merit scholarship, Pell Grant awards, math and English remediation, and controls for missing values. Standard errors appear in parentheses. Completed degrees include certificates, associates and bachelor’s degrees.

Significant over 90% CI, **95% CI, and ***99% CI.

The baseline persistence rate after 6 months is 58%. This persistence rate is lower than that of the overall college population, possibly due to the fact that many of these students are older, nontraditional students. In contrast to the uncoached persistence rate of 58%, the retention rate among coached students was 63%. The difference is significant over a 99% confidence interval. The relative effect is about a 9% increase in retention. When we control for covariates, the treatment effect is constant at about 5 percentage points.

In Column 2, we examine 12-month retention. Here the persistence rates for coached and noncoached students were 48.8% and 43.5%, respectively. The treatment effect does not change as we include covariates in Panel B. The estimated effect represents a 12% increase in college retention.

The results after 6 and 12 months occur at a time when, in most cases, the treatment is still active. Coached students during this period are receiving phone calls from their coaches. Columns 3 and 4 show the results after 18 and 24 months. By this point, the coaches are no longer contacting the students. The treatment is over, yet we still find effects. After 18 months, the treatment effect was 4.3 percentage points representing a 15% increase in retention in this sample, and after 24 months, the treatment effect was 3.4 percentage points representing a 14% increase in persistence. These differences are all statistically significant over a 99% confidence interval. Moreover, these results do not change when we control for age, gender, ACT score, high school GPA, SAT score, on- or off-campus residence, receipt of a merit scholarship, Pell Grant awards, math and English remediation.

For a subsample of students (three lottery cohorts), we observe whether the student completed college after the start of the treatment. InsideTrack worked with a variety of students, and degree completion could mean the completion of a certificate, an associate’s degree, or a bachelor’s degree. All three lottery cohorts included in this analysis come from 4-year colleges, though we do not observe whether these are proprietary, nonprofit, or public colleges. Across the three lottery cohorts, the average completion rate among the control group is 31%. The treatment effect is 4 percentage points and is statistically significant over a 90% confidence interval.

These graduation results only strengthen our results on retention. In our analysis in Table 3, we have only included students who are were attending the university after 6, 12, 18, or 24 months. Some students may have completed a degree within the first 6 to 12 months, and these students would not appear to be attending. Our enrollment data did not include these individuals who might have already graduated. If we were to amend our results in Table 3 by redefining persistence as being persistence at Time X or eventual graduation, then the estimated effects become slightly stronger.¹³

The estimates in Table 3 are estimates of the intention to treat. Converting these estimates to estimates of the effect of the treatment on the treated is more difficult. The intention to treat in this case—assignment to coaching—is binary, but the actual treatment reflects a student’s self-selected dosage (i.e., how many meetings the student will allow). It is difficult to measure dosage treatment effects since the counterfactual (i.e., how many meetings would have taken place in the control group had they been treated) is not observable for the control group.

We can make rough estimations of this treatment on the treated effect using an instrumental variables model. In this model, we identify the exogenous portion in the variation in treatment by regressing a dosage metric (e.g., received more than five contacts) on the treatment assignment and using this predicted contact in our treatment model. Using this method, we compare the outcomes for students who received a specific dosage level from coaches while using the randomization as an instrument for having received this level of dosage. For example, if we suppose that the treatment really becomes effective after five contacts, then using the instrumental variables model, the estimated effect of the treatment on the treated after 12 months would be roughly 6.4 percentage points. If we were to suppose that the treatment really becomes effective after 10 contacts, then the estimated effect of the treatment on the treated after 12 months would be roughly 9.9 percentage points. The instrumental variables estimates, as well as the first stage estimates (the effect of randomization on different measures of contact) are presented in Appendix Table A4.

Ideally, we would like to compare the cost efficiency of the measured effects to the effects found in other related services. Unfortunately, we know of no study of a scaled-up student service which provides an estimate of the cost-effectiveness of such services. By far the most researched and popular policy focused on retention is student aid, and compared with prior studies of student aid, the measured effects of coaching on persistence (and completion) are large. For example, Goldrick-Rab, Harris, Benson, and Kelchen (2011) examined a randomized experiment where students were given money for attending college without seeing any impact on persistence. Other studies of persistence found that need-based financial aid can modestly improve college persistence (e.g., Bettinger, 2004; S. M. Dynarski, 2003). These articles note that retention rates increase by 3 percentage points per US$1,000 of aid. In her study of merit-based aid, S. Dynarski (2005) found that state scholarships led to 5 to 11 percentage point increases in college persistence. In the case of the Georgia scholarships, the average expenditure was roughly US$2,500 per year. There is no evidence that the effects persist once students are no longer eligible for aid. In 2004 and 2007, InsideTrack charged about US$500 per semester.¹⁴ Over the course of two semesters, the costs of increased financial aid and coaching are the same. However, the effects are stronger for coaching and show persistence at least 1 year following the end of the treatment.

Robustness

The balance in the randomization and the failure of covariates to reduce the treatment effect suggest that the results are somewhat robust. One worry might be that a single lottery or single year could somehow account for the treatment effects. In Table 4, we estimate treatment effects separately for each lottery. We focus on the 12-month retention rate and the 24-month retention rate.

Table 4

Treatment Effects on Persistence Over Time by Lottery

Lottery	12-month persistence	24-month persistence	Lottery	12-month persistence	24-month persistence
1.	.078***	.020	10.	.052	—
2.	.057**	.039**	11.	.091**	—
3.	.043*	.050**	12.	−.055	—
4.	.050**	.050**	13.	.162***	.054
5.	.040	.029	14.	.054	−.010
6.	.072*	—	15.	.136**	—
7.	.018	.066**	16.	.062	.047
8.	.023	−.017	17.	.000	.058
9.	.058**	—

Significant over 90% CI, **95% CI, and ***99% CI.

Fifteen of the 17 lotteries show positive treatment effects after 12 months (lottery 12 shows a nonsignificant negative effect and lottery 17 shows a nonsignificant null effect). Nine of these observed positive effects are statistically significant at least at the 90% confidence level. The positive treatment effects are somewhat uniform around the average treatment effect of 5 percentage points. Two lotteries show effects in excess of 10 percentage points.

After 24 months, we only observe treatment effects in 11 of the 17 lotteries. Among the treatment effects after 24 months that we observe, four are positive and statistically significant with the maximum observed effect around 6.6 percentage points. Five are positive but not statistically significant; three of these five are larger in magnitude than the average treatment effect across all sites. Two are negative with the lowest observed effect at −1.7 percentage points.

The lesson from Table 4 is that the treatment effects are not arising because of one specific lottery. The observed effects are quite similar across sites. Broadly speaking, the results suggest that the program is having a consistent effect across sites.¹⁵

Another possibility is to check whether there are differences in treatment effects across years. If, for example, InsideTrack were to have different levels of effectiveness in different types of schools, we might expect some differences in treatment effects depending on whether InsideTrack’s client base is similar across years. If these differences are large enough, then 1 year’s impacts might explain the overall effects, but as we show in Table 5, the effects are balanced across years. Except in one case (2004 cohorts after 24 months), the treatment effects are all positive and significant for both samples across the different time horizons. The effects appear somewhat smaller in the 2007 cohort although the differences are not statistically different except in the estimates of retention after 6 months. The effects seem to be somewhat balanced over time suggesting that the program’s effects are not being driven by 1 year.

Table 5

Treatment Effect by Year

	6-month retention	12-month retention	18-month retention	24-month retention
Control Mean	.617	.479	.381	.356
2004 Lotteries
Treatment effect	.088*** (.020)	.070*** (.020)	.068*** (.021)	.030 (.020)
Covariates	Yes	Yes	Yes	Yes
n	1,774	1,745	1,520	1,524
2007 Lotteries
Control Mean	.573	.426	.265	.217
Treatment effect	.044*** (.008)	.049*** (.009)	.037*** (.010)	.034*** (.009)
Covariates	Yes	Yes	Yes	Yes
n	11,808	11,808	9,629	9,629

Significant over 90% CI, **95% CI, and ***99% CI.

Heterogeneity in Treatment Effects

In Table 6, we investigate whether the effects differ for males and females. In Panel A, we report the effects for females, and in Panel B, we report the effects for males. After 6 months, the treatment effects were 2.5 percentage points for females and 6.1 percentage points for males. The difference is statistically significant. After 12 months, the treatment effects are 4.5 and 5.4 percentage points for females and males, respectively. After 18 months, the treatment effects are 3.3 and 4.7 percentage points for females and males, respectively. The impacts of coaching on persistence are not significantly different across genders after 12 or 18 months. The impacts after 24 months are 2.2 and 4.7 percentage points for females and males, respectively. These differences are statistically significant.

Table 6

Treatment Effects on Retention Over Time by Gender

	6-month retention	12-month retention	18-month retention	24-month retention
Females
Control mean	.661	.497	.346	.299
Treatment effect (SE)	.025** (.012)	.045*** (.013)	.033** (.014)	.022* (.013)
n	6,045	6,045	4,740	4,744
Males
Control mean	.536	.403	.260	.215
Treatment effect	.061*** (.012)	.054*** (.012)	.047*** (.012)	.047*** (.011)
n	6,479	6,480	5,457	5,457

Note. When included, covariates include age, ACT score, high school GPA, SAT score, on- or off-campus residence, receipt of a merit scholarship, Pell Grant awards, math and English remediation, and controls for missing values. Regressions include fixed effects for lottery. Standard errors appear in parentheses.

Significant over 90% CI, **95% CI, and ***99% CI.

The difference between the noncoached and coached groups was always greater for males than for females. While males persisted at rates lower than their female peers, student coaching had larger effects for males. Two of the four differences in treatment effects were statistically significant. Male completion rates typically lag behind females and have been somewhat insensitive to interventions. There appears to be some evidence that the effect is larger for males, suggesting that this type of student coaching could reduce gender gaps in completion.

In Table 7, we examine the effects of the program for different age groups. We find that the estimated treatment effects have similar magnitudes across different age groups. After 6 months, the treatment effects are about 3.7 percentage points for students 30 and below and about 6.2 percentage points for students older than 30. The treatment effects are 5.2 and 4.4 percentage points, respectively, after 12 months. After 18 months, the treatment effects are 4.0 and 3.4 percentage points for students 30 and below and above 30, respectively. After 24 months, the treatment effects are 4.1 and 2.4 percentage points, respectively. All of the estimates are positive and only the treatment effect on older students after 24 months is statistically insignificant.

Table 7

Treatment Effects on Retention Over Time by Age

	6-month retention	12-month retention	18-month retention	24-month retention
Students 30 or below
Control Mean	.600	.438	.234	.184
Treatment effect (SE)	.037*** (.010)	.052*** (.011)	.040*** (.012)	.041*** (.011)
n	7,850	7,850	5,671	5,671
Students above 30
Control Mean	.513	.400	.311	.266
Treatment effect	.062*** (.017)	.044*** (.017)	.034** (.016)	.024 (.015)
n	3,958	3,958	3,958	3,958

Note. When included, covariates include age, gender, ACT score, high school GPA, SAT score, on- or off-campus residence, receipt of a merit scholarship, Pell Grant awards, math and English remediation, and controls for missing values. Regressions include fixed effects for lottery. Standard errors appear in parentheses.

Significant over 90% CI, **95% CI, and ***99% CI.

Conclusion

In the typical economic model of higher education, we assume that students know how to behave. We assume that they know how to study, how to prioritize, and how to plan. However, given what we know about rates of college persistence, this is an assumption that should be called into question. Across all sectors of higher education, more needs to be known about how to increase college persistence. Literature in economics, education, and sociology suggests that student coaching may be one way to help students succeed in college.

We find exactly this. While coaching was taking place during the 1st year, coached students were about 5 percentage points more likely to persist in college. This represents a 9% to 12% increase in retention. We also find that the effect of coaching on persistence does not disappear after the treatment. Coached students were 3 to 4 percentage points more likely to persist after 18 months and 24 months. These represented roughly a 15% increase in college retention among our sample. All of these effects were statistically significant. For the three campuses for which we have degree completion data, we find that coached students had graduation rates 4 percentage points higher than uncoached students after 4 years.

Given that many previous studies have found results that dissipate after the end of treatment, the persistence of these effects merits further study.¹⁶ Because the mechanisms through which InsideTrack coaching affects student behavior are not well understood, explaining why these effects persist while others do not is based on informed speculation. It could be that this type of conversation, focusing on personal struggles and obstacles, lends itself to lasting changes. Similarly, the proactive nature of the intervention might have impact on students who would not respond to other kinds of support. This question of the persistence of these effects needs more investigation.

These strong results that point to the potential of student coaching are bolstered by a favorable financial analysis. When we compared the costs and benefits of student coaching to programs that target financial aid, we find that student coaching leads to larger effects than financial aid and are much less costly to implement. The persistence of the effects after the treatment period and impact on completion only increases the relative cost-effectiveness.

The results also help us to better understand recent interventions which included a counseling component. For example, in the Opening Doors initiative, students were provided financial incentives and counseling. While economists have stressed the incentives as being important in the observed effects, the regular contact from a college counselor may have been the operative mechanism by which effects occurred.

In addition, Angrist et al. (2009) found that students who had access to incentives and counseling had higher academic performance in college. They, however, did not find any effect of counseling by itself. There are two key differences between InsideTrack and the intervention studied by Angrist et al. One is that the counseling was voluntary in the treatment studied by Angrist et al. Students had to seek out the counselors. In the case of InsideTrack, the coaching remains voluntary but the counselors attempt to find the students and provide both proactive and continuing outreach to the students. Another key difference is that the advisers in the Angrist et al. study were trained upper class students, not full-time coaches, and were not supported by the process and technology infrastructure that InsideTrack utilizes.

Our study is one of the first studies to use random assignment to evaluate the effects of student coaching and additional study is warranted. Research in other educational evaluations (e.g., Bettinger & Long, 2009; Dee, 2005) suggests that the traits of high school and college instructors influence student outcomes. It would be interesting to know whether there are specific characteristics of the college coaches which increase their efficacy. We also do not know the specific types of coaching services and the specific actions of coaches which are most effective in motivating students.

Further study can also reveal how student coaching might affect other student populations. Our study includes public, private, and proprietary institutions, and it includes a broad range of students including students who are pursuing associate’s degrees, and bachelor’s degrees. While the sample with whom InsideTrack works represents the broad range of college students, we cannot observe all of the unique characteristics of students in our samples, and even if we could, we do not have enough power to identify the effects on important subgroups. We do have power to identify the effects on males and females and younger and older students. We find that the effects do not vary by age; the effects on older students and younger students are similar. While the effects are positive for both males and females, we do find some evidence that the effect is larger for males. As such, it could reduce some of the disparities in college completion that exist by gender.

In an era when college retention is receiving increased attention in public policy and the media, our article provides strong evidence that college coaching is one strategy that can improve retention and graduation rates.

Footnotes

Appendix

Table A4

Instrumental Variables Estimates of the Effect of Contact With Coach on Retention

	Number of contacts	Number of meetings	At least 5 contacts	At least 10 contacts	Contact within first week of first term
A. First stage	First stage estimates
Randomization	11.055***	7.336***	0.771***	0.464***	0.644***
	(0.122)	(0.098)	(0.006)	(0.007)	(0.007)
B. IV estimates	Dependent variable = 12 month retention
	Control mean = 0.426 (0.495)
Number of contacts	0.004***
	(0.001)
Number of meetings		0.006***
		(0.001)
At least 5 contacts			0.064***
			(0.012)
At least 10 contacts				0.099***
				(0.018)
Contact within first week of first term					0.073***
					(0.013)
n	11,808	11,808	11,808	11,808	11,808

Significant over 90% CI, **95% CI, and ***99% CI.

Authors’ Note

The authors thank Brent Evans, Eric Taylor, Jon Valant, and three anonymous reviewers for helpful comments. The study described in this article was an independent analysis of data provided by InsideTrack; the research team has no financial relationship with InsideTrack.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

Authors

ERIC P. BETTINGER is an associate professor in the Stanford University School of Education. His research interests include economics of education, student success and completion in college, teacher characteristics and student success in college, and effects of voucher programs on both academic and nonacademic outcomes.

RACHEL B. BAKER is a doctoral candidate at the Stanford Graduate School of Education. Her research focuses on college sorting, student success and completion in college, and college program choice.

References

Adelman

Gonzalez

(2006). The toolbox revisited: Paths to degree completion from high school through. Washington, DC: U.S. Department of Education.

Angrist

Lang

Oreopoulos

(2009). Incentives and services for college achievement: Evidence from a randomized trial. American Economic Journal: Applied Economics, 1(1), 136–163.

Bettinger

(2004). How financial aid affects persistence. In Hoxby

(Ed.), College choices: The economics of where to go, when to go, and how to pay for it (pp. 207–238). Chicago, IL: The University of Chicago Press.

Bettinger

Long

(2009). Addressing the needs of under-prepared students in higher education: Does college remediation work? Journal of Human Resources, 44, 736–771.

Bettinger

Long

Oreopoulos

Sanbonmatsu

(2012). The role of application assistance and information in college decisions: Results from the H&R block Fafsa experiment. The Quarterly Journal of Economics, 127, 1205–1242.

Bloom

Sommo

(2005). Building learning communities early results from the opening doors demonstration at Kingsborough community college. New York, NY: MDRC.

Brock

Richburg-Hayes

(2006). Paying for persistence. Early results of a Louisiana scholarship program for low-income parents attending community college. New York, NY: MDRC.

Calcagno

J. C.

Long

B. T.

(2008, July). The impact of postsecondary remediation using a regression discontinuity approach: Addressing endogenous sorting and noncompliance (NBER Working Paper No. 14194). Cambridge, MA: National Bureau of Economic Research

Carrell

Sacerdote

(2012, July). Late interventions matter too: The case of college coaching in New Hampshire. UC Davis mimeo.

10.

Dee

T. S.

(2005). A teacher like me: Does race, ethnicity, or gender matter? American Economic Review, 95(2), 158–165.

11.

Deil-Amen

Rosenbaum

J. E.

(2003). The social prerequisites of success: Can college structure reduce the need for social know-how? The Annals of the American Academy of Political and Social Science, 586(1), 120–143.

12.

Deming

Dynarski

(2008). The lengthening of childhood. Journal of Economic Perspectives, 22(3), 71–92.

13.

Dynarski

(2005, April). Finishing college: The role of schooling costs in degree attainment (Working paper). Cambridge, MA: Harvard University.

14.

Dynarski

S. M.

(2003). Does aid matter? Measuring the effect of student aid on college attendance and completion. American Economic Review, v93(1), 279–288.

15.

Gallagher

(2010). National survey of counseling center directors. American College Counseling Association. Retrieved from http://www.iacsinc.org/NSCCD%202010.pdf

16.

Gansemer-Topf

A. M.

Schuh

J. H.

(2006). Institutional selectivity and institutional expenditures: Examining organizational factors that contribute to retention and graduation. Research in Higher Education, 47, 613–642.

17.

Goldrick-Rab

(2010). Challenges and opportunities for improving community college student success. Review of Educational Research, 80, 437–469.

18.

Goldrick-Rab

Harris

Benson

Kelchen

. (2011, July). Conditional cash transfers and college persistence: Evidence from a randomized need-based grant program (Institute for Research on Poverty Working Paper. 1393–11).

19.

Kern

C. W.

Fagley

N. S.

Miller

P. M.

(1998). Correlates of college retention and GPA: Learning and study strategies, testwiseness, attitudes, and ACT. Journal of College Counseling, 1, 26–34.

20.

Long

B. T.

(2008, April). What is known about the impact of financial aid? Implications for policy (An NCPR Working Paper No 52). New York, NY: National Center for Postsecondary Research.

21.

Obama

(2009, February). Address to Joint Session of Congress. Speech presented in Washington, DC.

22.

OECD. (2007). Tertiary education graduation rates—education: Key tables from OECD. Retrieved from http://www.oecd-ilibrary.org/education/tertiary-education-graduation-rates_20755120-table1

23.

Robbins

S. B.

Lauver

Davis

Langley

Carlstrom

(2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, 261–288.

24.

Scott-Clayton

(2011, January). The shapeless river: Does a lack of structure inhibit students’ progress at community colleges (CCRC Working Paper No. 25). New York, NY: Columbia University.

25.

Scrivener

Weiss

M. J.

(2009). More guidance, better results? New York. NY: MDRC.

26.

Sommo

Mayer

A. K.

Rudd

Cullinan

(2012). Commencement day: Six-year effects of a freshman learning community program at Kingsborough community college. New York, NY: MDRC.

27.

Thaler

R. H.

Sunstein

C. R.

(2009). Nudge: Improving decisions about health, wealth, and happiness. New York, NY: Penguin Books.

28.

Tinto

(1975). Dropout from higher education: A theoretical synthesis of recent research. Review of educational research, 45, 89–125.

29.

Tinto

(1998). Colleges as communities: Taking research on student persistence seriously. The Review of Higher Education, 21(2), 167–177.

30.

Turner

S. E.

(2004). Going to college and finishing college. Explaining different educational outcomes. In Hoxby

(Ed.), College choices: The economics of where to go, when to go, and how to pay for it (pp. 13–62). Chicago, IL: University of Chicago Press.

31.

Zeidenberg

Jenkins

Calcagno

J. C.

(2007, June). Do student success courses actually help community college students succeed? (CCRC Brief No 36). New York, NY: Columbia University.