Increasing Community College Graduation Rates: A Synthesis of Findings on the ASAP Model From Six Colleges Across Two States

Abstract

This paper presents new estimates of the effects of the City University of New York’s (CUNY’s) Accelerated Study in Associate Programs (ASAP) model, evaluated using a randomized controlled trial first in New York and later through a replication in Ohio. It presents longer-term effects of CUNY ASAP in New York, showing that the program’s effect on associate’s degree receipt persisted through 8 years and likely represents a permanent increase in degree receipt. It also presents an analysis from the pooled study samples in New York and Ohio. The findings indicate that the program had consistent effects on degree receipt across the two states but also for somewhat different levels of service contrast, such as the number of additional advising visits.

Keywords

community colleges program evaluation postsecondary education experimental research multi-site studies

Postsecondary education provides an important stepping-stone to well-paying jobs and the middle class for many low-income individuals. The nation’s community colleges play a vital part in that effort, serving about 40% of all postsecondary students and a disproportionate number of students from families with low-incomes (Ginder et al., 2019, Table 3). They are accessible, typically with open admission policies, and they are, on average, much less expensive than 4-year colleges or private 2-year colleges.

Unfortunately, however, graduation rates from community colleges are low. Among first-time, full-time, degree-seeking students entering public 2-year schools, only 25% graduate within 3 years (McFarland et al., 2019). Policymakers and higher education administrators across the nation are striving to increase those rates. Such increases would benefit students and society in an economy that increasingly demands a highly educated workforce.

There have been numerous reforms designed to increased students’ persistence in and completion of community college. These include financial supports, enhanced advising, learning communities, pedagogical reforms, and messaging campaigns (see, for example, Angrist et al., 2009; Attewell et al., 2012; Bettinger & Baker, 2014; Bettinger et al., 2012; Castleman & Page, 2015; Goldrick-Rab et al., 2012; Miller et al., 2020; Patel et al., 2013; Richburg-Hayes et al., 2009; Scrivener et al., 2015; Scrivener & Weiss, 2009; Weiss et al., 2019). These reforms often include one or two changes to business as usual, such as increased financial aid or increased aid combined with enhanced advising, and typically last for one or two semesters. When evaluated, many of these reforms led to modest increases in student progress and, occasionally, modest effects on graduation.

Among the tested policies, one program model stands out: the City University of New York’s (CUNY’s) Accelerated Study in Associate Programs (ASAP). Designed and implemented by CUNY to help more students graduate and to help them graduate more quickly, ASAP is a comprehensive, integrated, 3-year student support program serving community college students from families with low-incomes. The program requires students to attend college full-time and encourages them to take remedial courses early, enroll in the summer, and graduate within 3 years. Students receive frequent advising, enhanced career services, and additional tutoring. Three forms of financial supports alleviate students’ financial need: a tuition waiver, free use of textbooks, and a monthly transportation benefit, which is contingent on participation in key program services. Students can also enroll in courses with other program students in convenient schedules. ASAP was evaluated at three CUNY colleges using a randomized controlled trial that started in 2009. The evaluation found that the program led to a substantial increase in graduation rates—increasing 3-year graduation rates by 18 percentage points. This exceptional result led to the replication of the program by three community colleges (at four campuses) in Ohio.

Three-year findings from the test in Ohio also showed a significant increase, of 16 percentage points, in graduation rates (Miller et al., 2020). This finding was very encouraging, given that the replication of findings from a given test of an educational intervention is the exception rather than the norm (see, e.g., Camerer et al., 2018). The findings from Ohio showed not only that the model could be implemented with a high degree of fidelity in another setting but that it could have similarly positive and large effects.

This paper extends the earlier results in several ways. First, it presents an update of the findings from CUNY ASAP through 8 years. An earlier paper documented effects through Year 6 and found that the impact on degree receipt was still large, at 10 percentage points. However, this impact had diminished relative to Year 3, as the control group “caught up” somewhat to the program group (Weiss et al., 2019). It was expected that degree receipt would increase to some extent for students in the control group, mirroring national trends in which many graduates take longer than 3 years to earn a degree. By tracking effects for an additional 2 years, the paper examines whether the effects persist or diminish further. The longer follow-up period also provides a more comprehensive look at the program’s effects on enrollment in 4-year colleges and the receipt of bachelor’s degrees.

Second, the paper synthesizes the 3-year findings from CUNY and the Ohio replication. For the first time, this study pools the data from the original CUNY evaluation and the Ohio replication. The pooled analysis supports the precise estimation of a broader treatment effect (the mean effect across six colleges from two states) of the ASAP model on 3-year graduation rates. Also, the pooled sample is better positioned than the individual studies to examine subgroup effects, or effects for different types of students.

Finally, the paper examines effects across the states and across the individual colleges. Although exploratory and based on only two states and seven campuses, this analysis provides an important initial look into how much implementation differences and differences in service contrast affect program impacts. In particular, the paper examines whether variation exists across places in effects on degree receipt and, if so, whether it is associated with differences in students’ use of key services.

In sum, the findings show that ASAP’s effect on associates’ degree receipt persisted through 8 years, although it did not lead to a long-term increase in bachelors’ degree receipt. The effects on associates’ degrees indicate that the program, in addition to helping some students graduate faster, also helped some students graduate who would never have done so without the additional support. The effects across both states through 3 years provide further evidence of the program’s effectiveness, for students in general and for different types of students. Finally, the effects by state and by campus suggest that there is a positive association between service contrast, or effects on advising visits, for example, and degree impacts. However, the variation in service contrast across the campuses suggest that, beyond some level, there may be diminishing returns. In other words, increasing advising by 15 visits per semester may not lead to much larger effects on degree receipt than increasing advising by eight visits per semester. Because the model costs more than the usual services offered to students, colleges will need to compare the resources needed to generate larger increases in service contrast with the additional degrees it generates. More broadly, the consistency of findings across the states and colleges suggest that the results are likely to generalize beyond these students and colleges, furthering the idea of ASAP as a national model to increase educational attainment.

Background

Researchers have identified a number of student-level challenges and institutional practices that underlie the low rates of persistence and completion at community colleges (for overviews, see Baum et al., 2013; Braxton, 2002; Calcagno et al., 2008). These factors include, but are not limited to, financial challenges, such as the costs of attending college; a shortage and underutilization of student support services; insufficient preparation for college-level work; and the competing demands of work, family, and school. These challenges are even more pronounced for students from families with low-incomes and students’ with insufficient preparation for college-level work due to powerful negative systemic influences on students’ prior educational preparation (e.g., absence of academically rigorous course offerings in the high school) who are less likely than other students to stay on track and graduate (Attewell et al., 2012; Crosta, 2014; Klempin, 2014).

First, although tuition and fees at community colleges are about one-third the cost of public 4-year colleges and universities (Baum & Ma, 2014), they still constitute a substantial investment for many students from families with low-incomes, especially when opportunity costs are considered. In addition, financial aid sometimes does not cover the full cost of attendance, leaving students to struggle to afford tuition, transportation, or textbooks. Moreover, students from families with low-incomes must negotiate a complex financial aid system to receive aid (Bettinger et al., 2012).

Second, many students’ placement test scores suggest that they are not academically prepared for college, as indicated by the fact that nearly 60% enroll in at least one developmental (or remedial) reading, writing, or mathematics course during community college (Bailey, 2009). It is worth noting, however, that there is evidence of extensive misplacement into developmental education (e.g., see Scott-Clayton et al., 2014). Placement into developmental courses is negatively associated with student persistence and success, as students referred to developmental education frequently drop out of their classes or leave college entirely, and those who remain make progress relatively slowly (Adelman, 2004; Attewell et al., 2006). Only 13% of community college students with developmental needs earn a certificate or degree within 3 years (U.S. Department of Education, n.d.).

Upon entering college, many students also struggle to navigate an unfamiliar college environment and need help figuring out which courses to take and in what order, how to register for classes and apply for financial aid, and what resources are available to help make the transition to college (Bailey et al., 2015). Community colleges are rarely able to support the kind of personalized and timely advising that students need and that students often get at selective 4-year colleges (Bound & Turner, 2007). For example, the National Academic Advising Association (2011) estimates that the median caseload of an adviser at public 2-year colleges is 441 students per adviser (compared with 260 students per adviser at public 4-year colleges), which limits the amount of advising students receive. Students who do not have access to an informed adviser may neglect to register for a required course, fail to secure the maximum financial aid for which they qualify, or make other missteps that could negatively impact their progress through college. Community college students, most of whom commute, have also been identified as less likely to identify with the college community than traditional undergraduates at 4-year institutions (Tinto, 1997).

Taken together, the numerous systemic issues, institutional practices, and student-level barriers that students from families with low-incomes face contribute to their low completion rates. While there is causal evidence about the efficacy of postsecondary interventions across the literature, most policies have had, at best, only modest effects.

For example, several studies have produced experimental evidence that financial aid-related reforms can positively, although often modestly, influence students’ academic progress (for example, see Angrist et al., 2009, 2010, 2014, 2016; Bettinger et al., 2012; Cohodes & Goodman, 2014; Deming & Dynarski, 2010; Goldrick-Rab et al., 2016; Mayer et al., 2015). As one example, MDRC evaluated the effectiveness of performance-based scholarships in a variety of college settings and with varying incentive structures and found a modest three percentage point impact on degrees earned through five years (Mayer et al., 2015). Goldrick-Rab et al. (2016) found that additional scholarship aid via the Wisconsin Scholar’s Grant led to an increase in second-year enrollment in college of about 3 to 4 percentage points.

Colleges have also tried many ways to better integrate new students. Learning communities, for example, co-enroll a cohort of students in two or more courses to foster connections among students and enable instructors to collaborate and integrate coursework. As another example, student success courses, which vary in their content, often aim to improve study skills, change students’ behaviors and attitudes, including increasing their awareness of their and others’ emotions, understanding their own learning styles, improving time management skills, and recognizing their responsibility for their own learning (Engstrom & Tinto, 2008; Rutschow et al., 2012; Weiss, Mayer, et al., 2015; Weiss, Visher, et al., 2015). Other strategies include interventions aimed at helping community college students stay enrolled continuously, including in summer sessions (Attewell & Douglas 2014; Attewell & Jang, 2013; Weiss, 2019). In one such study, one group of students received a messaging campaign to encourage summer enrollment and another group received a messaging campaign plus a last-dollar tuition grant. Experimental evaluations of these approaches indicate that some have produced modest improvements in academic outcomes.

As another example, a fairly robust experimental literature on enhanced academic advising interventions finds positive, although often modest, effects on students’ academic outcomes using a variety of modes of advising (for examples, see, Avery et al., 2014; Bailey et al., 2016; Barr & Castleman, 2017; Bettinger & Baker, 2014; Carrell & Sacerdote, 2013, 2017; Evans et al., 2017; Scrivener & Weiss, 2009).

Two more recently studied programs provide more comprehensive and longer-term supports for students and have led to larger effects. One study is of the Early College High School initiative, a model that offers dual high school and college enrollment and a variety of supports to help students move into and complete college. Students in the program were substantially more likely to earn an associate’s degree than students not in the program (Berger et al., 2014). A second example comes from an evaluation of the Stay the Course program at a community college in Texas, which offered 3 years of ongoing comprehensive case management, provided by trained social workers, along with access to emergency financial assistance. The evaluation found large effects on degree receipt among the 20% of the study sample that took up the program (treatment-on-the-treated effects), although the effects were primarily for women (Evans et al., 2017).¹ The effects among program participants contain substantial estimation error, given the sample size and program participation rate.

Thus, the weight of the evidence suggests that short-term, light-touch interventions are typically not enough to substantially improve longer-term outcomes, such as a degree receipt, and that more comprehensive interventions that address multiple barriers are needed.² ASAP is one such program.

The ASAP Model and Implementation

The ASAP Model

Developed by the CUNY, ASAP is a comprehensive program that provides students with up to 3 years of financial and academic support and other support services to address multiple barriers to success, with the goal of helping more students graduate within 3 years. Along with those supports and services, however, is an obligation, since students are required to attend full-time and to participate in key program services. The original CUNY ASAP model included the following components.

Student Services

Students receive comprehensive advisement from an adviser with a small caseload, career information from a career and employment services staff member, and tutoring services separate from the usual college tutoring services. Each of these services is provided by ASAP-dedicated staff who work only with students in the program. ASAP advisers, who have caseloads of 60 to 80 students,³ provide support on a wide variety of academic and personal topics, including academic planning, balancing school with other responsibilities, accessing campus services, interacting with professors, staying on track to graduate, and dealing with personal issues. Students are required to meet with their adviser twice per month and to meet with careers services staff once per semester. Students in remedial courses or on academic probation are required to participate in tutoring once per week.

Course Enrollment

Students may enroll in blocked courses, or courses scheduled back-to-back, in their first year, with two goals. First, is minimizing the time required on campus so that students, many of whom work, can enroll full-time. Second, is ensuring ASAP students could take some of their classes with other ASAP students, fostering a sense of community. During the first few semesters, students also enroll in an ASAP seminar, covering topics such as goal-setting, study skills, and academic planning.⁴ Students can also register for courses early so that they can create convenient schedules and get seats in the courses they need.

Financial Supports

Students receive a tuition waiver that fills any gap between federal and state financial aid and college tuition and fees. Students also receive free use of textbooks and free MetroCards for use on public transportation.

Requirements and Messages

Students are required to attend college full-time during the fall and spring semesters and are encouraged to take courses during the winter and summer sessions. ASAP staff also encouraged students to complete developmental education early and to graduate within 3 years.

When the model was replicated in Ohio, some adjustments were made to fit the local context and to align with the evolved version of the ASAP model that CUNY was operating at the time. For example, CUNY offered program group students an ASAP-only student success seminar in their first few semesters, whereas the Ohio program did not create a special seminar course for program group students, but enrolled program group students into specific sections of existing student success courses. The Ohio program also did not offer monthly metro cards to students (worth over $100 in New York), given the lower use of public transportation, but instead provided $50 gas/grocery gift cards to participating students. In addition, CUNY provided textbooks free of charge to participating students. Although one Ohio college provided free textbooks, two of the colleges provided students with vouchers of up to $300 per semester to cover the cost of textbooks. Finally, the Ohio programs used a triage model of advising, in which the advising requirements differed depending on the students’ academic standing.⁵ Students in the high-need group were required to meet with their assigned adviser twice per month, while other students were required to meet less often, although usually at least once per month.

Finally, both programs used a Management Information System, created specifically for the program, to track student outcomes, participation, and progress. CUNY provided support to staff at colleges in both sites on how to track and use these data for monitoring and improving program outcomes.

Participating Colleges

ASAP was tested at 3 of the 24 colleges in the CUNY system. Each serving from 15,000 to 20,000 students, they are the largest of CUNY’s community colleges and were willing to participate in a randomized controlled trial, in which eligible students would be randomly assigned to a program group eligible to participate in ASAP or a control group eligible for the college’s standard services.

The ASAP demonstration in Ohio was tested at 3 of the state’s 23 community colleges. One of the colleges serves about 20,000 students, while the other 2 are smaller, at 9,000 to 11,000 students. These colleges were selected primarily based on administrators’ support for the program, their willingness to take part in a random assignment evaluation, and their capacity to reach the desired sample size goals. See Appendix Table S1 for selected characteristics of the 6 colleges.

Implementation

To ensure strong implementation and fidelity, the CUNY Office of Academic Affairs, which administered the program along with the participating colleges, monitored the implementation of the program at CUNY. It also provided technical assistance to the Ohio colleges, leading a workshop on the program model, helping the colleges develop action plans for implementation, and providing regular and continuing direct technical assistance and training in the ways each program component could be implemented. Implementation research was conducted periodically throughout the 3-year program period in each study. Overall, ASAP was well implemented in both New York and Ohio, and the program created a substantial service differential between students in ASAP and those eligible for the colleges’ usual services.

Figure 1 presents students’ self-reported use of three key program services from a survey administered to all study participants in both states 1 year after program entry. Panel A presents effects on incidence, or whether the student ever used a particular service during the first year. Panel B presents effects on the intensity of use, or the number of times the student used the services, by semester.

Figure 1.

(A) Ever received key service. (B) Number of times students received each key service.

First, consider advising. The data in panel A show that the program led to a moderate increase in the percent of students who ever met with an advisor, in part because most students meet with an advisor at least once. The notable difference is in the intensity of advising, where program group students met with an adviser on average 15 times during the second semester, compared with 3 times for control group students.

For tutoring and career services, the program led to larger increases in incidence, in part because fewer students use these services in the absence of the program. The program increased the number of students who ever used tutoring by 24 percentage points, for example, and program group students participated in nine sessions on average during the second semester, compared with three sessions for students in the control group. Note that these averages are for the entire program and control groups, including zeros for those who never participated. Thus, the number of tutoring sessions among those who ever used tutoring is much larger than shown in the figure.

In terms of financial assistance (not shown in figure), most students across both states reported receiving assistance with the cost of textbooks, and most received the monthly incentive (gift cards in Ohio and transportation cards in New York). Few students in either state received the tuition waiver, given that federal and state aid typically covered the cost of tuition. Students in the program group were also more likely than their control group counterparts to receive messages from advisers and other staff on the importance of attending full-time and graduating within 3 years.

Finally, blocked scheduling was implemented only partially. In New York, students were offered blocked courses at two of the colleges and were encouraged to take at least one of the courses in the block if they could not take the full set. Transcript data show that, although only about a third of program group students enrolled in a full block of courses, a majority of them were enrolled in one or more courses with four or more other ASAP students. Blocked scheduling was not formally implemented in Ohio given the difficulty of identifying overlapping courses for large groups of students. As discussed below, eligibility for the study in Ohio was expanded to include students with up to 24 earned credits, meaning that some of them had already completed required courses. Instead, advisers informally tried to guide program group students into courses with each other. Transcript data indicate that about half of the program group students in Ohio enrolled in one or more courses with at least four other program group students.

In sum, at noted earlier, the ASAP model was well implemented in both New York and Ohio. Although the Ohio program made minor modification to the model to fit local context, it contained the key elements of ASAP. In addition, even though the Ohio programs did not implement formal blocked scheduling and it was only partially implemented in CUNY, any effects of ASAP are more likely attributable to its other program components.

The Evaluation

Participating Students

Students were eligible for the program and study if they met several criteria: had family income below 200% of the federal poverty level or were eligible for a Pell grant, were new to college or had earned fewer than 12 credits in CUNY and 24 credits in Ohio, were willing to attend college full-time, and were in a major that could be completed within 3 years.⁶ The two studies differed with respect to developmental education requirements. For CUNY, the evaluation only included students who had one or two developmental education requirements, although a separate quasi-experimental evaluation conducted by CUNY staff included students without these requirements (Strumbos & Kolenovic, 2017). In Ohio, in contrast, eligibility was extended to students with or without development education requirements at study entry. Both programs excluded students with extensive (more than two) developmental education requirements, given that they were unlikely to earn a degree within 3 years.

Students at CUNY were recruited for the study and randomly assigned just prior to the spring 2010 semester and the fall 2010 semester. Students in Ohio were recruited and randomly assigned just prior to the spring 2015, fall 2015, and spring 2016 semesters. The study samples include 896 students for CUNY and 1,501 students for Ohio.⁷

Table 1 reports selected characteristics of the sample at baseline and, for comparison, characteristics of community college students nationwide.⁸ First, consider the two states. The students in the Ohio study differed from those in New York in several ways. For example, more Ohio sample students were nontraditional students, for example, they were older or had children (although they were less likely to work full-time, another marker of nontraditional status). The New York sample also includes more Hispanic students and fewer white students than the Ohio sample, reflecting in part the differences in the demographic makeup of the participating cities. Fewer Ohio sample students had developmental requirements than the CUNY sample students, as expected given different eligibility criteria for study entry in each state.

Table 1

Sample Characteristics for the CUNY ASAP, Ohio Programs, Pooled Samples, and National Population of Community College Students

Characteristic	CUNY ASAP	Ohio Programs	Pooled	National
Characteristic	Sample	Sample	Sample	Sample
Gender (%)
Male	38	36	37	44
Female	62	64	63	56
Age (%)
19 or younger	57	47	51	-
20–23	22	22	22	-
24 or older	20	31	27	44
Average age (years)	21	23	23	28
Race/ethnicity^a (%)
Hispanic	44	10	22	19
White	10	46	32	56
Black	34	35	35	16
Other^b	12	10	11	9
Living with parents (%)	74	58	64	-
Has children (%)	15	27	23	32
First person in the family to attend college (%)	30	34	33	-
Diplomas/degrees earned (%)^c
High school diploma	74	87	82
General Education Development (GED) certificate	21	12	15
Currently employed (%)	31	60	49	69
Currently employed full time (%)	7	16	12	33
Nontraditional student^d (%)	36	47	43	-
Has developmental education requirements (%)	98	74	83	68
Sample size	896	1,501	2,397

Source. Calculations using baseline information form data and placement test data from the CUNY Institutional Research Database (IRDB) and the ASAP Demonstration study colleges; IPEDS; and Stats in Brief, U.S. Department of Education, February 2017.

Note. Italics indicate statistics calculated only for a subset of respondents. Distributions may not add to 100% because of rounding.

Respondents who said they were Hispanic and chose a race are included only in the “Hispanic” category. ^bThe “Other” category includes Asian/Pacific Islander, Native American/Alaska Native, multiracial, and other races and ethnicities. ^cDistributions may not add to 100% because categories are not mutually exclusive. A small percent of sample members are excluded from these measures because they earned a certificate, technical or other degree and we are unable to determine if they had earned a GED or High School diploma. ^dNontraditional students are defined as those who were 24 years old or older, worked 35 or more hours per week, had children, or did not receive a high school diploma and were not enrolled in high school at the time of random assignment. Students are listed as nontraditional if they fit any of these characteristics.

The pooled sample is shown in the third column. The pooled sample is racially diverse, with no racial majority. Just over half of the sample self-identified as Black or Hispanic. Fifty-one percent of the students were of “traditional” college age (aged 19 or younger), and just over 40% were nontraditional students.

The students participating in the evaluation look roughly similar to community college students nationwide, with some exceptions. In terms of similarities, the majorities of the pooled and national samples are women, 20% to 30% have children, many are working, and a large majority have developmental education requirements. Students nationwide, however, are older than the study sample and more likely to work full-time. The program’s eligibility requirements may have led to these differences, given that the full-time attendance requirement may have discouraged full-time workers, who tend to be older students, from enrolling in the study.⁹

Data and Methods

We use data from several sources. First, students in the studies in both states filled out a baseline survey prior to random assignment, providing demographic and other information. Data on credit and degree completion at the participating colleges and at colleges within the broader CUNY and Ohio systems were obtained through transcript data provided by CUNY and the three participating Ohio colleges. Data on enrollment and degree receipt were also obtained from the National Student Clearinghouse, which covers students’ enrollments in nearly all postsecondary institutions throughout the United States, not just the participating colleges.

Finally, surveys were administered to study participants in both sites approximately 1 year after they were randomly assigned. The survey, fielded to the full study samples in both states, covered topics such as sample members’ participation in and experiences with student services, expectations and engagement in college, employment, and financial aid and other financial issues.¹⁰

We use the following equation to estimate the effects of being offered ASAP:

Y_{i j} = α + T_{i j} β + X_{i j} λ + C_{i j} γ_{j} + ε_{i j},

where Y_ij denotes the outcome for individual i in college j. T_ij is an indicator variable for assignment to the program group, eligible for ASAP, and the vector X_ij includes individual covariates that are expected to be correlated with the outcomes of interest, such as race, age, gender, and developmental education requirements. The vector C includes randomization block indicators, one for each campus and cohort combination. The coefficient β is the key parameter of interest and represents the effect of the program offer. Given that individuals were assigned at random to receive the offer to enroll in ASAP, this regression-based estimator of β is unbiased with respect to our target of inference (the estimand), which is the average effect across individuals in the evaluation sample.¹¹

Key outcomes include enrollment in 2- and 4-year colleges and degree receipt (associate’s degrees and bachelor’s degrees). The evaluation estimates the effects on key outcomes of the offer to participate in ASAP, or intent to treat estimates. Treatment-on-the-treated estimates are not presented, given that over 90% of students in the program group experienced at least one semester of the program. However, these estimates (of the effect of participating in at least some of the program, rather than the effect of the offer to participate in the program) can be viewed as roughly 10% larger than the estimates shown in this paper. Finally, no weighting is applied when estimating effects for the pooled sample.

Earlier Findings

ASAP’s effects in both states, through Year 3 in Ohio and Year 6 in New York, have been reported separately in earlier publications (Miller et al., 2020; Weiss et al., 2019). The CUNY program was found to increase 3-year degree receipt by 18 percentage points, from 22% for the control group to 40% for the program group. Rates of degree receipt continued to increase through Year 6 for both the program and control groups, and the differential between the two groups diminished in size. However, the impact at the end of Year 6 was still sizable, at 10 percentage points (p < .01). ASAP also increased enrollment rates and credit accumulation during the program period, or the first 3 years.

The Ohio program similarly led to large effects on enrollment, credit accumulation, and degree receipt by the end of Year 3. At that point, 35% of the students in the program group had earned a degree, compared with 19% of students in the control group, for an increase of 16 percentage points.

Update on CUNY ASAP Effects Through 8 Years

The earlier paper on CUNY ASAP showed that the effects on associate’s degree receipt diminished over time but remained sizable at the end of 6 years (Weiss et al., 2019). That paper was also the first to show small, positive effects on the receipt of bachelor’s degrees, with increases of 2 to 3 percentage points in semesters 8 and 9. However, these effects dissipated over time, as the control group caught up. This section updates those findings, to assess whether the stability in both levels and effects apparent by the end of Year 6 continued through Year 8.

Although a fade out of effects occurs between years 3 and 6, it is not obvious that impacts on degree receipt would remain stable after Year 6, given continued college enrollment after Year 6 and the fact that a fair number of students take longer than 6 years to earn a degree. Thus, a more complete assessment of ASAP’s effects on degree receipt can be obtained with the 8-year results.

Figures 2 and 3 present effects on enrollment and degree receipt. In semesters 2 through 4, during the program period, enrollment rates in 2-year colleges are higher for the program group than the control group, differences that are statistically significant. In the first two semesters of the post-program period (or semesters 7 and 8) enrollment in 2-year colleges is lower for the program group than the control group, given their higher rates of degree receipt and higher rates of enrollment in 4-year colleges. However, by the last few semesters of the 8-year period, enrollment rates at 2-year colleges have flattened out and fallen to less than 5% for both research groups. Enrollment in 4-year colleges has also flattened out by that time, at around 15%, with no meaningful differences between the two groups during the last 2 years of follow-up.

Figure 2.

(A) Enrollment rate at 2-year colleges. (B). Enrollment rate at 4-year colleges.

Figure 3.

(A) Degree receipt at 2-year colleges. (B) Degree receipt at 4-year colleges.

Effects on degree receipt are separated into associate’s and bachelor’s degrees. The main goal of the program is to increase the receipt of associate’s degrees. A few more students earned associate’s degrees over the additional 2 years of follow-up, but estimated effects on this outcome are stable in semesters 12 through 16, at about 12 percentage points.

In terms of bachelor’s degree receipt, about 25% of both research groups had earned a degree by the end of Year 8. The program led to small and statistically significant increases in semesters 8 through 10, but no effects thereafter.

The findings suggest that the program’s effect on bachelor’s degree receipt was to speed up receipt among students who would have earned it anyway. In contrast, the stability of the estimated impact on associate’s degree receipt during the last six semesters of follow-up increases our confidence that ASAP increased associate’s degree receipt among a sizable group (12 percentage points) of students who would not have earned an associate’s degree otherwise. The fact that the estimated effect on associate degree receipt peaked at 18 percentage points implies that, at a minimum, ASAP’s effect of speeding up degree receipt, among those who would have graduated anyway, is at least 6 percentage points This speeding up of degree receipt is notable and can reduce costs for colleges and students; however, the long-term increase degree receipt is what can drive the benefits of the program, given that the gains from higher education accrue over an individual’s lifetime.

Findings From the Pooled Sample

The remainder of the paper examines effects through 3 years using the New York and Ohio samples, estimating effects for the pooled sample, for subgroups, by state and by college. This section presents effects for the combined sample. Figure 4 presents the results for enrollment and degree receipt at any 2-year college.¹² Enrollment rates at any school fall steadily after the first semester, but less rapidly for students in the program group. By semester 2, for example, only 73% of students in the control group were still enrolled, compared with 84% in the program group, for a difference of 11 percentage points. Effects are similar in size in semesters 3 and 4 but become negligible after that point.¹³

Figure 4.

Enrollment in associate’s degree-granting colleges and associate’s degree receipt.

Study participants begin earning degrees by semester 3, and the increase is more rapid for students in the program group. By the end of semester 6, about 36% of students in the program group had earned a degree (nearly all associate’s degrees), compared with 18.8% of students in the control group, for a substantial increase of nearly 17 percentage points (with a 90% confidence interval ranging from 14 percentage points to 20 percentage points). As noted, this effect is nearly a doubling of degree receipt and is among the largest observed to date for a college intervention. The estimate of program effect is also for a broader population.

Enrollment rates fall over time as students either graduate or leave school without a degree. Thus, the program’s effects on enrollment are somewhat difficult to interpret in later semesters, given that they are estimated for the full sample and include students who have graduated.

Although not shown, enrollment rates in 4-year colleges gradually increase each semester, and almost 14% of control group students are enrolled in semester 6. The program increased that rate by an estimated 6.5 percentage points (p < .001).

Overall, the rate of transfer to 4-year colleges for both research groups is low, although perhaps not surprising, given the barriers to entry faced by students from families with low-incomes. National data indicate that only about 33% of community college students transfer to a 4-year institution within 6 years, despite the fact that the majority of them (as with this study sample) enter college intending to pursue a bachelor’s degree (Jenkins & Fink, 2016).

Effects for Subgroups of Students

As shown above, for the pooled sample, the findings show that ASAP led to a notable increase in degree receipt and an increase in 4-year college enrollment by the end of Year 3. This section examines whether effects vary for different types of students. Community college students are a diverse group, varying in their levels of academic preparation upon entry, their age, family status, and work status, for example. A significant share of them are “nontraditional” students, for example, meaning that they do not fit the profile of the single, childless, young adult entering college straight from high school. Subgroup analyses help to assess whether ASAP can be effective for a wide variety of student types, which speaks to the generalizability of the findings.

Table 2 presents the results.¹⁴ The top rows present effects by development education requirements at study entry. This subgroup was pre-specified in each study as the sole confirmatory subgroup, given that it is a key question of interest for which the study teams desired rigorous evidence. The remaining rows present effects for several subgroups as part of an exploratory analysis. Tests of effects across these dimensions (age, gender, high school diploma status, race/ethnicity, and work status) are of interest in their own right, but any findings should be interpreted with some caution and as suggestive for further study since these subgroups are exploratory (Schochet, 2008).

Table 2

Degrees Earned and 4-Year College Enrollments After 3 Years Variation in Effects by Student Characteristics

Characteristic	Earned degree				Enrolled in a four-year college
Characteristic	Control group (%)	Difference (percentage point)	SE	p value for subgroup difference	Control group (%)	Difference (percentage point)	SE	p value for subgroup difference
Developmental education requirements				.189				.445
None	25.5	17.6	5.2		21.0	1.0	4.6
One	22.5	21.3	3.3		14.8	7.6	2.8
Two or more	16.2	13.5	2.6		11.8	6.9	2.2
Gender				.206				.739
Female	20.9	17.9	2.3		15.1	6.0	2.0
Male	19.2	13.1	2.9		12.1	7.0	2.5
Age				.241				.193
19 or younger	20.0	17.6	2.5		14.9	7.6	2.2
20–23	20.9	10.4	3.9		13.6	1.7	3.2
24 or older	19.8	18.4	3.7		11.7	9.3	3.0
High school diploma at study entry				.372				.307
No	17.5	13.0	4.2		13.0	3.0	3.6
Yes	20.6	17.2	2.0		14.1	7.1	1.7
Race/ethnicity				.305				.754
Hispanic	22.1	10.3	4.0		14.4	4.6	3.3
White	23.7	19.5	3.4		13.3	7.4	2.7
Black	14.9	18.4	3.0		12.9	9.0	2.7
Other	18.1	18.5	5.9		15.3	5.3	5.2
Employed at study entry				.736				.939
No	21.3	15.9	2.6			6.3	2.2
Yes	18.7	17.1	2.6		14.1	6.5	2.3

Source. Calculations using data from the CUNY Institutional Research Database (IRDB), the ASAP Ohio Demonstration study colleges, and the National Student Clearinghouse (NSC).

Note. Estimates are adjusted by random assignment blocks and selected baseline characteristics. Rounding may cause slight discrepancies in calculating differences and sums. The H-statistic test was used to test for statistically significant differences in impact estimates across different subgroups.

The first set of columns presents effects on degree attainment, and the second set of columns shows effects on 4-year college enrollment after 3 years. ASAP increased degree receipt for all subgroups examined—effects for each group are large and statistically significant. Although estimated effects are somewhat smaller for certain groups, such as those without a high school diploma at entry and Hispanic students, they are not statistically significantly different from estimated effects for their subgroup counterparts. In other words, we do not find clear evidence that ASAP was more effective for one subgroup compared with another. Notably, the precision of the estimates of variation in effectiveness among subgroups is limited; consequently, lack of evidence of differential effectiveness should not be interpreted as strong evidence that effects are very similar among subgroups.

Effects on 4-year college enrollment are also positive for all groups, although not always statistically significant. There are two notable differences, although again, the variation across groups is not statistically significant. First, there are the smaller estimated effects for students without developmental education requirements. This may be because these students were more likely to attend a 4-year college in the absence of the program, as shown by rates for the control group, although the small sample size for this group makes the estimate imprecise.¹⁵ The program also did not appear to increase enrollment rates in 4-year colleges for students without a high school diploma at entry—again, this finding may be meaningful, or may simply reflect an imprecise impact estimate for a very small subgroup. Similar to other students, the majority of students without a high school diploma reported at study entry that they expected to obtain at least a bachelor’s degree. Thus, the lack of effect for this group is not due to lower aspirations.

Effects by State

The ASAP model was created by CUNY in 2007 and was operating at six of its community colleges before the evaluation began in 2009. The Ohio colleges worked closely with CUNY on the design and implementation of the program, but as expected, the replication was not exact. In addition, as shown below, the Ohio programs led to a smaller service contrast than CUNY. These factors, along with the different context in which the program was tested, warrant a comparison of effects across the two states.

Table 3 presents the results. The first two rows present effects on degree receipt and 4-year enrollment, using data from administrative records for the full sample. The next set of rows presents data from the survey subsample on receipt of key services (those shown for the pooled sample in Figure 1) and perceptions of support at school.¹⁶ The program had very similar estimated effects on both key outcomes. CUNY ASAP increased degree receipt by 17.3 percentages points, compared with 15.9 percentage points for Ohio. Effects on 4-year college enrollment are also very similar.

Table 3

Academic, Service, and Engagement Effects, by State

Outcomes	CUNY ASAP		SE	Ohio programs		SE	p value for subgroup difference
Outcomes	Control group	Difference	SE	Control group	Difference	SE	p value for subgroup difference
Academic outcomes (%)
Earned a degree	22.7	17.3	3.0	18.4	15.9	2.3	.725
Enrolled in a 4-year college	18.0	6.8	2.7	11.8	5.7	1.9	.739
Service contrast
Times met with adviser	3.8	17.3	1.4	5.0	8.3	0.9	.000
Times met with tutor	4.2	8.5	1.1	3.4	4.9	0.8	.010
Times met with career services	1.1	3.8	0.4	0.7	1.7	0.3	.000
Student engagement (%)
Had an employee to turn to for advice	58.4	28.0	3.2	72.6	16.5	2.5	.005
Had supports/services needed	56.9	32.7	3.1	81.0	10.2	2.2	.000
Sample size	445	896		695	1,501

Source. Calculations using data from the CUNY Institutional Research Database (IRDB), the ASAP Ohio Demonstration study colleges, the National Student Clearinghouse (NSC), and the MDRC student surveys.

Note. Estimates are adjusted by random assignment blocks and selected baseline characteristics. Sample sizes for specific outcomes may vary because of missing values. Rounding may cause slight discrepancies in calculating differences and sums.

As noted in the introduction, the similarity of effects is striking, given the difficulty of replicating original findings (the “replication crisis,” as it is sometimes called). But the similarity is also notable given that the students, contexts, and service contrast differed across the two sites. For example, more students in Ohio than in New York were nontraditional students, e.g., older, or with children. This difference in the sample is notable, as there was some concern that the model, with its full-time attendance requirement, would not work for these types of students. Also, fewer students in Ohio than in New York had developmental education requirements.

The context also differed in terms of management structure. CUNY is the largest urban community college system in the nation and is fairly centralized. Management of the program was led centrally by a dedicated ASAP team in the CUNY Office of Academic Affairs. Ohio’s community colleges, in contrast, are decentralized, meaning that the colleges largely implemented the program independently.

Finally, the similar findings are also noteworthy, given that the program in New York appears to have created a larger service contrast than the program in Ohio (as indicated by students’ responses to a survey). For example, program group students in New York met with their advisor on average 17.3 more times in the first semester than their control group counterparts, compared with a difference of 8.3 visits in Ohio. Control group advising visits were fairly similar across the two states, suggesting that the difference in effects is not driven by different baseline levels. Recall that Ohio implemented a triage model of advising, although this difference would primarily affect advising sessions in the second and later semesters. In addition, Ohio advisers had much larger caseloads than in CUNY, at the time of the study.

Differences in impacts on tutoring are not as large, but still sizable. Part of this difference could be driven by the fact that the CUNY sample consisted entirely of students with development education requirements, for whom tutoring was required under ASAP, whereas only 75% of the Ohio student had developmental education requirements. However, when the analysis for the Ohio study was restricted to students with development education requirements, the pattern of results was similar to that shown in the table.¹⁷

The final two rows present effects on students’ perceptions of support. ASAP in New York led to much larger changes. That program, for example, led to a 33-percentage point increase in the number of students who felt that they had adequate supports and services to succeed in school. The increase in Ohio was 10 percentage points. Part of the difference in effects may be due to the relatively high level of support already felt by students in Ohio, as indicated by responses for the control group students, leaving less room for improvement.¹⁸

The findings suggest that there may be diminishing returns to increased receipt of a given student service. In other words, perhaps encouraging students to visit their advisor an additional eight times is sufficient to generate the observed increase in graduation rates, with more visits generating little added effect. The next section looks at this issue from another angle by assessing effects for the individual college campuses.

Effects by College Campus

Although the New York programs were managed by CUNY, the programs in both states were ultimately specific to a given campus, with its particular environment, program director, advisors, other staff, and services. This section presents effects by campus. The estimates for the pooled sample and by state provide an overall average estimate of ASAP’s effectiveness. Estimating effects by campus provide further evidence on the replicability of the model in different settings.¹⁹ A caveat to this analysis is the smaller sample sizes at each individual campus, limiting the statistical power of tests of individual site effects and tests for variation across sites.

First consider effects on degree receipt through three years, as shown in Figure 5. The estimates range from 5 percentage points to 21 percentage points, and six of the seven estimates are greater than 11 percentage points and statistically significant, as illustrated by the confidence intervals. Although there is some variation in estimated effects across the campuses, it is not more than would be expected by chance if the true effects were the same across the campuses (p = .486 for the test of cross campus differences in effects). The fact that we cannot reject the null hypothesis of similar effects is encouraging and provides stronger evidence that the model can work in a range of settings.

Figure 5.

Associate’s degree effects by college/campus, through three years.

Figure 6 presents estimates by campus of the service contrast. The figure presents estimated effects on the number of times students participated in the three key services: advising, tutoring, and career services. It also presents effects on the students’ average feelings of support at the campus (estimated as an average of the impact on the two support outcomes shown in Table 3). The campuses are ordered in each graph by the size of their estimated effects on degree receipt, with the smallest effects on the left side and the largest effect on the right side.

Figure 6.

Effects on service contrast.

The figure shows notable variability in service contrast, variability that is statistically significant for each outcome. For example, the program-induced increase in advising was 20 visits at campus seven, compared with seven visits at campus two. The differences in impacts on perceptions of support are especially large.

The variation across campuses in service contrast raises the question of whether these effects are correlated with effects on degree receipt. Figure 7 presents the results. Each panel in the figure plots estimated impacts on degree receipt (on the vertical axis) against a given estimated service contrast (e.g., increase in the number of advising sessions attended) on the horizontal axis. The size of each bubble is proportional to the size of the sample it represents. In addition, a weighted least squares regression line is estimated over the seven observations and plotted in each graph, in which estimates based on larger samples are given greater weight.

Figure 7.

Association between service contrast and impacts on degree receipt.

Note that this analysis is non-experimental, and any association observed does not necessarily imply that the service contrast caused the impact. Other factors about the campuses, which may be associated with both outcomes, are not controlled for in the analysis. For example, the colleges vary in size, with some serving over 20,000 students and others less than 10,000 students. (Appendix Table S1 presents characteristics of the colleges, rather than the individual campuses, given that campus-level data are not available). Other differences include the demographic composition of the student population, in terms of race/ethnicity and age, and the rate at which students at a given college graduate. A range of campus-level features might contribute to program impacts, although the number of campuses in this study is too small to support a formal investigation.

The figures highlight two points. First, the positive sloping regression lines suggest that there is a positive, albeit weak, association between the estimated effect on the service contrast and the estimated effect on degree impacts. For example, for every three additional advising visits the graduation effect increased by around one percentage point.

Second, despite the positive relationships, effects on degree receipt are still quite large even with substantially smaller service contrasts. For example, consider the relationship between ASAP’s effect on advising visits during the first semester and ASAP’s effects on degree completion. The weighted regression line suggests that the cluster of three campuses where the program caused an increase of five to eight additional advising visits had degree effects of around 15 percentage points. The three campuses where ASAP caused an increase of over 16 additional advising visits had degree effects closer to 17 percentage points (this average includes the one outlier campus). Thus, from a cost perspective, there may be room to reduce the resources put into the service contrast (such as number of advising visits) and still achieve large effects. Put differently, the question might be “how does the cost of moving from five additional advising visits to 18 additional visits, for example, compare with the benefits of increasing degree receipt by an additional 3 percentage points?”

In sum, the findings by state presented in the previous section suggested that, beyond some initial positive amount, additional impacts on students’ interaction with staff may not be necessary to achieve large effects on degree receipt. However, the more disaggregated findings presented here support the idea that there is some positive relationship between service contrast and degree receipt. This analysis is only suggestive, however, given the small number of campuses, the small sample size for each campus, and the fact that the studies were not designed to estimate causal relationships between these two factors. A more formal analysis is warranted when data are available from subsequent studies.

Conclusion

This paper has presented an update of findings from CUNY ASAP through 8 years and a synthesis of findings through 3 years from Ohio and New York. ASAP is an unusually comprehensive, integrated, 3-year program that requires full-time enrollment, offers several forms of student supports and financial supports, and includes messaging around remedial course taking, enrollment intensity, and timely graduation. The longer-term findings for New York show that the effects on associate’s degree receipt narrow over time, as the control group catches up, but remain sizable and stable by the end of 8 year, at 12 percentage points. Effects on 4-year enrollment and bachelor’s degree receipt narrowed completely by the end of the period, suggesting that the main effect of the program on these outcomes was to help a small subset of students achieve them more quickly.

The pooled analysis for both states showed that, through 3 years, the ASAP model had large positive effects on receipt of associate’s degrees—36% of students offered the program graduated, compared with 18.8% of students offered typical college services. At the end of Year 3, the program had also increased enrollment in 4-year colleges. The program increased degree receipt for a variety of student types, including women and men, those with developmental education requirement and those without, and those with a high school degree at study entry and those without a degree. The program also had large effects in a variety of settings. The consistency of these findings suggest that the results are likely to generalize beyond these students and colleges.

An analysis by state also showed similar effects on degree attainment, even with some differences in implementation and service contrast. The Ohio programs led to smaller increases in advising and tutoring than CUNY ASAP, for example, but had similar-sized effects on degree receipt. These findings raised the questions of whether such large increases in student services, as observed for CUNY ASAP, are necessary to achieve substantial gains in graduation. However, the findings across the individual campuses suggest that there is some positive relationship between service contrast (or increases in advising and tutoring) and impacts on degree receipt. The issue of how program impacts vary with program components warrants further study, as it speaks to program improvement and cost.

The findings reiterate that this model of comprehensive supports and requirements can lead to large and lasting increases in degree receipt. But providing these supports to students has a cost. The direct cost of the program in Ohio, for example, was about $1,800 per student per year. The cost of the original CUNY model was higher than that, at about $4,700 per student per year, in part because of higher New York salaries and costs but also because of more intensive advising requirements and lower advisor caseloads. These costs are roughly in line with the costs of other comprehensive interventions, such as the Stay the Course initiative mentioned earlier (with a cost of about $2,000 per student per year) or the Valley Initiative for Development and Advancement initiative ($5,500 per student per year), which provides comprehensive counseling and financial assistance to help individuals complete certificates or associate’s degrees in high demand occupations (Rolston et al., 2017).

CUNY ASAP’s costs have gone down in recent years, due to economies of scale as it has expanded to serve more students, and due to the adoption of a triage approach to advising and larger caseloads per advisor. In addition, although a formal analysis was not conducted for either study, some part of these costs would likely be recouped through formula funding in most states that increases with enrollment and degree receipt. Of course, the costs of any intervention must be considered relative to any benefits it generates. From a societal perspective, a key benefit to consider is the increased earnings that come with an associate’s degree, which would accrue over the working lives of participants (Belfield & Bailey, 2017).

Despite the additional cost, CUNY has expanded ASAP substantially since the original evaluation. With funding from the City of New York, the model now operates at more colleges within the CUNY system and serves a broader group of students. In Ohio, in contrast, only one of the three colleges in the original study has sustained the program. Nationwide, colleges’ ability to implement and sustain ASAP will depend on funding support from the states or other sources.

Another option for addressing the costs is to consider reducing the intensity of certain components or dropping certain components entirely. The study evaluated the effects of the complete package, making it impossible to identify which, if any, components were responsible for its effects. Nonetheless, the Ohio results suggest that a triage approach to advising and tutoring can work, which is CUNY’s current approach, and that very large average increases in advising and tutoring observed in the CUNY evaluation may not be necessary to increase degree receipt. In addition, the Ohio colleges and one New York college did not formally implement blocked scheduling, raising the question of whether that component is necessary to the model.

Other questions raised by the findings are related to generalizability and scaling. The first question is whether the model could be effective for a broader group of students. The eligibility requirements are fairly broad, but the requirement to attend full-time, for example eliminated students who cannot attend at that level, for reasons of work, child care, etc. (although, as noted, the ASAP model led to large increases in full-time enrollment, indicating that many students can attend full-time with the necessary support). The recently evaluated “Stay the Course” model did not require full-time enrollment and had large positive effects on degree receipt, although the effects were largely for women, and the small sample sizes mean that there is a fair amount of sampling error around the estimates. Nonetheless, one question is whether the full-time requirement could be modified in some way. Given the importance of full-time enrollment for making steady progress through school, the program would probably not want to allow students who can attend full-time the option of attending less (Adelman, 2004). But perhaps a version of the model could be tested that brings in and serves students who cannot.²⁰

Another group that might benefit from the program is students in fields of study that take longer than 3 years to complete, such as those in the large field of allied health. For good reason, given that the goal of the program was to increase 3-year degree attainment, these students were not part of the program. But it is easy to imagine that they would benefit from the comprehensive supports and services provided by the ASAP model. Students with significant remedial needs were also excluded from the program, given that they also would be unlikely to graduate within 3 years. Perhaps some version of ASAP could help these students stay on track and earn a degree.²¹

The findings and analysis highlight the importance of replication in the study of education interventions, which might be achieved through the implementation of an existing model in a new site or via a research design that involves many sites at once. Replication efforts and multi-site trials also provide the ability to assess how program impacts might vary with local context (e.g., size of college or local economy) or with specific program components (e.g., number of advising visits or blocked scheduling). If there is interest in assessing the effect of local factors, evaluation might be deliberate in the selection of sites to provide adequate variability in local context. Assessing the effects of program components might be achieved through planned variation in components across sites or via differences in actual implementation.

A final and related point is that, although the findings shown here are very encouraging, the replication of the ASAP model in one other state is not definitive evidence of its broad effectiveness. More information is needed on the ability to implement the model and find similar effects in different places and for different types of students. As CUNY continues to replicate the model and provide technical assistance to colleges, as they did for the Ohio replication, similar results may be expected.²² But as community colleges across the country develop and implement their own programs borrowing from ASAP and without CUNY’s technical assistance, it will be important to understand if they can achieve similar results

Footnotes

Appendix

Table S1

College Characteristics for the CUNY ASAP and Ohio Programs Evaluation.

Characteristic	Cincinnati State Technical and Community College	Lorain County Community College	Cuyahoga Community College	Borough of Manhattan Community College	Kingsborough Community College	LaGuardia Community College
Campus structure	1 main campus + 3 other locations around the city	1 main campus + 5 satellite centers across the county	4 main campuses + 8 satellite locations across the county	1 main campus	1 main campus	1 main campus
Number of students	8,807	11,042	23,900	22,534	18,606	17,569
Full-time students (%)	26	27	31	65	60	58
Male (%)	44	39	40	41	44	41
Black (%)	24	9	25	30	32	17
Other minorities (%)	17	18	20	58	33	69
Under 24 years of age (%)	55	70	61	68	74	64
Campus setting	City: large	City: small	City: large	City: large	City: large	City: large
Geographic location	Southwest Ohio	Northeast Ohio	Northeast Ohio	New York City	New York City	New York City
Graduation rate (%)	15	23	15	14	25	15
Undergraduate students receiving Pell Grants (%)	33	36	38	79	69	72
Sample size (total = 2,397)	467	513	521	401	325	170

Source. Study colleges’ websites and MDRC calculations using data from the Integrated Postsecondary Education Data System (IPEDS).

Note. Two campuses in this study, Tri-C East and Tri-C West are combined into one college, Cuyahoga Community College. IPEDS data for campuses in each state reflect a year within each evaluation period: 2017 in Ohio and 2010 in New York.

Acknowledgements

The authors thank program staff and leaders across all six colleges for their roles in the original demonstration at the City University of New York (CUNY) colleges (Donna Linderman, Christine Brongniart, Mary Hiebert, Lesley Leppert, Diana Strumbos, and Zineta Kolenovic) and the replication at the Ohio colleges (Brett Visger, Katrina Rugless, Matthew Mercado, Miria Batig, and Angela Moore).

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding for the long-term follow-up of CUNY’s ASAP was provided by the Institute of Education Sciences, U.S. Department of Education, through grant R305A160273 to MDRC. Opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

Notes

Authors

CYNTHIA MILLER is a senior fellow in MDRC’s Economic Mobility, Housing, and Communities Policy Area. Her research focuses on the evaluation of programs to increase employment and income of low-wage workers and young adults. She also conducts research on programs to increase college completion among students from families with low-incomes.

MICHAEL J. WEISS is a senior fellow in MDRC’s Postsecondary Education Policy Area. He has over a decade of experience evaluating programs designed to improve community college students’ success rates. He also conducts methodological research to improve the quality of randomized controlled trials.

References

Adelman

(2004). Principal indicators of student academic histories in postsecondary education, 1972–2000. U.S. Department of Education, Institute of Education Sciences.

Angrist

Hudson

Pallais

(2014). Leveling up: Early results from a randomized evaluation of post-secondary aid (No. w20800). National Bureau of Economic Research.

Angrist

Hudson

Pallais

(2016). Evaluating post-secondary aid: Enrollment, persistence, and projected completion effects (Working Paper No. 23015). National Bureau of Economic Research.

Angrist

Lang

Oreopoulos

(2009). Incentives and services for college achievement: Evidence from a randomized trial. American Economic Journal: Applied Economics, 1(1), 1–28.

Angrist

Oreopoulos

Williams

(2010). When opportunity knocks, who answers? New evidence on college achievement awards (No. w16643). National Bureau of Economic Research.

Attewell

Douglas

(2014). Increasing the academic momentum of community college students. Society for Research on Educational Effectiveness conference presentation.

Attewell

Heil

Reisel

(2012). What is academic momentum? And does it matter? Educational Evaluation and Policy Analysis, 34(1), 27–44.

Attewell

Jang

S. H.

(2013). Summer coursework and completing college. Research in Higher Education Journal, 20. https://www.aabri.com/manuscripts/131522.pdf

Attewell

Lavin

Domina

Levey

(2006). New evidence on college remediation. The Journal of Higher Education, 77(5), 886–924.

10.

Avery

Howell

J. S.

Page

(2014). A review of the role of college counseling, coaching, and mentoring on students’ postsecondary outcomes: Research brief. College Board.

11.

Bailey

Bashford

Boatman

Squires

Weiss

Doyle

Valentine

J. C.

, et al. (2016). Strategies for postsecondary students in developmental education. Institute of Education Sciences, What Works Clearinghouse.

12.

Bailey

Jaggars

S. S.

Jenkins

(2015). What we know about guided pathways: Helping students to complete programs faster. Community College Research Center, Teachers College, Columbia University.

13.

Barr

Castleman

(2017). The bottom line on college counseling [Working paper]. https://collegecompletionnetwork.org/resources/the-bottom-line-on-college-counseling#:~:text=The%20study%20found%20that%20Bottom,the%20first%20cohort%20of%20students.&text=The%20program%20emphasizes%20college%20affordability,full%20cost%20of%20college%20a

14.

Bailey

(2009). Rethinking developmental education in community college. Community College Research Center, Teachers College, Columbia University.

15.

Baum

Kurose

McPherson

(2013). An overview of American higher education. Future of Children, 23(1), 17–39.

16.

Baum

(2014). Trends in college pricing 2014. The College Board.

17.

Belfield

Bailey

(2017). The labor market returns to sub-Baccalaureate College: A review. Center for Analysis of Postsecondary Education and Employment, Community College Research Center, Teachers College, Columbia University.

18.

Berger

Turk-Bicakci

Garet

Knudson

Hoshen

(2014). Early college, continued success: Early college high school initiative impact study. American Institutes for Research.

19.

Bettinger

E. P.

Baker

R. B.

(2014). The effects of student coaching: An evaluation of a randomized experiment in student advising. Educational Evaluation and Policy Analysis, 36(1), 3–19.

20.

Bettinger

E. P.

Long

B. T.

Oreopoulos

Sanbonmatsu

(2012). The role of application assistance and information in college decisions: Results from the H&R block FAFSA experiment. Quarterly Journal of Economics, 127(3), 1205–1242.

21.

Bound

Turner

(2007). Cohort crowding: How resources affect collegiate attainment. Journal of Public Economics, 91(5–6), 877–899.

22.

Braxton

J. M.

(2002). Reworking the student departure puzzle. Vanderbilt University Press

23.

Calcagno

J. C.

Bailey

Jenkins

Kienzl

Leinbach

(2008). Community college student success: What institutional characteristics make a difference? Economics of Education Review, 27, 632–645.

24.

Camerer

C. F.

Dreber

Holzmeister

T. H.

Huber

Johannesson

. . . Altmejd

(2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637–644.

25.

Carrell

Sacerdote

(2017). Why do college-going interventions work? American Economic Journal: Applied Economics, 9(3), 124–151.

26.

Carrell

S. E.

Sacerdote

(2013). Late interventions matter too: The case of college coaching New Hampshire (No. w19031). National Bureau of Economic Research.

27.

Castleman

Page

(2015). Summer nudging: Can personalized text messages and peer mentor outreach increase college going among low-income high school graduates? Journal of Economic Behavior & Organization, 115, 144–160.

28.

Cohodes

S. R.

Goodman

J. S.

(2014). Merit aid, college quality, and college completion: Massachusetts’ Adams scholarship as an in-kind subsidy. American Economic Journal: Applied Economics, 6(4), 251–285.

29.

Crosta

P. M.

(2014). Intensity and attachment: How the chaotic enrollment patterns of community college students relate to educational outcomes. Community College Review, 42(2), 118–142.

30.

Deming

Dynarski

(2010). College aid. In Levine

P. B.

Zimmerman

D. J.

(Eds.), Targeting investments in children: Fighting poverty when resources are limited (pp. 283–302). University of Chicago Press.

31.

Engstrom

C. M.

Tinto

(2008). Learning better together: The impact of learning communities on the persistence of low-income students. Opportunity Matters, 1, 1–17.

32.

Evans

W. N.

Kearney

M. S.

Perry

B. C.

Sullivan

J. X.

(2017). Increasing community college completion rates among low-income students: Evidence from a randomized controlled trial evaluation of a case management intervention (No. w24150). National Bureau of Economic Research.

33.

Ginder

S. A.

Kelly-Reid

J. E.

Mann

F. B.

(2019). Enrollment and employees in postsecondary institutions and financial statistics and academic libraries, fiscal year 2017 (First look provisional data NCES 2018-002). National Center for Education Statistics. https://nces.ed.gov/pubs2019/2019021REV.pdf

34.

Goldrick-Rab

Harris

Kelchen

Benson

(2012). Need-based financial aid and college persistence experimental evidence from Wisconsin. Institute of Research on Poverty.

35.

Goldrick-Rab

Kelchen

Harris

D. N.

Benson

(2016). Reducing income inequality in educational attainment: Experimental evidence on the impact of financial aid on college completion. American Journal of Sociology, 121(6), 1762–1817.

36.

Greenberg

Robert

Wiseman

M. M.

(1994). Multisite employment training program evaluations: A tale of three studies. ILR Review, 47(4), 679–691.

37.

Jenkins

P. D.

Fink

(2016). Tracking transfer: New measures of institutional and state effectiveness in helping community college students attain bachelor’s degrees. https://ccrc.tc.columbia.edu/publications/tracking-transfer-institutional-state-effectiveness.html

38.

Klempin

S. C.

(2014). Redefining full-time in college: Evidence on 15-credit strategies. https://ccrc.tc.columbia.edu/publications/redefining-full-time-in-college.html#:~:text=To%20complete%20an%20associate%20degree,increase%20rates%20of%20on%2Dtime

39.

Logue

A. W.

Douglas

Watanabe-Rose

(2019). Corequisite mathematics remediation: Results over time and in different contexts. Educational Evaluation and Policy Analysis, 41(3), 294–315.

40.

Mayer

A. K.

Patel

Gutierrez

(2015). Four-year effects on degree receipt and employment outcomes from a performance-based scholarship program in Ohio [MDRC working paper]. MDRC.

41.

McFarland

Hussar

Zhang

Wang

Hein

Diliberti

Forrest Cataldi

Bullock Mann

Barmer

(2019). The condition of education 2019 (NCES 2019-144). U.S. Department of Education, National Center for Education Statistics. https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2019144

42.

Miller

Headlam

Manno

Cullinan

(2020). Increasing community college graduation rates with a proven model: Three-year results from the accelerated study in associate programs (ASAP) Ohio demonstration. MDRC.

43.

National Academic Advising Association. (2011). Advisor load statistics from the NACADA national survey of academic advising: Table 6.28. National Academic Advising Association.

44.

Patel

Richburg-Hayes

de la Campa

Rudd

(2013). Performance-based scholarships: What have we learned? Interim findings from the PBS demonstration. MDRC.

45.

Richburg-Hayes

Brock

LeBlanc

Paxson

Rouse

C. E.

Barrow

(2009). Rewarding persistence: Effects of a performance-based scholarship program for low-income parents. MDRC.

46.

Rolston

Copson

Gardiner

(2017). Valley initiative for development and advancement: Implementation and early impact report (Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation Report 83). U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation.

47.

Rutschow

E. Z.

Cullinan

Welbeck

(2012). Keeping students on course: An impact study of a student success course at Guilford technical Community College. MDRC.

48.

Schochet

P. Z.

(2008). Technical methods report: Guidelines for multiple testing in impact evaluations [NCEE report]. National Center for Education Evaluation and Regional Assistance.

49.

Scott-Clayton

Crosta

P. M.

Belfield

C. R.

(2014). Improving the targeting of treatment: Evidence from college remediation. Educational Evaluation and Policy Analysis, 36(3), 371–393.

50.

Scrivener

Weiss

M. J.

(2009). More guidance, better results? Three-year effects of an enhanced student services program at two community colleges. MDRC.

51.

Scrivener

Weiss

M. J.

Ratledge

Rudd

Sommo

Fresques

(2015). Doubling graduation rates: Three-year effects of CUNY’s accelerated study in associate programs (ASAP) for developmental education students. MDRC.

52.

Scrivener

Weiss

M. J.

Sommo

(2012). What can a multifaceted program do for community college students? Early results from an evaluation of accelerated study in associate programs (ASAP) for developmental education students. MDRC.

53.

Strumbos

Kolenovic

(2017). Six-year outcomes of ASAP students: Transfer and degree attainment [ASAP evaluation brief]. City University of New York.

54.

Tinto

(1997). Classrooms as communities: Exploring the educational character of student persistence. The Journal of Higher Education, 68(6), 599–623.

55.

U.S. Department of Education, National Center for Education Statistics. n.d. QuickStats. https://nces.ed.gov/datalab/quickstats. Accessed on July 5, 2018.

56.

Weiss

(2019). How can community colleges increase student use of year-round Pell Grants? Two proven strategies to boost summer enrollment. MDRC.

57.

Weiss

Mayer

Cullinan

Ratledge

Sommo

Diamond

(2015). A random assignment evaluation of learning communities at Kingsborough Community College: Seven years later. Journal of Research on Educational Effectiveness, 8(2), 189–217.

58.

Weiss

Ratledge

Sommo

Gupta

(2019). Supporting community college students from start to degree completion: Long-term evidence from a randomized trial of CUNY’S ASAP. American Economic Journal: Applied Economics, 11(3), 253–297.

59.

Weiss

Visher

Weissman

Wathington

(2015). The impact of learning communities for students in developmental education: A synthesis of findings from randomized trials at six community colleges. Educational Evaluation and Policy Analysis, 37(4), 520–541.