An On-Ramp to Student Success: A Randomized Controlled Trial Evaluation of a Developmental Education Reform at the City University of New York

Abstract

Most community college students are referred to developmental education courses to build basic skills. These students often struggle in these courses and college more broadly. CUNY Start is a prematriculation program for students assessed as having significant remedial needs. CUNY Start students delay matriculation for one semester and receive time-intensive instruction in math, reading, and writing with a prescribed pedagogy delivered by trained teachers. The program aims to help students complete remediation and prepare for college-level courses. This article describes the results of an experiment at four community colleges (n ~ 3,800). We estimate that over 3 years, including one semester that students spent in the program and two-and-a-half years after the program was complete, CUNY Start substantially increased college readiness, slightly increased credit accumulation, and modestly increased graduation rates (by increasing participation in CUNY’s highly effective Accelerated Study in Associate Programs [ASAP]).

Keywords

community colleges educational reform postsecondary education program evaluation experimental design

Community colleges play an important role in U.S. higher education. In fall 2019, more than 5,000,000 students were enrolled in public 2-year colleges (National Student Clearinghouse Research Center, 2019). Most students who enter community college, however, are deemed academically underprepared for college-level courses and are referred to noncredit developmental (remedial) education courses to build their math, reading, or writing skills. One national study estimated that 59% of students entering community colleges were referred to developmental math courses, and 33% were referred to developmental reading courses (Bailey et al., 2010). These students struggle in the developmental education courses and in college more broadly. Nationwide, only 13% of students who take a developmental education course graduate within 3 years (U.S. Department of Education, National Center for Education Statistics, n.d.). Graduation rates are lower still among triple-remedial students, that is, students referred to developmental education across math, reading, and writing (see, for example, Ginder et al., 2015).

The low success rates among students referred to developmental education has sparked interest in developmental education reform. Although there is a growing body of experimental research on the effectiveness of such reform efforts, there remains a dearth of experimental evidence on programs that increase long-term outcomes (e.g., credits earned or degree attainment through 3 years) for students referred to developmental education (Bailey et al., 2016). Even less evidence exists regarding how to help triple-remedial students (Boatman & Long, 2018).

Hoping to improve the success rates of its least prepared students, the City University of New York (CUNY) developed CUNY Start. CUNY Start’s full-time program was designed for incoming students assessed as needing remediation in three subject areas: math, reading, and writing. CUNY Start lasts one semester, and its short-term goal is to help students fulfill their developmental education requirements, while preparing them for college-level courses. Its long-term goal is to improve academic outcomes, including graduation rates. Students enrolling in CUNY Start’s full-time program defer college matriculation—which starts when students first enroll in degree programs—for one semester to engage in time-intensive instruction in math, reading, and writing with a prescribed instructional approach. Students in CUNY Start’s part-time program defer matriculation for one semester to engage in time-intensive instruction in math or in reading and writing. CUNY Start also provides advising, tutoring, and a weekly seminar that teaches students skills they need to succeed in college. Students pay only US$75 for the program.

Among college developmental education reforms, CUNY Start is uncommon in that it explicitly aims to recruit triple-remedial students. CUNY Start’s services are also uncommon with respect to their greater level of time intensity—with 26.5 hours of class time required per week for the full-time program and 13.5 hours of class time required per week for the part-time program—and their curricular and pedagogical focus, including a detailed, standardized curriculum and a semester-long apprenticeship period for instructors new to CUNY Start. Add to this its low cost to students and integrated student support services, and the fact that it occurs before students matriculate, and CUNY Start is clearly an innovative approach to grappling with the low success rates of students referred to developmental education.

This article presents 3-year findings from a randomized controlled trial (RCT) on the effectiveness of CUNY Start. The evaluation was designed to answer the following questions:

Research Question 1 (RQ1): What is the effect of the offer to participate in CUNY Start on students’ academic progress and completion, compared with that of participating in the colleges’ standard courses and services (including the colleges’ standard developmental education courses)? Do the effects vary among student populations and settings?

Research Question 2 (RQ2): How is CUNY Start implemented, to what degree is it implemented with fidelity to the CUNY Start model, and to what degree are there differences between the program and the colleges’ standard courses and services?

Research Question 3 (RQ3): What are the costs associated with CUNY Start; how do the costs compare with the costs of the standard courses and services; and is CUNY Start cost effective?

This article focuses on the 3-year effects of CUNY Start on students’ academic progress and completion (RQ1). It briefly summarizes the implementation and cost findings (RQs 2 and 3). Detailed implementation findings are discussed in Scrivener et al. (2018), and detailed cost analyses are discussed in Community College Research Center (forthcoming).

Developmental Education Reforms

There are a variety of student barriers and institutional practices that help explain the low completion rates of community college students in general, and those referred to developmental education programs in particular (for overviews, see Baum et al., 2013; Braxton, 2002; Calcagno et al., 2008). Although it is beyond the scope of this article to review all the obstacles, we briefly describe those that CUNY Start aims to address, noting reforms, many with features related to CUNY Start, that attempt to address these barriers. The barriers include the way developmental education courses are taught (i.e., content and pedagogy), the time it takes to complete these courses, a shortage and underutilization of student support services, and financial issues. There is also a question as to whether developmental courses are of value at all.

Content and Pedagogy

Many developmental education classes focus on building discrete skills using decontextualized content (Grubb & Gabriner, 2013; Perin, 2013). Courses often use lecture-based pedagogy, in which students are recipients of information but have limited active learning opportunities (Rutschow & Schneider, 2011). Traditional lecture-based instruction, when compared with “active learning” approaches, may inhibit academic performance. Furthermore, these approaches often fail to engage and prepare students for college-level work (Grubb & Gabriner, 2013).

To improve pedagogy and content, many colleges have implemented multiple math pathways (i.e., sets of linked courses designed to teach students math content relevant to their degree requirements and program of study), self-paced learning models, and flipped classrooms (in which students are exposed to content outside of class, while most in-class time is devoted to activities, projects, and discussions; Rutschow & Mayer, 2018). By moving toward an instructional approach that positions students as active learners who regularly participate in small-group and whole-class activities, and teachers serve as facilitators of the learning process rather than lecturers, it is expected that students will be more engaged and develop deep and transferable knowledge, stronger academic identities, and an increased commitment to college. Past research suggests teaching approaches that encourage active learning are more effective (Stigler & Hiebert, 2009). Two recent experimental evaluations of interventions done with math pathways that change the content and/or pedagogy of developmental math found positive impacts on students’ successful completion of college-level math (Rutschow et al., 2019) and graduation rates (Logue et al., 2019).

Reducing the Time to Complete Developmental Courses

Taking multisemester noncredit developmental education course sequences may slow down students’ progress toward graduation (Bettinger & Long, 2005; CUNY Task Force on Developmental Education, 2016). These courses increase the number of requirements students must complete to earn a degree, increasing the time it takes them to earn a degree (Bettinger & Long, 2009).

To reduce the time it takes to complete developmental education requirements, colleges are implementing compressed coursework in math, integrated developmental reading and writing courses, and corequisite remediation (i.e., students take a college-level course concurrently with a developmental education course that supports their learning in the main course; Rutschow & Mayer, 2018). Nonexperimental research on offering developmental courses on a compressed time frame suggests improved progress through developmental education and increased course pass rates, grades, and rates of persistence (Rutschow & Schneider, 2011). A quasi-experimental study of the Accelerated Learning Program (ALP) at Community College of Baltimore County—a corequisite remediation intervention—found that students who participated in ALP had improved rates of courses attempted and completed (Jenkins et al., 2010). Furthermore, evaluations of community college and high school programs that incorporate significant increases in time spent in targeted subject areas provided evidence of the effectiveness of this approach at raising student success (Jenkins et al., 2010; Kemple et al., 2005; Nomi & Allensworth, 2009).

A Shortage and Underutilization of Student Support Services

Students referred to developmental education often benefit from receiving adequate forms of support to address nonacademic challenges and to develop the academic behaviors, metacognitive skills, and academic confidence associated with college success (Bailey et al., 2015; Scrivener et al., 2012). However, there is a broad student support challenge at community colleges—for example, the National Academic Advising Association estimates that the median caseload of an adviser at public 2-year colleges is 441 students (Robbins, 2013)—limiting the amount of advice students can receive.

There is a robust experimental literature on various forms of student support services. Experimental evidence on advising interventions finds positive, although often modest, effects on students’ academic progress with the use of a variety of modes of advising (Avery et al., 2014; Barr & Castleman, 2017; Bettinger & Baker, 2014; Carrell & Sacerdote, 2017; Evans et al., 2017; Oreopoulos & Petronijevic, 2016). Other student support strategies with some limited experimental evidence of efficacy include tutoring (see, for example, Sommo et al., 2014) and student success courses (see, for example, Weiss et al., 2011). There also is experimental evidence that learning communities, in which students in two or more classes form cohorts, can have a modest positive effect on student outcomes (Richburg-Hayes et al., 2008; Visher et al., 2012; Weiss et al., 2015).

Financial Issues

Setting aside developmental education, the cost of attending college is a challenge for many students (Deming & Dynarski, 2009). Taking noncredit course sequences leads students to incur additional tuition costs, and in the process, expend a portion of their financial aid (CUNY Task Force on Developmental Education, 2016). Furthermore, taking (and often repeating) noncredit course sequences can put students at risk of failing to meet satisfactory academic progress requirements for maintaining federal aid eligibility (Scott-Clayton & Schudde, 2017; U.S. Department of Education, Office of Federal Student Aid, n.d.b). Federal aid also has time limits: Students are only eligible to receive the equivalent of 6 years of Pell Grant funding (U.S. Department of Education, Office of Federal Student Aid, n.d.a). The additional time and aid required to complete developmental education requirements may inhibit academic progress. Numerous experimental evaluations provide causal evidence that finance-related forms can improve student academic progress, although often modestly (Angrist et al., 2010, 2014; Bettinger et al., 2012; Cohodes & Goodman, 2014; Deming & Dynarski, 2009; Goldrick-Rab et al., 2016; Mayer et al., 2015).

Broadly speaking, many developmental education reform strategies have yielded improved short-term outcomes for students (see, for example, Adelman, 2004; Attewell et al., 2006; Bailey et al., 2010, 2016; Jenkins et al., 2009; Rutschow & Schneider, 2011). Yet, when the U.S. Department of Education’s What Works Clearinghouse sought to create a practice guide for serving students referred to developmental education, only six strategies were recommended—none with a “strong” evidence base, three with a “moderate” evidence base, and three with a “minimal” evidence base (Bailey et al., 2016). There is a need for more rigorous evidence on the effects of programs designed to improve the outcomes of students referred to developmental education.

Referral to Developmental Education

Developmental education courses themselves may be an impediment to academic progress. Valentine et al. (2017) conducted a systematic review of the effects of referral to developmental education compared with referral to college-level courses. They meta-analyzed data from 11 regression discontinuity design (RD) studies at 2- and 4-year institutions. On average, they found that referral to developmental education had a negative effect on (a) students’ probability of passing the college-level course in which remediation was “needed,” (b) college-level credits earned, and (c) degree attainment. Strictly interpreted, these RD studies apply to students who were “on the margin” of college readiness; however, it is likely that results apply to students falling within at least 0.5σ of the cut score (Bloom et al., 2020). Moreover, very recent evidence from an experimental evaluation of a multiple measures placement system further suggested that referral to developmental education negatively affects students’ academic progress (Barnett et al., 2020).

The combined weight of the evidence indicates that for the populations where causal evidence exists, on average, referral to developmental education negatively affects students’ academic progress. For students scoring well below placement test cut scores who do not have high school grade point averages (GPAs) suggesting college readiness, much less is known about the relative effectiveness of referral to developmental versus college-level coursework.

Developmental Education Reform at CUNY

About 80% of first-time CUNY associate degree-seeking freshmen are assessed as needing remediation (CUNY Office of Academic Affairs, 2015). During this study, this assessment was based primarily on students’ scores on the CUNY Assessment Tests in math, reading, and writing (see the subsection “Data and Measures of Academic Outcomes” for more information on how CUNY assessed students’ skills). At CUNY’s community colleges, the 3-year graduation rate for first-time, full-time students referred to remediation in at least one subject was 14.2%, which applies to students who began school in fall 2011. This rate drops to just 6.5% for triple-remedial students MDRC calculations based on data from CUNY; CUNY Office of Institutional Research and Assessment, 2015).

As is the case at most community colleges, developmental education at CUNY has typically comprised multilevel, multisemester noncredit course sequences in math, reading, and writing. Students are required to take up to three semesters of developmental education courses. Most math courses are taught using lecture-based direct instruction with little emphasis on academic skill building (e.g., studying and note taking), and reading and writing are typically taught separately (see Table 2 for more information).

As reforms have been explored nationwide, CUNY’s leaders have also focused on rethinking the university system’s approaches, and CUNY and its colleges have implemented many different changes. For example, during the period of this study, CUNY was working to revamp its developmental education, introducing alterations in how students are assessed and referred to courses, as using a single assessment has been critiqued (see, for example, Scott-Clayton, 2012); in how students qualify to move out of developmental education; and in the content and pedagogy of the courses themselves. CUNY Start is part of these reform efforts as a prematriculation intervention. It is unique in its targeting of triple-remedial students and students assessed as having significant developmental education needs.¹

The CUNY Start Model

CUNY Start began in 2009 as a small, intensive program that targeted students with General Educational Development (GED) certificates (now, high school equivalency diplomas) who had been assessed as having very weak math, reading, and writing skills. It expanded in 2010 to also serve high school graduates.

Eligible Population

Students whose placement test scores result in their referral to remediation in math, reading, or writing are eligible for CUNY Start.² CUNY Start offers a full- and a part-time program. The full-time program is only open to students referred to remediation in math and at least one other subject area (reading, writing, or both). The part-time program, which provides instruction in math or reading and writing, is open to students referred to remediation in one or more subjects. Although any student who is referred to remediation is eligible for CUNY Start (and was eligible for the evaluation), CUNY Start focuses on recruiting triple-remedial students. Colleges also prioritize recruiting students referred to two developmental math courses.

Program Components

Administration, Cost, and Structure

At the time of the study, the CUNY Start programs were situated in each college’s continuing education division, which offers a range of courses outside the academic departments. Administrators in CUNY’s central Office of Academic Affairs (OAA) manage the program, working closely with CUNY Start directors at each college. Professional development staff based within CUNY’s OAA provide training to CUNY Start instructors and advisers and serve as primary curriculum developers.

CUNY Start students pay only US$75 for the semester, which includes the cost of all course materials. Because the program is offered prior to matriculation, students preserve their financial aid for future semesters.

CUNY Start uses a cohort model in which students co-enroll in math, reading, and writing courses, and a student success seminar, like some learning communities programs. Such grouping is hypothesized to foster stronger connections among students and between students and instructors (Visher et al., 2008).

CUNY Start’s program is time intensive, enabling students to complete remedial requirements in a single semester. The full-time program provides up to 26.5 hours of instruction per week during its one semester: 12 hours of math, 12 hours of integrated reading and writing, and 1.25 to 2.5 hours in the college success seminar.³ CUNY Start’s part-time program provides 12 hours of instruction in math or reading and writing, and 1.5 hours in the seminar. Students do not take college-level courses while in CUNY Start. This instructional intensity enables a dramatic increase in the amount of time that students spend actively engaged with learning activities such as problem-solving.

Math Instructional Approach

The core of CUNY Start is time-intensive instruction in math and reading/writing using standardized curriculum and pedagogy, which were created by experienced faculty members and CUNY Start professional development staff members. Curricula are regularly refined by the CUNY Start professional development team based on instructor feedback. The math curriculum integrates arithmetic and algebra. Various research on math instruction with adults and older adolescents (U.S. Department of Education, 2011) and national college readiness math standards (National Council of Teachers of Mathematics, 2000; National Governors Association Center for Best Practices, 2010; National Mathematics Advisory Panel, 2008) recommends that math instruction focus on developing students’ math proficiencies (e.g., problem-solving and adaptive reasoning) and conceptual understanding (e.g., comprehension of math concepts, operations, and relations). The CUNY Start math curriculum models these tenets by emphasizing conceptual understanding, real-world learning, and building academic skills such as studying and note taking.

CUNY Start’s math pedagogy is designed to give students a deep understanding of math concepts through student-centered instruction, requiring instructors to limit their use of lectures and, instead, promote student discussion. Rather than the traditional remedial pedagogy, characterized by rote procedural learning and lecture, CUNY Start’s math pedagogy positions students as active participants in their own learning, gives them time to think about and struggle with the concepts, and encourages them to speak and respond to each other. Central to the pedagogy is the technique of questioning, in which instructors ask specific open-ended questions to stimulate student thinking and discussion. Such features are intended to increase the relevancy of subject matter in service of deeper understanding and knowledge transfer (Perin, 2013), which have been shown to have positive effects on student learning (Hiebert et al., 2003; Stein et al., 2009).

Reading/Writing Instructional Approach

CUNY Start teaches reading and writing integrated in one class to reinforce the relationship between the two and to allow students to move more quickly through their developmental education requirements. To engage students, the curriculum uses a range of culturally relevant texts, including nonfiction articles and essays in the beginning of the semester, and short stories and longer works of fiction later in the semester.

Like CUNY Start math instructors, CUNY Start reading/writing instructors rely on student-centered instruction. Research on reading and writing instruction suggests that students should receive explicit instruction in reading comprehension and writing strategies, which are modeled and scaffolded by instructors with a gradual release of responsibility to students through repeated practice (Graham & Hebert, 2010; Graham & Perin, 2007a, 2007b; National Reading Panel, 2000b). These practices are further underscored by K–12 national language arts standards, which emphasize a development of students’ conceptual understanding, critical thinking skills, and ability to apply learned knowledge in real-world settings (National Governors Association Center for Best Practices, 2010).

To emphasize these techniques, CUNY Start reading/writing instructors specifically employ the cognitive apprenticeship model, in which instructors aim to help students learn the habits and techniques of proficient readers and writers (Collins et al., 1988). For example, instructors sometimes “think aloud” to model the processes used by proficient readers and writers—such as the process of identifying evidence to support a claim or theme, to identify the author’s point of view, or to identify how specific parts of a story are connected to others.

Student Support

CUNY Start’s student support is integrated into the classroom. The mandatory college success seminar, led by a CUNY Start adviser, aims to help students develop skills to balance school and life, solve problems, advocate for themselves, and view themselves as learners. CUNY Start advisers, who are responsible for far fewer students (about 75 students) than non-CUNY Start advisers (about 600 students), also meet with students outside the seminar to give them support during the program and to plan for their matriculation. CUNY Start math tutors and writing assistants provide help to students inside and outside of class.

Instructor Hiring and Training

CUNY Start hires instructors based on their content and pedagogical knowledge, and their openness to the prescribed curriculum and pedagogy. Once hired, CUNY Start instructors participate in a semester of apprenticeship and training before they teach their own CUNY Start classes. During this period, they participate in periodic training lead by CUNY Start professional developers and are assigned to the classroom of an experienced CUNY Start instructor on a full-time basis to observe and eventually assist with instruction to learn about the CUNY Start curriculum and pedagogy. After that semester, all instructors continue to receive ongoing training, classroom observation, and feedback from CUNY Start professional developers.⁴ A growing body of evidence on faculty development in higher education suggests that models that embed professional learning opportunities into instructor’s work portfolios and are tied closely to everyday practice can increase faculty buy-in (Bickerstaff & Edgecombe, 2012; Bragg & Barnett, 2009; Edgecombe et al., 2013).

Theory of Change

CUNY Start’s components are designed to work together to improve students’ outcomes. The math and reading/writing curricula and pedagogies are central to the model; the instructional intensity, instructor training and professional development, and student services are critical to facilitating and supporting the instruction (see Bickerstaff & Edgecombe, 2019, for a discussion of how the program components interact).

CUNY Start’s underlying theory of change posits that students with substantial developmental education requirements are best served through a time-intensive, cohort-based model, in which a group of students who join the program at the same time take all of their classes together. It is designed to build academic preparedness and college skills before matriculation. The program is intended to make students more engaged in their coursework, help them view themselves as learners, help them make greater connections with their peers, give them the support they need to succeed, and prepare them for college-level work. The program’s low cost to students is expected to make it easier for them to participate, and the fact that the program is provided before students’ matriculate allows them to preserve their financial aid for future courses.

The program’s designers hypothesize that compared with students in standard college courses, a higher proportion of CUNY Start students will build their basic skills and complete any required developmental education, and that they will do so more quickly. Because CUNY Start students spend a semester building their basic skills before matriculating, they are expected to earn fewer college credits in the short-term. The hypothesis, however, is that if greater amounts of CUNY Start students complete their developmental education and become college ready more quickly, over the longer term, they will remain enrolled at higher rates, accumulate more college-level credits, and graduate more frequently.

Study Setting

The CUNY system consists of 25 institutions, including community colleges, 4-year colleges, and graduate and professional institutions; it serves over a quarter-of-a-million matriculating students each semester (CUNY, n.d.a). In spring 2015, when the study began, CUNY Start was operating at seven CUNY colleges (at the time this article was written, it was operating at 10). Of those seven, the evaluation took place at four: Borough of Manhattan Community College, Kingsborough Community College, LaGuardia Community College, and Queensborough Community College. These colleges were chosen because each had operated the program for at least 2 years, had sufficient program infrastructure to continue operating it, and was willing and able to participate in a random assignment evaluation. These are also the four largest CUNY community colleges by total enrollment.

Borough of Manhattan Community College is in lower Manhattan. It served about 29,000 students when the study started, including students in credit programs leading to a degree and students in continuing education programs. Kingsborough Community College is in Brooklyn. It served about 25,000 students at the start of the study. LaGuardia and Queensborough community colleges are both located in Queens and served about 36,500 and 19,000 students, respectively, at the start of the study (CUNY Office of Institutional Research and Assessment email to author, January 3, 2018; CUNY Office of Institutional Research and Assessment, 2018). At Borough of Manhattan and Queensborough community colleges, most students take full-time course loads, whereas about half of Kingsborough and LaGuardia students do. All four colleges offer a wide range of associate’s degree programs that prepare students to transfer to 4-year colleges or enter professional careers. As mentioned previously, at the time of the study, CUNY Start was housed in each college’s continuing education division.

Method

Recruitment and Random Assignment

CUNY Start has a robust set of outreach strategies for recruiting students after they have applied and been accepted to CUNY. Before each semester, CUNY Start staff members contact eligible students by email, mail, and phone, and attend college orientation and registration events to describe the program and, during this study, the evaluation. Interested students attend CUNY Start information sessions at the colleges to learn more. Students then participate in a structured intake process with CUNY Start staff members to help them understand the program’s structure and time demands. The goal is for students to make an informed decision about participation (Kim et al., in press).

During the evaluation, students who agreed to participate in CUNY Start and the study signed an informed consent form and were then randomly assigned to either the program group or control group in real time via a web portal controlled by the research team. Program group students were offered the opportunity to participate in CUNY Start. Control group students were offered the opportunity to participate in all their colleges’ other programs and services, just not CUNY Start (see Table 2 for more information on the standard college courses and services). Three cohorts of students are part of the study, one prior to the spring 2015, fall 2015, and spring 2016 semesters. Immediately prior to random assignment, all students indicated whether they intended to participate in CUNY Start’s part- or full-time program if they were assigned to the program group (this does not reflect whether they intended to enroll part- or full-time if assigned to the control group).

The analytic sample includes 3,835 individuals (see detailed discussion of the analytic sample in the section “Characteristics of the Sample”). The probability of being assigned to the program group was 78%, on average. This uneven random assignment ratio was, by design, chosen to minimize the burden on program staff (who had to overrecruit) and to increase the likelihood that all program slots would be filled, while ensuring the research goals could be met (e.g., achieving adequate statistical power). Also, by design, the probability of being assigned to the program group was slightly greater in the spring than in the fall.⁵ Around 81% of program group members participated in CUNY Start and less than 1% of the control group participated. We focus analyses on estimating the effect of the program offer (i.e., intention to treat). If someone is interested in the effect of CUNY Start on those who complied with their assignment (i.e., the complier average causal effect), they can take the presented estimated effects and divide by 0.80 (Angrist et al., 1996; Bloom, 1984; Imbens & Angrist, 1994).

Estimation Model

We use the following estimation model to obtain regression-adjusted estimates of the average causal effect of the offer to participate in CUNY Start, among people in the evaluation:

Y = \sum_{j = 1}^{J} α_{j} \times R_{j} + \sum_{c = 1}^{C} γ_{c} \times X_{c} + β \times T + ε

(1)

Here, Y represents a target outcome (e.g., credits earned); R_j is a vector of J random assignment block indicators (one for each unique college by cohort by intention to enroll part vs. full-time); X_c is a vector of student baseline characteristics, including pretest scores, random assignment date, and binary indicators for any missing characteristic; and T is set equal to one if a student was assigned to the program, and zero otherwise. Robust (Huber-White) standard errors are used in all analyses. Analyses for all academic outcomes at all time points presented include all 3,835 students, unless otherwise noted (e.g., subgroup analyses).

Data and Measures of Academic Outcomes

The main data source for academic outcomes was CUNY’s Institutional Research Database (IRDB). From the IRDB, we received CUNY Start application data (including demographic and other background information);⁶ CUNY Start participation data, skills test data, transcript data (including the courses that students enrolled in and their grades); and degree data (including credentials earned by students at CUNY institutions). We also received data from the National Student Clearinghouse (NSC), which includes enrollment and credentials earned by students at any of the postsecondary institutions that report to the NSC. NSC data cover 99% of college enrollments across all postsecondary institutions in the United States (National Student Clearinghouse, n.d.). IRDB and NSC data were received for all students in the study who appeared in the relevant data. The data covered, at a minimum, spring 2015 (the program semester for the first cohort) through winter 2019 (3 years after the third cohort’s program semester).

Several academic outcome measures were created to evaluate CUNY Start’s effects on academic progress and credential completion, as will be described.

Enrollment

A student was considered enrolled if that student participated in CUNY Start or matriculated in a CUNY institution according to IRDB data or matriculated in a non-CUNY institution according to NSC data.

College Readiness

Entering students at CUNY’s colleges can demonstrate that they are prepared for college-level courses in several ways. They can submit scores from their SAT, ACT, or New York Regents’ exams, which are statewide standardized exams in core high school subjects. Students can also take tests administered by CUNY in math, reading, and writing, known as the CUNY Assessment Tests.⁷ Finally, individual colleges can grant students exemptions on a case-by-case basis to allow them to take college-level courses in each subject area. Students in developmental education courses were generally required to pass exit tests to demonstrate their proficiency.⁸ In our analysis, students were considered college ready in a given subject area (math, reading, or writing) if they passed a placement or exit exam in that subject area, passed the highest level course in that subject area’s developmental education course sequence, or passed a college-level course (including corequisite courses) in that subject area at a CUNY institution, according to IRDB data.⁹

Completion of Gateway Courses

A gateway course is defined as the first college-level math or English course a student completes. We focused on CUNY courses that may meet CUNY’s general education (Pathways) requirements in math and English.¹⁰ A student was considered to have completed a gateway course if that student received a D or better in a course at a CUNY institution, according to IRDB data.

Cumulative College-Level Credits

College-level credits were defined as credits earned at a CUNY institution, according to IRDB data, that may be applied to satisfy CUNY’s graduation requirements. Credits were considered earned if a student received a D or better in a course, and credits were considered attempted if a student remained enrolled in the course through the census date. Credits associated with remedial education courses were not included in college-level credits because they are not degree applicable.

Credential Completion

A credential was defined as a certificate, associate’s degree, bachelor’s degree, or graduate degree earned at a CUNY institution according to IRDB data or at a non-CUNY institution according to NSC data.

Outcomes were calculated for each semester. Borough of Manhattan and Queensborough community colleges generally have 14-week main sessions (i.e., during the fall and spring) and 4- to 8-week intersessions (i.e., during the winter and summer), while Kingsborough and LaGuardia generally have 12-week main sessions and 6-week intersessions. For all four institutions, we combined each main session with its subsequent intersession to create one semester. Thus, there are two semesters per year at each college, with one semester generally beginning in late August or early September and another beginning between late January and early March.

Three key confirmatory outcomes were specified prior to the start of the study. Confirmatory outcomes provide answers to sharply focused research questions that address “the study’s pre-specified central hypotheses” (Schochet, 2008, p. 3). Number of college-ready subject areas is our confirmatory measure of college readiness. It captures progress and completion of developmental education course requirements—key milestones toward degree completion and the direct goal of the program. Cumulative college-level credits earned is a confirmatory measure of student academic progress. It is a proxy for progress toward a degree because students typically must earn at least 60 college-level credits to earn an associate’s degree. Finally, completion of a degree or certificate is a confirmatory outcome. It is the long-term outcome in which decision-makers are often most interested.¹¹

Subgroups

Subgroup analyses were conducted to examine whether CUNY Start’s impacts varied based on four student characteristics (for additional subgroups, see Supplemental Appendix in the online version of the journal).

Full-Time Versus Part-Time

We estimated effects for students who intended to enroll in the full-time program, which included up to 26.5 hours of instruction time per week, and for students who intended to enroll in the part-time program, which included 13.5 hours of instruction time per week. This difference in intervention experience could lead to differential effectiveness. Notably, a student’s intended level of attendance in the CUNY Start program was only weakly correlated with their observed enrollment intensity in non-CUNY Start courses.¹² Furthermore, triple-remedial students were more likely to express an intention to participate in the full-time CUNY Start program, so this characteristic is somewhat conflated with the amount of remedial need.

Triple-Remedial Versus Fewer Remedial Requirements

Number of college-ready subject areas at baseline was examined because it is an indicator of a student’s level of risk of not achieving the study’s confirmatory outcomes, an important subgroup to consider in almost any study, with equity implications. Moreover, this subgroup analysis was motivated by the policy importance of understanding whether programs can make a difference for community college students with the greatest assessed developmental education needs, an understudied group. In addition, past research has found differential program effectiveness based on the number and types of remedial referrals (Boatman & Long, 2018; Hodara, 2015).

Intended College

We examined whether program effects vary by college because it is a key step in understanding the replicability of a program and the generalizability of its effects. All else being equal, a program with consistently positive effects across sites is more likely to be effective in similar contexts beyond the study than a program with large differences in effects across sites. Recent evidence across the education spectrum has shown that some interventions’ effects can vary substantially across settings (Weiss et al., 2017).

Race/Ethnicity

Finally, we considered whether effects vary by race and ethnicity. Race is of importance because Black and Hispanic students have traditionally been underrepresented in postsecondary education; they enroll disproportionately in community colleges, relative to public universities; and persistent race-based achievement gaps are a policy priority (McFarland et al., 2019).

Characteristics of the Sample

There were 42,232 incoming students at the study colleges during the study’s program semesters (spring 2015, fall 2015, and spring 2016). Among those students, 32,462 were assessed as having initial remedial needs or had an unknown remedial status, making them technically eligible for CUNY Start. There were 4,434 CUNY Start applicants, 41 of whom requested to be removed from the study or did not return a signed informed consent form, leaving 4,393 CUNY Start applicants for whom we have data. One meaningful difference between the CUNY Start-eligible population and CUNY Start applicants is that the applicants were about 30 percentage points more likely to have been triple-remedial (and, thus, were more likely to require remediation in each of the three subject areas; see Supplemental Table A1 in the in the online version of the journal). This is consistent with the program’s targeted focus on recruiting students assessed as having the most remedial needs. Otherwise, the observed characteristics of CUNY Start applicants were similar to the eligible population.

Among the 4,434 students who applied to participate in CUNY Start, 3,873 were randomly assigned. The other 561 students (13% of applicants) were “exempted,” meaning they were guaranteed a spot in CUNY Start.¹³ Among the students who were randomly assigned, 38 (0.8% of program group students and 1.6% of control group students) were removed from the analytic sample either because they asked to withdraw from the study or because we did not receive their signed informed consent forms.¹⁴ The analytic sample consisted of 3,835 students, including 2,997 program group students and 838 control group students.

Characteristics of program and control group members at the time of random assignment are shown in Table 1. The differences between the two research groups and the p values associated with those differences are provided. In Table 1, “Gender,” “Race/ethnicity,” and “Native language” were derived from CUNY Start application data received prior to random assignment. Additional application data that were received by CUNY after random assignment were excluded and treated as missing due to the possibility (however unlikely) that their collection was dissimilar across the two research groups.

Table 1

Baseline Characteristics of Evaluation Sample, by Research Group

Characteristic (%)	Program group	Control group	Difference	p value
Gender				.216
Female	48.3	46.2	2.1
Male	36.7	40.0	−3.3
Missing	15.0	13.8	1.2
Age (years)				.691
19 or younger	47.6	48.6	−1.0
20–23	30.4	28.9	1.5
24 or older	22.0	22.5	−0.5
Race/ethnicity^a				.278
Hispanic	35.9	36.9	−1.0
White	5.9	5.7	0.2
Black	25.3	27.4	−2.1
Asian or Pacific Islander	9.1	6.8	2.3
Other^b	6.4	7.0	−0.6
Missing	17.5	16.3	1.1
Native language				.620
English	46.5	49.0	−2.6
Spanish	18.6	17.9	0.7
Other	19.6	18.7	0.8
Missing	15.4	14.4	1.0
Number of subject areas requiring developmental education				.777
Zero (fully college ready)	0.2	0.4	−0.2
One	15.2	15.4	−0.2
Two	33.4	33.2	0.2
Three	51.2	51.0	0.2
Required developmental education in the subject area
Math	94.6	94.2	0.3	.696
Reading	63.4	64.3	−0.9	.637
Writing	77.7	76.4	1.3	.419
Intended college^c				1.000
College A	41.6	41.6	0.0
College B	16.5	16.5	0.0
College C	30.4	30.4	0.0
College D	11.5	11.5	0.0
Intended CUNY Start program^c				1.000
Full-time	61.4	61.4	0.0
Part-time	38.6	38.6	0.0
Sample size (total = 3,835)	2,997	838

Source. CUNY Start application data, MDRC random assignment data, and test data from CUNY’s Administrative Data Warehouse.

Note. Values are weighted to account for random assignment ratios across random assignment blocks. Rounding may cause slight discrepancies in sums and differences. CUNY = City University of New York.

Respondents who selected “Hispanic” for their ethnicity and chose another race category are included only in the “Hispanic” category. Respondents who did not select “Hispanic” for their ethnicity and chose more than one racial category are included in the “Other” category. ^b“Other” includes multiracial, Native American/Alaskan native, and other racial/ethnic categories. ^cRandom assignment occurred within blocks by cohort, intended college, and intended program.

The analytic sample included more women than men and was divided almost equally between students aged 19 years or younger and students aged 20 years or older. Black and Hispanic students made up most of the sample, and a substantial proportion of the sample had a native language other than English. The sample was split almost evenly between students who had developmental education requirements in all three subject areas and those with fewer requirements, with more students having such requirements in math than in reading and writing. In addition, fewer students scored near the threshold to be considered college ready in math than those who scored near the threshold in reading and writing, indicating that students in the sample not only were more likely to have a developmental education requirement in math but also had deeper requirements in math than in the other two subject areas.¹⁵ Overall, 61% of the sample intended to participate in the full-time CUNY Start program. Two colleges (Colleges A and C) accounted for 72% of the evaluation sample.¹⁶

There were no meaningful differences between program and control group members on any of the baseline characteristics shown in Table 1, nor on any of the baseline characteristics provided in Supplemental Table A2 in the in the online version of the journal. Furthermore, an omnibus F test showed that students’ baseline characteristics were not jointly predictive of their research group (p = .885), indicating that program and control group students were not systematically different on observable characteristics. Random assignment should also produce research groups that are similar on unobserved characteristics, such as learning style and tenacity.

Program Implementation and Service Contrast

From spring 2015 to spring 2016, the evaluation team collected information on the implementation of CUNY Start and the standard offerings at the four colleges using several data sources, including interviews with administrators and instructors, observations of classrooms, and surveys of students¹⁷ and instructors. Overall, CUNY Start was implemented with fidelity to the program model, and there was a substantial contrast between the program and the colleges’ standard courses and services, including their standard developmental education courses. Using data from the implementation research, Table 2 compares components of the program with those of the standard college courses and services, and this section briefly summarizes those differences (detailed implementation findings are available in Scrivener et al., 2018).

Table 2

Highlights of CUNY Start and Standard College Courses and Services

Component	CUNY start	Standard college courses and services
Administration, cost, and structure
Administration	Situated in continuing education	Situated in the academic affairs division
Cost to student	US$75; students not eligible for financial aid	Full-time tuition US$2,400; most students eligible for financial aid
Structure	One semester of developmental math, reading, and writing; could not take college-level courses that semester; up to 26.5 hours of instruction per week in the full-time program	Up to three semesters of developmental math, reading, and writing; students could take some college-level courses; 12 to 16 hours of instruction per week for a full-time student
Developmental math
Curriculum	Arithmetic and algebra integrated; emphasizes conceptual understanding; assignments include activities that develop students’ academic skills	Arithmetic and algebra taught separately; academic skill-building activities not prevalent
Pedagogy	Mostly student-centered instruction	Mostly lecture-based instruction
Developmental reading/writing
Curriculum	Reading/writing integrated; writing assignments designed to help students process and respond to reading material	Reading/writing typically not integrated; writing assignments in upper-level courses include research synthesis papers
Pedagogy	Mostly student-centered instruction	Mostly student-centered instruction
Student support
College success seminar	Mandatory; 78% of surveyed students took a seminar	Typically not mandatory; 29% of surveyed students took a seminar
Advising	Student-to-adviser ratio was 75:1; 82% of surveyed students had at least one one-on-one advising session; median student reported two sessions	Student-to-adviser ratio was ~ 600:1; 65% of surveyed students had at least one one-on-one advising session; median student reported one session
Tutoring	45% of surveyed students received tutoring; median student reported three tutoring sessions	34% of surveyed students received tutoring; median student reported four tutoring sessions
Instructor hiring and training
Hiring	Instructors hired based on content and pedagogical knowledge and openness to CUNY Start instructional approach	Instructors typically hired based on content knowledge
Training	Most instructors participated in an apprenticeship; continuing professional development was common and regular	Most instructors not trained up-front; continuing professional development was common but less regular and intensive

Source. Field research data; instructor and student survey data; CUNY (n.d.c); discussions with CUNY staff members.

Note. CUNY = City University of New York.

Administration, Cost, and Structure

CUNY Start and standard developmental education courses are situated in different parts of the colleges. Students pay far less for CUNY Start as a prematriculation intervention than for courses that charge standard tuition and fees.

CUNY Start provides many more hours of classroom instruction than the standard course offerings for students referred to developmental education. As mentioned above, CUNY Start’s full-time program provides up to 26.5 hours of instruction per week and the part-time program provides 13.5 hours of instruction. In contrast, students who are not in CUNY Start might take multiple developmental education courses over multiple semesters, with each course typically meeting 3 to 6 hours per week. Students can take some college-level courses at the same time, and a full-time student usually receives 12 to 16 hours of instruction per week.

Math Instructional Approach

CUNY Start’s math instructional approach is markedly different from standard developmental math instruction. Standard developmental math courses typically teach arithmetic and algebra separately and academic skill-building activities are not prevalent. Instructors in standard remedial math classes tend to use more lecturing than CUNY Start math instructors, who rely more on questioning and student discussion (for more detail, see Bickerstaff & Edgecombe, 2019).

Reading/Writing Instructional Approach

Standard developmental reading and writing were not integrated in three of the four colleges in the study, and the courses did not tend to use the cognitive apprenticeship model. In both CUNY Start and standard developmental reading and writing, instructors rely on student-centered instruction, so the pedagogical difference between CUNY and non-CUNY Start instruction is less substantial in reading and writing than it is in math.

Student Support

CUNY Start’s student support is more integrated into the classroom than is typical at community colleges, and, overall, program group students received somewhat more support than control group members. Typically, non-CUNY Start students are not required to participate in a college success seminar. Results from a survey of students during their first semester in the study found that 78% of program group students took a seminar, compared with 29% of control group students. In the survey, 82% of program group students reported that they had at least one one-on-one advising session, compared with 65% of control group students. The median program group student reported attending two advising sessions, and the median control group student reported attending one session—a small difference. Overall, 45% of program group members reported that they received tutoring, compared with 34% of control group students. The median program group student reported meeting with a tutor 3 times, and the median control group student reported meeting with a tutor 4 times.¹⁸

Hiring and Training

Typically, college instructors are hired primarily based on their content knowledge (and, of course, their academic credentials). In contrast, as mentioned above, CUNY Start hires instructors based on their content and pedagogical knowledge, teaching experience, and openness to the prescribed curriculum and pedagogy, and it provides a great deal of training to those hired. Most CUNY Start instructors who were surveyed for the study had participated in apprenticeships before teaching a course for the first time, and almost all reported receiving comments on their instruction. In contrast, most non-CUNY Start developmental education instructors who were interviewed did not report participating in training before teaching their first course. Many reported receiving some kind of professional development, but they participated in it for fewer hours than CUNY Start instructors.

Program Effects on Academic Outcomes

This section presents estimates of the effect of the opportunity to participate in CUNY Start on students’ academic progress and completion during the 3 years after they entered the evaluation, including one semester when CUNY Start services were offered to program group members (program semester) and 2.5 years after the program was complete (postprogram semesters). We focus on the students’ progress and completion of developmental education, college-level credit accumulation, and degree completion. For detailed tables, see the Supplemental Appendix in the online version of the journal.

Summary

The key findings are summarized in Table 3. The first two columns show the regression-adjusted average outcomes for the program and control groups (with the standard deviations of continuous outcomes in parentheses below the average outcomes). The third column presents the difference between the means for the two groups, which is the estimated average effect of being offered the opportunity to participate in CUNY Start. The last two columns represent the lower and upper bounds of a 90% confidence interval for the estimated effect.

Table 3

Impact Estimates on Confirmatory Outcomes After 3 Years

Outcome	ProgramGroup	ControlGroup	Difference	90% CI
Outcome	ProgramGroup	ControlGroup	Difference	LB	UB
College-ready subject areas (out of 3)	2.25	1.99	0.26	0.20	0.33
College-ready subject areas (out of 3)	(1.05)	(1.10)
Cumulative college credits earned	22.37	21.02	1.35	−0.16	2.86
Cumulative college credits earned	(25.32)	(24.30)
Degree or certificate completion (%)	14.5	11.4	3.1	1.0	5.1
Sample size (total = 3,835)	2,997	838

Source. MDRC calculations from CUNY Institutional Research Database (IRDB) and National Student Clearinghouse (NSC) data.

Note. Standard deviations of continuous outcomes are in parentheses. Rounding may cause slight discrepancies in sums and differences. Estimates are adjusted by random assignment blocks and select baseline characteristics. CI = confidence interval; LB = lower bound; UB = upper bound; CUNY = City University of New York.

At the end of 3 years, CUNY Start increased the number of subject areas in which students were college ready by an estimated 0.26 subjects (p < .001)—a 13% increase over the control group average of 1.99 subject areas in which students were college ready. This finding is consistent with the more easily interpretable result that CUNY Start increased the proportion of students who were college ready in all three subject areas by 16 percentage points (p < .001)—a large effect.

The estimated effect of CUNY Start on cumulative college-level credits earned was a modest 1.35 credits (p = .140), with a 90% confidence interval ranging from −0.16 to 2.86 credits. As intended by design, program group students earned fewer credits in the first semester after random assignment (the program semester) and then caught up and surpassed the control group during the subsequent 2.5 years.

After 3 years, 14.5% of program group students had earned a degree or certificate, compared with 11.4% of control group students (p = .015). As described later in further detail, CUNY Start enabled more program group students to become eligible for and participate in CUNY’s Accelerated Study in Associate Programs (ASAP), a separate and highly effective 3-year program. Most of CUNY Start’s graduation effect is likely due to this increase in ASAP participation. We proceed by examining these findings in more detail.

Enrollment

We begin by looking at CUNY Start’s effect on enrollment because it provides important context for all subsequent findings. Figure 1 shows enrollment in CUNY Start or any college across 3 years, or six semesters, for students offered CUNY Start (i.e., the program group, squares) and students not offered CUNY Start (i.e., the control group, circles). The estimated effect of CUNY Start, in percentage points, is plotted in diamonds with bars above and below that represent 90% confidence intervals.

Figure 1.

Enrollment in CUNY Start or any college, by semester.

CUNY Start increased enrollment in the first, second, and third semesters after random assignment by 10.6, 6.4, and 5.2 percentage points, respectively (p values ranging from <.001 to .008). The effects on enrollment in the fourth, fifth, and sixth semesters were smaller—3.1, 1.7, and 1.1 percentage points, respectively (p values ranging from .106 to .571).¹⁹ Program group students may have enrolled at higher rates than control group students in the program semester because they found CUNY Start more appealing than business as usual, as it offered them the possibility of making substantial progress through developmental education at a low cost and without dipping into their financial aid, or because CUNY Start provided them with resources to make registration easier.

Progress Through Developmental Education

CUNY Start had a substantial positive impact on students’ progress through developmental education. Figure 2 plots the confirmatory outcome Number of college-ready subject areas across 3 years.

Figure 2.

Progress through developmental education, by semester.

At the end of the first semester (the program semester), program group students were college ready in 1.87 subject areas, whereas control group students were college ready in 1.38 subject areas. The estimated impact of 0.49 subject areas after one semester (p < .001) decreased to 0.26 by the end of 3 years (p < .001), as control group students narrowed the gap during the five postprogram semesters. The downward trend of the effect flattened by the end of 3 years, suggesting that a full catch up by the control group is not expected.

Figure 3 shows the percentages of program and control group students who were college ready in math, reading, writing, and all three subject areas, plotted across 3 years.

Figure 3.

College readiness, by subject and semester.

The pattern in Figure 2 is again exhibited in Figure 3 for the percentage of students who were college ready in each subject area. Among the three subject areas, CUNY Start had the largest impact on college readiness in math, with a 32.1 percentage point effect after one semester that decreased and leveled off at 18.0 percentage points at the end of 3 years (p < .001 at both time points). Impacts on college readiness in reading and writing were smaller, at 8.0 and 9.4 percentage points after one semester (p < .001 for both subject areas) and decreasing to 2.9 and 5.5 percentage points at the end of 3 years (p = .046 and p < .001), respectively. As one might expect, the impact on full college readiness (i.e., college readiness in all three subject areas) followed a similar pattern, peaking at 24.9 percentage points after one semester and decreasing to 15.9 percentage points at the end of 3 years (p < .001 at both time points).²⁰

Completion of Gateway Courses

CUNY Start succeeded in reducing or eliminating students’ developmental education requirements. To have an impact on long-term academic outcomes, however, the program must prepare students not only to pass a test or course that deems them college ready but also to succeed in college-level courses. Whether program group students were more likely than control group students to complete a math or English course that satisfies requirements for graduation—that is, a gateway course—is the first milestone at which we can test if the program succeeded in preparing students for college-level coursework. Figure 4 shows the percentages of program and control group students who completed (a) a math gateway course, (b) an English gateway course, and (c) both types of gateway courses.

Figure 4.

Completion of gateway courses, by semester.

By the end of 3 years, CUNY Start increased the percentage of students who completed a gateway course in math and English by 5.2 and 4.4 percentage points, respectively (with respective p values of .004 and .022). In the first semester, when program group students were enrolled in CUNY Start and therefore not attempting any college-level courses, control group students completed gateway courses in each subject at higher rates. By the end of the second semester, however, program group students surpassed their control group counterparts with respect to gateway course completion in each subject. These trends peaked after the third semester, and the positive effects were maintained through 3 years.

Recall that CUNY Start’s effect on college readiness in math (18.0 percentage points) was substantially larger than its effect on college readiness in reading and writing (2.9 and 5.5 percentage points, respectively). Given this, it is surprising that its effects on completing math and English gateway courses were similar—each around 5 percentage points—representing a narrowing of the gap between the program and control groups in math and its maintenance in English.

The narrowing of the gap in math occurred in two steps. First, CUNY Start’s effect on attempting a gateway math course by the end of 3 years was 10.9 percentage points (not shown)—a 7.1 percentage point decrease from the 18 percentage point effect on math college readiness. Second, CUNY Start’s effect on completing a math gateway course by the end of 3 years was 5.2 percentage points—a 5.7 percentage point further narrowing of the gap from 10.9 percentage points. This further narrowing occurred because gateway math course pass rates were higher in the control group than in the program group among those students who attempted a math gateway course (not shown). Comparing pass rates among this subset of students requires careful interpretation. It may be that traditional developmental math better prepared students to pass gateway math courses, or it may be that the type of students CUNY Start induced to attempt gateway math were less likely to pass a math gateway course regardless of the method of developmental math instruction they received. In any case, CUNY Start’s large impact on college readiness in math was muted by the fact that control group students narrowed the gap when it came to attempting and passing gateway math courses. However, by causing more students to attempt a gateway math course, CUNY Start caused more students to complete a gateway math course, despite lower pass rates among those who attempted such a course.

Progress Through College-Level Courses

CUNY Start intends to prepare students not only to complete gateway courses but to complete multiple college-level courses and accumulate credits that lead to a college credential. Figure 5 shows the average number of cumulative college-level credits earned at CUNY, a confirmatory outcome of the study, across 3 years. Figure 5 visually represents CUNY Start’s theory of change well. The program’s designers hypothesized that the students assessed as being the least prepared for college are best served by delaying matriculation for one semester and intensively addressing their developmental education requirements. By doing so, CUNY Start students will earn fewer college-level credits in the short-term, but then be able to accumulate college-level credits more quickly in the long-term. A reasonable concern, however, is that it might not be possible for CUNY Start students to earn college-level credits quickly enough in the postprogram semesters to make up for the credits foregone in the program semester.

Figure 5.

Cumulative college-level credits earned, by semester.

Program group students and control group students accumulated college-level credits in a pattern consistent with CUNY Start’s theory of change. Program group students, on average, earned 1.88 fewer credits than control group students in the first semester after random assignment (the program semester; p < .001). In Semesters 2 through 5, program group students earned more credits than control group students in each semester (p values ranging from <.001 to .015), and, in Semester 6, program and control group students earned a similar number of credits (p = .615, not directly shown in figure). This increased pace of credit accumulation meant that program group students caught up to control group students by the end of the third semester, and had earned more credits than control group students by the end of Semesters 4 to 6, although these differences are fairly small in magnitude (with respective p values of .230, .106, and .140). The evidence does not support concerns that CUNY Start students are not able to catch up after delaying matriculation. However, the estimated effect on college credit accumulation through 3 years is small, with a 90% confidence interval that ranges from just below zero to a moderate three credits. The plateauing trend in the effect estimate suggests that any credit effects have largely been achieved.

Credential Completion

A program that does not substantially increase the average number of cumulative college-level credits earned at the end of 3 years may nonetheless shift some students across the thresholds necessary to earn a degree or certificate. Figure 6 shows the percentage of program group students and control group students who completed a degree or certificate across 3 years.

Figure 6.

Credential completion, by semester.

Over the first five semesters, there was no meaningful difference between the percentage of program group students and control group students who had completed a degree or certificate. However, by the end of the sixth semester, 14.5% of program group students and 11.4% of control group students had completed a credential, resulting in a 3.1 percentage point impact (p = .015). We received enough degree data to examine one additional main session (i.e., a spring or fall semester) after the 3-year mark. With an additional semester, the estimated impact of CUNY Start grew to 4.1 percentage points (19.0% vs. 14.9%; p = .003; not shown).²¹ These results are notable because single-semester higher education programs rarely produce impacts on graduation (Bailey et al., 2016).

It seems surprising that CUNY Start had a positive effect on the likelihood of completing a credential within 3 years but had a small estimated effect on cumulative college-level credits earned at the end of 3 years. To better understand these two seemingly contradictory findings, we examined the distribution of cumulative college-level credits earned at the end of 3 years, by research group (see Supplemental Figure A1 in the in the online version of the journal). At the end of 3 years, 14.5% of program group students and 11.9% of control group students had earned 60 or more college-level credits (p = .034), the number of credits typically required to earn an associate’s degree, closely mirroring rates of credential completion. Furthermore, a higher proportion of program group students than control group students earned 50 to 59 credits within 3 years (6.9% vs. 5.7%; p = .194), which is consistent with the finding presented above that CUNY Start’s effect on completing a credential increased by 1 percentage point in the semester immediately following the third year.²²

A Remark on Mechanism

CUNY’s ASAP is an unusually comprehensive, long-lasting (services are provided for up to 3 years), and highly effective student success initiative available to students who are fully college ready or have no more than two outstanding developmental education course requirements. An RCT evaluation of ASAP found that the program substantially increased 3-year graduation rates (Weiss et al., 2019).

Most CUNY Start students are not ineligible for ASAP until their developmental education requirements have been reduced. However, CUNY promotes CUNY Start as a “pathway” to ASAP and says that it “ensure[s] that eligible students have the opportunity to transition from CUNY Start or Math Start to ASAP seamlessly” (CUNY, n.d.b, pp. 1–2). Given ASAP’s success and its partnership with CUNY Start, it was essential to examine CUNY Start’s effect on ASAP participation and to consider how much of CUNY Start’s effects on 3-year outcomes is likely the direct effects of CUNY Start versus indirect effects resulting from enabling more students to participate in ASAP.

CUNY Start succeeded in substantially increasing ASAP participation.²³ While very few students in either research group participated in ASAP in the first semester after random assignment, by the end of the second semester, 23.2% of program group students had participated in ASAP, compared with only 5.3% of control groups students who had done so. This 17.9 percentage point difference grew to a 19.0 percentage point difference over the following four semesters (p < .001 at both time points; see Supplemental Table A11 in the in the online version of the journal). CUNY Start likely increased ASAP participation for two reasons. First, CUNY Start caused more students to become eligible for ASAP by enabling them to meet ASAP’s developmental education eligibility requirements. Second, the close relationship between CUNY Start and ASAP succeeded in creating a pipeline, increasing the likelihood that ASAP-eligible students joined the program.

A random assignment evaluation of CUNY’s ASAP (at three of the four colleges participating in this study) showed that the program had a 14.2 percentage point impact on completing a degree or certificate within five semesters (Weiss et al., 2019).²⁴ Assuming that this estimated effect applies to the 17.9% of students who were successfully induced by CUNY Start to join ASAP in the second semester, we estimate that ASAP participation accounted for roughly 2.5 percentage points (i.e., 80%) of CUNY Start’s 3.1 percentage point effect on completing a degree or certificate. Taking a similar approach with cumulative college-level credits, we estimate that program group students earned 1.26 more credits than control group students based on ASAP participation—close to the total estimated effect of 1.35 credits. It is unlikely, however, that ASAP participation explains CUNY Start’s effect on progress through developmental education, as that impact emerged in the first semester, before program group students began participating in ASAP. The practical implications of ASAP’s likely role in these findings are elaborated upon in the discussion session.

Subgroup Findings

The above results presented the average estimated effect of CUNY Start for the full sample; however, different types of students may have responded differently to the program. The results presented below document the estimated effects for subgroups of students and the extent to which CUNY Start’s impacts varied for different types, or subgroups, of students. Subgroup effects are presented for four student characteristics: (a) intended CUNY Start program,²⁵ (b) number of college-ready subject areas at random assignment, (c) intended college,²⁶ and (d) race/ethnicity. The first three subgroups were specified in a preanalysis plan as confirmatory. The fourth (race/ethnicity) is included because of its policy relevance. Impact estimates for additional subgroups can be found in the Supplemental Appendix in the online version of the journal.

Figures 7, 8, and 9 show CUNY Start’s respective estimated impacts on the number of college-ready subject areas, college-level credits earned, and credential completion after 3 years for each subgroup. The estimated effects are plotted with their 90% confidence intervals.

Figure 7.

Number of college-ready subject areas (out of three) after 3 years, by subgroup.

Figure 8.

College-level credits earned after 3 years, by subgroup.

Figure 9.

Credential completion after 3 years, by subgroup.

As shown in Figure 7, CUNY Start had a positive estimated effect on the number of college-ready subject areas after 3 years for all subgroups. CUNY Start’s effect on college readiness was greater for students who were college ready in zero subject areas at baseline (i.e., triple-remedial students; p value for the difference in effects = .030)—the students with the most room for improvement—narrowing the equity gap. There is some suggestive evidence that the program was more effective for full-timers than part-timers, with respect to college readiness (p = .132) and evidence that the effects varied by college (p = .100).

Figure 8 shows that for most subgroups examined, the estimated effects of CUNY Start on college-level credits earned were small, positive, and not statistically significant at conventional levels, with a few exceptions for subgroups with relatively small sample sizes and thus imprecise impact estimates (e.g., College B and White students, where the estimated effects were negative). While the estimated effects vary substantially—for example, from −0.82 to 2.86 within “Intended college” and from −1.10 to 2.75 within “Race/ethnicity”—there is not clear evidence that the true effects vary between subgroups within any of the student characteristics.

Perhaps the one exception is the number of college-ready subject areas at random assignment, where triple-remedial students may have benefited more from the program than those with fewer remedial needs (p value for the difference in effects = .217). CUNY Start is estimated to have increased the number of college-level credits earned by 2.45 credits (p = .053) for triple-remedial students—a potentially important finding given these students also experienced larger-than-average effects in college readiness. Meanwhile, the estimated effect for students who had fewer developmental education requirements at baseline was near zero. We are cautious to overinterpret this finding, and, as will be shown next, there is no evidence that credential completion effects were different based on college readiness at baseline.

The estimated effects of CUNY Start on completing a degree or certificate were positive for most subgroups examined, as shown in Figure 9. The estimated effects did not vary discernibly based on any of the four characteristics examined (the p values for differences in effects ranged from .527 to .896). As expected, the outliers were the small racial subgroups (e.g., White, Asian, and other), for which impacts were imprecisely estimated as shown by the wide confidence intervals.

Overall, the subgroup analyses provide suggestive evidence that CUNY Start was more effective for triple-remedial students—an important finding given that there is little research on how to help this population succeed in higher education (Boatman & Long, 2018). There is little clear evidence of any other effect variation.

CUNY Start’s Cost

A detailed cost analysis of CUNY Start was conducted by Community College Research Center (forthcoming). Using the ingredients method (Levin et al., 2017) and the budgetary method, Community College Research Center (forthcoming) estimated the per-student costs to operate CUNY Start and compared these costs with the educational investment for control group students. This comparison yields the incremental costs of CUNY Start; that is, the extra resources colleges needed to implement this reform. Over 3 years, an additional US$2,230 was invested in students offered a place in CUNY Start. CUNY Start resulted in an additional US$340 devoted to students’ developmental education, US$790 spent on additional college-credit taking (which was fully offset by the additional revenue generated from this additional credit taking), and US$1,100 spent for increased ASAP participation. Through 3 years, this increased investment led to, as already noted, an estimated 15.9 percentage point increase in full college readiness, 1.35 additional college-level credits, and a 3.1 percentage point increase in credential completion. With respect to college readiness and award completion, Community College Research Center (forthcoming) found that CUNY Start lowered the cost-effectiveness ratios—that is, CUNY Start was more efficient at producing outcomes than business as usual. The author aptly points out that it is important for colleges and society to consider their willingness to invest in each earned outcome.

Community College Research Center (forthcoming, p. 16) noted that

the funding systems for CUNY Start and BAU [Business-As-Usual] are not exactly the same. CUNY Start is distinctive in its student pricing: students pay only a nominal fixed fee of $75 to enroll in all developmental education coursework in the program semester. For BAU, students pay tuition of $200 per credit. Additionally, the CUNY Start program is mostly funded upfront by the University until such time as the Central Office is able to submit for reimbursement from New York State for continuing education programming and recoup University funding provided to the colleges to run their programs.

According to CUNY leadership, CUNY Start generates New York State Full Time Equivalent (FTE) reimbursements based on seat time—this funding covers most budgetary needs and the university supplements the rest.

Discussion

CUNY Start is an innovative approach to help students assessed as academically underprepared for college. As the implementation research found, the program differs in several important ways from standard college courses and services, including in its cost to the students; its instructional intensity; its math and reading/writing curriculum; its pedagogy, especially in math; its instructor hiring and training; and the support systems it offers for students.

Results from our RCT show that over a 3-year follow-up period, the program had a substantial positive effect on students’ progress through and completion of developmental education, the outcomes that CUNY Start was designed to affect most directly. At the end of 3 years, CUNY Start increased the proportion of students who completed all their developmental education requirements by 16 percentage points. The increase in completing developmental education was particularly large in math, which is striking because developmental math is a barrier that prevents many students from earning a degree (see Attewell et al., 2006). CUNY Start also successfully enabled many more students to participate in CUNY ASAP, a comprehensive program that markedly boosts graduation rates (Weiss et al., 2019). Finally, CUNY Start had a small positive estimated effect on college-level credits earned and graduation rates. Importantly, most of CUNY Start’s total effect on college-level credits earned and graduation after six semesters is likely attributable to CUNY Start increasing ASAP participation and ASAP participation improving these outcomes, rather than the direct effect of CUNY Start alone—the significance of this is discussed in more detail below.

CUNY Start targets students who are assessed as needing remediation in math, reading, and writing, and, notably, our analyses found that the program was effective for these triple-remedial students. This is important because, as mentioned above, many other developmental education reforms target students with fewer remedial education requirements and/or are unavailable to students with the lowest levels of skill in a subject area.

With respect to increasing college readiness, the program’s most immediate goal, CUNY Start is a promising reform. With respect to increasing graduation rates, the results are more complex. In both cases, it is important to consider the extent that the results likely apply beyond the evaluation sample and study institutions. When considering generalizability, the students, institutions, alternative services to which CUNY Start was compared, and context are all relevant—program effects may vary based on each of these factors (Weiss et al., 2014). We discuss each factor, in turn.

Students

Students in the study were all interested in participating in CUNY Start. Students pursuing the full-time program could dedicate up to 26.5 hours of time to classes per week—far more hours than are spent by matriculated students. Even students interested in the part-time program could dedicate 13.5 hours to classes per week. The relatively greater quantity of class time may be one of the keys to CUNY Start’s effect on progress through developmental education, and yet this same aspect of the program may be a barrier to participation for many students. Thus, the program may not be the right fit for all students referred to developmental education—and, indeed, it was not intended to be the right fit for all developmental education students at CUNY. Reassuringly, among students who were interested in the program and could commit to the required time, our subgroup analyses provide suggestive evidence that CUNY Start can work for a variety of different types of students.

Institutions

The study occurred within the context of four community colleges in the CUNY system, and the program operates as a consortium program comprising CUNY OAA and partner colleges. The colleges are located in a large urban setting and have a diverse mix of students with respect to characteristics, including race/ethnicity, country of origin, and languages spoken at home. In addition, during the period discussed in this report, CUNY required students to pass skills tests to exit developmental education, whereas many other community colleges have different policies. It is uncertain whether the institutional and personal barriers to student success that exist at CUNY and CUNY Start’s approaches to addressing these barriers are fully applicable to other community college contexts.

Alternative Services

Relatedly, as Holland (1986, p. 946) aptly noted, “the effect of a cause is always relative to another cause.” In other words, there is no singular effect of CUNY Start—it depends on what it is being compared with. In the present study, we examine the effectiveness of CUNY Start relative to CUNY’s alternative offerings at the time—some of which were described earlier. As is the case in many colleges and college systems, at CUNY, the landscape of developmental education has been and remains in flux. CUNY is modifying how students are referred to developmental education, moving away from primarily relying on a placement test and toward a system using multiple measures and no placement test. Because this change is about who is referred to developmental education and CUNY Start is a program option for students referred to developmental education, this reform could easily be complementary, although it may change how many and what types of students are eligible for CUNY Start. In addition, corequisite remediation is increasing within CUNY with the plan to move away from zero-credit postmatriculation offerings over the next several years, whereas corequisite remediation was rarely offered during this study. Consequently, even within CUNY, care must be used when interpreting this study’s findings in the presence of new alternatives, which were not in existence during the present study (recall that the study’s program semesters were spring 2015, fall 2015, and spring 2016).

Context

The graduation effect mostly resulted from CUNY Start enabling greater participation in the highly effective CUNY ASAP. In the CUNY context, providing CUNY Start to move students into ASAP is a viable strategy for increasing developmental education completion and, eventually, graduation. It is worth noting that in addition to providing an intentional pathway to ASAP, CUNY Start also caused more students to become eligible to participate in ASAP because students with significant developmental education course requirements (those who are assessed as needing more than two developmental education courses) are ineligible for ASAP. It is worth considering how the current CUNY Start-to-ASAP pipeline might compare with modifying the ASAP eligibility requirements to allow students with greater remedial education requirements to enroll directly in ASAP. CUNY ASAP’s estimated effect on 3-year degree completion for students referred to one (or fewer) remedial education courses is 22 percentage points. For students referred to two or more remedial education courses, the estimated effect is 16 percentage points (Weiss et al., 2019). Might ASAP’s effects for students referred to remediation in math, reading, and writing be greater than the 3 percentage points estimated here? (There would be cost implications.)

What about a college considering CUNY Start that does not have ASAP, or something similar, to follow? The evidence indicates that the CUNY Start model is well positioned to yield substantial improvements in college readiness in the absence of ASAP or something similar. However, the evidence from this study does not indicate that the CUNY Start model will meaningfully increase college-level credit accumulation or graduation rates in the absence of ASAP or something similar. It is rare for a one-semester intervention, alone, to meaningfully increase graduation rates, and our findings suggest that CUNY Start is no exception.

Developmental education in general, and developmental math in particular, is considered a critical barrier to graduation. In this study, CUNY Start caused 16 percentage points more students to complete all developmental education requirements and 18 percentage points more students to complete math developmental requirements, yet the program’s direct effect on graduation rates (i.e., absent the increased participation in CUNY ASAP) is near zero. This suggests that while developmental education may be a barrier to completion for many students, it is likely not the only barrier to completion. Figuring out ways to enable more students to complete developmental education is only a first step toward substantially boosting graduation rates. This aligns with prior research, mentioned earlier, that found that there are multiple institutional and individual factors that help explain low graduation rates at community colleges.

As is often the case, the present evaluation suggests a few areas ripe for further research. It is not known whether specific components of the CUNY Start model made more of a difference in students’ outcomes than others, yielding important questions for future research. Increased class time spent on the developmental education disciplines in a compressed time frame may help students focus their learning and gain the subject skills more quickly, even in the absence of CUNY Start’s harder-to-implement components (such as the math pedagogy or the specific curricula)—this is worth exploring. At the same time, the interactive and participatory curricular and pedagogical reform, particularly in math, is quite different from the direct instruction approaches that are common in many developmental education classrooms—Would these instructional reforms make a difference if used in a less time-intensive intervention?

Overall, this study’s findings demonstrate that with the right mix of reforms, students with a variety of developmental education requirements can make considerable progress through remedial education. CUNY Start’s innovative model of delaying college matriculation, and focusing intensively on building students’ skills and providing them forms of support, yielded meaningful positive effects on their completion of developmental education. Importantly, although the program only lasted for one semester, the effects remained large after 3 years, and delaying matriculation did not set back program group students in terms of college-level credits. Furthermore, students in the program made progress in developmental education while preserving financial aid for potential use on future courses. CUNY’s resource guide (Kim et al., in press) can provide information for colleges considering trying the program or something similar on their own campus. In a broader context of developmental education reform, CUNY Start is an intriguing program model to consider for those focused on increasing college readiness.

Supplemental Material

sj-docx-1-epa-10.3102_01623737211008901 – Supplemental material for An On-Ramp to Student Success: A Randomized Controlled Trial Evaluation of a Developmental Education Reform at the City University of New York

Supplemental material, sj-docx-1-epa-10.3102_01623737211008901 for An On-Ramp to Student Success: A Randomized Controlled Trial Evaluation of a Developmental Education Reform at the City University of New York by Michael J. Weiss, Susan Scrivener, Austin Slaughter and Benjamin Cohen in Educational Evaluation and Policy Analysis

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This manuscript and the evaluation on which it is based are funded by the Institute of Education Sciences at the U.S. Department of Education through Grant R305H140065. The findings and conclusions in this manuscript do not represent the official positions or policies of the funder.

ORCID iD

Susan Scrivener

Notes

Authors

MICHAEL J. WEISS is a senior associate in MDRC’s Postsecondary Education Policy Area. He has over a decade of experience evaluating programs designed to improve community college students’ success rates. He also conducts methodological research to improve the quality of randomized controlled trials.

SUSAN SCRIVENER is a senior associate in MDRC’s Postsecondary Education Policy Area. She has over two decades of experience evaluating programs in education and social programs for low-income individuals.

AUSTIN SLAUGHTER is a research analyst in MDRC’s Postsecondary Education Policy Area. He leads data, methodological, and cost work on evaluations of community college programs that seek to improve graduation rates for low-income and at-risk students.

BENJAMIN COHEN is a research assistant in MDRC’s Postsecondary Education Policy Area. He works on evaluations of developmental education and student support programs at community colleges.

References

Adelman

(2004). Principal indicators of student academic histories in postsecondary education, 1972–2000. Institute of Education Sciences, U.S. Department of Education.

Angrist

J. D.

Autor

Hudson

Pallais

(2014). Leveling up: Early results from a randomized evaluation of post-secondary aid (NBER working paper series, no. 20800). National Bureau of Economic Research.

Angrist

J. D.

Imbens

G. W.

Rubin

D. B.

(1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434), 444–455.

Angrist

J. D.

Oreopoulos

Williams

(2010). When opportunity knocks, who answers? New evidence on college achievement awards (NBER working paper series, no. 16643). National Bureau of Economic Research.

Attewell

P. A.

Lavin

D. E.

Domina

Levey

(2006). New evidence on college remediation. Journal of Higher Education, 77(5), 886–924.

Avery

Howell

Page

(2014). A review of the role of college counseling, coaching, and mentoring on students’ postsecondary outcomes. College Board.

Bailey

T. R.

Bashford

Boatman

Squires

Weiss

Doyle

Valentine

J. C.

LaSota

Polanin

J. R.

Spinney

Wilson

Yeide

Young

S. H.

(2016). Strategies for postsecondary students in developmental education. Institute of Education Sciences, What Works Clearinghouse.

Bailey

T. R.

Jaggars

S. S.

Jenkins

(2015). Redesigning America’s community colleges: A clearer path to student success. Harvard University Press.

Bailey

T. R.

Jeong

D. W.

Cho

S.-W.

(2010). Referral, enrollment, and completion in developmental education sequences in community colleges. Economics of Education Review, 29(2), 255–270.

10.

Barnett

Cullinan

Kopko

Belfield

(2020). Who should take college-level courses? Impact findings from an evaluation of multiple measures assessment. New York, NY: Center for the Analysis of Postsecondary Readiness. Retrieved from https://ccrc.tc.columbia.edu/media/k2/attachments/multiple-measures-assessment-impact-findings.pdf.

11.

Barr

Castleman

(2017). The bottom line on college counseling [Working paper]. Texas A&M University.

12.

Baum

Kurose

McPherson

(2013). An overview of American higher education. Future of Children, 23(1), 17–39.

13.

Bettinger

E. P.

Baker

R. B.

(2014). The effects of student coaching: An evaluation of a randomized experiment in student advising. Educational Evaluation and Policy Analysis, 36(1), 3–19.

14.

Bettinger

E. P.

Long

B. T.

(2005). Remediation at the community college: Student participation and outcomes. New Directions for Community Colleges, Number, 129, 17–26.

15.

Bettinger

E. P.

Long

B. T.

(2009). Addressing the needs of underprepared students in higher education: Does college remediation work? Journal of Human Resources, 44(3), 736–771.

16.

Bettinger

E. P.

Long

B. T.

Oreopoulos

Sanbonmatsu

(2012). The role of simplification and information in college decisions: Results from the H&R Block FAFSA experiment. Quarterly Journal of Economics, 127(3), 1205–1242.

17.

Bickerstaff

Edgecombe

(2012). Pathways to faculty learning and pedagogical improvement. Inside Out, 1(3), 1–4.

18.

Bickerstaff

Edgecombe

(2019). Teaching matters and so does curriculum: How CUNY Start reshaped instruction for students referred to developmental mathematics (CCRC working paper no. 110). Community College Research Center, Teachers College, Columbia University.

19.

Bloom

H. S.

(1984). Accounting for no-shows in experimental evaluation designs. Evaluation Review, 8(2), 225–246.

20.

Bloom

H. S.

Bell

Reiman

(2020). Using data from randomized trials to assess the likely generalizability of educational treatment-effect estimates from regression discontinuity designs. Journal of Research on Educational Effectiveness, 13(3), 488–517.

21.

Boatman

Long

B. T.

(2018). Does remediation work for all students? How the effects of postsecondary remedial and developmental courses vary by level of academic preparation. Educational Evaluation and Policy Analysis, 40(1), 29–58.

22.

Bragg

D. D.

Barnett

(2009). Lessons learned from breaking through: In brief. Office of Community College Research and Leadership, College of Education, University of Illinois at Urbana-Champaign.

23.

Braxton

J. M.

(2002). Reworking the student departure puzzle. Vanderbilt University Press.

24.

Calcagno

J. C.

Bailey

Jenkins

Kienzl

Leinbach

(2008). Community college student success: What institutional characteristics make a difference? Economics of Education Review, 27, 632–645.

25.

Carrell

Sacerdote

(2017). Why do college-going interventions work? American Economic Journal: Applied Economics, 9(3), 124–151.

26.

Cohodes

S. R.

Goodman

J. S.

(2014). Merit aid, college quality, and college completion: Massachusetts’ Adams Scholarship as an in-kind subsidy. American Economic Journal: Applied Economics, 6(4), 251–285.

27.

Collins

Brown

J. S.

Newman

S. E.

(1988). Cognitive apprenticeship: Teaching the Craft of reading, writing, and mathematics. Thinking: The Journal of Philosophy for Children, 8(1). https://apps.dtic.mil/dtic/tr/fulltext/u2/a178530.pdf

28.

Community College Research Center. (forthcoming). Cost and cost-effectiveness analysis of CUNY Start [Manuscript in publication]. Columbia University.

29.

City University of New York. (n.d.a). About. https://www.cuny.edu/about

30.

City University of New York. (n.d.b). CUNY start, math start, ASAP, accelerated study in associate programs [Brochure]. http://www1.cuny.edu/sites/asap/wp-content/uploads/sites/8/2018/10/CS_MS_ASAP_Brochure.pdf

31.

City University of New York. (n.d.c). CUNY start program overview. http://www.cuny.edu/sites/cunystart/program/cuny-start1

32.

City University of New York Office of Academic Affairs. (2015). Remediation at CUNY: Update to the CUE Council. https://www.cuny.edu/wp-content/uploads/sites/4/page-assets/about/administration/offices/undergraduate-studies/coordinated/CUERemediationUpdate12-16-2015OUS.pdf

33.

City University of New York Office of Institutional Research and Assessment. (2015). Three-year graduation rates of full-time, first-time freshmen enrolled in associate or certificate programs by gender: Fall 2011 entering cohort. City University of New York.

34.

City University of New York Office of Institutional Research and Assessment. (2018). Total enrollment by undergraduate and graduate level, full-time/part-time attendance, and college. City University of New York.

35.

City University of New York Task Force on Developmental Education. (2016). Report of the CUNY task force on developmental education. https://www2.cuny.edu/wp-content/uploads/sites/4/page-assets/about/administration/offices/undergraduate-studies/developmental-education/Proposed-Recommendations-of-RTF-06.17.16.final_.pdf

36.

Deming

Dynarski

(2009). Into college, out of poverty? Policies to increase the postsecondary attainment of the poor (NBER working paper no. 15387). National Bureau of Economic Research.

37.

Edgecombe

Jaggars

S. S.

Baker

E. D.

Bailey

(2013). Acceleration through a holistic support model: An implementation and outcomes analysis of FastStart@CCD. Community College Research Center, Teachers College, Columbia University.

38.

Evans

W. N.

Kearney

M. S.

Perry

B. C.

Sullivan

J. X.

(2017). Increasing community college completion rates among low-income students: Evidence from a randomized controlled trial evaluation of a case management intervention (NBER working paper no. 24150). National Bureau of Economic Research.

39.

Ginder

S. A.

Kelly-Reid

J. E.

Mann

F. B.

(2015). Graduation rates for selected cohorts, 2006-11; student financial aid in postsecondary institutions, academic year 2013-14; and admissions in postsecondary institutions, fall 2014; first look [Provisional data]. National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education.

40.

Goldrick-Rab

Kelchen

Harris

D. N.

Benson

(2016). Reducing income inequality in educational attainment: Experimental evidence on the impact of financial aid on college completion. American Journal of Sociology, 121(6), 1762–1817.

41.

Graham

Hebert

(2010). Writing to read: Evidence for how writing can improve reading (Carnegie Corporation time to act report). Alliance for Excellent Education.

42.

Graham

Perin

(2007a). What we know, what we still need to know: Teaching adolescents to write. Scientific Studies of Reading, 11(4), 313–335.

43.

Graham

Perin

(2007b). Writing next: Effective strategies to improve writing of adolescents in middle and high school. Alliance for Excellent Education.

44.

Grubb

W. N.

Gabriner

(2013). Basic skills education in community colleges: Inside and outside of classrooms. Routledge.

45.

Hiebert

Gallimore

Garnier

Givvin

K. B.

Hollingsworth

Jacobs

Miu-Ying Chui

Wearne

Smith

Kersting

Manaster

Tseng

Etterbeek

Manaster

Gonzales

Stigler

(2003). Teaching mathematics in seven countries: Results from the TIMSS 1999 video study. National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education.

46.

Hodara

(2015). The effects of English as a second language courses on language minority community college students. Educational Evaluation and Policy Analysis, 37(2), 243–270.

47.

Holland

P. W.

(1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945–960.

48.

Imbens

G. W.

Angrist

J. D.

(1994). Identification and estimation of local average treatment effects. Econometrica, 62(2), 467–475.

49.

Jenkins

Jaggars

S. S.

Roksa

(2009). Promoting gatekeeper course success among community college students needing remediation: Findings and recommendations from a Virginia study [Summary report]. Community College Research Center, Teachers College, Columbia University.

50.

Jenkins

Speroni

Belfield

Jaggars

S. S.

Edgecombe

(2010). A model for accelerating academic success of community college remedial English students: Is the Accelerated Learning Program (ALP) effective and affordable? (CCRC working paper series). Community College Research Center, Teachers College, Columbia University.

51.

Kemple

J. J.

Herlihy

C. M.

Smith

T. J.

(2005). Making progress toward graduation: Evidence from the talent development high school model. MDRC.

52.

Kim

Simon

Horenstein

Webber

(in press). CUNY Start: A resource guide for reforming developmental education in the college transition space. City University of New York.

53.

Levin

H. M.

McEwan

P. J.

Belfield

Bowden

A. B.

Shand

(2017). Economic evaluation in education: Cost-effectiveness and benefit-cost analysis. SAGE.

54.

Logue

A. W.

Douglas

Watanabe-Rose

(2019). Corequisite mathematics remediation: Results over time and in different contexts. Educational Evaluation and Policy Analysis, 41(3), 294–315.

55.

Mayer

A. K.

Patel

Rudd

Ratledge

(2015). Designing scholarships to improve college success. MDRC.

56.

McFarland

Hussar

Zhang

Wang

Hein

Diliberti

Forrest Cataldi

Bullock Mann

Barmer

(2019). The condition of education 2019. National Center for Education Statistics.

57.

National Student Clearinghouse. (n.d.). About the Clearinghouse. https://www.studentclearinghouse.org/about/

58.

National Council of Teachers of Mathematics. (2000). Principles and standards for high school mathematics. National Council of Teachers of Mathematics.

59.

National Governors Association Center for Best Practices. (2010). Common core state standards. Council of Chief State School Officers.

60.

National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the national mathematics advisory panel. U.S. Department of Education.

61.

National Reading Panel. (2000). Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction. National Institute of Child Health and Human Development.

62.

National Student Clearinghouse Research Center. (2019). Term enrollment estimates: Fall 2019. National Student Clearinghouse Research Center.

63.

Nomi

Allensworth

(2009). “Double-dose” algebra as an alternative strategy to remediation: Effects on students’ academic outcomes. Journal of Research on Educational Effectiveness, 2(2), 111–148.

64.

Oreopoulos

Petronijevic

(2016). Student coaching: How far can technology go? (NBER working paper 22630). National Bureau of Economic Research.

65.

Perin

(2013). Literacy skills among academically underprepared students. Community College Review, 41(2), 118–136.

66.

Richburg-Hayes

Visher

M. G.

Bloom

(2008). Do learning communities effect academic outcomes? Evidence from an experiment in a community college. Journal of Research on Educational Effectiveness, 1(1), 33–65.

67.

Robbins

(2013). Implications of advising load. In Carlstrom

(Ed.), 2011 National survey of academic advising (Monograph no. 25). National Academic Advising Association.

68.

Rutschow

Mayer

(2018). Early findings from a national survey of developmental education practices. Center for the Analysis of Postsecondary Readiness, Teachers College, Columbia University.

69.

Rutschow

Schneider

(2011). Unlocking the gate: What we know about improving developmental education. MDRC.

70.

Rutschow

Sepanik

Deitch

Raufman

Dukes

Moussa

(2019). Gaining ground: Findings from the Dana Center Mathematics Pathways Impact Study. MDRC.

71.

Schochet

P. Z.

(2008). Technical methods report: Guidelines for multiple testing in impact evaluations (NCEE 2008-4018). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

72.

Scott-Clayton

(2012). Do high-stakes placement exams predict college success? (CCRC working paper no. 41). Community College Research Center, Teachers College, Columbia University.

73.

Scott-Clayton

Schudde

(2017). Performance requirements in need-based aid: What roles do they serve, and do they work? [Research brief]. Center for Analysis of Postsecondary Education and Employment, Teachers College, Columbia University.

74.

Scrivener

Gupta

Weiss

M. J.

Cohen

Cormier

M. S.

Brathwaite

(2018). Becoming college-ready: Early findings from a CUNY Start evaluation. MDRC.

75.

Scrivener

Weiss

M. J.

Sommo

(2012). What can a multifaceted program do for community college students? Early results from an evaluation of Accelerated Study in Associate Programs (ASAP) for developmental education students. MDRC.

76.

Sommo

Boynton

Collado

Diamond

Gardenhire

Ratledge

Rudd

Weiss

M. J.

(2014). Mapping success: Performance-based scholarships, student services, and developmental math at Hillsborough Community College. MDRC.

77.

Stein

M. K.

Smith

M. S.

Henningsen

M. A.

Silver

E. A.

(2009). Implementing standards-based mathematics instruction: A casebook for professional development. Teachers College Press.

78.

Stigler

J. W.

Hiebert

(2009). The teaching gap: Best ideas from the world’s teachers for improving education in the classroom. Free Press.

79.

U.S. Department of Education. (2011). Building on foundations for success: Guidelines for improving adult mathematics instruction. Office of Vocational and Adult Education.

80.

U.S. Department of Education, National Center for Education Statistics. (n.d.). QuickStats. https://nces.ed.gov/pubs2020/2020522.pdf

81.

U.S. Department of Education, Office of Federal Student Aid. (n.d.a). Calculating Pell Grant lifetime eligibility used. https://studentaid.gov/understand-aid/types/grants/pell/calculate-eligibility

82.

U.S. Department of Education, Office of Federal Student Aid. (n.d.b). Staying eligible. https://studentaid.gov/understand-aid/eligibility/staying-eligible

83.

Valentine

J. C.

Konstantopoulos

Goldrick-Rab

(2017). What happens to students placed into developmental education? A meta-analysis of regression discontinuity studies. Review of Educational Research, 87(4), 806–833.

84.

Visher

M. G.

Wathington

Richburg-Hayes

Schneider

(2008). The learning communities demonstration: Rationale, sites and research design. National Center for Postsecondary Research.

85.

Visher

M. G.

Weiss

M. J.

Weissman

Rudd

Wathington

H. D.

(2012). The effects of learning communities for students in developmental education. MDRC.

86.

Weiss

M. J.

Bloom

H. S.

Brock

(2014). A conceptual framework for studying the sources of variation in program effects. Journal of Policy Analysis and Management, 33(3), 778–808.

87.

Weiss

M. J.

Bloom

H. S.

Verbitsky-Savitz

Gupta

Vigil

A. E.

Cullinan

D. N.

(2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Journal of Research on Educational Effectiveness, 10(4), 843–876.

88.

Weiss

M. J.

Brock

Sommo

Rudd

Turner

M. C.

(2011). Serving community college students on probation: Four-year findings from Chaffey College’s Opening Doors Program. MDRC’s.

89.

Weiss

M. J.

Ratledge

Sommo

Gupta

(2019). Supporting community college students from start to degree completion: Long-term evidence from a randomized trial of CUNY’s ASAP. American Economic Journal: Applied Economics, 11(3), 253–297.

90.

Weiss

M. J.

Visher

Weissman

Wathington

(2015). The impact of learning communities for students in developmental education: A synthesis of findings from randomized trials at six community colleges. Educational Evaluation and Policy Analysis, 37(4), 520–541.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.17 MB