How Effective Are Community College Remedial Math Courses for Students With the Lowest Math Skills?

Abstract

Objective: This article examines the effectiveness of remediation for community college students who are identified as having the lowest skills in math. Method: We use transcript data from a state community college system and take advantage of a regression discontinuity design that compares statistically identical students who are assigned to the lowest level of the math sequence that consists of three remedial courses versus the next lowest level that consists of two courses. Results: The results suggest that for the students with the lowest preparation in math, the longest developmental sequence offers little benefit and may even reduce the likelihood of earning a degree to certificate within 4 years. Contributions: This study is one of the first attempts to compare the academic outcomes of students assigned to long sequence of developmental math education to students with similar academic skills but assigned to shorter developmental math sequence. Results from this study can therefore help inform the national effort in reforming remedial education, especially in terms of whether shortening the long remedial sequence would either benefit or harm the academic outcomes of students who are least prepared for college-level coursework.

Keywords

remedial coursework math education developmental education degree completion

Existing studies consistently highlight the importance of cognitive and numeracy skills for labor market success (Banks & Oldfield, 2006; Murnane, Willett, & Levy, 1995; Rivera-Batiz, 1992). Yet, nationally, large proportion of high school graduates are unprepared to take college-level courses and are placed into remedial (also known as developmental) courses upon college entry (Bailey, Jeong, & Cho, 2010). Remedial placement rates are particularly high among students who enroll at community colleges. According to the Beginning Postsecondary Students (BPS) 6-year follow-up data from 2009, about 68% of students who began at a public 2-year college took one or more remedial courses within 6 years after their initial college entry (Scott-Clayton & Rodriguez, 2015). Traditional developmental programs consist of a set of multiple courses that students must enroll in sequentially. Students at the lowest levels are often required to complete at least three semesters of developmental coursework for the corresponding subject area. Despite the excessive amount of time and costs associated with remedial education for both colleges and individuals, college completion rates remain low among students who are required to take remedial coursework—Only 20% of students referred to developmental math and 37% of students referred to developmental reading or writing go on to pass the entry-level college-level course in the relevant subject (Bailey et al., 2010).

These poor academic outcomes raise questions about the effectiveness of the traditional long sequence of remediation to address the needs of students who are identified as having low math preparation. Yet, the methodological challenge to answering this question is whether the poor completion outcomes of students assigned to the bottom level of the remedial sequence reflects the ineffectiveness of the remedial courses, or whether these students would have had even worse outcomes if they were not assigned to remedial courses. Thus, studies that address baseline differences between students, and compare the academic outcomes of students assigned to longer sequence of developmental education with students with similar academic skills but assigned to shorter developmental sequence, could inform ongoing national discussions on developmental education reform strategies. In recent years, several causal studies on the effectiveness of remediation have attempted to examine whether taking an additional course in the remedial sequence improves students’ academic outcomes (e.g., Bettinger & Long, 2009; Boatman & Long, 2017; Calcagno & Long, 2008; Hodara, 2012; Lesik, 2007; Martorell & McFarlin, 2011; Scott-Clayton & Rodriguez, 2015). Most of these studies focused exclusively on evaluating the effectiveness of the highest level of the remedial sequence and did not consider the effect of remediation on students who are least prepared for college coursework.

A priori, it is likely that the effect of being required to take an additional class in remedial math is different for students with very low math skills compared with students just at the margins of remediation. Yet, the possible benefit of providing longer sequences of developmental math to the academically least prepared students in math is far from clear. On one hand, remedial education represents human capital investment and, for students who have very low math skills, taking longer sequences of remediation might be particularly helpful in improving academic skills that are necessary for subsequent college-level coursework (Becker, 1994). On the other hand, students at the lowest levels of developmental education are often required to complete at least three semesters of developmental coursework. Developmental education sequence length increases the opportunity cost of schooling, in that students need to spend extra time and resources on developmental education instead of in the labor market gaining wages and working experience (Hodara & Xu, 2016). Long developmental sequences may also delay students’ progress toward a degree as students assigned to developmental education cannot enroll in most college-level credit-bearing courses until they complete their developmental requirements (Melguizo, Bos, & Prather, 2011). In view of the huge costs associated with developmental education, there has already been a national push for decreasing the requirement for developmental education and shortening the traditional long developmental sequence has increasingly become a popular approach. Yet, whether shortening the sequence may result in less desirable student outcomes among least prepared students is largely unknown.

This study examines the causal impacts of the lowest level of remedial math on student academic outcomes. We examine the effectiveness of the third level of math remediation (which in most colleges is referred to as “Pre-Algebra” and is often the lowest level of remediation offered at a college), compared with the middle level of remediation math using data from the Virginia Community College System (VCCS) that includes 23 community colleges. We focus on the 2004 cohort, when this state had fairly traditional placement policies and developmental education sequences similar to those of other states nationwide (Hodara, Jaggars, & Karp, 2012). Therefore, results from this study can largely help inform the national effort in reforming remedial education, especially in terms of whether shortening the long remedial sequence would either benefit or harm the academic outcomes of students who are least prepared for college-level coursework.

Theoretical Framework and Existing Literature

Our theoretical framework for understanding the results of remedial education distinguishes between two models: the assistance and hindrance models (Scott-Clayton & Rodriguez, 2015). The assistance model hypothesizes that developmental education helps students with inadequate preparation catch up with other students by developing their academic skills and knowledge to college ready standards. As a result, although enrollment in remediation may delay college coursework, students should benefit over the long term, such as with a greater chance of passing college-level courses and higher rate of degree or certificate attainment (McCabe, 2003).

However, the assistance model is based on several assumptions that are yet to be verified. The first assumption is that there are well-defined criteria for what students need to know to be ready for college-level courses and graduation and that remedial courses offer that content. Second, it is assumed that remedial assessment correctly identifies students who can benefit from the remedial content. A third assumption is that any benefit in acquiring the relevant knowledge and skills outweigh the additional financial burden to the students, the additional opportunities for students to leak out of the system, and any stigma associated with being assigned to remediation (Scott-Clayton & Rodriguez, 2015).

In contrast, the hindrance model suggests that any benefit from skills and knowledge development in the remedial courses is outweighed because of one or more of the following costs to the students: (a) direct costs of taking additional courses that do not contribute toward a degree or certificates, (b) the cost of additional time spent in the classroom as well as delaying access to college-level courses in the subject of remediation, and (c) the psychological cost of feeling stigmatized for not being ready for college-level coursework (Papay, Murnane, & Willett, 2016). Moreover, the negative impacts of imposing such burden on students may be even stronger if these courses do not develop students’ skills that are useful to subsequent college-level coursework (Hodara & Xu, 2016). Indeed, through an examination of 169 developmental courses at 29 college in California, Grubb (2013) found that the drill and practice emphasized in most developmental courses are taught in a decontextualized way that do not directly connect to any particular field of study. As a result, these courses fail to “clarify for students the reasons for or the importance of learning these subskills” (p. 52) and may not effectively achieve their intended goal to prepare students for subsequent college coursework. A related possibility is that assignment to remediation might discourage or stigmatize by sending a message to students that they are not college material (Papay et al., 2016; Steele & Aronson, 1995).

A growing volume of studies have recently used quasi-experimental designs to draw causal inferences about the impact of developmental coursework (e.g., Bettinger & Long, 2009; Boatman & Long, 2017; Calcagno & Long, 2008; Hodara, 2012; Lesik, 2007; Martorell & McFarlin, 2011; Scott-Clayton & Rodriguez, 2015; Xu, 2016). The majority of these studies focused on evaluating the effectiveness of developmental education on students at the margin of needing it; overall, these studies had mixed findings regarding the impacts of developmental math on students’ academic outcomes. Using data from the state of Ohio, Bettinger and Long (2009) found positive effects for assignment to math remediation on college persistence. Calcagno and Long (2008) used data from Florida and relied on a regression discontinuity design (RDD), comparing students who scored just below and just above the threshold for remediation on the placement score. They found no positive effects from math remediation on long-term outcomes such as earning college credits or graduating, despite positive effects on increasing persistence to the second semester. Similarly using data from Texas, Martorell and McFarlin (2011) found no positive effects from being assigned to math remediation on academic outcomes including persistence, graduation, transfer, or postcollege wages for students at the margins of remediation.

Only three studies to date (Boatman & Long, 2017; Hodara, 2012; Xu, 2016) explored the causal impact of developmental coursework on students with much lower levels of preparation and all used RDDs. Based on a large administrative data from an anonymous community college system, Hodara (2012) examined the effect of developmental reading and writing for English as a second language (ESL) students. She found that developmental education does not help college success for students who place into the highest level of English remediation; but, for the language minority student with lower levels of academic preparation, assignment to two developmental English subjects versus one had a limited positive impact on his or her college outcomes. Yet, language minority students only constitute a small proportion of community colleges students, and therefore, it is unclear whether the benefits of long sequence of developmental coursework are generalizable to the majority of students enrolled in developmental education.

Xu (2016) used college administrative data from VCCS and compared outcomes of students who scored barely above the various cutoff scores for developmental English with those who scored barely below. The results suggest that the impacts of developmental English differ by the level of assignment. While the estimated effects are generally small in magnitude and statistically insignificant for students on the margin of needing developmental coursework, lower level (and therefore longer) developmental sequences lead to negative impacts on various academic outcomes for students with very low skills, including a lower first-year retention rate, a lower probability of ever attempting a college-level English course, a lower number of college-level credits earned, and a lower probability of earning any degree or certificate within 5 years of initial enrollment. For example, being assigned to the lower level developmental reading increases the probability of dropping out within the first year by 6 percentage points, and reduces the probability of ever embarking on college-level English course by 7 percentage points. For more distal outcomes, students just below the cutoff score and therefore assigned to lower level reading, compared with those just above the cutoff score, earned 5 fewer total credits and were less likely to earn any degree or transfer to a 4-year college by 7 percentage points. However, Xu (2016) focused on developmental English in her study and did not explore the impacts of math developmental education on student academic outcomes.

Boatman and Long’s (2017) study is the only quasi-experimental study that examined the effect of being assigned to remedial classes for students assigned to different levels of remedial math. Focusing on the 2000 community college entrants in Tennessee, the authors found the largest negative effects for students who were assigned to only one remedial class. For students with much lower preparation who were assigned to taking multiple remedial classes, the negative effects of being assigned to a lower level of remediation were smaller and were in some cases not significant. Specifically, being assigned to the lowest level of math remediation decreases college credits completed by 1.2 credits and reduces the likelihood of earning a credential by about 5 percentage points. These findings have important policy implications as they suggest that being assigned to longer sequence of remedial math hinders students’ academic progression. Therefore, it would be of great policy interest to examine whether findings regarding different levels of remedial math would also hold in other states. The current study responds to the national calls for more evidence on the impacts of developmental education by adding evidence on the impacts of the lowest level of developmental math sequence in Virginia.

Developmental Education in VCCS

Virginia is one of the several states that administer a multitiered statewide placement system to assign students to different levels of math, reading, and writing developmental courses. During the period of this study, developmental education in Virginia resembled remediation in most states where students are assigned to three (and in some cases four)¹ levels of a remedial sequence based on the students’ performance on a standardized test. Under the traditional model, when students enrolled in one of Virginia’s community college campuses, they were required to take a placement test to help the colleges determine students’ need for remediation. If a student was assigned to any of the three levels below college-level math, the student would be prevented from enrolling in a college-level course in math, but would be permitted to take as many as three classes in other subject areas. While Virginia colleges did not have a centralized policy for exempting students from placement testing, some of the colleges allowed test exemptions if the student scored above a specific threshold on the SAT or the ACT tests.

The placement test that was used in 2004 for remedial placement in math in Virginia was the COMPASS test, which is a computer adaptive placement test currently used by several states for the purpose of remedial placement (see Hughes & Scott-Clayton, 2011, for detailed information on remedial assessment in different states). The math section of the test first determines the level at which students should receive a comprehensive assessment, based on an initial preliminary assessment. Students who are assessed to have the lowest skills are then given questions from the Prealgebra section of the test. If a student is given the Prealgebra test, then the student is either assigned to the lowest level of math remediation or to the middle level of math remediation, based on the student’s score on the test and each college’s policy and threshold score. Developmental courses tend to cover similar general topics (Grubb, 2013).

While VCCS had statewide assessment and placement policies, individual colleges tended to implement their own policies regarding the cutoff scores for different levels of remediation. In math, the 23 colleges in Virginia used different thresholds for assigning students to the lowest level versus the middle level of math remediation, and the cut scores covered a fairly broad range. For the bottom versus the middle level of remedial math, the assignment threshold ranges from 29 to 40 points in a 100-point scale.

In Virginia, the lowest level of developmental math covered arithmetic, the middle level introductory Algebra, and the highest level, intermediate Algebra, prepared students for college Algebra. The third-level remediation course is called Arithmetic or Prealgebra and includes topics of arithmetic principles and computations. According to the 2004-2005 course catalogs from Northern Virginia Community College (NVCC; 2004), the course addresses “whole numbers fractions, decimals, percent, measurements, graph interpretation, geometric forms, and applications.” Students who score high enough on the Prealgebra test are assigned to the middle level of developmental math, which in most colleges is referred to as Algebra I or Introductory Algebra. Examples of topics discussed in Algebra I based on the NVCC 2004-2005 course catalog include “real numbers, equations and inequalities, exponents, polynomials, Cartesian coordinate system, rational expressions, and applications” (NVCC, 2004). In the three-course sequence for math developmental education, students at the lowest levels needed to complete at least three semesters of developmental coursework before they could enroll in college math and courses that have math prerequisites.

Data

The data set used in this analysis includes 24,664 first-time community college students who first enrolled in one of Virginia’s 23 community colleges in summer or fall of 2004.² Of students who have a valid placement test score, 5,440 students took the Prealgebra section of the COMPASS exam. To assess the effectiveness of the lowest level of math remediation, we limited the sample to students who took the Prealgebra section of the test.³

For each student, the data set includes information about the type of degree a student is seeking, as well as students’ gender, age, and race and whether a student has taken college-level courses while attending high school (i.e., dual enrollment status).⁴ The two academic outcomes that we examined were whether the student took and passed a gatekeeper math course in 4 years and whether the student received a certificate or degree after 4 years. Gatekeeper is a term that refers to the first required college-level course; in math in Virginia, the required college-level math course varied by program of study but includes algebra, precalculus, calculus, quantitative reasoning courses, and discipline-specific courses such as business math.

Table 1 shows the mean of different background characteristics and outcomes for the sample of students who have taken the Prealgebra section of the test. While the majority of colleges use student placement test scores to make remedial assignment, three colleges include a special policy of decision zones for more flexible remedial assignment. Specifically, for students within a certain range of placement test scores around the cutoff, counseling services are provided and the counselors could recommend students either to the higher or lower level of the developmental sequence. As we will explain in more detail later, this imposes threats to our identification strategy. We included all colleges in our main analyses but conducted separate robustness checks on a subsample of colleges excluding the three that implement the decision zone. Summary statistics of this subsample are presented in the right panel of Table 1. For both the full sample and the subsample that excludes colleges with decision zones, students who place into remedial math have generally poorer outcomes. Among students in all colleges, only 10% of students who were required to take the Prealgebra section of the test passed the gatekeeper math course within 4 years and only 11% earned a certificate or an associate degree within 4 years.

Table 1.

Sample Characteristics and Outcomes.

Variable	General sample		Schools with no DZ or retesting
Variable	M	SE	M	SE
Student characteristics
Age	23.0	0.1	22.9	0.1
Male	34%	0.006	33%	0.009
Black	31%	0.006	33%	0.009
Asian	3%	0.002	3%	0.003
Hispanic	4%	0.003	4%	0.003
Participation in dual enrollment	5%	0.003	7%	0.005
Intend to pursue an academic credential	57%	0.007	48%	0.009
Outcomes
Received a credential	11%	0.004	11%	0.006
Passed gatekeeper math	10%	0.004	8%	0.005
Number of students	5,394		2,864
Number of colleges	23		15

Source. Author calculations using data from Virginia Community College System.

Note. The sample is limited to students who have valid test scores on the Prealgebra section for the math placement test. DZ = decision zone.

Addressing Ability Sorting: RDD

In analyzing the effectiveness of the lowest level of math remediation, it is important to fully control for all the observable and unobservable differences between students who took the lowest versus the middle level of remedial math (in Virginia referred to as Prealgebra and Intermediate Algebra accordingly). Following the existing literature (e.g., Boatman & Long, 2017; Calcagno & Long, 2008; Martorell & McFarlin, 2011), we used a RDD. Taking advantage of the fact that each college in Virginia used a well-documented score threshold for placement into different levels of remediation, we compared outcomes of students who scored just above the assignment threshold with the outcomes of students who scored just below the threshold. Students around the threshold are expected to be fairly similar in their academic preparation, but yet sharply differ in developmental coursework assignment. As a result, the RDD coefficient can be then interpreted as the causal impact of the intervention for students on the margin of passing the cutoff (Levin & Calcagno, 2008).

If assignment to remediation predicted enrollment, we would have estimated a reduced form equation such as Equation 1.

\begin{matrix} Y = η + B above + κ (Compass Distance \times Below) \\ + λ (Compass Distance \times above) + δ X + ε, \end{matrix}

where below is an indicator that the student scored below the threshold for the lowest level of remediation and was thus assigned to the third lowest level of math and above is an indicator that the student scored above the threshold and is assigned to the middle level of remediation. Compass distance is the distance between a student’s score on the COMPASS test and the cutoff score of the college that the student is enrolled in. As mentioned earlier, in Virginia during the period of this study, each college chose its own assignment threshold; therefore, to obtain a standardized score for each student, we subtracted each student’s test score from the cutoff score of the student’s reported college of attendance. X is a vector of covariates including gender, race, age, as well as indicators for prior participation in dual enrollment, and whether the student has indicated intent to pursue an academic or vocational credential upon entry. ε is an idiosyncratic error term. The two separate interactions between COMPASS distance and whether the student has scored above or below the assignment threshold (Compass Distance × Below and Compass Distance × Above) allow for the effect of distance on the outcome to vary depending on whether the student has scored above or below the cutoff score.

Addressing Noncompliance: Fuzzy Regression Discontinuity (FRD) Design

Equation 1 provides the estimates of the effect of scoring below the college’s remediation assignment threshold for the lowest level of remediation, not the effect of enrolling in the lowest level of remediation. If there was compliance with placement recommendations based on students’ compass score and the assignment threshold, the two effects would have been identical. However, not all students who are assigned to the lowest level of remediation enroll in that course (noncompliance); therefore, assignment does not predict enrollment. Instead, we used the placement score as an instrument to predict enrollment in the lowest level versus the middle level of math remediation using the framework of a fuzzy RDD.

The FRD includes a two-stage equation as shown in Equations 2a and 2b below. Equation 2a uses the discontinuity rule as an instrument to predict the effect of scoring below the threshold on enrolling in the lowest level of remedial math. Then, in the second stage, Equation 2b, we estimated the effect of predicted assignment to the lowest versus the middle level of remediation, on different outcomes. We used the following outcomes as indicators of student success: completing the first gatekeeper course in math with a grade of C or higher and earning a credential in 4 years.

\begin{matrix} P = λ + ψ (below) + (Compass Distance \times Below) \\ + φ (Compass Distance \times Above) + X ϕ + ε, \end{matrix}

\begin{matrix} Y = α + β (P) + ς (Compass Distance \times Below) \\ + π (Compass Distance \times Above) + X δ + ε . \end{matrix}

In the equations above, P represents actual enrollment in the lowest level of remediation (Prealgebra), P represents predicted enrollment in Prealgebra, and all other variables are the same as defined in Equation 1. We estimated the two-stage model, with both a linear and a squared term, measuring the distance between a student’s test score and the cut score at the student’s college.⁵ While it is straightforward to estimate the linear regressions within a given range of scores (or bandwidth) around a cutoff point, a critical question is the selection of the bandwidth within which the analysis should be conducted. Lee and Lemieux (2010) specified the tradeoff between precision and bias when finding an optimal bandwidth. On one hand, using a larger bandwidth yields more precise estimates; on the other hand, the linear specification is less likely to be accurate when a larger bandwidth is used, which can bias the estimate of the treatment effects.

To identify the optimal bandwidth, we used the cross-validation procedure developed by Imbens and Lemieux (2008). The basic idea behind this procedure is to identify a bandwidth within which the functional form fits the data in an optimal way. Specifically, we estimated a linear regression to predict a given outcome variable within a set of different bandwidths. The bandwidth that minimizes the summation of the squared residuals then represents the best fit of the regression model to the data. The preferred bandwidth that we obtained using this particular procedure ranges depending on the cutoff explored and the outcome used, where most of them are around ±8 points. Accordingly, we reported results using an ±8-point bandwidth but conducted sensitivity analysis using slightly narrower bandwidths (±6 points) and one with a wider bandwidth (±10).

Validity of the RD Design

There are two main assumptions underlying the validity of the RDD: First, there should be a discontinuity in the probability of treatment at the cutoff score. In other words, scoring just below the college’s threshold score should significantly and substantively increase the likelihood that a student will enroll in the lowest remedial class compared with if the student had scored just above the threshold. This first assumption is testable. Figure 1 shows the probability of enrolling in the lowest versus the middle level of remediation by students’ score on the Prealgebra section of the COMPASS test. As we would expect, there is a discontinuity in the probability of taking Prealgebra at the threshold score; the scores are centered at 0. Estimating the first stage in Equation 2a indicates that scoring below the cutoff score in each college increases a student’s likelihood of enrolling in Prealgebra by 48% and that the difference is significant at p < .01.

Figure 1.

Probability of enrolling in Prealgebra by compass score.

Another condition that is necessary to exist for unbiased estimation using RD is that observable and unobservable student characteristics do not change discontinuously on either side of the threshold score. Unfortunately, it is impossible to test this assumption for unobservable student characteristics; however, if observable student characteristics are discontinuous at the assignment cutoff, then we have reason to believe that students are not randomly distributed on either side of the assignment threshold.⁶ Table 2 shows the average student demographic characteristics, the intent to earn an academic credential, and prior dual enrollment participation for the global sample as well as for the more limited samples including students who score within 6 points around their colleges’ cutoff score. When we compared all students who scored above the Prealgebra assignment threshold (referred to as “all above” in Table 2) with all the students who score below the threshold (referred to as “all below” in Table 2), there were statistically significant, but small, differences in student characteristics. However, when we compared the differences among the two groups for students within a 6-point score range from their college’s cutoff score, there were almost no statistically significant observable differences among these students, suggesting that, at least based on observed differences, students who scored 6 points above versus 6 points below the colleges’ assignment threshold are similar. This is suggestive evidence that with a narrow bandwidth of 6 points around the cutoff score, we would expect to be comparing similar students.

Table 2.

Student Characteristics by Assessment Score.

Variable	All below	All above	Difference	(+6)	(−6)	Difference
Age	22.443	23.574	1.131*	22.650	22.696	−0.046
Male	0.385	0.305	−0.080*	0.312	0.340	−0.029
Black	0.249	0.372	0.123*	0.023	0.033	−0.011
Asian	0.026	0.026	0.000*	0.031	0.052	−0.021
Hispanic	0.027	0.044	0.017*	0.331	0.320	0.011*
Participation in dual enrollment	0.064	0.037	−0.027*	0.058	0.043	0.014
Intend to pursue an academic credential	0.597	0.543	−0.054*	0.595	0.561	0.035

Source. Author calculations using data from Virginia Community College System.

The differences are significant at p values below .10. The sample is limited to students who have valid test scores on the Prealgebra section for the math placement test. “All above” and “all below” refer to samples that include all students who scored above and the sample that include all the students who scored below their colleges’ threshold on the Prealgebra test, respectively.

The main shortcoming of studies that use RDD, which this study partly overcomes, is that the results are only generalizable to students who score around the assignment threshold and not to students with much higher or lower skills. For example, in Florida and Texas, all the community colleges share a centralized placement cut score for remedial assignment. As a result, the estimates from a RDD would only speak to the students who fall within a limited range of placement test scores. The Virginia context, where developmental assignment cutoff scores vary across colleges, helps make the results generalizable across a wider range of scores.

Having multiple assignment cut scores can be a potential challenge for this analysis. If students chose their college based on cut score to avoid remediation, then we would have endogenous sorting around the cutoff score, which would lead to biased estimates. Qualitative research reveals that community college students are generally unaware of college remediation policies or even the existence of remedial assessment before enrolling in college (Venezia, Bracco, & Nodine, 2010). In Virginia, colleges are not located close enough to one another to allow much choice for students with regard to which college to attend.⁷ Several other characteristics of the Virginia system pose potential challenges for this analysis. As mentioned earlier, three colleges in Virginia have a range of scores in which an academic advisor can help students determine which level of remediation to enroll in. The decision zones can lead to endogenous sorting of students around the cutoff score in the colleges that have this policy because students who score in the range of the decision zone are not randomly assigned to classes, but are instead assigned to classes based on characteristics that may be unobservable to the researcher but are observable to the college academic advisor—those unobserved characteristics can determine college outcomes. To address this complication, we tested the robustness of the results, excluding the colleges that have decision zones, by reestimating the results after excluding the three colleges that report having decision zones.

Similar to other states such as Florida, a main concern using Virginia data is that some colleges may allow students to retake the COMPASS exam and only record the highest COMPASS score. Because more motivated students may be more likely to retake the exam, retesting can cause endogenous sorting around the cutoff score. As there is no central documentation of which colleges allow retesting, we directly tested whether colleges show evidence of endogenous sorting by examining the histogram of test scores around the cutoff scores. We identified five colleges that show evidence of possible retesting because there is a stacking of test scores just above the cutoff score. To test whether or not this potential bias is driving the results, we reran the analysis after excluding those five colleges that show evidence of possible stacking of test scores above their cutoff score. The results were fairly consistent with the main findings.

Results

We estimated the effect of enrolling in three levels below college-level math versus the effect of enrolling in the middle level of college math on two different college outcomes, including the likelihood of passing a gatekeeper course in math and of receiving an associate degree or a certificate (hereon referred to as a credential) within 4 years after initial college enrollment.⁸ Table 3 shows models with different specifications that estimate the effect of enrolling in the third lowest level of math remediation versus the next highest level using Equation 2b. Model 1 represents the global results that include all students who took the Prealgebra test regardless of COMPASS score, but which controls for observable differences among students and the distance between the students’ scores and their colleges’ placement cut scores. Models 2, 3, 4, and 5 limit the comparison to students who scored within different bandwidths around the cutoff score for their college including ±10 points, ±8 points, and ±6 points, respectively. Model 5 reports the global results but adds separate squared terms for COMPASS scores above and below the cutoff to allow for a nonlinear relationship between the COMPASS score and college outcomes; it also adds college fixed effects that control for all the observable and unobservable differences among colleges by comparing students within the same college.⁹

Table 3.

Instrumental Variable Estimates of Enrolling in the Third Lowest Level of Math Remediation.

	Bandwidth	Received a credential in 4 years		Passed gatekeeper math
	Bandwidth	Estimated effect	n	Estimated effect	n
Model 1	Global	−0.0884* (0.050)	5,110	−0.0472* (0.028)	5,440
Model 2	[±10]	−0.081 (0.094)	2,930	−0.023 (0.047)	3,118
Model 3	[±8]	−0.139 (0.116)	2,470	−0.011 (0.056)	2,627
Model 4	[±6]	−0.146 (0.126)	1,918	0.010 (0.074)	2,047
Model 5^a	Global	−0.0884* (0.050)	5,110	−0.044 (0.049)	5,110

Source. Author calculations using data from Virginia Community College System.

Distance from cutoff squared and college fixed effects. Standard errors are shown in parentheses. All specifications control for distance from the cut score and covariates including gender, age, and intent including academic/transfer or workforce credential, as well as dual enrollment status.

p < .10. **p < .05. ***p < .01.

As Table 3 shows, enrolling in Prealgebra reduces the likelihood that a student will earn a credential within 4 years. According to these results, enrolling in the lowest versus the middle level of remedial math reduces the likelihood of earning a credential in 4 years by 9 to 15 percentage points. These results are highly robust to different bandwidths and specifications. By contrast, when the outcome of interest is the likelihood that the student passed gatekeeper math, the results are not robust and the coefficients change sign with choice of bandwidth; thus, even though most of the coefficients are negative, we cannot make firm conclusions from those results about the potential effects of the lowest level of remediation on the likelihood of passing gatekeeper math.

Finally, as mentioned previously, a few colleges implement decision zone policy in assigning students into different levels of remedial coursework. This might violate the key assumption underlying RDD if certain types of students are more likely to be recommended out of the lowest level of remedial math than others. To address this concern, we conducted a robustness check by excluding the three colleges that included decision zones.

A related issue is the possibility of retesting. If colleges allow students to retake the placement exam, we would be concerned that certain types of students might be more likely to take advantage of this opportunity to change their original remedial assignment. Unfortunately, there is no clear documentation regarding which colleges allow retesting. Although interviews with the central VCCS administrators indicated that retesting is typically not allowed in all the VCCS colleges, we cannot rule out the possibility that retesting might occur in certain colleges. Therefore, we conducted a separate robustness check by excluding colleges where retesting might be allowed. Specifically, we plotted the histogram of Prealgebra test scores for each college to identify the colleges in which there is stacking of test scores just above the cutoff score, which would suggest manipulation of remedial assignment through retesting. We identified five colleges with discontinuity of test scores around the college’s assignment threshold; we then conducted a robustness check on a subsample of colleges excluding those five.

Table 4 shows the results from the two robustness checks: a subsample that excludes colleges with decision zone and another subsample that excludes colleges that show evidence of retesting. We can see that the results for obtaining a credential in 4 years are highly robust when we excluded either colleges with decision zones or colleges with retesting. These results closely approximate the main result, although they increase the range of the negative effects, suggesting it could lie between −3% and −20%. The results for passing gatekeeper math, by contrast, are highly sensitive to which colleges are excluded from the sample. Based on both the main model and the sensitivity checks, we can conclude that the Virginia data suggest that enrolling in the lowest level of remediation as opposed to the middle level reduces students’ likelihood of receiving a credential, but that is unclear how enrollment in the lowest level of remediation affects the likelihood of passing the gatekeeper course in math.

Table 4.

Sensitivity Checks.

	Bandwidth	Excluding students at colleges with decision zones				Excluding students at colleges with potential retesting
		Received a credential in 4 years^a		Passed gatekeeper math^b		Received a credential in 4 years		Passed gatekeeper math
		Estimated effect	n	Estimated effect	n	Estimated effect	n	Estimated effect	n
Model 1	Global	−0.107* (−0.059)	3,361	0.003 (−0.048)	5,440	−0.0980* (−0.050)	4,971	−0.058 (−0.049)	4,971
Model 2	[±10]	−0.027 (−0.089)	1,978	0.065 (−0.073)	3,118	−0.106 (−0.095)	2,847	−0.026 (−0.091)	2,847
Model 3	[±8]	−0.100 (−0.099)	1,683	0.079 (−0.082)	2,627	−0.155 (−0.118)	2,402	−0.035 (−0.113)	2,402
Model 4	[±6]	−0.109 (−0.098)	1,306	0.040 (−0.081)	2,047	−0.197 (−0.130)	1,873	−0.127 (−0.127)	1,873

Source. Author calculations using data from Virginia Community College System.

Covariates and distance from cutoff.

Covariates and linear and squared distance from cutoff. Standard errors are shown in parentheses. All specifications include controls for the linear distance from the cutoff score and covariates including gender, age, intent including academic/transfer or workforce credential, as well as dual enrollment status.

p < .10. **p < .05. ***p < .01.

Discussion and Conclusion

While a growing number of studies have examined the causal impacts of developmental coursework on student academic outcomes, almost all of them focus on comparing students at the margin of needing remediation. Yet, traditional developmental programs typically consist of a set of multiple courses that students must enroll in sequentially and little is known about whether increased dosage through lengthy developmental sequence could indeed benefit students with very low skills. This study asks the following causal question: Would students with the lowest skills in math have better or worse outcomes if they were required to take only two remedial classes, rather than three? The answer from Virginia’s data is that, at least for students who are at the margins of being assigned to the lowest level of remediation, being required to taking the longest remedial math sequence does not provide any benefit in terms of either completing the gatekeeper math or receiving an associate degree.

Besides our study, Boatman and Long (2017) provided the only other causal estimate in the literature that includes students who are required to take multiple remedial classes in math; they found that being assigned to the lowest remedial math reduces the likelihood of credential completion by 5 percentage points compared with similar students assigned to the comparatively shorter sequence. We also identified negative impacts of the longest sequence of developmental math on a timely awarding of credentials although the estimate did not reach statistical significance when we restrict the sample to the narrow band around the cutoff score. Yet, considering that the requirement of one additional remedial math course implies substantial economic, psychologic, and academic burdens on students, as well as tremendous volume of resources spent at the institutional level, the lack of any positive impact on student academic progress for the lowest level of remediation raises serious questions regarding the benefit of these lengthy remedial sequences for academically underprepared students. Therefore, the consistent negative impacts of the longest sequence of developmental math on student academic outcomes raise serious concerns from a policy perspective. If such impacts hold true across other community college systems, it would imply that the economic, psychologic, and academic burdens imposed by the traditional lengthy developmental sequences might in fact harm students’ academic outcomes rather than benefiting them. Considering that students from disadvantaged background are overrepresented in remedial education (e.g., Bettinger & Long, 2009), these results also imply that the traditional remedial sequence could strengthen, rather than ameliorate, educational inequality.

During the past few years, there has been ongoing national efforts in reforming traditional remedial sequence and our results provide further motivation for these efforts, particularly for shortening the sequence length. Colleges may consider combining multiple levels of developmental coursework into a shorter sequence to reduce the time it takes for students to complete their developmental math requirements. Another popular approach to reforming developmental math is to modularize the content by dividing it up into distinct topics that address specific competencies or skills (Gardenhire, Diamond, Headlam, & Weiss, 2016). These topics are typically taught as a series of one-credit modules, and students are required to take only the modules where the diagnostic placement test indicates a need for improvement and that are required for the students’ field of study. The goal therefore is to provide a more targeted support for students’ individual needs and reduce the time required to complete developmental education.

Current reforms to accelerate remedial sequence have already been in place across all colleges at VCCS since 2009. Starting from the spring of 2012, developmental math has been taught as a series of nine one-credit modules, and students are only required to take the modules in which the diagnostic placement test has indicated a need for improvement and is required for the students’ field of study (Asera, 2011). The reforms also aim to improve the actual course content and delivery by better aligning remedial coursework with college-level math courses, and also by providing more professional development for course instructors (Developmental Education Task Force, 2010). A recent study that reports findings on early impacts of the developmental math reforms in Virginia indicate that modularized developmental math curriculum enables students to skip portions of the developmental curriculum and make faster progress to college-level math than would have been possible preredesign (Bickerstaff, Fay, & Trimble, 2016).

In terms of implications for future research, a shortcoming of this and other previous causal studies is that it is impossible to know exactly why the traditional long developmental sequence is ineffective. It is likely that the negative effects of having to expend additional time and money offset or even outweigh any positive effects of instruction. If that were the case, these null to negative effects of long sequence of remediation could be present even when the remedial curriculum develops students’ academic skills. Yet, in addition to the length of the sequence, there are a number of other possible explanations for the negative impacts of the lowest remedial math on student outcomes. One plausible explanation for poor outcomes could be that students who are assigned to the lowest level of remediation have increased the time to degree substantially, which increases the chance that students drop out from college before attaining a degree (Boylan & Saxon, 1999). Finally, it is also possible that the algebra-based content of the developmental math courses may not have been tied closely to the college-level courses and therefore are not very useful for subsequent academic learning. Qualitative analysis is thus needed to provide more information about the specific mechanisms that contribute to the success or failure of remedial education.

In addition, it is somewhat perplexing that we identified negative impacts of the longest math developmental sequence on degree attainment in some of our model specifications without identifying any evidence of such negative impacts on short-term academic outcomes such as math gatekeeper completion. One possible explanation is that we have comparatively short follow-up window to track students’ graduation. Community colleges typically expect students to complete a degree or certificate within 2 years. Although we have allowed for 200% of that normal graduation time, a nontrivial proportion of students were still enrolled in college by the end of the tracking period. Considering that students assigned to developmental education cannot enroll in most college-level credit-bearing courses until they complete their developmental requirements, being assigned to the longest math developmental sequence may substantially delay students’ progress toward a degree (Melguizo et al., 2011). Therefore, future studies with data that have a longer tracking period may wish to explore whether the observed gap in graduation rate decreases or even disappears over time.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Notes

Author Biographies

Di Xu is an assistant professor of Educational Policy and Social Context at the University of California, Irvine. Xu holds a PhD in economics and education from Teachers College, Columbia University. Her research examines the impacts of educational programs and policies on student academic performance, persistence, and degree completion at the postsecondary educaiton level, with a particular focus on community college students.

Mina Dadgar is the director of Research at the Career Laddres Project (CLP) and holds a PhD in economics and education from Teachers College, Columbia University. Dadgar works directly with the CLP leadership team and partners to create a research agenda that ensures that data and research is used efficiently to examine opportunities for improving improving low-income and first-generation students’ opportunities to finish college and find long lasting careers.

References

Asera

(2011). Innovation at scale: How Virginia community colleges are collaborating to improve developmental education and increase student success. Washington, DC: Jobs for the Future. Retrieved from http://www.jff.org/publications/innovation-scale-how-virginia-community-colleges-are-collaborating-improve

Bailey

Jeong

D. W.

Cho

S. W.

(2010). Referral, enrollment, and completion in developmental education sequences in community colleges. Economics of Education Review, 29, 255-270. doi:10.1016/j.econedurev.2009.09.002

Banks

Oldfield

(2006, March). Understanding pensions: Cognitive function, numerical ability and retirement saving (IFS Working Paper W06/05). London, England: The Institute for Fiscal Studies. doi:10.1920/wp.ifs.2006.0605

Becker

(1994). Human Capital: A Theoretical and Empirical Analysis with Special Reference to Education (3rd ed.). Chicago, IL: The Univeristy of Chicago Press.

Bettinger

Long

(2009). Addressing the needs of under-prepared college students: Does college remediation work? Journal of Human Resources, 44, 736-771. doi:10.3368/jhr.44.3.736

Bickerstaff

Fay

M. P.

Trimble

M. J.

(2016, May). Modularization in developmental mathematics in two states: Implementation and early outcomes (Working Paper No.87). New York, NY: Community College Research Center, Teachers College, Columbia University.

Boatman

Long

B. T.

(2017). Does remediation work for all students? How the effects of postsecondary remedial and developmental courses vary by level of academic preparation. Educational Evaluation and Policy Analysis. Advance online publication. doi:10.3102/0162373717715708

Boylan

Saxon

(1999). What works in remediation: Lessons from 30 years of research (Prepared for The League for Innovation in the Community College). Boone, NC: National Center for Development Education. Retrieved from http://www.ncde.appstate.edu/reserve_reading/what_works.htm

Calcagno

Long

(2008, April). The impact of postsecondary remediation using a regression discontinuity design: Addressing endogenous sorting and noncompliance (NBER Working Paper No.14194). Cambridge, MA: National Bureau of Economic Research.

10.

Developmental Education Task Force. (2010). The critical point: Redesigning the developmental math education in Virginia’s community colleges. Retrieved from http://cdn.vccs.edu/wp-content/uploads/2013/07/The_Critical_Point-DMRT_Report_082010_pdf.pdf

11.

Gardenhire

Diamond

Headlam

Weiss

(2016). At their own pace: Interim findings from an evaluation of computer-assisted, modular approach to developmental math. New York, NY: MDRC. Retrieved from http://files.eric.ed.gov/fulltext/ED567012.pdf

12.

Grubb

W. N.

(with Gabriner

). (2013). Basic skills education in community colleges: Inside and outside classrooms. New York, NY: Routledge.

13.

Hodara

(2012). Language minority students at community college: How do developmental education and English as a second language affect their educational outcomes? (Doctoral dissertation). Retrieved from ProQuest Dissertations & Theses Full Text database. (Order No. 1014026879)

14.

Hodara

Jaggars

S. S.

Karp

M. M.

(2012, November). Improving developmental education assessment and placement: Lessons from community colleges across the country (CCRC Working Paper No. 51). New York, NY: Community College Research Center, Teachers College, Columbia University.

15.

Hodara

(2016). Does developmental education improve labor market outcomes? Evidence from two states. American Educational Research Journal, 53, 781-813. doi:10.3102/0002831216647790

16.

Hughes

K. L.

Scott-Clayton

(2011). Assessing developmental assessment in community colleges. Community College Review, 39, 327-351. doi:10.1177/0091552111426898

17.

Imbens

Lemieux

(2008). Regression discontinuity designs: A guide to practice. Journal of Econometrics, 142, 615-635. doi:10.1016/j.jeconom.2007.05.001

18.

Lee

D. S.

Lemieux

(2010). Regression discontinuity designs in economics. Journal of Economic Literature, 48, 281-355. doi:10.1257/jel.48.2.281

19.

Lesik

(2007). Do developmental math programs have a causal impact on student retention? An application of discrete-time survival and regression-discontinuity analysis. Research in Higher Education, 48, 583-608. doi:10.1007/s11162-006-9036-1

20.

Levin

Calcagno

J. C.

(2008). Remediation in the community college: An evaluator’s perspective. Community College Review, 35, 181-207. doi:10.1177/0091552107310118

21.

Martorell

McFarlin

(2011). Help or hindrance? The effects of college remediation on academic and labor market outcomes. The Review of Economics and Statistics, 93, 436-454. doi:10.1162/REST_a_00098

22.

McCabe

R. H.

(2003). Yes we can!: A community college guide for developing America’s underprepared. Phoenix, AZ: League for Innovation in the Community College.

23.

Melguizo

Bos

Prather

(2011). Is developmental education helping community college students persist? A critical review of the literature. American Behavioral Scientist, 55, 173-184. doi:10.1177/0002764210381873

24.

Murnane

R. J.

Willett

J. B.

Levy

(1995). The growing importance of cognitive skills in wage determination. Review of Economics and Statistics, 78, 251-266.

25.

Northern Virginia Community College Course Catalog. (2004). Math course descriptions (Northern Virginia Community College Course Catalog 2004-2005). Retrieved from http://www.nvcc.edu/catalog/cat2004/descript/subject.asp?subject=MTH

26.

Papay

J. P.

Murnane

R. J.

Willett

J. B.

(2016). The impact of test score labels on human-capital investment decisions. Journal of Human Resources, 51, 357-388. doi:10.3368/jhr.51.2.0713-5837R

27.

Rivera-Batiz

F. L.

(1992). Quantitative literacy and the likelihood of employment among young adults in the United States. The Journal of Human Resources, 27, 313-328.

28.

Scott-Clayton

Rodriguez

(2015). Development, discouragement, or diversion? New evidence on the effects of college remediation policy. Education Finance and Policy, 10, 4-45. doi:10.1162/EDFP_a_00150

29.

Steele

C. M.

Aronson

(1995). Stereotype threat and the intellectual test performance of African Americans. Journal of Personality and Social Psychology, 69, 797-811.

30.

Venezia

Bracco

K. R.

Nodine

(2010). One shot deal? Students’ perceptions of assessment and course placement in California’s community colleges. San Francisco, CA: WestEd.

31.

(2016). Assistance or obstacles? The impact of different levels of English remediation on under-prepared students in community colleges. Educational Researcher, 45, 496-507. doi:10.3102/0013189X16683401