Abstract
We use regression discontinuity design to examine the impact of placing into developmental math on science, technology, engineering, and mathematics (STEM) participation in community colleges and whether these relationships differ for underrepresented racially minoritized (URM), women, STEM-oriented, and STEM-aspiring students. Results show that lower math placement was a deterrent to both math progression and STEM participation, especially for those at the margin of placement in transfer-level math. Subgroup analyses suggest that lower math placement may have supported women, and to a lesser extent URM students, in completing transferable STEM credits. For STEM-oriented students, we find suggestive evidence that lower placement deterred them from accumulating transferable credits. We discuss how community colleges can support these students and bolster the community college STEM pathway.
Retaining college students in science, technology, engineering, and mathematics (STEM) fields can increase STEM degree production and bolster competitiveness in the U.S. workforce (President’s Council of Advisors on Science and Technology, 2012). As such, attrition from STEM pathways and inequality in STEM participation and attainment have motivated ample scholarship on ways to fortify the STEM pipeline. Researchers have importantly focused on such topics as financial incentives for STEM majors (Evans, 2017), developing student interest and engagement in STEM topics through academic exposure (Crisp et al., 2009), advising and early college interactions and momentum (Wang, 2013b, 2015), fostering math and science identities (Sax et al., 2015), and on creating welcoming academic environments with supportive campus and departmental climates (Good et al., 2012; Rincón & George-Jackson, 2016).
There has surprisingly been less focus on structural barriers to STEM participation such as math prerequisites and postsecondary math remediation. This is important because research has documented how students build STEM momentum and STEM self-efficacy as they take science and math courses in high school (Gottfried & Bozick, 2016; Wang, 2013a, 2015). However, this momentum may fade as students, particularly those in the nation’s community colleges, encounter remedial/developmental math coursework and math prerequisites required for entry into STEM fields. This may be particularly acute for students of color, who attend community colleges at higher rates and are therefore more subject to remediation than academically equivalent peers at 4-year institutions (Attewell et al., 2006).
Furthermore, most studies of postsecondary math remediation have focused on generic outcomes such as gatekeeper course completion, dropout, year-to-year persistence, credit accumulation, and degree attainment (e.g., Bettinger & Long, 2009; Clotfelter et al., 2015; Lesik, 2006). Some have examined learning gains (Scott-Clayton & Rodriguez, 2015) and wages (Martorell & McFarlin, 2011), but there is scant research examining the impact of math remediation on STEM participation. These connections are important to consider because evidence suggests that math remediation may be more of a diversion rather than a gateway to enrollment in STEM courses (Bryk & Treisman, 2010).
This study investigates whether assignment to a developmental/remedial math course affects the likelihood of STEM participation in community college. We do so using transcript data from a large urban community college district (LUCCD) in California that serves about 200,000 students each year, the vast majority of whom are students of color. With access to demographic information, assessment and placement results, and course enrollment records, we link math course placement to enrollment in STEM courses for 8 years after initial community college enrollment. Specifically, we capitalize on the system of placement cutoffs used to assign students to various math courses in the developmental math sequence and use a regression discontinuity (RD) design to determine the impact of these course assignments. We focus on STEM-relevant outcomes such as completion of the transfer-level math (TLM) course and progress toward STEM credentials. We also estimate the impact of lower placement on non-STEM credit accumulation to establish (a) whether the effect on STEM outcomes is statistically distinguishable and unique from the effect on non-STEM outcomes and (b) whether lower level math placement diverted students from STEM pathways. If students scoring just below the placement cutoff differ from their peers scoring just above with respect to their subsequent STEM participation, then this can be interpreted as a consequence of developmental math placement.
We conduct subgroup analyses to determine whether there are differences in the relationship between developmental math placement and STEM participation for underrepresented racially minoritized (URM) students and women. We define URM students as those who are underrepresented in STEM fields, including Black, Hispanic, and Native American students. While we would have liked to include Pacific Islanders and other Asian students who are underrepresented in STEM, we are unable to identify these students with existing data. Previous studies show that URM students are more likely to populate lower levels of developmental math (Fong & Melguizo, 2017), less likely to take advanced STEM courses in community college (Bahr et al., 2017), and more susceptible to attriting from STEM pathways (Chen & Soldner, 2013). However, studies have not yet determined the causal link between math remediation and advanced STEM course-taking among women and URM students. This analysis is important because placement in lower level math may work to confirm women and URM students’ doubts about their “fit” in STEM fields (Ben-Zeev et al., 2017; Walton et al., 2015) and subsequently more strongly deter them from STEM pathways relative to their White, Asian, and/or male peers. This study can therefore provide new evidence on whether math remediation impacts underrepresented students’ entry into STEM pathways. Investigating this link is timely given the ongoing need to diversify the STEM talent pool (National Science Foundation, 2017).
A unique contribution of our study is that we also examine differences in the impact of math remediation for students with varying STEM backgrounds. We use students’ high school transcripts and community college application data to identify two nonmutually exclusive student groups: STEM-oriented students, who completed significant amounts of STEM coursework in high school and may therefore have developed considerable STEM self-efficacy and momentum (Wang, 2015), and STEM-aspiring students, who indicated interest in STEM on their college application forms. 1 We include and distinguish between these two groups because preparation and aspiration are both important predictors of STEM participation (Britner & Pajares, 2006). STEM career aspiration is a commonly used construct in STEM education research, as it captures motivation and interest (e.g., Saw et al., 2018). We assume in the context of our study that STEM-aspiring students may be more interested in taking STEM courses and pursuing STEM credentials in college, and may therefore be differentially affected by remedial math placement. However, research also shows that STEM aspirations are volatile and can change over time (Saw et al., 2018). We therefore also focus on students who may be more likely to enter and succeed in STEM coursework based on their academic profile and identify these students as STEM-oriented students. STEM-oriented students, having demonstrated success in prior STEM coursework, may be more likely to pursue STEM fields and therefore also be differentially affected by placement in math remediation.
We find that developmental math was not beneficial for students at the cusp of placing into TLM. Compared with students who placed directly into TLM, students who placed lower into intermediate algebra were less likely to take TLM, often a requirement for STEM courses, and less likely to complete 18 transferable STEM credits, a requirement for transfer to a 4-year college in an STEM field. In addition, students who missed the cutoff and placed lower into elementary algebra were less likely to accumulate STEM credits relative to students who placed above into intermediate algebra. However, for students who were placed in arithmetic and pre-algebra—the two lowest levels of the developmental math sequence—we find mostly null effects.
Our subgroup analyses revealed differential effects depending on the level of remediation for women, URMs, STEM-aspiring, and STEM-oriented students. At upper levels of the developmental math sequence (e.g., placing into elementary algebra or intermediate algebra), we find that lower math placement delayed math progress for all students irrespective of their gender or race. However, we find suggestive evidence that missing the TLM cutoff and placing lower particularly deterred STEM-oriented students from accumulating transferable credits. In contrast, at lower levels of the developmental math sequence (e.g., arithmetic or pre-algebra), we find that lower placement increased the likelihood that women and, to a lesser extent, URMs took upper-level STEM courses. We perform additional analyses to correct for multiple hypothesis testing and find that our results remain consistent.
In the following section, we describe the conceptual framework undergirding the study and the community college policy context. We then discuss our sharp RD strategy and explain why this method is appropriate. Finally, we present results and conclude with policy implications.
Conceptual Framework
As we explore the relationship between developmental math and STEM outcomes in community colleges, we draw upon Wang’s (2015, 2017) theory of STEM momentum. Wang’s model defines STEM momentum as progressing forward in STEM courses with direction and force during the early stages of students’ academic trajectory (Wang, 2015). Specifically, the theory underscores the importance of establishing early momentum in predicting STEM degree attainment in community college. In addition to focusing on the level of achievement and academic effort expended during the early stages of students’ college trajectory, the theory defines structural barriers that students face in community college.
Based on this model, community colleges provide access to STEM careers by cultivating students’ STEM momentum (Wang, 2015). At the same time, community college students may face counter-momentum friction due to several factors. Students may be overwhelmed by the range of possible curricular pathways, be uninspired by decontextualized instruction, receive inadequate advising, experience financial barriers, and/or find themselves in developmental courses (Wang, 2017). Specifically, the theory suggests that students would lose any momentum they developed in high school if they are trapped in lengthy developmental courses.
While the STEM momentum theory emphasizes the structural forces, social-cognitive career theory (SCCT) details the motivational linkages related to STEM participation. Rooted in Bandura’s (1986) social-cognitive theory, SCCT examines the interaction between the self and the environment and their influence on thoughts, beliefs, emotion, and behavior within the context of career development (Lent et al., 1991, 1994, 2003). SCCT hypothesizes that individuals become interested in and eventually pursue a career based on three interlinked social-cognitive processes: self-efficacy, outcome expectation, and goals (i.e., aspirations). Specifically, SCCT suggests that individuals aspire to enter STEM fields if they feel efficacious in STEM subjects and hold beliefs that pursuing STEM will produce valued outcomes (i.e., outcome expectations). The model supposes that students with the goal of pursuing STEM will take courses and make educational decisions that align with their goals. Among various theorized sources of STEM self-efficacy, research documents that prior achievement in math and science courses is among the biggest (Britner & Pajares, 2006; Lent et al., 1991; Wang, 2013a). Displaying mastery in STEM subjects, or having an STEM orientation, is linked to STEM success through increased levels of STEM self-efficacy (Britner & Pajares, 2006).
SCCT acknowledges that supports and barriers influence career development in general terms, but the theory does not define what supports and barriers mean: factors that are pertinent in the community college context. Moreover, the theory does not adequately capture the continuous, cumulative nature of educational progress. Therefore, integrating STEM momentum and SCCT allows us to consider how developmental education may affect the STEM outcomes of STEM-oriented students and STEM-aspiring students. We do not directly measure psychological variables in the study, but rather draw from SCCT to motivate our inquiry into these two groups of students.
Both models allude to the importance of students’ racial/ethnic and gender identities in affecting STEM aspiration, orientation, and participation, but do not explicitly acknowledge these characteristics. Research shows that few women and URMs major in STEM, and among those who do, they are more likely to leave STEM pathways than their White, Asian, or male peers (Chen & Soldner, 2013). Thus, personal characteristics such as race/ethnicity and gender may be important moderating variables that influence the link between self-efficacy, outcome expectations, and goals during the experience of math remediation. For example, the stereotype threat literature (Steele, 1997; Steele & Aronson, 1995) sheds some light on why students with marginalized identities are less likely to pursue STEM fields. The idea behind stereotype threat in this context is that there exists a prevailing bias that women and URMs are not “STEM material” in society, and STEM-oriented or STEM-aspiring women and URMs may underperform due to anxiety induced by these stereotypes. Relatedly, research shows that mathematical confidence and self-concept are also gendered (Sax et al., 2015), with women being 1.5 times more likely to be dissuaded from pursuing STEM fields, even after accounting for mathematical ability (Ellis et al., 2016). Not feeling a sense of belonging in mathematics-intensive fields and STEM departments on campus may also deter female and minoritized students from enrolling in and persisting in math and other STEM coursework (Good et al., 2012; Rincón & George-Jackson, 2016). In this manner, placement in remediation might especially exacerbate anxiety and feelings of inadequacy among female and URM students, and/or signal an unwelcoming climate, and subsequently impact STEM participation.
We draw from these three theories to ground our research design. Specifically, this study posits that students developed their STEM goals based on their judgments of their ability to perform well in STEM subjects (i.e., self-efficacy) and their belief that pursuing STEM is linked to valued outcomes (i.e., outcome expectations). We note that students’ self-efficacy is based on their cumulative exposure and achievement in math and science subjects from high school (i.e., STEM orientation). While SCCT specifies the motivational linkages and mechanisms driving STEM participation, STEM momentum theory underscores that STEM participation occurs within structural constraints and may be a byproduct of students’ cumulative effort in math and science subjects. College math remediation may be one source of counter-momentum friction and, as such, a detriment to STEM participation. Given the literature on racialized and gendered aspects of STEM participation, we pay particular attention to the dynamics of these variables in the relationships we explore between math remediation and STEM outcomes.
Math Remediation in STEM Pathways
The conceptual link between college math remediation and STEM outcomes in community college should not be surprising as there are a number of studies linking math performance to STEM participation and attainment. For example, studies show that students with a strong foundational training in math are less likely to drop out and more likely to persist in college (Adelman, 2006), and more likely to enter occupations requiring substantial mathematical and cognitive skills typically characteristic of STEM fields (Goodman, 2019). Some studies looked specifically at the correlation between STEM courses-taking in high school and declaring a STEM major in college (Gottfried & Bozick, 2016; Wang, 2013a, 2013b), and other studies examined the relationship between college course-taking, majoring in STEM fields, and bachelor’s degree attainment in STEM (Wang, 2015). In each study, math was a key factor in STEM participation.
Developmental math is a particular concern in STEM pathways as it is a gatekeeper course for most community college students. Several studies, many of which employed RD designs, found that developmental math placement led to a decreased likelihood of remaining enrolled in community college (Boatman & Long, 2018) and virtually no effect on labor market outcomes (Martorell & McFarlin, 2011). Other studies found that the impact of remediation can vary by institutional context (Melguizo et al., 2016) and by students’ level of academic preparation (Boatman & Long, 2018; Scott-Clayton & Rodriguez, 2015). Overall, a meta-analysis of RD studies showed that placing into developmental education has a negative and statistically significant effect on the probability of passing the college-level course in which remediation was needed, college credits earned, and attainment (Valentine et al., 2017).
Although a majority of students begin their college journey in developmental math, there lacks empirical evidence on whether developmental math serves as a diversion from pursuing and persisting in STEM. Developmental math is an important juncture in STEM pathways as approximately 65% of first-time California community college enrollees start their college trajectory in developmental math, and a high proportion of these students are from minoritized backgrounds (Rodriguez et al., 2017). However, Bahr et al. (2017) found that California community college students in STEM pathways who started off in the lowest nondevelopmental course (college algebra) were much less likely to advance to the next level than those who began one level higher (trigonometry). Although that study provided a useful curricular map of STEM pathways in community college, the authors excluded students who placed in developmental math. Thus, the effect of math remediation in STEM pathways is an underexplored area of research. This study adds to this literature by examining the impact of developmental math on STEM participation in community colleges. We focus on four community colleges in a district in California.
Policy Context
Between 2005 and 2008, most community colleges in California used a placement test to assign students to developmental or college-level courses and supplemented the test score with points from various additional measures, also known as “multiple measures” (Melguizo et al., 2014; Ngo & Kwon, 2015; Rodriguez et al., 2017). This meant that nearly all degree-seeking students, regardless of major, took the placement test before enrolling in math and English. Across the California community colleges, the four developmental math levels listed in order are arithmetic, pre-algebra, elementary algebra, and intermediate algebra. TLM follows this and has intermediate algebra as a prerequisite. Students could decide to enroll in the level in which they placed, or in any lower level, but were not allowed to enroll in the level above unless they completed a process to challenge their results or retested at least one semester later. 2 About 10% of students in our sample retested. To avoid bias associated with using retest scores, we use students’ first test score in all analyses.
Given the decentralized nature of the California community college system, each college set different cut-scores to determine where their students should be placed in the developmental math sequence. 3 As a result, there was a range of cut-scores used to place students into the four developmental math levels and TLM. For example, students would need to receive a score of 60.5 or higher to place in intermediate algebra at one college, whereas at another college, students would need a score of 80 or higher. While students were placed using different cut-scores depending on the college, all students in these colleges during the time frame of our study took the same computer-adaptive test called ACCUPLACER. Furthermore, STEM majors and non-STEM majors in the same college faced the same cutoffs.
Critical to the internal validity of this study design, students did not know the cut-scores that determined placement. Furthermore, if administrators systematically thought that students who scored just below the cutoff had higher math ability and exempted those students, the placement exam score itself may be correlated with STEM participation. In another scenario, there would be selection bias around the cutoff if students who were more motivated, better test-takers, and/or were more meticulous were systematically clustered above the placement cutoff. These scenarios are unlikely because students took a computer-adaptive test that automatically adjusted depending on how each student responded to previous questions.
Data and Sample
The data used in this study are linked longitudinal transcript data obtained through partnerships with a LUCCD and a large urban school district (LUSD) in the same metropolitan area. The LUCCD enrolls more than 200,000 students each year, a significant portion of whom are low-income, and/or first-generation college students. LUSD graduates about 37,000 students each year, with about one third of students enrolling in LUCCD (see Appendix Table A1, available in the online version of the article). We construct a unique student-level data set of LUSD students who enrolled in a LUCCD college within 3 years of high school graduation during 2005 to 2008 and who were not concurrent high school students. The LUSD-LUCCD data set includes high school as well as community college course-taking information, placement test score results, and demographic data for each student found in both systems. We are able to track these students’ community college outcomes through 2016, and so we follow each cohort for 8 years.
Outcomes
The momentum literature underscores the importance of initial academic course load and effort expended in pursuing academic outcomes (Attewell et al., 2012; Wang, 2015, 2017). Drawing from this definition, we examine steady progression through the developmental math sequence. Specifically, we examine the following indicators of math progression: (a) taking elementary algebra, the math requirement for an associate’s degree at the time; (b) taking intermediate algebra, a common prerequisite for advanced math and science courses; and (c) taking TLM, a course required for transfer. Next, we examine (d) STEM credits attempted in 3 years, (e) non-STEM credits attempted in 3 years, (d) STEM credits completed in 6 years, and (e) non-STEM credits completed in 6 years to explore STEM attainment and whether developmental education deters students from STEM coursework. Finally, we also generate (f) a dichotomous indicator of attempting 18 STEM credits overall. 4 Students who aim to transfer as a junior with an STEM major need at least 18 semester STEM credits. 5 STEM credits are defined as all transferable math and science courses taken in community college and non-STEM credits are defined as transferable social sciences, humanities, and English credits.
Examining STEM participation rather than summative measures such as degree attainment and/or transfer rates is especially informative in the community college setting. Several studies document gains in the labor market from taking just a few community college courses (Dadgar & Trimble, 2015; Hodara & Xu, 2016). In particular, accumulating STEM credits is a proxy for learning relevant skills for success in the knowledge economy and warrants attention irrespective of whether students graduate with an STEM degree. Indeed, several reports underscore the growing importance of an STEM-focused education even if students decide not to work in an STEM field (National Science Board, 2015, 2018).
Four Focal Colleges
Our initial sample includes 30,208 LUSD-LUCCD students who took the math placement test but for our analyses we focus on 11,352 students who took the math assessment and enrolled at one of four LUCCD colleges—colleges A, B, C, and D—between 2005 and 2008. We chose these four community colleges because they administered the same math placement test (ACCUPLACER) during the same time span, allowing us to pool students who attended different colleges, and before placement policy changes were enacted. Among these 11,352 students, 524 placed in TLM, 2,281 placed in intermediate algebra, 2,680 placed in elementary algebra, 3,801 placed in pre-algebra, and 2,066 placed in arithmetic.
Table 1 examines the generalizability of our analytical samples relative to all LUSD-LUCCD students. Specifically, Table 1 compares the demographic breakdown of all LUSD-LUCCD students with students from colleges A, B, C, and D as well as with students by levels of developmental math placement. First, we observe that the characteristics of the students at the four focal colleges are similar to LUSD-LUCCD students. Specifically, about half of the students were women, more than 80% of students were identified as URMs, and over a third of the students intended to transfer. Looking at the different math remediation levels, there were notable demographic trends. For one, fewer URMs and women populated the intermediate algebra or TLM levels than the elementary or pre-algebra, or arithmetic levels. Also, perhaps unsurprisingly, more students who placed in intermediate algebra or TLM displayed STEM orientation and transfer intent than students who placed in arithmetic or pre-algebra. These statistics suggest that students who placed in upper levels displayed stronger academic preparation and were more interested in STEM than students who placed in lower levels.
Demographic Breakdown of LUSD-LUCCD Students, 2005–2008
Note. The numbers only include students who were enrolled between 2005 and 2008 and who were not concurrent high school students. LUSD = large urban school district; LUCCD = large urban community college district; ABCD = students attending one of the four colleges within this LUCCD; TLM = transfer-level math; IA = intermediate algebra; EA = elementary algebra; PA = pre-algebra; AR = arithmetic; URM = underrepresented racially minoritized; STEM = science, technology, engineering, and mathematics; HS = high school; GPA = grade point average.
Empirical Strategy
RD
We use an RD in which assignment to treatment is determined by a predictor, or running variable (Imbens & Lemieux, 2008; Lee & Lemieux, 2010). Specifically, if there is a policy or some administrative decision that creates a discontinuity at some threshold of the predictor, those who fall slightly below or slightly above the threshold are as good as randomly assigned. The predictor in this study is the math placement test score. Students who just missed the placement cutoff were assigned to varying levels of developmental math whereas students who scored higher than the cutoff were placed one level above.
Because there are four cut-scores that determine placement into arithmetic, pre-algebra, elementary algebra, intermediate algebra, and TLM, we run four separate RD analyses for students around these cut-scores. In Table A2 and Figure A1 (available in the online version of the journal), we show the sample size of those included in the analyses of each math cutoffs. Notably, students did not know the criteria that determined which math level they would place; therefore, there should not be any selection around the threshold. In effect, students who just missed the cutoff and placed lower and those who just exceeded the cutoff and placed above were similar in expectation.
We pool together the students from the four focal colleges by standardizing student scores within college and then centering scores around each cutoff. Even though the four colleges administered the same tests, they have set different cutoffs to place students (e.g., as mentioned above, a score of 60.5 for intermediate algebra placement in one college vs. 80 in another). Standardizing the running variable first and then centering using the cut-scores allows us to account for the fact that it is easier to attain a given cutoff at one campus relative to another campus. We do this for all four cutoffs to pool the colleges together and increase the sample size and generalizability of this study.
We estimate a local linear regression separately for each of the four cutoffs using the 0.5 standard deviation (SD), 0.75 SD, and 1 SD. Because observations that are far away from the cutoff may impact parametric estimates, local linear regression estimator relaxes the assumptions about the functional form away from the cutoff (Calonico et al., 2014; Calonico et al., 2017; Imbens & Kalyanaraman, 2012). We also calculated the optimal bandwidth using the method described by Calonico et al. (2017). The bandwidths varied by model and outcome, ranging between 0.47 SD and 1.03 SD. We therefore present estimates for 0.5 SD, 0.75 SD, and 1.0 SD and provide optimal bandwidth estimates for each outcome in Appendix Tables A6 and A7 (available in the online version of the journal).
The RD model is presented in the following equation:
where
Figure 1 shows the percentage of students who were placed at each level according to their placement results (converted to standard deviation units). The fraction of students who place in upper-level math sharply increases as scores pass the threshold, indicating the appropriateness of a sharp RD. However, placement does not neatly correspond to enrollment. While compliance with placement results is higher at upper levels of math placement, we still see that enrollment in the course is not 100% (see Appendix Figure A2 in the online version of the journal). Therefore, all of our results should be interpreted as the effect of math placement or the intent-to-treat estimate.
Math placement across placement score distribution.
Subgroup Analysis
There are reasons to hypothesize that remediation may affect women and URMs differently when examining STEM participation. Research shows that women and URM students are more susceptible to attriting from STEM pathways than men and URM students (Chen & Soldner, 2013). Also, missing the cutoff and placing in lower levels of developmental math may inadvertently send a signal to women and URMs that they are not equipped to pursue STEM. Therefore, math remediation may especially discourage women and URMs from pursuing and persisting in STEM fields.
In contrast, we hypothesize that STEM-aspiring and STEM-oriented students may take more STEM courses relative to their peers despite lower placement. These students developed STEM momentum in high school by taking advanced math and science courses (whom we call STEM-oriented) or started college with a clear interest in STEM on their college application (whom we call STEM-aspiring). We surmise that STEM-oriented and STEM-aspiring students may be more likely to persist in STEM pathways than their peers without an interest or background in STEM. The equation (2) shows how we assess subgroup differences in STEM participation.
where, as before,
Given the large number of outcomes and numerous heterogeneity analyses, it is possible that some results are significant by chance. We therefore subject all of our inferences to multiple hypothesis corrections. Our supplemental analysis calculates the expected proportion of rejections that are Type I errors (Anderson, 2008; Benjamini & Hochberg, 1995). The estimated sharpened q values, or the false detection rate, adjust the p value accordingly. These results are available in Appendix Tables A9, A10, and A11 (available in the online version of the journal).
RD Validity
We check to see that students’ placement status is quasi-randomly assigned close to the cutoff. If we observe a discontinuity, we would be worried that students near the threshold behaved differently in unobserved ways. First, we estimate whether the density of the running variable is smooth using the kernel density estimator along four pooled cut-scores (Cattaneo et al., 2018). We also plot densities in Figure 2 and conducted the McCrary (2008) density test. There are no visible jumps around the threshold along all four placement cutoffs, and none of the estimated discontinuities were statistically significant.
Discontinuity in the density of the running variable across four placement cutoffs.
In addition, Table 2 displays the comparison of sample means across a range of baseline characteristics along the four cutoffs. There are noticeable differences in means when comparing students in upper levels of the math sequence compared with students in lower levels of the math sequence. First, more women and URM students populated lower placement levels relative to upper levels. Second, the average high school GPA of students around the intermediate algebra/TLM was higher than the average high school GPA of students in the lowest two levels of the math sequence (pre-algebra/elementary algebra and arithmetic/pre-algebra). We test for continuity in these variables across the RD thresholds by substituting each variable as the outcome in the RD equation described above. At a ±0.75 SD unit bandwidth, we did not find meaningful differences around the cutoffs. Overall, students who placed lower, on average, were similar with respect to race, gender, achievement, and aspiration as students who placed above at each cutoff.
Test of Baseline Equivalence Around Levels of Cutoff, ±0.75 SD Bandwidth
Note. The coefficient estimates the likelihood of observing a significant treatment effect across various characteristics around ±0.75 SD bandwidth. All estimations include college-by-semester-term fixed effects. IA = intermediate algebra; TLM = transfer-level math; EA = elementary algebra; RD = regression discontinuity; URM = underrepresented racially minoritized; STEM = science, technology, engineering, and mathematics; HS = high school; GPA = grade point average.
Nonenrollment
Not all students who took the placement test subsequently enrolled in a college course, and we do not observe the outcomes of these nonenrollees. One concern this produces with our estimation strategy is that subsequent outcomes may be due to nonenrollment rather than the treatment (placement in lower level math). To explore this concern, we first examine whether placement affects students’ decision to enroll in college. Figure 3 and Appendix A3 (available in the online version of the journal) indicate that remedial assignment did not affect students’ college enrollment rate at the margin, aligning with prior literature that developmental placement does not discourage enrollment in community college (Martorell et al., 2015).
Fraction of students enrolled in college after math assessment.
Similar to previous RD analyses, we recode missing outcomes to zeros (see, for example, Martorell & McFarlin, 2011; Scott-Clayton & Rodriguez, 2015). The zero in one of our outcomes, ever enrolling in a TLM course, could represent students who enrolled in college but never took a TLM course, those who dropped out or withdrew, or those who delayed their enrollment altogether beyond the time frame of our data. By recoding missing outcomes to zeros, we are able to preserve our initial identification and estimate a causal intent-to-treat effect. Nevertheless, we elaborate on the potential selection bias associated with differential take-up of remedial assignment and how this may affect our estimates in the sensitivity section.
Results
We start by inspecting graphical displays of discontinuities at each of the four cutoffs, and for each of the outcomes of interest. We first create the mean value of each dependent variable within 0.15 standard deviation units as the pre-specified bin and fit a separate linear regression line at each side of the normalized value of zero. Upon examining all of the graphs, we find the most visible discontinuities around the intermediate algebra/TLM cutoff. Therefore, we display the scatter plots of the intermediate algebra/TLM cutoff here and include the scatter plots of the remaining math levels in Appendix Figures A3 (available in the online version of the journal). Figure 4 shows the relationship between placing lower into intermediate algebra or above into TLM on math progression and STEM outcomes.
Discontinuity at the intermediate algebra/transfer-level math cutoff on math progression and STEM participation.
By visual inspection, we see that students who placed in intermediate algebra were less likely to ever take TLM. Also, placing above, into TLM, appears to be associated with attempting more transferable STEM units and attempting at least 18 transferable STEM credits in the long-term, a requirement for transfer to a 4-year institution. Below, we estimate the magnitude of the jumps that we see in Figure 4, as well as those for the other math cutoffs.
Main Estimation Results
In Table 3 we estimate the magnitude of the discontinuities that we visually observe in Figure 4. We focus on the results at the ±0.75 SD bandwidth, but show the ±0.5 SD and ±1.0 SD bandwidths to assess sensitivity to bandwidth size. Examining the effects of lower placement on students’ progression through the developmental math sequence, we see a significant negative effect on math progression at the upper end of the developmental math sequence. Students who placed lower into intermediate algebra (vs. TLM) and into elementary algebra (vs. intermediate algebra) were less likely to have taken college-level math and TLM. Specifically, students who placed in intermediate algebra were 18.7 percentage points (p < .01) less likely to have ever taken TLM at the ±0.75 SD bandwidth, with estimates of similar magnitude for the other bandwidths. Similarly, placing into elementary algebra instead of intermediate algebra resulted in 12.6 percentage points (p < .01) decreased probability of ever taking intermediate algebra for students around the ±0.75 SD bandwidth. This finding is notable because intermediate algebra is a common prerequisite for advanced STEM courses. In contrast, examining the pre-algebra/elementary algebra cutoff and the arithmetic/pre-algebra cutoff, we do not find any robust significant results. This suggests that students who placed below were just as likely to attempt upper-level math courses as students who placed directly into the higher level course.
The Effects of Developmental Math Placement on Math Progression and STEM Participation
Note. The RD coefficient estimates a local linear regression with college-by-semester-term fixed effects. The coefficient estimates the likelihood of observing a significant treatment effect on outcomes around the specified bandwidth. Standard errors are in parentheses. STEM = science, technology, engineering, and mathematics; EA = elementary algebra; IA = intermediate algebra; TLM = transfer-level math; PA = pre-algebra; AR = arithmetic; RD = regression discontinuity.
p < .10. *p < .05. **p < .01. ***p < .001.
In terms of STEM attainment, we see suggestive evidence that students were discouraged from pursuing STEM pathways due to lower placement. We find that the effect of lower math placement on STEM participation trends negatively, although most of the results do not reach statistical significance. In particular, we note that compared with students who placed in TLM, students who placed lower into intermediate algebra were less likely to complete STEM credits in 6 years and were less likely to have attempted at least 18 transferable STEM units. The results also suggest that elementary algebra placement led to a decrease in STEM credits attempted. After 3 years, students placed in elementary algebra attempted, on average, about two fewer STEM credits than their peers placed in intermediate algebra, and this gap persisted after 6 years.
Subgroup Results: Women and URM Students
Women
Table 4 displays the differential effect of developmental math placement for women relative to men across all four cutoffs. The results differ depending on the developmental math level. Examining the cutoffs at the upper end of the developmental math sequence, elementary algebra/intermediate algebra and intermediate algebra/TLM, women compared with men were no more or no less likely to take transferable STEM courses because of lower math placement. In contrast, at the pre-algebra/elementary algebra cutoff, we find evidence that lower placement may have helped increase women’s STEM participation relative to men. Specifically, Table 4 indicates that women who placed lower into pre-algebra attempted, on average, about two (3.411–0.970) more transferable STEM credits in 3 years and completed four (7.062–3.068) more transferable STEM credits in 6 years relative to men. This pattern holds even after adjusting for the probability of false rejections (Appendix Table A11 in the online version of the journal).
Differential RD Effects for Women
Note. Standard errors are in parentheses. RD = regression discontinuity; EA = elementary algebra; IA = intermediate algebra; TLM = transfer-level math; STEM = science, technology, engineering, and mathematics; PA = pre-algebra; AR = arithmetic.
p < .05. **p < .01. ***p < .001. †p < .10.
URMs
Next, Table 5 displays the differential effect of developmental math placement for URMs relative to their White and Asian peers across the various math cutoffs at the upper end of the developmental math sequence, most of the interaction coefficients are statistically insignificant, suggesting that lower placement was a deterrent for all students irrespective of race. URM students who placed in elementary algebra were about 20 percentage points less likely to take intermediate algebra relative to their peers who placed directly into intermediate algebra. At lower levels of the developmental sequence (i.e., pre-algebra vs. elementary algebra and arithmetic vs. pre-algebra), we find positive effects, suggesting that lower placement may have been beneficial for URM students relative to non-URM students. However, we caution against decisive interpretations of these results as the coefficients become statistically insignificant once we account for multiple hypothesis testing (Appendix Table A10 in the online version of the journal).
Differential RD Effects for Underrepresented Racially Minoritized Students
Note. We define underrepresented racially minoritized students as those who are underrepresented in STEM fields, including Black, Hispanic, and Native American students. While we would have liked to include Pacific Islanders and other Asian students who are underrepresented in STEM, we are unable to identify these students with existing data. Standard errors are in parentheses. RD = regression discontinuity; EA = elementary algebra; IA = intermediate algebra; TLM = transfer-level math; STEM = science, technology, engineering, and mathematics; URM = underrepresented racially minoritized; PA = pre-algebra; AR = arithmetic.
p < .05. **p < .01. ***p < .001. †p < .10.
Taken together, developmental math appears to largely have similar effects irrespective of race and gender. We find evidence that women, and to a lesser extent URMs, who placed lower into arithmetic or pre-algebra were more likely to attempt upper-level math or transferable STEM courses relative to their same-demographic peers placed one level above. However, at the upper end of the developmental math sequence, we find no evidence that lower placement improved URM and women’s math progress and STEM participation. In fact, most of the effects on math progression and STEM participation, while imprecisely measured, trend negatively suggesting at best a null effect and at worst a negative effect for women and URM students near the TLM cutoff.
Subgroup Results: STEM Aspiration and Orientation
STEM Aspiration
Table 6 shows the differential effect of developmental math placement for STEM-aspiring students relative to their non-STEM-aspiring peers across all four cutoffs. According to our conceptual framework, we would expect to see positive interaction coefficients for STEM outcomes if STEM aspiration provided a source of motivation for students to pursue STEM irrespective of math placement results. However, we generally do not see evidence of this at any math cutoffs, as nearly all of the RD interaction coefficients are insignificant. STEM-aspiring students who placed lower performed no differently with respect to STEM outcomes compared with their non-STEM-aspiring peers.
Differential RD Effects for STEM-Aspiring Students
Note. STEM-aspiring students are those who indicated interest in STEM on their application. Standard errors are in parentheses. RD = regression discontinuity; STEM = science, technology, engineering, and mathematics; EA = elementary algebra; IA = intermediate algebra; TLM = transfer-level math; URM = underrepresented racially minoritized; PA = pre-algebra; AR = arithmetic.
p < .05. **p < .01. ***p < .001. †p < .10.
While we largely do not find evidence that STEM aspiration buffered against the negative effects of lower placement, these students made behavioral choices that aligned with their goals. Looking at the main effects, STEM-aspiring students were more likely to attempt transferable STEM credits in both the short- and long-term. We do not see corresponding evidence on non-STEM units attempted, indicating that the behavioral choices were uniquely directed toward achieving STEM outcomes.
STEM Orientation
Table 7 shows the differential effect of developmental math placement for STEM-oriented students across all four cutoffs. Again, we would expect positive interaction coefficients for STEM outcomes if STEM self-efficacy and momentum provided a source of motivation to persist in the STEM pathway. Our results indicate that STEM-oriented students did not take more transferable STEM courses despite lower placement. We find suggestive evidence that lower placement may have served as a discouragement for STEM-oriented students who placed in intermediate algebra versus TLM. STEM-oriented students who placed in intermediate algebra attempted about five (−10.23+5.5) fewer non-STEM credits than non-STEM-oriented students within 3 years of enrollment. Although statistically insignificant, we also observe negative coefficients when examining STEM and non-STEM credits completed in 6 years, and the probability of completing 18 STEM credits.
Differential RD Effects for STEM-Oriented Students
Note. STEM-oriented students are those who have taken at least 3 years University of California–approved math courses and passed all of them with a C or higher and at least 2 years of University of California–approved science courses and passed all of them with a C or higher. Standard errors are in parentheses. RD = regression discontinuity; STEM = science, technology, engineering, and mathematics; EA = elementary algebra; IA = intermediate algebra; TLM = transfer-level math; URM = underrepresented racially minoritized; PA = pre-algebra; AR = arithmetic.
p < .05. **p < .01. ***p < .001. †p < .10.
Similarly, at the elementary/intermediate algebra cutoff, we find suggestive evidence that lower math placement may have discouraged STEM-oriented students from accumulating transferable credits by Year 3 of enrollment. STEM-oriented students disproportionately attempted fewer STEM and non-STEM credits than their non-STEM-oriented peers if placed lower into elementary algebra. For example, STEM-oriented students who placed in elementary algebra attempted 3.5 (−2.799 to 0.798) fewer transferable STEM credits and 5.3 (−5.695+0.463) fewer transferable non-STEM credits than STEM-oriented students who placed in intermediate algebra.
Finally, our findings suggest that lower placement may have also hindered STEM-oriented students placed furthest away from TLM. Pre-algebra placement relative to elementary algebra placement resulted in about 11 percentage points decreased likelihood of taking intermediate algebra and TLM—courses that are typically required for entry into STEM courses. However, our subgroup results pertaining to STEM-oriented students lose statistical significance once we subject our analyses to multiple hypothesis corrections (Anderson, 2008). Nevertheless, the interaction coefficients trend negative, and we therefore more cautiously interpret these findings as suggestive rather than conclusive (Appendix Table A11 in the online version of the journal).
Sensitivity Analyses
We conducted several sensitivity analyses to examine how sensitive the estimates are to the choice of bandwidth and functional form, and whether we see effects when we use a pseudo-cutoff. We also elaborate on concerns related to our sample definition of community college enrollees. Table A4 (available in the online version of the journal) shows the results using a second-order polynomial as the functional form. Similar to our main results, we see that the clearest effects are concentrated on taking an upper-level math course. Students who missed the cutoff and placed lower were less likely to take college-level math and TLM across all levels of developmental math. Next, we conducted a falsification exercise with a pseudo/placebo-cutoff by treating one side of the threshold as its own sample and looking for the discontinuity at the median of the running variable (Imbens & Lemieux, 2008). We re-estimate the same RD model using a ±0.75 SD around the pseudo-cutoff (see Table A5 in the online version of the journal). As expected, we do not see any significant RD effects on both math progression and STEM attainment. This exercise ensures that the results are attributable to the college-specific cutoffs.
Next, in Tables A6 and A7 (available in the online version of the journal), we examined sensitivity to the choice of bandwidths. Similar to other scholars (e.g., Castleman et al., 2017), the main analysis is based on a midpoint from a range of bandwidths specified after estimating developmental math progression and STEM participation outcomes. We examined the sensitivity of the results by calculating the optimal bandwidth (Calonico et al., 2017). The direction and magnitude of the estimations are robust to different specifications of the bandwidth. We present the most pertinent robustness results in the Online Appendix, and the rest are available upon request.
Finally, we redo our analysis conditioning on the sample of enrollees. Most of the nonenrollees are concentrated among those at the lower end of the developmental math sequence. There are fewer nonenrollees around the intermediate algebra/TLM cutoff compared with those around the arithmetic/pre-algebra cutoff. By recoding the missing outcomes to zero, we assume that the average outcomes of these students on STEM participation is zero. 6 Recoding missing outcomes to zero is essentially the worst-case scenario as it is highly improbable that all of the nonenrollees would have performed poorly (Lee, 2009).
Given that we “impute” zeros in our main analyses, our estimates may well be downward biased. We note two scenarios in which we may be presenting downward biased estimates. The first scenario is that we may have underestimated the benefits to lower placement. Specifically, we observe a positive effect from lower placement among certain students who were placed in the lowest levels of developmental math. Therefore, the true benefits to lower placement among those around the lowest math levels may be smaller than what we present. The second scenario is that we may have underestimated the costs to lower placement. We observe null effects on many outcomes although the coefficients are largely negative, but the true effect may be more negative than what we present.
Table A8 (available in the online version of the journal) shows the results without imputing zeros for missing outcomes. Conditioning on the sample of enrollees, we find patterns similar to the main results. For the coefficients that are statistically significant, the direction and magnitude correspond to our main results. For instance, the negative effects on STEM participation for students at the intermediate algebra/TLM and elementary algebra/intermediate algebra cutoffs are consistent. In addition, the gap in early math progression and STEM participation between URMs and women relative to men and non-URMs remained comparable when examining all students as well as only enrollees. These results are available upon request.
Limitations
The aforementioned bandwidth analyses help to address one key limitation of RD estimates—they only apply to a narrow bandwidth around the treatment cutoff. Even though consistency across bandwidths suggests that the results can be applied to more students around the placement cutoffs, the results nevertheless only apply to students scoring near the cutoff in these four colleges during the time period of the study, and we caution against generalization to different contexts. From a policy perspective, students who are on the margin of being remediated are the student population most likely to be impacted by different policy schemes as opposed to those who are farther away from the cutoffs (i.e., more academically prepared or academically underprepared). Nevertheless, our study examines the effect of lower placement at various points of the test score distribution, thereby investigating the effect of placement beyond test ability. Given that we examined the effect at multiple points of the test score distribution, we note that our effects may have broader generalizability than RD analyses that examine one cutoff.
Discussion
Bolstering the STEM pipeline, particularly for underrepresented students, has been on the national education policy agenda for the last few decades. Community colleges are seen as a linchpin in this effort to increase the number of diverse STEM graduates, but they are also faced with the reality that many community college students are referred to developmental math. While some regard developmental education as a way to prepare students for college-level courses (Bettinger & Long, 2009), others show that it is a source of discouragement or diversion from college progress (Martorell & McFarlin, 2011; Scott-Clayton & Rodriguez, 2015). In this study, we examined whether structures like developmental education, which are supposed to equip students to be successful in college-level STEM courses, precluded community colleges from developing diverse students’ STEM talents. With access to student records and college applications, we closely looked at those for whom experiences of math remediation may be most consequential with respect to STEM participation—women, URM students, students who may be STEM-oriented, and those who have STEM aspiration. This study therefore provides new evidence on the ongoing debate regarding the merits of developmental education, and in particular, its role in STEM pathways and participation.
Our results show that lower math placement, and subsequently having to take additional math prerequisites prior to STEM entry, was a deterrent to both math progression and STEM participation, especially for those near the TLM cutoff. These effects were largely not found at the lower levels of the developmental math sequence. This study underscores how barely missing the cutoffs to more advanced math levels had large implications for STEM participation.
Aligning with what the theory suggests, STEM-aspiring students at community colleges attempted more transferable STEM courses than their non-STEM-aspiring peers. Therefore, we found evidence that STEM-aspiring students made educational decisions that align with their STEM-related goals. Nevertheless, we did not find evidence that STEM aspiration buffered against the negative effects of lower math placement at the margin of the cutoff across all developmental math levels. If community colleges aim to increase STEM transfer rates, one method might be to identify STEM-aspiring students from their community college enrollment forms and determine optimal pathways from the point of math placement to STEM degree attainment or transfer so that these students can sustain their momentum.
As for students with significant high school STEM coursework, lower placement deterred these STEM-oriented students from accumulating both STEM and non-STEM credits. Like research showing that feedback about academic ability affects STEM attrition and dropout decisions (T. Stinebrickner & Stinebrickner, 2012; R. Stinebrickner & Stinebrickner, 2014), it is possible that feedback about STEM potential possibilities, here in the form of a math course placement, might have deterred students from pursuing STEM fields. Moreover, recent evidence suggests that there is a lack of alignment between high school and community college readiness standards (Melguizo & Ngo, 2020). Another means of improving STEM participation among STEM-oriented students may be to increase alignment between high school and community college standards so that these signals are more aligned and the transition more seamless (Park et al., 2020). If community college counselors were to identify STEM orientation and achievement based on high school transcripts and place students based on this information, then STEM-oriented students may be more likely to proceed directly into transfer-level courses and not experience setbacks in the STEM pathway.
Finally, we found that the RD results differed for students at the pre-algebra/elementary algebra cutoff. Women in particular who placed in pre-algebra completed four more transferable STEM credits and five more transferable non-STEM credits than men. These results indicate that the effect of lower math placement differs when examining academically underprepared students, corroborating other research findings (Boatman & Long, 2018; Scott-Clayton & Rodriguez, 2015). Incoming students with varying levels of academic preparation will likely have different academic needs. In a policy environment where community colleges are moving away from requiring developmental education, it will be important for community colleges to provide targeted academic support for incoming women to increase their STEM participation. Co-requisite models, which can provide this type of academic support as students enroll directly in college-level courses, may be an effective strategy to meet the needs of students who would otherwise benefit from developmental coursework (Logue et al., 2019).
Taken together, the findings of the study show that developmental math does introduce a structural source of counter-momentum friction in early math progression, particularly for those students at the margin of TLM. By not placing directly into TLM, and therefore having to complete the STEM course prerequisite, these students were significantly less likely to earn transferable STEM credits. This supports the findings of Wang (2015) that community colleges should help cultivate students’ early STEM momentum to usher more students into STEM pathways. Removing barriers is one way to do so, and a number of states have already moved toward making developmental education optional (e.g., Florida) or very limited (e.g., California), and/or offering college statistics as an alternative pathway. We encourage community college practitioners engaging with these developmental education reforms to identify STEM-oriented and STEM-aspiring students early on to maintain their STEM momentum.
Improving STEM participation in the nation’s community colleges, a system that is more likely to serve students who are underrepresented in STEM fields, remains a national education policy priority. Although some benefits were found for certain groups of students, overall, the study showed that developmental math may produce frictions in the STEM pathway, even for students with expressed STEM aspirations and significant STEM preparation. Given initiatives across the nation to reform developmental education, it is important to consider how changes to policy and practice may impact both general community college outcomes and the promise of the community college STEM pathway.
Supplemental Material
sj-docx-1-epa-10.3102_0162373720973727 – Supplemental material for The Effect of Developmental Math on STEM Participation in Community College: Variation by Race, Gender, Achievement, and Aspiration
Supplemental material, sj-docx-1-epa-10.3102_0162373720973727 for The Effect of Developmental Math on STEM Participation in Community College: Variation by Race, Gender, Achievement, and Aspiration by Elizabeth S. Park and Federick Ngo in Educational Evaluation and Policy Analysis
Footnotes
Acknowledgements
We thank Tatiana Melguizo, Xueli Wang, and the participants of AERA and AEFP for their comments and feedback on the paper. Opinions are those of the authors alone and do not necessarily reflect those of the granting agency or of the authors’ home institutions.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by an EAGER grant from the National Science Foundation (NSF) (2015-1544254), the Haynes Foundation, and the USC Russell Endowed Fellowship.
1
More specifically, science, technology, engineering, and mathematics (STEM)-oriented students are those who have taken at least 3 years of University of California–approved math courses and passed all of them with a C or higher and at least 2 years of University of California approved science courses and passed all of them with a C or higher. STEM-aspiring students are those who indicated STEM as their intended major in their college application.
2
Ten percent (n = 1,173) of students retested of which 29% (n = 338) retested into a higher math level. Therefore, only 3% of the students who retested placed in a higher math level than their initial math course. Most students who retested ended up placing into lower levels. Retesters have to retest if they do not enroll within a semester or within a year.
3
In Fall 2019, California community colleges were required to implement the mandates of Assembly Bill 705, which require the use of high school measures in lieu of placement tests unless colleges provide evidence that the placement test is a more effective instrument.
4
We acknowledge that completion hinges on whether students attempt the course. Therefore, there is inherent selection bias in examining completion outcomes because not all students may attempt and complete the course. For this reason, we examine students’ attempted credits within 3 years to determine early STEM momentum and completed within 6 years to assess completion.
6
The percentage of those in arithmetic (AR), pre-algebra (PA), elementary algebra (EA), and intermediate algebra (IA) that are missing and that we recoded to zero are as follows: 15% of students who placed in AR, 13% of students who placed in PA, 9% of students who placed in EA, 7% of students who placed in IA, and 2% of students who placed in transfer-level math (TLM).
Authors
ELIZABETH S. PARK is a postdoctoral scholar at the University of California, Irvine. Her research areas include the high school to college transition, STEM education, community colleges, and program evaluation.
FEDERICK NGO is an assistant professor in the Department of Educational Psychology and Higher Education at the University of Nevada, Las Vegas. His research examines the impacts of higher education policy, with a focus on college access and success for community college students.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
