Does Classroom Composition Matter? College Classrooms as Moderators of Developmental Education Effectiveness

Abstract

This study investigates how the impact of developmental education is moderated by classroom composition. Drawing on a regression discontinuity design, we used data from 3,429 community college students, nested within 223 classrooms, to explore the extent to which classroom and instructor characteristics moderated the effect of developmental English students’ performance in a successive, college-level English course. Our results suggest developmental English students’ college-level course performance benefited from developmental education but was modified by classroom and instructor characteristics. On average, the impact of participating in the developmental English program was greater when developmental students were enrolled in classrooms that contained a higher proportion of developmental students. Achievement for developmental students was also enhanced when developmental students enrolled in classrooms taught by full-time instructors. Findings suggest that after underprepared students completed developmental English, classroom composition in the first, college-level English course had significant influence on developmental students’ performance.

Keywords

classroom composition developmental education student success regression discontinuity design remediation

Across the United States, colleges and universities provide developmental education, or remediation, on a massive scale.¹ It has been estimated that each fall semester, higher education admits approximately 700,000 freshmen who need remediation, with an annual associated cost that exceeds US$2 billion (Parsad, Lewis, & Greene, 2003; Strong American Schools, 2008). The National Center for Education Statistics (Parsad et al., 2003) estimates 42% of entering freshman at public, 2-year and 20% at 4-year institutions are not academically equipped to succeed in one or more college-level subject areas. To meet these students’ academic needs, approximately three quarters of community colleges and universities provide developmental education, designed to “develop basic skills to proficiency levels required for success in regular college-level, college-credit courses” (American Association of Community Colleges [AACC], 2000, “Remedial Education Defined,” para. 1; Parsad et al., 2003).

With such high student demand, and with the allocation of significant human and financial capital, developmental education is often at the center of debates about the suitability of offering below college-level curricula versus opening access to higher education (Ignash, 1997; Kozeracki, 2002; McCabe, 2000; Moss & Yeaton, 2006; Oudenhoven, 2002; Parker, 2007). Questions regarding the appropriateness of remediation in higher education often escalate when economic resources become scarce. Under increased scrutiny, remedial programs are regularly targeted for funding reductions or even eliminated. Facing opposition, proponents of developmental education are challenged to provide evidence that such programs add value worthy of their expense.

To understand whether remedial programs are effective, prominent organizations committed to developmental education recommend research begin by examining within program components. For example, the Achieving the Dream (AtD) initiative endorses the use of institutional and national data to identify developmental education interventions that result in developmental student success in college-level courses. AtD’s goals are consistent with the AACC definition of effective developmental education. AACC’s (2000) position states, “effective remedial education programs provide educational experiences that begin at the student’s level of ability and development, build the academic and personal skills necessary to succeed in subsequent courses” (“Remedial Education Programs,” para. 2). However, as the AACC position maintains, to assess the impact of remediation on student readiness, researchers use measures that occur after developmental education has been completed (Boylan & Saxon, 2005).

Developmental education studies commonly use intermediate or long-term indicators of program efficacy. Martorell and McFarlin (2011) utilized remedial education data from both 2- and 4-year institutions in Texas and found developmental education had a small negative effect on the number of credits attempted and 1-year persistence. This study also found upward transfer to a 4-year institution, earning a degree, and later earnings were not influenced by developmental program completion. In Florida, Calcagno and Long (2008) analyzed statewide community college data and found developmental education had no significant effect on degree completion, transfer rates or earning college-level credits. However, they did find that developmental education increased the likelihood of year-to-year persistence and the total number of credits completed.

Short-term student success measures have also been used to study remediation. Using data from 57 AtD colleges, Bailey, Jeong, and Cho (2010) found that even after successful completion of developmental education, between one fifth and one quarter of developmental students who enrolled in the first, subsequent college-level mathematics or reading courses did not pass. Similarly, Bahr’s (2012) study of 105 of California’s community colleges found that for developmental students who completed their assigned remedial coursework, just 58% to 60% of math and 63% to 68% of writing students passed the subject-related college-level course on the first attempt. Calcagno and Long (2008) also noted remedial students were slightly less likely to pass college English when compared with students who did not need remediation, and no difference was found in the odds of passing math.

Based on the mixed results of these large-scale studies, effectiveness of developmental education is regularly questioned. However, it is important to note that these studies are not without limitations. State- or national-level evaluations will obscure variation between institutional-level effectiveness measures. More specifically, among each sample of colleges and universities, effect estimates will certainly be dispersed, with beneficial program effects hidden within the overall estimate. In addition, long-term outcomes, such as degree completion and post-graduation income, provide a general understanding of program effects but they do little to provide evidence about immediate ability gains for the specific subject matter that remediation was recommended. Consequently, remedial students’ grades in content-related, entry-level, college-level course provide an immediate and important metric of developmental education effectiveness (Scott-Clayton & Rodriguez, 2012).

Developmental education students’ transition into college-level coursework is a critical juncture for facilitating developmental students’ long-term goals (AtD, 2012; Boylan & Bonham, 2011). Once remedial prerequisites are complete, developmental students are shifted out of classrooms occupied with classmates who possess similar academic ability and are permitted to enroll into college-level classrooms alongside students who were deemed academically prepared for college-level courses. While developmental education completion fosters student persistence, it does not guarantee success in subsequent, content-related, entry-level college coursework (Bahr, 2012; Bailey et al., 2010; Calcagno & Long, 2008). This is important, as it is possible that developmental education students’ postprogram success is influenced by the college-level classroom factors. In addition, just as statewide developmental education effect estimates shroud institutional differences, institutional-level program effect estimates will also conceal important within-institution variation that occurs by classrooms.

Developmental Student Transition Into College-Level Classrooms

Once underprepared students complete their remedial prerequisites, enrollment and successful performance in the successive college-level course becomes an important hurdle to overcome. Course failure at this juncture can impede academic progress and decrease the likelihood of student persistence (Bailey, Jeong, & Cho, 2010). Yet, whether college-level classroom attributes influence developmental students’ postremediation academic success has received little attention.

Meaningful classroom interactions are influential components of the learning environment that facilitate learning and intellectual development (Astin, 1984; Pascarella & Terenzini, 1991, 2005). Both student and instructor characteristics influence delivery of course content and impact the nature of class interactions. Thus, inside classrooms, developmental students are subjected to a dynamic set of social conditions that may hamper or facilitate academic success (Umbach & Wawrzynski, 2005).

Classroom environment can be altered by students’ level of academic readiness. Academically advanced students are more participatory and are more likely to control class discussions to meet their intellectual needs, potentially abandoning the needs of developmental students (Fritschner, 2000). Williams and Clark (2004) found that students’ uneasiness with self-perceived academic ability, as well as the nature of teachers’ feedback, strongly correlates with course performance and participation. These researchers found underprepared students experience anxiety about how peers and instructors observe their class contributions. As a result, lower academic ability may exacerbate developmental student self-doubt and insecurities, based primarily on fears of being viewed as unintelligent.

Weighing the risk of embarrassment, developmental students evade class participation or avoid asking for clarification from faculty (Cox, 2011; Weaver & Qi, 2005). Avoidance of this sort is significant because limited student–faculty contact has been linked to lower educational gains and less satisfaction with academic progress (Kuh & Hu, 2001). Furthermore, developmental students’ lower academic ability increases risk for dropout, which may be thwarted by supportive faculty contact (Bailey, 2009; Smart, 2010). Thus, developmental students’ persistence can benefit from classroom engagement and integration into the college community (Astin, 1984, 1993; Kuh, Cruce, Shoup, Kinzie, & Gonyea, 2008; Tinto, 1997).

Course content can also be swayed by the level of scholastic aptitude within the classroom. In a study of more than 250 community college classrooms, Grubb (1999) found that instructors altered their course material when confronted with students who could not complete the assigned work. He observed that when faced with a sizable number of underprepared students, instructors regularly lowered standards, essentially converting college-level courses into remedial courses. Therefore, concentrated underpreparedness may produce qualitatively distinct classrooms, where instructors accommodate to meet the average level of class ability. Despite these findings, remedial research has overlooked how the classroom ratio of developmental to nondevelopmental students impacts developmental and nondevelopmental student achievement.

Other student attributes also influence the classroom learning environment. When mixed with a classroom of younger students, older students shoulder the responsibility of facilitating discussions (Howard & Baird, 2000; Howard, James, & Taylor, 2002). Older students are also perceived by younger students to be more competent. Fritschner (2000) observed that older undergraduate students were viewed by classmates as more motivated and responsible, often creating an age-based social divide within the classroom. However, scholars have uncovered conflicting evidence regarding gender differences in the classroom. While some researchers found classroom environment dissuades female participation (Hall & Sandler, 1982; Morris & Daniel, 2008), other investigators have not confirmed gender differences regarding classroom involvement (Howard et al., 2002; Weaver & Qi, 2005). Consequently, peer relationships and student ability possess potentially powerful factors that influence student outcomes and create status hierarchies that exasperate developmental education students’ vulnerabilities.

Beyond the influence of classmates, instructors also shape developmental students college-level classroom experience and performance (Umbach & Wawrzynski, 2005). Analyses of student–faculty exchanges have confirmed that increased faculty contact enhances students’ self-concept, perceived ability, and achievement (Deil-Amen, 2011; Kim & Sax, 2009; Komarraju, Musulkin, & Bhattacharya, 2010). Nevertheless, research on community college faculty and classroom processes remains unclear (Kim & Sax, 2011; Twombly & Townsend, 2008). In addition, access to faculty is likely contingent upon pragmatic issues, such as employment status. Typically, part-time instructors will spend less time on campus, making them less accessible for student consultation (Outcalt, 2000, 2002). Instructors’ years of teaching experience has been positively correlated with higher levels of achievement (Carrell & West, 2010). Therefore, limited faculty–student contact, or less faculty teaching experience, related to employment status can negatively impact developmental education students’ academic achievement.

The purpose of this study was to gain a more comprehensive description of the effect of English developmental education on students’ postremediation academic success. Our investigation focused on two primary research goals. First, we assessed the overall impact of developmental English on underprepared students’ first, subsequent college-level English course grade. Second, we examined the extent to which student and instructor characteristics moderated the effect of developmental English education.

Method

Study Design

Most colleges and universities in the United States use cut-scores on standardized placement tests to identify students who are in need of additional instruction before entering college-level coursework (Parsad et al., 2003). In our study, we build directly on the use of such standardized placement tests and cut-scores by using the regression discontinuity design to evaluate the effect of remediation on students’ college-level English grades (Calcagno & Long, 2008; Martorell & McFarlin, 2011; Moss & Yeaton, 2006; Moss, Yeaton, & Lloyd, in press; Thistlethwaite & Campbell, 1960).

Regression discontinuity designs require that study participants are assigned to control or treatment conditions based solely on falling above or below a threshold on a pretreatment assignment variable (Shadish, Cook, & Campbell, 2002; Trochim, 1984). This assignment mechanism creates a systematic difference between the treatment and control groups, with the source of the difference being precisely known. As a result, researchers can statistically adjust for the identified difference between groups, induced by the cut-score assignment, by estimating the relationship between the cut-score variable and outcome for the treatment and control groups (e.g., through regression analysis). By comparing the estimated regression line for participants who scored at or above the cut-score (control group in our study) with participants who scored just below the cut-score (treatment group in our study), researchers are able to assess the presence of a discontinuous relationship between the regression lines, or treatment effect. The fundamental principle underlying the regression discontinuity design is that participants who scored near both sides of the cut-score are similar, except that one group received treatment.

Sample

Our sample was drawn from a large, suburban community college in the Midwest and included all course sections (classrooms) of the first, college-level English course (n = 223) offered during two consecutive semesters. Overall, these sections consisted of 3,429 students and were taught by 82 different instructors. Approximately one third of students in this sample were non-White (32%), 45% were eligible for financial aid, and 52% were full-time students (Table 1). The average class size was 16 students. Finally, 40% of the instructors were employed full-time, with an average of 12 years teaching experience, and 5% were self-identified non-White (Table 1).

Table 1.

Student, Instructor, and Classroom Composition Characteristics.

Variable	M	SD
Student characteristics (n = 3,429)
Grade point	2.72	1.27
Received financial aid	0.45	0.50
Treatment group	0.21	0.41
Student age	23.68	7.43
Placement score	11.31	27.72
Full-time student	0.52	0.50
Non-White student	0.32	0.47
Female student	0.52	0.50
Classroom characteristics (n = 223)
Average classroom overall GPA	2.70	0.55
Proportion of class who received financial aid	0.45	0.16
Proportion of class DE	0.20	0.14
Average student age	23.60	2.63
Average placement score	11.76	9.20
Proportion of class full-time	0.52	0.19
Proportion of class non-White	0.32	0.21
Proportion of class female	0.52	0.15
Class size	16.08	3.16
Female student	0.52	0.50
Instructor characteristics (n = 82)
Faculty age	51.46	13.51
Full-time faculty	0.40	0.49
Years teaching	12.70	11.06
Non-White faculty	0.05	0.21
Female faculty	0.55	0.50

Note. GPA = grade point average; DE = developmental English.

Procedure

Upon admission to the college, students were required to take the COMPASS (ACT, 2003) English placement test to determine need for remediation. Based on the ACT’s recommendation, students with a combined reading and writing score less than 150, on a scale ranging from 2 to 196, were classified as developmental English students. These students were required to complete a developmental English course before they were allowed to enroll in the first, college-level English course. In contrast, students who scored at or above 150 were classified as nondevelopmental English students, and were not required to take a developmental English course. As a result the placement policy, students with scores below the placement test cut-score formed our developmental treatment group, whereas students at or above the cut-score served as a nondevelopmental control group.

The developmental program’s primary purpose was to prepare developmental students for college-level English studies. The program included instruction focused on increasing basic English skills (e.g., grammar, spelling, and punctuation). Concurrently, program content promoted student awareness and self-critique regarding the process of reading and writing as methods of improvement. Once the program was successfully completed, developmental students were deemed prepared for college-level English and could enroll in English courses along with nondevelopmental students. The first, college-level English course after developmental English was introduction to composition. This English composition course required students read and analyze essays, as well as practice expository writing. Collegewide, the English faculty agreed upon two main course outcomes, students’ increased ability to effectively communicate and enhanced critical thinking.

Our outcome variable was ordinal grade point for both developmental and nondevelopmental students in the first, college-level English composition course that followed developmental English. Course grades included 11 possible values ranging from a low of 0.0 (F) to 4.0 (A), including all half grades. Developmental students’ grades in the first, college-level content-related course are considered an important marker of program effectiveness (Boylan & Bonham, 2011; Scott-Clayton & Rodriguez, 2012). Colleges and universities regularly use course grades as preconditions for course advancement, gaining entry into academic programs, and to assess student learning.

Analytic Approach

We assessed our research questions in two phases. In the first phase, we investigated the average impact of the developmental program on college-level English performance by comparing grade differences among students near the cut-score. To do so, we drew on both nonparametric and multilevel analyses and describe students’ course grades as a function of placement score. As students were nested within classrooms, in the second phase, we drew on cross-level interactions within a multilevel framework to examine the extent to which the proportion of developmental English students in a classroom moderated the effect of developmental English.

To address our first research question, regarding the overall impact of developmental English, we first compared grade differences for treatment and control students around the cut-point using a nonparametric smoother (Nadaraya–Watson kernel estimator with a normal kernel density; Nadaraya, 1964; Watson, 1964). Analytically, this approach allowed us to examine the relationship between students’ grades and placement scores using locally weighted grade averages without assuming a functional form (e.g., linear). Recent literature on regression discontinuity designs has advocated for the use of nonparametric methods because differences between treatment groups may be a function of nonlinear relationships rather than a discontinuity or treatment effect (Bloom, 2012; Schochet et al., 2010; Shadish et al., 2002).

To provide further evidence of the effect of developmental English and describe the extent the effect varied across classrooms and instructors, we also estimated the overall effect of developmental English using an ordinal multilevel model (Raudenbush & Bryk, 2002). Because students were nested within classrooms or sections, which were in turn nested within instructors, we modeled students’ grades as a function of three levels. At the first level, we considered how individual student characteristics related to changes in student performance using

log (\frac{p (Y_{i j k} \leq m)}{p (Y_{i j k} > m)}) = η = β_{0 j k} + β_{1 j k} S_{i j k} + β_{2 j k} S_{i j k}^{2} + δ_{j k} T_{i j k} + β_{3 j k} T_{i j k} S_{i j k} + \sum_{m = 2}^{M - 1} D_{m i j k} θ_{m} .

Here, Y_ijk was the college-level English grade for student i in section j taught by instructor k, S was the adjusted placement score (centered by subtracting 150 from each score) with corresponding coefficient β₁, s² was the quadratic of the placement score with coefficient β₂, T was the treatment group indicator with corresponding effect δ, and TS was the treatment by placement score interaction with β₃ as the corresponding coefficient. Furthermore, m represented grade categories with corresponding category indicators, D_mijk and thresholds, θ_m.

At Level 2, we considered how students’ performance varied across classrooms and how the treatment effect from developmental English participation varied across classrooms using

\begin{array}{l} β_{0 j k} = γ_{00 k} + u_{0 j k}, \\ δ_{j k} = Δ_{0 k} + u_{δ j k} . \end{array}

Here, we first allowed the average student grade point, β_0jk to vary across college-level English classrooms, with γ_00k indicating the mean classroom grade for instructor k, and u_0jk summarizing the classroom-specific deviations from this mean, with a normal distribution centered at zero and variance of $σ_{β_{0}}^{2}$ . We also allowed the average treatment effect, δ_jk, to vary across classrooms with Δ_0k representing the average treatment effect for classrooms served by instructor k, and with u_δjk describing the classroom-specific treatment effect deviations with normal distribution centered at zero and variance of $σ_{δ}^{2}$ .

Similarly, at Level 3, we considered how students’ performance varied across instructors and how the treatment effect from developmental English participation varied across instructors using

\begin{array}{l} γ_{00 k} = π_{000} + r_{00 k}, \\ Δ_{0 k} = ξ_{00} + r_{δ k} . \end{array}

Here, we allowed the average student grade, γ_00k, to vary across instructors, with π₀₀₀ indicating the overall, adjusted average grade and r_0jk summarizing the instructor-specific deviations from this mean, with a normal distribution centered at zero and variance of $σ_{γ_{0}}^{2}$ . We allowed the average treatment effect to vary across instructors with $ζ_{00}$ representing the overall average treatment effect and r_δk describing the instructor-specific treatment effect deviations, with normal distribution centered at zero and variance of $σ_{Δ}^{2}$ .

To investigate our second research goal, concerning the moderating role of the proportion of developmental education students in a classroom, we extended the aforementioned multilevel model by introducing covariates at each level. At Level 1 (student level), we expanded Equation 1 to become

\begin{array}{l} log (\frac{P (Y_{i j k} \leq m)}{P (Y_{i j k} > m)}) = η = β_{0 j k} + β_{1 j k} S_{i j k} + β_{2 j k} S_{i j k}^{2} + δ_{j k} T_{i j k} + β_{3 j k} T_{i j k} S_{i j k} \\ + \sum_{q = 4}^{Q} β_{q j k} X_{i j k} + \sum_{m = 2}^{M - 1} D_{m i j k} θ_{m} \end{array} .

Here we retain the same notation, but now included X_ijk as student level covariates with coefficients β_qjk. At Level 2 (classroom level), we expanded equation such that

\begin{array}{l} β_{0 j k} = γ_{00 k} + \sum_{p = 1}^{P} γ_{p k} W_{j k} + u_{0 j k}, \\ δ_{j k} = Δ_{0 k} + λ_{k} P D E_{j k} + u_{δ j k} . \end{array}

We continue with the same notation but now introduce PDE_jk as the proportion of developmental education students in class j taught by instructor k with λ_k as its coefficient (the magnitude of the moderating effect). Finally, Level 3 (instructor level) becomes

\begin{array}{l} γ_{00 k} = π_{000} + \sum_{n = 1}^{N} π_{n} Z_{k} + r_{00 k}, \\ Δ_{0 k} = ζ_{00} + ξ F_{k} + r_{δ k} . \end{array}

Similar to previous levels, we introduce covariates for instructor k as Z_k with corresponding coefficients π_n, while F is an indicator for full-time status with coefficient $ξ$ .

Results

We describe the results in two sections. First, we partition the variance in course grades across students, classrooms, and instructors and describe the average impact of the developmental education program using parametric and nonparametric estimators. Second, we investigate the extent to which the classroom proportion of developmental students and full-time employment status of an instructor moderate the impact of developmental education on college-level English performance.

Impact of English Developmental Education

To understand the extent to which student course grades varied across classrooms and instructors, we partitioned the observed variance in course grade across students, classrooms, and instructors. We found that students’ grades varied significantly across classrooms and instructors, suggesting significant clustering both among students within the same classroom and among classrooms taught by the same instructor (see Table 2). Namely, our analyses indicated that grade differences among students partially stemmed from differences among instructors and classrooms.

Table 2.

Variance Components for Multilevel Model.

Random effect		Unconditional	Conditional
Class	Intercept ( $σ_{β_{0}}^{2}$ )	0.41	0.02
	Treatment ( $σ_{δ}^{2}$ )	1.44	0.08
Instructor	Intercept ( $σ_{γ_{0}}^{2}$ )	0.43	0.01
	Treatment ( $σ_{Δ}^{2}$ )	0.05	0.03

Next, we evaluated the degree to which developmental English improved underprepared students’ grades in college-level English. We first investigated grade difference between students in the area near the cut-score by plotting average course grades as a function of placement score using our nonparametric estimator (Bloom, 2012; Imbens & Lemieux, 2008). Evident from Figure 1, there was a large discontinuity, or jump, at the cut-score for students who had received developmental English (left side of the cut-score) versus students who did not require remediation (right side of cut-score). On the original grade point scale, the observed discontinuity or difference between the regression lines for the treatment and control groups was approximately 0.25.

Figure 1.

College-level English performance, as a function of placement score, using nonparametric estimators.

We found similar results using the aforementioned ordinal multilevel model. Our results indicated that participating in the developmental program, on average, increased the log odds of receiving a better grade by a factor of 0.48 (Table 3, Model 1). In other words, students who participated in the developmental program were 1.62 times more likely to receive a higher grade than students who did not participate in the program.² Furthermore, our analyses indicated that although there was an overall positive effect of developmental education on student performance, this effect varied considerably across classrooms and instructors (see variance components for treatment effect at each level for treatment in Table 2). In this way, our analyses suggested that the benefit of developmental education participation was moderated by classroom and instructor characteristics.

Table 3.

Results of Multilevel Ordered Regression Models for Student Performance in College-Level English.

	Coefficient (SE)
	Model (1)	Model (2)
Student characteristics
Grade thresholds
A− to A	1.73* (0.05)	3.70* (0.23)
B+ to A−	1.01* (0.05)	2.98* (0.22)
B to B+	0.60* (0.05)	2.46* (0.22)
B− to B	0.01 (0.05)	1.81* (0.22)
C+ to B−	−0.32* (0.05)	1.45* (0.22)
C to C+	−0.61* (0.05)	1.15* (0.22)
C− to C	−0.99* (0.05)	0.74* (0.22)
D+ to C−	−1.22* (0.05)	0.50* (0.22)
D to D+	−1.29* (0.06)	0.42 (0.22)
F to D	−1.60* (0.06)	0.11 (0.22)
Placement score	0.02* (0.00)	0.01* (0.00)
Placement score (quadratic)	0.00 (0.00)	0.00 (0.00)
Treatment effect (discontinuity)	0.48* (0.07)	0.01 (0.11)
Non-White student		−0.54* (0.04)
Received financial aid		−0.00 (0.01)
Full-time student		−0.21* (0.03)
Female student		0.34* (0.03)
Student age		0.04* (0.01)
DE student by placement score	−0.01* (0.00)	−0.01* (0.00)
Classroom peer characteristics
Proportion of class DE		−0.77* (0.27)
SD of placement score		−0.01* (0.00)
Proportion of class non-White		−0.28* (0.12)
Average placement score		0.00 (0.01)
Class size		0.04* (0.01)
Proportion of class who received financial aid		0.87* (0.14)
Proportion of class full-time		0.26* (0.11)
DE student by proportion of class DE		1.02* (0.32)
Instructor characteristics
Years teaching		0.03* (0.00)
Full-time faculty		−0.04 (0.05)
Non-White faculty		0.14 (0.10)
DE student by full-time faculty		0.38* (0.10)

Note. DE = developmental English.

p < .05.

Assessment of Research Design Validity Threats

The validity of our regression discontinuity estimates depend on the extent to which our regression models correctly describe the relationship between course performance and the placement test score. To assess the extent to which our results were sensitive to misspecification, we first evaluated our nonparametric estimates using several different kernel densities and multiple bandwidths. Similarly, we re-specifiedthe regression model using linear, quadratic, and cubic terms and their respective interactions with the treatment variable (Shadish et al., 2002). We found evidence that both the magnitude of the effect and its uncertainty were insensitive to choices of densities, bandwidths, or functional form specification. Together, the parametric and nonparametric results provide corroborating evidence of developmental English effectiveness.

The validity of regression discontinuity design estimate also depends on the validity of the treatment assignment mechanism. For example, manipulation of the placement test could introduce bias into the treatment effect estimate. In our study, scores on the placement test were computer generated, with scores being directly uploaded into the college’s student information system without any handling by the testing center staff. We also explored the possibility of administrative assignment crossovers by analyzing the presence of asymmetric densities in test scores between the developmental education and nondevelopmental education students at the cut-score. Using the McCrary (2008) test, we found that densities of developmental education and nondevelopmental education scores to be comparable near the placement threshold (t = .004, p > .05). Consequently, both the administrative procedures and the density analysis indicate test manipulation was unlikely.

Finally, we investigated the extent to which our results were sensitive to attrition. In particular, if a large number low ability developmental education students enrolled in the college-level English dropped out, it is possible that our estimate of the effectiveness of English developmental education is upwardly biased. To investigate this possibility, we compared placement scores of the 428 noncompleters in our original sample with that of course completers. We found no significant difference in placement score means between these two groups of students, which suggested a low likelihood that attrition due to academic ability compromised our results.

Classroom Moderators

Having established underprepared English students significantly benefited from developmental English participation, and that this effect varied considerably across classrooms and instructors, we examined the extent to which contextual features (i.e., student and instructor classroom characteristics) moderated the impact of participating in developmental English. Put differently, we tried to explain why developmental English had a greater effect in some classrooms, and with some instructors, than others. To do so, we developed a multilevel model that analyzed student performance differences among classrooms by classroom and instructor characteristics.

From this multilevel model (Table 3, Model 2), we found positive and significant relationships at the student level between course performance and placement score, as well as between gender and age. In contrast, student non-White status and full-time enrollment demonstrated significant negative associations with achievement.

At the classroom level, the proportion of developmental students in the classroom was related to lower classroom performance. Greater classroom placement score heterogeneity was associated with lower classroom course performance. While positive outcome relationships were evident for the proportion of class who received financial aid or were full-time, a negative relationship was observed for the proportion of non-White students in the classroom. Unexpectedly, larger class size was also associated with better performance. Together these covariates explained approximately 95% of the variation in student performance attributable to differences among classrooms (see Table 2, for unconditional and conditional variance components).

We had less success identifying significant predictors of differences among instructors in terms of students’ performance. Although collectively, the instructor characteristics considered explained more than 97% of grade variation at the instructor level, only teaching experience had a significant positive relationship with performance.

Having framed an overall model of achievement, we then examined the degree to which two variables moderated the effect of developmental English. In evaluating the moderating role of the proportion of developmental education students in classroom, we found that students who participated in developmental English had better subsequent course performance when enrolled in classrooms with a higher proportion of developmental students. More specifically, our analyses indicated that for every 10% increase in the proportion of developmental students in a classroom, the log odds of developmental students receiving a better grade were increased by a factor of 0.102.³ In total, the difference among classrooms, in terms of the proportion of developmental students in a classroom, accounted for more than 94% of the variation in treatment effectiveness at the classroom level (comparing the unconditional and conditional variance components for $σ_{π_{1}}^{2}$ see Table 2). That is, virtually all the variability in treatment effectiveness among classrooms was explained by the proportion of developmental students.

We found a similar moderating effect for instructor full-time employment status. Developmental students were more successful when selecting course sections taught by full-time instructors. Overall, developmental English students were 1.46 (exp[0.38]) times more likely to receive a higher grade if they selected a classroom taught by a full-time instructor, compared with students enrolled in a classroom led by a part-time instructor (Table 3, Model 2). The differences among instructors in terms of the full- and part-time employment status accounted for 40% of the variation in treatment effect at the instructor level (comparing the unconditional and conditional variance components for $σ_{β_{1}}^{2}$ ; see Table 2).

Discussion

Developmental education research has largely overlooked how the postremediation classroom environment moderates the effect of remediation. To our knowledge, this is the first study to explore whether, if after remediation, developmental English students’ achievement and developmental education’s effect was modified by classroom contextual factors. Employing regression discontinuity design, we observed underprepared students’ performance in college-level English courses benefited from developmental program participation and that the effect significantly varied by classroom and instructor. In contrasting alternative models, individual level results from both models were largely parallel with the exception of the treatment effect. In particular, analyses using the simpler model (i.e., Table 3, Model 1), which ignored how the treatment effect systematically interacted with classroom and instructor characteristics, tended to draw an incomplete picture of developmental English effect. Namely, the simpler model assumed treatment effects were uniform across classrooms and instructors that possessed different characteristics.

After successful completion of developmental English, students’ academic progress was tempered by proportion of development students enrolled in the first, college-level English course. On average, student performance declined when the proportion of developmental students in the classroom increased. However, we found that developmental English students enrolled in classrooms that contained a large proportion of developmental students demonstrated significantly higher average course performance than developmental students who enrolled in classrooms with few developmental students. In other words, developmental students’ performance in college-level English prospered when grouped in classes that contained many other developmental students.

Several classroom factors could also explain why the proportion of developmental students in classrooms alters developmental student performance. One possible explanation is that when developmental students enrolled in classrooms dense with other developmental students, participation was less threatening, thereby having a positive influence on achievement (Cox, 2011; Weaver & Qi, 2005). Course instruction may also be influenced by higher proportions of developmental students. That is, when classrooms contained more uniform ability, instructors were able to adopt teaching strategies that more efficiently addressed the average classroom academic need (Grubb, 1999). It could also suggest that the pedagogical knowledge used for effective instruction of developmental students might be different from that formed for nondevelopmental students. Therefore, it may be difficult for instructors to maintain both types of specialized knowledge.

Although logistically challenging, policy makers might consider strategies that assemble a balanced ratio of developmental to nondevelopmental students within college-level English classrooms or attempt to create groupings of developmental students. For example, advocates of learning communities argue that facilitating connectedness between academically comparable students will foster developmental student success or reduce attrition (Brock & LeBlanc, 2005; Tinto, 1997). With this approach, developmental students are paired with peers who possess similar academic aptitude, with the expectation that classroom engagement and social cohesion will increase, while concurrently aligning course content with skill level (Zachry & Schneider, 2010).

Continued academic support or specialized instruction might reduce residual academic weaknesses that endure after developmental education. Providing postremediation support in the form of additional academic counseling, tutoring, and monitoring could be used to target developmental students’ needs and has been found to promote higher academic performance and facilitate student persistence (Waycaster, 2001). Complicating these policies are concerns that tracking students into developmental curriculum further exacerbates educational stratification that place developmental students at additional risk for dropout or poor grades (Kozeracki, 2005). Therefore, there remains a delicate balance between developmental students’ long-term academic needs with policies that generate unnecessary barriers.

Developmental English students had better college-level English performance when they enrolled in courses taught by full-time instructors. This finding is similar to recent research that found both teaching experience and full-time work status was associated with higher student performance (Carrell & West, 2010; Outcalt, 2000, 2002). Full-time, versus part-time, employment may provide instructors with more opportunity to gain valuable experience and competencies that address institutional remedial need and, because of time on campus, is also likely to result in instructors being more accessible when developmental students need additional instruction (Umbach & Wawrzynski, 2005).

Study Limitations and Future Directions

Our results are limited in a number of ways. Although our sample size was large, we use data from one institution, which resides within a suburban environment, and contains student and classroom characteristics that may not generalize to other settings or student populations. We also note that our findings focus on developmental English and are potentially unrelated to other developmental education disciplines. In spite of concluding that classrooms are associated with developmental students’ academic performance, we have no metrics of which student–student or student–faculty interactions were important in maintaining the positive impact of developmental education. Future research might consider collecting qualitative data to understand effective classroom instructional practices within the classroom “black box” (Grubb, 2001, p. 5), as well as explore other developmental education subject matter.

Conclusion

Developmental education policies and practices generally embrace the belief that once underprepared students complete remedial coursework, no further intervention or academic support is necessary. Our findings suggest that placing emphasis on underprepared students’ developmental course completion is an important but incomplete strategy to facilitate academic success in college-level English courses. Developmental English students’ achievement was conditional upon postprogram classroom variables, leading us to contend that the integration of developmental English students into college-level English courses should be monitored. If we are to understand how developmental education works, and with whom, analyses that offer more comprehensive descriptions of variables that impact developmental education outcomes will better leverage existing information to advance developmental education research and practice. This process is challenging and requires both institutional and instructor commitment rooted in rigorous research design strategies.

Footnotes

Acknowledgements

We would like to thank Ozan Jaquette and Bill Yeaton for providing feedback on an early draft of this manuscript.

Author Note

Brian Moss and Ben Kelcey share first authorship.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Notes

Author Biographies

Brian G. Moss is an instructor in the Department of Sociology and School of Social Work at Wayne State University. He conducts research focused on methods used to enhance causal inference, specifically the use of experimental and quasi-experimental research designs to study at-risk education and health populations.

Ben Kelcey is an assistant professor in the College of Education Criminal Justice, & Human Services at the University of Cincinnati. His research interests include the development of measurement and quantitative research methods to understand effective teaching and teachers.

Nancy Showers is the Executive Director of Institutional Research, Quality and Planning at Oakland Community College. Her current responsibilities include formulation and implementation of policies governing institutional planning, performance excellence, and college-wide research activities.

References

Achieving the Dream. (2012, January 12). One goal—Success for ALL community college students. Retrieved from http://achievingthedream.org/institutional_change/one_goal

ACT. (2003). COMPASS/ESL reference manual. Iowa City, IA: Author.

American Association of Community Colleges. (2000). AACC position statement on remedial education. Retrieved from www.aacc.nche.edu/About/Positions/Pages/ps08012000.aspx

Astin

A. W.

(1984). Student involvement: A developmental theory for higher education. Journal of College Student Personnel, 25, 297-308.

Astin

A. W.

(1993). What matters in college: Four critical years revisited. San Francisco, CA: Jossey-Bass.

Bahr

P. R.

(2012). Deconstructing remediation in community colleges: Exploring associations between course-taking, course outcomes, and attrition from the remedial math and remedial writing sequences. Research in Higher Education, 53, 661-693. doi:10.1007/s11162-011-9243-2

Bailey

(2009). Challenge and opportunity: Rethinking the role and function of developmental education in community college. New Directions for Community Colleges, 145, 11-30. doi:10.1002/cc.352

Bailey

Jeong

Cho

(2010). Referral, enrollment, and completion in developmental education sequences in community colleges. Economics of Education Review, 29, 255-270. doi:10.1016/j.econedurev.2009.09.002

Bloom

H. S.

(2012). Modern regression discontinuity analysis. Journal of Research on Educational Effectiveness, 5, 43-82. doi:10.1080/19345747.2011.578707

10.

Boylan

H. R.

Bonham

(2011). Criteria for developmental education program evaluation. Retrieved from http://ncde.appstate.edu/sites/ncde.appstate.edu/files/evaluation.pdf

11.

Boylan

H. R.

Saxon

D. P.

(2005). What works in remediation: Lessons from 30 years of research. Boone, NC: National Center for Developmental Education.

12.

Brock

LeBlanc

(2005). Promoting student success in community college and beyond: The opening doors demonstration. New York, NY: Manpower Demonstration Research Corporation.

13.

Calcagno

J. C.

Long

B. T.

(2008). The impact of postsecondary remediation using a regression discontinuity approach: Addressing endogenous sorting and noncompliance (No. 14194). Cambridge, MA: National Bureau of Economic Research. doi:10.3386/w14194

14.

Carrell

S. E.

West

J. E.

(2010). Does professor quality matter? Evidence from random assignment of students to professors. Journal of Political Economy, 118, 409-432. doi:10.1086/653808

15.

Cox

R. D.

(2011). The college fear factor: How students and professors misunderstand one another. Boston, MA: Harvard University Press.

16.

Deil-Amen

(2011). Socio-academic integrative moments: Rethinking academic and social integration among two-year college students in career-related programs. Journal of Higher Education, 82, 54-91. doi:10.1353/jhe.2011.0006

17.

Fritschner

L. M.

(2000). Inside the undergraduate classroom: Faculty and students differ on the meaning of student participation. Journal of Higher Education, 71, 342-362. doi:10.2307/2649294

18.

Grubb

W. N.

(1999). Honored but invisible: An inside look at teaching in the community colleges. New York, NY: Routledge.

19.

Grubb

W. N.

(2001). From black box to Pandora’s box: Evaluating remedial/developmental education. New York, NY: Columbia University, Teachers College, Community College Research Center.

20.

Hall

Sandler

(1982). The classroom climate: A chilly one for women? Washington, DC: Association of American Colleges.

21.

Howard

J. R.

Baird

(2000). The consolidation of responsibility and students’ definitions of situation in the mixed-age college classroom. Journal of Higher Education, 71, 700-721. doi:10.2307/2649159

22.

Howard

J. R.

James

G. H.

III Taylor

D. R.

(2002). The consolidation of responsibility in the mixed-age college classroom. Teaching Sociology, 30, 214-234. doi:10.2307/3211384

23.

Ignash

J. M.

(1997). Who should provide postsecondary remedial/developmental education? New Directions for Community Colleges, 100, 5-20. doi:10.1002/cc.10001

24.

Imbens

G. W.

Lemieux

(2008). Regression discontinuity designs: A guide to practice. Journal of Econometrics, 142, 615-635. doi:10.1016/j.jeconom.2007.05.001

25.

Kim

Y. K.

Sax

L. J.

(2009). Student-faculty interaction in research universities: Differences by student gender, race, social class, and first-generation status. Research in Higher Education, 50, 437-459. doi:10.1007/s11162-009-9127-x

26.

Kim

Y. K.

Sax

L. J.

(2011). Are the effects of student–faculty interaction dependent on academic major? An examination using multilevel modeling. Research in Higher Education, 52, 589-615. doi:10.1007/s11162-010-9209-9

27.

Komarraju

Musulkin

Bhattacharya

(2010). Role of student-faculty interactions in developing college students’ academic self-concept, motivation, and achievement. Journal of College Student Development, 51, 332-342. doi:10.1353/csd.0.0137

28.

Kozeracki

C. A.

(2002). ERIC review: Issues in developmental education. Community College Review, 29, 83-100. doi:10.1177/009155210202900405

29.

Kozeracki

C. A.

(2005). Responsive developmental education. New Directions for Community Colleges, 129, 1-4. doi:10.1002/cc.180

30.

Kuh

G. D.

Cruce

T. M.

Shoup

Kinzie

Gonyea

R. M.

(2008). Unmasking the effects of student engagement on first-year college grades and persistence. Journal of Higher Education, 79, 540-563. doi:10.1353/jhe.0.0019

31.

Kuh

G. D.

(2001). The effects of student-faculty interaction in the 1990s. Review of Higher Education, 24, 309-332. doi:10.1353/rhe.2001.0005

32.

Martorell

McFarlin

(2011). Help or hindrance? The effects of college remediation on academic and labor market outcomes. Review of Economics and Statistics, 93, 436-454. doi:10.1162/REST_a_00098

33.

McCabe

(2000). No one to waste: A report to public decision-makers and community college leaders. Washington, DC: Community College Press.

34.

McCrary

(2008). Manipulation of the running variable in the regression discontinuity design: A density test. Journal of Econometrics, 142, 698-714. doi:10.1016/j.jeconom.2007.05.005

35.

Morris

L. K.

Daniel

L. G.

(2008). Perceptions of a chilly climate: Differences in traditional and non-traditional majors for women. Research in Higher Education, 49, 256-273. doi:10.1007/s11162-007-9078-z

36.

Moss

B. G.

Yeaton

W. H.

(2006). Shaping policies related to developmental education: An evaluation using the regression-discontinuity design. Educational Evaluation and Policy Analysis, 28, 215-229. doi:10.3102/01623737028003215

37.

Moss

B. G.

Yeaton

W. H.

Lloyd

(in press). Estimating the effectiveness of developmental mathematics by embedding a randomized experiment within a regression discontinuity design. Educational Evaluation and Policy Analysis. doi:10.3102/0162373713504988

38.

Nadaraya

E. A.

(1964). On estimating regression. Theory of Probability and Its Applications, 9, 141-142. doi:10.1137/1109020

39.

Oudenhoven

(2002). Remediation at the community college: Pressing issues, uncertain solutions. New Directions for Community Colleges, 117, 35-44. doi:10.1002/cc.51

40.

Outcalt

(2000). ERIC review: Community college teaching—Toward collegiality and community. Community College Review, 28, 57-70. doi:10.1177/009155210002800204

41.

Outcalt

(2002). A profile of the community college professorate, 1975-2000. New York, NY: Routledge Falmer.

42.

Parker

T. L.

(2007). Ending college remediation: Consequences for access and opportunity (ASHE/Lumina Policy Briefs and Critical Essays, No. 2). Ames: Department of Educational Leadership and Policy Studies, Iowa State University.

43.

Parsad

Lewis

Greene

(2003). Remedial education at degree-granting postsecondary institutions in fall 2000 (NCES 2004–010). Washington, DC: U.S. Department of Education, National Center for Education Statistics.

44.

Pascarella

Terenzini

(1991). How college affects students: Findings and insights from twenty years of research. San Francisco, CA: Jossey-Bass.

45.

Pascarella

Terenzini

(2005). How college affects students: A third decade of research (Vol. 2). San Francisco, CA: Jossey-Bass.

46.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: SAGE.

47.

Schochet

Cook

Deke

Imbens

Lockwood

Porter

Smith

(2010). Standards for regression discontinuity designs. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_rd.pdf

48.

Scott-Clayton

Rodriguez

(2012). Development, discouragement, or diversion? New evidence on the effects of college remediation (No. w18328). Cambridge, MA: National Bureau of Economic Research.

49.

Shadish

Cook

Campbell

(2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.

50.

Smart

J. C.

(2010). Differential patterns of change and stability in student learning outcomes in Holland’s academic environments: The role of environmental consistency. Research in Higher Education, 51, 468-482. doi:10.1007/s11162-010-9163-6

51.

Strong American Schools. (2008). Diploma to nowhere. Washington, DC: Author.

52.

Thistlethwaite

D. L.

Campbell

D. T.

(1960). Regression discontinuity analysis: An alternative to the ex post facto experiment. Journal of Educational Psychology, 51, 309-317. doi:10.1037/h0044319

53.

Tinto

(1997). Classrooms as communities: Exploring the educational character of student persistence. Journal of Higher Education, 68, 599-623. doi:10.2307/2959965

54.

Trochim

W. M. K.

(1984). Research design for program evaluation: The regression-discontinuity approach. Beverly Hills, CA: SAGE.

55.

Twombly

Townsend

B. K.

(2008). Community college faculty: What we know and need to know. Community College Review, 36, 5-24. doi:10.1177/0091552108319538

56.

Umbach

P. D.

Wawrzynski

M. R.

(2005). Faculty do matter: The role of college faculty in student learning and engagement. Research in Higher Education, 46, 153-184. doi:10.1007/s11162-004-1598-1

57.

Watson

G. S.

(1964). Smooth regression analysis. Sankhyā: The Indian Journal of Statistics, Series A, 26, 359-372.

58.

Waycaster

(2001). Factors impacting success in community college developmental mathematics courses and subsequent courses. Community College Journal of Research and Practice, 25, 403-416. doi:10.1080/106689201750192256

59.

Weaver

R. W.

(2005). Classroom organization and participation: College students’ perceptions. Journal of Higher Education, 76, 570-601. doi:10.1353/jhe.2005.0038

60.

Williams

R. L.

Clark

(2004). College students’ ratings of student effort, student ability and teacher input as correlates of student performance on multiple choice exams. Educational Research, 46, 229-239. doi:10.1080/0013188042000277304

61.

Zachry

E. M.

Schneider

(2010). Building foundations for student readiness: A review of rigorous research and promising trends in developmental education. National Center for Postsecondary Research, Teachers College, Columbia University. Retrieved from http://www.postsecondaryresearch.org/conference/PDF/NCPR_Panel%203_ZachrySchneiderPaper.pdf