A Look Inside Online Educational Settings in High School: Promise and Pitfalls for Improving Educational Opportunities and Outcomes

The Promise for and Evidence on Digital Instruction

Digital instructional programs are marketed to state and local educational agencies on the premise that they will expand quality learning opportunities for students and enhance instructional practices for teachers, albeit perhaps with expectations for conditions and capacities in place that may be unreasonable for many large, resource-constrained school districts to attain (Burch & Good, 2014). Arguments for the adoption of online instructional programs include more personalization of course content and tailoring of instruction to student experiences and skill levels; access to more diverse learning resources and wider course offerings (within and outside the classroom); and fostering greater student engagement and motivation for learning and more connected learning opportunities (Cavanaugh et al., 2007; Collins & Halverson, 2009; Darling-Hammond & Bransford, 2007). In addition, new strategies and forms of assessment are being built into online instructional programs, with more rapid feedback loops, structured forms and processes for monitoring student progress, and greater access to assessment information for teachers, parents, and students (Burch et al., 2016; Halverson et al., 2015). Indeed, many digital learning initiatives begin with a plan to deliver instruction through blended learning approaches that combine digital and face-to-face content delivery as a way to marry the flexibility, access, and monitoring and feedback tools of online instruction with the social aspects, individualization, and contextual benefits of face-to-face instruction (Enyedy, 2014; Osguthorpe & Graham, 2003). Existing research emphasizes the importance of continued live interaction between teachers and students as online instruction is adopted, as well as more collaborative rather than independent interactions with online instructional components (Means et al., 2010; VanLehn, 2011; Zhao, Lei, Yan, Tan, & Lai, 2005).

Despite the promising features embedded in online instructional programs, prior research finds mixed (positive and negative) and wide variation in effects on student achievement across schools and student subgroups, as well as limited implementation of the program features that are intended to enhance learning and instructional quality (Ahn & McEachin, 2017; Chingos, Griffiths, & Mulhern, 2017; Cole, Kemple, & Segeritz, 2012; Margolin, Kleidon, Williams, & Schmidt, 2011; Pane, Steiner, Baird, Hamilton, & Pane, 2017). Pane et al. (2017) pointed out that most personalized learning efforts have been implemented in school and classroom settings that continue to employ traditional models of large-group instruction, while the modal approach to personalization involves student self-pacing and some additional choices of media for learning, due in part to the limited capacity of teachers to organize classrooms in ways that support more innovative digital learning (Margolin et al., 2011).

Research focusing specifically on online course-taking shows that a large share of high schools adopt online course instruction primarily for credit recovery and realize relatively low rates of course completion, generally in the range of 30% to 55% (Carr, 2000; Roblyer, 2006; Simpson, 2004; Stevens, Frazelle, Bisht, & Hamilton, 2016; Viano, 2018). The few studies that examine the implementation of online learning at the secondary level emphasize the importance of staff with training to provide instructional support and monitor student progress (Hannum, Irvin, Lei, & Farmer, 2008; Stevens et al., 2016). A study of online course-taking for credit recovery in Montana found that only a handful of schools provided extensive student support, with the role of most teachers limited to addressing classroom management and technology access issues (Stevens et al., 2016). This study also identified student engagement and attendance as critical factors, yet there has been little attention in the literature as to which students enroll in and complete courses online or what influences the extent to which they engage and how well they perform. Another concern with rapid growth rates in online instruction (more than 100% in some states) is the disproportionate share of students with disabilities and of color taking these courses (Corry, Dardick, & Stella, 2016; Smith & Basham, 2014). In Arizona, online school options are seen as one way to increase access and reduce dropout rates for Hispanic or Latino students, despite a lack of conclusive evidence that online schools meet the needs of this particular student population or increase their high school graduation rates (Corry et al., 2016). More recently, allegations have been made of inflated high school graduation rates attributed in part to low standards in and misuse of online credit recovery programs in school districts in Chicago, Nashville, New York, San Diego, and Washington, D.C. (Malkus, 2018).

Heppen et al. (2017) conducted one of the first studies using an experimental design to assess the effectiveness of online credit recovery course-taking in high school compared with a face-to-face option. The study followed students in 17 Chicago Public Schools who failed algebra in their first year of high school and were randomly assigned to retake the course in the summer through either an online course provider or a traditional face-to-face course. Of these students, 90% were Hispanic or African American, 86% were free- or reduced-price lunch eligible, and 12% qualified as students with disabilities. Heppen et al. (2017) described the various ways the courses might have differed when offered online versus face-to-face—such as content and sequencing, staffing intensity, interactions between instructors and students, and grading and feedback—with more variation expected across these dimensions within the face-to-face setting. They reported that the online instruction was delivered mainly in school computer labs, with most communication between students and teachers occurring asynchronously. When comparing student outcomes following the summer courses, they found that students in the online course had significantly lower end-of-course posttest scores and lower credit recovery rates compared with those in the face-to-face course. Contrary to touted advantages of online instructional models for facilitating personalized learning, the authors surmised that the online course may have been less effective in adapting to students’ individual needs, in that these courses lacked flexibility for addressing gaps in students’ initial skills and understanding of algebra. More rigorous evidence, such as this across a wider variety of online course-taking and in more grades and settings, is needed to determine if these results generalize beyond this particular online course and context.

Theoretical Framing

This longitudinal, mixed methods study of online course-taking in high schools is situated within a broader research investigation of digital learning in K–12 public schools. In formulating the logic model shown in Figure 1 to guide our research, we drew on sociotechnical and sociocultural theories. Sociotechnical theory takes into consideration the technical properties of the educational technology and examines how the technology users enact them in practice, positing that individuals and their social settings will shape the types and intensity of their use through recurring interactions (Orlikowski, 2000). Sociocultural theory similarly focuses on understanding student learning and development through their interactions in educational settings, but it places a greater emphasis on social and cultural norms and processes that influence how students use technology and whether and how they draw on other individuals and resources in the classroom to support their learning with it (Nasir & Hand, 2006). Sociocultural theory also stresses that we should not lose sight of how educational settings, particularly in large urban school districts such as the one we study, are interleaved with larger societal forces such as those that perpetuate poverty, discrimination, and inequality. These two theories and the logic model developed through this framing guided the focus of our data collection—including classroom observations of instructor and student use of the technology and their behaviors and interactions that varied around physical settings and resources such as instructional and technical supports (see Figure 1)—as well as our analysis of student online user behaviors, outputs (e.g., time on task) and short-term outcomes (e.g., course completion, credit accumulation, and grades).

OPEN IN VIEWER

Figure 1.

Logic model for assessing equity and effects of online instruction in high schools.

As our in-depth investigation of digital learning progressed and we uncovered patterns in the targeting and implementation of high school online course-taking, we sharpened our theoretical framing to hone in on concerns about equity in access to quality learning opportunities and instructional supports for the historically underserved student populations that are frequently prioritized for online learning in low-resource, urban school districts. In particular, we bring in theory on categorical inequality that describes how education systems create social categories and sort students into groups such as classrooms and academic tracks, often by ability, which in turn influence the educational environments and resources made available to them for learning (Domina, Penner, & Penner, 2017; Massey, 2007). For example, in our partner district, as we discuss below, we observed that high school students are typically assigned to take courses online after failing a course in a traditional classroom or being removed for behavioral problems or special educational needs. These students often go to an online learning lab rather than a regular classroom for instruction, which effectively segregates them according to their academic performance and institutionalizes their exclusion from better performing peers. As Domina et al. (2017) elaborate, this social categorization of students may influence not only influence students’ access to academic content and resources but also their own identity formation and social status in ways that can exacerbate negative stereotypes and reinforce racial and class disparities in educational processes that extend beyond students’ time in school. Relating this theoretical perspective to our logic model in Figure 1, we explore how categorical inequalities emerge in how students are prioritized for online course-taking and in the physical settings where online instruction is accessed and the resources and supports provided to teachers and students to create quality learning opportunities with technology.

Study Samples and Data

This study is set in an urban, Midwestern school district, where approximately one quarter of students in Grades 9 to 12 access course instruction through an online instructional program, both during and outside the school day (up from about 5% of all high school students when use of the program began in 2010–2011). Data from the technology vendor on online course-taking in this district are currently available through the 2016–2017 school year, including detailed information on student use in each online session, course information, and progress toward course completion and performance. We link these data to student record data that include demographic information, absences and suspensions, credits earned and GPA (grade point average), ACT (American College Test) scores, and standardized test scores, as well as to measures of high school characteristics.

In the analyses of quantitative data, we focus on online instruction occurring in 46 high schools over the 2013–2014, 2014–2015, 2015–2016, and 2016–2017 school years. We linked student district records to the vendor’s online instructional program records with match rates ranging from 80.1% to 86.6% as follows: (1) 4,676 online course-takers (81.4%) matched with 1,648,380 session-level vendor records in 2013–2014, (2) 5,175 online course-takers (86.6%) matched with 2,142,340 session-level vendor records in 2014–2015, (3) 4,976 online course-takers (83.3%) matched with 1,599,852 session-level vendor records in 2015–2016, and (4) 5,250 online course-takers (80.1%) matched with 1,510,189 session-level vendor records in 2016–2017. Table 1 presents basic demographic characteristics for our study sample, comparing all high school students in this district with online course-takers with linked district-online vendor records. The simple descriptive statistics in Table 1 suggest that a higher proportion of students taking courses online identify as Black and low-income. Online course-takers in the district are also absent more often from school and have lower average fall math and reading test scores than the overall district high school student population.

OPEN IN VIEWER

Table 1

High School Student Characteristics (Study District)

	2013–2014		2014–2015		2015–2016		2016–2017
	All High School Students	District-Online Vendor Linked Records	All High School Students	District-Online Vendor Linked Records	All High School Students	District-Online Vendor Linked Records	All High School Students	District-Online Vendor Linked Records
Total number of students	20,984	4,676	20,581	5,175	21,922	4,976	22,147	5,250
Asian (%)	6	2	6	2	6	2	6	3
Black (%)	62	68	62	66	60	68	61	67
Hispanic (%)	20	20	20	22	22	20	23	22
White (%)	10	10	12	8	10	8	9	7
Other race (%)	0	0	0	0	0	0	1	1
Female (%)	48	46	48	46	48	46	49	46
English language learner (%)	8	6	8	6	8	4	16	12
Free lunch–eligible (%)	79	82	82	86	74	75	76	77
Student with special needs (%)	22	22	22	22	22	24	23	23
Percentage of days absent	18	22	17	20	20	26	20	29
Average test score: Fall math	222.13	218.95	216.72	216.30	727.58	714.81	712.92	703.09
Average test score: Fall reading	214.99	213.04	209.90	209.49	677.78	656.32	633.12	614.09
Average test score: Spring math	224.36	220.28	219.49	217.13	738.54	716.80	746.65	734.82
Average test score: Spring reading	216.04	213.20	210.73	208.62	669.80	636.41	693.39	671.50

Note. Online course-takers without linked district records had shorter course durations and lower on-time course passing rates. A table with the details of the comparison of these two groups (with and without linked records) is available from the authors.

The strength of qualitative research methods lies in its focus on the creation of meaning, situating the research in natural settings or contexts, and the holistic and rich nature of its data sets (Miles & Huberman, 1994). Qualitative approaches allow us to examine the conditions and processes through which a phenomenon like online credit recovery interacts with students, teachers, and the persistent characteristics of classrooms and schools (e.g., structures, beliefs, perspectives, culture). In our effort to better understand the nuances of digital learning in high schools, this study draws on rich data collected in 158 observations of student online course-taking in 18 high schools across the 2014–2015, 2015–2016, and 2016–2017 school years. While there is considerable variation in the percentage of students taking courses online in the high schools over time, the particular high schools observed included approximately 90% of all the students who took courses online in the district. We used a standardized observation instrument developed to evaluate the nature of digital tools and their implementation in digital and blended instructional settings (Burch et al., 2016). The rubric evaluates the extent to which the instructional session facilitates quality learning opportunities for students using a set of indicators or dimensions for rating the entire learning experience (see the full instrument in the Appendix, available in the online version of the journal) along a 5-point Likert-type (0–4) scale. The instrument also records narrative comments for each dimension and an in-depth narrative vignette. We capture total instructional time, time on task, the extent to which the format facilitates live interaction between instructors and students, and the functionality/operability of the technology, which map to many of the specific inputs, activities, and outputs shown in the logic model in Figure 1. As we do not identify the students observed, we do not directly link the data collected in the observations to the student-level district provider data. All research team members conducting observations went through a series of reliability trainings to ensure consistency in ratings and the nature of and constructs captured via our qualitative fieldnotes.

In addition, we conducted 24 structured interviews with district- and school-level administrators and support staff over 2 years to characterize and understand how malleable factors such as organizational capacity, staffing, training, support decisions, and policy guidance and requirements for implementing the technology influence access to, and the effectiveness of, the online instructional tool in increasing student learning. The interviews addressed teacher background, training and experience with technology, how the online instructional program is used in the classroom and integrated with other instructional practices, support received for using the program, impediments to their access and effective use by students, and needs for additional resources to improve program use and student outcomes (corresponding to inputs and activities in our logic model, particularly resources and supports in technology enactment). Consistent with recent calls for better implementation studies of digital K–12 initiatives (e.g., Pane, 2018), our qualitative approach adds a nuanced understanding of conditions surrounding digital learning in K–12 classrooms, a detailed description of implementation processes, and context and texture to the quantitative analysis of program participation and outcomes.

Study Measures

Treatment and Control Measures

We conceptualize and investigate “treatment” in online instruction in two primary ways to address our first two research questions, assessing (1) who accesses courses online (for at least one course during high school) when targeted or offered the opportunity and (2) how students are using (interacting with) the online instructional program in the educational setting, including these measures of user behavior: session and course duration, number of courses in which a student enrolled, number of sessions per course, activities completed per day, idle-to-active time per session, and the percentage of sessions taking place outside the regular school day. We begin by investigating the extent to which student demographic characteristics (see Table 1 again) and their academic status and experiences (i.e., course failure, credits earned, GPA, absences, suspensions, and test score performance) predict the likelihood that they will take a course online. We also factor in school characteristics using school fixed effects as well as publicly available data that describe the type of high school (alternative, charter, neighborhood, and specialty schools), administrator type (principal, assistant or co-principal, teacher/teacher leader, etc.), school calendar, geographic location, and other attributes. Administrative and support staff conveyed in interviews that decisions about how to organize online course-taking and how intensively it is used are typically made at the school level and vary from year to year (as discussed below and shown in Table 2).

OPEN IN VIEWER

Table 2

Percentage of Students Taking Courses Online by High School and School Year

High Schools (HS)	Percentage of Students Taking Courses Online
Offering Online Courses	2013–2014	2014–2015	2015–2016	2016–2017
HS A	11.72	1.68	5.80	39.06
HS B	38.58	51.56	49.46	68.41
HS C	26.27	19.95	14.97	8.00
HS D	27.47	39.37	25.82	28.27
HS E	52.42	64.90	62.75	64.88
HS F	26.75	28.73	22.89	29.46
HS G	0.34	7.58	16.84	23.08
HS H	12.83	10.00	23.14	17.82
HS I	22.58	58.54	8.82	0.00
HS J	11.06	0.00	1.40	0.85
HS K	4.86	5.03	0.64	3.56
HS L	0.18	0.18	0.51	5.68
HS M	10.81	35.06	30.09	36.14
HS N	29.78	14.48	13.35	46.48
HS O	0.00	28.14	49.79	35.75
HS P	35.42	64.00	51.26	47.22
HS Q	37.62	31.91	29.98	42.47
HS R	30.40	32.28	31.30	22.48
HS S	17.70	34.74	34.06	30.54
HS T	12.51	26.46	20.21	18.72
HS U	93.44	87.58	86.60	85.04
HS V	16.13	18.09	30.96	27.80
HS W	45.42	39.52	19.83	14.00
HS X	53.66	46.64	19.13	21.46

Note. Percentages are reported only for schools with a total of at least 100 students taking online courses over the 4-year study period.

We next draw on the rich data available on student user behaviors and level of engagement with the online instructional program to elicit some descriptive typologies of student use (or “user types”), using k-means cluster analysis. In this way, we differentiate student use based not only on their individual (or historical group member) characteristics (Catterall, 1998) and prior academic performance but also on factors in the implementation of online course-taking that may be more malleable at the classroom level for improving student use of the online instructional program (see again Figure 1).

Mixed Methods Analysis

Quantitative Analysis of Student Outcomes

In our analysis of the potential effectiveness of online course-taking in improving student outcomes, we aim to adjust for the selective differences between students who take courses online and those who do not and for factors that affect types and intensity of use over time in models examining associations between how the online instructional system is used and student outcomes. In particular, we control for student “baseline” demographic characteristics and academic status and experiences (described above), as well as important time-varying high school characteristics. Our primary estimation strategy employs panel data models, where we pool data across the four school years (2013–2014 through 2016–2017) and adjust for stable student, school, grade, and year factors (fixed effects) to identify the average effects of online course-taking in high school. More than 38% of high school students in our sample took at least one course online during high school, and of those taking courses online, close to two thirds did so in only 1 year; another quarter took courses online in 2 years, and the other 12% took courses online in three or more years of high school.

In a typical fixed effects model for estimating average effects of online course-taking on student-level outcomes in a given school year, we would estimate

A jst = α D jt + β 1 X 1 jt + β 2 A jst − 1 + β 3 P st + π s + μ gt + ε jst

(1)

where A_jst is the achievement/intermediate outcome of student j attending school s in year t; D_jt is an indicator if the student accessed instruction online in year t; X_1jt are student characteristics at the start of the school year in which instruction is accessed online (including student demographics, percentage of students absent in prior year, special educational needs, etc.); A_{jst− 1} is the prior year assessment/outcome measure; P is the percentage of students in a given school who access online instruction (as shown in Table 2); π _s is a school fixed effect that captures school attributes that are stable over time; μ _gt are grade by year fixed effects, and ε _jst is the random error term.

In the above model, the coefficient on the online instruction variable indicates, on average, if there was an association (positive or negative) between online course-taking and student intermediate outcomes in these settings, controlling for student characteristics and time-invariant school and/or classroom and grade year effects, as well as time-varying rates of online instruction use in schools. This model would only identify effects of online course-taking if it was reasonable to assume that no other unobserved, time-varying factors influenced online course-taking and student educational outcomes (the conditional independence assumption).

With longitudinal (panel) data that we employ in our estimation (i.e., following students over four school years), we improve on this method of identification by adding student fixed effects to our models as follows, where δj is the student fixed effect:

A jst = α D jt + β 1 X 1 jt + β 2 A jst − 1 + β 3 P st + δ j + π s + μ gt + ε jst

(2)

Identification of the average effect of online course-taking in this model comes from students who take courses online in some but not all years that we observe them in high school (Heinrich & Nisar, 2013), which is the case for many of the students in our sample.

As the assumption that no other unobserved, time-varying factors (at student, school, and grade levels) had influenced online course-taking and student educational outcomes is a relatively strong one, we also estimated inverse probability weighting models with regression adjustment (IPWRA), a double-robust estimator that aims to align the observed characteristics of online course-takers and nonusers at baseline in assessing the relationship of online course-taking and its intensity of use to student outcomes. This doubly robust estimation method uses probability weights from a model that predicts treatment status (i.e., online course-taking or the number of years of online course-taking) to obtain outcome-regression parameters that account for the fact that each student is observed in only one of the potential outcomes. The estimated inverse-probability weights are used to fit weighted regression models of the outcome for each treatment level and to obtain predicted outcomes for each student. Average treatment effects (ATE) are then computed from these estimates of treatment effects.

The multivalued treatment model that is used to estimate the effects of intensity of online course-taking is shown in Equation (3). Again defining D_t,i as a binary variable that equals 1 if student i is in a given treatment state and 0 if not, the model we estimate is as follows:

ATE ^ t = 1 / ( n ∑ i = 1 n [ D t , i ) / ( p ^ t ( X i ) Y i + ( 1 − D t , i ) / ( p ^ t ^ ( X i ) ) μ ^ t ( X i ) ] − 1 ) / ( n ∑ i = 1 n [ 1 − D t , i ) / ( 1 − p ^ t ( X i ) Y i + ( 1 − ( 1 − D t , i ) ) / ( 1 − p ^ t ( X i ) ) μ ^ 0 ( X i ) ] = Δ ^ ( t ) − Δ ^ ( 0 )

(3)

In the above formula, p ^ t ( X i ) is the estimated propensity score for treatment t and μ ^ t ( X i ) estimates μ t ( X i ) = E [ Y ( t ) | X ] for t ∈ { 0 , 1 , … , T } . The ATE is estimated in a three-step procedure, where the true propensity score p_t(X_i) is estimated first, in this case with a multinomial logit model; the true regression model μ _t (X_i) is estimated next, and then they are combined as in Equation (3) to calculate the final result. The primary advantage of IPWRA is that the estimate for the ATE is consistent if either the model for the propensity score or for the potential outcome regression is correctly specified (the doubly robust property). We estimate the IPWRA models in part to provide a robustness check on the fixed effects model results, but we do not claim to have overcome all limitations to the validity of causal inference due to selective differences between online course-takers (and the intensity of online course-taking) and nonusers.

Who Takes Online Courses, and How Intensively?

Table 2 shows that the proportion of high school students in the district taking online courses in a given high school ranged considerably by school, as well as within high schools over time, from 0 to a high of more than 93%. Interviews with district staff and teachers pointed to numerous factors that might influence the proportion of students participating in online course-taking, including administrative decisions about the types of educational programs offered at schools and the staffing and management of online instruction, the student bodies served, and policy changes in the district over these years that addressed who should be directed to take courses online. Sometimes substantial swings in the proportion of students taking courses online were related to administrative (e.g., principal) changes or space or staffing constraints. In addition, several of the high schools with the highest rates of online course-taking serve particular student populations, such as pregnant and parenting students and those returning to the classroom from incarceration or expulsion (the latter with 85% to 93% of the student body taking courses online over the four study years). Indeed, descriptive statistics confirm that the highest proportion of online course-takers were in alternative schools (on average more than 31% of their student body), with citywide specialty schools the next highest proportion (more than 11% on average). In addition, nearly all students using the online instructional program were in the same three (of eight) school zones that follow the district’s early calendar year start time, and a number of the high school zip codes with relatively higher proportions of online course-takers are in inner-city, high-poverty areas.

Table 3 presents the results of the logistic regression estimated to predict student use of the online instructional program over time (with pooled data), including school, grade, and year fixed effects. Among the student characteristics examined, a student suspension increased the odds of online course-taking by approximately 15%. With regard to student characteristics, Asian students, English language learners, students with disabilities, and those absent more frequently were less likely to take any courses online, although measures of their prior educational performance were relatively stronger predictors of online course-taking. In particular, students who failed a course in the prior year had 126% higher odds of taking a course online, affirming the “credit recovery” focus of online instruction in the district. This emphasis is also suggested by the fact that the odds of online course-taking increased as a student entered higher grades and were 101% and 89% higher, respectively, in Grades 11 and 12 than in Grade 9. The separate year-to-year logistic regressions predicting online course-taking for each of the four school years showed the same statistically significant patterns of relationships. The odds of course-taking associated with a course failure in the prior year were highest (186% higher) in the most recent (2016–2017) year.

OPEN IN VIEWER

Table 3

Logistic Regression Predicting Student Online Course-Taking, 2013–2014 to 2016–2017 School Years

Student and School Characteristics	Odds Ratios	p Values
Female	1.044	.206
Black	1.022	.736
Asian	0.591	.000
Hispanic	1.003	.964
Other race	1.136	.476
English language learner	0.808	.002
Free/reduced lunch	1.062	.170
Special educational needs	0.882	.006
Percentage absent	0.629	.000
Suspended	1.153	.002
Failed credit	2.264	.000
Prior year GPA	0.623	.000
Prior year credits earned	0.982	.203
Fall reading test score	1.006	.810
Fall math test score	1.008	.745
Grade 10	1.191	.002
Grade 11	2.015	.000
Grade 12	1.886	.000
Principal school administrator	0.686	.000
Citywide school transportation	0.673	.000
2014–2015	0.938	.102
2015–2016	2.065	.004
2016–2017	0.975	.569

Note. N = 23,277, pseudo R-squared = 15.14%, school fixed effects not reported in table. Boldfaced values indicate statistically significant estimates at α = .05. GPA = grade point average.

We found that including school fixed effects accounted for more of the between-school variation than including specific measures of school attributes, with the exception of measures indicating whether a principal was the lead administrator in the school and citywide school transportation. These were (separately) statistically significantly, negative predictors of online course-taking in high schools and were thus retained in the models. The school fixed effects coefficients showed that students in the alternative high school that serves those transitioning back from incarceration or expulsion had the highest (about 3,400% higher) odds of online instructional program use among high schools in the district.

As noted earlier, we also predicted the number of years of online course-taking by a given high school student (0–4 years) using a multinomial logit regression with school fixed effects. These results, presented in Table 4, show that students taking online courses for 1 to 3 years (vs. no years) were statistically significantly more likely to be Black and economically disadvantaged (eligible for free or reduced lunch), although these demographic characteristics did not predict online course-taking for 4 years. Prior course failure is still the strongest predictor of online course-taking for each category (other than some specific schools), with the odds increasing steadily each year to nearly 700% greater odds for students taking courses online for 4 years. In light of this finding, we also predicted the probability that students ever failed a course during the time we observed them in high school, including demographic characteristics and school fixed effects. These results indicated negative, statistically significant associations between being female, Asian, and a student with special needs and course failures, and positive, statistically significant associations between course failures and being Black, Hispanic, other race, an English language learner, economically disadvantaged, suspended, and a higher proportion of school absences. In other words, the students being directed to take courses online—or being grouped into online learning labs after failing courses and segregated from their peers who were performing better—are more likely to be poor, students of color, English language learners, and those missing school for disciplinary or other reasons.

OPEN IN VIEWER

Table 4

Predicting Total Years of Student Online Course-Taking, 2013–2014 to 2016–2017 School Years

	1 Year vs. 0 Years		2 Years vs. 0 Years		3 Years vs. 0 Years		4 Years vs. 0 Years
Student Characteristics (N = 40,027)	Odds Ratios	p Values	Odds Ratios	p Values	Odds Ratios	p Values	Odds Ratios	p Values
Female	1.038	.327	1.109	.024	1.102	.130	1.040	.719
Black	1.211	.004	1.246	.008	1.557	.001	1.122	.541
Asian	0.642	.000	0.556	.000	0.722	.157	0.260	.005
Hispanic	1.081	.296	1.010	.913	1.566	.001	0.900	.578
Other race	1.508	.056	1.204	.507	2.290	.018	1.794	.167
English language learner	1.189	.000	1.214	.000	1.180	.017	1.083	.475
Free/reduced lunch	0.969	.696	0.640	.000	0.414	.000	0.900	.672
Special educational needs	0.720	.000	0.765	.000	0.765	.001	1.086	.497
Percentage absent	0.586	.000	0.384	.000	0.220	.000	0.118	.000
Suspended	1.188	.000	1.145	.005	1.074	.265	0.941	.583
Failed credit	2.510	.000	3.944	.000	5.912	.000	7.930	.000
Prior year GPA	0.710	.000	0.680	.000	0.673	.000	0.642	.000
Prior year credits earned	0.994	.499	0.943	.000	0.892	.000	0.834	.000

Note. Full results available from authors (school fixed effects not reported in table). Boldfaced values indicate statistically significant estimates at α = .05. Pseudo R-squared = 11.14%. GPA = grade point average.

How Are Students Using the Online Instructional Program?

In addressing our second research question concerning how students interact with the online course instructional program, we examine student user behaviors in course sessions to better understand the use and implementation of online course-taking and inform policies and strategies at district, school, or classroom levels for improving student use and educational outcomes. This approach also recognizes the potential for diverse types of student users that could mask important variation in associations between online course-taking and student outcomes. Using all 4 years of student data (approximately 7 million observations) in the k-means cluster and discriminant function analysis, we identified four typologies of student users that we characterize as follows: engaged users, moonlighters, nominal exerters, and incompatible users. Below, we describe these four groups of student users in terms of their course-taking behaviors. Table 5 presents descriptive statistics on these user behaviors, as well as the types of courses taken online and students’ demographic characteristics by user type.

OPEN IN VIEWER

Table 5

Course-Taking Behaviors, Courses Taken, and Student Demographics by User Type, 2013–2014 to 2016–2017

	Engaged Users (N = 12,591)		Moonlighters (N = 3,050)		Nominal Exerters (N = 2,999)		Incompatible Users (N = 413)
	M	SD	M	SD	M	SD	M	SD
Course duration (hours)	122.68	109.48	110.18	102.27	64.29	91.11	39.91	117.45
Completed activities (per day)	4.27	4.47	5.30	6.94	3.62	6.10	2.06	2.74
Session time (minutes)	79.03	54.79	98.21	63.63	61.10	53.87	49.87	42.32
Idle/session time ratio	0.21	0.10	0.26	0.14	0.46	0.19	0.39	0.25
Number of sessions (per course)	45.55	41.01	43.71	43.02	17.78	20.23	1.95	0.83
Number of courses	2.08	1.87	2.18	1.84	1.65	1.44	1.08	0.29
Percentage night school	3.56	5.34	80.26	11.99	4.18	8.44	10.34	27.69
Percentage absent	1.04	5.52	0.95	5.42	1.73	7.94	1.28	6.20
Percentage free/reduced lunch	82.07	38.36	77.06	42.05	81.00	39.24	81.59	38.83
Percentage special educational needs	22.80	41.96	17.73	38.20	26.98	44.40	20.69	40.58
Percentage English language learner	6.37	24.42	5.85	23.48	4.52	20.77	7.93	27.07
Percentage female	45.64	49.81	49.55	50.01	41.59	49.30	45.02	49.84
Percentage Black	69.11	46.21	65.41	47.58	71.63	45.09	66.32	47.34
Percentage Hispanic	20.09	40.07	21.91	41.37	17.42	37.93	19.93	40.02
Percentage White	7.48	26.31	9.23	28.95	8.60	28.05	8.59	28.07
Percentage 9th grade	21.68	41.21	18.38	38.74	30.53	46.06	37.55	48.51
Percentage 10th grade	24.09	42.76	22.79	41.95	24.03	42.73	24.55	43.12
Percentage 11th grade	32.59	46.88	32.77	46.95	29.03	45.40	25.63	43.74
Percentage 12th grade	21.64	41.18	26.06	43.91	16.42	37.05	12.27	32.87
Standardized fall reading test score	−0.11	0.90	0.02	0.91	−0.26	0.94	−0.21	1.04
Standardized fall math test score	−0.15	0.90	−0.07	0.92	−0.30	0.91	−0.25	0.90

Note. M = mean; SD = standard deviation.

Focusing on the first seven rows of Table 5, distinct patterns emerge in user behaviors across the four user types. The engaged users (the largest of the four groups) and moonlighters both have longer course durations and time in their course sessions, and they completed more activities per day (on average) and spent less time idle in the online course-taking system. The primary distinction between the moonlighters and engaged users (reflected in our labeling of the user groups) is that the moonlighters were working on their online courses mostly outside the regular school day (more than 80% of their time in online sessions). The nominal exerters were comparatively less productive users of the online course-taking system with close to half of their time in sessions spent idle (not interacting with the system) and less than half as many sessions per course as the engaged users and moonlighters. The fourth and final group, the incompatible users, was the smallest group and was only identified in the analysis in the first of the four school years (2013–2014). The incompatible users took only one course on average, spent the least amount of time in their course and online sessions, and completed the fewest number of activities in their courses per day.

While the incompatible users were especially ineffectual in their online course-taking, we observed high idle/session time ratios of 0.21 or greater across all four user groups. The following excerpt from a classroom observation describes a typical online course-taking session of a student with a high proportion of idle or unproductive time in session:

The student entered a direct instruction lesson on scenes in Romeo and Juliet, but his cell phone was also in his hand, and he was texting. Some announcements were coming in loudly over the PA system, and students were walking around. The student toggled between his phone and the lesson on the screen, texting while the lecture played and talking to a student nearby. He was playing a game on his phone and attending less and less to the lecture. The instructor came by and told him to take notes, but he did not follow through.

The student in the above observation had full access to the technology and instructional environment, and there was an instructor encouraging the student to work effectively. The student demonstrated a preference for the content available through his cell phone over the video lecture, however, and as we frequently observed, the instructor was ineffective in minimizing this distraction. District policy did not allow teachers to take away phones, but in the 2014–2015 school year, some schools started using cell phone pouches that allowed teachers to lock phones inside, so students could hold onto but not access their phones. Teachers reported initial increases in course progress that filtered off as students learned how to open the pouches.

The demographic characteristics of the different user groups and the types of courses they were taking online (see again Table 5) also shed some light on the distinctions among them. For example, more than 60% of the incompatible users were underclassmen (in 9th and 10th grades), and higher percentages of them were taking math and language arts courses (about half in total) versus electives. We frequently heard in interviews with classroom instructors that they believed this younger group of users was less compatible with the online course-taking system:

Some of the underclassmen are in here all day and it is not working very well; underclassmen do not appreciate the opportunity of making up classes. They do not work. . . . They are not at the reading level of the program.

Recognizing that underclassmen were often unprepared academically and seemingly less motivated to make progress in their online courses, the school district began discouraging (and disabling) online course-taking among 9th and 10th graders, which aligns with our finding that this user group no longer emerged in the cluster analysis after the 2013–2014 school year.

There are other important demographic differences across the four user types as well. The nominal exerters were absent more frequently, were more likely to have special educational needs, were more likely to be Black and male, and, across the four groups, they were the least well-prepared academically in terms of their baseline (fall) reading and math scaled test score averages. Alternatively, the moonlighters were the least likely to qualify for free or reduced priced lunch, least likely to have special educational needs, and were the most academically prepared for online course-taking. Moonlighters spent the longest time on average per course session (nearly 100 minutes), which likely reflects their ability to consistently access the online course system outside of school (unconstrained by bell schedules) and also their slight advantage socioeconomically. The more academically disadvantaged students, alternatively, were less likely to engage with and progress in online courses. As we discuss further below, instructor capacity and willingness to support students in online course-taking outside of regular school hours also likely affected students’ ability to make progress in online courses outside of school.

Associations Between Student Online Course-Taking Behaviors and Course Performance

Table 6 summarizes the results of fixed effects regressions (pooled across the four school years) that adjust for student, school, grade, and year fixed effects in assessing the relationship between course-taking behaviors and students’ online course performance—course pass rates, on-time completion of courses, course grades, and the rate at which courses were disabled by instructors—over time, while also controlling for student demographic characteristics (see the specific measures in Table 3) and prior test score performance. The results show a number of strong, consistent (statistically significant) predictors of online course outcomes across the four course performance measures and years. Students who spent more time in their courses and completed more activities per day (fitting the profile of the engaged users) appeared to be more successful in online course-taking and were less likely to have their courses disabled for lack of progress. Alternatively, the higher the proportion of time that they spent idle in their course sessions, the worse they did in all course outcomes. For example, for each additional percentage point of time idle, on average, course pass rates fell by about one third of a percent and on-time pass rates by about one fifth; this is relative to average course passing rates of about 30% and on-time pass rates of about 17% across the four school years. Course grades (on a scale of 0–100) were about 0.42 points lower for each additional percentage point of time idle.

OPEN IN VIEWER

Table 6

Relationship of Student Online Course-Taking Behaviors to Course Performance

Online Course-Taker Behaviors (N = 8,531)	Course Pass Rate	On-Time Pass Rate	Overall Grade	Course Disabled Rate
Course duration (hours)	0.032	0.006	0.045	−0.049
	(0.008)	(0.008)	(0.005)	(0.010)
Completed activities (per day)	1.474	0.637	0.623	−1.310
	(0.237)	(0.226)	(0.145)	(0.290)
Idle/session time ratio	−35.387	−20.413	−41.802	20.409
	(6.812)	(6.507)	(4.184)	(4.184)
Session time (minutes)	0.138	0.065	0.155	−0.212
	(0.050)	(0.047)	(0.030)	(0.030)
Number of sessions (per course)	0.200	0.095	0.080	−0.049
	(0.024)	(0.023)	(0.015)	(0.015)
Number of courses	3.333	3.346	0.954	−1.354
	(0.582)	(0.556)	(0.358)	(0.358)
Percentage night school	0.001	0.011	−0.001	0.021
	(0.036)	(0.034)	(0.022)	(0.022)

Note. Standard errors in parentheses. Estimates from fixed effects regressions (with student, school, grade, and year fixed effects). Coefficients in boldface are statistically significant at α = .05.

Table 7 shows how these critical student course-taking behaviors changed over time by the four typologies of student users identified. It is notable that for the three groups identified in each of these school years, the course duration (in hours) steadily declined, and students were completing their courses in fewer sessions. Activities completed per day also declined, which would be expected with fewer hours spent in online courses. The analyses presented in Table 6 showed that these three course-taking behaviors were associated with better online course outcomes, and online course performance (e.g., passing rates and on-time completion rates) were continually improving over time. It is plausible that these patterns could reflect more efficient use of the online course-taking system over time; the fraction of all students falling into the nominal exerters category declined from 33% in 2013–2014 to only 7% in 2016–2017; the incompatible users fell out of the classifications as a group, and the moonlighters (engaging in most of their online course sessions outside the school day) nearly doubled over this period. Another possible explanation relates to a substantial increase over time in the proportion of students taking and passing course pretests—from passing rates of 26% in 2013–2014 to 67% in 2016–2017—which allowed students to “test out of” and bypass some or all parts of course instruction (and thereby complete courses in fewer sessions). This raises the question of whether course passing rates imply mastery of content—that is, learning that would be reflected in other measures of academic progress. We turn now to further explore how these student interactions with the online course-taking system affected students’ academic progress and intermediate academic outcomes, comparing online course-takers over time (in traditional and online courses) and with students who did not engage in online course-taking in high school.

OPEN IN VIEWER

Table 7

Student Online Course-Taker Typologies and Course Performance by School Year

	Engaged Users					Moonlighters
School Year	n (%)	Course Duration (Hours)	No. of Sessions (per Course)	Idle/Session Time Ratio	Completed Activities (per Day)	n (%)	Course Duration (Hours)	No. of Sessions (per Course)	Idle/Session Time Ratio	Completed Activities (per Day)
2013–2014	2,483 (46)	191.03	57.82	0.26	5.08	718 (13)	145.86	43.29	0.32	5.64
2014–2015	4,449 (79)	124.13	47.54	0.19	3.97	903 (16)	125.49	50.70	0.22	5.48
2015–2016	3,280 (72)	104.90	45.82	0.20	4.22	613 (13)	100.74	47.82	0.25	5.52
2016–2017	2,379 (69)	96.56	40.86	0.22	3.87	816 (24)	90.88	39.97	0.25	4.98
	Nominal Exerters					Incompatible Users
School Year	n (%)	Course Duration (Hours)	No. of Sessions (per Course)	Idle/Session Time Ratio	Completed Activities (per Day)	n (%)	Course Duration (Hours)	No. of Sessions (per Course)	Idle/Session Time Ratio	Completed Activities (per Day)
2013–2014	1,803 (33)	86.39	20.38	0.44	2.99	413 (8)	42.15	1.97	0.38	2.14
2014–2015	310 (5)	36.25	10.16	0.55	4.38	0.00
2015–2016	648 (14)	45.17	19.90	0.45	5.54	0.00
2016–2017	238 (7)	31.77	13.25	0.53	2.69	0.00

Associations Between Online Course-Taking and Intermediate Academic Outcomes

As discussed above, with panel data available (over 4 years) and variation in online course-taking within students and schools over time, we estimated fixed effects models to examine associations between online course-taking and intermediate academic outcomes. These models allow us to adjust for stable student, school, grade-level, and year factors, while controlling for other student characteristics, pretreatment measures of the outcomes, and time-varying school characteristics, including the percentage of online course-takers in each school (see again the covariates in Table 3). Panel A in Table 8 presents the fixed effects and IPWRA estimates of average effects (associations) of any online course-taking to test scores, GPA, and credits earned at the end of the school year, along with standard errors (statistically significant coefficients in bold). As seen in Panel A, the average associations between online course-taking and intermediate outcomes are mostly negative, although only the associations between math and reading test scores and online course-taking are statistically significant. The fixed effects and IPWRA estimates are highly comparable. They suggest that, on average, online course-taking is not benefitting students or reflecting real learning, and some students may even be set back in their learning (as suggested by lower average test scores).

OPEN IN VIEWER

Table 8

Estimated Average Associations Between Online Course-Taking and Student Intermediate Academic Outcomes

	Credits Earned	Grade Point Average	Math Test Score	Reading Test Score
Panel A: Results by method	N = 47,165	N = 39,712	N = 20,218	N = 19,891
IPWRA	−0.020	−0.011	−0.043	−0.063
	(0.027)	(0.011)	(0.014)	(0.015)
Fixed effects	0.040	−0.004	−0.039	−0.050
	(0.027)	(0.011)	(0.020)	(0.022)
Panel B: Fixed effects models with treatment interactions	N = 47,165	N = 39,712	N = 20,218	N = 19,891
Online course-taker	−0.250	−0.124	−0.052	−0.055
	(0.037)	(0.015)	(0.024)	(0.026)
Grade 11 × Online course-taker	0.486	0.184	0.044	−0.014
	(0.046)	(0.018)	(0.035)	(0.038)
Grade 12 × Online course-taker	0.442	0.222	−0.357	0.008
	(0.056)	(0.023)	(0.150)	(0.166)
Panel C: IPWRA models with intensity (years) of online course-taking interactions	N = 45,699	N = 38,187	N = 19,510	N = 19,224
1 vs. 0 years	−0.095	−0.087	−0.026	−0.021
	(0.024)	(0.010)	(0.011)	(0.013)
2 vs. 0 years	−0.085	−0.107	−0.043	−0.016
	(0.032)	(0.012)	(0.015)	(0.016)
3 vs. 0 years	−0.009	−0.082	−0.060	−0.095
	(0.055)	(0.019)	(0.034)	(0.033)
4 vs. 0 years	−0.155	−0.112	−0.194	−0.161
	(0.139)	(0.042)	(0.058)	(0.058)

Note. Estimation methods are fixed effects regressions and IPWRA. Data provided are coefficient estimates with standard errors in parentheses. Coefficient estimates in boldface are statistically significant at α = .05. IPWRA = inverse propensity score weighting with regression adjustment.

In light of instructors’ comments about the incompatibility of the online course-taking system for underclassmen and the district’s change in policies to discourage use among 9th and 10th graders, we also estimated fixed effects models with interactions between online course-taking and grade level (distinguishing 11th and 12th graders from the underclassmen). These results (see Panel B in Table 8) show negative and statistically significant average associations between online course-taking and all four intermediate outcomes. The interactions between online course-taking and the indicators for 11th and 12th grades, however, are positive and statistically significant for credits earned and GPA. When combining the treatment and interaction estimates, the associations between online course-taking and these two outcomes are positive: 11th graders earned 0.236 more credits and 12th graders earned 0.192 more credits on average in online course-taking, and their GPAs were accordingly 0.06 grade points (11th graders) and 0.098 grade points (12th graders) higher on average, as failed courses were replaced with online course grades. Alternatively, the results of these models appear to substantiate the finding that there were no significant increases in student learning through online course completion, as measured by their reading and math (scaled) scores at any grade level.

In our final empirical analysis, we examine how intensity (number of years) of online course-taking in high school is associated with student intermediate outcomes, using the doubly robust IPWRA estimation approach that adjusts for selective differences in the level of course taking (0–4 years). In Table 4, we showed that students taking online courses for 1 to 3 years (vs. 0 years) were statistically significantly more likely to be Black and economically disadvantaged, although prior course failure was the strongest predictor for all levels of online course-taking. The findings of this analysis, presented in Panel C of Table 8, show increasingly negative and statistically significant associations between more years of online course-taking and academic outcomes, where students taking courses online for 4 years (vs. 0 years) appear to experience the largest penalties (particularly in terms of GPA and test scores). This subgroup of students (taking courses online all 4 years) constitutes about 1.25% of all high school students and 6% of the online course-takers over this period (in this district) and, therefore, represents an extreme form of ability grouping of academically struggling students who were confined primarily to an online instructional environment that provided few supports for their learning, as the qualitative findings suggest below. These larger estimated negative effects on student achievement associated with online course-taking in all 4 years (−0.194 SD [standard deviation] for math scaled scores and −0.161 SD for reading scaled scores) also align with the findings of Ahn and McEachin (2017), who examined patterns in student achievement outcomes in e-schools and found that e-school high school students performed worse on standardized tests (−0.230 SD for math and −0.128 SD for reading) relative to their peers in traditional classroom settings.

Insights From Qualitative Analysis on Lack of Student Academic Gains

The classroom observations and interviews with teachers, as well as discussions of the findings with district staff, suggested possible reasons for the lack of positive (and some negative) associations between online course-taking and student achievement. One consistent concern reported in teacher interviews was the low reading levels among students directed into online course-taking. One teacher indicated that many students enrolled in the courses were at 3rd- to 5th-grade reading levels and that the mismatch between their reading levels and the level of reading required in the online courses was a “big demotivator.” In addition, for students for whom English is a second language, teachers found that the language accommodations in the online course-taking system were not adequate. A teacher pointed out that the translation function in the system occurs in text (not voice) format, so students have to be able to read the text while the online instructor is talking in English. However, it is more often the case that students understand the spoken language but do not know or learn the written (native) language.

Another recurrent challenge was the apparent lack of accommodations in the online course-taking system for students with special educational needs. Teachers indicated that they typically did not have access to information about student individual education plans (IEPs) or extra resources to support their needs; as one teacher explained,

I have someone with an IEP in my second hour . . . nothing in [the online course-taking system] really accommodates them. They expect the teachers to accommodate them.

Some instructors made efforts to meet these students’ needs, particularly if they had experience or training in special education. For example, one teacher printed the transcripts of online videos and had students highlight them, while another found practice tests and worked with students outside the online system on content support. Several teachers also mentioned a resource room where special education students could work on online courses with their IEP teacher, although this depended on whether the IEP teacher possessed technical knowledge required to support use of the online course-taking system:

I think that many of the special ed kids are frustrated. I have a very good relationship with my special ed teacher, so a lot of the kids that have me for class go to the resource room to work on this. She is also trained [in the system] and has her own account, which makes a difference.

We also saw many student behaviors that suggested a lack of engagement in the learning process, such as the following examples from classroom observations:

The student did not interact much at all with the software (i.e., didn’t progress through the screen). The aide checked in with her at the beginning of class and told her to get going, that “she is smart and can do it.” There was no direct interaction with a teacher after that point. The student just talked to another student next to her. The student would click on a screen when the teacher walked by, otherwise she would just stare at the screen and talk with her friend.

Other students appeared to be somewhat engaged in getting through their online courses, but not necessarily in learning. We observed distinct differences between active and passive engagement among students, where in the latter case, they moved through the program, sometimes taking notes, but not necessarily engaging with the content, as shown below:

The student was working on a lesson on the Mongol Empire when the observation began, reading a source document and taking notes. He entered an assessment mode (about halfway through the observation) and went to Google to search for answers to the questions. In some cases, the student copied and pasted the exact assessment question into Google to find the answers.

Observations such as these raise the question of whether the goal of online credit recovery is to simply provide students an opportunity to earn credits needed to graduate, or whether an important aim is to provide an alternative instructional environment and flexibility for students who may not have been successful in a traditional classroom setting to master content needed for life after graduation. These findings also raise concerns about whether students taking courses online are learning at the same level as those in traditional classrooms as they earn course credit. Two teachers, both with substantial (multiyear) experience as instructors in online course-taking classrooms, commented on the low attendance rates in the online instructional settings, which both pegged at about 25%. One of these instructors was very direct in stating “the students aren’t learning anything” [in their online courses].

One of the barriers to students mastering content in the online setting may relate to the fact that some classroom instructors struggled to help students when they were challenged in their online courses, particularly in subjects outside their content expertise. In an interview, one teacher explained her difficulty with providing math support:

As a non-math person, I find it difficult. I can do it if I watch the whole video, but I don’t have the time to watch the entire video to answer the questions with a student.

We observed a lack of math content expertise limiting the efficacy of instructor assistance more often than in other subjects. In response to this concern, some schools have placed content teachers in lab settings where online courses are accessed, and one school grouped course subjects by period so all students were working on math modules that session, allowing content teachers to come in and provide extra instructional help to more students. To date, the efficacy of this strategy has been mixed. For example, in one observation, the content teacher was reported to be effectual and engaged, while another sat in a corner working on his own stuff, and a third just gave the students the answers.

Finally, for some of the students, academic progression may be a secondary aim to the implicit goal of the online instructional program in providing a “safe space” for students who might otherwise not be in school. This theme came out in observations and interviews, particularly in contexts where lab instructors were managing the intersection of the classroom with students’ complicated lives. For example, one teacher was describing how “these kids have so much baggage and drama in their lives” when a pregnant and parenting youth coordinator came into the room to speak with two of the students. One of the students had only one class to finish but was not progressing well. The teacher pointed to another student who she said had emotional problems and had made very little progress. The student came into the classroom, put her head down, and slept throughout the period without logging onto a computer. The instructor indicated that she was 19 already but at the 11th grade level in terms of her credits and that she would probably “just sit here until she is 21 and will call it a day.” Another teacher explained during an observation that if a student is making progress, she does not harass them. “This lab becomes a place for EBD (emotionally, behaviorally, disabled) students to decompress for a period so they are better able to deal with their other classes.” In fact, some teachers went out of their way to extend support to students outside the regular school day:

The teacher will take emails from kids until 9:00 pm at night (and often much later) to unlock or help them progress through a course. He showed me an email from 12:30 am the previous night and said, “If kids are motivated enough to work at home, the least I can do is respond.”

In interviews and research briefings, teachers and other staff suggested that if it were not for the online course-taking option, some of these students would not be in school at all or would be disruptive in the regular classroom. In one classroom observation, a teacher pointed out that a student who was sleeping in the room had completed his courses and had nowhere else to go for the rest of the school year. More than one instructor used the term dumping ground to describe how students came to be placed in their classrooms for online course-taking. Ultimately, analyses of students’ longer-term outcomes, including high school completion and postsecondary education and labor market outcomes, may shed further light on whether, or the extent to which, these students benefit or are harmed through online course-taking.

Strategies Employed in Schools to Improve Online Course-Taking

The school district partnering in this research is aware of many of the challenges and constraints to successfully implementing online course-taking and credit recovery in this context. Through classroom observations, interviews, and research briefings with district staff, we identified specific strategies they are pursuing to address some of the concerns highlighted above and improve online course-taking and student outcomes. These include specific guidelines and directives for implementing online course-taking, for both students and instructors, which were rolled out in the 2015–2016 school year:

The following excerpts from observations of online course-taking in classrooms illustrate some of these practices:

The teacher was emphasizing to the students that they needed to strive for the goal of completing three percent of their coursework per week. He told them to focus more and to take advantage of the resources they have both during and after the school day to work in [the online instructional system]. He asked to see their notes when they requested access to an assessment (quiz).

By setting weekly progress goals in conjunction with weekly one-on-one student-teacher check-ins, teachers provided students regular feedback and directed them toward more manageable goals. Indeed, we saw many examples of creative and concerted efforts by teachers to follow the district’s guidance and improve online course-taking supports, including teachers who tracked and encouraged student progress toward goals using charts and incentives (e.g., certificates, rewards) and those who developed their own instructional materials to aid student interactions with the system. One teacher, for example, created a “March Madness” competition for his students in different class periods to motivate their progress toward individual goals.

The process of taking notes can help students in learning content and provide focus for future studying, as well as support their successful completion of online course assessments. The practice of asking students to show notes before allowing access to an online assessment was intended to increase the likelihood that students would view and interact with the instructional content and resources available online and also discourage behaviors such as guessing answers or otherwise attempting to complete the quiz without learning the content. Our classroom observations in the school years since this policy was first implemented suggest, however, that it is inconsistently enforced, and many students simply ignore teacher directives to take notes during instructional videos. In the effort to provide students with more structure for note-taking, one teacher even went as far as to watch the online courses himself and create “guided notes” to help students identify the material in the online instructional videos that would be important to know for the end-of-course assessments.

Students who did not meet goals established by teachers to incentivize adequate course progress often had their online courses disabled, and some were reassigned to an alternative class setting where they could receive more in-person and one-on-one assistance from an instructor. The policy intent was to prevent students from spending extensive time in online courses without making progress toward course completion.

During the session, the instructor told students that if they save a quiz, the answers are locked in and cannot be changed later. He offered to check their answers before they save. He stated: “Everyone is behind, and no one seems to have a sense of urgency.” Another teacher entered the room to make an announcement. He said that students who don’t get to 12 percent by next Thursday will have their accounts suspended. He tells the students that he is available every day from 4–6 p.m. (Mon.–Thurs.) and that [the online instructional system] is available 24/7.

Above, the instructor offered to check student answers before they submitted an assessment. This practice, encouraged by the district, made instructors aware of the questions that students did not answer correctly. On occasion, this led to reteaching and targeted, blended instruction to help students better understand the content underlying the questions they had answered incorrectly. More often, however, instructors told students which questions they had answered incorrectly, in which case some students went back to review their notes, but more commonly, students used a process of elimination to determine the answers (for the questions then known to be wrong).

Limited resources undoubtedly constrained the implementation of these practice guidelines and other instructional supports (e.g., live teacher interactions) for online course-taking. One teacher indicated that she was supposed to have 20 students in the classroom, but that on any given day, she might have as many as 45 students; she reported having 65 students in her classroom the previous year. Another teacher explained how a high student-teacher ratio limited his interactions with students:

We need smaller class sizes than we have; I think I could do it well with 35. One class we saw was 74 students; ideally, we would have 25–30 students. We need more time for one-on-one interactions with the students.

Our descriptive analysis of classroom observation data—which revealed a negative association between observed student-teacher ratios and student digital citizenship (i.e., appropriate use of technology)—aligned with this teacher’s insight. A few instructors also described problems using the LAN, explaining that it has not consistently worked with some of the operating systems in use (e.g., Chrome). And in cases where substitute teachers were present but lacking experience with or the ability to log into the online course-taking system or use the LAN, instructional supports might be completely absent, as seen in the following classroom observation:

The first thing the sub said as students were coming in was, “I can’t check your work today. I can’t help you.” The sub had no way to interact with the resources, and therefore, had no real interaction with the students.

Observations such as this underscore our strong concerns about the potential for differential access to quality learning experiences between online and traditional classroom environments and the perpetuation of racial and class disparities through ability grouping in online learning environments, given the disparate access to fundamental instructional supports and learning opportunities we have observed in these settings.