Abstract
School districts required under No Child Left Behind (NCLB) to provide supplemental educational services (SES) to students in schools that are not making adequate yearly progress rely heavily on the private sector to offer choice in services. If the market does not drive out ineffective providers, students may not gain through SES participation. We estimate SES provider effects on students’ math and reading achievement in an urban school district that accounts for a significant share of participating students. We expect this research to inform education policy on the viability of policy interventions employing a private market model to improve public sector outcomes, including the reauthorization of Title I and district tutoring interventions under NCLB and after federal waivers from NCLB.
Introduction
The U.S. No Child Left Behind (NCLB) Act was designed to increase the achievement of economically disadvantaged students by introducing greater choice, flexibility, and accountability in public education. Key legislative provisions include offering educational choice to those in persistently low-performing schools; empowering parents with more information about the quality of their children’s schools; holding states, districts, and schools accountable for student achievement by identifying and imposing requirements on schools in need of improvement; and targeting federal funds on effective practices for improving teacher and school quality. 1 As in the decentralization of government functions in other policy areas (public assistance, publicly funded training, child welfare, etc.), the expectation is that involving the private sector in public services delivery will bring about a more efficient, innovative, competitive, results-oriented, and responsive public sector that better meets diverse public preferences, values, and needs (Frederickson & Smith, 2003; Rivlin, 1992).
We study the NCLB provision that requires schools that are not making adequate yearly progress for 3 years to offer children in low-income families the opportunity to receive extra academic assistance (or tutoring), known as supplemental educational services (SES). The primary responsibility for implementing SES lies with school districts, which largely rely on the private sector to offer eligible students a range of choices for SES. NCLB obligates school districts to set aside 20% of their Title I funding for SES and lays out criteria and guidelines for state and local educational agencies in approving SES providers and arranging for their services. 2 Importantly, it requires state and local educational agencies to measure provider effectiveness in increasing student achievement and to use this information to withdraw approval of ineffective providers (Heinrich, 2010).
NCLB also directs school districts to determine eligibility for SES using the same information used for making within-district Title I allocations (e.g., free school lunch eligibility), and schools then notify families of their children’s eligibility. If more eligible students sign up than there are funds available for serving them, districts establish additional eligibility criteria, frequently based on student special needs and academic performance. However, some eligible students do not follow through in registering for and attending SES, and others stop attending before their SES dollar allocation is expended. Therefore, selection into treatment (or who gets tutored in SES) and for how long is likely influenced by student characteristics as well as program type and administration (Heinrich, Meyer, & Whitten, 2010; Steinberg, 2011). This makes it enormously challenging for researchers, and even more so for school districts with limited resources, to identify effects of SES providers on student achievement while controlling for the effects of other classroom and school interventions.
Chicago Public Schools (CPS) accounts for a disproportionately large share of the students eligible for SES in the United States, and as we discuss further in the following, CPS has consistently made an effort to evaluate SES effects, including provider effectiveness. We use the most recently available longitudinal data from CPS—the 2007–2008, 2008–2009, 2009–2010, and 2010–2011 school years—to estimate provider-specific effects and the effects of different types of SES providers using alternative estimation techniques and subsamples and under differing assumptions about student and provider selection. Based on prior literature and our conceptual framework (discussed in the following section), we hypothesize that there are positive effects of attending SES on the achievement of eligible students that will increase with hours attended, provider characteristics will be correlated with SES effectiveness, and some specific providers will be more effective than others in delivering SES and increasing student achievement.
We employ four alternative approaches to estimate the effects of SES and different type of SES providers on changes in student test scores while controlling for student selection into SES and/or into different provider types. The SES provider types that we examine are: district versus non-district providers, online provision, on-site versus off-site providers, and for-profit versus not-for-profit (nonprofit or public) provision. 3 We use gains in test scores as our outcome in school value-added, student fixed effects, school and student fixed effects, and propensity score matching models. We control for school and student time invariant characteristics using these four strategies. Each of these modeling approaches makes somewhat different assumptions about selection into SES, and as we are estimating different types of effects (e.g., provider specific, provider type), the analytical samples differ to some extent as well. Thus, while we look for overall consistency in the results, we expect some differences as well.
In general, we find that the CPS district provider delivers significantly more hours of tutoring to students who register to receive SES, and there appears to be strong linkage between hours of SES attended and increases in student achievement. In addition, the CPS district provider is an on-site provider, and we also find that SES delivered by on-site providers is effective in increasing student achievement. Students attending SES with online providers and for-profit providers gain less in reading and math as compared to other off-line and not-for-profit providers, respectively. We also identify a number of providers that consistently have positive and statistically significant effects on student achievement. Overall, while average effect sizes are small (approximately .09 in reading and .06 in math) relative to other educational interventions, there is some limited evidence that the SES program contributes to improving student achievement.
In the next section, we discuss our conceptual framework and some background information, followed by a review of the literature to date on SES effectiveness. We then describe the data and estimation approaches that we employ in the analysis of SES effects and present some descriptive statistics on the different types of SES providers and the characteristics of CPS students they serve. We follow with a discussion of the results of the estimation of SES effects for different provider types and specific providers in CPS and conclude with policy implications.
Conceptual Framework and Background
Consistent with broader public sector management trends and the classic market paradigm, an important component of recent initiatives to improve K–12 education effectiveness are efforts to introduce market-driven management and increase pressures on educational institutions to develop new strategies for improving student learning and educational outcomes. In the context of SES, measuring provider performance and disseminating information on provider effectiveness should foster a more competitive market for services, contribute to more informed student choices, encourage innovative approaches to service delivery as providers compete for market share, and squeeze out inefficient or ineffective providers through choice.
Indeed, a large number of diverse organizations with widely varying hourly rates, service costs, tutor qualifications, tutoring session length, instructional strategies, and curricula have entered the market to compete for the opportunity to provide SES. These include national and local organizations, for-profit and nonprofit providers, online and off-line providers, those offering services on-site at the schools (and off-site), and in some cases, schools districts engaging directly in SES provision. Burch, Steinberg, and Donovan (2007) described the market for SES as “a very new market where hundreds of firms are flocking to take advantage of the promise of sizeable revenues” (p. 121). NCLB explicitly discourages states from taking any actions that might limit the supply of providers and range of choices available to parents. 4
Private companies have long been involved in the delivery of K–12 public educational services, from textbooks and instructional supports to testing and evaluation, tutoring, and more. More recently, the increasing participation of for-profit entities in direct services provision through vouchers for school choice, charter schools, and mandatory out-of-school-time interventions has drawn criticism of the prospects for “profiteering” in public education, where for-profit firms are viewed as willing to compromise on quality and to shortchange students to better their bottom line (Horn, 2011). The primary opposing view counters that private firms have considerable potential to cultivate critically needed innovation in educational practice; that is, they are more likely to have the capability and incentives to rapidly expand successful practices and approaches, attract the required financial and human capital, and more cost-effectively deliver educational services.
In the context of this debate, Peterson (2003) recounts the compromise between Capitol Hill conservatives who supported vouchers as a key lever of accountability through choice and liberal politicians who opposed the encroaching private sector role in K–12 public education that led to the creation of SES under NCLB. SES allowed for the “back door” entry of private providers-for-profit, nonprofit, secular, and religious into public schools that were failing to make adequate yearly progress while preserving an important role for school districts in arranging access for students to SES and in contracting with private providers. School districts identified as “in need of improvement” were, for the most part, prohibited from directly providing SES, on the premise that if they were not effective during the regular school day, they would be unlikely to do better after school. Private providers, alternatively, might benefit from the fact that SES is voluntary rather than compulsory, allowing them to potentially work with a more motivated group of students enrolling in an afterschool academic program. In addition, they are free to hire and fire teachers/tutors (unencumbered by the typical union rules) and have broad leeway in program structure, focus, and curricular design.
School districts have assumed the major responsibility for disseminating information on SES providers approved by state educational agencies and operating in the district, although to date, this has largely consisted of provider self-reported information on their attributes and effectiveness (Heinrich et al., 2010). SES providers also market their services directly to parents and students. Some school district accountability and evaluation units have attempted to measure program and provider effectiveness with administrative and student record data, as in Chicago Public Schools. They face important challenges, however, in properly evaluating student- and provider-level SES effects, given that participation in SES is voluntary among eligible students.
In general, school districts have few resources for monitoring SES providers and little leverage for dismissing them, 5 and they have criticized state educational agencies for their lack of responsiveness to reported problems with providers, including fraud and ineffectiveness. Chicago Public Schools has been particularly proactive in its efforts to disseminate available information to parents and school principals about SES provider effectiveness and to develop district policies that support monitoring of providers that use its school facilities for service delivery. Still, in the absence of accurate and fairly complete information on SES provider performance, states and districts have little capability or leverage for disciplining the market (i.e., sanctioning or disqualifying ineffective providers), and the benefits of choice in a competitive market are unlikely to be realized if the purchasers (eligible students or their parents) have inadequate information for choosing providers.
Review of Literature on SES Effectiveness
Previous research on out-of-school-time programs reports mixed findings on the effectiveness of these programs in improving student outcomes (Halpern, 2003; Little, 2007). Many after-school programs, particularly those with a greater focus on recreational than educational activities, have been shown to have minimal effects on students’ academic progress (Hollister, 2003). SES, however, was designed to explicitly address students’ educational needs, and the large literature on other after-school/tutoring programs confirms their potential to increase student achievement with sufficient hours of tutoring (Dynarski et al., 2004; Halpern, 2003; Lauer et al., 2004; Little, 2007; Vandell et al., 2005). Yet to date, little is also known about what types or attributes of SES programs are effective and what policies at state or local levels can maximize the potential benefits of SES for eligible students.
Early studies of SES effects on student achievement were primarily descriptive and focused on the challenges of implementing the services in an evolving market (Burch et al., 2007), while more recent studies have sought to empirically estimate the effects of SES on student achievement. Chatterji, Kwon, and Sng (2006) estimate the effects of SES in one New York school and found small positive effects. Evaluations conducted by Chicago Public Schools in 2003–2004, 2004–2005, 2006–2007 and Jones (2009) reported larger gains in reading and mathematics for students receiving at least 40 hours of tutoring and for students in Grades 4 through 8 who were not English language learners and who received at least 30 hours of SES tutoring. A study by the Los Angeles Unified School District (Rickles & Barnhart, 2007) found fairly small program effects, attributed primarily to improved performance by elementary students. Studies in Minneapolis (Heistad, 2005) and Milwaukee Public Schools (Heinrich et al., 2010), where average SES hours attended are particularly low, did not find statistically significant, positive effects of SES participation. Zimmer, Gill, Razquin, Booker, and Lockwood (2007) reported average increases in math test score gains of .09 standard deviations for students attending any SES across the eight sites in their study, and Springer, Pepper, and Ghosh-Dastidar (2009) likewise estimated increases in test score gains of about .09 standard deviations in mathematics (and .076 standard deviations in reading). In an alternative specification that accounted for those who registered but did not attend SES, however, Springer et al. did not find statistically significant effects on reading for students attending SES. Springer et al. caution that very few studies rigorously adjust for student selection into SES, identifying only four studies besides their own that did (Heistad, 2005; Heinrich et al., 2010; Zimmer, Christina, Hamilton, & Prine, 2006; Zimmer et al., 2007).
There are likewise few studies that rigorously examine the effects of specific SES providers on students’ academic outcomes (Jones, 2009; Muñoz, Potter, & Ross, 2008; Potter, Ross, Paek, Pribesh, & Nunnery, 2006; Ross, Potter, Paek, & McKay, 2008; Zimmer et al., 2007). This is especially problematic given that it was the explicit intent of NCLB to hold providers accountable by giving students and parents the necessary information on provider performance to exercise choice and realize the benefits of a competitive market for services.
Jones (2009) used multilevel modeling to explore SES provider effects on students who attended SES, controlling for student- and school-level characteristics. 6 He reported moderate effects of attending SES and positive effects of several individual providers in Chicago. Studies by Zimmer et al. (2007) and Socias, deSousa, and Le Floch (2009) used a difference-in-differences strategy with value-added models to estimate SES provider effects across multiple districts after the introduction of district providers in 2006–2007. The Zimmer et al. study found that participation in SES had positive effect on students’ achievement in reading and math, with students participating for multiple years realizing larger gains. Although the Zimmer et al. study did not estimate the effects of specific providers, they did estimate the effects of a district provider and reported mixed results. Socias et al. found that the district SES provider had no effect on student achievement in Anchorage and Hillsborough.
The Center for Research on Education Policy (CREP) conducted several, multiyear studies in Kentucky, Tennessee, and Louisiana on provider-specific effects using a matched treatment-comparison strategy. Students attending SES were matched on observable characteristics to schoolmates who were eligible for SES but did not participate. They found mixed results for the overall effect of attending SES across different states. Specifically, Muñoz et al. (2008) analyzed student-level achievement for those who attended SES in Kentucky and found no significant effects for any individual provider or for all providers combined. Similarly, Ross et al. (2008) found three providers were significantly worse than the comparison group in math (suggesting the potential for selection bias that is not adequately adjusted through matching) but no effects for any providers in reading for the 2007–2008 school year in Tennessee. Finally, Potter et al. (2006) found most students in Louisiana who were served by SES providers did no better or worse than their counterparts who were not served.
The gap in knowledge that we aim to fill is to identify not only effective providers but also their attributes and approaches in delivering SES that contribute to their success. This should help policymakers better direct the resources that school districts are required to set aside for publicly funded tutoring. Although we realize that these results are based primarily on a single school district and have limited generalizability, CPS has one of the largest numbers of students eligible for and receiving SES, accounting for 10% of all SES recipients in the nation’s public schools in 2008–2009. It is also one of a small number of school districts identified as in need of improvement that successfully petitioned the federal government to directly provide SES. We more fully explore the implications of direct service provision by districts in the discussion of our findings and the concluding section.
Data and Methods
We obtained school record data for all students eligible for SES in CPS for the school years 2007–2008, 2008–2009, 2009–2010, and 2010–2011. The longitudinal database that we constructed includes student test scores, demographics, and information on their registration for and participation in the SES programs. These data allow us to construct measures of students’ SES attendance with specific providers, including the number of hours of SES attended and total expenditures from provider invoices. The district also provided information on the SES providers, including whether they were online, off-site, or on-site; district or non-district; for-profit or nonprofit, which allows us to explore the types of organizations and methods of service delivery that may contribute to improving student outcomes.
To construct the key outcome measures of student achievement gains (or changes) in student test scores, we use data from standardized tests (Illinois Standardized Achievement Tests; ISAT). For each grade and year, we construct z-scores using the district mean and standard deviation, so that the test scores are comparable across grades and years. Table 1 shows the number of CPS students who are eligible, registered, and attended SES for the different grades and years as well as the percentage of students with missing scores. Across the three panels and school years, the distribution of characteristics for those who are eligible, register for, and attend SES look similar to the subset of those with gain scores. Therefore, the missing data (other than Grade 3) 7 will be treated as random.
Number of Students Eligible, Registered for Supplemental Educational Services (SES) and Who Attended SES, With and Without Gain Scores, for the 2008–2009, 2009–2010, and 2010–2011 School Years
Characteristics of Eligible Students and SES providers
Table 2 shows the characteristics of students who are eligible and registered for SES and who attended SES for the 3 school years for which we estimate SES effects. These measures, along with the student’s grade year and school attended, serve as the core set of control variables intended to account for selection into SES in the estimation of SES effects. Our choice of these variables reflects specific criteria used by CPS in prioritizing eligible students for SES, as well as other characteristics we expect might be associated with the likelihood that eligible students will register for and attend SES. Per the law, the primary criterion for most school districts is free lunch eligibility, but as the number of students eligible for SES has expanded, districts have specified additional criteria for registrations, such as grade level, past performance in school, and student special needs. For example, in the 2009–2010 school year, CPS established the following hierarchy of criteria for prioritizing registrations among eligible students: (a) free/reduced lunch eligible; (b) all students in Grades 1 through 3 and in high school and English language learners (ELL) and students with disabilities (SWD) in Grades 4 through 8; and then (c) students in Grades 4 through 8 by reading stanine. The new emphasis on registering SWD and ELL students in 2009–2010 is reflected in the larger percentage of SWD and ELL students that registered that year (vs. 2008–2009), even though the percentage eligible were approximately the same from year to year. We also control for student absences in the prior school year, as we find, not surprisingly, that students who are less likely to attend regular school are also significantly less likely to attend SES after school. 8 In addition, students who attended SES in a prior school year are significantly more likely to attend SES again.
Student Characteristics of Those Who Are Eligible, Registered, and Attended Supplemental Educational Services (SES; percentage unless otherwise indicated)
If after controlling for the student characteristics shown in Table 2, other factors influencing participation in SES are random, then our estimates of the effects of SES should approximate the true effects. It is not possible to verify, however, that there are no unobserved, selective differences between eligible students who participate in SES and those who do not (and serve as comparison group members in our analysis). A well-executed random assignment experiment would be needed to assume statistical equivalence between students participating in SES and eligible nonparticipants. For example, if there were consistently greater demand for SES than funding, school districts might be persuaded to allocate opportunities to participate in SES by randomly assigning some fraction of the eligible students to receive services and others to a control group of students who would not be invited to participate. This would facilitate a causal analysis of the effects of SES on student outcomes. In the absence of random assignment, we can only estimate potential effects or associations that are suggestive of a possible causal interpretation of the findings, which we undertake employing rigorous econometric methods that adjust for selection. 9
Table 3 shows the characteristics of CPS students by SES provider type in the 2008–2009, 2009–2010, and 2010–2011 school years. A few of the notable differences across provider types include a higher proportion of ELL students served by the district provider, a lower proportion of ELL students served by online providers, and a substantially larger proportion of students with disabilities receiving services from off-site providers. Students with disabilities were also significantly more likely to attend SES with online providers in 2009–2010. These differences in student characteristics across different service provider types were also confirmed in the first stage propensity score matching model (discussed in the following section) that predicted SES attendance with different provider types. 10
Characteristics of Chicago Public Schools Students Served by Different Types of Supplemental Educational Services (SES) Providers in the 2008–2009, 2009–2010, and 2010–2011 School Years (percentages unless otherwise indicated)
Estimation Strategies
In our estimation of the average effects of SES and of attending SES with different types of providers on changes in student test scores, we adjust for student selection in registering for SES and/or their enrollment with particular types of providers (see again Table 3). In addition, we estimate individual, provider-specific effects for SES providers serving at least 30 students in a given school year (and conditional on having data for at least 30 students). SES providers serving fewer than 30 students are combined in a small-provider measure, allowing us to estimate the average effect of smaller providers relative to larger ones. We employ the following four strategies to address possible selection bias.
Value-Added Model
One way that education studies deal with selection is using value-added models. The formal value-added model we employ is specified in Equation 1. The value-added strategy allows us to control for other classroom and school interventions that are fixed over time while identifying provider characteristics. For example, if there is a reading intervention at a school and those students also attend SES, failing to control for the intervention (school fixed effect) would bias the results. The outcome measure is the achievement gain made by a given student, which accounts for the possibility that students with similar characteristics might enter SES with different underlying achievement trajectories (as reflected in their prior test scores). We estimate the following equation,
where Ajst is the achievement of student j attending school s in year t; SESjt is an indicator function if the student j attended SES in year t; Xjt−1 are student characteristics that include student demographics, percentage absent in prior year, retained in prior year, and attended SES in prior year; πs is school fixed effect; μgt are grade by year fixed effects; and Ejst is the random error term. Identification in this specification comes from the average gain in student achievement after controlling for student characteristics and school and grade year effects.
Student Fixed Effects Model
The value-added model assumes that selection depends on observed student characteristics. Hence, controlling for them allows us to deal with self-selection. However, if selection is on some unobserved or unmeasured characteristics of the students, then a value-added strategy could still lead to biased results. The student fixed effects model controls for all time-invariant characteristics of a student, including those that are not observed or measured. The following model of an educational production differs from Equation 1 in that it includes student fixed effects (δj) instead of school fixed effects,
When we take the first difference of Equation 2, we eliminate the student fixed effect (δj), and the model estimates the average difference between the gains made by students attending SES with the gains made by similar students in CPS who were likewise eligible for SES. This formulation imposes some restrictions (or assumptions) that are important to note. First, the impact of students’ prior experience does not deteriorate over time. This implies, for example, that the effect of the quality of kindergarten has the same impact on student achievement no matter the grade. The second assumption is that the unobserved effect of attending SES only affects the level but not the rate of growth in student achievement. A concern with this restriction is that if students with lower growth are more likely to choose to attend SES, then this type of selection may bias the estimates obtained from a gains model.
In order to relax this restriction, the following equation is estimated,
This approach to estimating the fixed effects model controls for any unobserved differences between students that are constant across time. The estimation of this model requires a first difference of Equation 3 and therefore needs three or more observations for each student. 11 As students self-select into the SES program, we deal with this selection by using the gain scores made by same student in the prior year. Identification of the average impact of SES in this model comes from students who participate in one or more but not all years. If these students differ in systematic ways from all students who attend SES, then the estimator gives a “local” effect (specific to students with these characteristics) instead of an average effect. In estimating provider-specific effects, identification comes from students who transfer from one SES provider to another over the period of observation. Therefore, it is important that we check the robustness of the model results using alternative estimation strategies. Table C.1 in Appendix C (in the online journal) shows the differences in characteristics between the students who are used for identification and those who are not in this estimation approach.
School and Student Fixed Effects Model
The base model for this estimation strategy is the combination of the two aforementioned methods. A school fixed effect (π s ) is added to Equation 3, which gives:
adding a school fixed effect controls for unmeasured, time-invariant school quality. For example, in CPS, school administrators have a role in choosing the providers that deliver services on-site at their schools. If principals invite providers that they believe are best suited to their students and school environments, provider effects may be correlated with unobservable school characteristics that might affect student performance. The inclusion of school fixed effects facilitates controlling for time-invariant school characteristics such as average school test scores, neighborhood attributes, parental involvement in the school, and peer composition, to the extent these are unchanging over time. The inclusion of student fixed effects effectively controls for student ability and other time-invariant student characteristics.
As discussed previously, identification of the average impact of SES in this model comes from students who participate in some but not all years, or in the estimation of provider-specific effects, from students who transfer from one SES provider to another, whereas the identification of the school effect comes from students who switch schools. This model is generally preferred over the value added and the student fixed effects models, as it controls for both school and student fixed effects. 12 On the other hand, if the students who switch are different across some time-variant, unobserved characteristics, the results from this strategy could still be biased.
Propensity Score Matching Model
The focus of our analysis using matching methods is the estimation of the differential effects of different types of providers for students participating in SES. We employ propensity score matching (PSM), a two-step process in which the probability of participation in SES (with a particular type of provider) is first estimated based on student characteristics (X), generating predicted probabilities of participation (propensity scores). The matching process is thereby reduced to a one-dimensional problem of comparing students who receive SES from a particular type of provider with students with similar propensity scores who participate with other providers. In other words, if SES participants and comparison group members have the same propensity scores, the distribution of X across these groups will be the same:
and students can be compared on the basis of their propensity scores alone, where D is the treatment of attending SES with a given type of provider.
In applying PSM, we invoke the conditional independence assumption, which implies that after controlling for observable characteristics (X), a student’s treatment status is unrelated to what his outcome would be in the counterfactual state (Rosenbaum & Rubin, 1983). The validity of this assumption depends largely on the set of variables or student characteristics (X) available for the estimation. There may be some unmeasured factors that influence participation with particular types of SES providers; what is important is that participation not be predictive of the outcome that would have occurred with another type of provider. In addition, because our outcome variables are defined as the difference between a pre-program and post-program measure, we use a panel form of the matching estimator (difference-in-difference matching) that allows for time-invariant, unobserved differences between SES participants and comparison students without biasing estimates of program impacts. In estimating this model, we make the assumption that conditional independence holds for the periods both before (t) and after (t1) treatment:
This model estimates the average difference between the gains made by students attending SES with a specific type of provider with the gains made by “matched” students attending with other providers, without putting a functional form on the gain equation (as in the case of student fixed effects). The control variables used in the first-stage matching model are the same as those included in the other modeling strategies (shown in Table 2). The primary PSM matching technique we apply in the second stage model is radius matching, which specifies a “caliper ” or maximum propensity score distance (.01 in our analysis) by which a match can be made. It uses not only the nearest neighbor within each caliper, but all comparison cases within the caliper (based on the specified distance), and the common support condition is imposed to exclude poor matches from the analysis. It is important to reiterate that the sample used in this analysis only includes students who attended SES; thus, the estimates produced are relative comparisons between providers (or types of providers) that show their differential effects on student achievement.
After-matching balancing tests suggested that the matching generally worked effectively; for each of the different estimations (by year and provider type), the covariates were fully balanced after matching, with just two exceptions in 2009–2010. In the estimation of the effects of on-site providers for students attending SES in 2009–2010, English language learners were less likely to attend SES with on-site providers (a small difference of .013 that was not reduced by matching), and in this same year, students with disabilities were significantly more likely to attend with off-site providers (a larger difference of .165 that was likewise not reduced by matching). There were no after-matching balancing concerns (statistically significant differences in covariate means) for 2008–2009 or 2010–2011.
Results of Analyses
Overall Effects of Attending SES
Tables 4 and 5 show the average effects of attending SES (using a dummy variable for any SES attendance) and the number of hours of SES attended, respectively. Irrespective of the variable or estimation strategy used, we find positive and statistically significant results of attending SES on math and reading achievement gains for CPS students. The effect size is very similar using the student fixed effects or school and student fixed effects strategies in reading and math. Table 4 shows that the effect size for reading is approximately .07 to .09 standard deviations, and for math, it is about .05 to .06. These effect sizes are about one-third of the average annual reading and math gains for elementary and middle school students (as reviewed by Hill, Bloom, Black, & Lipsey, 2007). 13
The Average Effect of Attending Supplemental Educational Services (SES; under alternative estimation strategies)
Average Effects of the Number of Hours of Supplemental Educational Services (SES) attended on Student Achievement in Math and Reading
Table 5 reports the effects of the number of hours of SES received on student achievement in math and reading. We find that there is a positive and statistically significant effect of an additional hour of SES on student achievement. Appendix A (in the online journal) shows histograms of the number of hours of SES received by students who attended SES in these 3 school years (2008–2009, 2009–2010, and 2010–2011). These histograms show distinct spikes in the distribution of SES hours attended, typically close to 40 and 60 hours of SES attended, reflecting, in part, that the number of hours attended is a function of the rate the providers charge and the maximum dollars allocated per student by CPS. For the average number of hours of SES received (approximately 40 hours), estimated effect sizes are comparable to those shown in Table 4. In Appendix B (in the online journal), we report findings on SES effects at the 40- and 60-hour attendance thresholds (using PSM methods) only for those students who attended SES (and for brevity, only for school year 2008–2009). In general, these results are consistent with prior studies (Lauer et al., 2004), which suggest effect sizes are larger for programs offering 45 or more hours of tutoring.
In addition, we also estimated generalized propensity score models 14 using data on the total number of hours of SES students attended over the years 2008 to 2011 to assess the cumulative effects of tutoring on students’ math and reading achievement (for those tutored). We are not able to attain balance across student characteristics at all intervals (e.g., quartiles or deciles) of hours of SES attended (likely due to unmeasured selection into different levels of SES attendance), and thus, we view these results as illustrative rather than causal. The graphical display of these results in Appendix B suggests a linear relationship between hours tutored and reading gains through about 55 hours of tutoring, with diminishing returns to additional hours of SES setting in around 60 hours of tutoring. Alternatively, the results for students’ math achievement suggest a steady, positive relationship between hours tutored and math gains through 80-plus hours.
Heterogeneous Effects of SES Providers
A diverse range of providers come and go from the tutoring market. If there are identifiable attributes of the more effective providers and this information is made available to key stakeholders in SES (e.g., state and local educational agencies, students, and parents), it could have the potential of increasing the effectiveness of SES (and tutoring services more generally) over time. We first discuss results that compare the CPS district provider with other (non-district) SES providers. Under federal regulations, school districts that have been identified for improvement are not eligible to provide SES, but CPS was one of a small number of districts granted waivers by the U.S. Department of Education to offer SES. Coming policy changes may allow an increasing number of school districts to engage in direct provision of SES, as well as other flexibility for innovative approaches to service provision.
There are several possible reasons we might expect to observe a different effect size for the district provider (A.I.M. High) compared to other SES providers. First, CPS uses only regular school-day teachers as tutors in its program, with the intent to provide continuity in learning from the regular school day to after-school instruction, as well as to take advantage of teacher knowledge about student needs. That said, if the instruction is less likely to be innovative or just more of the same from the school day, this feature might not benefit students. In addition, we know from our analysis of information on SES provider rates in CPS and other large urban districts that CPS not only charges a relatively low hourly rate as a district provider (approximately $28 per hour in 2009–2010), but it also appears to have influenced rate setting among other (non-district) providers in the Chicago area. We observe the same non-district providers operating in other districts charging as much as twice the rate they charged for an hour of their services in CPS. Because the hourly rate charged directly affects the number of hours of SES students can receive before reaching the district maximum per-student allocation, CPS students are attending more hours of SES (compared to students in other districts). In this context, it is possible that a district provider could contribute to higher hours of SES tutoring received and correspondingly, to greater program effectiveness.
The first analysis includes both SES-eligible students who attended and those who did not attend SES in the sample, so that the estimated effects are for district providers and other (non-district) providers relative to outcomes for eligible students who did not receive SES. (Alternatively, when we restrict our sample to include only students who attended SES, our estimated effects are differential effects between the district and other providers.) The value-added (with school fixed effects), the student fixed effects, and school with student fixed effects results (see Table 6) all show statistically significant effects of CPS district-provided services on students’ math and reading achievement relative to students who do not receive SES. The coefficients are the changes (measured in standard deviations from district average reading and math test scores) in an average student’s outcome that can be expected if the student participates in SES. The estimated coefficients are consistently larger for district versus non-district providers, suggesting that attending SES with a district provider may generate a larger effect on student achievement than attending with a non-district provider. We explicitly test this in additional analyses discussed in the following and presented in Table 11. Although there are a few differences in estimated effect sizes (estimates from the fixed effects estimation approaches are slightly larger), they generally represent about one-third the average annual student gain scores.
The Average Effects of Attending Supplemental Educational Services (SES) With District Provider Versus a Non-District Provider
Tables 7 and 8 present the provider-specific effect sizes for the district provider and other providers who have at least 30 students attending SES with them; smaller providers (serving less than 30 students) are grouped into a single “small provider” indicator. There is fairly strong agreement among the estimates of value-added and both types of fixed effect models results, with a handful of providers standing out as particularly effective in both sets of results and/or across the 3 years (i.e., the district provider A.I.M. High, Newton Learning, Orion’s Mind, School Service Systems, and SES of Illinois). With few exceptions, the provider-specific effect sizes from the school and student effects models are very close to those of the student fixed effects models.
Effects of Attending Supplemental Educational Services (SES) With the District Provider and Other Providers on Reading Gains
Effects of Attending Supplemental Educational Services (SES) With the District Provider and Other Providers on Math Gains
In Appendix B (in the online journal), we report an average effect size of approximately .06 that was statistically significant for SES providers delivering 40 or more hours of tutoring to students. The CPS district provider was getting significantly more hours of SES to the students it served (an average of 48 hours, and nearly twice as many as other providers). Thus, it is not surprising that the effect size of the district provider is approximately twice the size of the average for all providers. 15 Effect sizes for Newton Learning, School Service Systems, and SES of Illinois are similarly large. These providers have different hourly rates, although because we view the hourly rate and number of hours as part of the treatment (as defined or designed by a given provider), we do not include number of hours attended as a covariate.
Tables 9 and 10 report average effects of the district and other provider types on CPS students’ reading and math achievement in 2008–2009, 2009–2010, and 2010–2011, estimated using value-added models, student fixed effects models, and school and student fixed effects models. Table 11 shows the differential effects of different types of providers for those students who attended SES using the three aforementioned estimation techniques and propensity score matching (respectively). Even though the samples differ to some extent (particularly for Table 11, where students attending with different types of providers are compared), the results are fairly consistent across specifications. On-site providers consistently have positive effects on student math and reading achievement (effect sizes of .042–.081), and students attending SES with the district provider generally outperformed other on-site providers (although the difference between district and non-district providers is not statistically significant, as verified by an F test, Prob > F = .152). 16 The results also suggest that online providers are generally less effective than other providers. The difference in the coefficients of online versus on-site are statistically significant at 5% significance level. The sample size for off-site providers was relatively small in 2008–2009 and 2009–2010 (75 and 91, respectively), although more than 2,000 students attended SES with off-site providers in 2010–2011. In 2009–2010, off-site providers served a larger percentage of SWD students, and thus, we suggest interpreting these results with some caution. 17
Effects of Attending Supplemental Educational Services (SES) With the District, Online, On-Site, and Off-Site Providers
Effects of Attending Supplemental Educational Services (SES) With the District, For-Profit, and Not-for-Profit Providers
Differences in Supplemental Educational Services (SES) Effects by Provider Types (for students who attended SES)
The effect sizes of attending SES with for-profit providers versus other nonprofit or district providers (see Table 10) suggest that for-profit providers are generally less effective than district/public providers in increasing student achievement, particularly for math. Students attending with for-profit providers gain about .03 standard deviations less than the district providers in reading and about .07 standard deviations less in math. Table 3 shows that only about 2.5% of students attended SES with a nonprofit provider, and another 10% to 16% attended with the district provider; clearly, the largest share of students attend SES with for-profit providers.
The results in Table 11 show the estimated differential effects between the district and other providers, as well as the effects of other provider types, from the analysis that only includes students who attended SES. As the analysis is done for only those students who attend SES (controlling for their selection into specific provider types in the PSM analyses), a smaller sample size leads to larger standard errors than in the fixed effects strategies and fewer statistically significant results. The differential impacts can also be calculated from the previous tables, and the results are consistent across these different estimation strategies.
The first four rows of results in Table 11 present the average differential effect between the district provider and other SES providers serving CPS students in 2008–2009, 2009–2010, and 2010–2011, estimated using value-added models, student fixed effects models, school and student fixed effects models, and propensity score matching (respectively). The results suggest that on average, students attending SES with the district provider gain more on reading tests (approximately .02–.04 standard deviations more where statistically significant) and more on math tests (approximately .02–.06 standard deviations) than students attending with non-district providers. We also restricted the sample to only students who attended SES with an on-site provider, either the district or another on-site SES provider, to determine if the district provider performance differed from that of other on-site providers. These findings (in the fifth to eighth rows of results in Table 11) suggest that the district provider generally outperformed other on-site providers, with students who attended SES on-site with the district provider realizing larger gains of .04 to .06 standard deviations more (where statistically significant) than other on-site providers.
Results of analyses comparing providers that deliver SES instruction online with other (off-line) providers (see Table 11) suggest that online providers are generally less effective than other providers, although the coefficients (differential effect sizes) are statistically significant only for the 2008–2009 school year and in two specifications. The estimated (negative) differential effects for 2008–2009 are highly comparable between the value-added and PSM models, suggesting students attending with online providers gain approximately .03 to .04 less than students attending SES with other providers. A final set of results comparing effect sizes of attending SES with for-profit providers versus other providers (nonprofit or public) again suggests that for-profit providers are generally less effective than nonprofit and district/public providers in increasing student achievement, particularly for math.
Conclusion
Supplemental educational services (out-of-school tutoring) are a core provision of NCLB, in which school districts are mandated to pay for the cost of provision of after-school tutoring for low income and disadvantaged students who attend schools that are not making adequate yearly progress for 3 years. A key feature of this mandate is its reliance on the private sector to offer eligible students greater choice in a competitive market that is expected to encourage innovative service approaches and squeeze out ineffective providers. Identifying tutoring provider effects on student achievement is essential to generating the information necessary for students and parents to make informed choices of tutoring providers, but efforts to estimate provider effects are complicated by the fact that participation is voluntary. In this article, we have drawn on nonexperimental methods to estimate the effects of SES providers on student achievement in a large urban school district (Chicago Public Schools), which accounts for a significant share of students receiving tutoring under NCLB.
The findings of our empirical analyses of the effects of SES providers who served eligible CPS students in the 2008–2009, 2009–2010, and 2010–2011 school years suggest that there is a statistically significant effect of attending SES on student achievement, particularly for those who receive at least 40 hours of tutoring. These effect sizes represent about one-third of the annual gains made by students in these schools, and the gains from tutoring generally increase with the number of hours of tutoring received.
Additionally, we find that the district provider is more effective than non-district and other on-site SES providers in increasing the math and reading test scores of students who attend SES, although we recognize that this effect may not generalize beyond CPS. In an ongoing qualitative component of our study that involves multiple large urban school districts (Heinrich et al., 2012), we have identified several distinctive features of the implementation of SES in CPS that might explain the district’s greater effectiveness, including the significantly lower hourly rate charged that allows more hours of tutoring for students, the use of regular school day teachers as tutors, and the use of school-based SES coordinators in monitoring and coordinating the delivery of tutoring services. Alternatively, we find that students receiving tutoring from online providers appear to gain less in reading and math than students attending with other providers. We are currently further documenting and analyzing the different types of digital/online providers (vs. nondigital providers), in an effort to identify empirically and qualitatively what contributes to these providers’ lower average effectiveness (e.g., do the online sessions involve interactions with a live tutor, or are they preloaded, self-directed sessions?). 18 This is particularly important given that some urban school districts have seen substantial expansions of online providers’ market shares of students attending SES. 19
Students receiving tutoring from for-profit providers gain less than those attending with nonprofit providers or the district provider, particularly in math. We also identified individual SES providers that were significantly more effective in producing math and reading gains for CPS students. These are a mix of for-profit and nonprofit SES providers, although each of them offers SES on-site (at the public schools that students attend).
Given that unmeasured differences in students who attend SES or attend with particular types of providers could still introduce bias in these results, we are encouraged by the fact that the findings are fairly consistent across the four different rigorous estimation methods that make different assumptions. We also note that our findings on provider-specific effects are consistent with the most recent CPS evaluation of SES as well (Jones, 2009), which identified many of the same providers as being among the most effective in CPS. Finally, we believe that our research has identified some basic characteristics of more effective approaches to the organization and management of SES programs that might be considered by other school districts seeking to improve tutoring outcomes, although we again caution that our findings should be viewed as associations (rather than causal effects).
By design, if NCLB or its successor initiatives (under recently granted waivers to states from NCLB provisions) are to achieve the broader goal of reducing the academic achievement gap through after-school tutoring for students in underperforming schools, tutoring program administrators need adequate and independent (not self-reported) information on provider effectiveness to guide students’ and parents’ choices. We have presented readily adaptable estimation strategies that can be used by states and school districts to generate information on tutoring provider effectiveness for students and parents that will help them to make better informed choices. Whether under NCLB or waivers from its provisions, many school districts across the country will continue to spend millions of Title I funds on tutoring for economically and academically disadvantaged students, and these findings will help to inform those who are looking for guidance in improving these services and their impacts on student achievement.
Finally, the different levels of SES program administration—primarily district and state—could improve their coordination in oversight and monitoring of tutoring provider performance to more fully realize the potential of the competitive market in improving student outcomes. For example, with the evidence from this and related studies, states and districts might introduce performance-based contracts to exert more control over provider rates, minimum levels of tutoring delivered, and other parameters of service delivery that this research suggests could contribute to improved outcomes. Indeed, under newly granted state waivers from NCLB provisions, some school districts are already establishing maximum hourly rates and/or other requirements that will ensure students are offered a minimum of 40 hours of tutoring (Heinrich et al., 2012). In this regard, we expect our study to more broadly speak to the viability of education and other policy interventions that employ a private market model to improve public sector outcomes.
Footnotes
Acknowledgements
We would like to thank the funder of this research, the Institute for Educational Sciences, PR/Award number: R305A090301, Education Policy, Finance and Systems Research Program, Goal 3. In addition, we express our most sincere thanks to the staff in the study district for their time and support for this project. We would also like to thank the staff at the Wisconsin Center for Education Research, UW-Madison for their help and especially Martina Chura, Curtis Jones, Hyun Sik Kim, Huiping Cheng, Patricia Burch, Annalee Good, and Molly Stewart. We also thank anonymous referees for very helpful comments on an earlier version of this manuscript.
Notes
C
H