Abstract
Few school choice evaluations consider students who leave such programs, and fewer still consider the effects of leaving these programs as policy-relevant outcomes. Using a representative sample of students from the citywide voucher program in Milwaukee, Wisconsin, we analyze more than 1,000 students who leave the program during a 4-year period. We show that low-performing voucher students tend to move from the voucher sector into lower performing and less effective public schools than the typical public school student attends, whereas high-performing students transfer to better public schools. In general, transferring students realize substantial achievement gains after moving to the public sector; these results are robust to multiple analytical approaches. This evidence has important implications for school choice policy and research.
S
Despite the attention devoted to student selection into various choice programs, there have been few studies explicitly concerned with students who select out of such programs. Even in an educational environment with several schooling options, some students will undoubtedly elect to remain in, or return to, the traditional public sector for a variety of possible reasons. Earlier studies by Cowen, Fleming, Witte, & Wolf (2012) and Howell (2004) have analyzed the characteristics of individuals who leave voucher programs and return to the traditional public sector, but to our knowledge, there has been no research that systematically considers the academic careers of former choice students after they have enrolled or reenrolled in public schools. 1
It is toward this understanding that we are focused in the current article. Drawing on data from the Milwaukee Parental Choice Program (MPCP) and Milwaukee Public Schools (MPS), this article has two objectives. First, we explore the characteristics of the public schools that students attend when they leave a voucher program and enter the public sector. We devote particular attention to analyzing the achievement levels and effectiveness of the public schools into which former voucher students enroll. Such an analysis can provide valuable insight into the motivations underlying students’ decisions to leave the voucher program as well as information about the factors that affect the specific public school into which a former choice student enrolls.
Perhaps inseparable from the school attendance decisions of students who leave a voucher program and enroll in public schools are their educational outcomes. How does such a transfer affect student achievement? Does the effect vary by the characteristics of the students or the public schools into which they enroll? There is little, if any, existing empirical evidence on such questions. This fact, coupled with competing theoretical considerations regarding the academic effects of intersector transfers, means that the issue must be resolved empirically; this resolution represents the second objective of this article.
Our results indicate that students who leave the voucher program and enroll in MPS are disproportionately disadvantaged relative to both their new public school peers and typical voucher students. After leaving the MPCP, low-achieving students tend to enroll in low-performing, less effective public schools, whereas high-achieving students generally attend higher performing, more effective schools in MPS. However, all students exhibit increased levels of achievement in both reading and mathematics after transferring, and the magnitudes of these increases are not negligible; on average, they are in the range of 0.15 to 0.20 standard deviations. Focusing on the average effect, however, masks the fact that the achievement effects of moving from the MPCP to MPS are somewhat larger for low-performing students than for their higher achieving peers.
These findings have implications for both research and policy. With respect to research, these results—when considered in concert with broader research on the MPCP—are consistent with the presence of heterogeneity in the effects of school choice programs. They also illustrate the importance of explicitly describing the identifying assumptions of a research design when estimating the effect of a choice policy on outcomes of interest. With respect to policy, the results inform issues related to the ability of families to select high-quality schools, the potential stratifying effects of school choice policy, and the relative effectiveness of public and private schools, at least for the population of students examined in this article; these implications are discussed in more detail in the concluding section of this article.
Background
Begun in 1990, the MPCP is the oldest publicly funded private school voucher program in the country. It is also the largest urban voucher program, having grown from about 340 students enrolled in 7 schools in 1990 to more than 20,000 students enrolled in more than 100 schools during the 2010–2011 school year (Wisconsin Department of Public Instruction, 2012). Studies of the program in its early years—between 1990 and 1995—uncovered either insignificant (Witte, 2000) or small positive effects on student achievement (Greene, Peterson, & Du, 1998; Lamarche, 2008; Rouse, 1998); the results are sensitive to various specification choices. Because of statutory provisions, no data were collected on the program between 1996 and 2005, rendering researchers unable to evaluate it during this period. However, as directed by the program’s 2005 reauthorization (2005 Wisconsin Act 125, S.B. 618, 2005-2006 Legislature; 2006), a more recent evaluation occurred between 2006 and 2011, with the results indicating that in three out of four follow-up years, the achievement growth of MPCP students was no different from the achievement growth of a matched group of MPS students (Witte, Carlson, Cowen, Fleming, & Wolf, 2011). However, the official evaluators discerned a small, positive effect of voucher usage on the likelihood of high school graduation and enrollment in 4-year colleges (Cowen, Fleming, Witte, Wolf, & Kisida, in press). During the conduct of this evaluation, it became clear that a consequential number of students were leaving the MPCP and enrolling in MPS (Cowen et al., 2012). Specifically, across a 4-year period—2006–2007 to 2009–2010—approximately one-third of the evaluation sample left the MPCP and enrolled in MPS.
Such a finding raises several policy-relevant issues, with the first-order questions concerning the characteristics of these students. What are their demographic profiles? How were they performing academically prior to leaving the MPCP? What types of schools were these students attending in the MPCP prior to leaving the voucher program? Such questions served as the basis of a paper by Cowen et al. (2012), which reported that low-performing students, African American students, and students attending schools with a high concentration of voucher students were disproportionately likely to leave the MPCP and enroll in the MPS. These findings echo results presented in Howell’s (2004) work, which demonstrates that low-performing students, African American students, and less religious students were disproportionately likely to leave a small, privately funded voucher program in New York City. In addition, Cowen et al. (2012) and Metcalf, West, Legan, Paul, and Boone (2003) have found evidence that students who leave voucher programs were more likely to have considered other sector options earlier in their careers, suggesting that such students are generally more mobile in the first place. Although these studies provide valuable insight into the characteristics of students who leave voucher programs, they provide little information on other important issues, such as the schools that students attend after enrolling in the public sector and their subsequent academic outcomes. Indeed, to our knowledge, there has been no systematic empirical work on these topics. 2
In the absence of any empirical evidence on the public school careers of former voucher students, one may rely on theoretical considerations to guide expectations. In our case, however, theory provides only weak guidance about the academic effects of leaving a voucher program and enrolling in public schools; plausible arguments can be marshaled in support of a wide variety of expectations. This ambiguity exists in part because myriad factors have the potential to affect the academic outcomes of students who transfer from private to public schools.
Any transfer from one school to another will by definition change the environment in which a student is educated. The move from a private school to a public one is no exception. Such a move will result in a new set of teachers and could result in an entirely different curriculum prioritizing different learning objectives. The administrative structure at the destination public school may be quite different from that in the private sector, resulting in a climate where teachers and nonteaching staff alike may be participating in a variety of different leadership roles. These factors—and possibly others—affect the overall quality of a school, which is an important determinant of academic outcomes.
Along with experiencing a change in academic environments, students may experience a change in the characteristics and quality of their peers. In the context of this study, a student who leaves a voucher program and enrolls in a public school will, by definition, move from being surrounded by students whose parents actively chose to enroll them in a private school to being surrounded by students whose parents did not make that choice. Additionally, the demographic, behavioral, and academic characteristics of a student’s peers may change after a move from the private to the public sector. A large body of research has demonstrated that peer characteristics and quality can affect students’ academic outcomes (Figlio, 2007; Hanushek, Kain, Markman, & Rivkin, 2003; Hanushek, Kain, & Rivkin, 2009; Hoxby, 2000; Lefgren, 2004; Zimmer & Toma, 2000).
Sector differences may also affect the academic outcomes of students who leave a voucher program and enroll in the public school system. First, achievement outcomes could be affected by differences in the general effectiveness of the two sectors. If the effects of a voucher program are positive (e.g. Barnard, Frangakis, Hill, & Rubin, 2003; Howell, Peterson, Wolf, & Campbell, 2006; Rouse, 1998), the effects of exiting the program and returning to the public sector may be negative. On the other hand, if voucher impacts are largely neutral (e.g., Metcalf et al., 2003; Witte, 2000) or disproportionately realized by a marginally more advantaged group of students (e.g., Wolf, Kisida, Guttmann, Rizzo, & Eissa, 2011), and if low performers are those most likely to leave in the first place (Cowen et al., 2012; Howell, 2004), then perhaps students who give up a voucher are simply returning to a schooling environment that is simply a better academic fit. There may also be policy differences—both formal and informal—between the two sectors that affect academic outcomes. Perhaps most notably, private schools—unlike public schools—are not subject to the accountability provisions of No Child Left Behind. 3 This disparity has the potential to influence many factors relevant to academic outcomes, a sampling of which may include subject emphasis, alignment between the curriculum and standardized tests (i.e., teaching to the test), and perhaps even retention policy.
Further relevant to the academic effects of intersector transfers is the literature demonstrating that switching schools often has a disruptive effect on academic achievement (Alexander, Entwisle, & Dauber, 1996; Engberg, Gill, Zamarro, & Zimmer, 2012; Hanushek, Kain, & Rivkin, 2004; Ingersoll, Scamman, & Eckerling, 1989; Kerbow, Azcoitia, & Buell, 2003; Lash & Kirkpatrick, 1990, 1994; Rumberger, Larson, Ream, & Palardy, 1999; South, Haynie, & Bose, 2007; Temple & Reynolds, 1999; Xu, Hannaway, & D’Souza, 2009; Zimmer et al., 2009). Regardless of the underlying reason, research routinely finds school transfers to have a negative effect on student achievement, at least initially. There is no reason to suspect that this factor will not be at play in the context of this article.
Finally, an analysis of the academic effects of intersector transfer must recognize the fact that switching schools—to say nothing of switching sectors—is unlikely to be an exogenous event. It is possible that students may be particularly likely to leave the voucher program and enroll in public school after what amounts to an inordinately poor academic year. In such a scenario, any observed increases in post-transfer achievement could be attributable to simple mean reversion. In a related vein, Rumberger et al. (1999) note that although student mobility occurs for any number of reasons, all transfers can usefully be classified as “strategic” or “reactive” in nature; each type of transfer has different implications for academic outcomes. 4
Taken together, the considerations discussed above do not support any absolute predictions about the effects of intersector transfers on academic outcomes. The various factors could interact in ways to produce academic outcomes that are—on average—worse, similar, or better for students who leave the voucher program and enroll in MPS. To the extent our data allow, we explore how the factors discussed above relate to any achievement changes that occur when a student leaves the voucher program and enrolls in public schools.
Data
We use two unique, high-quality data sets to track students in both sectors, collect demographic data on each student and their schools, and link students to their math and reading standardized test scores. The first of these data sets consists of panel data on a sample of MPCP students collected as part of the state-mandated evaluation of the MPCP that began in 2006 (2005 Wisconsin Act 125). The data set contains a representative sample of approximately 2,700 students enrolled in the MPCP during the 2006–2007 school year. This sample consisted of students in Grades 3 through 8, stratified by grade, and the population of ninth graders. It was refreshed with samples of third grade students in the voucher program in both 2007–2008 and 2008–2009, thus totaling nearly 3,500 unique student observations. 5 In addition to student demographics, the data set contains student achievement measures from the reading and mathematics portions of the Wisconsin Knowledge and Concepts Examination (WKCE), which is a criterion-referenced exam administered in Grades 3 through 8 and Grade 10 that the State of Wisconsin uses to comply with federal No Child Left Behind requirements. 6 The data also include an identifier for the census tract in which the students resided in the year that they entered the sample, along with the longitude and latitude of the census tract.
To track former MPCP students in MPS, we examined a second data set, a repeated cross-section containing information on the universe of students enrolled in MPS who took the WKCE in the fall of the 2006–2007, 2007–2008, 2008–2009, 2009–2010, or 2010–2011 academic years. In addition to the WKCE results, the data set contains additional valuable information, including standard student demographics, such as gender, race, grade, free or reduced-price lunch (FRL) status, English language learner status, and special education status. It also identifies the school attended by each student, which allows us to generate school-level characteristics—for all test takers in the school—such as average school achievement in reading and math, the percentage of female students, the racial composition of the school, the percentage of students eligible for FRL, and the percentages of students who are English language learners or receive special education services. 7
Tables 1 and 2 provide information on the grades and demographics, respectively, of students who left the MPCP and enrolled in MPS. For purposes of comparison, Table 2 also presents characteristics for both the population of MPS students and the full MPCP evaluation sample. A plurality of students—more than 31%—left the MPCP and enrolled in MPS in 9th grade. About 12% to 13% of former MPCP students were first observed in MPS in 4th or 10th grade, and fewer than 10% of students were first observed in MPS in each of the remaining grades. In line with the results of Cowen et al. (2012), Table 2 illustrates that students who leave the MPCP and enroll in MPS are—relative to both the population of MPS and the full MPCP evaluation sample—disproportionately Black, low achieving, eligible for FRL, and designated for academic special needs. In sum, the students who leave the MPCP and enroll in MPS are among the most disadvantaged along multiple dimensions. The following section explores the characteristics of the MPS schools into which they enroll.
First Grade Recorded in MPS for Former MPCP Students
Note. MPS = Milwaukee Public Schools; MPCP = Milwaukee Parental Choice Program.
Demographics of MPS and Former MPCP Students
Note. MPS = Milwaukee Public Schools; MPCP = Milwaukee Parental Choice Program; WKCE = Wisconsin Knowledge and Concepts Examination.
Figures are based on 3,398 (for free/reduced-priced lunch), 2,519 (for special needs), and 3,518 (for English language learner) sample sizes, as these classifications are not maintained by all MPCP schools.
Destination Public Schools
With more than 100 elementary schools, 70 middle schools, and 40 high schools, MPS possesses a large and variable set of schools that students could plausibly attend after leaving the voucher program and enrolling in the public sector. This sizable and diverse set of schools, coupled with the fact that MPS policy allows families substantial latitude in selecting the school their child will attend, renders it instructive to examine the characteristics of the MPS schools that former MPCP students attend.
School Characteristics
Table 3 presents average demographic and academic characteristics of the MPS schools that former MPCP students attend in their 1st year in MPS. School characteristics are presented separately for students in Grades 3 through 8 and 9 through 12. In addition, results are presented separately for all students, low-performing students, and high-performing students. 8 Average school characteristics for all other MPS students are presented for purposes of comparison.
School Characteristics for Current MPS Students and Former MPCP Students
Note. MPS = Milwaukee Public Schools; MPCP = Milwaukee Parental Choice Program; WKCE = Wisconsin Knowledge and Concepts Examination; NA = not applicable. All cells are mean proportions except average WKCE scores and average value-added estimates. Average WKCE scores are school-aggregated average scores standardized using the student-level MPS mean and standard deviation. Average value-added estimates are school-level estimates standardized using the school-level MPS mean and standard deviation.
The demographic characteristics presented in Table 3 are straightforward, but the academic characteristics—average WKCE scores and school value added—may require further explanation. Average school achievement levels are calculated using the data set containing the universe of MPS students who took the WKCE between 2006–2007 and 2010–2011. Separately for reading and math, each student’s scale score was standardized using the MPS student-level mean and standard deviation from the proper year and grade. These standardized scores were then aggregated to the school level to obtain each school’s average reading and math achievement levels.
Average achievement levels are an important indicator of school quality, but these levels reflect many factors, including the demographic composition of the school, neighborhood location, and school history. A second measure of quality, one based on school value added across the 5 years of data we observe, may provide a purer indicator of school effectiveness. To obtain school value-added, we estimated the following:
In this model, Y represents the average z score on the reading and math portions of the WKCE for student i in school j at time t. This average achievement is modeled as a function of a vector of lagged reading scores,
Interpretation of the value-added and average achievement measures differ in a subtle but important way. The average achievement measure indicates the extent to which former MPCP students are enrolling into MPS schools with higher- or lower-achieving students. The value-added measures, on the other hand, indicate the extent to which students are enrolling into schools that are more or less effective at increasing student performance.
The results presented in the top panel of Table 3 indicate that—relative to all MPS students in Grades 3 through 8—former MPCP students attend MPS schools that have, on average, lower enrollments and significantly larger proportions of students who are Black and eligible for FRL. The results also indicate that former MPCP students attend schools with lower average achievement than the typical MPS student. Furthermore, the value-added estimates demonstrate that former MPCP students attend schools that are somewhat less effective than the schools attended by the average MPS student. 10 Similar patterns are observed at the high school level, but the differences are smaller in magnitude. 11
As noted earlier, Table 3 disaggregates each comparison for low- and high-achieving students. Within each sector, low-performing students attend schools with lower enrollments and larger proportions of students who are Black and eligible for FRL, relative to their higher-achieving peers. Additionally, relative to high-achieving students, the schools of low-achieving students have lower average achievement and are estimated to be less effective. Across sectors, it is clear that—for both low- and high-achieving students—former MPCP students attend schools that are both lower performing and less effective than those attended by the typical low- or high-performing MPS student, respectively.
Student Choice Sets
To further explore the quality of the MPS schools attended by former MPCP students, we estimate a simple model of school attendance. The first step in this analysis involves constructing the choice set of MPS schools that are theoretical options for each ex-MPCP student to attend in the year they first enroll in MPS. For each student, we define the choice set as all schools that serve the student’s grade and were in operation during the year the student first enrolled in MPS. We recognize that features of MPS policy, to say nothing of practicality, likely results in this defined choice set being an overstatement of a student’s operative choice set. With respect to policy, although MPS provides families with substantial latitude in selecting the school their child will attend, the district does not provide transportation to schools outside of a student’s transportation region. Similarly, whereas the enrollment process for most MPS schools occurs in January, there are approximately two dozen middle and high schools that enroll students under a separate “early admission” program that takes place in the fall. 12 Practically, families are unlikely to seriously consider attending schools located a substantial distance from their residence.
However, two reasons underlie our belief that this definition of the choice set represents the most cautious analytical option. First, any effort to constrain the choice set based on seemingly reasonable decision rules—such as excluding early admission schools or schools beyond a given transportation region—invariably resulted in the exclusion of schools that students attend, which is clearly problematic. 13 Such exclusions are consistent with the fact that although policy and practicality likely serve to limit a student’s operative choice set, they do not formally prevent attendance at any school. Thus, our expansive definition of the choice set is consistent with MPS’s stated policy of allowing parents to send their child to any MPS school with available seats, regardless of whether it is in their neighborhood catchment area. Second, many of the policy- and practicality-induced limitations are related to a student’s distance from a given school, and we are able to account for this factor in the models that follow.
After constructing this choice set for each student, we estimate a (conditional) logistic regression model containing student fixed effects, which account for the student-specific set of schooling options within our broad framework. The model can be generally written as follows:
where S represents schools in an individual’s choice set, i and j index individuals and schools, respectively, A represents a measure of school quality (we estimate models where quality is measured as average achievement levels as well as school value added),
We estimate Equation 2 via maximum likelihood with standard errors clustered at the student level. The model is estimated separately for students in Grades 3 through 8 and 9 through 12. In addition, we estimate the model separately for high-performing students and low-performing students. Results from the estimation of these models are presented in Table 4. For all students in Grades 3 through 8, after accounting for distance, there is no relationship between student attendance and average achievement levels in a school but a small positive relationship between student attendance and school value added; students are no more likely to attend a school with low average achievement than with high average achievement, but they are slightly more likely to attend more effective schools.
Coefficients and Standard Errors From Logit Models Predicting School Attendance Decisions
Note. All models contain student fixed effects. Robust standard errors clustered at the student level in parentheses below coefficients.
p < .1. **p < .05. ***p < .01.
The results for all students mask notable differences in the relationship between student attendance and school quality for low- and high-performing students. Specifically, conditional on distance and given their set of plausible schooling options, high-performing students in Grades 3 through 8 are more likely to attend schools that are both higher performing and more effective. Low-performing students, on the other hand, are more likely to attend schools with lower average achievement, although there is no relationship between student attendance and school value added. At the high school level, there is a positive relationship between student attendance and average achievement levels; this relationship holds for all students, low-performing students, and high-performing students, although the relationship is stronger for high-performing students than for low-performing students. As noted earlier, we are unable to obtain reliable school value-added estimates at the high school level. Finally, consistent with previous studies (Hastings, Kane, & Staiger, 2009; Carlson, Lavery, & Witte, 2011), as well as the issues of policy and practicality described previously, the results indicate that distance plays a large constraining role in school attendance. However, the fact that the second-order term is positive and significant suggests that the constraining role of distance is not absolute, which one may expect given MPS’s stated school attendance policy.
Considered as a whole, the results presented in this section provide a variety of information on a basic descriptive question: What type of public schools do students attend when they leave a voucher program? A second question logically follows: What happens to their performance when they arrive?
Achievement Impacts of Moving From the MPCP to MPS
Addressing this next question involves exploring the relationship between student achievement and transferring from the MPCP to MPS. Before proceeding, however, we caution readers from interpreting this analysis as one specifically designed to provide evidence on the general effectiveness of the public versus private sector. Our achievement results focus uniquely on students who exit private schools and enroll in the public sector, comparing academic outcomes within these individuals over time. 14
In the Background section, we identified several factors that have the potential to affect the academic outcomes of students who transfer from private to public schools. We noted that these factors could interact in different ways to produce academic outcomes that are—on average—worse, similar, or better for students who leave the voucher program and enroll in MPS. Given the competing theoretical considerations, we estimate a set of models to gain empirical insight into the matter. The first we specify as follows:
In this model, i and t index students and time, respectively, A represents a measure of standardized student achievement, MPS is a variable indicating whether student i had transferred to MPS by time t, π represents a vector of grade-by-year fixed effects, γ is a student fixed effect, and ε is an error term. The fact that students are tested only once in high school—in 10th grade—complicates efforts to assess the achievement effects of transferring from the MPCP to MPS during high school. As a result, we estimate Equation 3 using the sample of students who were first observed in MPS in Grades 3 through 8. We estimate separate models for reading and math.
Equation 3 accounts for time-invariant student characteristics, and thus the estimate of MPS should be interpreted as the effect of transferring into MPS on the level of a student’s achievement. If unobserved, time-varying student characteristics are correlated with the decision to transfer, this estimate could be biased (see Bifulco & Ladd, 2007). Consequently, to assess the robustness of our results to alternative modeling approaches, we estimate a second model—a value-added model—that measures the effect of MPS enrollment on a student’s academic growth rather than his or her achievement level:
where Ait−1 is student achievement at time (t – 1) and the remaining contents of the model were described previously. 15 Separate models are again estimated for reading and math. Because Equation 4 contains a measure of lagged achievement, the analytic samples for Equations 3 and 4 are necessarily different. To assess the extent to which any divergent results from the fixed-effects and value-added approaches are the result of different analytic samples—as opposed to resulting from the different modeling techniques—we also estimate Equation 3 using the value-added analytic sample. A final note on these samples is warranted. Although the analytic sample for Equation 3 contains an average of 3.5 observations per student, these observations are split approximately evenly between the pre- and post-transfer periods. Consequently, we have an average of just fewer than 2 post-transfer observations for the average student, meaning we must interpret the results as relatively short-term impacts, particularly given the fall administration of the WKCE. This does not limit our ability to address the question of what happens to students after transferring, but it does imply that we can say little about their longer term career trajectories as they persist in public school. Additional analyses in the following section attempt to address the impact of post-transfer sample size on these results.
Results from the estimation of these models are presented in Table 5. Whether we consider the fixed-effects estimates of Equation 3 or the value-added estimates of Equation 4, these results indicate that former voucher students exhibit significant achievement increases in both reading and math after they transfer to the public schools. Because outcomes are standardized against the MPS mean, we may interpret the coefficients in Table 5 directly as effect sizes. In most cases, the magnitude of these estimates is substantial, comparing favorably to the effects of several well-known interventions, such as class size reduction (Krueger, 1999). Table 5 also disaggregates the results by student achievement level. These specifications indicate that, if anything, low-achieving students benefit from transferring from the MPCP to MPS to a greater extent than high-achieving students; in each subject, and across both estimation strategies, the MPS coefficient estimate for low-achieving students is larger than that for high-achieving students. In one sense, these results may come as somewhat of a surprise, given that the previous section demonstrated that low-performing students attend lower performing, less effective schools than their high-achieving peers. We explore this issue further—and discuss the broader implications of the results presented in Table 5—in the following sections.
Coefficients and Standard Errors From Models Predicting Student Achievement
Note. MPS = Milwaukee Public Schools. Robust standard errors in parentheses below coefficients. All models contain grade-by-year fixed effects.
p < .1. **p < .05. ***p < .01.
Additional Specifications and Possible Explanations
The finding that students exhibit a substantial increase in achievement after transferring from the MPCP to MPS is robust to a variety of specifications and analytical approaches. With such consistency, the next natural set of questions concerns the factors that may be responsible for the observed achievement increases. The Background section identified several factors that have the potential to affect the academic outcomes of students who leave a voucher program and enroll in the public sector; this section is devoted to exploring possible relationships between those factors and the results observed in Table 5.
Regression to the Mean or Single-Year Impacts
Previous studies have demonstrated that students often leave voucher programs after one or more years of poor academic performance (Howell, 2004; Cowen et al., 2012). Such a finding raises the possibility that the positive MPS achievement results may be attributable to simple mean reversion (e.g. Ashenfelter, 1978). If this is the case, the implications of the results presented in Table 5 are far different than if the observed achievement increases were attributable to another, perhaps more fundamental, factor.
We investigated the possibility of mean reversion in several ways. First, following Imberman (2011), we recovered residuals from a model predicting achievement as a function of all variables in Equation 3 save the MPS indicator; the residuals essentially represent student achievement demeaned by student and year-by-grade fixed effects. The mean residuals, which are plotted in Figure 1 along with their 95% confidence intervals, reveal no evidence of a 1-year dip prior to transfer in either subject. To the contrary, there is some evidence that achievement steadily declined in the years prior to transfer but rose in the 1st years after students transferred to MPS, although this rise appears to taper off by the 3rd year post-transfer for the small sample of students we observe that late in the data. Although present across both subjects, this pattern is somewhat more distinct for math than for reading.
Student achievement before and after transfer to Milwaukee Public Schools.
Our second approach replaces the measure of lagged achievement in Equation 4 with a measure of twice-lagged achievement. If students transferred after an especially poor academic year, then twice-lagged achievement may be a better measure of their true underlying ability and thus provides a better estimate of the true effect of transferring from the MPCP to MPS. Table 6 illustrates that the math results are robust to the inclusion of a measure of twice-lagged achievement. In reading, the point estimate on the MPS coefficient remains positive but fails to reach statistical significance. The failure to reach significance is attributable, at least in part, to the fact that estimating a model with twice-lagged achievement substantially reduces the size of the analytic sample.
Results of Regression to the Mean Analyses
Note. MPS = Milwaukee Public Schools. Robust standard errors in parentheses below coefficients. All models contain grade-by-year fixed effects.
p < .1. **p < .05. ***p < .01.
Our final set of specifications informs an exploration of possible mean reversion, but it is also relevant to an analysis of possible heterogeneity in the achievement effects of transferring from the MPCP to MPS. First, we estimated—net of student fixed effects—year-specific coefficients for MPS enrollment for each of the first 2 years after transfer. If mean reversion were responsible for the results in Table 5, then we would expect the coefficients for the 1st-year post-transfer indicator to be larger than the coefficients for the 2nd-year post-transfer indicator. The results—presented in the second panel of Table 6—demonstrate no such pattern. In general, the differences between the estimates for the 2 years are slight and exhibit no systematic pattern. Second, we estimated Equations 3 and 4 using our sample after dropping each student’s first post-transfer observation. The results of this analysis—presented in the bottom panel of Table 6—are positive and significant in the fixed-effects specifications and positive but insignificant in the growth specifications. Similar to the twice-lagged approach described earlier, the growth specification loses a considerable number of student-year observations, which is likely partially responsible for the lack of significance. 16
The results presented in Figure 1 and Table 6 provide a generally consistent substantive story. Prior to transferring to MPS, students experience a multiyear slide in achievement. After enrolling in the public schools, students exhibit a notable increase in their math and reading scores. The achievement growth occurs most intensely in the 1st year post-transfer but appears to continue into the 2nd year as well. Considered as a whole, the evidence indicates that the results presented in Table 5 are not attributable, at least wholly, to a reversion to the mean after an uncharacteristically poor academic year in the MPCP.
Sector Differences
The Background section identified sector differences as a factor that could affect the academic outcomes of students transferring from the MPCP to MPS. Although previous research provides mixed evidence that the two sectors in Milwaukee differ significantly in their general effectiveness (Greene et al., 1998; Lamarche, 2008; Rouse, 1998; Witte, 2000), there are other differences that could be relevant to the observed achievement increases. Perhaps most notable is the fact that during the period our data span, NCLB accountability provisions applied to MPS schools but not to the private MPCP schools. Thus, the stakes for students’ WKCE performance were higher in MPS than they were in the MPCP.
The results presented in the second panel of Table 6, coupled with the fact that the WKCE is administered in November of each school year, provide evidence relevant to the possible influence of the differential test stakes. Specifically, the results demonstrate that students exhibit a substantial increase in achievement during their 1st year in MPS. Given the timing of the WKCE administration, coupled with the evidence on the generally similar effectiveness of the two sectors, it would be surprising if these achievement increases were fully attributable to knowledge increases stemming from MPS enrollment. Seemingly more plausible is a scenario where these 1st-year achievement increases are attributable to MPS placing a heavier emphasis on WKCE performance than on the MPCP. 17 Although the results presented in Tables 5 and 6 are fully consistent with this conjecture, the evidence is clearly not definitive. 18
A second possible difference between the MPCP and MPS concerns student classification and retention policy. Anecdotal evidence from the formal evaluation of the MPCP has indicated that relative to MPS, private schools in Milwaukee are more averse to taking actions that may bring negative attention to a student, such as formally labeling students as requiring special education services or retaining students in grade (Wolf, Witte, & Fleming, 2012). Given this evidence, it is possible that at least some of the achievement gains observed in Table 5 are attributable to greater rates of retention in grade after transferring to MPS. These gains could occur if retention was beneficial in its own right or simply by virtue of having students take the same test for a 2nd consecutive year. To analyze whether the achievement gains are attributable to greater rates of retention, we assess (a) whether students are more likely to be retained after transferring to MPS and (b), if so, whether the results in Table 5 persist after accounting for student retention.
Table 7 reports linear probability estimates of the effect that transferring to MPS has on the probability of student retention. 19 The table includes results from various specifications, but the findings are consistent: Former MPCP students are between 2 and 9 percentage points more likely to be retained after enrolling in MPS, depending on the estimation strategy and the analytic sample. These results indicate that given the same students, retention is a strategy more commonly employed in public schools than in private schools, but they do not appear to explain our results in Table 5; the results are robust to specifications that include an indicator for student retention. 20
Coefficient and Standard Errors for MPS Variable From Linear Probability Models Predicting Student Retention
Note. MPS = Milwaukee Public Schools. Models without student fixed effects contain an MPS indicator, year and grade fixed effects, and measures of special needs, English language learner, and free and reduced-price lunch status. Models containing student fixed effects contain an MPS indicator. Standard errors adjusted for clustering in parentheses below coefficients.
Change in Educational Environment
The Background section noted that any transfer from one school to another is likely to result in changes in the environment in which a student is educated and that such changes have the potential to affect academic outcomes. Assessing changes in the educational environment—and by extension, relating these changes to the observed achievement increases—requires data on (a) the MPCP schools that students attended and (b) the MPS schools into which students enroll. The Data section presented the characteristics of the MPS schools into which students enroll, but our data—and all other data of which we are aware—do not permit calculation of reliable measures of quality or demographic composition of the MPCP schools. 21 Consequently, we are unable to discern how students’ transfers from the MPCP to MPS may have changed their educational environments.
Our inability to assess the extent to which transferring from the MPCP to MPS resulted in a change in school or peer quality highlights the potential benefits of more thorough data collection on student achievement and the demographic composition of private schools, particularly those receiving public funding for their operations. Access to private school data permitting estimation of changes in school and peer quality would inform several issues arising from the analyses in the previous section. Most centrally, it would allow for an assessment of the relationship between changes in school or peer quality and changes in achievement when students transfer from the MPCP to MPS. It would also allow for inquiry into other—perhaps less central but no less important—questions. For example, Table 5 demonstrates that despite enrolling into MPS schools that are lower performing and less effective, low-achieving students exhibit greater achievement gains after enrolling in MPS than their higher achieving peers. A reasonable hypothesis for explaining this empirical finding holds that the low-achieving students attended very low-quality MPCP schools and thus experienced larger changes in school and peer quality than did higher achieving students after transferring to MPS. However, until data exist to empirically test this—or any other—hypothesis, it remains just that.
Disruptive Effects of Switching Schools
As noted previously, a large body of literature has demonstrated that switching schools exerts a negative effect on academic achievement, at least in the years immediately following the switch. In the context of this study, the fact that students’ achievement increased when they left the MPCP and enrolled in MPS does not preclude the possibility that switching schools exerted a disruptive effect on achievement; such effects were apparently overwhelmed by factors—a number of possibilities were discussed earlier—working to increase achievement. Unfortunately, our data do not permit isolation of the specific contribution of the disruptive effects of switching schools on the achievement results presented in Table 5. Attempting to determine the relative achievement-related contributions of the various factors discussed in this section—and perhaps others—represents a natural line of future inquiry.
Discussion and Conclusion
Substantively, the analyses presented earlier tell a compelling, multipart story. After leaving the MPCP, students tend to enroll in MPS schools that are lower performing and less effective than those attended by the typical MPS student. Perhaps not surprisingly, high-achieving transfers enroll into higher quality MPS schools than do their lower achieving peers. In general, all students who transfer to the public sector realize significant achievement gains after doing so, although these gains are estimated to be larger for low-performing students than for their higher achieving peers—a somewhat surprising finding, given the characteristics of the schools attended by these two groups of students.
As made clear in the previous section, the observed achievement increases resulting from a move to MPS from the MPCP could be attributable to any of several factors, including sector differences and changes in school or peer quality. To the extent our data allow, we investigated the possible explanatory power of each of these factors. However, because of data limitations, the specific contributions of these factors—and thus the precise mechanism driving these results—remains within an ill-defined black box, just as it often is on the private or charter side in studies that consider switches to those alternatives. Regardless of the mechanism, however, the findings presented in this article have important implications for research and policy.
With respect to research, the results presented in this article generate a set of analytical implications for school choice research. The striking feature of these results is not their indication that students who exit the voucher program are low performers—such a pattern is demonstrated elsewhere by Cowen et al. (2012) and Howell (2004)—but that switchers exhibit achievement increases after arriving in the public sector. Given other results demonstrating little difference in the performance of a representative sample of voucher students and a matched panel of MPS students across 5 years (Witte, Carlson, Cowen, Fleming, & Wolf, 2011), this implies that a substantial degree of treatment heterogeneity may exist within the MPCP and choice programs more generally. Such heterogeneity implies that even unbiased estimates of average treatment effects of school choice programs may not provide the entire set of policy-relevant information. For example, estimates generated from designs leveraging student transfers—such as fixed-effects approaches—may be smaller than the effects for a broader population of students. Similarly, estimates based on students who remain in their respective sectors for a fixed period—as may be the case in matching designs—seem likely to be larger than what would be observed in expectation across the population of potential participants. Such a reality suggests that evaluators would do well to explicitly identify their estimand and discuss the population to which their findings may be generalizable. It also illustrates the benefits of attempting to empirically assess the extent of treatment heterogeneity.
Along with these research implications, the findings presented in this article also have implications for policy. First, they provide evidence—albeit equivocal—on the ability of students to attend high-quality schools that result in improved academic outcomes. On one hand, our results demonstrate that former MPCP students attend MPS schools that are lower performing and less effective than the schools attended by the typical MPS student. On the other hand, our results also illustrate that given the set of schools that a student could theoretically attend, and accounting for the constraint of distance, there was a slight positive relationship between school effectiveness and student attendance; this relationship was strongly positive for high achieving students and nonexistent for low-achieving students. Furthermore, students exhibit significant achievement gains after transfer to the public sector, and these gains are larger for low-achieving students than for their higher achieving peers. Together, these results raise several questions. Are low-achieving students leaving particularly ineffective MPCP schools and—even though they attend MPS schools that are on average less effective—experiencing an increase in school quality after transferring to MPS? Could the observed achievement increases for low-achieving students be even larger if they attended a higher-quality MPS school? Do MPS schools vary in their effectiveness for low- and high-performing students? Addressing such questions is either beyond the scope of this article or not possible given our data but would be a natural topic for future research.
Concerns have been voiced that granting families the ability to choose their schools will result in greater stratification along academic and demographic dimensions. The results presented in this article demonstrate that such concerns may be warranted: Low-achieving students attend MPS schools that are, by accepted measures, of lower quality than the MPS schools attended by their higher achieving peers. At the same time, however, low-achieving students exhibit greater achievement gains than do transfers who exhibit better performance, a fact that complicates a normative interpretation of any potential stratification.
Finally, our results can inform debates on public and private school effectiveness by suggesting that the answers to such questions may be different for different students. This may be particularly true in environments where several schooling options are available to most students. Throughout this article, we have noted that the official evaluation of the MPCP found very little difference in the performance of representative voucher students and a matched panel of MPS students across 5 years, suggesting that on average, these sectors are doing a comparable job of educating typical students. However, the achievement gains observed for students who leave the MPCP and enroll in MPS imply that there exists a class of students who performed poorly in the voucher sector and who are subsequently served at least as well, if not better, by the traditional public sector. Few school choice supporters would suggest that private or charter schools are for everyone, and in the context of high between-sector mobility that we and other scholars have observed, it would be troubling if the average student who exited a choice program suffered, or continued to suffer, achievement losses after the transfer. Our results indicate that such a negative impact is not apparent, on average. This would be consistent with the more optimistic interpretations of a mature high-choice environment: Students sort between schooling options until they settle on the option—public or private—that is best suited to their needs.
Footnotes
Acknowledgements
The authors are grateful to Patrick Wolf and John Witte for access to these data and for collaboration on the larger project from which these data are drawn. The authors would also like to thank three anonymous reviewers whose comments and suggestions resulted in an improved article. All errors are the sole responsibility of the authors listed above.
Notes
Authors
DEVEN CARLSON is an Assistant Professor of Political Science at the University of Oklahoma. His research interests include education policy, social policy, and policy analysis.
JOSHUA M. COWEN is an Assistant Professor in the Martin School of Public Policy and Administra-tion at the University of Kentucky; 433 POT Lexington, KY 40506-0027;
DAVID J. FLEMING is an Assistant Professor in the Political Science Department at Furman University. His research examines the effect of parenthood and education policy on parents’ views of education, social capital, and political orientations.