On the Threshold: Impacts of Barely Passing High-School Exit Exams on Post-Secondary Enrollment and Completion

Abstract

Many states use high-school exit examinations to assess students’ career and college readiness in core subjects. We find meaningful consequences of barely passing the mathematics examination in Massachusetts, as opposed to just failing it. However, these impacts operate at different educational attainment margins for low-income and higher-income students. As in previous work, we find that barely passing increases the probability of graduating from high school for low-income (particularly urban low-income) students, but not for higher-income students. However, this pattern is reversed for 4-year college graduation. For higher-income students only, just passing the examination increases the probability of completing a 4-year college degree by 2.1 percentage points, a sizable effect given that only 13% of these students near the cutoff graduate.

Keywords

educational policy educational reform accountability high stakes testing quasi-experimental analysis regression discontinuity

Ensuring that high school graduates have the skills to succeed in college or careers has been a central focus of educational policy over the last two decades. States have pursued a variety of policy instruments to achieve this goal. In 13 states, including Florida, New York, Texas, and Virginia, all students must pass exit examinations in core subject areas (typically mathematics and English language arts) to earn a high school diploma. Massachusetts, the site of this study, has had such exams in place since the early 2000s, affecting 70,000 students in each year’s graduating cohort. In recent years, several states—including California—have moved away from these policies because of fears that they provide barriers to graduation. Understanding the consequences of these examinations—both positive and negative—is critical for decisions about whether to continue these policies and, more importantly, for designing policy responses that mitigate negative impacts.

We study one equity consequence of these examinations. By design, states must define a level of proficiency that qualifies students for a high school diploma and specify a cut score on the test that represents this level. As such, these exit examinations necessarily assign students with essentially equal skills to either pass or fail based on whether their scores fall just above or below the minimum passing score. This potential threat to within-group equity—treating essentially similar students differently—means that students may have different outcomes based on their position relative to the cutoff. We leverage this design feature to draw causal claims about the impact of barely passing or failing the examination on student outcomes.

Our past work in this same context has shown that barely failing the examination substantially reduces the probability of high-school graduation for urban low-income students but not for higher-income students. Here, we extend these analyses, using data from five successive cohorts of students to estimate impacts on college graduation. This analysis reveals a nuanced story: while we continue to find impacts on high school graduation for low-income students but not their higher-income peers, we find the opposite pattern for college graduation. Barely passing the 10th grade mathematics exam increases the probability of 4-year college completion by about 20% (2.2 percentage points) for higher-income students scoring near the cutoff, but has no impact on this outcome for low-income students.

These effects are economically meaningful because of the large financial returns to 4-year college degrees. Among young adults, 4-year college degree holders now command annual earnings that are 62% higher, on average, than those of terminal high school graduates (National Center for Education Statistics, 2019). Taken together, our results suggest that there are meaningful consequences of barely passing the examination, although on different educational attainment margins, for both low-income and higher-income students.

Background and Context

Advocates see three types of benefits from exit examinations. First, exams should increase motivation for students to work hard in school and build human capital. Second, exams should increase motivation for educators to develop the skills and knowledge of all students. Third, an exit-exam requirement should signal to employers that high school graduates have certain baseline skills, thereby increasing the economic value of a diploma (Betts, 1998). Therefore, the underlying theory of change is that an exit-exam requirement should improve students’ educational attainments and labor market opportunities (Evers & Walberg, 2002).

Critics object that exit exams put unnecessary stress on students and violate two dimensions of educational equity: equity across groups and equity within groups. The negative impact on equity across groups is that students living in poverty, those who have learning disabilities, or those who are English learners have greater difficulty in passing the exit exams than other students (Kornhaber & Orfield, 2001). The potential violation of equity within groups is that by specifying minimum passing score cutoffs, exit exams inevitably treat differently similar students with essentially equal proficiency whose scores fall just on either side of the cutoff. Students who fail by a point or two face greater hurdles to graduation than students who just pass and may be discouraged by their failing score. They may also be subject to local policy responses, such as being funneled into remedial courses instead of continuing into college preparatory courses, perhaps with longer-term impacts (Holme, 2008; Holme et al., 2010; Sipple et al., 2004).

To be clear, we do not explore the overall impact of exit examinations—which may be positive for many students—or the across-group equity concerns in this paper. There is a long literature on this topic, using variation within a state over time or across states (see Holme et al., 2010, for a detailed review; Dee & Jacob, 2006; Kyoore, 2019; Reardon et al., 2010; Warren et al., 2008).

Instead, we focus on the consequences of exit exams for equity within groups. Researchers have explored within-group equity consequences by applying regression-discontinuity designs to compare the outcomes of students on either side of the passing cutoff, who are essentially equivalent on their underlying academic skills. The state’s exogenously assigned cutoff creates a natural experiment by dividing these otherwise similar students into two groups: those who just passed the exam, thereby satisfying the state’s graduation requirement, and those who barely failed it. Any later differences in outcomes for students in these categories reflect the causal impact of barely passing for students at the margin. These impacts are unintended consequences of the exit exam, because these students are actually of similar ability.

The results from the extant research on within-group equity have been mixed. Using data from Texas, Martorell (2005) found no effects of barely passing the 10th grade exit exam on the probability of graduating from high school. Reardon et al. (2010) reported broadly similar results in California. In contrast, other research has found consequences of barely passing an exam. Polson (2018) found that students who just passed an exit exam in Texas responded by taking fewer courses in their senior year of high school and committing fewer disciplinary infractions than those who just failed. Ou (2010) found that barely passing an exit exam increased the probability of graduating from high school for students in New Jersey. In previous work in Massachusetts, we found that just passing the exam increased the probability of on-time high-school graduation by 8 percentage points for low-income urban students in the class of 2006, although not for other groups of students in that cohort (Papay et al., 2010). We also found that just passing an exit exam increased the probability of enrolling in college by 3 to 5 percentage points (Papay et al., 2014).

We know of no studies that have estimated impacts of barely failing an exit examination on longer-run educational attainments, such as graduation from a 4-year college. One relevant study is by Clark and Martorell (2014), who focused on the last-chance examination that students who had failed earlier attempts took in 12th grade. They found that failing this examination reduces the probability of high-school graduation and college enrollment, but not college completion or earnings. However, they focus on students who did not pass on earlier attempts and persisted in school—an important group but not all students affected by these policies. The broader effects of high-stakes testing on longer-term outcomes are crucial for educational policymakers to understand.

Massachusetts Context

Beginning with the high-school graduating class of 2003, the 10th grade Massachusetts Comprehensive Assessment System (MCAS) mathematics and English Language Arts tests have functioned as high-school exit exams. Students must pass both tests to receive a high school diploma. The state attempts to make the examination as minimal a barrier to graduation as possible: it allows students to take the tests without time constraints, to retake them repeatedly if they fail, and to appeal the decision in several ways.¹ Students who satisfy local graduation requirements but do not pass the test can earn a Certificate of Attainment.

While students must pass both tests to graduate, the passing thresholds for math and ELA are located in different places in the distribution of scores. Among the 2003 to 2007 test-takers studied here, 7.7% of students failed the ELA test on their first attempt while almost 13% failed the math exam. Because our prior work found impacts in math but not ELA, and because so many more students fail the math exam, we focus our presentation here on math.² Given that students face both constraints, we do explore whether the effect of failing the math exam depends on students’ performance in ELA. Most students who fail go on to retake the test, and most who retake it do indeed pass. Because the retake decision is endogenous, we focus our analysis on each student’s first attempt at the exit examinations.

Methods

Dataset and Sample

The Massachusetts Department of Elementary and Secondary Education (DESE) has compiled a comprehensive database that follows students longitudinally through high school. This database includes MCAS scores, information on student demographics and school attendance, and high-school graduation dates. The state also collects information on college enrollment and graduation from the National Student Clearinghouse (NSC). These NSC data include students from nearly all colleges and universities (public and private, 2 year, and 4 year) in the United States. Student records are merged by the NSC using names and dates of birth. The match rate in Massachusetts approaches 95% in recent years (Dynarski et al., 2015).

In our analytic sample, we include the 345,936 students who first took the 10th grade MCAS examinations as sophomores in 2003 through 2007, approximately 70,000 students each year. We retain in our sample students who took the test and subsequently dropped out or transferred out of the state’s public school system. We have complete NSC records on these students for 9 years after they took the MCAS.

In Table 1, we present descriptive statistics and outcome means for four groups: the full sample of first-time 10th grade test-takers, the subsample of students who scored within two raw-score points of the passing threshold on the math exam (about 7% of students), and this same subsample disaggregated by family income (which we define based on whether students ever qualified for free or reduced-price lunch). Students near the cutoff differ from the full sample in important ways. Over half of this group is low-income, and the proportions of Black students, Hispanic students, students with disabilities, and students attending urban schools are higher than in the sample as a whole. Students near the cutoff graduate from high school and go on to college at substantially lower rates than the average test-taker, with only 13% enrolling in a 4-year college or university within 4 years of the MCAS and only 9% graduating within 9 years.

Table 1

Sample Proportions of All First-Time 10th Grade Test-Takers and Those Within Two Raw-Score Points of the Passing Threshold on the 10th Grade Mathematics Exit Examination for Key Outcomes and Demographic Indicators, 2003 to 2007

Variable	All students	Students within 2 raw-score points of the passing cutoff
Variable	n = 345,936	Alln = 25,284	Low-incomen = 14,494	Higher-incomen = 10,790
Outcomes
Graduated high school (MCAS+3)	0.87	0.75	0.70	0.81
Enrolled in any college (MCAS+4)	0.69	0.43	0.38	0.50
Enrolled first in 2-year college (MCAS+4)	0.20	0.30	0.28	0.33
Enrolled first in 4-year college (MCAS+4)	0.49	0.13	0.10	0.17
Graduated from 4-year college (MCAS+9)	0.39	0.09	0.06	0.13
Demographics
Asian	0.05	0.03	0.04	0.01
African-American	0.08	0.16	0.23	0.06
Hispanic	0.10	0.19	0.30	0.04
White	0.77	0.62	0.42	0.88
English language learner	0.04	0.07	0.12	0.01
Students with disabilities	0.14	0.28	0.25	0.33
Female	0.50	0.51	0.52	0.49
Urban	0.26	0.42	0.58	0.19
Low-income	0.34	0.57	1.00	0.00
MCAS performance
Passed both	0.85	0.53	0.48	0.60
Passed math but failed ELA	0.02	0.09	0.11	0.06
Passed ELA but failed math	0.07	0.29	0.29	0.30
Failed both	0.05	0.08	0.10	0.04

Notes. MCAS = Massachusetts Comprehensive Assessment System 10th grade mathematics test; (MCAS+3) indicates that the outcome was measured 3 years after students took the test for the first time; ELA = English Language Arts.

We note stark differences between the low-income and higher-income students near the cutoff, as shown in the last two columns of Table 1. While their MCAS scores imply that these students possess similar academic skills, the low-income group has consistently lower levels of average later educational attainment. Black and Hispanic students, and those attending urban schools, comprise the majority of this group, while the higher-income group is almost 90% White with fewer than 20% attending urban schools.

Measures

Our primary outcomes are students’ educational attainments. We created a set of dichotomous outcome variables that indicate whether the student graduated from a public high school in Massachusetts (HSGRAD), enrolled in a 4-year college or university (COLL), and graduated from a 4-year college or university (COLLGRAD). Each outcome is time-delimited based on the year the student first took the MCAS math exam. For HSGRAD, a student is coded as 1 if they graduated from a Massachusetts public high school within 3 years of taking the MCAS in the spring of their 10th grade year (akin to a 5-year graduation rate). The time horizons for COLL and COLLGRAD are within 4 and 9 years of the MCAS, respectively.³ Our high-school graduation outcome defines students who transfer to a private high school or out-of-state school as non-graduates, as we are unable to track them further in our data, while our college outcomes include students regardless of where they completed high school.

The key predictors for our regression-discontinuity approach derive from student performance on the MCAS math exam. We centered students’ raw scores⁴ (MATH^C) such that a student with the minimum passing score that year has a centered score of 0.⁵ We also created a dichotomous version of this predictor, PASS_MATH, to indicate on which side of the pass–fail cutoff the student’s score lay.

We categorize students as low-income or higher-income based on whether they had ever been eligible for free or reduced-price lunch as Massachusetts public-school students.⁶ In 2005, for example, the maximum annual income for reduced-price lunch eligibility for a family with two adults and two children was $36,641 (this is equivalent to approximately $48,000 in 2019 dollars). The dataset also contains information about student demographic characteristics, including race/ethnicity and gender, as well as indicators for whether the student was an English learner or special education student, and whether the student attended a high school in one of the state’s 22 urban school districts or 21 urban charter high schools.

Analytic Approach

We follow the approach described in Papay et al. (2016) to estimate the causal impact of just passing the mathematics exit exam on the first attempt. By examining students immediately on either side of the cut score on the forcing variable, the 10th grade mathematics exam score ( $M A T H_{i}^{C}$ ), we compare outcomes for two groups of students—those who scored at the cut score and passed and those (hypothetical) students who scored at the cut score yet failed. For illustration, we focus on the population probability of graduating from college within 9 years of taking the test (COLLGRAD_i):

γ_{a b o v e} = \lim_{M A T H_{i}^{C} \to 0^{+}} [P (C O L L G R A D_{i} = 1) | M A T H_{i}^{C}]

γ_{b e l o w} = \lim_{M A T H_{i}^{C} \to 0^{-}} [P (C O L L G R A D_{i} = 1) | M A T H_{i}^{C}]

If the cut score is established exogenously, then students just on either side of the cut score must be equal in expectation, and the estimated difference between these parameters provides an unbiased estimate of the average causal impact of classifying students as passing (as opposed to failing) for students at the cut score, in the population (Lee & Lemieux, 2010; Murnane & Willett, 2011). Because the classifications are applied rigidly such that all students who score below the cut-off on the forcing variable fail and all students who score above the cut-off pass, the discontinuity is sharp.

Regression-discontinuity designs rest on a key assumption: the exogeneity of the cut score. We perform the usual checks of this assumption and include the details in the Appendix. As in Papay et al. (2010), we inspect the histogram of the raw scores on the mathematics exam for evidence of a discontinuity at the passing threshold. We also perform density tests described in McCrary (2008) and Frandsen (2017), and find no evidence of manipulation on the forcing variable. Finally, we check for balance of key covariates around the passing cutoff and find no significant differences in the average characteristics of students who just passed and those who just failed.

We estimate local linear regression models of the following form, using COLLGRAD_i as the dependent variable in this example:

\begin{array}{l} p (C O L L G R A D_{i} = 1) = β_{0} + β_{1} M A T H_{i}^{C} \\ + β_{2} P A S S_M A T H_{i} + β_{3} M A T H_{i}^{C} \\ \times P A S S_M A T H_{i} + γ^{'} X_{i} + δ_{t} + ε_{i} \end{array}

for student i. The causal effect of interest is $β_{2}$ , which represents the difference in the probabilities of graduating from a 4-year college for students who just passed the MCAS and otherwise similar students who just failed. If its estimated value is statistically significant and positive, we can conclude that just passing the exam causes the student’s probability of graduating from college to increase discontinuously, on average, in the population. We include the covariates described above (X_i), as well as the fixed effect of cohort ( $δ_{t}$ ), to improve precision, although we find quite similar results without controls.

We fit our models within an optimal bandwidth, h*, which we calculate using the cross-validation procedure described by Imbens and Lemieux (2008).⁷ The optimal bandwidths are 3 raw test score points for high-school graduation and 2 points for all college outcomes. More recent approaches to bandwidth selection (e.g., Calonico et al., 2014; Imbens & Kalyanaraman, 2012) assume independent and identically distributed data and are therefore not optimal for use with discrete running variables, as we have here. However, we assess the sensitivity of our results to choice of bandwidth and present the results in a later section. Following Lee and Card (2008), we cluster our standard errors on MCAS score point to account for our discrete forcing variable.

As noted above, our prior work revealed that the causal impact of just passing the math MCAS exam on the probability of high-school graduation is different for low-income students than for their higher-income peers. We extend this work to investigate whether the effects on college enrollment and graduation differ by family income as well. We do this by fitting models with the full set of interactions between the predictors MATH^C, PASS_MATH, and LOWINC, as follows:

\begin{array}{l} p (C O L L G R A D_{i} = 1) = α_{0} + α_{1} M A T H_{i}^{C} \\ + α_{2} P A S S_M A T H_{i} \\ + α_{3} M A T H_{i}^{C} x P A S S_M A T H_{i} \\ + α_{4} L O W I N C_{i} + \end{array}

α_{5} M A T H_{i}^{C} x L O W I N C_{i} + α_{6} P A S S_M A T H_{i} x L O W I N C_{i} +

α_{7} M A T H_{i}^{C} x P A S S_M A T H_{i} x L O W I N C_{i} + π^{'} X_{i} + δ_{t} + ε_{i}

Here, $α_{2}$ represents the causal effect of passing for higher-income students, while the linear combination $α_{2}$ + $α_{6}$ represents the impact for low-income students.

Findings

High-School and College Graduation

Consistent with our prior work, we find a positive impact of just passing the math exam on the first attempt on the probability of high-school graduation. The effect is about two percentage points in the full sample of test-takers, but, as the results in the second column of Table 2 show, the effect is concentrated among low-income students. As in Papay et al. (2010), there appears to be no difference between the high-school graduation rates of higher-income students just on the two sides of the passing threshold. However, for students from low-income families, the impact of just passing the mathematics exit exam on the probability of graduating from high school is 3.3 percentage points. This is a meaningful but relatively modest effect, given the graduation rate of about 70% for low-income students near the cutoff. We also replicate our prior finding of particularly strong high-school graduation impacts for low-income urban students (approximately five percentage points).⁸

Table 2

Estimated Causal Effects of Passing the 10th Grade Exit Examination in Mathematics, as Opposed to Failing It, on the Probability of Selected High-School and College Outcomes for Students at the Margin of Passing, for All Students and by Family Income

Group	High-school graduation	Any college enrollment	Four-year college enrollment	Four-year college graduation
	(MCAS+3)	(MCAS+4)	(MCAS+4)	(MCAS+9)
	h* = 3	h* = 2	h* = 2	h* = 2
All students	0.0194**	0.0244**	0.0173**	0.0094**
All students	(0.0028)	(0.0011)	(0.0006)	(0.0018)
Low-income	0.0329**	0.027*	0.0008	0.0002
Low-income	(0.0067)	(0.0065)	(0.0015)	(0.0014)
Higher-income	−0.0017	0.0208*	0.0408**	0.0207*
Higher-income	(0.0043)	(0.0066)	(0.0036)	(0.0064)
N	35,304	25,284	25,284	25,284

Notes. MCAS is the Massachusetts Comprehensive Assessment System 10th grade mathematics test; (MCAS+3) indicates that the outcome was measured 3 years after students took the test for the first time. Standard errors clustered on raw-score point are in parentheses; h* indicates the optimal bandwidth used in the regression-discontinuity model for each outcome.

p < .01. *p < .05. +p < .1.

We find quite different patterns for 4-year college graduation. As shown in the far-right column of Table 2, just passing the math exam on the first attempt increases the probability of graduating from a 4-year college by approximately one percentage point (or 11%). However, this effect is concentrated among higher-income students. Among students from more advantaged families, just passing the math exit exam increases the probability of 4-year college graduation by two percentage points. Given that the graduation rate among these higher-income students near the cutoff is only about 13%, this is quite a substantial effect.⁹

Importantly, we find confirmatory visual evidence of these effects. In Figure 1, we plot the relationship between math exit exam scores and both the probabilities of high-school graduation (Panel A) and 4-year college graduation (Panel B). For students near the cutoff, passing the examination on the first attempt substantially increases the probability that low-income students graduate from high school and the probability that higher-income students graduate from college. Visually, these effects appear as a discontinuity in the relationship between the outcome and the math score at the cutoff.

Figure 1.

Sample mean probabilities of high-school graduation within 3 years (top panel) and graduation from a 4-year college within 9 years (bottom panel) at raw-score points near the passing threshold on the mathematics high-school exit examination, by family income.

We observe no discontinuities at the cutoff in the other plots in Figure 1. The evidence from the plots and from the fitted local linear probability models is consistent: The impact of just passing the math exit exam operates at different margins of educational attainment. The findings of our earlier work suggested that family advantage effectively insulated higher-income students from the effects of just failing the MCAS on their first attempt, as there was no evidence of an impact on their probability of high-school graduation. But tracking them further through the educational pipeline reveals that passing the math exam does impact their educational attainments, increasing their probability of graduating from a 4-year college or university.

This positive impact on the probability of 4-year college graduation could derive from 2 sources: a higher rate of 4-year college enrollment and a higher rate of persistence in college among students who do enroll. As seen in the third column of Table 2, for higher-income students, just passing the exit exam on the first attempt increases the probability of enrolling in a 4-year college by more than four percentage points. We find no evidence that just passing the exit exam affects the probability of persistence to graduation from a 4-year college for students from higher-income families.¹⁰ Thus, we explore in more detail the differential impacts on 4-year college enrollment.

College Enrollment

The literature on the determinants of college enrollment is substantial and highlights key gaps in access between low-income and higher-income students. In our data from Massachusetts, low-income students are indeed less likely to attend college than higher-income students, even for students with the same MCAS scores (Papay et al., 2020). We see similar patterns for students scoring near the passing cutoffs (see Table 1).

Overall, only 10% of low-income students near the cutoff initially enroll in a 4-year college or university, due at least in part to the serious financial barriers to college matriculation. Moreover, relatively few individuals near the passing cutoff successfully completed the pathway from a 2-year college to a Bachelor’s degree within the period of observation. Of the 4012 low-income students within two score points of the math cutoff who initially enrolled in a 2-year college, only 264 (6.5%) graduated from a 4-year college within 9 years after taking the grade-10 MCAS exams.¹¹ Thus, just passing the exit exam on the first attempt exerts a somewhat different impact on college-going for low-income students than for higher-income students. For the former group, it increases the probability that they enroll in a 2-year college, while for the latter it induces them to enroll in a 4-year college. We will examine the pathways taken by Massachusetts students who enroll in 2-year colleges in more detail in future work.

Interestingly, there is no income-based gap in students’ college-going plans at the time they take the MCAS. To measure students’ expectations, we use responses to a question from a survey administered at the beginning of the 10th grade MCAS testing session each year. This multiple-choice item asks students about what they plan to do when they finish high school. The response choices include: attend a 4-year college, attend a community college or technical school, join the military, work/other, and “I have no plans right now.”

Table 3 includes the distribution of responses for all students and for those within two raw-score points of the mathematics passing threshold, disaggregated by family income. While there are striking differences in post-secondary plans between the low- and higher-income groups overall, low-income and higher-income students near the cutoff do not differ, on average, in their plans at the time they take the MCAS. This finding is consistent with national data showing a narrowing “expectations gap” by family income (Jacob & Linkow, 2011).

Table 3

Post-Graduate Plans for All Students and Those Within Two Raw-Score Points of the Passing Threshold on the 10th Grade Mathematics Exit Examination, 2003 to 2007, by Family Income and Passing Status

Post-graduate plans	All students		Students within 2 raw-score points of the passing cutoff
Post-graduate plans	Low-incomen = 118,662	Higher-incomen = 227,274	Low-incomen = 14,494	Higher-incomen = 10,790
Four-year college	0.46	0.64	0.37	0.39
Community college	0.11	0.07	0.15	0.16
Military	0.04	0.03	0.05	0.06
Other	0.11	0.06	0.14	0.15
No plans/don’t know	0.09	0.06	0.10	0.11
No response	0.19	0.14	0.20	0.14

Notes. Responses taken from student questionnaire administered as part of the Massachusetts Comprehensive Assessment System (MCAS) exams. Cell entries are the proportions of respondents selecting each answer choice.

While these groups of students do not differ in their expectations, they do differ in the subsequent realization of those plans. Higher-income students who expect to attend a 4-year college are more likely to enroll than are low-income students with the same plans and test scores. In results reported in Table 2, we find similar impacts of barely passing the exit exam by income on enrollment in any college, but dramatic differences in the type of college students attend. Just passing the exit exam increases the probability of enrolling in a 2-year, but not a 4-year, college for low-income students.

This difference might reflect a purely mechanical effect, because public 4-year institutions in Massachusetts require a high-school diploma to matriculate. If the exit-exam requirement is preventing some low-income students from graduating from high school, then they would not be eligible to enter most 4-year institutions unless they earned a high-school equivalency credential. However, among students who score just below the passing cutoff on their first attempt, nearly all retake the test at least once, as shown in Table 4. While low-income students pass their first retest at a lower rate than their higher-income peers, most go on to eventually pass (82% of low-income students and 89% of higher-income students). Among the over 8,600 low-income students who barely failed on their first attempt, only 223 failed to graduate but did earn a certificate of attainment, meaning they had completed all local graduation requirements but never passed the exit exam.

Table 4

Retest Behavior and Success for All Students Who Failed and Those Within Three Raw-Score Points of the Passing Threshold on the 10th Grade Mathematics Exit Examination, 2003–2007, by Family Income

Retest status	All students who failed	Students who failed but scored within 3 raw-score points of the passing cutoff
Retest status	n = 43,450	Alln = 13,968	Low-incomen = 8,621	Higher-incomen = 5,347
Ever retested	0.89	0.93	0.92	0.94
Passed on first retest	0.34	0.52	0.47	0.59
Ever passed on retest	0.70	0.84	0.82	0.89

Note. Cell entries are the sample proportions of students within each category.

This evidence indicates that the differential impact we observe on college-going is not purely a mechanical effect, in which low-income students who otherwise would have graduated from high school and enrolled in college are prevented from doing so because they cannot pass the test. Rather, the exit-exam requirement appears to induce somewhat different responses among students near the passing cut-off, depending on their family income. We address the question of whether these responses reflect encouragement or discouragement effects in the discussion and conclusion.

Sensitivity Analyses

Choice of Bandwidth

We test the sensitivity of our main findings in three ways. First, we perform the usual checks for robustness to bandwidth selection, presented in Table 5. Panel A includes models for high-school graduation within 3 years of the test. The impact estimates, overall and by income, are generally consistent across the range of bandwidths from 2 to 6. Our main result, an average impact of 2 to 3 percentage points for low-income but not higher-income students, is robust to the choice of bandwidth.

Table 5

Estimated Causal Effects of Passing the 10th Grade Exit Examination in Mathematics for Students on the Probability of High-School and College Graduation for Students at the Margin of Passing, for Different Bandwidths by Subgroup

Group	Bandwidth (h)
Group	2	3	4	5	6
Panel 1: High-school graduation
All students	0.0156**	0.0194**	0.0165**	0.0120**	0.00725
All students	(0.0022)	(0.0028)	(0.0031)	(0.0035)	(0.0048)
Low-income students	0.0198**	0.0329**	0.0230**	0.0171**	0.0110
Low-income students	(0.0027)	(0.0067)	(0.0047)	(0.0051)	(0.0069)
Higher-income students	0.0106**	−0.00169	0.00572	0.00373	0.000826
Higher-income students	(0.0013)	(0.0043)	(0.0040)	(0.0035)	(0.0041)
Panel 2: Four-year college graduation
All students	0.0094**	−0.0012	0.0048	0.0052	0.0038
All students	(0.0018)	(0.0051)	(0.0035)	(0.0035)	(0.0043)
Low-income students	0.00015	−0.0107	−0.0062	−0.0045	−0.0061
Low-income students	(0.0014)	(0.0059)	(0.0045)	(0.0040)	(0.0041)
Higher-income students	0.0207 *	0.0097	0.0188*	0.0174*	0.0166*
Higher-income students	(0.0064)	(0.0073)	(0.0082)	(0.0067)	(0.0069)
N	25,284	35,304	45,419	55,190	64,925

Note. Standard errors clustered on raw-score point are in parentheses. Results using the optimal bandwidths appear in bold.

p < .01. *p < .05. +p < .1.

We see similar consistency in the models for 4-year college graduation, shown in Panel B. Across bandwidths, the impact estimates for all students and for the low-income subgroup are very small, and we replicate the key finding of a causal effect on college graduation for higher-income students at nearly all bandwidths from 2 to 6. The effect appears to be 1.5 to 2 percentage points on average, depending on the bandwidth chosen.

Multiple Forcing Variables

To this point, we have modeled the effect of just passing the mathematics exam without regard to the student’s passing status on the English Language Arts (ELA) exam. Because students in the 2003 to 2007 cohorts were required to pass both to graduate, the impact of just passing the math exam could depend on a student’s performance on the ELA exam (Papay et al., 2011; Reardon & Robinson, 2012). If a student failed both exams on their first attempt, for example, the hurdle to high-school graduation would be higher than if a student failed one but not the other. Failing both tests could have a more demoralizing effect on students’ college aspirations, and the remedial coursework in advance of retests might afford students few opportunities to complete college preparatory classes as juniors and seniors. This is a potentially important source of impact heterogeneity.

While most 10th grade students in Massachusetts pass both tests on their first attempt, this is not true of the sample for our regression-discontinuity models, which includes students who scored within two or three points of the mathematics passing threshold. Sixty percent of students in this group passed the math exam, and among these students, 86% also passed the ELA exam. Among those who failed math, 77% passed the ELA exam and 23% failed both tests. In other words, many more students fail the math exam than the ELA test.

We test whether our main findings depend on students’ passing status on the ELA exam by refitting our models with a dummy for PASS_ELA and the full set of its interactions with MATH^C, PASS_MATH, and LOWINC. The results of this analysis, shown in Table 6, indicate that impacts are largely concentrated in the group of students who passed the ELA test on their first attempt, which is most of the students in the sample.

Table 6

Estimated Causal Effects of Barely Passing the 10th Grade Mathematics Exit Examination for Students on the Margin of Passing, by Family Income and Their Performance Category on the English Language Arts (ELA) Examination

Group	Passing status	High-school graduation	Any college enrollment	Four-year college enrollment	Four-year college graduation
	On	(MCAS+3)	(MCAS+4)	(MCAS+4)	(MCAS+9)
	ELA exam	h* = 3	h* = 2	h* = 2	h* = 2
All students	Passed	0.0168**	0.0264**	0.0290**	0.0124**
	Passed	(0.0021)	(0.0009)	(0.0015)	(0.0017)
	Failed	0.0194	−0.00468	−0.0382**	−0.00622*
	Failed	(0.0129)	(0.0106)	(0.0026)	(0.0021)
Low-income	Passed	0.0344**	0.0264**	0.0076**	−0.0011
students	Passed	(0.0044)	(0.0047)	(0.0008)	(0.0015)
	Failed	0.0231	0.0035	–0.0290**	0.0033
	Failed	(0.0247)	(0.0174)	(0.0054)	(0.0029)
Higher-income students	Passed	−0.0052*	0.0269*	0.0551**	0.0273**
	Passed	(0.0019)	(0.0072)	(0.0040)	(0.0056)
	Failed	0.0118	−0.0314*	−0.0678**	−0.0356+
	Failed	(0.0242)	(0.0086)	(0.0054)	(0.0149)
N		34,903	25,006	25,006	25,006

Note. MCAS is the Massachusetts Comprehensive Assessment System 10th grade mathematics test; (MCAS+3) indicates that the outcome was measured 3 years after students took the test for the first time. Standard errors clustered on raw-score point are in parentheses; h* indicates the optimal bandwidth used in the regression-discontinuity model for each outcome.

p < .01. *p < .05. +p < .1.

Plots of the relationships between math exam scores and the probabilities of high-school and college graduation provide confirmatory visual evidence. In Figure 2, the top panel presents high-school graduation plots for low-income students who passed and for those who failed the ELA test; the bottom panel includes the corresponding 4-year college graduation plots for higher-income students. In both cases, the discontinuities evident in Figure 1 appear only in the plot for the students who passed the ELA test.

Figure 2.

Sample mean probabilities of high-school graduation within 3 years (top panel) and graduation from a 4-year college within 9 years (bottom panel) at score points near the passing threshold on the mathematics high-school exit examination, by family income and passing status on the English Language Arts examination.

Discussion and Conclusion

In summary, we find convincing evidence that 10th graders with essentially the same proficiency on the state mathematics test have significantly different high school and college outcomes simply because they are categorized as “passing” or “failing” the exam, and that these impacts operate at different margins for low-income and higher-income students. Using additional cohorts of data, we replicate our previous finding that barely passing the exam on the first attempt increases the probability of graduating from high school for low-income (particularly urban low-income) students near the cutoff, and again observe no impacts on higher-income students. However, this pattern is reversed for 4-year college graduation. For higher-income students, but not low-income students, just exceeding the “passing” threshold on the exit exam increases the probability of completing a 4-year college degree by about 2.1 percentage points, a large effect given that only 13% of these students near the cutoff graduate.

Our results have important implications for understanding the within-group equity consequences of exit examination policies and underscore the importance of looking at long-term consequences of educational policies as well as the shorter-term consequences. Barely passing the examination on the first attempt induces both low-income and higher-income students to enroll in college. However, despite the fact that the official signal these students receive is identical—they just passed the examination—it induces low-income students to attend 2-year colleges and higher-income students to attend 4-year colleges.

Importantly, the equity consequences of these findings depend critically on whether they stem from encouragement or discouragement. We cannot determine conclusively whether the effects are the product of an encouragement effect associated with barely passing the exam, a discouragement effect associated with just failing it, or a combination of the two. For example, among low-income students, those who just pass may become more confident about their abilities and motivated to obtain their diplomas. Alternatively, it may be that just failing the exam leads students to drop out or take additional time to retest successfully and complete other local high-school graduation requirements. These individual reactions may intersect with organizational responses by the schools students attend (Holme et al., 2010). In some schools, students receive substantial support to persist in school and retake the tests, while in others students who fail do not receive such positive messages and may be relegated to remediation courses that do not help them build academic skills. The same variation occurs among students who pass the test: in some schools, students who pass may be provided additional organizational encouragement to enroll in college-preparatory courses, while in others these students may not receive any extra guidance or support.

Similarly, if barely passing the math exit examination encourages low-income students who would otherwise not have gone to college to enroll in a 2-year college, the impact may be positive. If, on the other hand, barely failing the examination induces low-income students who would have enrolled in a 4-year college to instead attend a 2-year college, we would expect negative impacts in terms of equity (Mountjoy, forthcoming).

While we have no way to disentangle these mechanisms definitively, we follow our previous approach in Papay et al. (2016) to provide suggestive evidence. Here, we leverage students’ prior performance on the MCAS tests as 8th graders and assume that students respond predominantly to a change in their test performance label. In other words, if students who failed in 8th grade expect to fail in 10th grade, passing the 10th grade examination would constitute new information that might influence their decision to pursue further education. Any impacts for this group of students may thus reflect an encouragement effect. On the other hand, if we assume that students who passed the mathematics test in 8th grade expect, on average, to pass in 10th grade, any effects concentrated among these students would reflect the discouragement of failing. For both of our key impacts (i.e., high-school graduation for low-income students and 4-year college graduation for higher-income students), estimated effects are concentrated among students who had failed the test in 8th grade. This suggests that barely passing the examination, instead of barely failing it, may result in encouragement effects that increase educational attainments.

As Massachusetts and other states revisit their high-school graduation requirements in the coming years, these results should inform the policy debate over exit examinations. While previous work identified impacts of just passing vs. just failing on low-income students, particularly those in urban schools, we find evidence of impacts on higher-income students when we track them through college graduation. All of these impacts represent unanticipated consequences of efforts to raise standards and prepare students for college and career success. Our results can allay, at least to some degree, the fear that these consequences have negative impacts on educational equity. Our analyses suggest that barely passing the examination on the first attempt improves outcomes for students who otherwise would not have graduated from high school or completed a 4-year college degree. They also suggest the need to provide similar encouragement to students who do not pass the examinations on their first attempt. These consequences need to be at the center of efforts to make standards-based reforms work for all students in the years ahead.

Footnotes

Appendix

We see no visual evidence of manipulation of the forcing variable in Figure A1. The distribution of scores around the pass-fail cutoff on the MCAS mathematics examination is smooth.

We also check for balance of key covariates around the passing cutoff by fitting separate local linear regression models (h*=3) with each covariate as the dependent variable. All models included the forcing variable (exit exam score), an indicator for passing, and their two-way interaction, as well as cohort fixed effects. The results reveal no statistically significant differences except for student attendance in the year prior to the 10th grade test. In Table A1, students within three points of the cutoff who passed had attended school at a rate about 0.25 percentage points higher, on average, than those who just failed.

We also perform density tests described in McCrary (2008) and Frandsen (2017) and find no evidence of manipulation on the forcing variable. Using a bin size of 1 for the McCrary test, we obtain a discontinuity estimate of .0157 with a standard error of .0172, meaning we fail to reject the null hypothesis of no manipulation on the forcing variable.

Frandsen (2017) argues that the McCrary density test can be misleading when the forcing variable is discrete, as in our case, and suggests an alternate test. The Frandsen test depends on the choice of the bound coefficient, k. With 28 support points within one standard deviation of the passing cutoff, we use values of k ranging from .002 to .01 and consistently fail to reject the null with this test as well.

Acknowledgements

The authors thank the Massachusetts Department of Elementary Secondary Education for their ongoing partnership in relation to this work. In particular, we thank Carrie Conaway, Matt Deninger, and Robert Lee, who facilitated data access and provided helpful guidance that informed this research. We also thank the editor and anonymous reviewers for their helpful comments.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305H190035 to Brown University, and the Spencer Foundation. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

Notes

Authors

JOHN P. PAPAY is an Associate Professor of Education and Economics at Brown University and the director of the Educational Opportunity in Massachusetts project. His research interests include policies designed to address inequalities in educational outcomes as students progress from high school to college and into the career and how policies affect teachers and their work, including teacher hiring, teacher development, and teacher retention.

ANN MANTIL is a postdoctoral research associate with the Annenberg Institute at Brown University. She studies charter schools, desegregation and school choice programs, and the equity impacts of K–12 educational policies and interventions.

RICHARD J. MURNANE, an economist, is the Thompson Research Professor at the Harvard Graduate School of Education, Cambridge, Massachusetts. His research interests include the consequences for education of changes in national economic systems and the impacts (both intended and unintended) of particular initiatives designed to improve the performance of the education sector.

References

Betts

J. R.

(1998). The impact of educational standards on the level and distribution of earnings. American Economic Review, 88(1), 266–275.

Calonico

Cattaneo

M. D.

Titiunik

(2014). Robust nonparametric confidence intervals for regression-discontinuity designs. Econometrica, 82, 2295–2326.

Clark

Martorell

(2014). The signaling value of a high school diploma. Journal of Political Economy, 122(2), 282–318.

Dee

T. S.

Jacob

B. A.

(2006, April). Do high school exit exams influence educational attainment or labor market performance? (Working paper 12199). National Bureau of Economic Research. https://www.nber.org/papers/w12199

Dynarski

S. M.

Hemelt

S. W.

Hyman

J. M.

(2015). The missing manual: Using national student clearinghouse data to track postsecondary outcomes. Educational Evaluation and Policy Analysis, 37(1_suppl), 53S–79S.

Evers

W. M.

Walberg

H. J.

(Eds.). (2002). School accountability. Hoover Institution Press.

Frandsen

(2017). Party bias in union representation elections: Testing for manipulation in the regression discontinuity design when the running variable is discrete. In Cattaneo

M. D.

Escanciano

J. C.

(Eds.), Advances in econometrics: Vol. 38—Regression discontinuity designs: Theory and applications (pp. 29–72). Emerald.

Holme

J. J.

(2008). High stakes diplomas: Organizational responses of California’s high schools to the state’s exit examination requirement. Research in the Sociology of Education, 14, 157–188.

Holme

J. J.

Richards

M. P.

Jimerson

J. B.

Cohen

R. W.

(2010). Assessing the effects of high school exit examinations. Review of Educational Research, 80(4), 476–526.

10.

Imbens

G. W.

Kalyanaraman

(2012). Optimal bandwidth choice for the regression discontinuity estimator. The Review of Economic Studies, 79(3), 933–959.

11.

Imbens

G. W.

Lemieux

(2008). Regression discontinuity designs: A guide to practice. Journal of Econometrics, 142(2), 615–635.

12.

Jacob

B. A.

Linkow

T. W.

(2011). Educational expectations and attainment. In Duncan

G. J.

Murnane

R. J.

(Eds.), Whither opportunity? Rising inequality, schools, and children’s life chances (pp. 133–162). Russell Sage Foundation.

13.

Kornhaber

M. L.

Orfield

(2001). High-stakes testing policies: Examining their assumptions and consequences. In Orfield

Kornhaber

M. L.

(Eds.), Raising standards or raising barriers? Inequality and high-stakes testing in public education (pp. 1–18). Century Foundation Press.

14.

Kyoore

(2019). The effect of exit exams on college enrollment: A state level analysis [Doctoral dissertation, University of Missouri].

15.

Lee

D. S.

Card

(2008). Regression discontinuity inference with specification error. Journal of Econometrics, 142(2), 655–674.

16.

Lee

D. S.

Lemieux

(2010). Regression discontinuity designs in economics. Journal of Economic Literature, 48(2), 281–355.

17.

Martorell

(2005). Does failing a high school graduation exam matter? [Working paper].

18.

McCrary

(2008). Manipulation of the running variable in the regression discontinuity design: A density test. Journal of Econometrics, 142, 698–714.

19.

Mirtcheva

D. M.

Powell

L. M.

(2009). Participation in the National School Lunch Program: Importance of school-level and neighborhood contextual factors. Journal of School Health, 79(10), 485–494.

20.

Mountjoy

(forthcoming). Community colleges and upward mobility. American Economic Review.

21.

Murnane

R. J.

Willett

J. B.

(2011). Methods matter: Improving causal inference in educational and social science research. Oxford University Press.

22.

National Center for Education Statistics. (2019). The condition of education 2019 (NCES 2019-144). https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2019144

23.

(2010). To leave or not to leave? A regression discontinuity analysis of the impact of failing the high school exit exam. Economics of Education Review, 29(2), 171–186.

24.

Papay

J. P.

Mantil

Murnane

R. J.

Donohue

McDonough

(2020). Lifting all boats? Accomplishments and challenges from 20 years of education reform in Massachusetts. Educational Opportunity in MA, Brown University. http://liftingallboatsma.org

25.

Papay

J. P.

Murnane

R. J.

Willett

J. B.

(2010). The consequences of high school exit examinations for low-performing urban students: Evidence from Massachusetts. Educational Evaluation and Policy Analysis, 32(1), 5–23.

26.

Papay

J. P.

Murnane

R. J.

Willett

J. B.

(2014). High-school exit examinations and the schooling decisions of teenagers: A multi-dimensional regression-discontinuity analysis. Journal of Research on Educational Effectiveness, 7(1), 1–27.

27.

Papay

J. P.

Murnane

R. J.

Willett

J. B.

(2016). The impact of test-score labels on human-capital investment decisions. Journal of Human Resources, 51(2), 357–388.

28.

Papay

J. P.

Willett

J. B.

Murnane

R. J.

(2011). Extending the regression-discontinuity approach to multiple assignment variables. Journal of Econometrics, 161(2), 203–207.

29.

Polson

(2018). TAKS-ing students? Texas exit exam effects on human capital formation. Economics of Education Review, 62, 129–150.

30.

Reardon

S. F.

Arshan

Atteberry

Kurlaender

(2010). Effects of failing a high school exit exam on course taking, achievement, persistence, and graduation. Educational Evaluation and Policy Analysis, 32, 498–520.

31.

Reardon

S. F.

Robinson

J. P.

(2012). Regression discontinuity designs with multiple rating-score variables. Journal of Research on Educational Effectiveness, 5(1), 83–104.

32.

Sipple

J. W.

Killeen

Monk

D. M.

(2004). Adoption and adaptation: School district responses to state imposed high school graduation requirements. Educational Evaluation and Policy Analysis, 26, 143–168.

33.

Warren

J. R.

Grodsky

E. S.

Lee

J. C.

(2008). State high school exit examinations and post-secondary labor market outcomes. Sociology of Education, 81, 77–107.