Regression Discontinuity in Prospective Evaluations

Introduction

Regression discontinuity (RD) analysis is receiving increasing attention for use in impact evaluation. ¹ RD analysis exploits situations in which a program assigns a binary status (e.g., a benefit or a requirement) based on the value of some “ranking variable,” where those with ranking variable scores on one side of a specified cutoff get the status (i.e., the benefit or requirement), while those on the other side do not. In this situation, comparing outcomes “very close” to the cutoff on either side gives a valid estimate of the causal impact of the status. The RD heuristic is straightforward: Sufficiently close to the cutoff, the impact of the program is large relative to any spurious correlation with the ranking variable. ² RD is considered the strongest possible design when random assignment is not possible (Cook, 2008; DiNardo & Lee, 2010). Similarly, the latest What Works Clearinghouse evidence standards designate well-conducted RD studies alongside randomized controlled trials as designs able to yield credible evidence of intervention effectiveness (Institute of Education Sciences, 2013).

Retrospective applications of RD proceed as follows: For the program or policy of interest, the evaluator first identifies an extant data set including measures of both the outcome of interest and the variable or variables determining eligibility. Using that data set, impact is then estimated by comparing outcomes for observations as close as possible to the eligibility cutoff. As we discuss in detail in the section on The Case for Prospective RD, deciding which observations should be included in the analysis—and, in particular, how far those observations may be from the cutoff—and what regression corrections should be made to account for the fact that observations are not exactly at the cutoff involves a trade-off between (apparent) precision and internal validity.

RD also has potential for prospective impact evaluation, especially when the evaluation involves primary data collection. Such prospective RD evaluation applications offer new opportunities—and raise new issues. This article considers these opportunities and issues in the context of a recent prospective RD evaluation of the U.S. Department of Agriculture (USDA) Food and Nutrition Service (FNS) Fresh Fruit and Vegetable Program (FFVP). FFVP provides free fresh fruits and vegetables to students outside of regular school mealtimes. The FFVP authorizing legislation mandated an impact evaluation; the legislation also required that the limited program funds be allocated to those schools in each state with the highest proportion of low-income students. This mandate renders a random assignment-based evaluation design infeasible but simultaneously suggests that an RD design is feasible. ³ Abt Associates Inc. implemented a prospective RD design to evaluate the impact of FFVP.

The balance of this article proceeds as follows. The subsequent section provides an overview of RD designs in theory and practice. The section on Gains From Prospective RD: Targeted Sample Selection discusses the first main advantage of prospective RD relative to retrospective RD, the ability to conduct targeted sample selection. The section on Gains From Prospective RD: Known Program Selection defines fuzzy RD and contrasts its role in retrospective and prospective RD. The section on Using RD to Estimate the Impact of FFVP describes how we implemented prospective RD in the FFVP evaluation and relates the results of that evaluation to the article’s claims about prospective RD. The Discussion section summarizes our broader lessons for implementing RD for prospective evaluation with primary data collection and outlines some directions for future research.

The Case for Prospective RD

This article considers RD as a design for prospective evaluations. In the prospective evaluation context, the random assignment design is often viewed as ideal for estimating causal impacts. However, in practice, random assignment is often infeasible.

For FFVP in particular, the governing statute requires that funds within each state are to be allocated “to schools with the highest percentages of low-income students, to the maximum extent practicable” (National School Lunch Act, Section 19, 42 U.S.C. 1770) at a level of US$50–US$75 per student per school year, equivalent to between US$1 and US$2 per week. This legislative requirement implicitly precludes a random assignment evaluation; that is, schools with a larger percentage of low-income students could not legally be randomized out of the program in favor of schools with a smaller percentage of low-income students.

The same provision also invalidates propensity score-matching approaches, because such strategies require substantial overlap between treatment and comparison groups (Shadish, Cook, & Campbell, 2001). Since, by statute, schools participating in FFVP should always have a higher percentage of low-income students than schools not participating in the program, no such overlap between participants and nonparticipants should be present in the FFVP case.

However, the same legislative provision invalidating these alternative evaluation strategies induces a discontinuity making RD analysis feasible: Within each state, applicant schools with percentages of low-income students higher than some cutoff will receive FFVP funds, while applicant schools with percentages of low-income students below that cutoff will not receive funding. In the remainder of this section, we further elucidate the practical advantages of prospective RD in an evaluation context, as a remedy to some of the inherent methodological challenges in implementing RD designs, using the FFVP evaluation as an illustrative case.

Figure 1 captures the essence of the RD intuition here illustrated with the FFVP evaluation example. Along the horizontal axis, we give the student income proxy that FNS selected to rank schools applying to FFVP—the percentage of students in each school eligible to receive free or reduced-price school lunch (FRPSL). In this example as illustrated, the eligibility “cutoff” is set at 70%; schools with an FRPSL percentage greater than 70% (above the cutoff) get FFVP; schools with an FRPSL percentage less than 70% (below the cutoff) do not get FFVP. Note that the exact cutoff is unknown when school applications are due, so it would be difficult for a school to manipulate its FRPSL percentage to make itself eligible. ⁴

OPEN IN VIEWER

Figure 1.

Motivating RD in the FFVP evaluation. RD = regression discontinuity.

Along the vertical axis, we plot the outcome of interest; here, total cups of fruits and vegetables consumed per child per day. In the field of the graph, we draw notional outcomes with and without the program. As drawn (and consistent with what we find; see subsequently), FFVP effectively increases fruit and vegetable consumption among students in participating schools; thus, the dotted line indicating consumption under FFVP is above the dashed line without FFVP. In addition, notional outcomes are posited to worsen with FRPSL percentage (i.e., students in higher FRPSL percentage schools consume fewer fruits and vegetables on average); thus, the two lines slope down and to the right (i.e., as FRPSL percentage rises). Then, the solid line plots the posited reduced form relation between FRPSL percentage that would be observed by the evaluator in the presence of the program with an infinitely large sample, eliminating sampling variability; that is, the lower/no FFVP line below (to the left of) the cutoff and the higher/FFVP line above (to the right of) the cutoff.

Note that we have deliberately avoided depicting the relationship between the ranking variable and the outcome of interest as exactly linear; as drawn, it is instead a slightly negative quadratic. In addition, we have drawn the gap between notional outcome with and without the program or policy as nonconstant; in the graph, the gap grows with FRPSL percentage. While we have no a priori reason to believe these particular deviations from a linear relationship between FRPSL percentage and outcomes, or from a constant FFVP impact, are correct, we draw the graph in this way to emphasize that any approach to estimation and any interpretation of the results of the estimation should be robust to those possibilities.

Finally, note the posited sharp discontinuity immediately at the cutoff in the observed relation between the ranking variable (in our example, FRPSL percentage) and the outcome of interest (in our example, cups of fruits and vegetables consumed). RD proceeds by exploiting that discontinuity. Sufficiently close to the cutoff, the scale of the discontinuity will dominate any change in the outcome associated with variation in the ranking variable (i.e., the slope of the plotted lines on either side of the cutoff), so simple comparisons of schools just below the cutoff (without FFVP) to schools just above the cutoff (with FFVP) will identify the impact of the program without bias.

The “at-the-cutoff” caveat is crucial. There is no a priori reason to believe that the impact should not vary with the location of the cutoff. However, RD is a local estimator. Inasmuch as impact does vary with the location of the cutoff, it is only appropriate to extrapolate RD estimates to points away from the cutoff under the assumption that—in the region to which one is extrapolating—impact estimates are (relatively) stable as the ranking variable varies. Thus, using RD is implicitly a choice of internal validity (i.e., valid estimates at the cutoff) over external validity (i.e., generalizability away from the cutoff).

Granted this caveat, however, several arguments in defense of RD nonetheless seem appropriate. First, conventional practice is to size studies to detect any impact. Internally, valid impact estimates at any point may represent a major contribution to knowledge. Second, existing evidence for variation in impacts is weak, probably in part because detecting such variation requires substantially larger samples than budgets typically allow. Thus, assuming homogenous impacts is a useful starting point. Third, impacts at the cutoff are exactly the quantity of interest when considering a particular policy change, that is, a (small) expansion or reduction in funding. Despite these points, the local nature of RD inference is a crucial caveat and should be kept in mind in interpreting and applying RD estimates.

Gains From Prospective RD: Targeted Sample Selection

The classic case for RD thus implicitly, but fundamentally, relies on the assumption that sampled units (individual schools, in the case of the FFVP evaluation) are indeed sufficiently close to the selection cutoff. For an outcome hypothesized to vary with the ranking variable (in our case, variation in average student fruit and vegetable consumption associated with school FRPSL percentage), the further from the cutoff, the larger is the plausible effect of the ranking variable relative to the fixed size of the true effect at the cutoff. Thus, as an evaluation samples children in schools further and further from the cutoff, observed differences between average outcomes with the program and average outcomes without the program will increasingly reflect not the true impact of interest—the effect of FFVP—but instead effects related to the association of outcomes with the ranking variable.

Consider, as an extreme case, a sample that includes mostly observations with either very low FRPSL percentage (e.g., less than 10%) or very high FRPSL percentage (e.g., near 100%). As shown in Figure 1, in such a sample, a simple comparison of mean outcomes in schools with and without the program would appear to show that FFVP decreases fruit and vegetable consumption. As shown, observed fruit and vegetable consumption in the sampled nonparticipating schools would average about two and a quarter cups versus one and a half cups in the sampled participant schools with the highest percentages of FRPSL students.

This line of argument emphasizes that, as discussed previously, the basis for the internal validity of RD is inherently local—it strictly applies only very near the cutoff. ⁵ However, in practice, particularly in applications relying on extant secondary data, large numbers of observations very near the cutoff are rarely available. The conventional analytic approach to this challenge is to explicitly model the relationship between the ranking variable and the outcome. When the true relationship of the outcome to the ranking variable (in the absence of the intervention) is exactly linear, then inclusion of a linear term in the ranking variable as a regressor when estimating impacts will effectively isolate the discontinuity as the primary causal factor, regardless of the position of sampled units along the ranking variable spectrum and the width of the observation window. When the true relationship is quadratic as illustrated, then the appropriate approach is to include linear and quadratic terms. This line of argument justifies including observations far from the cutoff.

In practice, there are three problems with this approach. First, as Goldberger (1972) shows, the ranking variable is—by construction—highly correlated with the program participation variable. This correlation imposes a strong power penalty when the ranking variable is included as a regressor. Goldberger’s analysis suggests that including a ranking variable increases the required sample sizes by 2.6 to 4 times relative to random assignment.

Second, the relation between the outcome and the ranking variable may not in fact be exactly linear (or quadratic, as depicted in Figure 1). If the analyst is willing nonetheless to assume that the relation is continuous (as is often, but not always, plausible), standard Taylor series approximation arguments ⁶ imply that locally a linear approximation is likely to be reasonable. However, the plausibility of the linear (or quadratic) approximation declines quickly with distance from the cutoff. As noted above, particularly in RD evaluations relying on extant secondary data, many included units may be far from the cutoff, implying that inclusion of a linear (or quadratic) term in the ranking variable may be insufficient in many applications.

Third, there may exist other programs or policies with rules that induce other discontinuities with respect to the ranking variable (or variables strongly correlated with the ranking variable). In the presence of other such discontinuities, the locally linear (or even locally continuous) argument does not follow (Shadish et al., 2001; Lee & Lemieux, 2010).

Formalizing the existing literature, Lee and Lemieux (2010) suggest addressing these three problems using a combination of an analogy to Taylor series approximation and a pretest strategy as follows. The analysis begins with some high-order polynomial, that is, not just a linear term but also a quadratic and cubic (and perhaps even higher order) polynomial. That model is estimated and the highest order term is tested for statistical significance. If the highest order term is significant, that model is used; if not, the highest order term is dropped and the process repeated (i.e., reestimate without the highest order term) until the highest order term is statistically significantly different from zero.

This algorithm provides a data-driven approach to determining how best to model the ranking variable. Subsequently, we employ this algorithm in our evaluation of FFVP. We note, however, that the internal validity of this approach rests on the untestable assumption that the polynomial in the ranking variable captures (nearly) all of the variation in the outcome with respect to the ranking variable. We also note that such pretest strategies usually have low power. Thus, while the pretest strategy is a constructive approach to the analytic challenge of specifying the polynomial, it is not ideal.

In practice, the data are generally insufficient to estimate the relationship between the ranking variable and the outcome with any precision. That lack of precision in the estimation of the relation to the ranking variable will then induce specification error into the estimate of the impact. As the observations are drawn farther from the cutoff and as the order of the polynomial increases, the plausibility of this implicit and crucial assumption falls rapidly. With a high-order polynomial and/or observations far from the ranking cutoff, the local RD argument no longer plausibly implies high internal validity for the RD estimator. RD should thus no longer be considered to have high internal validity when the observations are “far” from the cutoff.

This discussion leads naturally to a first observation on the inherent advantages of prospective RD. In evaluations relying on extant secondary data, by necessity the evaluator is constrained to use whatever data are available—often only a small fraction of the full population. In such a sample, many of the units closest to the cutoff will not be included in the secondary data sample. For any given sample size, then, the included observations in an RD application relying on extant secondary data will represent a wider range of the ranking variable than would be necessary if we could purposively select for the analysis sample only those observations closest to the cutoff.

This low sampling fraction requires a wider window (i.e., a greater range of observations included in the analysis sample) to generate sufficiently large samples. This wider window in turn induces a cascading series of negative analytic consequences for the evaluator. A wider window increases the likelihood that a high-order polynomial will need to be included in the impact regression. The associated loss in statistical power and internal validity implies that a still larger number of units will need to be included—further increasing the range of the ranking variable in the analytic sample and additionally exacerbating the associated loss in statistical power and internal validity. As we argue in greater detail in the section on Gains From Prospective RD: Known Program Selection, prospective RD applications can deliberately select for primary data collection the observations closest to the cutoff, thereby minimizing the cascading costs of a wider window.

Gains From Prospective RD: Known Program Selection

The previous section and the illustration in Figure 1 depict the classic motivation for RD when the ranking variable exactly determines which units are selected to participate in the program. In practice, simple examination of the data will often suggest that the ranking variable is not in fact the only determinant of program participation—that is, there is often some overlap in the ranking variable between participating and nonparticipating units. In some cases, this phenomenon arises from measurement error in the observed ranking variable: The value of the ranking variable observed by the evaluator is not identical to that employed in making the program selection decision. Another common cause is incomplete program take-up: If the evaluation sample includes units that do not apply for the program, and if the application process is not observed by the evaluator, some units that appear to be eligible based on the ranking variable will not participate. More broadly speaking, this is a special case of a situation in which some additional (and potentially unobservable) criteria beyond the ranking variable alone factor into program application, eligibility, and/or selection determination.

In retrospective RD, the standard strategy to address the inexact association between the ranking criterion and the actual participation is to apply instrumental variable (IV) techniques, sometimes referred to in the literature as “fuzzy RD” (Imbens & Lemieux, 2008; Lee & Lemieux, 2010). In particular, assuming the program will have impacts on only the units that actually participated, the analysis can employ the logic of the Bloom estimator (Bloom, 1984) for partial compliance.

Estimate the difference in outcomes between units on either side of the designated cutoff—regardless of actual program participation status. In essence, that first-stage estimate will be “diluted” by the presence of nonparticipant units in the analytic sample on the participation side of the cutoff.

IV estimation with a binary indicator for status relative to the cutoff instrumenting for participation is the regression analog of this approach (Heckman, Smith, & Taber, 1998; Heckman, Hohmann, & Smith, 2000).

Note, however, that this IV approach implies (another) power penalty. Consider the group of schools in the analytic sample on the participation side of the cutoff. The estimated impact on these schools unconditional on actual participation will be the product of the “true” impact on participating schools and the actual fraction of participating schools in this group. The required sample size for detecting the true impact then increases with the square of the reciprocal of the fraction of participating schools. Thus, participation levels of 88%, 50%, or 32%, respectively, among schools on the participation side of the cutoff, correspond to required sample sizes 1.5, 4, or 10 times as large, respectively, as the sample needed in the case where all units on the participation side of the cutoff actually participate. These larger samples have two more negative implications for the evaluator. First, larger samples imply (often much) higher costs. Data collection costs often comprise the bulk of total costs for a prospective evaluation. Thus, total evaluation costs will often increase nearly proportionately with the increase in sample size.

Second, for an extant data source, larger samples imply a wider range of the ranking variable. This wider range for the ranking variable leads to escalating power penalties as the order of the polynomial that must be included as a regressor increases. These power penalties in turn imply both further required increases in the sample size and decreased internal validity of the resulting RD estimates.

The FFVP evaluation illustrates the potential advantages of prospective RD to address this problem in the context of incomplete program take-up—as well as the challenges in doing so. One might have expected the FFVP evaluation to be an ideal and pure application of RD: the legislation specifically requires using a cutoff. The reality was much more complicated. Prospective RD, particularly in understanding the selection process, allowed improved precision and greater internal validity.

As noted, the award of FFVP to a school was to be based on poverty. USDA regulations interpreted school poverty as FRPSL percentage. Information on school FRPSL percentage is publicly available in the common core data (CCD). ⁷ Given the list of all schools in a state and their FRPSL percentages from the CCD for a given school year, and the list of schools receiving FFVP funding in that year, one can approximately reverse-engineer the selection cutoff for each state—as the lowest FRPSL percentage for a funded school.

However, despite the apparent clarity of the legislation, FRPSL does not exactly determine which schools operated FFVP. Some schools with FRPSL percentages above that reverse-engineered cutoff value—those that did not apply or whose applications were rejected (see Olsho, Klerman, & Bartlett, 2011)—did not in fact receive FFVP funds. This discrepancy arises in small part from the fact that the actual FRPSL percentage varies slightly over time due to shifts in student enrollment. The FRPSL percentage used by each state in making FFVP allocation decisions for a given school year may therefore differ slightly from that published in the CCD.

In the course of the FFVP evaluation, we noticed discrepancies between the CCD FRPSL percentages and participation status. As official evaluators, in these cases, we simply contacted the states and requested information on actual FRPSL percentages used to determine eligibility. This ability to directly contact states effectively eliminated this potential source of measurement error and associated analytic concerns.

Nonetheless, even using the exact FRPSL percentage cutoffs provided by states, starting from the cutoff and in the direction of eligibility, not all schools with FRPSL percentages above the cutoff actually implemented FFVP. This phenomenon was driven by the actual process through which schools were selected to participate; in that selection process, FRPSL percentage was not the only criterion employed.

The FFVP statute required that schools submit an application for FFVP funds in order to be selected to participate in the program. FNS rules allowed states to review each school’s FFVP application in making funding determinations. An official FNS Program Note (U.S. Department of Agriculture, Food and Nutrition Service, 2009) explicitly forbade the use of most secondary criteria (e.g., geographical balance) in screening school applications. However, that guidance explicitly identified the following four allowable reasons to pass over high FRPSL percentage schools in favor of lower FRPSL percentage schools:

school fails to meet the deadline for application completion;

States therefore reviewed school applications with these secondary criteria in mind when deciding whether to approve or reject. ⁸

Approved school applications were then ranked according to FRPSL percentage to determine the final funding status. States worked their way down the list until they had allocated all available FFVP funds. In other words, the cutoff was not known ex ante: At the time of preparing an application, a school would not know where its FRPSL percentage fell relative to the cutoff, since that was determined only once all applications had been received and reviewed.

Our status as official and prospective evaluators gave us a window into what otherwise would have been the “black box” of the FFVP selection process. As such, we were able to build a sample consisting of those schools—with FRPSL percentages above the cutoff—that got FFVP funds, and “comparable” schools below the FRPSL percentage cutoff that did not get FFVP funds, where comparable schools are those that applied for FFVP funds and whose applications were not rejected by states for the reasons listed above (e.g., not meeting the deadline). That is, the only reason for nonparticipation among schools in our comparison sample was the fact that their FRPSL percentages fell below the selection cutoff in their state, since nonapplicant schools with FRPSL percentages above the cutoff and “rejected” schools with FRPSL percentages above the cutoff were not included in our evaluation sample. Because we confined our analysis to the group of schools whose applications were accepted, no Bloom adjustment was necessary, no power/sample size penalty was imposed, the required window of FRPSL values in the estimation sample did not grow, and the overall plausibility of RD as a solution to issues of internal validity was preserved.

Stated in more general terms, the problem inducing the need to apply IV is as follows: An evaluator can usually identify which units above the cutoff got the program via a simple perusal of the list of participants. However, an evaluator will often not be able to identify the units below the cutoff that would have gotten the program if additional funding had been available and the same ranking criterion had been applied.

Moreover, as Olsho, Klerman, and Bartlett (2011) discuss, variations in the standard selection process may further compromise the evaluator’s ability to identify an appropriate comparison sample below the cutoff:

Program administrators may not retain information on all applicant schools or related application scoring criteria. For example, they may discard unsuccessful applications.

In practice, an evaluator will need to deal with additional complications like these when applying prospective RD. This implies that despite a program-induced cutoff, even prospective RD will sometimes need to use IV methods.

This discussion illuminates the second key advantage of prospective RD, that is, applying RD prospectively (and, in particular, as an official evaluator), it is generally possible to discuss unit selection with program administrators—ideally in “real time” as they are making selection and allocation decisions—enabling a clear understanding of details of the selection process.

Moreover, the evaluator may be able to subtly shift the selection process, for example, by working with program administrators to ensure that:

all decision materials are retained and all selection decisions are clearly documented,

With this information in hand, the evaluator will usually be able to identify the units below the cutoff that differ from the selected units only because of the ranking variable, thereby avoiding the necessity to apply IV methods in estimating impacts. Again, this ultimately allows selection of a smaller sample within a narrower range of the ranking variable, maximizing RD’s internal validity and minimizing data collection costs.

Using RD to Estimate the Impact of FFVP

The previous sections have motivated prospective RD using the example of the FFVP evaluation. In particular, the discussion emphasized that prospective RD allows collecting data on the observations closest to the cutoff, thereby increasing precision and improving internal validity. This section illustrates how those ideas interact.

The previous section emphasized the ability of a prospective implementation of FFVP to select observations near the cutoff. For FFVP, we randomly chose 16 states and then 256 schools in those states (where the numbers of schools per state was selected to be proportional to the total program size). Within those schools, we randomly selected classrooms and then children. Randomly selected children reported detailed dietary intake via a diary-assisted 24-hr recall interview. The final sample included 5,560 students in 252 schools. (See Olsho et al. (2011) and Bartlett et al. (2013) for a complete description of sample selection.)

Figure 2 shows the maximum distance to the cutoff from above and below for schools within each state in our sample. Thirty-eight of the 252 schools in which we performed data collection were more than two- and a-half percentage points away from the state-specific cutoff; our preferred analytic sample (hereafter termed the “restricted near-cutoff sample”) excludes those schools, yielding 4,696 students in the 214 schools that remain.

OPEN IN VIEWER

Figure 2.

Maximum distance to the percentage FRPSL cutoff by state, from above and below. FRPSL = free or reduced-price school lunch.

Impact estimation then proceeds via linear regression as specified in Equation 1:

Y s , i = α + d s π + R ρ + X s , i β + Z s γ + ϵ s , i

1

where s indexes schools, i indexes students within schools, Y is some nutritional outcome, d is an indicator variable for treatment status (= 1 if treatment school, = 0 if comparison school), R is the ranking variable, X is a vector of individual student-level observed covariates (not affected by the intervention; both s and i subscripts), Z is a vector of school-level covariates (only an s subscript), ∊ is a random error term, and α, π, ρ, β and γ are parameters to be estimated. In this specification, the crucial parameter is π, the estimate of the impact of FFVP at the cutoff. Specifically, we pool data across states and specify the ranking variable, R, as the distance in percentage points from the FRPSL percentage in each sample school to the relevant state-specific FRPSL percentage cutoff (see Black, Galdo, & Smith, 2007 for a similar approach). ⁹

Following the literature, this specification includes the ranking variable. However, as noted in the section on The Case for Prospective RD, including the ranking variable as a regressor typically imposes a large cost in statistical power. We conjecture that given the way we have selected the schools for the RD sample, it is not necessary to include R at all. Because we drew our samples from entire states, most selected schools (particularly within our preferred restricted near-cutoff sample) are within a few percentage points of the FRPSL from the corresponding state cutoff, implicitly obviating the need to adjust for the ranking variable in our models. In the results section below, we provide evidence on this conjecture for our data. Specifically, we report results both excluding and including a linear term in the ranking variable.

Table 1 provides estimates for four different RD specifications. Estimates are presented for two different samples: (i) the full analytic sample and (ii) the restricted near-cutoff sample of schools within two- and a-half percentage points of the FRPSL cutoff in each sample state. For each sample, we present two sets of estimates—without and with the ranking variable.

OPEN IN VIEWER

Table 1.

Impact of FFVP on Consumption of Fruits and Vegetables.^a

Sample	Restricted near-cutoff subsample (n = 4,696)		Full analytic sample (n = 5,560)
Ranking variable included?	No	Yes	No	Yes
T mean (regression adjusted; SE)	2.39 (0.06)		2.37 (0.05)
C mean (regression adjusted; SE)	2.07 (0.05)		2.04 (0.05)
Estimated impact (T − C); SE	0.32***; 0.08	0.32***; 0.08	0.33***; 0.07	0.33***; 0.07
Ranking variable; SE	N/A; N/A	−0.05; 0.07	N/A; N/A	−0.03; 0.02

Note. SE = standard error; N/A = not applicable; T = treatment group; C = comparison group. Regression adjustment using sample characteristics. ^acup equivalents per day.

Asterisks indicate statistical significance for regression coefficients: *p < .10. **p < .05. ***p < .01. (One-sided test for increase in fruit and vegetable consumption.)

In neither sample is the test on the coefficient for the ranking variable statistically significant at the 95% confidence level. The lack of clear evidence of significance for the ranking variable is as expected, given the narrow range of values for FRPSL percentage (even in the full sample) made feasible by our prospective RD approach. Note that there is slightly stronger—but still not statistically significant—evidence for including the ranking variable with the broader sample. Furthermore, the literature supports using this pretest strategy and excluding the ranking valuable when the analyst cannot reject that its slope is zero (Lee & Lemieux, 2010). Our preferred specification therefore excludes the ranking variable and includes only schools within two- and a-half percentage points of the FRPSL cutoff. We note that for this FFVP example, including the ranking variable does not materially affect the magnitude of the impact estimates. ¹⁰

The point estimate for the impact of FFVP on total fruit and vegetable consumption per day is robust across all four specifications (range 0.32–0.33 cups) and always statistically significant at conventional levels (p-values below 0.001). In this example, probably because the range of the ranking variable is so narrow, including the ranking variable has only trivial effects on the standard error and t-statistic for the treatment effect. However, as noted, our tests do not support the need for this variable’s inclusion.

Discussion

This article has used the application of RD to an impact evaluation of the USDA FFVP to discuss some issues in—and, in particular, advantages of—using RD in a prospective evaluation.

In particular, we argued that prospective RD yields two advantages. First, rather than relying on observations in some existing secondary data set, prospective RD with primary data collection allows the evaluator to purposively target data collection to those observations closest to the cutoff. Second, the close connection with program administrators involved in site/participant selection—ideally, before and during site/participant selection—affords the evaluator clear advantages in understanding the selection process, including identification of the exact ranking variable precisely as applied, and an insight into secondary selection criteria that may have been employed. In the case of FFVP, such a connection to program administrators allowed us to identify an appropriate group of comparison schools below the FRPSL percentage cutoff (those schools that had applied and had applications deemed acceptable by the state).

Together, these two advantages of prospective RD allowed us to collect data on many fewer observations, and those observations are much closer to the cutoff value. Consistent with our conjecture, our observations are so close to the cutoff point that we cannot reject the no-ranking variable specification in favor of the linear ranking variable specification. The literature suggests that including the ranking variable hurts precision. In our FFVP example, the range of the ranking variable is so narrow that including it has only trivial effects on precision.

Finally, this narrower range of values of the ranking variable is much more consistent with the local justification of the RD approach; that is, comparing outcomes for observations just on either side of the cutoff. This increases the plausibility of the claims of strong internal validity for RD.

The bureaucratic state often induces discontinuities in eligibility for discrete benefits. RD exploits these discontinuities, holding out the promise of impact estimates with strong internal validity. However, those claims of strong internal validity require large numbers of observations very close to the cutoff point. In practice, these requirements suggest that—even given a clear discontinuity—RD will often be infeasible or that the application of RD will require using observations so far from the cutoff point as to lead to questions about the actual internal validity of the resulting RD estimates. These issues are less salient with “prospective RD,” where the evaluator can learn the details of the selection process (so IV is unnecessary) and sample the units closest to the cutoff. Knowing the selection process and selecting the observations closest to the cutoff allow for much, much smaller samples. These smaller required samples and lower cost will make prospective RD feasible for more program evaluations.