How Three-Arm Random Assignment Within Sites can Improve Non-Experimental Cross-Site Estimates of the Relationship Between Program Characteristics and Impact

Abstract

This article considers a new method, called Cross-Site Attributional Model Improved by Calibration to Within-Site Individual Randomization Findings (abbreviated as “CAMIC”), which seeks to reduce bias in analyses of the contributions of a program’s various design, implementation, and contextual characteristics to its overall impacts. It requires a multi-site experiment where some (or all) sites randomize individuals to one of three arms: a standard treatment group, an enhanced treatment group (that receives the standard treatment plus a program “enhancement”), or a control group (that has no access to the program). A recent evaluation—that of the Health Profession Opportunity Grants (HPOG) program—provides a motivating example of this design and its potential for both methodological and substantive lessons; and the article considers other evaluations that would be fitting for deploying and learning from CAMIC. We conclude that the promise for CAMIC lies in situations where the correlations between the selected program enhancement and alternative program characteristics of interest are relatively high, implying that producing an experimental estimate of the enhancement can reduce bias in the estimation of other non-randomized program characteristics.

Keywords

impact analysis program evaluation experiments causal inference

Introduction

Experimental evaluations—in which study participants are randomly assigned to treatment and control groups within sites—provide researchers with a method for understanding the overall effectiveness of programs without selection bias. However, once we understand the magnitude and sign of program impacts, the next question is why the program did or did not have its intended effect. Programs that operate in many locations—and the multi-site evaluations that accompany them—offer an opportunity to “get inside the black box” to explore how program characteristics determine its impact on outcomes of interest. They can do so by leveraging both planned and natural variation in the program world. Beyond multi-site experiments, multi-armed experiments offer greater potential still.

For example, randomly assigning individuals within each site either to a standard treatment group or to an “enhanced” treatment group that is offered the standard treatment plus a program enhancement is a multi-armed design that has both planned and natural variation. The planned variation comes from the three-armed experiments where the two treatment arms represent variants of the program; and the natural variation comes from the fact that the evaluation takes place in multiple locations. Researchers can produce an experimental estimate of the influence of the program enhancement on impact magnitude via the multi-armed experiment. But, the program components themselves are not randomized to the locations within which evaluations take place. Although they could be, the program’s design (choices about the components that are part of the program) is not often randomized. Indeed, there are relatively few circumstances in which randomizing at the program level would be feasible or desirable. Further, some characteristics of a program—such as the nature of its implementation (as generated by the dynamism of a program manager or office culture, e.g.,) cannot be randomized. Instead, it is routinely the case that each site chooses its own configuration of program components to adopt, each possesses its own set of implementation features, and each operates within its own context with its own target populations. In a multi-site evaluation, design and implementation characteristics tend to vary naturally across sites.

In this case, researchers will rely on that naturally occurring cross-site variation in program characteristics to produce non-experimental estimates of the influence of program characteristics on impact magnitude. Early examples of this sort of work include Greenberg et al. (1993; 1994) and Bloom et al. (2003) who pooled multi-site job training and welfare reform experiments to assess how certain program design and implementation characteristics associated with experimentally based impact estimates. As those researchers pointed out, impact estimates may be subject to omitted variable bias if there are unmeasured or omitted site-level variables that are correlated with the program characteristics included in the model and that influence treatment impact magnitude (e.g., Bloom et al., 2001; Moulton et al., 2014). For example, a local program manager’s enthusiasm and leadership might be associated with both the choice to offer access to peer support groups as part of their job training; and those same traits may also lead the program to have larger or smaller impacts. In this situation, the manager’s traits are associated both with what is offered and with impact magnitude. If the researcher fails to control for these traits (which are often difficult to measure), then cross-site non-experimental estimates of the relationship between program characteristics and impact magnitude will be biased.

This article considers whether having an experimental estimate of the contribution that one program enhancement makes to impact magnitude can help to reduce the bias in cross-site non-experimental estimates of the relationship between other program characteristics and impact. More specifically, the approach we use is called the Cross-Site Attributional Model Improved by Calibration to Within-Site Individual Randomization Findings Method (abbreviated as “CAMIC”). The CAMIC method uses the difference between experimental and non-experimental estimates of the program enhancement to choose the statistical model with the least-biased non-experimental estimate of the program enhancement. The method then uses this model to estimate the influence of program characteristics for which random assignment did not take place.

The article proceeds as follows: first, we discuss two common approaches that prior researchers have used to estimate the influence of program characteristics on impact magnitude when those characteristics are not randomized. Next, we explain how multi-site, multi-armed experiments can eliminate bias in estimates of the effect of randomized program components and, in turn, potentially reduce bias in estimates of the effect of non-randomized program characteristics, via the CAMIC Method. Finally, we offer some evidence on how the CAMIC method performs in simulations, and discuss implications for evaluation research and program practice.

Methods for Analyzing the Contribution of Program Characteristics to Impacts

Two methods have commonly been used in recent years as analytic means for assessing the contribution of program characteristics to impacts in the context of experimentally designed evaluations: (1) a multi-level modelling approach, and (2) an instrumental variables approach. Both of these use an experimental evaluation design, essentially estimating the relationship between the site-level program characteristics and experimental site-level program impacts. This section describes briefly each of these two approaches and then describes how multi-site, multi-armed experiments provide a way to eliminate bias in estimates of randomized program enhancements. Capitalizing on the multi-site, multi-armed design, the next section will discuss the CAMIC approach, a distinct kind of “within-study-comparison” (WSC) opportunity to improve the non-experimental analysis by leveraging experimental results.

Multi-Level Modeling of the Contribution of Program Characteristics to Impacts

To inform how program characteristics contribute to impact magnitude, a number of studies have taken advantage of the naturally occurring cross-site variation in the specific services offered by programs (program components) and in how these services are delivered (implementation features), while controlling for other contextual influences (e.g., Bloom et al., 2001, 2003, 2005; Dorsett & Robins, 2014; Godfrey & Yoshikawa, 2012; Greenberg & Robins, 2011). These analyses commonly use multi-level modeling based on Bryk and Raudenbush (1992) to compute non-experimental estimates of the relationship between site-specific program characteristics and experimentally estimated impact magnitude. This method of relating program characteristics to impact requires the assumption that the model is not subject to omitted variable bias in that there are no site-level variables that are both correlated with the program characteristics included in the model and influence treatment impact magnitude (e.g., Moulton et al., 2014).

This kind of analysis originated in the work of Greenberg et al. (1993), who may have been the first to categorize the questions of interest as being about “prying the lid from the black box.” That is, impact analyses treat “the program” as a “black box,” an unknown space within which varied activities would take place. The black box nature of the characteristics surrounding program implementation begs the questions: which of those activities, and which implementation and contextual factors, are responsible for any observed program impacts? Placing a multi-site evaluation in a multi-level framework provides a means for answering those “black box” questions, at least correlationally (Greenberg et al., 1994).

Using Site-by-Treatment Interactions in an Instrumental Variables Analysis

More recently, similarly motivated analyses have used each of the site-by-treatment interactions that exist in a multi-site experiment as instruments for estimating the contribution of site-level characteristics (e.g., Bos & Granger, 2000; Kling et al., 2007; Magnusson & McGroder, 2002). One key instrumental variables assumption is the exclusion restriction. In this context, the exclusion restriction means that the effect of the program on the outcome must be mediated—and mediated alone—by the program characteristic of interest. However, programs are commonly multifaceted—especially those subject to these “prying the lid from the black box” sorts of questions—which makes it difficult to separate the effects of one program characteristic from the effects of another characteristic using instrumental variables (Gennetian et al., 2002). An additional challenge of using instrumental variables is that, if the impacts of the treatment on the program characteristic do not vary significantly across sites, then the use of multiple instruments may lead to substantially decreased precision and increased finite sample bias (Reardon et al., 2013).

A Multi-Site, Multi-Armed Experiment’s Distinctive Method

Going a step further, the impact evaluation of the first round of Health Profession Opportunity Grants (HPOG 1.0) included design elements that could eliminate bias in estimates of the contribution of specific program characteristics (Peck et al., 2018). The impact evaluation assesses whether providing access to career pathways training for healthcare occupations improves participant outcomes overall while also examining what about the multi-faceted intervention associates with impacts. To do this, eligible individuals were randomized to the HPOG treatment group or to a control group that did not have access to HPOG-funded services. Importantly, in some sites, there were two treatment groups, where one treatment group had access to HPOG while the second treatment group had access to HPOG enhanced with one of three selected program components: (1) emergency assistance for specific needs, (2) noncash incentives designed to encourage desirable program outputs and outcomes, and (3) facilitated peer support groups. This design enabled the researchers to produce experimental estimates of the contribution of each of these enhancements—peer support, emergency assistance, and noncash incentives—to the HPOG Program’s impact.¹

As described further below, the HPOG evaluation design allowed for both experimental and non-experimental evidence on the relative effectiveness of each of these program components. As a result, HPOG involves a special kind of WSC that provides an opportunity for additional methodological learning. The goal of other WSC studies (beginning with LaLonde, 1986) has been to learn which approach to measuring program impact, subject to selection and other sources of bias using observational data, best replicates an experimental finding for the same impact quantity. A series of such studies has begun to point evaluators toward the conditions that yield more-reliable non-experimental findings than other options do (see Cook et al., 2008; Glazerman et al., 2003).²

The CAMIC Method

The CAMIC method was conceived in the HPOG 1.0 Impact Study as a possible way of improving on the Multi-level Modeling of the Contribution of Program Characteristics to Impacts Method by employing a WSC-style approach, thereby reducing bias in non-experimental estimates of the contribution of selected program characteristics to overall impacts (Bell et al., 2017). It does so by identifying the most useful measures to include as covariates in the statistical model.

As depicted by Figure 1, consider the experimental evaluation design, where two sets of sites exist, with respective designs as follows:

• Design in Sites that Randomize Individuals to an Enhanced Treatment Group. Individuals in a subset of sites are randomly assigned to one of three arms: a standard treatment group, an enhanced treatment group (that receives the standard treatment plus an enhancement component), or a control group (that has no access to the program). For example, HPOG’s emergency assistance enhancement added access to emergency funds to meet needs stemming from imminent eviction from housing, utility shutoff, vehicle repair needs, and childcare needs to the standard intervention.

• Design in Sites that Do Not Randomize Individuals to an Enhanced Treatment Group. Individuals in another subset of sites are randomly assigned into one of (at least) two arms: a standard treatment group (where the treatment does not include the program component that was—in other sites—randomized to as an enhancement) and a control group.³

Figure 1.

CAMIC design requirements.

Given this design, the CAMIC method has the potential to reduce bias in cross-site non-experimental estimates of program components and implementation features by completing the following steps.

Step 1

Experimentally estimate the impact of the program enhancement (e.g., the emergency assistance program enhancement) using the sample limited to sites that randomize individuals to an enhanced treatment group using three-armed random assignment (as depicted in the top panel of Figure 1). An experimental estimate of the impact of the program enhancement can be computed in these sites by comparing mean outcomes for individuals randomly assigned to the enhanced treatment group with mean outcomes for individuals randomly assigned to the standard treatment group.

More specifically, we can produce an experimental estimate of the enhancement in these sites by estimating the following model

Y_{j i} = α_{0} + β_{0} {T E}_{j i} + π_{e} E_{j i} + {{v_{j} + μ_{j} {T E}_{j i} + ω_{j} E_{j i} + ε}_{j i}}

(1)

In this equation, Y is the outcome of interest for individual i in site j; ${T E}_{j i}$ is an overall treatment indicator which denotes whether the individual was assigned to either the standard treatment or enhanced treatment; and $E_{j i}$ is an indicator for whether the individual was assigned to the enhanced treatment group. The model also includes a series of error terms. See Table 1 for definitions of the terms included in these equations and all models presented throughout the manuscript.

Estimating equation (1) through linear regression, we obtain—among other things—an estimate ${\hat{π}}_{e}^{X}$ of $π_{e}$ straight from the experiment, based on purely random variation in which individuals receive a program that includes the enhancement component e and which individuals receive a program that does not. The X superscript denotes the unbiased experimental nature of this estimate.

Table 1.

Definition of Model Terms.

Term	Definition
Outcome and covariates
$Y_{j i}$	The outcome measure for individual i from site j
${T E}_{j i}$	The treatment group indicator (1 for those individuals assigned to the standard treatment or enhanced treatment group, and 0 for those individuals assigned to the control group; this is labeled “TE” for the combination of standard “treatment” and “enhanced” treatment)
$E_{j i}$	The enhanced treatment group indicator (1 for those individuals assigned to the enhanced treatment group, and 0 otherwise; this is labeled “E” for “enhanced” treatment)
$P_{e j}$	An indicator for whether the enhancement component was included as part of the program in site j. In this example, $P_{e j}$ = 1 in three-arm sites and = 0 in two-arm sites.
$P_{m j}$	Program characteristic m for site j, m = 1,. . ., M (these are labeled “P” for “program”)
$X_{n j}$	Additional site-level covariate n for site j, n = 1,. . ., N. These site-level covariates are included in the model to reduce bias in other site-level measures of interest, and may include program characteristics of secondary interest and local context measures (e.g., the unemployment rate in the area where the site is located).
Model coefficients
$α_{0}$	The grand mean control group outcome
$β_{0}$	The grand mean impact of the standard treatment
$π_{e}$	The grand mean impact of being offered the enhanced intervention inclusive of component e, rather than the standard intervention without e
$π_{m}$	The influence of program characteristic m on impact magnitude, m = 1,. . ., M
$φ_{n}$	The influence of bias-reducing covariate n on impact magnitude, n = 1,. . ., N
$κ_{n}$	The influence of the bias-reducing covariate on the control group mean outcome.
Error terms
$ε_{j i}$	A random component of the outcome for each individual
$μ_{j}$	A random component of the standard intervention impact for each site
$v_{j}$	A random component of the mean outcome for each site
$ω_{j}$	A random component of the enhanced intervention’s incremental impact for each site

Step 2

In this step, we calculate several non-experimental estimates of the contribution of the program enhancement to impact magnitude using cross-site variation in whether the enhancement is included in a given site’s program (this follows the Multi-level Modeling of the Contribution of Program Characteristics to Impacts Method from above). To produce this estimate, the sample is limited to (1) the control and enhanced treatment arms from sites that conducted three-armed random assignment using the enhancement and (2) the control and standard treatment arms from sites that did not randomize to the enhancement.⁴ We use variations of the following model to produce non-experimental estimates of the contribution of the enhancement component to impact magnitude

Y_{j i} = α_{0} + β_{0} {T E}_{j i} + π_{e} P_{e j} {T E}_{j i} + \sum_{m} π_{m} P_{m j} {T E}_{j i} + \sum_{n} φ_{n} X_{n j} {T E}_{j i} + \sum_{n} κ_{n} X_{n j} + {v_{j} + μ_{j} {T E}_{j i} + ε_{j i}}

(2)

In this equation, Y is the outcome of interest for individual i in site j and ${T E}_{j i}$ is an overall treatment indicator which denotes whether the individual was assigned to either the standard treatment or enhanced treatment. $P_{e j}$ is an indicator for whether the enhancement component (e.g., emergency assistance) was included as part of the program in site j (i.e., $P_{e j}$ equals 1 in three-arm sites and equals 0 in two-arm sites); $P_{m j}$ captures all other key program charactersitics of interest which the researcher wants to ensure are included in the model subject to degrees of freedom limitations (e.g., the researchers may be interested in the extent to which tuition is covered by the program affects impact magnitude, even though tuition assistance is not offered as a randomized enhancement); and $X_{n j}$ captures all other site-level covariates that could be included in the model to reduce bias (e.g., the unemployment rate).⁵ These site-level program characteristics are all multiplied by the treatment indicator, and the resulting interaction terms capture the influence of a given site-level measure on the magnitude of the program’s impact. The model also includes a series of error terms, as defined in Table 1.

Estimating equation (2) through linear regression, we obtain a non-experimental estimate of the influence of the enhancement component on impact magnitude ${\hat{π}}_{e}^{N}$ of $π_{e}$ using cross-site variation in whether the enhancement is part of the program in a given site. The N superscript denotes the non-experimental nature of this estimate which may be subject to bias. Additionally, we obtain non-experimental estimates of the influence of other key program characteristics of interest ${\hat{π}}_{1}^{N}$ ,..., ${\hat{π}}_{M}^{N}$ .

Bias in the estimates of the influence of the enhancement component $P_{e j}$ and the influence of other key program characteristics $P_{m j}$ will arise when these program characteristics are correlated with omitted site-level factors that also influence impacts. For example, in the case of the HPOG impact evaluation these factors—often omitted because they are unobserved in the data—may include aspects of the program that administrators choose (e.g., an unmeasured program component or implementation feature), aspects of the program context that are beyond their control (e.g., the local unemployment rate), and other unobserved factors that influence both the program components and implementation features and the impacts of the program (e.g., the raw talent of the program leadership). For all of these types of unobserved factors, if the program characteristics of interest are correlated with unobserved factors, then the estimated influence of those characteristics will reflect the influence of the unobserved factors (Bell et al., 2017). Appendix A details the circumstances that can cause omitted-variable bias in estimates of the contributions of program characteristics to site-level impacts.

Importantly, in this step, we produce several different estimates of the non-experimental estimate of the influence of the enhancement component on impact magnitude ${\hat{π}}_{e}^{N}$ (and ${\hat{π}}_{1}^{N}$ ,..., ${\hat{π}}_{M}^{N}$ ) by varying the set of bias-reducing site-level covariates $X_{n j}$ included in the model, given degrees of freedom limitations.⁶

Step 3

The next step in implementing the CAMIC method is to identify the version of the equation (2) model from Step 2 that minimizes the difference between the experimental estimate of the program enhancement’s effect computed in Step 1 and non-experimental estimate of the program enhancement’s effect computed in Step 2 (using the best “similarity” metric available from WSC scholarship; e.g., Steiner & Wong, 2018), where bias is measured subject to sampling variability as $| {\hat{π}}_{e}^{N} - {\hat{π}}_{e}^{X} |$ .

Step 4

Apply the “best” model from Step 3 to estimate the contribution of other program characteristics of interest $P_{m j}$ (when these other components or features do not vary randomly among individuals within sites or across sites) to assess their contribution to overall impacts with potentially reduced bias.

CAMIC Method Performance

Bell et al. (2017) report on simulations that investigate whether the CAMIC method is expected to accomplish its goal of reducing bias in non-experimental estimates of the influence of program characteristics on a program’s total impact. The simplified framework used for the simulations focuses on the site-level relationship between the true impact of a program ( $∆$ ) and three program components ( $P_{1}, P_{2}, P_{3}$ ), all of which are correlated with an unobserved, site-level factor ( $μ$ ).⁷ This relationship is given by

∆_{j} = π_{0} + π_{1} P_{1 j} + π_{2} P_{2 j} + π_{3} P_{3 j} + μ_{j} + ε_{j},

(3)

where

• $∆_{j}$ is the impact of the program implemented by site j.

• $π_{0}$ is the mean impact of the program.

• $P_{1}$ is a continuous measure of the extent to which program $j$ implemented an “enhancement component” for which we can obtain an experimental estimate of its influence on impact, as depicted by $P_{e j}$ in equation (2).

• $P_{2}$ is a continuous measure of the extent to which program $j$ implemented a non-randomized “focal component” for which we are interested in minimizing bias, as depicted by $P_{m j}$ in equation (2).

• $P_{3}$ is a continuous measure of a “bias-reduction component,” the effect of which is not directly of interest, but can be included in the model to potentially reduce bias in estimates of the influence of the non-randomized focal component. $P_{3}$ is depicted by $X_{n j}$ in equation (2).

• $π_{m}$ is the influence of program component $m$ on the program’s impact.

• $μ_{j}$ is a site-specific, unobserved factor that may be correlated with observed program components, as depicted by $F_{j}$ in Appendix A.

• $ε_{j}$ is an error term unrelated to the observed program components.

As is the setting for the CAMIC method’s potential application, suppose we observe an unbiased measure of the true value of the enhancement component $π_{1}$ and that our goal is to obtain an estimate of focal component $π_{2}$ . Then, the CAMIC method uses the unbiased estimate of $π_{1}$ to select the least biased specification between the following models

\begin{array}{c} M o d e l 1 : & ∆_{j} = π_{0}^{1} + π_{1}^{1} P_{1 j} + π_{2}^{1} P_{2 j} + π_{3}^{1} P_{3 j} + u_{j}^{1} \end{array}

(4)

\begin{array}{c} M o d e l 2 : & ∆_{j} = π_{0}^{2} + π_{1}^{2} P_{1 j} + π_{2}^{2} P_{2 j} + u_{j}^{2} \end{array}

(5)

As discussed above and shown in Appendix A, bias arises in non-experimental cross-site estimates of the influence of program characteristics on impact magnitude when the program characteristics are correlated with site-level factors that also influence impacts but that are omitted from the analysis. Similarly, in this simplified framework the key parameters that determine the bias are the correlations among the three observed program characteristics (the enhancement component, focal component, and the bias-reduction component), the correlation between each observed program characteristic with the unobserved factor, and the true influence of the bias-reduction component on impact. To calculate bias, one must select a value for each of these seven parameter values.8^,9^,10

To understand the CAMIC method’s potential to reduce bias in our estimate of the focal component $π_{2}$ , we seek to understand whether the least biased specification for the enhancement component $P_{1}$ is also the least biased specification for the focal component $P_{2}$ . Bell et al. (2017) first identified possible values for the correlation between the enhancement component, focal component, and the bias-reduction component and referred to each combination of correlations in these components as a scenario. Then, for each scenario, they ran 500 simulations to capture a broad range of values of the unobserved parameters. They calculated the proportion of these simulations that are favorable to the CAMIC method for a particular scenario. A simulation is considered favorable to the CAMIC method if the model specification that produced the least biased estimate of the enhancement component (whether it be model 1 or model 2) is also the model that produced the least biased estimate for the non-randomized focal component of interest and unfavorable otherwise. This structure allows researchers to consider which of the scenarios is most similar to the correlations in observed program components they might observe in their data.

Table 2 presents an excerpt from Bell et al.’s (2017) Exhibit 9, and it reveals the following: when the correlation between the enhancement component and the focal component is high (ρ = 0.70) and the correlation between each of these two components and the bias-reduction component is low (ρ = 0.25) to moderate (ρ = 0.50), the CAMIC method performs favorably in more than 80% of the simulations (Scenarios A and B in Table 2). In contrast, when the correlations are high (ρ = 0.70) across the board (as is the case for Scenario C) CAMIC performs favorably in only 29% of the simulations. When the correlation between the enhancement component and the focal component is low (ρ = 0.25) to moderate (ρ = 0.50), the findings are inconclusive in that the CAMIC method sometimes performs favorably and sometimes performs unfavorably (Scenarios D through M).

These findings imply that the CAMIC approach—calibrating the model specification by comparing experimental and non-experimental estimates of the enhancement component—can reduce bias (and improve upon standard practice) in some other, non-randomized focal component only when there is a meaningful correlation between the enhancement component and the focal component. Without that correlation, CAMIC cannot be expected to do any better than standard practice. Further, good performance of the CAMIC method requires that the correlation between each of these two components and a third bias-reduction component is low (ρ = 0.25) to moderate (ρ = 0.50). If the intercorrelations among the enhancement component, the focal component, and the additional bias-reduction component are all high (ρ = 0.70), the CAMIC method performs poorly as it may suffer from multicollinearity.

Table 2.

Results for Selected Scenarios, Focused on the Relative Magnitude of the Correlations.

	Correlations between…			Of the 500 simulations: Favorable to the CAMIC method
Scenario	Enhancement component and focal component ( $ρ_{12}$ )	Enhancement component and bias-reduction component ( $ρ_{13}$ )	Focal component and bias-reduction component ( $ρ_{23}$ )	N	Percent
Scenarios with high correlation between the enhancement component and focal component that are favorable to the CAMIC method
A	0.70	0.50	0.25	416	83
B	0.70	0.25	0.50	416	83
Scenarios with high correlation between the enhancement component and focal component that are unfavorable to the CAMIC method
C	0.70	0.70	0.70	144	29
Scenarios with low to moderate correlation between the enhancement component and focal component are inconclusive
D	0.25	0.25	0.25	200	40
E	0.25	0.50	0.25	200	40
F	0.25	0.25	0.50	204	41
G	0.25	0.50	0.50	238	48
H	0.25	0.70	0.50	272	54
I	0.25	0.50	0.70	272	54
J	0.50	0.25	0.25	156	31
K	0.50	0.50	0.25	214	43
L	0.50	0.25	0.50	274	55
M	0.50	0.50	0.50	188	38

Source: Excerpt from Bell et al. (2017), Exhibit 9.

In advance of implementing the CAMIC method, these observations suggest that researchers compute the correlation between the enhancement component and the focal component of interest to ensure that these two program characteristics are highly correlated.¹¹ If this criterion is satisfied, then the researcher could deploy the CAMIC method, calibrating the model specification using only those additional bias-reduction components that have a low (ρ = 0.25) to moderate (ρ = 0.50) correlation with the enhancement component and the focal component of interest.

These simulation findings ignore the role that variance might play in the CAMIC method. Although the CAMIC method seeks to select the model that yields the least biased estimate of the non-experimental estimate of the enhancement component, noise in the estimates may result in selecting the wrong model. This limitation of the CAMIC method is not reflected in the simulations, where perhaps further analysis or application is needed.

Discussion and Conclusion

This article has discussed the problem of bias that arises when estimating the influence of program characteristics that have not been randomized, but instead are selected and implemented in ways and in contexts that correlate with their relative contributions to overall impacts. Both multi-level analytic methods and instrumental variable estimation are approaches that analysts use to pry the lid from the black box—to use Greenberg et al.’s (1993) phrase—and attempt to ascertain what it is about a program that is responsible for its impacts.

Beyond these analytic approaches, recent years have seen an increase in the numbers of experimental evaluations that include both multiple sites and multiple treatment arms. Through design, if individuals are randomly assigned to either a standard treatment group or an enhanced treatment group that receives the standard treatment plus a program enhancement, then the bias in the estimate of the randomized program enhancement can be eliminated. Advancing further still, this kind of design offers additional analytic opportunities for bias reduction in the non-experimental estimation of the contribution of other non-randomized program characteristics to overall impacts.

Past examples of evaluations that share the multi-site plus multi-arm experimental design include the following. The U.S. Department of Health and Human Services-funded National Evaluation of Welfare-to-Work Strategies (NEWWS) covered many sites, where some sites included three-armed experiments testing competing models of welfare reforms and other sites included an enhanced treatment arm designed to estimate the contribution of case management to the treatment’s overall impact (Hamilton et al., 2001). Similarly, the U.S. Social Security Administration’s Benefit Offset National Demonstration (BOND) also included—in its national, multi-site design—a third treatment arm that randomized eligible individuals within a subset of the study sample to have access to support services (Gubits et al., 2014). The U.S. Department of Housing and Urban Development’s First-Time Homebuyer Education and Counseling Demonstration also randomized to a control group and two treatment groups, with about 5800 individuals randomized across 28 metropolitan areas, with service access to 63 local and two national service providers (DeMarco et al., 2017). These studies and others with similar designs—randomized treatments, many sites, data on varying design and implementation—have the potential to use the CAMIC method to explore bias reduction in estimates of non-randomized program features, thereby improving understanding of why or how an intervention had its observed effect. Going forward, we encourage researchers to do so, where fitting.

This article contributes to the literature in two main ways. First, it explores the use of a multi-site, multi-armed experimental design as a way to produce unbiased estimates of an enhancement component. In addition, it suggests a method that could reduce bias in the cross-site non-experimental estimates of non-randomized program characteristics, in the case where the study design includes three-armed randomization that allows for experimental analysis of the contribution of at least one specific program characteristic. Over time, CAMIC method-based tests of cross-site impact attribution specifications may yield consistent results with enough replications. Given the investment needed to carry out this kind of evaluation design (with both multiple arms and multiple sites), we recognize that it will take time to amass replicate evidence of the influence of program components on impacts. Certainly as the state of the science of within-study comparisons improves as well, we should gain knowledge regarding the calibration of non-experimental to experimental results.

The evolution of CAMIC as a method among those in the evaluation toolkit has implications both for evaluation research and for program practice. Future evaluation research should consider opportunities to test CAMIC in settings where the design is fitting: multi-site, multi-armed experiments, where at least a subset of the sites randomize individuals to either a standard treatment group, an enhanced treatment group (that receives the standard treatment plus a program “enhancement” or modification), or a control group. This design allows the researcher to compute both experimental and non-experimental estimates of the influence of the program enhancement, and CAMIC may be able to leverage that evidence in order to improve the estimation of the contribution of other non-randomized program characteristics of interest. The CAMIC method is likely to achieve its promise of reducing bias in estimates of the relationship between a focal/non-randomized program component and program impact only when the correlation between the randomized program enhancement and the focal/non-randomized program component of interest is high and the correlation between each of these two components and alternative bias-reducing covariates are low to moderate.

One important consideration for researchers interested in using data from multi-site randomized experiments to estimate the influence of site-level program characteristics on program impacts is the sample size at the site-level (Moulton et al., 2014). Because it can be expensive to conduct evaluations in many varied locations, most past experiments have been confined to a small number of sites. Having a small number of sites severely limits researchers’ ability to conduct cross-site comparisons due in part to lack of statistical power. Additionally, having a small number of sites increases the potential for omitted variable bias of non-experimental estimates due to the need to leave out key site-level variables in order to conserve degrees of freedom. Acknowledging that the minimum sample size required to achieve a given level of statistical power depends on a number of factors, including the expected effect size and the amount of measurement error, Green (1991) suggests that the sample size for regression analysis should be at least 50 + 8 m, where m is the number of independent variables (VanVoorhis & Morgan, 2007). This limitation on the number of site-level predictors included implies that analysts need to be selective in which site-level measures enter into the analytic model; and the CAMIC method provides one way to be selective.

Another consideration when designing a multi-site evaluation is the cost and related tradeoffs between competing design and analytic options. We do not recommend that the potential application of the CAMIC method be the reason that an evaluator decides to implement a multi-site, multi-armed experimental design. Instead, the policy questions are what should determine the design. In the situation where an evaluation is already designed to meet the multi-site, multi-armed criteria that CAMIC requires, the added cost of applying CAMIC is likely to be marginal. CAMIC can be thought of as another tool for “within-study-comparison” and could be a promising low-cost method for choosing an improved model specification for non-experimental analyses that inform why or how an intervention works.

Indeed, to the extent that the CAMIC method helps reduce bias in estimates of the influence of non-randomized program characteristics on impact magnitude in real-world applications, CAMIC also has implications for program practice: program managers can expect more reliable estimates of the relative contributions of program characteristics to overall impacts. For example, the proliferation of this line of analysis in practice means that the field should be able to learn how programs operate differently in strong versus weak economic conditions, which aspects of multi-faceted programs are important drivers (or suppressors) of impact, which implementation features are essential to a program’s success, and how participant compositional factors influence (or not) programs’ effectiveness. These are practical lessons that inform administrative decisions on the ground as well as higher-level policy decisions about what to fund.

Footnotes

Acknowledgments

The authors are grateful input on this draft from Larry Buron and Eleanor Harvill at Abt Associates, and from Nicole Constance, Hilary Bruck, and Amelia Popham at ACF/OPRE. Eleanor Harvill and Stephen Bell are co-authors on a prior iteration of this work, and we are grateful to their input on developing and testing the CAMIC method. The views expressed in this article do not necessarily reflect the views or policies of the HHS/ACF/OPRE.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the U.S. Department of Health and Human Services (HHS), Administration for Children and Families (ACF) (No. HHSP23320095624WC), Office of Planning, Research and Evaluation (OPRE).

ORCID iD

Laura R. Peck

Notes

Source of Omitted-Variable Bias in Non-Experimental Cross-Site Estimates of the Influence of Program Characteristics

Estimating the equation (2) model will produce estimates of the $π_{m}$ that capture the relationship between the influence of program characteristic $P_{m}$ on impact magnitude, for m = 1,…, M. Omitted-variable bias arises in this analysis if a site-level factor F_j exists that (a) influences impact magnitude, (b) is not included as a covariate in equations (2), and (c) correlates with one or more of the program characteristics $P_{m j}$ .

To see how bias arises, consider the following version of equation (2) that includes a term for the omitted site-level factor F_j and the coefficient $λ$ captures the influence of the omitted factor on impact magnitude (see Table 1 for definitions of the other terms included in this equation) (2)

Y_{j i} = α_{0} + β_{0} {T E}_{j i} + π_{e} P_{e j} {T E}_{j i} + \sum_{m} π_{m} P_{m j} {T E}_{j i} + \sum_{n} φ_{n} X_{n j} {T E}_{j i} + {λ F}_{j} {T E}_{j i} + {v_{j} + μ_{j} {T E}_{j i} + ε_{j i}}

When this equation is estimated with maximum likelihood methods (Bryk & Raudenbush, 1992) with $F_{j} {T E}_{j i}$ omitted as in equation (2), the estimates of the contribution of every program characteristic $P_{m}$ to impact magnitude are asymptotically biased. As an example, it can be shown that the asymptotic expectation of ${\hat{π}}_{m}$ in the special case where $F_{j}$ correlates with program component $P_{r j}$ but not with any of the other right-hand-side variables in equation (2ʹ), is

p l i m ({\hat{π}}_{r}) = π_{r} + λ \frac{C o v (P_{r j} {, F}_{j})}{V a r (P_{r j})},

(see Wooldridge, 2002, pp. 61–62). This equation aligns with the usual econometric formula for omitted-variable bias, although it gives the relationship at the limit as sample size goes to infinity. The bias of

{\hat{π}}_{r}

can be derived from this probability limit as

b i a s ({\hat{π}}_{r}) = p l i m ({\hat{π}}_{r}) {- π}_{r} = λ \frac{C o v (P_{r j} {, F}_{j})}{V a r (P_{r j})},

because Cov(P_rj, F_j ) ≠ 0,

{\hat{π}}_{r}

from the mis-specified equation (2) model is a biased estimate of

π_{r}

even in very large samples, with bias that approaches

λ \frac{C o v (P_{r j} {, F}_{j})}{V a r (P_{r j})}

as sample size goes to infinity. The same argument can be made for all of

{\hat{π}}_{1}

,...,

{\hat{π}}_{M}

References

Bell

S. H.

Harvill

E. L.

Moulton

S. R.

Peck

(2017). Using within-site experimental evidence to reduce cross-site attributional bias in connecting program components to program impacts, OPRE report #2017-13. Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Bloom

Hill

C. J.

Riccio

J. A.

(2005). Modeling cross-site experimental differences to find out why program effectiveness varies. In Bloom

H. S.

(Ed.), Learning more from social experiments (pp. 37–74). Russell Sage Foundation.

Bloom

H. S.

Hill

C. J.

Riccio

J. A.

(2001). Modeling the performance of welfare-to-work programs: The effects of program management and services, economic environment, and client characteristics. Manpower Demonstration Research Corporation.

Bloom

H. S.

Hill

C. J.

Riccio

J. A.

(2003). Linking program implementation and effectiveness: Lessons from a pooled sample of welfare-to-work experiments. Journal of Policy Analysis and Management, 22(4), 551–575. https://doi.org/10.1002/pam.10154

Bos

Granger

(2000). Estimating the effects of day care use on children’s school readiness: Evidence from the new chance demonstration. Manpower Demonstration Research Corporation.

Bryk

Raudenbush

(1992). Hierarchical linear models: Applications and data analysis methods. Sage.

Cook

T. D.

Shadish

W. R.

Wong

V. C.

(2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27(4), 725–750. https://doi.org/10.1002/pam.20375

DeMarco

Fiore

Moulton

Bocian

D. G.

Peck

Ballinger

Cuenca

Rothschild

Stephen

(2017). The first-time homebuyer education and counseling demonstration baseline report: Study design and implementation. U.S. Department of Housing and Urban Development, Office of Policy Development and Research.

Dorsett

Robins

P. K.

(2014). A multilevel analysis of the impacts of services provided by the U.K. employment retention and advancement demonstration. Evaluation Review, 37(2), 63–108. https://doi.org/10.1177/0193841X13517383

10.

Gennetian

L. A.

Bos

J. M.

Morris

P. A.

(2002). Using instrumental variables analysis to learn more from social policy experiments. Manpower Demonstration Research Corporation.

11.

Glazerman

Levy

D. M.

Myers

(2003). Nonexperimental versus experimental estimates of earnings impacts. The Annals of the American Academy of Political and Social Science, 589(1), 63–93. https://doi.org/10.1177/0002716203254879

12.

Godfrey

E. B.

Yoshikawa

(2012). Caseworker recipient interaction: Welfare office differences, economic trajectories, and child outcomes. Child Development, 83(1), 382–398. https://doi.org/10.1111/j.1467-8624.2011.01697.x

13.

Green

S. B.

(1991). How many subjects does it take to do a regression analysis? Multivariate Behavioral Research, 26(3), 499–510. https://doi.org/10.1207/s15327906mbr2603_7

14.

Greenberg

Meyer

R. H.

Wiseman

(1993). Prying the lid from the black box: Plotting evaluation strategy for welfare employment and training programs. University of Wisconsin-Madison, Institute for Research on Poverty.

15.

Greenberg

Meyer

R. H.

Wiseman

(1994). Multi-site employment and training evaluations: A tale of three studies. Industrial and Labor Relations Review, 47(4), 679–691. https://doi.org/10.1177/001979399404700411

16.

Greenberg

Robins

P. K.

(2011). Have welfare-to-work programs improved over time in putting welfare recipients to work? Industrial and Labor Relations Review, 64(5), 910–920. https://doi.org/10.2307/41343706

17.

Gubits

Lin

Bell

S. H.

Judkins

(2014). BOND implementation and evaluation: First- and second-year snapshot of earnings and benefit impacts for stage 2. (Deliverable 24c.5) U.S. Social Security Administration.

18.

Hamilton

Freedman

Gennetian

Michalopoulos

Walter

Adams-Ciardullo

Gassman-Pines

(2001). National evaluation of welfare-to-work Strategies: How effective are different welfare-to-work approaches? Five-year adult and child impacts for eleven programs. U.S. Department of Health and Human Services, Administration for Children and Families, Office of the Assistant Secretary for Planning and Evaluation and U.S. Department of Education, Office of the Deputy Secretary, Planning and Evaluation Service, Office of Vocational and Adult Education.

19.

Kling

J. R.

Liebman

J. B.

Katz

L. F.

(2007). Experimental analysis of neighborhood effects. Econometrica, 75(1), 83–119. https://doi.org/10.1111/j.1468-0262.2007.00733.x

20.

LaLonde

(1986). Evaluating the econometric evaluations of training with experimental data. American Economic Review, 76(4), 604–620.

21.

Magnusson

K. A.

McGroder

(2002). The effects of increasing welfare mothers’ education on their young children’s academic problems and school readiness. Northwestern University.

22.

Moulton

Bell

S. H.

Peck

L. R.

(2014). Social policy impact pathfinder (SPI-path) analytic suite: SPI-Path|Site user guide. Abt Associates Inc.

23.

Peck

L. R

Hilary

Nicole

(2022). Insights from the health profession opportunity grant program’s three-armed, multi-site experiment for policy learning and evaluation practice. Evaluation Review. Advance online publication. https://doi.org/10.1177/0193841X221133420

24.

Peck

L. R

Werner

Harvill

Litwok

Moulton

Fountain

A. R.

Locke

(2018). Health profession opportunity grants (HPOG 1.0) impact study interim report: Program implementation and short-term impacts, OPRE report #2018–16a. Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

25.

Reardon

S. F.

Unlu

Zhu

Bloom

H. S.

(2013). Bias and bias correction in multi-site instrumental variables analysis of heterogeneous mediator effects. Journal of Educational and Behavioral Statistics, 36(1), 53–86. https://doi.org/10.3102/1076998613512525

26.

Steiner

P. M.

Kim

(2016). The mechanics if omitted variable bias: Bias amplification and cancellation of offsetting biases. Journal of Causal Inference, 4(2), 20160009. https://doi.org/10.1515/jci-2016-0009

27.

Steiner

P. M.

Wong

V. C.

(2018). Assessing correspondence between experimental and nonexperimental estimates in within-study comparisons. Evaluation Review, 42(2), 214–247. https://doi.org/10.1177/0193841X18773807

28.

VanVoorhis

C. R. W.

Morgan

B. L.

(2007). Understanding power and rules of thumb for determining sample sizes. Tutorials in Quantitative Methods for Psychology, 3(27), 43–50. https://doi.org/10.20982/tqmp.03.2.p043

29.

Wong

V. C.

Steiner

P. M.

Anglin

K. L.

(2018). What can be learned from empirical evaluations of nonexperimental methods? Evaluation Review, 42(2), 147–175. https://doi.org/10.1177/0193841X18776870

30.

Wooldridge

J. M.

(2002). Econometric analysis of cross section and panel data. The MIT Press.