Insights on Variance Estimation for Blocked and Matched Pairs Designs

Abstract

Evaluating blocked randomized experiments from a potential outcomes perspective has two primary branches of work. The first focuses on larger blocks, with multiple treatment and control units in each block. The second focuses on matched pairs, with a single treatment and control unit in each block. These literatures not only provide different estimators for the standard errors of the estimated average impact, but they are also built on different sets of assumptions. Neither literature handles cases with blocks of varying size that contain singleton treatment or control units, a case which can occur in a variety of contexts, such as with different forms of matching or poststratification. In this article, we reconcile the literatures by carefully examining the performance of variance estimators under several different frameworks. We then use these insights to derive novel variance estimators for experiments containing blocks of different sizes.

Keywords

causal inference potential outcomes precision finite sample inference randomization inference Neymanian inference

1. Introduction

Beginning with Neyman and Fisher, there is a long literature of analyzing randomized experiments by focusing on the assignment mechanism rather than some generative model of the data. One major family of experimental designs in this literature is the blocked randomized experiment, where units are grouped to hopefully create homogenous collections, and then treatment assignment is randomized within each group (see Fisher, 1926). Ideally, this process gives a higher precision estimate of the overall average treatment effect as compared to a completely randomized design.

In the potential outcome causal literature (as in Imbens & Rubin, 2015; Rosenbaum, 2010),¹ much of the prior work on randomized experiments has focused on two forms of blocking: blocking where there are several treated and control units in each block and blocking where there is exactly one treated and one control unit in each block (matched pairs). See, for example, Imai et al. (2008) or Imbens (2011) for treatments of large blocks and Abadie and Imbens (2008) or Imai (2008) for treatments of matched pairs. This literature, for the most part, has a gap: It has not extensively treated the cases where researchers have generated groups of varying size but where there is still only one treated and/or one control in some of the blocks, which we call the “hybrid design.” Recent textbooks such as Field Experiments: Design, Analysis and Interpretation (Gerber & Green, 2012) and Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction (Imbens & Rubin, 2015) do not propose a clear answer for Neyman-style variance estimation in this case. While obtaining a point estimate for the overall average treatment effect is straightforward in this context, assessing the uncertainty of such an estimate is not. Currently one would instead have to turn to Fisher-style permutation tests, which typically rely on constant treatment effect assumptions, or regression-based approaches, which can be biased and usually require assumptions as to the residual error structure. We build on prior work to fill this gap by providing novel methods for conducting Neyman-style analyses for this more general hybrid design. The approach to causal inference used in this work also has strong connections to the survey sampling literature, as treated in, for example, Särndal et al. (2003) or Cochran (1977).

This gap is important as hybrid experiments with blocks of different sizes, and different numbers of treated and control units within the blocks, can easily arise in many modern social science experiments. For example, multisite trials in education often have several sites (e.g., districts) with only a few schools in each site. Many matching methods used in observational studies generate hybrid designs as well. For instance, coarsened exact matching (CEM; Iacus et al., 2012) can lead to many variable-sized blocks, some of which have singleton treatment or control units. “Full matching,” which identifies collections of units that are similar on some baseline covariates (Hansen, 2004; Rosenbaum, 1991), creates variable-sized blocks, each with exactly one treated or one control unit. Our approach allows for a Neyman-style analysis in these contexts. See Section 6 for more on these applications.

There are several different models used for Neyman-style causal inference. The first, the finite sample model, takes the sample of units in the experiment as fixed, using the assignment mechanism as the sole source of randomness. Other so-called super-population or population models assume that the units in the experimental sample come from some larger population; this can induce additional uncertainty that needs to be accounted for. With blocking, there is the further complication of how the blocks in the final experimental sample are formed. There can be fixed blocks in which every unit inherently belongs to one of a finite number of blocks; flexible blocks made by the experimenter once a sample is obtained; and structural blocks that capture natural groupings of units. There are also several possible sampling mechanisms beyond the classic simple random sampling of units typically presumed, such as sampling from strata corresponding to the blocks or sampling entire blocks rather than individual units. We believe these variants in how blocks are formed and sampled has caused the gap of the hybrid design: Because much of the current literature uses different frameworks tailored to the specific special cases of either large blocks or matched pairs, it is not easily extended as the variance and variance estimators differ across these variants. As part of our work we carefully outline the common frameworks used and discuss how they are different from each other and how they connect to different types of blocks. We also analyze the performance of uncertainty estimation for all cases.

Recent work by Fogarty (2018) has also addressed some of these issues. In particular, Fogarty presents a method for estimating variance with small blocks of variable size, not just matched pairs. His estimators share some similarities with ours, though they are distinct and we note the difference in bias in Section 3.4. He also makes explicit the issue of differing results under different population and sampling frameworks by comparing multiple settings. In our article, we tackle the issue of creating a cohesive hybrid estimator for experiments with large and small blocks and do not focus on the use of covariates to model treatment effect heterogeneity.

In Section 2, we set out our notation and discuss blocked randomization. We begin with the finite sample framework because it is a building block for the infinite population frameworks. Section 3 provides methods for estimating uncertainty in the case of large blocks, small blocks, and the hybrid of the two and gives their bias under the finite sample framework. We then, in Section 4, provide true variance formula and the performance characteristics of the variance estimators for several infinite population frameworks. Section 5 contains finite sample simulation studies to illustrate estimator performance, and Section 6 illustrates estimation in two data examples. For clarity in presentation, we have moved the derivations of provided formulae to the Supplementary Material, available in the online version of the journal. To use these methods in practice, we refer readers to our R package, blkvar (Miratix & Pashley, 2020, on github). Sample scripts demonstrating its use and replicating our simulations are also available at https://scholar.harvard.edu/lmiratrix/insights.

2. Overall Setup and Notation

We use the Neyman–Rubin model of potential outcomes (Rubin, 1974; Splawa-Neyman et al., 1923/1990). We assume the Stable Unit Treatment Value Assumption of no differential forms of treatment and no interference between units (Rubin, 1980). Consider an experimental sample of n units. In a completely randomized experiment, the entire collection of the units is divided into a treatment group and a control group by taking a simple random sample of $p n$ units as the treatment group and leaving the remainder as control. In a blocked randomized experiment, our sample is divided into K blocks, formed based on some pretreatment covariate(s), with n_k units in block k. Each block k is then treated as a mini-experiment, with a fixed number of $p_{k} n_{k}$ units being randomly assigned to treatment and the rest to control, independently of the other blocks.

The sample average treatment effect (SATE) is the typical estimand in so-called finite sample inference, which takes our sample as fixed, leaving the assignment mechanism as the only source of randomness. Under blocking, the SATE within block k, for $k = 1, . . ., K$ , is

τ_{k, S} = \frac{1}{n_{k}} \sum_{i : b_{i} = k} (Y_{i} (t) - Y_{i} (c)),

where $Y_{i} (t)$ and $Y_{i} (c)$ are the potential outcomes for unit i under treatment and control, respectively, and where b_i indicates the block that unit i belongs to. The overall SATE (see Imbens & Rubin, 2015, p. 86) is then

τ_{S} = \frac{1}{n} \sum_{i = 1}^{n} (Y_{i} (t) - Y_{i} (c)) .

In this work, we consider two estimators for the SATE (and later the population average treatment effect [PATE]), one typically used for complete randomization and one for blocked randomization. Define the variable Z_i as $Z_{i} = t$ if unit i is assigned treatment and $Z_{i} = c$ if unit i is assigned control, for $i = 1, . . ., n$ . Let $I_{Z_{i} = t}$ be the indicator that unit i received treatment, n_t be the total number of treated units, and n_c be the total number of control units. So, $n_{t} = \sum_{i = 1}^{n} I_{Z_{i} = t}$ , $n_{c} = n - n_{t}$ . Similarly, let $n_{t, k}$ , $n_{c, k}$ indicate these values within block k. Define $Y_{i}^{obs} = Y_{i} (Z_{i})$ as the outcome we observe for unit i given a specific treatment Z_i. The blocked randomization estimator is then a weighted average of simple difference estimators for each block

{\hat{τ}}_{(B K)} = \sum_{k = 1}^{K} \frac{n_{k}}{n} {\hat{τ}}_{k},

with the

{\hat{τ}}_{k} = \frac{1}{n_{t, k}} \sum_{i : b_{i} = k} I_{Z_{i} = t} Y_{i} (t) - \frac{1}{n_{c, k}} \sum_{i : b_{i} = k} (1 - I_{Z_{i} = t}) Y_{i} (c),

$k = 1, . . ., K$ , being simple difference estimators within each block.

In general, ${\hat{τ}}_{(B K)}$ is unbiased, with

E [{\hat{τ}}_{(B K)} | S] = τ_{S},

with $E [\hat{M} | S]$ the expected value of some estimator $\hat{M}$ for a given, fixed, finite sample $S$ over the blocked randomization. It is describing and estimating the variance of ${\hat{τ}}_{(B K)}$ that is more tricky. This assessment is the goal of this article, but first we need to introduce a few more useful concepts.

An important aspect of blocking is how the blocks are formed. Explicit articulation of block formation will be useful when we discuss asymptotic properties of our estimators and will also be used to differentiate the various population frameworks in Section 4. We identify three primary ways that blocks are formed:

Fixed blocks: It occurs when the total number of blocks and the covariate distribution of blocks is fixed before looking at the sample. For example, blocking that occurs on a single categorical covariate.

Flexible blocks: It occurs when the covariate distribution and total number of blocks may not be known before looking at the sample’s covariates. For example, if there are many covariates or continuous covariates and matching or discretizing is used to form blocks.

Structural blocks: It occurs when units have some natural grouping such that the blocks are self-contained. The members of each block are fixed, and if a block is represented in the sample, typically all members of that block are in the sample. For example, twins or classrooms.

Note that structural blocks are often thought of as clusters. With clusters, however, treatment assignment is commonly assigned at the cluster level, whereas we are focusing on treatment assigned within cluster. We use “structural block” to clarify this difference.

3. Variance Estimation

We next discuss how to estimate a blocked estimator’s variance, an integral part of obtaining standard errors and confidence intervals. We discuss from a Neyman–Rubin randomization perspective. See online Supplementary Material A for a discussion of alternative variance estimators (such as from linear models) that make additional assumptions on the data structure. We first investigate bias under a finite sample framework and extend to other frameworks in Section 4.

We start by giving the true variance in the finite sample. To do so, we need some additional notation. The mean of the potential outcomes for the units in the sample under treatment z for block k is

{\bar{Y}}_{k} (z) = \frac{1}{n_{k}} \sum_{i : b_{i} = k} Y_{i} (z),

the sample variance is

S_{k}^{2} (z) = \frac{1}{n_{k} - 1} \sum_{i : b_{i} = k} {(Y_{i} (z) - {\bar{Y}}_{k} (z))}^{2},

and the sample variance of the individual-level treatment effects is

S_{k}^{2} (t c) = \frac{1}{n_{k} - 1} \sum_{i : b_{i} = k} {(Y_{i} (t) - Y_{i} (c) - τ_{k, S})}^{2} .

For the finite sample, the variance of ${\hat{τ}}_{k}$ within a block is well known (see Imbens, 2011; Imbens & Rubin, 2015):

var ({\hat{τ}}_{k} | S) = \frac{S_{k}^{2} (t)}{n_{t, k}} + \frac{S_{k}^{2} (c)}{n_{c, k}} - \frac{S_{k}^{2} (t c)}{n_{k}} .

Summing these across the independent blocks, with the weights for block sizes, gives an overall variance of

var ({\hat{τ}}_{(B K)} | S) = \sum_{k = 1}^{K} \frac{n_{k}^{2}}{n^{2}} var ({\hat{τ}}_{k} | S) = \sum_{k = 1}^{K} \frac{n_{k}^{2}}{n^{2}} (\frac{S_{k}^{2} (t)}{n_{t, k}} + \frac{S_{k}^{2} (c)}{n_{c, k}} - \frac{S_{k}^{2} (t c)}{n_{k}}) .

For blocked experiments, the type of variance estimator one would use in the finite sample depends on the sizes of blocks one has. In cases where we have at least two treated and two control units in each block, we can directly extend classic results for completely randomized experiments by using them within each block and weighting (see, e.g., Imbens, 2011; Miratrix et al., 2013; Mukerjee et al., 2018). In particular, we can estimate each variance component of Equation 2 as

{\hat{σ}}_{k}^{2} = \hat{var} ({\hat{τ}}_{k}) = \frac{s_{k}^{2} (c)}{n_{c, k}} + \frac{s_{k}^{2} (t)}{n_{t, k}},

with $s_{k}^{2} (z)$ the sample variance of the units within block k under treatment z. Then we can combine these to get the plug-in variance estimator of

{\hat{σ}}_{(B K)}^{2} = \hat{var} ({\hat{τ}}_{(B K)}) = \sum_{k = 1}^{K} \frac{n_{k}^{2}}{n^{2}} (\frac{s_{k}^{2} (c)}{n_{c, k}} + \frac{s_{k}^{2} (t)}{n_{t, k}}) .

This gives a conservative estimate due to the dropping of the $S_{k}^{2} (t c) / n_{k}$ terms. Some tightening is possible by exploiting features such as differences in the shape of the observed treatment and control outcome distributions; for example, see Aronow et al. (2014), Chapter 6 of Imbens and Rubin (2015), or Schochet (2016). We call this the “big block” style of blocking and the “big block” estimator.

For the “small blocks” case, where our blocks have only one treated unit or one control unit, we need to use an alternative approach as we cannot estimate the variance for a treatment arm with a single unit. Our approach is presented below. To give some background, the analytical problems that arise when estimating the variance in matched pairs experiments, especially when working in the finite sample framework, have been lamented by many statisticians (see, e.g., Imbens, 2011). The issues arise from the fact that there is no way to estimate the within pair variance with only one unit assigned to treatment and one unit assigned to control in each pair. Previous work has found conservative estimators, however, which we build on. For instance, Imai (2008) showed that the standard matched pairs estimator is biased in the finite sample setting and put bounds on the true variance. The RCT-YES R package and documentation (Schochet, 2016) also provides a conservative variance estimator for the matched pairs design (as well as estimators for blocked designs); this is discussed more in online Supplementary Material A.3. For a hybrid experiment with both big and small blocks, we combine results to create an overall variance estimator.

3.1. Small Block Experiments With Equal-Size Blocks

When we have small blocks of the same size, we can directly use the usual variance estimator in the matched pairs literature (e.g., Imai, 2008) as a variance estimator for ${\hat{τ}}_{(B K)}$ , no matter what the block sizes are, as also noted by Fogarty (2018). This gives a variance estimator of

{\hat{σ}}_{(SMALL / s)}^{2} = \frac{1}{K (K - 1)} \sum_{k = 1}^{K} {({\hat{τ}}_{k} - {\hat{τ}}_{(B K)})}^{2} .

This estimator directly estimates the variance of the overall block treatment effect estimator rather than estimating the variance for each individual block and then weighting. We will see that, depending on the framework used, this estimator can give positively biased estimates if the true $τ_{k}$ tends to differ across blocks.

3.2. Small Block Experiments With Varying Size Blocks

For experiments with small blocks of varying sizes, we offer two variance estimators. The first directly extends the standard matched pairs estimator by grouping the blocks by size into J groups and using Equation 5 for each group. We then weight and combine to get an overall variance estimator.

Stratified Small Block Variance Estimator:

{\hat{σ}}_{(SMALL / m)}^{2} = \frac{1}{{(\sum_{j = 1}^{J} m_{j} K_{j})}^{2}} \sum_{j = 1}^{J} {(m_{j} K_{j})}^{2} {\hat{σ}}_{(SMALL), j}^{2},

where K_j is the number of blocks of size m_j and

{\hat{σ}}_{(SMALL), j}^{2} = \frac{1}{K_{j} (K_{j} - 1)} \sum_{k : n_{k} = m_{j}} {({\hat{τ}}_{k} - {\hat{τ}}_{(SMALL), j})}^{2},

with ${\hat{τ}}_{(SMALL), j} = \sum_{k : n_{k} = m_{j}} {\hat{τ}}_{k} / K_{j}$ . That is, grouping by the same size allows for using the equal size block estimator above. While straightforward, this is not ideal because it requires at least two blocks of each size in the overall experiment to estimate each ${\hat{σ}}_{(SMALL), j}^{2}$ . See online Supplementary Material E.1 for further detail.

The second approach allows the variance of all of the small blocks to be estimated at the same time, without requiring multiple blocks of the same size.

Unified Small Block Variance Estimator:

{\hat{σ}}_{(SMALL / p)}^{2} = \sum_{k = 1}^{K} \frac{n_{k}^{2}}{(n - 2 n_{k}) (n + \sum_{i = 1}^{K} \frac{n_{i}^{2}}{n - 2 n_{i}})} {({\hat{τ}}_{k} - {\hat{τ}}_{(B K)})}^{2} .

For ${\hat{σ}}_{(SMALL / p)}^{2}$ to be defined and guaranteed conservative, no one block can make up half or more of the units. We derived this estimator using the basic form of the matched pairs variance estimator as a weighted sum of the squared differences between the estimated average block treatment effects and the estimated overall average treatment effect. The weights then come from a simple optimization (see online Supplementary Material F) and partially account for the different blocks having different levels of precision when estimating the variance of the block-level impacts. This estimator has similar finite sample properties to the standard estimator for blocks of the same size (Equation 5). In particular, it is also conservative and unbiased when the block average treatment effects are all the same. When block sizes are all the same, this reduces to the usual matched pairs type estimator.

3.3. Hybrid Experiments

When doing variance estimation in a hybrid blocked design, we can split the blocks up into small blocks and big blocks. Grouping the big and small blocks together allows us to write the causal effect estimand as a combination of two estimands for our two different types of block sizes. Let there be $n_{s b}$ total units in small blocks in the sample. Then

τ_{S} = \frac{n - n_{s b}}{n} τ_{(BIG), S} + \frac{n_{s b}}{n} τ_{(SMALL), S},

where

τ_{(BIG), S} = \frac{1}{n - n_{s b}} \sum_{k : n_{t, k} \geq 2, n_{c, k} \geq 2} n_{k} τ_{k} and τ_{(SMALL), S} = \frac{1}{n_{s b}} \sum_{k : n_{t, k} = 1 or n_{c, k} = 1} n_{k} τ_{k} .

The estimator for the overall treatment effect can also be written as

{\hat{τ}}_{(B K)} = \frac{n - n_{s b}}{n} {\hat{τ}}_{(BIG)} + \frac{n_{s b}}{n} {\hat{τ}}_{(SMALL)} .

For finite sample inference, we can similarly break down the variance, and estimator of the variance, of ${\hat{τ}}_{(B K)}$ because the block estimators are independent due to the block randomized treatment assignment.

Hybrid Variance Estimator:

\hat{var} ({\hat{τ}}_{(B K)}) = \frac{{(n - n_{s b})}^{2}}{n^{2}} \hat{var} ({\hat{τ}}_{(BIG)}) + \frac{n_{s b}^{2}}{n^{2}} \hat{var} ({\hat{τ}}_{(SMALL)}) .

Here, we would use ${\hat{σ}}_{(B K)}^{2}$ (Equation 4) over just the big blocks for $\hat{var} ({\hat{τ}}_{(BIG)})$ and either ${\hat{σ}}_{(SMALL / m)}^{2}$ (Equation 6) or ${\hat{σ}}_{(SMALL / p)}^{2}$ (Equation 8) over just the small blocks (with the appropriate assumptions for just the small blocks) for $\hat{var} ({\hat{τ}}_{(SMALL)})$ . Thus, when we have small blocks, we can estimate the variance for those small blocks separately and use the usual blocking estimator on the larger blocks, essentially treating these as two separate experiments and combining with appropriate weights in the end. Alternatively, one could use ${\hat{σ}}_{(SMALL / m)}^{2}$ or ${\hat{σ}}_{(SMALL / p)}^{2}$ for all blocks, but we do not recommend this for the finite sample.

3.4. Finite Sample Bias of the Variance Estimators

In the finite setting, all of the above estimators are conservative and are only unbiased in specific circumstances. Each block is a miniature complete randomized experiment. For such experiments, ${\hat{σ}}_{k}^{2}$ is known (Imbens & Rubin, 2015, p. 92; Splawa-Neyman et al., 1923/1990) to have bias

E [{\hat{σ}}_{k}^{2} | S] - var ({\hat{τ}}_{k} | S) = \frac{S_{k}^{2} (t c)}{n_{k}} .

If all of the blocks have at least two treated and two control units, we can extend this result to ${\hat{σ}}_{(B K)}^{2}$ (Equation 4), which has bias

E [{\hat{σ}}_{(B K)}^{2} | S] - var ({\hat{τ}}_{(B K)} | S) = \sum_{k = 1}^{K} \frac{n_{k}}{n^{2}} S_{k}^{2} (t c) .

This extends readily to the big block component of the hybrid estimator by only including in the sum those blocks that are big, changing the n² in the denominator by ${(n - n_{s b})}^{2}$ , and weighting appropriately.

For the small blocks of varying sizes, we have two main results. In presenting these, we assume that the whole sample is made up of small blocks, though, as with the bias of ${\hat{σ}}_{(B K)}^{2}$ , the extension to the small block component of the hybrid estimator is straightforward. See online Supplementary Material E.2 and F for proofs. The first is a Corollary to classic results on matched pairs (see, e.g., Imai, 2008):

Corollary 3.4.1: The bias of ${\hat{σ}}_{(SMALL / m)}^{2}$ (Equation 6) under the finite framework is

E [{\hat{σ}}_{(SMALL / m)}^{2} | S] - var ({\hat{τ}}_{(SMALL)} | S) = \sum_{j = 1}^{J} \frac{K_{j} m_{j}^{2}}{n^{2} (K_{j} - 1)} \sum_{k : n_{k} = m_{j}} {(τ_{k, S} - τ_{(SMALL), S, j})}^{2} .

The above extends prior results for ${\hat{σ}}_{(SMALL / s)}^{2}$ for matched pairs (see Imai 2008; Imbens and Rubin, 2015, p. 227, or, for a more general case, Fogarty, 2018). ${\hat{σ}}_{(SMALL / m)}^{2}$ is conservative and unbiased when the average treatment effect is the same for all blocks of the same size (similar to the unbiased result from Imai (2008) for ${\hat{σ}}_{(SMALL / s)}^{2}$ ).

For ${\hat{σ}}_{(SMALL / p)}^{2}$ we have

Theorem 3.4.1: The bias of ${\hat{σ}}_{(SMALL / p)}^{2}$ (Equation 8) under the finite framework is

E [{\hat{σ}}_{(SMALL / p)}^{2} | S] - var ({\hat{τ}}_{(SMALL)} | S) = \sum_{k = 1}^{K} \frac{n_{k}^{2}}{(n - 2 n_{k}) (n + \sum_{i = 1}^{K} \frac{n_{i}^{2}}{n - 2 n_{i}})} {(τ_{k, S} - τ_{(SMALL), S})}^{2},

assuming no blocks have $n_{k} \geq n / 2$ .

If the average treatment effect is the same across all small blocks then this estimator is unbiased, and if there is heterogeneity, it is conservative. This is a distinction from the behavior of the variance estimator suggested in section 4.2 of Fogarty (2018) for use with variable size small blocks without covariates, in which even with the average treatment effect being the same across all small blocks, the bias is strictly greater than zero.

Remark: Both small block estimators are conservative, which raises the question of whether one is superior. The constant in front of each term of the bias of both estimators is of order $n_{k}^{2} / n^{2}$ . Then, we expect the bias of ${\hat{σ}}_{(SMALL / m)}^{2}$ to be less than the bias of ${\hat{σ}}_{(SMALL / p)}^{2}$ when the treatment effects of blocks of similar sizes are similar because the variance of impacts within blocks of a given size will be smaller than across all of the blocks. However, ${\hat{σ}}_{(SMALL / m)}^{2}$ has the drawback that it can only be used when we have at least two blocks of each small size.

The improved potential performance of ${\hat{σ}}_{(SMALL / m)}^{2}$ when there is homogeneity within block sizes does suggest that we could group blocks in some other way if we had prior knowledge of which blocks were most similar. That is, ${\hat{σ}}_{(SMALL / m)}^{2}$ relies on the blocks being equal size, so the weights factor out of the sum to give the expression for the cross-block estimate of variation. But we could first subdivide our blocks based on some similarity measure and apply ${\hat{σ}}_{(SMALL / p)}^{2}$ to each group, combining the parts with the hybrid weighting approach. This could make ${\hat{σ}}_{(SMALL / p)}^{2}$ less conservative while maintaining its validity.

Mukerjee et al. (2018) create a general framework for a class of conservative Neyman variance estimators that extends to a variety of causal estimands and estimators in the finite sample context. Of our estimators, ${\hat{σ}}_{(B K)}^{2}$ is directly shown as an example in their paper, and ${\hat{σ}}_{(SMALL / m)}^{2}$ can be shown to fall under their framework as well, as we show in online Supplementary Material B. The hybrid of these two can then also be included. Interestingly, it appears that ${\hat{σ}}_{(SMALL/ p)}^{2}$ does not fall within their framework, and instead, we need to rely on our own methods and derivations. See online Supplementary Material B for more details on these connections.

For our estimators, how conservative the estimators are may vary with blocks sizes. In the case where all blocks are the same size, when we have blocks with m control units and one treated unit, as m increases the variance of the treatment effect estimator will decrease, as we are getting a more precise estimate for the control units. However, the form of the bias of ${\hat{σ}}_{(SMALL / s)}^{2}$ remains the same. Therefore, with large m the bias of ${\hat{σ}}_{(SMALL / s)}^{2}$ due to treatment effect heterogeneity becomes larger relative to the true variance. This intuition extends to the variable size case as well. In these cases, alternative variance estimation strategies, such as discussed in online Supplementary Material A, may become more appealing.

The type of blocks also impacts whether the bias of these estimators go to zero as sample size increases. For instance, one might argue for the use of ${\hat{σ}}_{(SMALL / p)}^{2}$ instead of ${\hat{σ}}_{(B K)}^{2}$ even if we have big blocks because the condition for unbiasedness for ${\hat{σ}}_{(SMALL / p)}^{2}$ (that all blocks have the same average treatment effect) could be considered less stringent than for ${\hat{σ}}_{(B K)}^{2}$ (that there is zero treatment variation within each block). However, with fixed blocks, the number of units within each block increases as sample size increases and the bias of ${\hat{σ}}_{(B K)}^{2}$ will go to zero, the standard result, but the bias of ${\hat{σ}}_{(SMALL / p)}^{2}$ will not, unless all of the blocks have the same average treatment effect. In this case, as the blocks grow to be big, we would use ${\hat{σ}}_{(B K)}^{2}$ .

In the hybrid setting, the overall bias will be a weighted sum of the biases for the big and small block components. Therefore, because the overall weighting depends on the block sizes, having a poor estimator for the small blocks may not have a large effect on the overall bias if small blocks make up only a small proportion of the sample.

There is no way to unbiasedly estimate variance within small blocks without additional structure or covariates. If we think that the treatment effects of different strata are not too far apart, then we suggest using one of the previous estimators. We at least know that the bias incurred is positive. However, if we have reason to believe that the treatment effects of different strata will be very far apart, a plug-in estimator, as discussed in online Supplementary Material A, may be more appropriate.

4. Infinite Population Frameworks

Up to this point, we have examined blocking in a finite sample framework, conditioning on the units in the experiment in question. In the literature, however, blocking has often been examined under a variety of infinite population frameworks. In particular, the matched pairs literature uses a framework where the blocks themselves are sampled from an infinite population of blocks, whereas the big block literature typically assumes stratified random sampling from a finite number of infinite size strata. Using different population frameworks will give different answers to important questions of what the true variance of the treatment effect estimate is and what the bias of our variance estimators are. In this section, we first discuss the literature related to variance estimation for infinite populations, identifying the apparent tensions that exist. We then systematically discuss different frameworks, deriving the true variance of the treatment effect estimators under each of them. We also evaluate the bias of the variance estimators introduced in Section 3. We focus on infinite superpopulations; finite superpopulations substantially larger than the sample would give similar results. We explore work pertaining to the use of linear models, such as Cochran (1953) and Lin (2013), in online Supplementary Material A.1. An important note is that in some cases, these sampling schemes are chosen for convenience and that the generalizability of the experiment to the population will depend upon the assumptions made in them being true. The sampling model may also be considered to serve as a conservative approach to finite sample inference (see Ding et al., 2017).

Related Work

For matched pairs experiments, Imai (2008) showed that with a superpopulation of an infinite number of structural blocks, specifically matched pairs, from which pairs are randomly sampled, the standard matched pairs variance estimator (Equation 5) is unbiased for the PATE. On the other hand, Imbens (2011) showed that the standard matched pairs variance estimator is biased in the setting where we have fixed blocks and units are drawn using stratified random sampling (see Section 4.3 for more on this setting). This is a clear example of how the population framework being used matters. We therefore advise practitioners to carefully consider what population and sampling structure they are assuming and to not simply assume a framework for convenience.

The general blocked design has been previously discussed in various forms. Imbens (2011) discussed blocking in the context of a superpopulation with a fixed number of strata from which units are sampled using a stratified sampling method. He formed unbiased estimators for the variance in this context, assuming that the blocks each have at least two units assigned to treatment and control. These results are similar to finite sample results discussed in Section 3 and will be discussed more in Section 4.3. Imai et al. (2008) analyzed estimation error and variance with the blocked design. Scosyrev (2014) also analyzed the blocked experiment in the finite sample and under two sampling frameworks, recognizing that the different settings resulted in different outcomes. Sävje (2015) analyzed flexible “threshold” blocking and made critical points about the importance of block structure and sampling design when analyzing blocked experiments, which we will echo and expand on.

4.1. Infinite Populations in General

Inference for the PATE typically takes the sample as a random sample from some larger population, as opposed to inference for the SATE discussed earlier that held the sample of potential outcomes as fixed. This makes estimation an implicit two-step process, estimating the treatment effect for the sample and extrapolating this estimate to the population. Frequently, in fact, the estimators themselves are the same as for finite sample inference even though the estimands are different.

Define the PATE as

τ = E [Y_{i} (t) - Y_{i} (c) | F],

where $F$ both indicates the block type and sampling framework. This is the same as the direct average of the unit-level treatment effects for all of the units in the population, as is commonly used (see Imbens & Rubin, 2015, p. 99), as long as our sampling mechanism is not biased. Here, we will only consider frameworks where the sampling scheme provides a sample that, on average, has the same average treatment effect as the population but note that bias from the sampling mechanism can be fixed using weighting if the sampling mechanism is known (see Miratrix et al., 2018).

Under blocking, the PATE within block k is

τ_{k} = E [Y_{i} (t) - Y_{i} (c) | b_{i} = k, F],

where, again, b_i indicates the block that unit i belongs to. It is possible that k indexes a (countably) infinite set of blocks in the case of some infinite population models.

Overall, using the law of total expectation and variance decompositions, we can generally obtain the properties of our estimators with respect to population estimands by first obtaining expressions for a finite sample and then averaging these expressions across the sampling distributions. In other words, we heavily exploit $E [\hat{M} | F] = E [E [\hat{M} | S] | F]$ , where $S$ is a sample obtained from $F$ , our population and sampling framework. Under any unbiased framework $F$ , we have the typical result (e.g. see Imbens, 2011)

E [{\hat{τ}}_{(B K)} | F] = E [τ_{S} | F] = τ .

There are several different frameworks that one might assume. These can generally be characterized by two primary features: the block types, which also dictates the population strata structure, and the sampling scheme. Note that the term strata is used for the population here analogously to blocks in the sample. We may obtain a sample using simple random sampling and then form blocks based on covariates postsampling and prerandomization, that is, flexible blocks. Or we may have fixed blocks (e.g., blood types) and use stratified sampling where we sample units from each population stratum. Finally, we may have structural blocks and conceptualize a population of an infinite number of these blocks (e.g., schools in an “infinite” population of schools) from which we randomly select a fixed number of blocks. As we show next, the bias of the variance estimators can differ depending on the framework assumed. We refer to frameworks using their sampling method as a shorthand, leaving the block type and population structure implicit.

4.2. Simple Random Sampling, Flexible Blocks

In this framework, denoted $SRS$ , units are sampled at random, without regard to block membership, from the population. In this context, we focus on the use of flexible blocks, for example, blocking using clustering on a continuous covariate or based on observed covariates in the sample obtained. Structural blocks do not make sense in this framework (e.g., one would always sample pairs of twins not individuals who are twins if we wish to run a twin study) and fixed blocks give rise to difficulties when the sample does not have units from all population strata. For blocked experiments with fixed blocks in this framework, see Scosyrev (2014).

The variance in this framework, using the basic variance decomposition, is

var ({\hat{τ}}_{(B K)} | SRS) = E [\sum_{k = 1}^{K} \frac{n_{k}^{2}}{n^{2}} (\frac{S_{k}^{2} (c)}{n_{c, k}} + \frac{S_{k}^{2} (t)}{n_{t, k}} - \frac{S_{k}^{2} (t c)}{n_{k}}) | SRS] + var (τ_{S} | SRS) .

The expectation is across the sampling and blocking process. SRS denotes the simple random sample and subsequent blocking of sampled units.

In this context, we have an unbiased variance estimator if we have all big blocks:

Theorem 4.2.1: The variance estimator

{\hat{σ}}_{SRS}^{2} = \sum_{k = 1}^{K} \frac{n_{k} (n_{k} - 1)}{n (n - 1)} (\frac{s_{k}^{2} (c)}{n_{c, k}} + \frac{s_{k}^{2} (t)}{n_{t, k}}) + \sum_{k = 1}^{K} \frac{n_{k}}{n (n - 1)} {({\hat{τ}}_{k} - {\hat{τ}}_{(B K)})}^{2},

is an unbiased estimator for $var ({\hat{τ}}_{(B K)} | SRS)$ , if $n_{c, k} \geq 2$ and $n_{t, k} \geq 2$ .

See online Supplementary Materials G for a derivation. The first term in the estimator looks similar to our usual big block estimator and captures part of the first term in our variance decomposition. The second term looks similar to our proposed small block estimator and accounts for the rest of the variation. While very similar to the estimator found in Scosyrev (2014), we have made adjustments to achieve unbiasedness of the estimator whereas Scosyrev focuses on consistency. Scosyrev also works with fixed blocks where the number of blocks is assumed known before sampling and weights are used to match the sample to the population proportions, as opposed to flexible blocks which allow random numbers of blocks that are created postsampling.

Remark: If we naively use ${\hat{σ}}_{(B K)}^{2}$ (Equation 4) our bias will be

E [{\hat{σ}}_{(B K)}^{2} | SRS] - var ({\hat{τ}}_{(B K)} | SRS) = \frac{1}{n} E [\sum_{k = 1}^{K} \frac{n_{k}}{n} S_{k}^{2} (t c) - S^{2} (t c) | SRS],

where $S^{2} (t c)$ is the sample variance of individual-level treatment effects across the whole sample. This result follows from the derivations in online Supplementary Materials G, and it implies that ${\hat{σ}}_{(B K)}^{2}$ could be anti-conservative in this setting if there is generally treatment variation across samples (making $S^{2} (t c) > 0$ ), but units put within the same block are nearly identical in terms of impacts (making $S_{k}^{2} (t c) \approx 0$ ). This could happen when the experimenter is successfully making homogenous blocks.

Similarly, if we use either of the small block variance estimators, the bias will be the difference between the expected finite sample bias for those estimators (which depends on treatment effect heterogeneity between blocks) and $E [S^{2} (t c) | SRS] / n$ , which corresponds to treatment effect heterogeneity across the whole population. Therefore whether these estimators are conservative or not depends upon the structure of the population and how the blocks are formed.

4.3. Stratified Sampling, Fixed Blocks

In the “stratified sampling” framework, denoted $F_{1}$ , there are K fixed strata of infinite size in the population. Then n_k units are randomly sampled from strata k (i.e., stratified random sampling is used). Here, we have fixed blocks. We assume that n_k is fixed and that $n_{k} / n$ is the population proportion of units in stratum k, for simplicity. Otherwise, a weighting scheme, as mentioned in Section 4.1, would be needed to create an unbiased estimator of the direct average of treatment effects in the population. This is the framework used in Imbens (2011) and Miratrix et al. (2013), who show the following result under equal proportions treated within each block, which simplifies the weights.

As in the finite sample, overall variance is a weighted sum of within block variances:

var ({\hat{τ}}_{(B K)} | F_{1}) = \sum_{k = 1}^{K} \frac{n_{k}^{2}}{n^{2}} var ({\hat{τ}}_{k} | F_{1}) = \sum_{k = 1}^{K} \frac{n_{k}^{2}}{n^{2}} (\frac{σ_{k}^{2} (c)}{n_{c, k}} + \frac{σ_{k}^{2} (t)}{n_{t, k}}),

with $σ_{k}^{2} (z)$ the population variance of the potential outcomes under treatment z in strata k.

As noted in Imbens (2011), the variance estimator of big blocks, ${\hat{σ}}_{(B K)}^{2}$ (Equation 4), is unbiased in this framework. The estimators for the variance of the small blocks, however, can have bias. We have two results pertaining to this. For presentation of results for small blocks, we assume that all blocks in the sample are small, but the results extend directly to just the small block component of the hybrid variance estimators.

First, as with the finite sample, we can extend results for ${\hat{σ}}_{(SMALL / s)}^{2}$ (see Imbens, 2011) to ${\hat{σ}}_{(SMALL / m)}^{2}$ .

Corollary 4.3.1: The bias of ${\hat{σ}}_{(SMALL / m)}^{2}$ (Equation 6) under the stratified sampling framework is

E [{\hat{σ}}_{(SMALL / m)}^{2} | F_{1}] - var ({\hat{τ}}_{(SMALL)} | F_{1}) = \sum_{j = 1}^{J} \frac{K_{j} m_{j}^{2}}{n^{2} (K_{j} - 1)} \sum_{k : n_{k} = m_{j}} {(τ_{k} - τ_{(SMALL)})}^{2} .

As with finite sample inference, this shows that ${\hat{σ}}_{(SMALL / m)}^{2}$ is a conservative estimator unless the average treatment effect is same across all small blocks of the same size, in which case it is unbiased. See online Supplementary Material E.2 for the derivation.

Second, for our new variance estimator we have the following result:

Corollary 4.3.2: The bias of ${\hat{σ}}_{(SMALL / p)}^{2}$ (Equation 8) under the stratified sampling framework is

E [{\hat{σ}}_{(SMALL / p)}^{2} | F_{1}] - var ({\hat{τ}}_{(SMALL)} | F_{1}) = \sum_{k = 1}^{K} \frac{n_{k}^{2}}{(n - 2 n_{k}) (n + \sum_{i = 1}^{K} \frac{n_{i}^{2}}{n - 2 n_{i}})} {(τ_{k} - τ_{(SMALL)})}^{2},

assuming no blocks have $n_{k} \geq n / 2$ .

This shows that ${\hat{σ}}_{(SMALL / p)}^{2}$ is also a conservative estimator (given no block makes up more than half the sample), and it is unbiased when the average treatment effect is the same across all small blocks. See online Supplementary Material F for a derivation.

4.4. Random Sampling of Strata, Structural Blocks

In the “random sampling of strata” framework, denoted $F_{2}$ , there are an infinite number of strata of finite size, that is, an infinite number of structural blocks. K strata are then randomly chosen to be in the sample, and randomization is done within each of the sample blocks. This setting, with equal block sizes, is often used in the matched pairs literature, such as in Imai (2008).

Within this framework, which blocks are included in the sample is itself random. Therefore, the variance estimator needs to capture not only the within strata variance but also the variance due to which strata are chosen to be in the sample. Furthermore, if the block sizes vary, the total number of units is random which introduces additional complexities.

For the more general variable-size version of this framework, the variance of ${\hat{τ}}_{(B K)}$ is

var ({\hat{τ}}_{(B K)} | F_{2}) = E [\sum_{k : B_{k} = 1} \frac{n_{k}^{2}}{n^{2}} (\frac{S_{k}^{2} (c)}{n_{c, k}} + \frac{S_{k}^{2} (t)}{n_{t, k}} - \frac{S_{k}^{2} (t c)}{n_{k}}) | F_{2}] + var (τ_{S} | F_{2}),

where B_k is the indicator that stratum k is included in the sample, with $B_{k} = 1$ indicating sample membership and $B_{k} = 0$ otherwise.

When blocks are of the same size, we can simplify the expression with $\frac{n_{k}^{2}}{n^{2}} = \frac{1}{K^{2}}$ , which is no longer random. If we have all blocks of the same size, then we can rewrite ${\hat{σ}}_{(SMALL / s)}^{2}$ (Equation 5) using sample inclusion indicators as

{\hat{σ}}_{(SMALL / s)}^{2} = \frac{1}{K (K - 1)} \sum_{k} B_{k} {({\hat{τ}}_{k} - {\hat{τ}}_{(B K)})}^{2},

and this is an unbiased estimator for $var ({\hat{τ}}_{(B K)} | F_{2})$ . This is simply the variance of the estimated block effect in the sample. Imai (2008) showed that this estimator is unbiased in this setting with an infinite population of matched pairs. See online Supplementary Material E.3 for the proof of this result extended to other small block types of equal size.

Variance estimators when the strata vary in size are more complicated. In particular, under this framework, there is a chance that there is only a single block of a given size, making the first variance estimator infeasible. If we condition on the number of strata drawn of each possible strata size, assuming that there are multiple strata of the each size in the sample, we obtain the following Corollary:

Corollary 4.4.1: In the conditioned case, assuming it is defined, ${\hat{σ}}_{(SMALL / m)}^{2}$ (Equation 6) is an unbiased estimator for $var ({\hat{τ}}_{(B K)} | F_{2})$ .

This result can be seen directly from the results in online Supplementary Material E.3.

Alternatively, if we are willing to assume that block size is independent of treatment effect, then we have the following more general result:

Theorem 4.4.1 (Unbiasedness of ${\hat{σ}}_{(SMALL / p)}^{2}$ given independence): In the random sampling of strata setting where block sizes are independent of block average treatment effects, ${\hat{σ}}_{(SMALL / p)}^{2}$ (Equation 8) is an unbiased estimator for $var ({\hat{τ}}_{(B K)} | F_{2})$ , assuming no blocks have $n_{k} \geq n / 2$ . The proof is in online Supplementary Materials F.2.

Remark: We may also consider an infinite number of strata of infinite size, as is commonly used in multisite randomized trials. This is the setting considered in Schochet (2016) and the RCT-YES software (Schochet, 2016) estimator discussed in online Supplementary Material A.3 could be used. The sampling scheme then has two steps: first sample the strata, then sample units from the strata. To discuss variance, we need to add a bit of notation. Let $τ_{S}^{*}$ denote the expectation of the treatment effect estimator given the blocks in the sample. That is, we fix which strata are in the sample and take the expectation over the sampling of units from the infinite size strata. So conditioning on which strata are in the sample we are in a stratified sampling setup. Let this framework be denoted by $F_{3}$ . Then, the variance of ${\hat{τ}}_{(B K)}$ is

var ({\hat{τ}}_{(B K)} | F_{3}) = E [\sum_{k : B_{k} = 1} \frac{n_{k}^{2}}{n^{2}} (\frac{σ_{k}^{2} (c)}{n_{c, k}} + \frac{σ_{k}^{2} (t)}{n_{t, k}}) | F_{3}] + var (τ_{S}^{*} | F_{3}) .

It is straightforward to extend the results of Corollary 4.4.1 and Theorem 4.4.1 to this case.

4.5. Discussion

While the variance formulas that we presented above share a similar structure with each other and the finite sample forms, there are important differences. In the finite sample framework (Equation 2), there is a term regarding treatment effect variation that reduces the variance due to the correlation of potential outcomes. This term is retained in the random sampling of strata framework of Section 4.4 but not in the stratified sampling framework of Section 4.3. This difference in the true variance implies that different variance estimators may be more appropriate in different settings. It also suggests comparisons of blocking to complete randomization under these different assumptions will also diverge; for further discussion on this, see Pashley and Miratrix (2020). In fact, this difference explains much of the apparent discrepancy between the matched pairs literature and the blocking literature.

Relatedly, different variance estimators can have different amounts of bias depending on the framework being used. The small blocks estimators ( ${\hat{σ}}_{(SMALL / m)}^{2}$ and ${\hat{σ}}_{(SMALL / p)}^{2}$ ) in the finite sample and the stratified sampling framework are unbiased if the average treatment effect is the same across all of the small blocks (or all of the small blocks of the same size for ${\hat{σ}}_{(SMALL / m)}^{2}$ ) and otherwise are more conservative as the variance of the average treatment effects across blocks increases. For the infinite number of strata framework, under some assumptions, all of our small block variance estimators are unbiased. We have no small block estimator that is guaranteed to be unbiased or conservative for the simple random sampling (flexible block) framework, though we present one for big blocks.

The big blocks estimator ( ${\hat{σ}}_{(B K)}^{2}$ ) in the finite sample is unbiased if the treatment effect is additive within each block and otherwise depends on the treatment effect heterogeneity within each block. In the stratified sampling framework, however, ${\hat{σ}}_{(B K)}^{2}$ will be unbiased.

Overall, only the framework of Section 4.4 of sampling structural blocks, with the additional assumption of independence of impact and block size given there, has unbiased variance estimators for a mixture of big and small blocks. This means that, without additional assumptions allowing for plug-in approaches, the hybrid estimators, where possible, will always be conservative.

5. Simulations

We compare different estimators of the variance for hybrid blocked experiments where there are a few big blocks and many small blocks in a finite sample context. We explore a context where 50% of our units are in small blocks, each with only one treated unit, and the remainder are in big blocks with at least two treated units. None of our blocks have many treated units due to only having approximately 20% of the units treated overall. (The 20% was approximate in order to create varying size small blocks to see the different performance of the hybrid estimators.) We have 15 blocks with sizes ranging from 3 to 20.

The simulations presented here are for the finite sample framework, as it is both a common mode of inference and a core building block to the population frameworks. These results, however, are largely applicable to the other settings. For instance, the biases for the small blocks variance estimators have the same form for the finite sample and the stratified sampling frameworks.

We considered our two hybrid estimators, which correspond to estimating the variance of the small blocks two different ways. We also considered two regression estimators: the HC1 sandwich estimate (Hinkley, 1977) from a linear model with fixed effects and no interaction between treatment indicator and blocking factor, and the standard variance estimate (inverse Fisher information) from a weighted regression, weighting each unit by the inverse probability of being assigned to its given treatment status in its block, multiplied by the overall proportion of units in its treatment group (this is a variant of the approach in Gerber and Green, 2012; see also Miratrix et al., 2020).² Note that the HC1 estimator is the “robust” estimator used in Stata (StataCorp, 2017) for estimating standard deviations.

In our simulations, we varied both to what extent blocking successfully separated units based on their potential outcomes under control and also on their treatment effects. The average potential outcome under control and the average treatment effect for each block were both negatively correlated with block size, so that smaller blocks had larger control potential outcomes and larger treatment effects. The correlation of potential outcomes within blocks was also varied between $ρ = 0, .5, and 1$ . See online Supplementary Material C for more on the data generating process.

We compared all of the variance estimators to the actual variance of the corresponding blocking treatment estimator in Figure 1 by looking at the percent relative bias ( $[mean ({\hat{σ}}_{*}^{2}) - var ({\hat{τ}}_{(B K)} | S)] / var ({\hat{τ}}_{(B K)} | S)$ ).³ The variation due to changing the between block difference in the mean of control potential outcomes was found to be minimal, so we average these differences on the plots. The two hybrid estimators, the one using ${\hat{σ}}_{(SMALL / m)}^{2}$ (Equation 6) ( ${Hybrid}_{m}$ ) for the small blocks and the one using ${\hat{σ}}_{(SMALL / p)}^{2}$ (Equation 8) ( ${Hybrid}_{p}$ ) for the small blocks, outperform the linear model estimators, especially as the treatment effect variation across the blocks increases. We see that ${Hybrid}_{m}$ also has lower bias than ${Hybrid}_{p}$ as treatment heterogeneity increases. This is because the value of treatment effects is correlated with block size and ${\hat{σ}}_{(SMALL / m)}^{2}$ groups variance estimation by block size. Weighted regression performance was generally similar to that of ${Hybrid}_{p}$ , although slightly anti-conservative for samples with low treatment heterogeneity when $ρ = 1$ .

Figure 1.

Simulations to assess variance estimators’ relative bias as a function of treatment variation across blocks. Each column represents a different value of $ρ$ , with values denoted at the top of the graph. The x-axis shows the standard deviation of block treatment effects. Dots indicate average over changes in control means for specific finite samples. FE stands for fixed effects.

For discussion of the variance of the variance estimators, see online Supplementary Material D. The variance estimators’ variances were found to be comparable, with weighted regression generally the most stable.

When comparing the performance of estimators, there is an important note about the linear model estimator: The sandwich estimate for a linear model is associated with a different treatment effect estimator than the others. In particular, a linear model with fixed effects is estimating a precision weighted estimate of the treatment effect across the blocks. It is well-known that as treatment heterogeneity increases, this estimator can become increasingly biased. See Raudenbush and Schwartz (2020) for a longer discussion on this and related estimators. This is not an issue for the weighted regression which, similar to adding interactions between treatment and block dummy variables, will recover ${\hat{τ}}_{(B K)}$ .

6. Data Example

One area where analysts are often faced with many small blocks of varying sizes is found in the matching literature. In particular, full matching (see Hansen, 2004; Rosenbaum, 1991) finds sets of similar units, with either one treated and several control or vice versa, that could be considered as-if randomized. After matching, a researcher could then analyze these data using permutation tests and associated sensitivity checks (see, e.g., Rosenbaum, 2010), but in this context, generating confidence intervals or standard errors using permutation inference would typically rely on a constant treatment effect assumption across the blocks. One might alternatively wish for a Neyman-style randomization analysis such as would be typically done for large block experiments to obtain inference for the average effect in the presence of treatment variation. The average treatment effect estimate is easy to obtain; it is the uncertainty estimation that causes the trouble. Our small block variance estimators fill this gap. To illustrate, we analyze a data set from the National Health and Nutrition Examination Survey (NHANES) 2013–2014 (Centers for Disease Control and Prevention [CDC] National Center for Health Statistics [NCHS], 2013–2014) given in the CrossScreening package (Rosenbaum & Zhao, 2017) in R statistical software (R Core Team, 2016). This data set was also used by Zhao et al. (2018) to analyze the effect of high fish consumption (defined as 12 or more servings of fish or shellfish in the previous month) versus low fish consumption (defined as 0 or 1 servings of fish or shellfish in the previous month) on a number of biomarkers.

Although Zhao et al. (2018) analyzed numerous outcomes, we focus on a measure of mercury (LBXGM), converted to the ${log}_{2}$ scale, as a simple illustration of our methods. We use unrestricted full matching to obtain a set of all small blocks of varying size. As in Zhao et al. (2018), we matched on smoking, age, gender, race, income, and education. We used Bayesian logistic regression through the brglm package (Kosmidis, 2017) and optmatch (Hansen & Klopfer, 2006) in R (R Core Team, 2016). This resulted in 197 blocks with only one treated or one control unit in each. Sizes of blocks ranged from 2 to 47. This type of matching would fall into the category of flexible blocks, and we here focus on estimation of the SATE for the finite sample.

There were some block sizes that were unique, so the hybrid estimator with ${\hat{σ}}_{(S M A L L / m)}^{2}$ could not be used. Alternate forms of full matching could potentially avoid this concern: Full matching can include additional restrictions, such as using only a portion of the control group or exact matching on some important covariates, which could make the block sizes more homogenous (Hansen & Klopfer, 2006); for simplicity, we do not explore this here. The blocking treatment effect estimate ( ${\hat{τ}}_{(B K)}$ ) was 2.45 but using a fixed effects model with no interaction the treatment effect estimate was 2.75. Looking at Table 1, we see that our hybrid estimator using ${\hat{σ}}_{(SMALL / p)}^{2}$ gave a much larger variance estimate (relative to the scale of the precision estimates) than the two linear model-based variance estimators.

Table 1.

Results of National Health and Nutrition Examination Survey (NHANES) (Full Matching), and Lalonde (CEM) for Different Estimation Strategies

	NHANES		Lalonde
Estimator	Estimate	$\hat{S E}$	Estimate	$\hat{S E}$
Hybrid blocking with ${\hat{σ}}_{(SMALL / m)}^{2}$	2.45	N/A	$560	$570
Hybrid blocking with ${\hat{σ}}_{(SMALL / p)}^{2}$	2.45	.20	$560	$606
Weighted regression	2.45	.11	$560	$560
Fixed effects regression (HC1)	2.75	.13	$425	$601

A second method for analyzing observational datasets where our variance estimators could be useful is CEM. CEM coarsens covariates used to match and then exactly matches to these coarsened variables (Iacus et al., 2012). We follow the example from the vignette of the cem package (Iacus et al., 2016) in R using the most automated version of CEM on the classic LaLonde data set (LaLonde, 1986), available in the cem package. This data set consists of individuals who received or did not receive a job training program with the outcome of interest as earnings in 1978. We use the unmodified version of the LaLonde data set, but otherwise follow the automated process for CEM laid out in the vignette to create blocks (we do not follow the analysis). This resulted in the creation of 69 blocks, some small and some big, with some ungrouped units being dropped. The blocking treatment effect estimate ( ${\hat{τ}}_{(B K)}$ ) was $560 but using a fixed effects model with no interaction the treatment effect estimate was $425. From Table 1, the precision estimates from all methods were similar though, again, the hybrid estimator using ${\hat{σ}}_{(SMALL / p)}^{2}$ was the largest and likely the most conservative.

7. Discussion

Blocking can be viewed under a wide variety of population frameworks ranging from a fixed, finite sample model to one where we envision the units as being sampled from a larger population in pre-set groups. Because different types of blocking tend to use different frameworks, there has not been good guidance on how to proceed when faced with some singleton units in some blocks and not in others.

We have worked to bring the different frameworks together in order to compare them systematically. We identified and compared the true variance of a blocking-based estimator under multiple settings, and created corresponding estimators of the impact estimator’s variance. We also provide simple, model-free variance estimators for two types of experiments that have not received much attention: blocked experiments with variable-sized blocks containing singleton treatment or control units and hybrid blocked experiments with large and small blocks combined. These contexts are quite common, frequently appearing in, for example, the matching literature. We analyzed the performance of both our new variance estimators and the classic variance estimators under different frameworks, identifying when they are unbiased or conservative. This investigation again illustrates how different sampling frameworks and block types can impact assessments of an estimator’s performance.

Future work includes extending these results to other population settings and sampling methods, in particular finding small block estimators for the setting of constructing blocks postsampling and prerandomization. Variance estimation is also a missing and needed piece in poststratification research, as noted in Miratrix et al. (2013). Although conditional answers for poststratification would correspond to the estimators presented in this work, the unconditional case remains an open extension.

Supplemental Material

Supplemental Material, blocking_variance_appendices - Insights on Variance Estimation for Blocked and Matched Pairs Designs

Supplemental Material, blocking_variance_appendices for Insights on Variance Estimation for Blocked and Matched Pairs Designs by Nicole E. Pashley and Luke W. Miratrix in Journal of Educational and Behavioral Statistics

Footnotes

Authors’ Note

Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation, the Institute of Education Sciences or the U.S. Department of Education.

Acknowledgments

The authors would like to thank Guillaume Basse, Avi Feller, Colin Fogarty, Michael Higgins, Luke Keele, and Lo-Hua Yuan for their comments and edits. We would also like to thank members of The Miratrix C.A.R.E.S. Lab and Donald B. Rubin’s research lab for their useful feedback on the project and Peter Schochet and Kosuke Imai for insightful discussion of this material. Finally, we thank anonymous reviewers for their helpful feedback.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research and/or authorship of this article: The research reported here was partially supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305D150040. This material is also based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE1745303.

Notes

References

Abadie

Imbens

G. W.

(2008). Estimation of the conditional variance in paired experiments. Annales d’Économie et de Statistique, 91/92, 175–187.

Aronow

P. M.

Green

D. P.

Lee

D. K.

(2014). Sharp bounds on the variance in randomized experiments. The Annals of Statistics, 42(3), 850–871.

Centers for Disease Control and Prevention (CDC). National Center for Health Statistics (NCHS). (2013–2014). National health and nutrition examination survey data. U.S. Department of Health and Human Services, CDC.

Cochran

W. G.

(1953). Matching in analytical studies. American Journal of Public Health and the Nations Health, 43(6_Pt_1), 684–691.

Cochran

W. G.

(1977). Sampling techniques. Wiley series in probability and mathematical statistics-applied (3rd ed.). John Wiley & Sons.

Cochran

W. G.

Cox

G. M.

(1950). Experimental designs. John Wiley & Sons.

Ding

Miratrix

L. W.

(2017). Bridging finite and super population causal inference. Journal of Causal Inference, 5(2), 1–8. https://doi.org/10.1515/jci-2016-0027

Fisher

R. A.

(1926). The arrangement of field experiments. Journal of Ministry of Agriculture, 33, 503–513.

Fogarty

C. B.

(2018). On mitigating the analytical limitations of finely stratified experiments. Journal of the Royal Statistical Society. Series B, 80(5), 1035–1056.

10.

Gerber

A. S.

Green

D. P.

(2012). Field experiments: Design, analysis and interpretation. Norton.

11.

Hansen

B. B.

(2004). Full matching in an observational study of coaching for the SAT. Journal of the American Statistical Association, 99(467), 609–618.

12.

Hansen

B. B.

Klopfer

S. O.

(2006). Optimal full matching and related designs via network flows. Journal of the American Statistical Association, 15(3), 609–627.

13.

Hinkley

D. V.

(1977). Jackknifing in unbalanced situations. Technometrics, 19(3), 285–292.

14.

Iacus

S. M.

King

Porro

(2012). Causal inference without balance checking: Coarsened exact matching. Political Analysis, 20(1), 1–24.

15.

Iacus

S. M.

King

Porro

(2016). cem: Coarsened exact matching (R package version 1.1.17). https://CRAN.R-project.org/package=cem

16.

Imai

(2008). Variance identification and efficiency analysis in randomized experiments under the matched-pair design. Statistics in Medicine, 27(24), 4857–4873.

17.

Imai

King

Stuart

E. A.

(2008). Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society: Series A, 171(2), 481–502.

18.

Imbens

G. W.

(2011). Experimental design for unit and cluster randomid trials. Conference International Initiative for Impact Evaluation, Cuernavaca.

19.

Imbens

G. W.

Rubin

D. B.

(2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge University Press.

20.

Kosmidis

(2017). brglm: Bias reduction in binary-response generalized linear models (R package version 0.6.1). https://CRAN.R-project.org/package=brglm

21.

LaLonde

R. J.

(1986). Evaluating the econometric evaluations of training programs with experimental data. The American Economic Review, 76(4), 604–620.

22.

Lin

(2013). Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique. Annals of Applied Statistics, 7(1), 295–318.

23.

Miratix

L. W.

Pashley

N. E.

(2020). blkvar. https://rdrr.io/github/lmiratrix/blkvar/

24.

Miratrix

L. W.

Sekhon

J. S.

Theodoridis

A. G.

Campos

L. F.

(2018). Worth weighting? How to think about and use weights in survey experiments. Political Analysis, 26(3), 275–291.

25.

Miratrix

L. W.

Sekhon

J. S.

(2013). Adjusting treatment effect estimates by post-stratification in randomized experiments. Journal of the Royal Statistical Society. Series B, 75(2), 369–396.

26.

Miratrix

L.W.

Weiss

Henderson

(2020). An applied researcher’s guide to estimating effects from multisite individually randomized trials: Estimands, estimators, and estimates [Unpublished manuscript].

27.

Mukerjee

Dasgupta

Rubin

D. B.

(2018). Using standard tools from finite population sampling to improve causal inference for complex experiments. Journal of the American Statistical Association, 113(522), 868–881.

28.

Pashley

N. E.

Miratrix

L. W.

(2020). Block what you can, except when you shouldn’t [Unpublished manuscript].

29.

Raudenbush

S. W.

Schwartz

(2020). Randomized experiments in education, with implications for multilevel causal inference. Annual Review of Statistics and Its Application, 7, 177–208.

30.

R Core Team. (2016). R: A language and environment for statistical computing. R Foundation for Statistical Computing.

31.

Rosenbaum

P. R.

(1991). A characterization of optimal designs for observational studies. Journal of the Royal Statistical Society. Series B, 53(3), 597–610.

32.

Rosenbaum

P. R.

(2010). Design of observational studies. Springer series in statistics. Springer.

33.

Rosenbaum

P. R.

Zhao

(2017). CrossScreening: Cross-screening in observational studies that test many hypotheses (R package version 0.1.1). https://CRAN.R-project.org/package=CrossScreening

34.

Rubin

D. B.

(1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701.

35.

Rubin

D. B.

(1980). Randomization analysis of experimental data: The Fisher randomization test comment. Journal of the American Statistical Association, 75(371), 591–593.

36.

Särndal

C.-E.

Swensson

Wretman

(2003). Model assisted survey sampling. Springer.

37.

Sävje

(2015). The performance and efficiency of threshold blocking. arXiv preprint arXiv:1506.02824. https://arxiv.org/abs/1506.02824

38.

Schochet

P. Z.

(2016). Statistical theory for the RCT-YES software: Design-based causal inference for RCTs (2nd ed.) (Technical Report (NCEE 2015-4011)). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Analytic Technical Assistance and Development.

39.

Scosyrev

(2014). Causal inference in block-randomized experiments: Analysis based on Neyman’s stochastic causal model [Unpublished manuscript].

40.

Splawa-Neyman

Dabrowska

D. M.

Speed

(1923). On the application of probability theory to agricultural experiments. Essay on principles. Section 9. Statistical Science, 5(4), 465–472. (original work published 1990)

41.

StataCorp. (2017). Stata Statistical Software: Release 15. StataCorp.

42.

C. F. J.

Hamada

M. S.

(2000). Experiments: Planning, analysis, and parameter design optimization. John Wiley & Sons.

43.

Zhao

Small

D. S.

Rosenbaum

P. R.

(2018). Cross-screening in observational studies that test many hypotheses. Journal of the American Statistical Association, 113(523), 1070–1084.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.27 MB