Considering Network Effects in the Design and Analysis of Field Experiments on State Legislatures

Introduction

Two recent streams of innovative research in legislative politics include the study of legislative networks and field experiments on legislatures—state legislatures, in particular. These two emerging approaches have evolved largely separate from one another, but we argue that they should be integrated due to the interdependence that arises between legislators based on processes such as cue-taking. In a study of cue-taking in roll call votes in the California Assembly, Masket et al. (2008) aptly summarize the importance of understanding sources of interdependence between legislators in accounting for legislative outcomes. Masket (2008, 302) notes that,

. . . there is a great deal of cue-taking in a legislature. Members defer in their judgment to trusted colleagues with expertise in particular issue areas.

Masket et al. (2008) find that a connection as informal as two legislators being desk mates in the legislative chamber increases the rate at which two legislators vote in agreement. Legislative networks research, which has grown significantly in recent years, has documented complex forms of interconnectedness that can be observed in patterns of cosponsorship (Fowler 2006; Kirkland 2011, 2013), shared campaign staff (Nyhan and Montgomery 2015), collaborative press events (Desmarais et al. 2015), and caucus comembership (Victor and Ringe 2009). Any of these networks, and other forms of connections discussed below, could serve as conduits of interdependence between legislators. What the legislative networks literature has been lacking is an approach to research design that is causally valid. Legislative networks literature provides theoretical justification for testing for interdependence, but the extent of interdependence between legislators is still an open question due to the challenges in identifying influence in networks with observational data (Shalizi and Thomas 2011).

Field experiments on state legislatures have emerged as a standard approach to causally valid research design in the study of legislators. Bergan (2009, 331) notes the value of experimentation for exactly this case, “Random assignment of legislators to treatment and control can eliminate the potential bias that results from groups strategically choosing whom to lobby.” Field experiments have explored the relationship between constituency opinion and roll call voting (Butler and Nickerson 2011), racial conditioning in legislator communications (Broockman 2013), and the effects of lobbying on roll call voting (Bergan and Cole 2015).

Despite the separate insights offered by legislative networks scholarship and legislative field experiments, there is a degree of incompatibility in the assumptions underlying approaches in these two literatures. The interdependence between actors that represents a central concept in legislative networks research poses a challenge to the use of field experiments to identify causal effects. Network-based interdependence (i.e., influence, contagion) violates the stable unit treatment value assumption (SUTVA)—the assumption that a unit’s outcome is unaffected by other units’ treatment statuses. SUTVA is a bedrock assumption in the conventional approach to causal identification via randomized experiments (Sekhon 2008). If we take recent research on the role of networks in legislative decision-making seriously, simple randomization of treatment is likely not a robust method, as networked interdependence between legislators poses a high likelihood of interference. As Sekhon (2008, 5) notes, “When SUTVA is violated, an experiment will not yield unbiased estimates of the causal effect of interest.”

Virtually all research on legislative networks is based on observational data, lacking in design-based causal identification strategies (see Rogowski and Sinclair 2012 for an exception). Due to the interconnectedness of actors, observational research on social networks presents myriad confounding problems that place considerable limits on the feasibility of causal identification (Shalizi and Thomas 2011). As such, confronting interference in legislative field experiments presents two related research opportunities. First, accounting for interference is a vital step in producing unbiased estimates of treatment effects in the presence of SUTVA violations. Second, studying interference in field experiments on legislators represents an approach to studying networked interdependence in legislatures with a more credible identification strategy than that which is attainable in observational research. A growing body of research seeks to study interference through experimental interventions on networks (e.g., Aral and Walker 2014; Bapna and Umyarov 2015; Bond et al. 2012; Gerber, Green, and Larimer 2008; Muchnik, Aral, and Taylor 2013; Paluck 2011; ben-Aaron, Denny, Desmarais, and Wallach, 2017). These studies follow a variety of approaches to designing the interventions and testing for interference effects. However, it is clear that the field has not, as of yet, converged upon a consistent methodological framework for testing for causal effects in the presence of interference.

In this article, we review and illustrate a recently developed method that can be used to test for both direct and interference effects in experiments. This methodology, developed by Bowers, Fredrickson, and Panagopoulos (2013), allows the researcher to test for causal effects in experiments while relaxing SUTVA. Beyond the review of this methodology, we offer three contributions in this article. First, we provide a typology of theoretical considerations that researchers can draw upon when formulating hypotheses regarding interference. Second, we provide a focused review of the networks through which scholars of legislative politics should consider specifying tests for interference. Third, we apply this methodology by analyzing data from past studies that involved field experiments on state legislatures.

A Design-Based Test for Network Effects Models

In this section, we review the methodology introduced by Bowers, Fredrickson, and Panagopoulos (BFP; 2013), which enables the researcher to test for both direct and interference effects, represented by models of effects. The five components required to test hypotheses using the BFP methodology include (1) a model of effects, (2) a network, (3) a randomization design, (4) a test statistic for evaluating the model of effects, and (5) a set of parameter values to evaluate. The model of effects describes how the treatment statuses assigned to subjects affect the direct recipients of treatment and any or everyone else in the experiment (e.g., a legislator assigned to treatment changes their behavior and that of their two closest neighbors in the network). The network provides the precise representation of the ties between units (e.g., a legislator’s two closest neighbors include those with whom they have cosponsored most frequently over the past 2 years). The randomization design gives the distribution according to which the treatment is assigned (e.g., each half of the Democrats and half of the Republicans are randomly assigned to receive a call from a constituent). The test statistic is a quantity that represents the difference between the outcome observed and the outcome that would have been expected under the hypothesized parameter values and model of effects (e.g., if the treatment effect increased the directly treated legislators by 2 and their neighbors by 1, we could average the absolute values of the t statistics calculated in comparing isolated legislators, directly treated legislators, and those who had treated neighbors). The test statistic should have a monotonic relationship to the presence of differences across experimental conditions (e.g., as a t statistic increases in absolute value, the differences between the samples increase). The set of parameter values is a large grid of values that reflects the bounds of what the researcher thinks the effects could have been (e.g., the treatment had an effect between reducing support for a bill by 30% and increasing support by 30%).

The BFP test is a randomization test (Basu 2011). The uncertainty in the outcomes in the experiment is attributed to the randomization distribution (i.e., the observed outcomes would have been different if and only if a different set of treatment assignments had been drawn from the randomization distribution). Given the components described above, the process for carrying out the BFP test follows these steps:

The higher the p value associated with a parameter value, the greater the evidence for that parameter value. The intuition for this is that, if a parameter value is close to the truth, we should be able to use it to remove differences in outcomes that are attributable to the treatment assignments. Recalculating the test statistics on randomized treatment vectors provides a distribution of the test statistics under the condition in which we know that the re-randomized treatment did not affect the observed outcomes.

We illustrate the BFP test with a simple toy example. Consider an experiment in which the population under study is a legislature of nine legislators, connected through a legislative collaboration network. The collaboration network is depicted in Figure 1. The outcome under study in the experiment is the percentage of the legislation sponsored by a legislator that focuses on a particular policy issue (e.g., auto emissions limits). The treatment in the experiment is a well-studied advocacy campaign that is designed to shift a legislator’s attention toward this issue. Suppose we know, through past experimental research, that the direct effect of this treatment is an increase by 10 percentage points in the percentage of sponsored legislation that focuses on this issue. We are only interested in studying the interference effects, and so we only randomize one legislator to the treatment to assure that there are some legislators who are isolated from the treatment (i.e., have no neighbors who are treated).

OPEN IN VIEWER

Figure 1.

Toy network of nine legislators.

We represent the indirect effect as a change in the percentage of sponsored legislation focusing on the issue that is induced when a neighboring legislator is assigned to treatment. We will set the true indirect effect to be a 5-percentage point increase in the percent of sponsored legislation focusing on the issue. Assume that Legislator 5 is assigned to treatment. Legislator 5 is tied to Legislators 4, 2, 8, and 6. We present the effects of the experiment in the first few columns of Table 1. The column Y(0) gives the hypothetical outcome that we would observe if the treatment was not administered; we created this by drawing a 0 or 10 uniformly at random. Y(Z) gives the outcome observed under the assignment of treatment to Legislator 5. The other columns give the implied values that would have been observed if the treatment were not administered, given the hypothesized indirect and known direct effect, and the legislator indicated in the subscript being assigned to treatment.

OPEN IN VIEWER

Table 1.

Data from Toy Example Experiment.

Legislator	Y(0)	Y(Z)	Y5(0)	Y1(0)	Y2(0)	Y3(0)	Y4(0)	Y6(0)	Y7(0)	Y8(0)	Y9(0)
Hypothesized Indirect Effect = −6
1	0.00	0.00	0.00	−10.00	6.00	0.00	0.00	0.00	0.00	0.00	0.00
2	10.00	15.00	21.00	21.00	5.00	15.00	21.00	15.00	15.00	15.00	15.00
3	0.00	0.00	0.00	0.00	0.00	−10.00	6.00	0.00	0.00	0.00	0.00
4	10.00	15.00	21.00	15.00	21.00	21.00	5.00	15.00	15.00	15.00	15.00
5	10.00	20.00	10.00	20.00	26.00	20.00	26.00	26.00	20.00	26.00	20.00
6	0.00	5.00	11.00	5.00	5.00	5.00	5.00	−5.00	11.00	11.00	5.00
7	10.00	10.00	10.00	10.00	10.00	10.00	10.00	16.00	0.00	10.00	10.00
8	10.00	15.00	21.00	15.00	15.00	15.00	15.00	21.00	15.00	5.00	21.00
9	10.00	10.00	10.00	10.00	10.00	10.00	10.00	10.00	10.00	16.00	0.00
		abs(t stat)	3.54	1.43	1.74	1.43	1.74	2.61	0.03	1.74	1.39
Hypothesized Indirect Effect = 6
1	0.00	0.00	0.00	−10.00	−6.00	0.00	0.00	0.00	0.00	0.00	0.00
2	10.00	15.00	9.00	9.00	5.00	15.00	9.00	15.00	15.00	15.00	15.00
3	0.00	0.00	0.00	0.00	0.00	−10.00	−6.00	0.00	0.00	0.00	0.00
4	10.00	15.00	9.00	15.00	9.00	9.00	5.00	15.00	15.00	15.00	15.00
5	10.00	20.00	10.00	20.00	14.00	20.00	14.00	14.00	20.00	14.00	20.00
6	0.00	5.00	−1.00	5.00	5.00	5.00	5.00	−5.00	−1.00	−1.00	5.00
7	10.00	10.00	10.00	10.00	10.00	10.00	10.00	4.00	0.00	10.00	10.00
8	10.00	15.00	9.00	15.00	15.00	15.00	15.00	9.00	15.00	5.00	9.00
9	10.00	10.00	10.00	10.00	10.00	10.00	10.00	10.00	10.00	4.00	0.00
		abs(t stat)	0.39	0.24	0.42	0.24	0.42	0.20	1.39	0.42	0.03

Note. Y(0) gives the hypothetical outcome that we would observe if the treatment was not administered. Y(Z) gives the outcome observed under the treatment (i.e., Legislator 5 was treated). Y _X (0) gives the hypothesized value of Y(0) if treatment was assigned to legislator X.

In terms of the test statistic, in the current example, we will use the absolute value of a t statistic calculated by testing the difference between legislators that were in the control condition and isolated from (i.e., not connected to) the treated legislator, and legislators that were in the control condition and exposed to the treated legislator. The vector of outcomes used to calculate the test statistic is the vector of outcomes that results after removing the hypothesized effect of the experiment. To give an example of this implied control outcome, consider Legislator 2 and a hypothesized indirect effect of −6 (i.e., a 6-percentage point reduction in the percent of sponsored legislation focusing on the issue). Legislator 2’s true control/baseline value is 10. Legislator 2 was exposed to the treated legislator in the experiment, and we, therefore, observed the outcome (i.e., Y(Z)) of 15 because the true indirect effect is 5. However, if we hypothesize that the indirect effect is −6, this implies that Legislator 2’s control outcome is 21, which we arrive at by subtracting the hypothesized indirect effect (−6) from Legislator 2’s observed value (15).

Using the BFP method, if the hypothesized parameter/effect value is close to the true value, the implied control outcomes will be similar across experimental conditions, because removing the hypothesized effects will adjust the outcomes accurately according to how they were affected by the experiment. The test statistic is calculated for each re-randomized treatment regime, and reported in the last row of the table. The evidence for a parameter value is given by the proportion of absolute t statistics calculated on the randomized treatment regimes that are larger than the absolute t statistic calculated on the observed data. The higher this proportion, the greater the evidence for the hypothesized parameter value. Furthermore, any parameter value for which this proportion is greater than α is included in the 100 × (1 − α)% confidence interval (CI; e.g., parameter values with p > .05 are included in the 95% CI).

Considering the example data presented in Table 1, we see that there is very little evidence for the hypothesized effect value of −6. As seen in the Y₅(0) column, and indicated by the absolute t-statistic value of 3.54, removing the hypothesized indirect effect does not remove differences between control legislators who were exposed to the treated legislator and those who were not. The observed test-statistic value is not smaller than any of the values calculated when treatment is artificially reassigned to any of the other nodes, meaning that there is effectively zero evidence for an indirect effect of −6, and that the value −6 would not be included in the CI at any level less than 100%. This should be reassuring, as the true parameter value is 5. In contrast, the evidence for an indirect effect value of 6 is quite strong. At this value, the test statistic calculated on the observed data is 0.39—less than the values calculated on half of the outcome columns that result from artificially re-assigning treatment to the other legislators. The parameter value 6 would be included in CIs with levels greater than 50%. Real-world application of the BFP methodology, which we illustrate below, is much more complicated than in this toy example (e.g., we typically do not know the values of any parameters, and test many more than two parameter values), but this toy example illustrates the steps of hypothesizing models of effects and assessing their evidence.

Considerations in Testing for Interference

In this section, we offer a novel set of recommendations regarding theoretical considerations that can be drawn upon by researchers when they design experiments in which they plan to test for interference and/or specify tests to be conducted on data from field experiments that have already been conducted on legislatures. One of the virtues of controlled experiments, in which treatment allocation is randomized, is that the randomization design can be used as the basis for inference in statistical tests (i.e., design-based or randomization-based inference; Little and Rubin 2000). Testing using the Bowers et al. framework still relies on design-based inference, as the stochastic nature of the outcomes is assumed to arise from the distribution based on which the treatment was randomized. However, the hypothesis being tested is formulated as a model of causal direct and spillover effects. As these models of effects are more complicated than the conventional form of effects considered in experiments, researchers must put more thought into the functional forms that describe the relationship between the treatment and outcome vectors. It is not possible to enumerate all of the choices available in specifying the model of effects, but we discuss a few salient dimensions below.

Network Selection

The methodology introduced by Bowers, Fredrickson, and Panagopoulos (2013) is applicable in any domain of experimental political science research in which interference is suspected, and the networks through which interference might occur can be measured. There are two features of legislative politics that render methodology for testing interference particularly useful. First, because legislatures operate according to explicitly majoritarian reward systems, and it is feasible for any legislator to bargain with his or her colleagues to achieve a legislative goal, legislators face particularly strong incentives to influence each other (Bernhard and Sulkin 2013; Ferejohn 1986; Matthews 1959). Second, there is an active literature on legislative networks that offers several options to consider when testing for interference effects (Desmarais et al. 2015; Kirkland and Gross 2014). Example legislative networks that have been studied include similarity in roll call voting (Kim and Barnett 2012), bill cosponsorship (Fowler 2006), overlapping committee membership (Porter et al. 2005), collaboration in press events (Desmarais et al. 2015), comembership in caucuses (Victor and Ringe 2009), the proximity of members of Congress’s DC offices (Rogowski and Sinclair 2012), follower-followee connections among members of Congress on Twitter (Peng et al. 2016), the similarity of campaign contributions received by candidates for state legislature (Masket and Shor 2015), a survey to measure collaboration and social networks among members of the Brazilian national legislature (Wojcik 2017), demographic similarity between legislators’ constituencies (Bratton and Rouse 2011), and connections between legislative staffers (Ringe, Victor, and Gross 2013). In Table 2, we list the different networks that researchers might consider when investigating interference in legislative networks. This list is drawn directly from the literature. Given a set of prospective networks, such as these, researchers must consider through which single network, or combination of networks, spillover will occur.

OPEN IN VIEWER

Table 2.

List of Legislative Networks Drawn from Past Research.

Networks in legislative politics	Example
Roll call voting similarity	Kim and Barnett (2012)
Bill cosponsorship	Fowler (2006)
Overlapping committee membership	Porter et al. (2005)
Collaboration in press events	Desmarais et al. (2015)
Ideal point similarity	Coppock (2014)
Comembership in legislative caucuses	Victor and Ringe (2009)
Legislative staff sharing	Ringe, Victor, and Gross (2013)
Spatial proximity of legislative offices	Rogowski and Sinclair (2012)
Relationships in online social networks (e.g., Twitter)	Peng et al. (2016)
Similarity in legislators’ campaign contributions	Masket and Shor (2015)
Social network surveys administered to legislators	Wojcik (2017)
Similarity in constituency demographics	Bratton and Rouse (2011)

The determination regarding which network(s) to consider in any particular application is, of course, best made by the researchers carrying out the application. Selecting which network(s) to test is much like selecting which variables to include when specifying a model—researchers should use a combination of theory and exploration. We discuss two dimensions of interference dynamics—exposure and uptake—that should help to inform this determination. Exposure refers to the degree to which the network governs legislators’ awareness regarding each others’ beliefs or behaviors. Uptake refers to the role of the network in determining which legislators would adopt each others’ beliefs or behaviors if exposed to them. ¹ Consider a legislator’s position on a major policy issue. It is likely that each legislator in a chamber is aware of each other legislator’s opinion on a major issue, so the network does not need to play a major role in exposure to govern interference. However, to influence each other on a major policy issue, legislators may need to see each other as closely aligned ideologically. For interference dynamics that do not require exposure through the network, but require uptake, researchers should look for networks that signal ideological similarity such as co-voting on bills. However, some interference dynamics for which uptake might be highly likely, such as reuse of issue framing in legislators’ public statements (Lin, Margolin, and Lazer 2016), or the adoption of strategies in responding to constituent requests (Grose, Malhotra, and Parks Van Houweling 2015), would require legislators to be exposed to each other through explicit communication channels. In applications where the network needs to play an important role in signaling exposure, networks such as Twitter-follower networks and caucus comembership may be more appropriate. We can also think of networks that would signal both ideological alignment and explicit communication ties, such as coparticipation in press events and frequent bill cosponsorship (especially early stage, or original cosponsorships).

Note that there are two categories of processes through which interference can occur— spread of the treatment through a network (e.g., an influential lobbying communication is sent to a legislator, and that legislator forwards the communication to others in their network) and spillover of effects (e.g., a lobbying communication influences a legislator’s vote, and others in that legislator’s network take cues from their vote). A useful thought experiment in selecting networks to use in tests of interference would be to consider which networks would facilitate the spread of treatments, and which networks would facilitate the spillover of effects. It is, of course, entirely possible that the researcher either hypothesizes that more than one network plays a role in interference, and/or is uncertain regarding which network best represents potential interference relationships. In the case that the researcher hypothesizes that multiple networks serve as vectors for interference, the researcher can either create a composite network (e.g., a network in which a tie indicates that two legislators serve on a committee together and frequently cosponsor the same legislation; e.g., Ansari, Koenigsberg, and Stahl 2011), or include multiple networks in the interference model. We caution that combining networks into a single composite would introduce measurement error if interference only occurred through one or a small number of networks used in the composite. In the case that multiple models are evaluated that include different networks, the researcher should be cognizant of issues related to robustness and multiple testing bias (i.e., the increased likelihood of Type 2 inferential errors when running multiple tests of the same hypothesis). In future methodological research, it would be valuable to develop a Bonferroni-style adjustment (Cabin and Mitchell 2000) to avoid multiple testing bias with the BFP methodology. ²

Interference Model Formulation

Unlike the review of legislative networks we provided in the previous section, our discussion here is applicable to research outside of legislative politics. In this section, we focus on the mathematical structure of the model that describes how interference flows through the network. The interference model is a function that takes as its input a treatment regime (i.e., a vector that indicates the control/treatment status of each node [e.g., legislator] in the network), a network structure, and the outcomes under the uniformity trial (i.e., the outcome values in the case where each node is assigned to control), and outputs a vector of node outcomes that is conditioned on the treatment regime via the network. In other words, the interference model transforms the uniformity trial into a vector of outcomes using the network and treatment regime. For a given focal node, the two components of the model that shape the change that results from the experiment include (1) the set of other nodes whose treatment status could influence the focal node via the network, and (2) the mathematical form of the function through which those other nodes’ treatment statuses affect the focal node. Given these two components, it is possible to calculate how any given treatment regime would affect a focal node’s outcome. We discuss two important considerations in formulating the interference model to be tested. First, we discuss the specification of the neighborhood, as defined on the network structure, of nodes whose treatment status may affect a focal node (e.g., a node’s outcome depends on the treatment statuses of all nodes that are at most two hops away). Second, we discuss the specification of the functional form through which neighbors affect a focal node (e.g., the outcome of a node is a linear function of the proportion of neighbors allocated to treatment).

In Table 3, we illustrate how varying the interference model can result in different effects on a focal node. We depict two definitions of the neighborhood—one in which all nodes within two hops of the focal node affect the focal node, and one in which all nodes within three hops of a focal node affect the focal node. We also depict two definitions of the functional form of the interference effects. In one definition, all nodes in the neighborhood affect the node equally. In the other functional form, the effect of neighbors on the focal node decays with the neighbors’ distance from the focal node. Combining these two dimensions results in four alternative interference models.

OPEN IN VIEWER

Table 3.

Alternative Models of Effects, Focusing on a Single Focal Node.

	Two-hop neighborhood	Three-hop neighborhood
Constant effect
Decaying effect

Note. The red triangle represents the focal node, on which the other nodes have various effects under each model. Square nodes are treated. The circle is a control node. The darker the node’s shading, the larger the effect it has on the focal node.

Neighborhood selection

Once the researcher decides which network—or combination of networks—to use in analysis, it is important to determine the neighborhood within which the effects of the treatment can be transmitted. For example, Bond et al. (2012) find that Facebook users’ voter turnout, as expressed on their Facebook walls, influences not only their Facebook friends’ turnout decisions, but also turnout of the friends of their friends. This means that the effects of a Facebook user’s turnout decision spread within a neighborhood of two hops through the friendship network. This specification decision becomes more complicated when the network is weighted (i.e., ties can take on many values rather than just being binary tie/no tie), as in the legislative networks that we consider in our applications. In the weighted network, case transmission is likely a function of connection strength, but may also disappear at some threshold (e.g., the level of ideological distance that indicates opposition between two legislators). In our consideration of state legislative networks, we specify the neighborhood in two ways when using the ideological similarity networks:

Entire network: Treatment effect can propagate through the entire network—proportional to ideological similarity—to affect the outcome of control units.

K-nearest neighbors: Treatment effects can spread to control units from their K nearest neighbors, varying the value of K.

The definition of neighborhood depends on substantive knowledge about the interaction in a certain network. For example, a state legislature is a relatively small and internally familiar community. As such, everyone may potentially communicate with everyone else regarding major legislative tasks and actions. However, in looking at interpersonal political communication networks among regular citizens, even the closest of friends may fail to communicate about an election or other major political event.

Interference effect specification

The above two specification steps—selecting the network and the neighborhood—determine which units play a role in the interference reflected in the hypothesized model. Diffusion model specification involves defining how the treatment effect spreads through the network. We highlight two considerations—the way in which treated and untreated neighbors factor into the interference effects, and the linearity of the interference model.

The first consideration regards whether a control unit influenced by the number of treated units with which it interacts (e.g., as in an epidemic network), or by the balance or proportion of its neighbors that are treated (e.g., as we would assume in a voting or opinion-spreading network). Bowers, Fredrickson, and Panagopoulos’s (2013) specification assumes treatment spreads as a function of the number of treated neighbors. Alternatively, the Voter Model—a classic mode of opinion dynamics in networks—assumes that the proportion of treated neighbors is the relevant quantity (Valentini, Hamann, and Dorigo 2014). This specification choice likely comes down to whether the researcher assumes that the treatment and the lack of it are equally powerful forces, or whether change in the outcome can only result from exposure to treated units. If untreated neighbors can offset the effects of treated neighbors, it is likely the proportion that matters. If units are influenced only by treated neighbors, it is likely the raw count of treated neighbors that is relevant.

Although a very familiar consideration in quantitative social science, functional form assumptions are also relevant in the specification of a model of network effects. It is important to determine whether the functional form of the propagation of treatment effect should be linear or nonlinear. Does the second treated neighbor have the same effect on a node’s outcome as the first treated neighbor, or does the effect diminish? Or, alternatively, is it a threshold effect that only manifests when the number of treated neighbors reaches a critical level (e.g., a model in which a unit adopts the majority opinion among its neighbors)? Coppock (2014) adopts a linear functional form in specifying the way in which legislators learning about their districts’ opinions affects the votes of ideologically similar legislators. Alternatively, the classic Susceptible-Infected-Recovered (SIR) Model in epidemiology assumes a model in which the probability of transmission increases at a decreasing rate with the number of exposed neighbors to which a unit is exposed (Dodds and Watts 2004).

Replication Analyses: Testing for Network Effects

To illustrate the BFP methodology, we re-analyze results from two field experiments on state legislatures—Butler and Nickerson (2011) and Bergan and Cole (2015). The replication of Butler and Nickerson (2011) builds directly off the work of Coppock (2014). In each of the replications, we test causal models that include network effects. To test these models, we must specify their functional forms and select the data to use in measuring the network. For each replication, we consider two definitions of both the network through which, and the functional form according to which, network effects are transmitted, as we do not have strong prior expectations regarding exactly which network or neighborhood should be included in the models of effects. Note, our replications are not intended to serve as a meta-analysis of interference in legislative field experiments, nor to provide evidence regarding whether there is or is not interference in state legislatures, generally speaking. Rather, the purpose of the replications is to illustrate the considerations, steps, and process of testing for interference using the data produced by the experiments we replicate.

We present each replication in a separate section below. For each analysis, we present point estimates from the BFP method (i.e., the vector of parameters with the highest p value), the 95% CI (i.e., the minimum and maximum parameter values that correspond to a p value more than .05), and the 90% CI.

Butler and Nickerson (2011)

Butler and Nickerson conducted an experiment on New Mexico legislators to study the effect on legislators’ votes of their constituents’ opinions regarding the votes. In 2008, a special session of the New Mexico Legislature was called to vote on a bill regarding proposed spending plans for a budget surplus—a tax rebate. Butler and Nickerson conducted a large-scale phone survey to gather constituent opinions from across the state. Using matched-pair randomization—matching in terms of political party—35 out of 70 legislators were assigned to the treatment group. Legislators in the treatment group were sent a letter containing the district-specific support for the proposed spending plan in their own districts. Butler and Nickerson find that the effects of the treatment on legislators’ votes were conditioned by the level of support for the measure indicated in the treatment message. In districts with high support for the tax rebate, the treatment had little effect. This is because legislators generally assumed that the tax rebate would be popular, and that constituents would support the measure. In districts with low support for the tax rebate (defined as districts with levels of support below the median district in percent supporting the bill), the treatment had a negative effect on the likelihood of voting in support for the measure, as legislators in low-support districts were presumably surprised and affected by the information that the plan was unpopular in their districts.

Coppock (2014) applied the BFP methodology to test for propagation of treatment in this experiment. The indirect effect estimates were not statistically significant (Coppock 2016), even when separating the sample into low- and high-support districts. In the network that Coppock analyzed, the tie between legislators was given by their ideological similarity. Using this network, each legislator’s outcome is affected by every other legislator’s treatment status, but with varying weight based on ideological similarity. Coppock used a linear model to represent the direct and indirect effects of the treatment on the outcome. Under the model assumed by Coppock,

y i , z = y i , 0 + β 1 z i + β 2 h i ( G , z ) ,

where y_i,z is the observed outcome (i.e., 0/1 indicating vote against/for the bill), y_i,₀ is the outcome for i that would have been observed if each legislator were assigned to the control, β₁ is the direct effect of the treatment, β₂ is the indirect/interference effect, and h_i(G, z) is a function of the network G and the vector of treatment assignments (z). Coppock defines h_i(G,z) as the excess or beyond-expected sum of ideological similarities of legislator i to legislators who were assigned to treatment. The steps used in calculating this “excess exposure” to the treatment are as follows:

Calculate W-NOMINATE ideology score (ideo) for each legislator using roll call vote data.

Calculate ideological similarity as S i m i l a r i t y i , j = 2 − | i d e o i − i d e o j | 2 .

Calculate raw exposure as R a w e x p o s u r e i = ∑ j = 1 n S i m i l a r i t y i , j × z j , j f = i .

Coppock introduces an adjustment for the expected exposures of legislators. Adjusting for expected exposure removes endogeneity in the network exposure function (h_i(G,z)). Exposures are simulated under a large number of re-randomizations of the treatment (10,000 in both Coppock’s and our application), and the average exposure value for each legislator is subtracted from the raw exposure value to give the excess exposure value (i.e., h_i(G, z) = Raw exposure_i − Expected exposure_i).

We follow Coppock in the adjustment of the exposure function for expected exposure. We extend the model of effects used by Coppock (2014) to incorporate differential effects of low- and high-support treatment in one model, rather than running the analysis for the two separate subpopulations. We take this approach because we assume that treatment of legislators in low-support districts can influence the outcomes of legislators in high-support districts and vice versa—a dynamic that is lost when splitting the sample into low- and high-support districts. The model of effects we use is

y i , z = y i , 0 + β 1 z i l i + β 2 z i ( 1 − l i ) + β 3 h i ( G , z × l ) + β 4 h i ( G , z × ( 1 − l ) ) ,

where l_i is an indicator (0/1) of whether legislator i is in a low-support district. This model form separates the direct and indirect effects based on whether the treatment delivered to the legislators incident to the treatment were in low- or high-support districts. We hypothesize that exposure to low-support treatment (either direct or indirect) will reduce support for the bill, and exposure to high-support treatment (either direct or indirect) will increase support for the bill.

In part, to build on what Coppock has already contributed with this replication, and also to focus more closely on networks that indicate a higher likelihood of contact/communication between legislators, we depart from Coppock in the definition of the networks. For each network, we consider both binary and weighted forms of the interference effect. First, we analyze a network in which two legislators are connected based on copartisanship and co-committee membership. In the binary version, legislator i is exposed to legislator j’s treatment status if legislator i is of the same party as legislator j and serves on at least one standing committee with legislator j during the session preceding the special session on the budget surplus. In the weighted version, legislator i is exposed to legislator j’s treatment status in proportion to the number of committees on which they served together if they are copartisans. We used the committee network based on the assumptions that (1) legislators will consider the preferences of constituents of their copartisans as relevant to their own votes, and (2) they are likely to be in contact with legislators with whom they regularly collaborate through committee assignments. ³ The second network we consider is defined by the cohorts in which two legislators arrived in the legislature. In the binary version of the cohort network, legislator i is exposed to legislator j’s treatment status if they are copartisans and were elected in the same year. In the weighted version of the cohort network, legislator i is exposed to legislator j’s treatment status in proportion to 1/(1 +|c_i − c _j|), where c_i is the year in which legislator i was elected. The weighted cohort analysis reflects a different network neighborhood than the co-committee membership network. If two legislators are copartisans, they are assumed to influence each other through cohort similarity (albeit at a rate that decreases with the difference between the two legislators’ cohorts). Cohort membership is used to proxy social ties between legislators. ⁴

The next detail we need to fill in when it comes to applying the BFP methodology is the test statistic used in the analysis. To refresh, the test statistic should be designed to quantify the degree to which the outcome is unrelated to the experimental conditions once the hypothesized effects have been removed from the observed outcome. We follow Coppock and Bowers, Fredrickson, and Aronow (2016) and use the following steps to calculate the test statistic:

Estimate the outcome under control for each observation as y_iˆ,₀ = y_i,z − [β₁z_il_i + β₂z_i(1 − l_i)+ β₃h_i(G, z × l)+ β₄h_i(G, z × (1 − l))], where the β’s are given by their hypothesized values.

Fit the regression equation y_iˆ,₀ = γ₀ + γ₁z_il_i + γ₂z_i(1 − l_i)+ γ₃h_i(G, z × l)+ γ₄h_i(G, z ×(1 − l), estimating the γ’s by ordinary least squares (OLS).

Set the test statistic equal to the residual sum of squares (RSS) = ∑ _i [y_iˆ,₀ − γ₀ − γ₁z_il_i − γ₂z_i(1 − l_i) − γ₃h_i(G, z × l) − γ₄h_i(G, z × (1 − l))]².

The intuition behind using the RSS is that, if the hypothesized parameter values remove the effect of the experiment from y_i,z, the RSS from regressing y_iˆ,₀ on variables defined by the model of effects will be high, as the effects of the experiment were removed from the dependent variable prior to running the regression on which the RSS is based.

The last detail of implementing the BFP framework regards the grid of hypothesized parameters over which p values are calculated. Because the testing process does not involve an optimization routine, there is no way for the parameter values to be selected automatically. However, standard optimization methods can be used to approximate point estimates around which to expand the grid of hypothesized values. In our applications, the model of effects has a linear form, and we can use linear regression to find the estimates around which to expand the grid. In terms of how far to expand the grid—there should be enough grid points that none of the point estimates are close to the boundaries of the grid.

Results for Butler and Nickerson (2011) data

The results of the Butler and Nickerson (2011) replication are presented in Table 4. The first notable result is that each CI in each model includes zero. None of the parameter estimates are statistically significantly different from zero at the .05 level. However, in most of the models, most of the parameters are of the expected sign. In each model, the effects of both low-support effects are negative, and the indirect high-support effect is positive. With the copartisan cohort network, in both the binary and weighted specifications, the indirect low-support effect is statistically significant at the .10 level. To interpret our results, we look to the estimates from the binary network of same cohort copartisans. The low-support effect of −.20 indicates that, for a legislator in a low-support district, being assigned to treatment reduces the probability of a legislator voting in favor of the bill by .20. Similarly, having a copartisan who was elected in the same cohort who is in a low-support district assigned to treatment, reduces the probability of voting in favor of the bill by .20. We note two central takeaways from this analysis. First, they represent moderate evidence of interference—especially via exposure to treated neighbors in low-support districts. Second, the BFP methodology exhibits relatively low statistical power, as the direct effects that are statistically significant in the original analyses by Butler and Nickerson are not significant in our analysis. This suggests that field experiments on legislatures in which the researcher is interested in studying interference may require larger sample sizes (e.g., via administering treatment in multiple states, on multiple legislative proposals, and/or across both chambers).

OPEN IN VIEWER

Table 4.

Results from Applying the BFP Methodology to the Replication of Butler and Nickerson (2011).

	Binary ties			Weighted ties
	Estimate	95% CI	90% CI	Estimate	95% CI	90% CI
Results for Committee Copartisans Network
Direct (L)	−0.25	[−0.50, 0.03]	[−0.50, 0.00]	−0.30	[−0.50, 0.05]	[−0.50, 0.01]
Direct (H)	0.00	[−0.25, 0.45]	[−0.20, 0.35]	0.00	[−0.30, 0.40]	[−0.30, 0.35]
Indirect (L)	−0.05	[−0.15, 0.10]	[−0.10, 0.05]	−0.03	[−0.10, 0.10]	[−0.10, 0.05]
Indirect (H)	0.10	[−0.05, 0.20]	[−0.03, 0.20]	0.05	[−0.10, 0.15]	[−0.05, 0.10]
Results for Cohort Copartisans Network
Direct (L)	−0.20	[−0.50, 0.10]	[−0.50, 0.05]	−0.20	[−0.50, 0.05]	[−0.50, 0.03]
Direct (H)	−0.10	[−0.40, 0.30]	[−0.35, 0.25]	−0.15	[−0.40, 0.15]	[−0.35, 0.15]
Indirect (L)	−0.20	[−0.50, 0.01]	[−0.50, –0.02]	−0.20	[−0.45, 0.05]	[−0.45, –0.02]
Indirect (H)	0.05	[−0.15, 0.35]	[−0.10, 0.35]	0.10	[−0.10, 0.35]	[−0.05, 0.30]

Note. (L) indicates the effect of low-support treatment, and (H) indicates the effect of high-support treatment. The outcome variable is 0/1, where 1 indicates a vote in favor of the proposal to create a tax rebate to distribute a budget surplus (voted on in the New Mexico Legislature 2008). p Values calculated using 1,000 randomization iterations. There are 70 legislators in this dataset/network. CI = confidence interval. BFP = Bowers, Fredrickson, Panagopoulos.

Bergan and Cole (2015)

The second dataset we work with comes from an experiment on the Michigan legislature. This experiment was conducted on legislators from both chambers, in the context of antibullying legislation. Legislators were stratified based on six background variables. The treatments were calls from constituents expressing their support for the proposed bill. Treatment was given in three different doses, which differed in the number of calls placed to the given legislator. The results were that this treatment had a significant effect on the final vote on the bill. They observed a 12-percentage point increase in the likelihood of voting in favor of the antibullying bill for those treated.

These data have not been analyzed for indirect effects previously. However, because legislators consider legislation in the context of open communication and debate, and for supporters to assure that legislation passes they need the votes of their colleagues, we expect treatment effects of roll call votes to be characterized by interference. In our replication of Bergan and Cole (2015), we use a very similar model of interference, except the indicators for low support and high support are replaced by indicators for Democrat (D) and Republican (R). Bergan and Cole (2015) tested for, but did not find evidence, that partisanship or ideology moderated the effect of the treatment, but we incorporate partisan moderators into our model to allow for different effects of the advocacy treatment by party. Our model for the Bergan and Cole (2015) replication takes the following form:

y i , z = y i , 0 + β 1 z i d i + β 2 z i ( 1 − d i ) + β 3 h i ( G , z × d ) + β 4 h i ( G , z × ( 1 − d ) ) ,

where d_i is a 0/1 indicator of whether legislator i is a Democrat. We use the same test statistic as in the Butler and Nickerson (2011) replication.

As in the Butler and Nickerson (2011) replication, we consider two different networks and functional forms. We again test the copartisan cohort network. ⁵ For the Bergan and Cole (2015) replication, we use cosponsorship instead of co-committee membership to measure formal legislative connection between legislators. In the binary form of the cosponsorship network, we include a tie between any two legislators who cosponsored two or more of the same bills. ⁶ In the weighted form of the cosponsorship network, the effect of j’s treatment status on i is proportional to the number of bills that were cosponsored by both i and j. We do not have theoretical expectations regarding the different effects of committee comembership and cosponsorship in New Mexico and Michigan, the difference is driven by data availability.

Results for Bergan and Cole (2015) data

The results of the Bergan and Cole (2015) replication are presented in Table 5. In this application, we find one statistically significant (at the .10 level) interference effect—a reduction in the probability of voting for the bill by .10 from indirect exposure to treated Democrats in the copartisan binary cohort network. The signs of the interference effects are mostly negative, which is counterintuitive, but could arise if exposure to other legislators who received the advocacy call resulted in awareness that the call was part of an advocacy campaign and not an independent contact from a constituent. The signs of the direct effect estimates are consistently positive, as was found in the original analysis. However, the statistical significance of the direct effect estimates is not robust to the choice of network used to model the interference effects. The direct effects for both Democrats and Republicans are statistically significant at the .05 level in the copartisan cohort network, but not significant with the cosponsorship network. To draw a conclusion regarding the direct effects in the current analysis, we need to either make a theoretical argument that the model based on the copartisan cohort network is more valid, or note the lack of robustness and need for further replication of the experiment to draw firm conclusions. The lack of robustness regarding direct effects in the current application provides further indication that, in experiments that may be subject to interference, we require larger samples to make sufficiently powerful inferences. Larger samples could be achieved by including more votes and/or legislatures in the experiment.

OPEN IN VIEWER

Table 5.

Results from Applying the BFP Methodology to the Replication of Bergan and Cole (2015).

	Binary ties			Weighted ties
	Estimate	95% CI	90% CI	Estimate	95% CI	90% CI
Results for Cosponsorship Network
Direct (D)	0.15	[−0.15, 0.40]	[−0.10, 0.35]	0.10	[−0.15, 0.40]	[−0.10, 0.40]
Direct (R)	0.00	[−0.10, 0.40]	[−0.10, 0.40]	0.10	[−0.05, 0.45]	[−0.03, 0.40]
Indirect (D)	0.00	[−0.10, 0.20]	[−0.10, 0.15]	0.05	[−0.03, 0.20]	[−0.03, 0.20]
Indirect (R)	−0.05	[−0.10, 0.10]	[−0.10, 0.10]	−0.15	[−0.20, 0.05]	[−0.20, 0.05]
Results for Cohort Copartisans Network
Direct (D)	0.40	[0.10, 0.50]	[0.10, 0.50]	0.20	[0.05, 0.50]	[0.10, 0.50]
Direct (R)	0.20	[−0.03, 0.45]	[0.00, 0.40]	0.50	[0.40, 0.50]	[0.45, 0.50]
Indirect (D)	−0.10	[−0.20, 0.00]	[−0.15, –0.01]	−0.10	[−0.15, 0.03]	[−0.10, 0.01]
Indirect (R)	−0.05	[−0.10, 0.05]	[−0.10, 0.05]	−0.20	[−0.20, 0.03]	[−0.20, 0.03]

Note. (D) indicates the effect of Democrat treatment, and (R) indicates the effect of Republican treatment. The outcome variable is 0/1, where 1 indicates a vote in favor of the antibullying bill (voted on in both chambers of the Michigan Legislature 2011). p Values calculated using 1,000 randomization iterations. There are 144 legislators in the dataset/network. CI = confidence interval. BFP = Bowers, Fredrickson, Panagopoulos.

Conclusion

In this article, we have made the case that scholars who run field experiments on state legislatures should consider testing for interference. We provide guidance in specifying these tests using the methods developed by Bowers, Fredrickson, and Panagopoulos (2013). Specifically, we discuss options for specifying the network(s) through which interference occurs, selecting the neighborhood of legislators who affect the legislator through the network(s), and specifying the functional form according to which the interference effects manifest. We illustrate this approach with two in-depth replications. We do not find consistent evidence for interference effects in our replications. Our mixed findings regarding interference effects are attributable to an actual lack of interference in some contexts, a misspecification of the model of effects (which could include using the wrong network[s]), or sample sizes that are too small. Nonetheless, these replications serve to illustrate the variety of choices researchers have to make when testing for interference effects in experiments on state legislatures.

The results from our replication of field experiments on legislatures underscore the importance and complexity of accounting for interference. The replication and extension of Butler and Nickerson (2011) exhibited moderate evidence for interference—through the copartisan cohort network, particularly. We also observed some evidence for interference through the copartisan cohort network in the re-analysis of data from Bergan and Cole (2015). One consistent finding from these two replications is that the copartisan cohort network exhibits the greatest evidence as a vector for interference. Our replication study is not intended to provide definitive evidence regarding whether or not state legislative field experiments are subject to interference effects. Rather, we illustrate a broad array of network and neighborhood definitions, and provide evidence that some experiments on state legislatures are characterized by interference effects, and some are not. Given that tools are now available for testing interference effects, researchers have little reason to assume SUTVA in legislative field experiments. In the replication materials for this article, we include an R package that implements functions for carrying out the testing methodology developed by Bowers, Fredrickson, and Panagopoulos (2013).

One shortcoming of our replication analyses is that the experiments were designed and data collected with a focus on direct effects, assuming SUTVA. We retrospectively constructed networks to use in testing for interference, relaxing SUTVA, which is not ideal as there are likely to be more appropriate networks for each individual application. In future state legislative field experiments, researchers should consider collecting network data that characterize the patterns of interdependence between legislators that are most relevant to their experiments. Furthermore, in each of the studies we consider, half of the observations were allocated to treatment, and treatment allocation was uniform-at-random (within blocks). This is not the optimal randomization design if the researcher is interested in testing for and identifying interference effects. In experiments designed for testing interference effects, the optimal proportion assigned to treatment is typically much lower than 50% (Bowers et al. 2018). Furthermore, researchers can use the networks through which they think interference occurs to design higher powered experiments that incorporate the network structure (Bowers, Fredrickson, and Aronow 2016). Higher powered experiments can, of course, also be achieved through an expanded sample size. As we have noted above, when testing for interference, it may be advisable for the researcher to design an experiment that is applicable to multiple legislative actions and/or multiple legislatures to expand the sample.