Agent-Based Models for Assessing Complex Statistical Models: An Example Evaluating Selection and Social Influence Estimates from SIENA

Abstract

Although agent-based models (ABMs) have been increasingly accepted in social sciences as a valid tool to formalize theory, propose mechanisms able to recreate regularities, and guide empirical research, we are not aware of any research using ABMs to assess the robustness of our statistical methods. We argue that ABMs can be extremely helpful to assess models when the phenomena under study are complex. As an example, we create an ABM to evaluate the estimation of selection and influence effects by SIENA, a stochastic actor-oriented model proposed by Tom A. B. Snijders and colleagues. It is a prominent network analysis method that has gained popularity during the last 10 years and been applied to estimate selection and influence for a broad range of behaviors and traits such as substance use, delinquency, violence, health, and educational attainment. However, we know little about the conditions for which this method is reliable or the particular biases it might have. The results from our analysis show that selection and influence are estimated by SIENA asymmetrically and that, with very simple assumptions, we can generate data where selection estimates are highly sensitive to misspecification, suggesting caution when interpreting SIENA analyses.

Keywords

SIENA networks stochastic actor–based model influence effects selection agent-based models simulated data simulation

Data simulation, as defined by Kéry and Royle (2016), is the “generation of random numbers from a stochastic process that is described by a series of distributional statements.”¹ There are many benefits of simulated data, such as testing the sensitivity of certain model parameters, or examining the robustness of an estimator and the effects of violations of estimator assumptions (Kéry and Royle 2016).² The value in using data simulation in the testing and benchmarking of novel analysis methodologies is that the ground truth conditions that generated the data are known. Furthermore, counterfactual experiments can be readily created allowing testing of methodologies against theoretical conditions for which real data are not available.

Typically, data are simulated by first specifying the statistical distributions (including possible covariate interaction) and experimental conditions (such as missing data or observer bias), after which parameter values are selected that result in the desired statistical outcomes. However, when studying problems that involve interactions between actors and nonlinear dependencies, statistical distributions and interactions might be too coarse to evaluate the robustness and validity of a statistical model (Manzo 2014).

In this article, we argue that ABMs are particularly useful to assess models when the phenomena under study is complex, which can be understood as behavior that “cannot be reduced to the simple sum, or aggregation, of individual behaviors” (Hoffer 2013). Agent-based models³ (ABMs) are effective for the generation of complex simulated data because they are typically designed by specifying the microrules for local interactions which lead to emergent or surprising behavior at the macrolevel (Kéry and Royle 2016; Matthews et al. 2007). Although ABMs have been increasingly accepted as a valid tool in social sciences to formalize theory (Squazzoni 2012), to propose mechanisms able to recreate regularities, and to guide empirical research (Chattoe-Brown 2013; Conte and Paolucci 2014; Macy and Willer 2002), we are unaware of previous research that uses ABMs to assess the robustness of statistical methods.

Through the generation of data that not only have specific desired distributions and interactions, but possibly other emergent structures and correlations, ABMs offer a unique opportunity to evaluate our analytic tools. This approach also allows increased flexibility in specifying the behavior of complex systems by including qualitative insights (Ghorbani, Dijkema, and Schrauwen 2015), formal model learning (Epstein 2006; Squazzoni 2012), and interactions between actors (Gibert and Hamill 2016). As a demonstration of this methodology, we use an ABM to generate data that is then used to assess the Simulation Investigation for Empirical Network Analysis (SIENA) estimation of selection and influence effects. SIENA is a network statistical model to estimate network dynamics for longitudinal network data collected as panel data. This particular example is relevant for several reasons. Firstly, SIENA is a relatively new and prominent methodology employed by a diverse and growing number of publications in social sciences. Secondly, we still know little about its robustness. Thirdly, while some work has been done testing some of SIENA’s assumptions, to our knowledge, no one has tested specific mechanisms for network dynamics (i.e., network selection and social influence). And finally, social dynamics exhibit a number of complexities that are poorly captured by linear models.

We divide the remainder of this article into seven sections. The first section briefly introduces and motivates SIENA and discusses some of its fundamental assumptions. In the second and third section, we describe our ABM and the scenarios used to assess SIENA’s performance. We also discuss the calibration and verification procedures implemented. In the fourth section, we present the SIENA specifications utilized to analyze the generated data, while the fifth section describes our experimental design. We then discuss the results showing the effectiveness of SIENA when estimating selection and influence, address the implications of our results, and motivate the use of ABMs to assess our statistical tools.

SIENA

Background and Motivation

A powerful and pervasive regularity in the social world is the interdependence between social connections and individual traits, behaviors, and attitudes. Those who are proximate or share the same social connections tend to be more similar than those who do not. This phenomenon is usually explained with two different processes: selection and influence. On the one hand, it is argued that the similarity of attributes and behavior is just the result of the selection of network partners, either because we are more likely to contact those who are proximate (i.e., those with whom we share the same geographic location or organizational context) or because we prefer similar peers due to cognitive mechanisms (Bishop 2009; McPherson, Smith-Lovin, and Cook 2001). On the other hand, a long line of investigation suggests that social networks affect and modify individual traits and behaviors as diverse as health, deviance and criminality, norms, educational attainment, smoking, and migration (Christakis and Fowler 2011).

Great effort has been paid to empirically identifying selection and social influence. Identifying which mechanism plays the strongest role can be decisive for understanding the evolution of a social system and, consequently, the success or failure of intervention programs. A policy that is successful in a context where selection is predominant may fail when influence is the main social force of change. Unfortunately, disentangling selection and social influence have proven to be extremely difficult (Shalizi and Thomas 2011; VanderWeele 2011). To infer selection or sorting, we would need dynamic microlevel data that capture changes in individual social networks and connections while behavior remains the same. To infer influence, in turn, we need data measuring changes in behavior while the individual social network remains the same. Furthermore, we would need to account for confounding processes that affect both behavior and network formation.

These challenges have not prevented the development of statistical methods aiming to identify selection and social influence.⁴ One of the most recent and prominent efforts to simultaneously model selection and influence is SIENA, a model proposed by Snijders (2001). This method has gained popularity over the years and been used in a number of publications estimating selection and influence for a wide range of behaviors and traits: delinquency (Knecht et al. 2010; Weerman 2011; Weerman, Wilcox, and Sullivan 2017), violence (Dijkstra, Berger, and Lindenberg 2011; Huitsing et al. 2014; Rulison, Gest, and Loken 2013), smoking (Haas and Schaefer 2014; Mercken et al. 2010; Steglich et al. 2012), drinking (Huang et al. 2014; Knecht et al. 2011; Mercken et al. 2012), drugs (Pearson, Sieglich, and Snijders 2006), depression (Schaefer, Kornienko, and Fox 2011; Van Zalk et al. 2010), obesity (Shoham et al. 2012; Simpkins et al. 2013; Zhang et al. 2015), educational achievement (Lomi et al. 2011; Rambaran et al. 2016), and among others.

Despite the increasing number of publications using this method, we know little about its robustness. SIENA estimates selection and influence by reproducing unobserved behavioral and network changes over time. However, we do not know how reliable and successful these imputations (and their assumptions) are. As Edmonds and Meyer (2013) put it, “When this technique (SIENA) is reliable and what its particular biases are, have not yet been established” (p. 729).

There exist simulation-based studies for SIENA, though. For instance, sensitivity analyses on different types of missing data using SIENA have been discussed in Huisman and Snijders (2003) and Huisman and Steglich (2008). Stadtfeld et al. (2017), on the other hand, examine how change of network ties and the composition of actors (leavers and joiners) affect SIENA’s statistical power. However, those studies use the same data generation mechanisms specified by SIENA. To our knowledge, no published work has assessed the robustness of SIENA estimates using alternative mechanisms.

We set out in this example to pose two questions: (1) How sensitive SIENA estimates are to different levels of misspecifications? and (2) Under what circumstances does SIENA yield valid estimates? To answer these questions, we design an ABM that generates synthetic data and reproduces some common aggregate regularities of social networks such as low network density, fat-tail distribution of degree, short paths, reciprocity, and clustering. Then, by controlling the parameters of the ABM (including whether influence or selection is enabled), we explore how well and under what conditions SIENA produces reliable and consistent estimates. Since we never know the optimal specification of SIENA, it is valuable to know how robust its estimates are to misspecification and to identify which of them are particularly consequential.

SIENA Description

Stochastic actor-based models have the primary aim of statistical inference of processes of network and behavior change.⁵ Although they can be thought of as ABMs because decisions (i.e., changes in ties and behavior) are made from the perspective of a focal actor, they also include elements of generalized linear statistical models not often seen in ABMs (Snijders and Steglich 2015).

SIENA’s authors argue that their technique overcomes critical limitations of traditional methods. First, SIENA estimates selection and influence as parallel processes using complete network data, enabling feedback between network and behavior changes (i.e., a coevolution model), and adjusting statistically for structural network tendencies, such as reciprocity and transitivity, that might explain the formation or destruction of ties. It is argued that not controlling for such tendencies might misleadingly lead us to infer sorting or assimilation effects (Steglich, Snijders, and Pearson 2010). Second, SIENA addresses the missing data problem that results from using discrete-time panel data. This reminds us of the identification problem of selection and social influence effects when using cross-sectional data. Using simulation to impute unobserved network and behavior changes in a continuous time framework, SIENA’s authors claim to facilitate identification (Steglich et al. 2010; Veenstra and Steglich 2012). However, this strategy does not overcome problems related to latent homophily (i.e., unobserved selection processes, see Shalizi amd Thomas 2011). Using observational data, we cannot exclude the possibility that perceived social influence concerning a variable $x$ might be the consequence of earlier selection choices on an unobserved variable u. The only way SIENA can actually disentangle influence and selection is under the assumption that the observed network and individual variables contain enough of the information that plays a role in the causal process, in addition to the correct specification of the functional forms.⁶

SIENA usually models repeated measures of the same network and group of actors, assuming they result from a continuous process of change. In this framework, a big change from one observation to the next is conceived as the accumulation of a sequence of small changes, restricting actors in the model to changing their behavior by a predefined step size, or their social networks by one tie at a time. Since the precise trajectory of small changes from one observation to another is unknown (missing data), the model relies on data augmentation procedures (imputation) and simulation-based inference. It also assumes that actors act independently without coordination. Change needs to occur slowly enough, so actors can make decisions in response to a surrounding network that remains approximately stable.⁷

As SIENA actors control their network ties and behavior, they have to determine when to make a decision and which decision to make. Two rate functions are used to determine when they make a decision: one for behavior and one for network. Both are exponential distributions independently estimated from the data.⁸ Repeated samples from these rate functions determine the time between decisions. Changes, on the other hand, are defined using deterministic objective functions. These model attractiveness of network states $x'$ to actor i. Given the current network $x$ and the vector of behavior scores for all actors $z$ , actor i will choose the next network state $x'$ , by changing any one of its current network ties that maximizes the function $f_{i}^{net} (x, x', z) + ε_{i}^{net} (x, x', z)$ , where $f^{net}$ is the network objective function and $ε^{net}$ is a random disturbance term for unexplained change. This implies actors observe and know the entire topology of the network when making their decisions. Similarly, actor i will incrementally change the value $z'$ (behavior) that maximizes $f_{i}^{beh} (x, z, z') + ε_{i}^{beh} (x, z, z')$ , where $f^{beh}$ is the behavior objective function, with the associated random disturbance term, $ε^{beh}$ . After making assumptions on the distribution of the random term (Gumbel), choice probabilities can be expressed in multinomial logit shape where the denominator corresponds to the sum over all possible next network or behavior states:

Pr (x \to_{i} x') = \frac{exp (f_{i}^{net} (x', z))}{\sum_{x ″} exp (f_{i}^{net} (x ″, z))},

Pr (z \to_{i} z') = \frac{exp (f_{i}^{beh} (x, z'))}{\sum_{z ″} exp (f_{i}^{beh} (x, z ″))} .

Both network and behavior objective functions are modeled using linear combinations of effects that represent associations between network, behavior, and exogenous variables. They are defined as:

f_{i} (x, x', z) = \sum_{h} β_{h} s_{h} (i, x, x', z),

where $s_{h}$ represents a given system effect (either network or behavior) with an associated weight parameter $β_{h}$ . The weight parameters are estimated from the data. They describe the preferences of system effects of actors. In other words, when $β_{h}$ is positive, system states with higher scores $s_{i h}$ are more likely than states with lower scores and vice versa. Moreover, these objective functions are homogeneous: all actors maximize their network and behavior statistics using the same function. Although it is possible to introduce actor-level heterogeneity through actor covariates, it requires knowledge on which actors are different.⁹ The evolution model is implemented using a stochastic simulation algorithm to generate network and behavior data according to a proposed dynamic process using the method of moments estimation routine.¹⁰

To provide an intuitive explanation of the output SIENA provides, we briefly review a toy example discussed by Steglich et al. (2010). This example includes both homophilous selection and influence (assimilation) effects and uses a behavior variable $z$ that ranges from 1 to 5, with an overall observed average over all waves of $\bar{z} = 3$ . The network objective function contains three structural effects (out-degree, reciprocity, and behavior similarity) with their corresponding estimated parameters:

f_{i}^{net} (x, x', z) = - 2.0 \sum_{j} {x'}_{i j} + 2.5 \sum_{j} {x'}_{i j} x_{j i} + 1.0 \sum_{j} {x'}_{i j} {sim}_{i j} .

The out-degree effect, $\sum_{j} {x'}_{i j}$ , is the tendency of actors to send out ties in the network. Its weight is −2.0. Disregarding other parameters, this means that, given an opportunity to change, the odds for any tie to be present versus absent are $exp (- 2.0) = 0.14$ . The network effect of reciprocity, $\sum_{j} {x'}_{i j} x_{j i}$ , describes the tendency of actors to reciprocate incoming ties by extending outgoing ties. The odds of reciprocating an incoming tie depend on both the reciprocity and the out-degree effects; therefore, the odds of reciprocating an incoming tie versus not doing it are $exp (- 2.0 + 2.5) = 1.65$ . Finally, the similarity effect, $\sum_{j} {x'}_{i j} {sim}_{i j}$ , is the tendency to have ties with similar alters rather than dissimilar ones, and its weight is 1.0, which corresponds to the odds of 2.72. That is, selecting similar actors is almost three times more likely than connecting dissimilar alters.¹¹

The behavior objective function, on the other hand, describes the preference of behavior values. It includes two terms describing an intrinsic behavior preference, and one accounting for the average similarity to neighbors:

f_{i}^{beh} (x, z, z') = - 1.0 ({z'}_{i} - \bar{z}) - 0.5 ({z'}_{i} - \bar{z})^{2} + 2.5 \frac{\sum_{j} x_{i j} {si m^{'}}_{i j}}{\sum_{j} x_{i j}} .

In this formulation, behavior is centered and ranges from −2 to +2. The linear and quadratic shape parameters of the intrinsic behavior preference define a parabolic objective function with a maximum centered at the score −1 (i.e., 2 in the original scale). In other words, the distribution of behavior over the long run will be unimodal with a maximum value at 2, indicating the intrinsic optimal behavior value. The odds of moving from the average score of 3 to the optimum score of 2 versus staying at 3 are $exp (- 1.0 [(2 - 3) - (3 - 3)] - 0.5 [(2 - {3)}^{2} - {(3 - 3)}^{2}]) = 1.65$ . That is, without accounting for actor similarity, moving down to 2 is more likely than staying at 3. The positive parameter for average similarity, on the other hand, suggests a tendency to behave in the same way as peers do (i.e., assimilation). If an actor has a behavior score of 3, and five friends each having a higher behavior score such that the average similarity is −0.25, we can compute the odds, due to similarity alone, of staying at 3 versus moving down to 2: $exp (2.5 \times - 0.25) = 0.535$ .¹² In this case, the preference is to stay at 3. Bringing the two effects together, we get the odds of $1.65 \times 0.535 = 0.883$ for moving down to 2 instead of staying at 3, indicating a slight preference to remain at 3, due to social effects.

Although network and behavior dynamics are being modeled independently, SIENA allows feedback between them. More specifically, both homophilous selection and influence effects are estimated simultaneously in the same model (although in different parts) and are controlled for each other (i.e., they are statistically separated). That is why SIENA’s authors claim their method allows disentangling selection and influence effects.

In sum, SIENA makes inferences on networks and behavior dynamics as if each actor maximizes linear combinations of network statistics, behavior, and covariates. This strategy is intended to include all the available information and overcome the difficulties of fitting and understanding a statistical model, but it does not mean the actual processes follow additivity or linearity, or that people select peers/friends by counting the number triangles or reciprocal ties, or actors have a clear notion of all their social connections and can make complicated computations to assess the short-term consequences of adding, keeping or subtracting ties. These differences between the actual processes and SIENA’s assumptions imply that SIENA's algorithm is impartial to the underlying data-generating microprocesses, provided network/behavior statistics are good summaries of the actual processes and produce valid inferences of small change trajectories.

To assess SIENA’s performance, the ideal would be to use ground-truth data; however, these data are hard to obtain in human populations. We would need fine-grained data with no (or very low) measurement error and be able to manipulate selection and social influence processes. An alternative, and more feasible, strategy is to use a generative model to create dynamic data in which we know the mechanisms at play and the small change trajectories (Epstein 2006). This generative model needs to be different from SIENA but still comparable. For this purpose, we create an ABM of behavior and network dynamics that differs somewhat from the mechanisms assumed by SIENA in order to assess its validity and robustness. At best, our model would be a better analogy of the actual processes, and at worst, would be just as different from the mechanisms specified by SIENA allowing to assess its performance under misspecification.

Influence Selection ABM (ISA): A Network and Behavior Dynamic ABM

We develop an ABM, called the ISA, that generates data matching as closely as possible the requirements of SIENA, allowing us to assess its inferences. It is important that any ABM used to test SIENA not follow its same actor behavior assumptions, as we already know the real processes do not follow SIENA’s algorithm. This is what we mean by misspecification: There is no direct translation between the generative processes of ISA and SIENA, so any SIENA specification would be just an approximation to the actual mechanisms, as occurs when analyzing real data. The question is whether SIENA is able to make accurate statistical inferences given this known difference.

While agent-based modeling cannot by itself solve the empirical estimation and identification problems of the selection and influence effects, it offers a unique opportunity to rigorously implement theoretical and empirical assumptions on these processes. By construction, all group- and individual-level variables defining agent attributes are perfectly known (i.e., there is no unobserved heterogeneity). Also, the way in which agents are linked and the composition of their local neighborhood are also completely transparent.

Our ABM is not intended to perfectly represent the actual processes at play but provides a simplified analogy of potential selection and social influence mechanisms. Although these simplified mechanisms might seem plausible, they are still artificial. If SIENA performs well using a different but simplified version of homophilous selection and social influence, we can argue SIENA’s estimates are robust to particular levels of misspecification (those given by our ABM). This does not confirm SIENA’s ability to produce valid selection and influence estimates in the real world. It does, however, increase our confidence in SIENA estimates, especially if our ABM is empirically appealing. If SIENA performs poorly, we can argue its ability to estimate selection and influence is limited and sensitive to the misspecifications given by our ABM. The next question would be which assumptions seem to be the most critical and how likely they are met in reality.

As we discussed in the previous section, some of the key SIENA assumptions and requirements include: (1) each microstep, consisting of a change to a single network edge or a behavior change of a discrete amount, does not depend on past changes (i.e., no coordination between agents, no memory), (2) the rate of change in network and behavior follows an exponential distribution (i.e., continuous time framework) and is constant across actors,¹³ (3) all actors share the same objective function when considering how to change their connections and behavior (i.e., low heterogeneity among actors), and (4) all actors are fully aware of the topology of the network when choosing how to change an tie. In addition, ideal data against which to fit SIENA will have no leavers or joiners, or missing data, though, SIENA can accommodate some of these issues.

With these assumptions in mind, we design ISA following four key design concepts. First, we follow the social circles model proposed by Hamill and Gilbert (2009, 2016) to define the social network dynamic. That model, inspired by Simmel’s (1902) theory, adapts the idea that people are embedded in social circles. It generates a social network by giving each agent a social reach (a radius r defining its social circle) of alter agents and produces personal (ego-centric) networks that are limited in size (low network density), vary in size between individuals, display right-skewed distributions, high clustering (i.e., communities), and are positively assortative by the degree of connectivity (i.e., well-connected agents tend to be connected to other well-connected agents).¹⁴ Second, homophilous selection occurs because agents prefer alters that are more similar in behavior: when creating a tie, they choose the agent with the most similar behavior value. Third, we model social influence using a very simple mechanism: agents choose an influencer among their connections and adjust their behavior to be incrementally more similar.¹⁵ Finally, to explore the consequences of heterogeneity for SIENA’s estimates, we include two important sources of heterogeneity in our ABM: social reach (size of the radius to select peers) and behavioral change independent from social influence.

Model Description

The purpose of our model is to simulate a simplified data-generating process to examine the behavior of the SIENA actor-oriented model, specifically the estimation of influence and selection effects within a changing social network. In order to allow agents to experience influence and selection, agents need to track their behavior and their social connections over time. Therefore, the primary characteristics of agents are their behavior (measured using a continuous scale from 0 to 1) and social contacts (connections that form their out-degree). Secondary characteristics are position in space and social circle radius (to capture new potential alters or destroy existing ties).

Agent actions include movement, creation/destruction of ties, and interaction. Regarding movement, each agent has the same step size $d_{move}$ so that at a rate $r_{move}$ , they choose a new random location $d_{move}$ away from their current position, moving to it at a constant speed of 100 units per model second. This movement procedure is important to control for network saturation in our model.

In parallel, at a rate defined by $r_{network}$ , agents perform a network decision. Figure 1 shows the action chart of network decisions (creation or destruction of ties). For each such decision, agents either create a new tie (with probability $p_{create}$ ) or break a current tie. When agents choose to create a new tie, they choose either the most similar agent by behavior (with probability $p_{sel}$ ) or a random agent within their social circle at their current position ( $d_{circle}$ ). When choosing the most similar agents, they can select either from within their social circle (with probability $p_{inside}$ ) or from the whole agent population. This latter rule operationalizes two types of homophily distinguished by McPherson et al. (2001): baseline and inbreeding. While baseline homophily poses constraints to the local social world of individuals who incidentally will establish connections with people who do not match their behavior (shortage of alter candidates), inbreeding homophily describes the tendency to choose similar peers above the opportunity set (e.g., social circle), recruiting alters from wider surrounding of ego. When breaking a tie, in turn, agents define, with probability $p_{outside}$ , if the tie to break will be randomly selected from alters exclusively outside the social reach $d_{circle}$ or from any of their connections.

Figure 1.

Network change action chart.

At a rate defined by $r_{int}$ , agents become influenced by another agent. This is done by moving their behavior value closer to that of a random agent from their social connections. To determine their new behavior value, $b_{e}^{*}$ , agents use the simple influence update rule:

b_{e}^{*} \leftarrow b_{e} + α (b_{a} - b_{e}),

where b is behavior of the (e)go or the (a)lter, and $α$ is the behavior change rate.

Agents are also allowed to make behavior and network changes independently from influence/selection, so that dynamic behavior and networks can still observed when influence or selection are disabled during the experiments. Independent behavior change is controlled by a separate rate, $r_{i b}$ . At each independent behavior change choice, agents use a behavior tendency, $β$ , to decide how to internally change behavior. This tendency is a binary value, selecting between two different internal patterns (triangular distributions): one that increases behavior on average and one that decreases on average. Independent network change, on the other hand, is controlled by $p_{sel}$ , the probability that an agent will choose selection versus random choice within the social circle when updating their network. At the end, the resulting social network from this implementation is a dynamic and directional graph.

The initial population is of 200 agents, randomly placed in a 500 × 500 unit toroidal space. Model time is continuous, meaning there is no fundamental clock “tick.”¹⁶ The initial social network is created by connecting each agent to two other randomly chosen agents. Both the social circle and initial behavior are defined using uniform distributions. For each agent, the chance of an increasing versus decreasing behavior tendency is 50 percent. Table 1 describes all the model parameters with their initialization values.

Table 1.

ISA Parameters.

Parameter	Anylogic Name	Symbol	Value	Description
Network dynamics
Selection effects
Network change rate (per day)	p_networkRate	$r_{tie}$	0.4/day	Average number of network change decisions per model day
Social circle reach (unit distance)	p_socialCircle	$d_{circle}$	uni(10,80)	The radius from current location of the social circle
Create ties chance	p_probCreateTies	$p_{create}$	0.50	Given a network change decision, the probability of creating a new tie vs breaking a current one
Break ties by circle chance	p_probBreakTiesDistance	$p_{outside}$	0.60	Given a break tie decision, the probability that it will be exclusively from outside the social circle
Selection tendency	p_probSelection	$p_{sel}$	0.60	Given a make tie decision, the probability an agent uses selection versus making tie randomly
Selection inside circle	p_probSelectionInsideCircle	$p_{inside}$	0.70	Given a make tie decision and the use of selection, the probability of selecting from within the social circle versus the whole population
Behavior dynamics
Influence
Interaction rate	p_interactionRate	$r_{int}$	0.125/day	Average number of influence decisions per model day
Behavior change rate	p_behaviorChangeRate	$α$	0.20	Strength of influence effect
Independent change rate	p_behaviorRate	$r_{i b}$	0.05/day	Average number of independent behavior change decisions per model day
Behavior tendency	p_probBehaviorChangeTendency	$β$	rand(0.50)	Selects between two default behavior change patterns. Both are triangular distributions from −0.2 to 0.2, but the mode is either 0.05 or −0.05
Other
Distance to move	p_distanceToMove	$d_{move}$	5.0	Distance moved per move decision
Movement rate	p_movementRate	$r_{move}$	1/day	Probability per model-day of making a move decision
Behavior initialization	p_myBehavior	$b_{init}$	uni(0,1)	Initialization value of agent behavior
Population size	p_populationSize	$pop$	200	Number of agents
Duration	p_durationSimulation	T	300 days	Length of model run
Measurement window	p_measurementUpdate	T_m	30 days	Time between measurements

Note: For distributions in the value column, these are resampled for each agent. Initial social network is created by connecting each agent to two other randomly chosen agents. uni(x, y) = a sample from a uniform distribution between x and y; rand(x) = true with a probability of x; ISA = influence selection ABM; ABM = agent-based model.

To generate the output data, the model simulates a measurement, outputting location, social network, and agent behavior data for each agent every $T_{m} = 30$ days. The model runs for $T = 300$ days. We implement ISA using AnyLogic Personal Learning Edition 8.2.2.¹⁷

ISA Scenarios, Calibration, and Verification

We generate data for six ABM scenarios. Table 2 displays their differences. While ISA calculates a strength value for each influence and selection, we have not defined a mechanism to either scale the strength of influence or selection, nor have we defined a method to calculate a strength comparable to SIENA (i.e., there is no a direct translation of ISA’s parameters and SIENA coefficients).¹⁸ As a result, when comparing ground truth with SIENA predictions, we measure simply presence or absence. Therefore, we exclude scenarios where selection and social influence are at play simultaneously. Table 3 displays descriptive statistics of the outcomes of 100 replications of each scenario (details of the experimental design are discussed in Experimental Design section).

Table 2.

ISA Scenarios.

Scenario	Selection	Influence	Radius
B	No	No	uniform(10,80)
I	No	Yes	uniform(10,80)
S	Yes	No	uniform(10,80)
B + CR	No	No	50
I + CR	No	Yes	50
S + CR	Yes	No	50

Note: B = baseline; S = selection; I = influence; CR = constant radius; ISA = influence selection ABM; ABM = agent-based model.

Table 3.

Descriptives by ISA Scenario (100 Replicates).

ISA Scenario	Statistics
ISA Scenario	Density	Degree (Avg.)	Reciprocity	Transitivity	Moran’s I Behavior	Moran’s I Radius	Jaccard	Stability Behavior
B	.03 (.00)	10.28 (.60)	.40 (.02)	.42 (.01)	.00 (.03)	.00 (.03)	.43 (.01)	.57 (.02)
I	.03 (.00)	10.32 (.64)	.40 (.02)	.42 (.01)	.22 (.06)	.00 (.03)	.43 (.02)	.40 (.02)
S	.03 (.00)	10.65 (.63)	.34 (.02)	.30 (.01)	.22 (.03)	−.01 (.03)	.40 (.02)	.57 (.02)
B + CR	.03 (.00)	11.08 (.40)	.60 (.01)	.49 (.01)	−.01 (.03)	— (—)	.39 (.01)	.57 (.02)
I + CR	.03 (.00)	11.15 (.42)	.60 (.01)	.49 (.01)	.28 (.08)	— (—)	.39 (.01)	.40 (.02)
S + CR	.03 (.00)	12.17 (.43)	.50 (.01)	.34 (.01)	.19 (.03)	— (—)	.38 (.01)	.57 (.02)

Note: Standard deviation is given in parentheses. B = baseline; S = selection; I = influence; CR = constant radius; ISA = influence selection ABM; ABM = agent-based model.

The first scenario (B) corresponds to our baseline where both (homophilous) selection and influence are disabled, and there is heterogeneity in agents’ social circles. This was done to ensure the model does not artificially generate nonzero selection or influence (measured by behavioral-network autocorrelation). Table 3 provides (network) statistics that characterize each scenario (averaged over four waves). Scenario B shows a network density of 0.03, average degree of 10.28,¹⁹ and high levels of reciprocity and transitivity.

To measure for any unintended selection or influence pressures, we use one of the most popular standardized measures for network autocorrelation, Moran’s I:

I_{M} = \frac{n \sum_{i j} x_{i j} (z_{i} - \bar{z}) (z_{j} - \bar{z})}{(\sum_{i j} x_{i j}) (\sum_{i} {(z_{i} - \bar{z})}^{2})},

where n is the number of agents, $z$ is the variable of interest, and $x$ is an element of a matrix of network weights. This coefficient is based on cross products of behavioral scores of relational partners. Values close to zero indicate that relational partners are not more similar than one would expect under random pairing, while values close to one indicate a very strong network autocorrelation. As expected, for scenario B, the Moran’s I coefficient is zero both for behavior and the size of social circles (radius). That is, there is no emergent property in our ABM that generates behavior or radius autocorrelation other than the selection and influence mechanism.

In order for SIENA estimates to be valid, SIENA’ authors indicate that the network must have a minimum amount of network stability over time. SIENA’s authors measure this using Jaccard index, that is, Jaccard distances between time-successive networks:

J = \frac{N_{11}}{N_{01} + N_{10} + N_{11}},

where $N_{h k}$ is the number of tie variables with value h in one wave and value k in the next wave (Ripley et al. 2017). All ISA scenarios in Table 2 satisfy the levels of network stability recommended by SIENA’s authors ( $J \geq 0.30$ ). Furthermore, our behavior stability measures look similar to those we find in empirical applications of SIENA (Steglich et al. 2010; Veenstra and Steglich 2012).

The ISA scenarios I and S enable influence or selection, respectively. The influence mechanism does not directly affect network dynamics, so network statistics between scenarios B and I are almost identical, as expected. This is not the case for scenario S, where ties are being defined through a selection process while behavior remains unaffected. As expected, network statistics are different from scenario B, slightly increasing the degree and reducing reciprocity, transitivity, and the Jaccard index. In scenario I, Moran’s coefficient for behavior is .22, and behavior stability is lower than scenario B because behavior is changing in a systematic way and not only through a random process (behavior tendency). For scenario S, Moran’s coefficient is .22, the same as scenario I but less noisy (i.e., smaller standard deviation). Importantly, both influence and selection mechanisms independently increase network autocorrelation.

In order to examine the consequences of agent heterogeneity, scenarios B + CR, I + CR, and S + CR (CR stands for constant radius) set all agents’ social circles to the same value (50). This changes the network structure, increasing degree, reciprocity, and transitivity. In other words, agents are more connected, increasing Moran’s coefficient for scenario I + CR (there are more chances of being influenced) and decreasing that coefficient in scenario S + CR (relatively less clustering of behavior).

How Different Is ISA from SIENA?

Along with the four SIENA’s assumptions discussed above, we examine how different SIENA and ISA are. First, ISA replicates some SIENA assumptions such as (1) a continuous time framework for changes in the network and behavior and (2) the speed of those changes coming from exponential distributions. However, our implementation of change is different. We set several and different rates for agents’ actions such as movement, change of ties, interaction between agents, and modification of behavior. SIENA, on the other hand, estimates one rate for network change and another for behavior modifications. In addition, while for SIENA keeping ties or no change is a valid choice in a discrete choice model, we do not define stability explicitly, but by the time between agents’ actions (e.g., moving/not moving, changing a tie/not changing a tie).

Second, ISA does not satisfy the assumptions of (3) agent heterogeneity or (4) awareness of the network topology. Agents in ISA are heterogeneous along dimensions that impact their network decisions (to create, keep, or destroy ties), and they do not use or have awareness of the complete network topology. They only use information from their agents in their social circles and social networks to create and destroy ties. Social circles vary by agent and their differences can be thought as heterogeneous agents’ propensities to send outgoing ties: given a particular neighborhood those agents with a bigger social circle are more likely to establish connections.

We think violations to assumptions (3) and (4) are not eccentric and it is worth examining their consequences. People are likely to have individual and contextual (unobserved) differences when defining their social networks. In addition, since our sensing and intelligence are limited, it seems reasonable to explore how sensitive SIENA is to alternative micromechanisms of network generation that do not assume perfect information and extraordinary computing abilities (Epstein 2014).

Finally, we should note that ISA generates perfect data, that is, no missing records, no compositional change or measurement error, and does not include any latent homophily process that might confound estimates of selection and influence.

SIENA Specification

Table 4 outlines the effects we include in the SIENA specifications. Each effect is placed within a specification grouping (column 2) to facilitate comparisons between experimental scenarios, as described below. The structural effects include network dynamics at the individual, dyad, and triad (group) level. We chose these effects through an iterative process in which we assess the goodness of fit (GOF) of our models and review the most common effects used in empirical studies.²⁰

Table 4.

Siena Effects by Specification.

SIENA Effect	Specification	Description
Network dynamics
Structural effects
Out-degree (density)	All	Actor i extending ties to alter j
Reciprocity	N	Actor i reciprocating ties to alter j
Transitive triplets	N	Actor i extending ties to alter j to whom he is indirectly tied
Transitive reciprocated triplets	N	Actor j reciprocating ties to ego i to whom he is directly and indirectly tied (via h)
Three cycles	N	Actor i extending ties to alter j to whom he is indirectly tied (via h)
Number at distance 2 twice	N	Tendency of actors to be tied indirectly through at two intermediaries (via h and k)
Geometrically weighted transitive triplets	N	Tendency of actors to be tied indirectly through at least one intermediary (via h) where the influence of extra intermediaries (k) decreases
In-degree-network (square)	N	Actors with many incoming ties attract more incoming ties
Out-degree-network (square)	N	Actors with many outgoing ties attract more incoming ties
Out-degree-activity (square)	N	Actors with many outgoing ties extend more outgoing ties
Distance and radius
Distance (log) (dyad varying covariate)	D	A tie between two actors is less likely when the dyadic distance is larger
Radius ego (log)	R	Actor i with higher values on radius (log) extends more outgoing ties
Selection effects
Behavior ego	S	Actor i with higher values on behavior extends more outgoing ties
Behavior alter	S	Actor i with higher values on behavior attracts more incoming ties
Behavior similarity	S	Actor i extends ties to alter j who has similar values on a covariate
Behavioral dynamics
Control effects
Linear shape	All	Tendency of actors to change in behavior
Quadratic shape	All	Tendency of actors to change in behavior
Behavior tendency attribute	All	Actor with a higher behavior tendency value have a higher average value in the behavior
Influence effects
Average alter	I	Actors whose alters have a higher average value of the behavior, have themselves a stronger tendency toward high values on the behavior

Note: The density effect is allowed to vary by wave in all the specifications. N = network structural effects; S = selection; I = influence; D = distance; R = radius; SIENA = Stochastic Actor-Oriented Model.

Below that, we include two effects directly related to ISA’s generative mechanism: distance and ego radius. The distance effect expresses the extent to which a tie between two actors is less likely when the dyadic distance is larger. This coefficient in our analyses is, as expected, negative and statistically significant. The ego radius effect expresses how actors with larger social circles extend more outgoing ties. The coefficient of this effect is, as expected, positive and significant. We decided to include the radius information only through the ego agent. Effects such a alter radius (those with higher radius have more incoming ties) and radius similarity (actors select peers based on radius) do not seem reasonable given the generative mechanisms of our ABM.²¹

Following SIENA authors’ recommendations, we use three selection parameters: ego behavior, alter behavior, and behavior similarity. The ego and alter effects are not significant in our analysis, suggesting no main effect of behavior on either network nominating activity or popularity. Behavior similarity (what we call selection), in turn, indicates descriptive evidence for homophilous selection based on behavior.

The behavioral dynamic part of the model is simpler. Two shape parameters are used to express basic distributional features of the behavior variable and as control variables. The linear shape parameter plays the role of an intercept, whereas the quadratic shape parameter controls for underdispersion (regression to the mean, when the parameter is negative) or overdispersion (polarization, when the parameter is positive). Together, the two shape effects can be interpreted as expressing the relative prominence of the specific values on their distribution. If the shape contribution to the objective function is U shaped, for instance, it suggests that changes in behavior are drawn to the extremes of the range, decreasing behavior of actors with low behavior, and increasing those with high behavior. SIENA’s authors recommend the quadratic shape effect when the behavior variable contains three or more possible values (Veenstra and Steglich 2012). We also adjust for the independent behavior change choice, a binary value (0/1) that defines different triangular distributions to modify behavior. Finally, we use the average alter effect to parameterize social influence. This coefficient suggests that actors whose alters have a higher average value of behavior will, themselves, also have a tendency toward higher behavior.²²

Table 5 summarizes the seven SIENA specifications we use, composed of the effect groupings indicated above and in column 2 of Table 4. We include selection (S) and influence (I) effects separately and together to explore how these parameters affected each other. We do similarly with radius (R) and distance (D) and add a model where the structural network part included only density (out-degree). Thus, we provide a relatively diverse set of specifications.

Table 5.

Siena Specifications.

SIENA Specification	Structural Network Effects (N)	Selection (S)	Influence (I)	Radius (R)	Distance (D)
N + S	Various	Yes	No	No	No
N + I	Various	No	Yes	No	No
N + S + I	Various	Yes	Yes	No	No
N + S + I + R	Various	Yes	Yes	Yes	No
N + S + I + D	Various	Yes	Yes	No	Yes
S + I + R + D	Density only	Yes	Yes	Yes	Yes
N + S + I + R + D	Various	Yes	Yes	Yes	Yes

Note: SIENA = Stochastic Actor-Oriented Model; N = network structural effects; S = selection; I = influence; D = distance; R = radius.

To fit each of these SIENA specifications, we took four measurements for each of the ABM scenarios (corresponding to model day 180, 210, 240, and 270) in order to ensure enough statistical power (Stadtfeld et al. 2017).²³ Our behavior variable is continuous, but SIENA currently only allows ordinal or categorical values when modeling behavior as a dependent variable. We converted our continuous behavior variable into one of 10 discrete levels (empirical applications usually use dependent variables with 4–10 categories).

Experimental Design

Table 6 shows all relevant combinations of the six ISA scenarios and the seven SIENA specifications, resulting in 18 experimental scenarios. As mentioned above, each of the six ISA scenarios is run 100 times to allow the Moran’s I network autocorrelation to stabilize.²⁴ Each ISA run outputs the social and behavioral landscape at four measurement times, which are then fed into the respective SIENA specification. This results in fitting SIENA 1,800 times, 100 for each of the 18 experimental scenarios. Those models that reach overall convergence are kept and included in Table 6.²⁵

Table 6.

Coverage Rate, Bias, and SIENA Estimates by Scenario and Specification (100 Replicates).

ISA Scenario	Siena Specification	Estimates (SIENA)
		Selection				Influence
		Coverage	Bias	$P r (β_{s} > 0)$	RE Estimate	Coverage	Bias	$P r (β_{i} > 0)$	RE Estimate
B	N + S + I	.85	0.16	—	.02 (.01)	0.93	.09	—	.01 (.01)
I	N + S	.46	1.75	—	.65 (.03)	—	—	—	—
I	N + S + I	.44	1.73	—	.68 (.04)	—	—	1.00	.49 (.01)
I	N + S + I + D	.64	1.31	—	.43 (.03)	—	—	1.00	.43 (.01)
I	N + S + I + R	.69	0.93	—	.39 (.04)	—	—	1.00	.53 (.01)
I	S + I + R + D	.79	0.25	—	.07 (.03)	—	—	1.00	.44 (.01)
I	N + S + I + R + D	.84	0.23	—	.07 (.04)	—	—	1.00	.46 (.01)
S	N + I	—	—	—	—	0.99	−.25	—	−.01 (.01)
S	N + S + I	—	—	1.00	.73 (.01)	0.98	−.25	—	−.01 (.01)
S	N + S + I + D	—	—	1.00	.94 (.01)	1.00	−.25	—	−.01 (.01)
S	N + S + I + R	—	—	1.00	.71 (.01)	0.98	−.25	—	−.01 (.01)
S	S + I + R + D	—	—	1.00	.91 (.01)	0.98	−.26	—	−.01 (.01)
S	N + S + I + R + D	—	—	1.00	.93 (.01)	0.99	−.25	—	−.01 (.01)
B + CR	N + S + I	.95	0.09	—	.01 (.01)	0.97	−.12	—	.00 (.01)
I + CR	N + S + I	.89	0.57	—	.19 (.04)	—	—	1.00	.60 (.01)
I + CR	N + S + I + D	.92	0.06	—	−.01 (.03)	—	—	1.00	.57 (.01)
S + CR	N + S + I	—	—	1.00	.65 (.01)	0.91	−.20	—	−.01 (.01)
S + CR	N + S + I + D	—	—	1.00	.80 (.01)	0.91	−.21	—	−.01 (.01)

Note: ISA scenario: B = baseline; I = influence; S = selection; CR = constant radius. Siena specification: N = network structural effects; S = selection; I = influence; D = distance; R = radius. Performance statistics: $β_{s}$ = behavior ego–alter interaction; $β_{i}$ = behavior average alter. $P r (β > 0)$ = probability confidence interval includes only positive values; RE estimate = average estimate of the true effect using a random-effects model; SIENA = Stochastic Actor-Oriented Model; ISA = influence selection ABM; ABM = agent-based model.

To evaluate SIENA's performance, we use two measures: standardized bias and coverage rate. Standardized bias is the raw bias (average parameter estimates across replications minus the true parameter value) divided by the standard deviation of the estimates across all replicates. For instance, a value of −0.5 means that the estimate on average falls one-half of a standard deviation below the parameter. Collins, Schafer, and Kam (2001) provide the rule of thumb that absolute values of 0.4 or higher are practically significant. Coverage rates are the proportion of replications whose 95 percent confidence interval includes the true parameter estimate. The actual coverage should be approximately equal to or greater than the nominal coverage rate (95 percent). According to (Collins et al. 2001), problematic coverage rates are below 90 percent. We also estimate aggregate coefficients for selection and influence effects by using meta-analysis (An 2015; Viechtbauer 2010), so that we can compare the magnitude of effects across SIENA specifications and estimate the probability of observing a positive effect $P r (β > 0)$ when it is present in our ABM. This probability might be interpreted as a measure of statistical power of selection/influence estimates, but we should note that it does not tell us anything about the bias of coefficients.

To assess the GOF of our models, we employ distributions of auxiliary statistics such as out-degree, in-degree, geodesic distance, triad census, and behavior and explore the differences (Mahalonobis distance) between the observed distributions (summed across the four waves of data) and the simulated distributions from SIENA (summed across 1,000 random networks). Most of our specifications reproduce well the degree and behavior distributions. GOF for geodesic distances and triad census, in contrast, is weak. The connection between bias and GOF, though, is not straightforward. While some of our best SIENA estimates (e.g., S + I + R + D) have poor GOF with statistics such as geodesic distance and triad census, specifications with a similar (or even better) GOF appear to have more biased estimates (e.g., N + S + I). Section GOF in the Supplemental Material (which can be found at http://smr.sagepub.com/supplemental/) provides details on the GOF of our models, although more research would be needed to explore the connection between bias and GOF in SIENA.

Results

Table 6 displays coverage rates, standardized biases by scenario and specification, and pooled SIENA estimates. We begin by examining ISA scenario B where selection and influence are disabled. Using the SIENA specification N + S + I (adjusting for structural network effects, selection, and influence), we find that selection bias is expectedly below the 0.40 threshold (0.16), but coverage is below 0.90 (0.85). Furthermore, the selection coefficient, although very close to zero, is statistically significant using standard hypothesis testing procedures. The influence estimate, on the other hand, shows no bias, good coverage, and RE estimate consistent with no influence effects. In other words, while N + S + I correctly reports no influence, the selection estimate is a bit more uncertain.

ISA scenario I enables the influence mechanism. This time, we should expect positive estimates of influence and close-to-zero estimates of selection. As discussed above, we use six SIENA specifications. The first one, N + S, adjusts only for structural network effects and homophilous selection. As expected, this specification shows high bias in the selection estimate as that coefficient does not account for influence, the actual mechanism generating network autocorrelation (as shown in Table 3). We then consider the specification N + S + I that adds an influence effect. In this case, we might expect a more accurate estimation of selection since SIENA is now allowed to adjust for influence. However, allowing SIENA to account for influence (N + S + I) does not improve the selection estimate. Bias remains practically the same, and the pooled selection estimate becomes slightly higher, while the influence estimate is positive and statistically significant.²⁶

We progressively continue adding information on the mechanisms of the network generation to the SIENA specification. N + S + I + D, for instance, adds a distance between dyads effect. Under this specification, bias is reduced (from 1.73 to 1.31), coverage increases (from 0.44 to 0.64), and the pooled selection coefficient decreases from 0.68 to 0.43. Similarly, N + S + I + R, incorporates a ego radius effect and reduces bias even more (from 1.73 to 0.93). Although selection coefficients remain positive and significant when we expect zero effect, we observe that including information of the generative network formation improves selection estimates.

Then, we consider including both ego radius and distance together using two different network specifications. First, we only adjust for radius and distance and remove all the structural network effects in SIENA except for density (S + I + R + D). In this case, bias decreases to 0.25 (below the 0.40 threshold), coverage remains below 0.90, and the pooled selection estimate is positive and statistically significant. Second, we add the structural network effects back (N + S + I + R + D) and adjusts for radius and distance simultaneously. Here, the general results are the same: the selection bias is below the threshold (0.23), coverage is slightly under 0.90, and the pooled estimate is positive and statistically significant under conventional levels ( $z = 2.04$ ). These two scenarios still falsely indicate a selection process, but its magnitude is considerably reduced relative to the other scenarios in this group. Overall, no SIENA specification was able to firmly reject the existence of a selection process. Influence estimation, on the other hand, despite some variability across specifications (0.43–0.53), is stable and positive (as expected).

All of these experiments were estimated on ground-truth data generated from ISA every 30 days. As discussed in Section S4, we conducted further comparisons varying instead this update period, using the N + S + I SIENA specification throughout. Choosing the shortest update period of five days resulted in coverage, bias, and pooled selection estimates better than all SIENA specifications other than S + I + R + D and N + S + I + R + D. This speaks to the importance of checking the sensitivity of estimates on the measurement updates period.

ISA scenario S enables selection instead of influence. As can be seen in Table 6, standardized biases for influence are all around −0.25, regardless of the SIENA specification, coverage rates never go below 0.98, and pooled coefficients are very close to zero (−0.01). In all cases, influence is correctly measured. Moreover, influence estimates do not change when including/excluding selection effects, similar to what we observed in ISA scenario I.

We finally examine the consequences of heterogeneity. The scenario designation CR removes radius heterogeneity by making social circles constant (i.e., all agents have the same radius). SIENA estimation using the baseline scenario (B + CR) shows biases and coverage rates consistent with no selection and influence effects, better than the baseline model with heterogeneity. Removing heterogeneity also improves selection bias when influence is enabled (I + CR), although selection coefficients are still positive and statistically significant under the N + S + I SIENA specification (the pooled selection effect is 0.19 with an standard error of 0.04). Adjusting for distance (N + S + I + D) results in a sufficiently high coverage (0.92), a sufficiently low selection bias (0.06) and a statistically insignificant selection estimate. With a homogeneous radius and selection enabled (ISA scenario S + CR), influence estimates are again correct and stable: bias ranges around −0.20, the coverage rates are slightly over 0.90, and the pooled estimates are close to zero (−0.01). Evidently, heterogeneity in the network generation process has strong consequences for the estimation of selection using SIENA, even when accounting for that source of heterogeneity (e.g., adjusting for variables such as ego radius).

Conclusion and Discussion

The goal of this article is to demonstrate how ABM-generated simulated data can provide novel and valuable insight into the testing of new methodologies. This was demonstrated with an ABM built to generate test data for the SIENA analysis tool. Specifically, we explore how sensitive SIENA’s selection and influence estimates are to misspecification and examine the conditions under which SIENA provides correct estimates.

The primary result from this work is that, while influence estimates were consistently robust to variation in SIENA specification and across all ISA scenarios, selection estimates were highly sensitive. As we increased the complexity of the SIENA specification, the selection bias did improve, however in order to obtain accurate estimates of non-selection, we needed to both incorporate knowledge of the data-generating process itself (by adding social circles and social distance to the SIENA specification) and reduce the heterogeneity in the ISA population (by enforcing a constant social radius). This is an important conclusion for two reasons. Firstly, SIENA claims to be able to obtain accurate selection estimates conditional on a model specification that is a good approximation to the data-generating process. Knowledge on the data-generating process is typically unavailable and including parameters that has the potential to correctly describe the network dynamic seems not to be enough to recover the true parameters. Secondly, social networks are marked by a great deal of heterogeneity. This latter result was expected given that SIENA assumes a homogeneous objective function when defining ties and behavior evolution. This method, however, demonstrates a way to quantify the degree of sensitivity to this assumption, allowing researchers to better understand when SIENA estimates might be overly biased.

In describing our hypothesis as to why selection is often overestimated while influence is robust, it is helpful to refer to Figure 2. It displays different dyad combinations and how they can change over time. This figure provides a very intuitive understanding of how SIENA proposes to separate influence from selection. If we measure a dyad in the state on the right in one measurement and the bottom in the next, it is thought to be easier to follow path {d} than path {c, b, a}; therefore, selection should be easily distinguishable from the confounding effects. However, with random behavior change as in our ISA scenarios, step {c} (from right to top) would be very likely since random behavior changes are more likely to make any nonconnected pair of similar agents less similar. Once in the top state (two dissimilar disconnected actors), step {b} is more likely for the same reason: randomly moving closer in behavior is less likely than randomly connecting. Finally, if influence is enabled in the ABM, step {a} from left to bottom is expected. This shows how selection could be observed when no selection is actually present.

Figure 2.

Configurations of similarity and friendship in a dyad. Start with the first left connection and follow through path (b, c, and d). This selection-only path is more complex than the influence path of simply taking (a). Hence, measurement of influence when it is not present is unlikely. Source: Veenstra and Steglich (2012).

As to why influence measures are robust, we start the same type of argument at the left configuration of Figure 2 (two dissimilar connected actors). Being able to accurately measure influence is akin to distinguishing between path {a} and path {b, c, d}. For similar reasons as above, step {c} (top to right) is relatively unlikely due to ISA’s random behavior change. Therefore, path {b, c, d}, which is a confounding path for measuring influence, is much less common than {a}, which is the path measuring influence, whereas path {c, b, a}, confounding for selection, could be nearly comparable to path {d}, true selection. This would result in influence estimates being much more robust than selection estimates.

A simple test to check this interpretation is to add a social norms mechanism to our model, such that step {c} becomes more equally likely from either direction, and see if the influence stability remains. In the Section S3, we provide evidence that supports this interpretation as influence becomes positive when adding social norms. These results also provide interesting insights regarding how independent changes in behavior can affect the robustness of selection and influence estimates.

Although SIENA allows feedback between selection and influence processes, the analysis of the data generated by our ABM suggests that SIENA measures selection and influence almost independently. In ISA scenario I, for instance, the estimation of selection remains unchanged when we add the influence effect. This might be also explained using Figure 2. As we discuss above, given two discrete data points (i.e., waves), inferring (unobserved) selection and influence trajectories at the same time seems very unlikely. In other words, simulated microchanges tend to prioritize either selection or influence inferences when imputing unobserved changes given two data points. If covariance between selection and influence inferences is close to zero, adjusting for each other will not have any consequences for the estimates. To fully validate this hypothesis, we would need more research, especially since selection is not well captured in many of our experimental scenarios, for reasons discussed above.

In sum, we recommend caution when using SIENA estimates. Our simulated ground-truth uses data generated from a simple ABM where no measurement error, missing or censored data, panel attrition or compositional change is present. Also, no latent homophily affecting selection or influence estimates was included. Analysis with real data might be even more challenging, and additional sources of bias might be overlaid on top of the conditions explored in this article. If we suspect that the problem we are studying is context dependent and heterogeneous, and we only have partial knowledge or no good theory on the key mechanisms that generates the network dynamic, the relative importance of selection and influence estimates provided by SIENA should be interpreted with extreme caution.

It could be argued that our ABM scenarios are too simple to be a valuable comparison against SIENA estimates. For example, social networks likely are very rarely, if ever, devoid of selection effects. How valuable, then, are discrepancies between SIENA estimates and ISA ground truth in light of this oversimplification? In response, we would point out that this research is not meant as the final word on SIENA’s limitations; certainly more work is required, work that includes a strength measure for each effect, thereby allowing a wider array of experimental scenarios for the ABM. As is typical when conducting “benchmarking” tests, it is wise to begin at the extremes of simplicity to allow for clear thinking of the mechanisms. This argument to simplicity does not invalidate the conclusion that SIENA appears to treat selection and influence asymmetrically.

Relatedly, while the SIENA estimates in this article are specific to how ISA was designed, it should be noted that the conclusions are potentially broadly applicable. The micromechanisms in ISA are certainly an oversimplification of real-world selection and influence processes. However, they were chosen to generate specific key population-level phenomena observed in the real world (such as social clustering). While the Generativist Manifesto (Epstein 2006) says that generating a macrobehavior is not sufficient to explaining the real-world mechanisms, it is nonetheless a necessary element. Since this plausible, though simple, mechanism demonstrates surprising sensitivity in SIENA estimates, further exploration under different generating mechanisms is warranted.

There are many future research paths suggested by this work. We could include more complex and realistic mechanisms to better understand robust or sensitive parameter regimes of SIENA. For instance, this work could be extended to study the impact of time, to observe when or how often we might want to take measurements, and how long SIENA’s predictions might remain valid. It would be also useful to examine how sensitive selection and influence SIENA estimates are to latent homophily. And finally, this approach would be very powerful when studying the effects of noise in the survey data. Independent changes in behavior affects influence estimates opening the question on how noise (e.g., measurement error, stochasticity) might bias downward social influence estimation when using SIENA. In this scenario, it would be of interest, for example, to examine whether it is more important to focus on accuracy in the social network or in the behavioral measure.

In conclusion, in order to be able to build confidence in our analysis techniques, especially when applied to complex and dynamic systems, it is important to be able to validate them against a diverse set of mechanisms that might plausibly represent real-world situations. With the use of modeling techniques such as ABM, a wide range of previously difficult problems becomes much more accessible. We have conducted a first step in this direction, but much more is to be done to assess complex statistical models like SIENA.

Supplemental Material

supplemental_material_resubmission - Agent-Based Models for Assessing Complex Statistical Models: An Example Evaluating Selection and Social Influence Estimates from SIENA

supplemental_material_resubmission for Agent-Based Models for Assessing Complex Statistical Models: An Example Evaluating Selection and Social Influence Estimates from SIENA by Sebastian Daza, and L. Kurt Kreuger in Sociological Methods & Research

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The University of Wisconsin–Madison researchers are supported by core grants to the Center for Demography and Ecology, University of Wisconsin (R24 HD047873) and to the Center for Demography of Health and Aging, University of Wisconsin (P30 AG017266).

Supplemental Material

Supplemental material for this article is available online.

Notes

References

W. E

. 2011. “Models and Methods to Identify Peer Effects.” Pp. 514–28 in The SAGE Handbook of Social Network Analysis, edited by Scott

Carrington

. Los Angeles: Sage.

W. E.

2015. “Multilevel Meta Network Analysis with Application to Studying Network Dynamics of Network Interventions.” Social Networks 43:48–56.

Bishop

2009. The Big Sort: Why the Clustering of Like-minded America Is Tearing Us Apart. 1st Ed . Boston, MA: Mariner Books.

Bruch

Atwell

. 2013. “Agent-based Models in Empirical Social Research.” Sociological Methods & Research 44:1–36.

Chattoe-Brown

2013. “Why Sociology Should Use Agent Based Modelling.” Sociological Research Online 18:3.

Christakis

N. A.

Fowler

J. H.

. 2011. Connected: The Surprising Power of Our Social Networks and How They Shape Our Lives—How Your Friends’ Friends’ Friends Affect Everything You Feel, Think, and Do. Reprint Ed. New York: Back Bay Books.

Collins

L. M.

Schafer

J. L.

Kam

C. M.

. 2001. “Comparison of Inclusive and Restrictive Strategies in Modern Missing Data Procedures.” Psychological Methods 6:330–51.

Conte

Paolucci

. 2014. “On Agent-based Modeling and Computational Social Science.” Frontiers in Psychology 5:668.

Dijkstra

J. K.

Berger

Lindenberg

. 2011. “Do Physical and Relational Aggression Explain Adolescents’ Friendship Selection? The Competing Roles of Network Characteristics, Gender, and Social Status.” Aggressive Behavior 37:417–29.

10.

Edmonds

Meyer

. 2013. Simulating Social Complexity. Berlin, Germany: Springer.

11.

Elliot

Fairweather

Olsen

Pampaka

. 2016. A Dictionary of Social Research Methods. 1st Ed. Oxford, England: Oxford University Press.

12.

Elsenbroich

Gilbert

. 2014. Modelling Norms. Dordrecht, the Netherlands: Springer.

13.

Epstein

J. M.

2006. “Generative Social Science: Studies in Agent-based Computational Modeling.” Generative Social Sciences 18:356–73.

14.

Epstein

J. M.

2014. Agent_Zero: Toward Neurocognitive Foundations for Generative Social Science. Princeton, NJ: Princeton University Press.

15.

Ghorbani

Dijkema

Schrauwen

. 2015. “Structuring Qualitative Data for Agent-based Modelling.” Journal of Artificial Societies and Social Simulation 18:2.

16.

Gibert

Hamill

. 2016. Agent Based Modelling in Economics. 1st Ed. West Sussex, England: Wiley.

17.

Haas

S. A.

Schaefer

D. R.

. 2014. “With a Little Help from My Friends? Asymmetrical Social Influence on Adolescent Smoking Initiation and Cessation.” Journal of Health and Social Behavior 55:126–43.

18.

Hamill

Gilbert

G. N.

. 2009. “Social Circles: A Simple Structure for Agent-based Social Network Models.” Journal of Artificial Societies and Social Simulation 12:3.

19.

Hamill

Gilbert

G. N.

. 2016. Agent-based Modelling in Economics. Hoboken, NJ: John Wiley.

20.

Hoffer

L. D.

2013. “Unreal Models of Real Behavior: The Agent-based Modeling Experience.” Practicing Anthropology 35:19–23.

21.

Huang

G. C.

Soto

Fujimoto

Valente

T. W.

. 2014. “The Interplay of Friendship Networks and Social Networking Sites: Longitudinal Analysis of Selection and Influence Effects on Adolescent Smoking and Alcohol Use.” American Journal of Public Health 104:e51–59.

22.

Huisman

Snijders

T. A. B.

. 2003. “Statistical Analysis of Longitudinal Network Data with Changing Composition.” Sociological Methods & Research 32:253–87.

23.

Huisman

Steglich

. 2008. “Treatment of Non-response in Longitudinal Network Studies.” Social Networks 30:297–308.

24.

Huitsing

Snijders

T. A. B.

Van Duijn

M. A. J.

Veenstra

. 2014. “Victims, Bullies, and Their Defenders: A Longitudinal Study of the Coevolution of Positive and Negative Networks.” Development and Psychopathology 26:645–59.

25.

Kéry

Royle

J. A.

. 2016. “Introduction to Data Simulation.” Pp. 123–43 in Applied Hierarchical Modeling in Ecology, 1st ed., Vol. 1, Chap. 4. Academic Press.

26.

Knecht

A. B.

Burk

W. J.

Weesie

Steglich

. 2011. “Friendship and Alcohol Use in Early Adolescence: A Multilevel Social Network Approach.” Journal of Research on Adolescence 21:475–87.

27.

Knecht

A. B.

Snijders

T. A.

Baerveldt

Steglich

C. E.

Raub

. 2010. “Friendship and Delinquency: Selection and Influence Processes in Early Adolescence.” Social Development 19:494–514.

28.

Leifeld

Cranmer

S. J.

. 2015. “A Theoretical and Empirical Comparison of the Temporal Exponential Random Graph Model and the Stochastic Actor-oriented Model.” arXiv preprint arXiv:1506.06696 .

29.

Lomi

Snijders

T. A. B.

Steglich

C. E. G.

TorlÃ

V. J.

. 2011. “Why Are Some More Peer than Others? Evidence from a Longitudinal Study of Social Networks and Individual Academic Performance.” Social Science Research 40:1506–20.

30.

Macy

M. W.

Willer

. 2002. “From Factors to Actors: Computational Sociology and Agent-based Modeling.” Annual Review of Sociology 28:143166.

31.

Manzo

2014. Analytical Sociology: Actions and Networks. 1st ed. New York: Wiley.

32.

Matthews

R. B.

Gilbert

N. G.

Roach

Polhill

J. G.

Gotts

N. M.

. 2007. “Agent-based Land-use Models: A Review of Applications.” Landscape Ecology 22:1447–59.

33.

McPherson

Smith-Lovin

Cook

J. M.

. 2001. “Birds of a Feather: Homophily in Social Networks.” Annual Review of Sociology 27:415–44.

34.

Mercken

Snijders

T. A. B.

Steglich

Vartiainen

de Vries

. 2010. “Dynamics of Adolescent Friendship Networks and Smoking Behavior.” Social Networks 32:72–81.

35.

Mercken

Steglich

Knibbe

de Vries

. 2012. “Dynamics of Friendship Networks and Alcohol Use in Early and Mid-adolescence.” Journal of Studies on Alcohol and Drugs 73:99–110.

36.

Olden

J. D.

Joy

M. K.

Death

R. G.

. 2004. “An Accurate Comparison of Methods for Quantifying Variable Importance in Artificial Neural Networks Using Simulated Data.” Ecological Modelling 178:389–97.

37.

Pearson

Sieglich

Snijders

. 2006. “Homophily and Assimilation among Sport-active Adolescent Substance Users.” Connections 27:47–63.

38.

Rambaran

J. A.

Hopmeyer

Schwartz

Steglich

Badaly

Veenstra

. 2016. “Academic Functioning and Peer Influences: A Short-term Longitudinal Study of Network-behavior Dynamics in Middle Adolescence.” Child Development n/a–n/a.

39.

Ripley

R. M.

Snijders

T. A.

Preciado

. 2017. Manual for Rsiena. Oxford, England: Department of Statistics, Nuffield College, University of Oxford.

40.

Rulison

K. L.

Gest

S. D.

Loken

. 2013. “Dynamic Social Networks and Physical Aggression: The Moderating Role of Gender and Social Status among Peers.” Journal of Research on Adolescence 23:437–49.

41.

Schaefer

D. R.

Kornienko

Fox

A. M.

. 2011. “Misery Does Not Love Company: Network Selection Mechanisms and Depression Homophily.” American Sociological Review 76:764–85.

42.

Shalizi

C. R.

Thomas

A. C.

. 2011. “Homophily and Contagion Are Generically Confounded in Observational Social Network Studies.” Sociological Methods & Research 40:211–39.

43.

Shoham

D. A.

Tong

Lamberson

P. J.

Auchincloss

A. H.

Zhang

Dugas

Kaufman

J. S.

Cooper

R. S.

Luke

. 2012. “An Actor-based Model of Social Network Influence on Adolescent Body Size, Screen Time, and Playing Sports.” PLos One 7:e39795.

44.

Simmel

1902. “The Number of Members as Determining the Sociological Form of the Group. i.” The American Journal of Sociology 8:1–46.

45.

Simpkins

S. D.

Schaefer

D. R.

Price

C. D.

Vest

A. E.

. 2013. “Adolescent Friendships, BMI, and Physical Activity: Untangling Selection and Influence through Longitudinal Social Network Analysis.” Journal of Research on Adolescence: The Official Journal of the Society for Research on Adolescence 23:537–49.

46.

Snijders

T. A. B.

2001. “The Statistical Evaluation of Social Network Dynamics.” Sociological Methodology 31:361–95.

47.

Snijders

T. A. B.

Steglich

C. E. G.

. 2015. “Representing Micro-macro Linkages by Actor-based Dynamic Network Models.” Sociological Methods & Research 44:222–71.

48.

Snijders

T. A. B.

Steglich

C. E. G.

Schweinberger

. 2007. “Modeling the Co-evolution of Networks and Behavior.” in Longitudinal Models in the Behavioral and Related Sciences, edited by van Montfort

Oud

Satorra

. Lawrence Erlbaum.

49.

Squazzoni

2012. Agent-based Computational Sociology. Chichester, England: Wiley.

50.

Stadtfeld

Snijders

T. A. B.

Steglich

van Duijn

M. A. J.

. 2017. “Statistical Power in Longitudinal Network Studies.” arXiv:1701.05177 [stat].

51.

Steglich

Sinclair

Holliday

Moore

. 2012. “Actor-based Analysis of Peer Influence in a Stop Smoking in Schools Trial (Assist).” Social Networks 34:359–69.

52.

Steglich

Snijders

T. A. B.

Pearson

. 2010. “Dynamic Networks and Behavior: Separating Selection from Influence.” Sociological Methodology 40:329–93.

53.

Van Zalk

M. H. W.

Kerr

Branje

S. J. T.

Stattin

Meeus

W. H. J.

. 2010. “It Takes Three: Selection, Influence, and De-selection Processes of Depression in Adolescent Friendship Networks.” Developmental Psychology 46:927–38.

54.

VanderWeele

T. J.

2011. “Sensitivity Analysis for Contagion Effects in Social Networks.” Sociological Methods & Research 40:240–55.

55.

Veenstra

Steglich

. 2012. “Actor-based Model for Network and Behavior Dynamics: A Tool to Examine Selection and Influence Processes.” Handbook of Developmental Research Methods 598–618.

56.

Viechtbauer

2010. “Conducting Meta-analyses in r with the Metafor Package.” Journal of Statistical Software 36:1–48.

57.

Weerman

F. M.

2011. “Delinquent Peers in Context: A Longitudinal Network Analysis of Selection and Influence Effects.” Criminology 49:253–86.

58.

Weerman

F. M.

Wilcox

Sullivan

C. J.

. 2017. “The Short-term Dynamics of Peers and Delinquent Behavior: An Analysis of Bi-weekly Changes within a High School Student Network.” Journal of Quantitative Criminology.

59.

Zhang

Tong

Lamberson

P. J.

Durazo-Arvizu

R. A.

Luke

Shoham

D. A.

. 2015. “Leveraging Social Influence to Address Overweight and Obesity Using Agent-based Models: The Role of Adolescent Social Networks.” Social Science & Medicine 125:203–13.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.44 MB