Generalizing the Network Scale-up Method

Abstract

The network scale-up method enables researchers to estimate the sizes of hidden populations, such as drug injectors and sex workers, using sampled social network data. The basic scale-up estimator offers advantages over other size estimation techniques, but it depends on problematic modeling assumptions. The authors propose a new generalized scale-up estimator that can be used in settings with nonrandom social mixing and imperfect awareness about membership in the hidden population. In addition, the new estimator can be used when data are collected via complex sample designs and from incomplete sampling frames. However, the generalized scale-up estimator also requires data from two samples: one from the frame population and one from the hidden population. In some situations these data from the hidden population can be collected by adding a small number of questions to already planned studies. For other situations, the authors develop interpretable adjustment factors that can be applied to the basic scale-up estimator. The authors conclude with practical recommendations for the design and analysis of future studies.

Keywords

hidden populations social networks sampling network scale-up method

1. Introduction

Many important problems in social science, public health, and public policy require estimates of the sizes of hidden populations. For example, in HIV/AIDS research, estimates of the size of the most at-risk populations—drug injectors, female sex workers, and men who have sex with men—are critical for understanding and controlling the spread of the epidemic. However, researchers and policymakers are unsatisfied with the ability of current statistical methods to provide these estimates (Joint United Nations Programme on HIV/AIDS 2010). We address this problem by improving the network scale-up method, a promising approach to size estimation. Our results are immediately applicable in many substantive domains in which size estimation is challenging, and the framework we develop advances the understanding of sampling in networks more generally.

The core insight behind the network scale-up method is that ordinary people have embedded within their personal networks information that can be used to estimate the sizes of hidden populations, if that information can be properly collected, aggregated, and adjusted (Bernard et al. 1989, 2010). In a typical scale-up survey, randomly sampled adults are asked about the number of connections they have to people in a hidden population (e.g., “How many people do you know who inject drugs?”) and a series of similar questions about groups of known size (e.g., “How many widowers do you know?”“How many doctors do you know?”). Responses to these questions are called aggregate relational data (McCormick et al. 2012).

To produce size estimates from aggregate relational data, previous researchers have begun with the basic scale-up model, which makes three important assumptions: (1) Social ties are formed completely at random (i.e., random mixing), (2) respondents are perfectly aware of the characteristics of their alters, and (3) respondents are able to provide accurate answers to survey questions about their personal networks. From the basic scale-up model, Killworth, McCarty, et al. (1998) derived the basic scale-up estimator. This estimator, which is widely used in practice, has two main components. For the first component, the aggregate relational data about the hidden population are used to estimate the number of connections that respondents have to the hidden population. For the second component, the aggregate relational data about the groups of known size are used to estimate the number of connections that respondents have in total. For example, a researcher might estimate that members of her sample have 5,000 connections to people who inject drugs and 100,000 connections in total. The basic scale-up estimator combines these pieces of information to estimate that 5 percent ( $5, 000 / 100, 000$ ) of the population injects drugs. This estimate is a sample proportion, but rather than being taken over the respondents, as would be typical in survey research, the proportion is taken over the respondents’ alters. Researchers who desire absolute size estimates multiply the alter sample proportion by the size of the entire population, which is assumed to be known (or estimated using some other method).

Unfortunately, the three assumptions underlying the basic scale-up model have all been shown to be problematic. Scale-up researchers call violations of the random mixing assumption barrier effects (Killworth et al. 2006; Maltiel et al. 2015; Zheng, Salganik, and Gelman 2006), they call violations of the perfect awareness assumption transmission error (Killworth et al. 2006; Maltiel et al. 2015; Salganik, Mello, et al. 2011; Shelley et al. 1995, 2006), and they call violations of the respondent accuracy assumption recall error (Killworth et al. 2003, 2006; Maltiel et al. 2015; McCormick and Zheng 2007).

In this paper, we develop a new approach to producing size estimates from aggregate relational data. Rather than depending on the basic scale-up model or its variants (e.g., Maltiel et al. 2015), we use a simple identity to derive a series of new estimators. Our new approach reveals that one of the two main components of the basic scale-up estimator is problematic. Therefore, we propose a new estimator—the generalized scale-up estimator—that combines the aggregate relational data traditionally used in scale-up studies with similar data collected from the hidden population. Collecting data from the hidden population is a major departure from current scale-up practice, but we believe that it enables a more principled approach to estimation. For researchers who are not able to collect data from the hidden population, we propose a series of adjustment factors that highlight the possible biases of the basic scale-up estimator. Ultimately, researchers must balance the trade-offs between the basic scale-up estimator, generalized scale-up estimator, and other size estimation techniques on the basis of the specific features of their research setting.

In Section 2, we derive the generalized scale-up estimator, and we describe the data collection procedures needed to use it. In Section 3, we compare the generalized and basic scale-up approaches analytically and with simulations; our comparison leads us to propose a decomposition that separates the difference between the two approaches into three measurable and substantively meaningful factors (equation 15). In Section 4, we make practical recommendations for the design and analysis of future scale-up studies, and in Section 5, we conclude with a discussion of the steps that follow. Appendices A to G in the online journal provide technical details and supporting arguments.

2. The Generalized Scale-Up Estimator

The generalized scale-up estimator can be derived from a simple accounting identity that requires no assumptions about the underlying social network structure in the population. Figure 1 helps illustrate the derivation, which was inspired by earlier research on multiplicity estimation (Sirken 1970) and indirect sampling (Lavallée 2007). Consider a population of seven people, two of whom are drug injectors (Figure 1a). In this population, two people are connected by a directed edge $i \to j$ if person $i$ would count person $j$ as a drug injector when answering the question “How many drug injectors do you know?” Whenever $i \to j$ , we say that $i$ makes an out-report about $j$ and that $j$ receives an in-report from $i$ .¹

Figure 1.

Illustration of the derivation of the generalized scale-up estimator. (a) A population of seven people, two of whom are drug injectors (shown in gray). A directed edge $i \to j$ indicates that $i$ counts $j$ as a drug injector when answering the question “How many drug injectors do you know?” (b) The same population, redrawn so that each person now appears twice: as a source of out-reports, on the left, and as a recipient of in-reports, on the right. This arrangement shows that total out-reports and total in-reports must be equal. (c) The same population again, but now some of the people are in the frame population $F$ and some are not. In real scale-up studies, we can only learn about out-reports from the frame population.

Each person can be viewed as both a source of out-reports and a recipient of in-reports, and in order to emphasize this point, Figure 1b shows the population with each person represented twice: on the left as a sender of out-reports and on the right as a receiver of in-reports. This visual representation highlights the following identity:

total out-reports = total in-reports .

Despite its simplicity, the identity in equation (1) turns out to be very useful because it leads directly to the new estimator that we propose.

In order to derive an estimator from equation (1), we must define some notation. Let $U$ be the entire population, and let $H \subset U$ be the hidden population. Furthermore, let $y_{i, H}$ be the total number of out-reports from person $i$ (i.e., person $i$ ’s answer to the question “How many drug injectors do you know?”). For example, Figure 1b shows that person 5 would report knowing one drug injector, so $y_{5, H} = 1$ . Let $v_{i, U}$ be the total number of in-reports to $i$ if everyone in $U$ is interviewed; that is, $v_{i, U}$ is the visibility of person $i$ to people in $U$ . For example, Figure 1b shows that person 5 would be reported as a drug injector by three people so $v_{5, U} = 3$ . Because total out-reports must equal total in-reports, it must be the case that

y_{U, H} = v_{U, U},

where $y_{U, H} = \sum_{i \in U} y_{i, H}$ and $v_{U, U} = \sum_{i \in U} v_{i, U}$ . Multiplying both sides of equation (2) by $N_{H}$ , the number of people in the hidden population, and then rearranging terms, we get

N_{H} = \frac{y_{U, H}}{v_{U, U} / N_{H}} .

Equation (3) is an expression for the size of the hidden population that does not depend on any assumptions about network structure or reporting accuracy; it is just a different way of expressing the identity that the total number of out-reports must equal the total number of in-reports. If we could estimate the two terms on the right side of equation (3)—one term related to out-reports ( $y_{U, H}$ ) and one term related to in-reports ( $v_{U, U} / N_{H}$ )—then we could estimate $N_{H}$ .

However, in order to make the identity in equation (3) useful in practice we need to modify it to account for an important logistical requirement of survey research. In real scale-up studies, researchers do not sample from the entire population $U$ , but instead they sample from a subset of $U$ called the frame population, $F$ . For example, in almost all scale-up studies the frame population has been adults (but note that our mathematical results hold for any frame population). In standard survey research, restricting interviews to a frame population does not cause problems because inference is being made about the frame population. In other words, when respondents report about themselves, it is clear to which group inferences apply. However, with the scale-up method, respondents report about others, so the group that inferences are being made about is not necessarily the same as the group that is being interviewed. As we show in Section 4.2, failure to consider this fact requires the introduction of an awkward adjustment factor that had previously gone unnoticed. Here, we avoid this awkward adjustment factor by deriving an identity explicitly in terms of the frame population. Restricting our attention to out-reports coming from people in the frame population, it must be the case that

N_{H} = \frac{y_{F, H}}{v_{U, F} / N_{H}},

where $y_{F, H} = \sum_{i \in F} y_{i, H}$ and $v_{U, F} = \sum_{i \in U} v_{i, F}$ . The only difference between equation (3) and equation (4) is that equation (4) restricts out-reports and in-reports to come from people in the frame population (Figure 1c). The identity in equation (4) is extremely general: it does not depend on any assumptions about the relationship between the entire population $U$ , the frame population $F$ , and the hidden population $H$ . For example, it holds if no members of the hidden population are in the frame population, if there are barrier effects, and if there are transmission errors. Thus, if we could estimate the two terms on the right side of equation (4)—one term related to out-reports ( $y_{F, H}$ ) and one term related to in-reports ( $v_{U, F} / N_{H}$ )—then we could estimate $N_{H}$ under very general conditions.

Unfortunately, despite repeated attempts, we were unable to develop a practical method for estimating the term related to in-reports ( $v_{U, F} / N_{H}$ ). However, if we make an assumption about respondents’ reporting behavior, then we can re-express equation (4) as an identity made up of quantities that we can actually estimate. Specifically, if we assume that the out-reports from people in the frame population only include people in the hidden population, then it must be the case that the visibility of everyone not in the hidden population is 0: $v_{i, F} = 0 for all i \notin H$ . In this case, we can rewrite equation (4) as

N_{H} = \frac{y_{F, H}}{v_{H, F} / N_{H}} = \frac{y_{F, H}}{{\bar{v}}_{H, F}} if v_{i, F} = 0 for all i \notin H,

where ${\bar{v}}_{H, F} = v_{H, F} / N_{H}$ .

To understand the reporting assumption substantively, consider the two possible types of reporting errors: false positives and false negatives. Previous scale-up research on transmission error focused on the problem of false negatives, where a respondent is connected to a member of the hidden population but does not report this, possibly because she is not aware that the person she is connected to is in the hidden population (Bernard et al. 2010). Because hidden populations such as drug injectors are often stigmatized, it is reasonable to suspect that false negatives will be a serious problem for the scale-up method. Fortunately, equation (5) holds even if there are false negative reporting errors. However, false positives—which do not seem to have been considered previously in the scale-up literature—are also possible. For example, a respondent who is not connected to any drug injectors might report that one of her acquaintances is a drug injector. These false positive reports are not accounted for in the identity in equation (5) and the estimators that we derive subsequently. If false positive reports exist, they will introduce a positive bias into estimates from the generalized scale-up estimator. Therefore, in Appendix A in the online journal we (1) formally define an interpretable measure of false positive reports, the precision of out-reports; (2) analytically show the bias in size estimates as a function of the precisions of out-reports; and (3) discuss two research designs that could enable researchers to estimate the precision of out-reports.

2.1. Estimating $N_{H}$ from Sampled Data

Equation (5) relates our quantity of interest, the size of the hidden population ( $N_{H}$ ), to two other quantities: the total number of out-reports from the frame population ( $y_{F, H}$ ) and the average number of in-reports in the hidden population ( ${\bar{v}}_{H, F}$ ). We now show how to estimate $y_{F, H}$ with a probability sample from the frame population and ${\bar{v}}_{H, F}$ with a relative probability sample from the hidden population.

The total number of out-reports ( $y_{F, H}$ ) can be estimated from respondents’ reported number of connections to the hidden population,

{\hat{y}}_{F, H} = \sum_{i \in s_{F}} \frac{y_{i, H}}{π_{i}},

where $s_{F}$ denotes the sample, $y_{i, H}$ denotes the reported number of connections between $i$ and $H$ , and $π_{i}$ is $i$ ’s probability of inclusion from a conventional probability sampling design from the frame population. Because ${\hat{y}}_{F, H}$ is a standard Horvitz-Thompson estimator, it is consistent and unbiased as long as all members of $F$ have a positive probability of inclusion under the sampling design (Sarndal, Swensson, and Wretman 1992); for a more formal statement, see Result B.1. This estimator depends only on an assumption about the sampling design for the frame population, and in Table D.2, we show the sensitivity of our estimator to violations of this assumption.

Estimating the average number of in-reports for the hidden population ( ${\bar{v}}_{H, F}$ ) is more complicated. First, it will usually be impossible to obtain a conventional probability sample from the hidden population. As we show below, however, estimating ${\bar{v}}_{H, F}$ requires only a relative probability sampling design in which hidden population members have a nonzero probability of inclusion and respondents’ probabilities of inclusion are known up to a constant of proportionality, $c π_{i}$ (see Appendix C.1 in the online journal for a more precise definition). Of course, even selecting a relative probability sample from a hidden population can be difficult.

A second problem arises because we do not expect respondents to be able to easily and accurately answer direct questions about their visibility ( $v_{i, F}$ ). That is, we do not expect respondents to be able to answer questions such as “How many people on the sampling frame would include you when reporting a count of the number of drug injectors that they know?” Instead, we propose asking hidden population members a series of questions about their connections to certain groups and their visibility to those groups. For example, each sampled hidden population respondent could be asked “How many widowers do you know?” and then “How many of these widowers are aware that you inject drugs?” This question pattern can be repeated for many groups (e.g., widowers, doctors, bus drivers). We call data with this structure enriched aggregate relational data to emphasize its similarity to the aggregate relational data that is familiar to scale-up researchers. An interviewing procedure called the game of contacts enables researchers to collect enriched aggregated relational data, even in realistic field settings (Salganik, Mello, et al. 2011; Maghsoudi et al. 2014).

Given a relative probability sampling design and enriched aggregate relational data, we can now formalize our proposed estimator for ${\bar{v}}_{H, F}$ . Let $A_{1}, A_{2}, \dots, A_{J}$ , be the set of groups about which we collect enriched aggregate relational data (e.g., widowers, doctors). Here, to keep the notation simple, we assume that these groups are all contained in the frame population, so that $A_{j} \subset F$ for all $j$ ; in Appendix C.4 in the online journal, we extend the results to groups that do not meet this criterion. Let $A$ be the concatenation of these groups, which we call the probe alters. For example, if $A_{1}$ is widowers and $A_{2}$ is doctors, then the probe alters $A$ is the collection of all widowers and all doctors, with doctors who are widowers included twice. Also, let ${\tilde{v}}_{i, A_{j}}$ be respondent $i$ ’s report about her visibility to people in $A_{j}$ and let $v_{i, A_{j}}$ be respondents $i$ ’s actual visibility to people in $A_{j}$ (i.e., the number of times that this respondent would be reported about if everyone in $A_{j}$ was asked about their connections to the hidden population).

The estimator for ${\bar{v}}_{H, F}$ is

{\hat{\bar{υ}}}_{H, F} = \frac{N_{F}}{N_{A}} \frac{\sum_{i \in s_{H}} \sum_{j} \tilde{υ_{i}}, A_{j} / (c π_{i})}{\sum_{i \in s_{H}} 1 / (c π_{i})},

where $N_{A}$ is the number of probe alters, $c$ is the constant of proportionality from the relative probability sample, and $s_{H}$ is a relative probability sample of the hidden population. Equation (7) is a standard weighted sample mean (Sarndal et al. 1992, Section 5.7) multiplied by a constant, $N_{F} / N_{A}$ . Result C.2 shows that this estimator is consistent and essentially unbiased,² when three conditions are satisfied: one about the design of the survey, one about reporting behavior, and one about sampling from the hidden population.

The first condition underlying the estimator in equation (7) is related to the design of the survey, and we call it the probe alter condition. This condition describes the required relationship between the visibility of the hidden population to the probe alters and the visibility of the hidden population to the frame population:

\frac{v_{H, A}}{N_{A}} = \frac{v_{H, F}}{N_{F}},

where $v_{H, A}$ is the total visibility of the hidden population to the probe alters, $v_{H, F}$ is the total visibility of the hidden population to the frame population, $N_{A}$ is the number of probe alters, and $N_{F}$ is the number of people in the frame population. In words, equation (8) says that the rate at which the hidden population is visible to the probe alters must be the same as the rate at which the hidden population is visible to the frame population. For example, in a study to estimate the number of drug injectors in a city, drug treatment counselors would be a poor choice for membership in the probe alters because drug injectors are probably more visible to drug treatment counselors than to typical members of the frame population. On the other hand, postal workers would probably be a reasonable choice for membership in the probe alters because drug injectors are probably about as visible to postal workers as they are to typical members of the frame population. Additional results about the probe alter condition are presented in the online appendices: (1) Result C.3 presents three other algebraically equivalent formulations of probe alter condition, some of which offer additional intuition; (2) Result C.4 provides a method to empirically test the probe alter condition; and (3) Table D.1 quantifies the bias introduced when the probe alter condition is not satisfied.

The second condition underlying the estimator ${\hat{\bar{v}}}_{H, F}$ (equation 7) is related to reporting behavior, and we call it accurate aggregate reports about visibility:

{\tilde{v}}_{H, A} = v_{H, A},

where ${\tilde{v}}_{H, A}$ is the total reported visibility of members of the hidden population to the probe alters ( $\sum_{i \in H} \sum_{A_{j} \in A} {\tilde{v}}_{i, A_{j}}$ ) and $v_{H, A}$ is the total actual visibility of members of the hidden population to the probe alters ( $\sum_{i \in H} \sum_{A_{j} \in A} v_{i, A_{j}}$ ). In words, equation (9) says that hidden population members must be correct in their reports about their visibility to probe alters in aggregate, but equation (9) does not require the stronger condition that each individual report be accurate. In practice, we expect that there are two main ways that there might not be accurate aggregate reports about visibility. First, hidden population members might not be accurate in their assessments of what others know about them. For example, research on the “illusion of transparency” suggests that people tend to overestimate how much others know about them (Gilovich, Savitsky, and Medvec 1998). Second, although we propose asking hidden population members what other people know about them (e.g., “How many of these widowers know that you are a drug injector?”) what actually matters for the estimator is what other people would report about them (e.g., “How many of these widowers would include you when reporting a count of the number of drug injectors that they know?”). In cases in which the hidden population is extremely stigmatized, some respondents to the scale-up survey might conceal the fact that they are connected to people whom they know to be in the hidden population, and if this were to occur, it would lead to a difference between the information that we collect ( ${\tilde{v}}_{i, A}$ ) and the information that we want ( $v_{i, A}$ ). Unfortunately, there is currently no empirical evidence about the possible magnitude of these two problems in the context of scale-up studies. However, Table D.1 quantifies the bias introduced into estimates if the accurate aggregate reports about visibility condition is not satisfied.

Finally, the third condition underlying the estimator ${\hat{\bar{v}}}_{H, F}$ (equation 7) is that researchers have a relative probability sample from the hidden population. Currently the most widely used method for drawing relative probability samples from hidden populations is respondent-driven sampling (Heckathorn 1997); see Volz and Heckathorn (2008) for a set of conditions under which respondent-driven sampling leads to a relative probability sample. Although respondent-driven sampling has been used in hundreds of studies around the world (White et al. 2015), there is active debate about the characteristics of samples that it yields (Bengtsson and Thorson 2010; Gile and Handcock 2010, 2015; Gile, Johnston, and Salganik 2015; Goel and Salganik 2010; Heimer 2005; Li and Rohe 2015; McCreesh et al. 2012; Mills et al. 2012; Rohe 2015; Rudolph et al. 2013; Salganik 2012; Scott 2008; Yamanis et al. 2013). If other methods for sampling from hidden populations are demonstrated to be better than respondent-driven sampling (e.g., see Karon and Wejnert 2012; Kurant, Markopoulou, and Thiran 2011; Mouw and Verdery 2012), then researchers should consider these methods when using the generalized scale-up estimator. Furthermore, researchers can use Table D.2 to quantify the bias that results if the condition requiring a relative probability sample is not satisfied.

To recap, using two different data collection procedures—one with the frame population and one with the hidden population—we can estimate the two components of the expression for $N_{H}$ given in equation (5). The estimator for the numerator ( ${\hat{y}}_{F, H}$ ) depends on an assumption about the ability to select a probability sample from the frame population (see Result B.1), and the estimator for the denominator ( ${\hat{\bar{v}}}_{H, F}$ ) depends on assumptions about survey construction, reporting behavior, and the ability to select a relative probability sample from the hidden population (see Result C.2).

We can combine these component estimators to form the generalized scale-up estimator:

{\hat{N}}_{H} = \frac{{\hat{y}}_{F, H}}{{\hat{\bar{υ}}}_{H, F}}

Result C.8 proves that the generalized scale-up estimator will be consistent and essentially unbiased if (1) the estimator for the numerator ( ${\hat{y}}_{F, H}$ ) is consistent and essentially unbiased, (2) the estimator for the denominator ( ${\hat{\bar{v}}}_{H, F}$ ) is consistent and essentially unbiased, and (3) there are no false positive reports.

One attractive feature of the generalized scale-up estimator (equation 10) is that it is a combination of standard survey estimators. This structure enabled us to derive very general sensitivity results about the impact of violations of assumptions, either individually or jointly. We return to the issue of assumptions and sensitivity analysis when discussing recommendations for practice (Section 4).

3. Comparison Between the Generalized and Basic Scale-Up Approaches

In Section 2, we derived the generalized network scale-up estimator by using an identity relating in-reports and out-reports as the basis for a design-based estimator. The approach we followed differs from previous scale-up studies, which have posited the basic scale-up model and derived estimators conditional on that model. In this section, we compare these two different approaches from a design-based perspective.

We begin our comparison by reviewing the basic scale-up model, which was used in most of the studies listed in Table 1. To review this model, we need to define another quantity: we call $d_{i, U}$ person $i$ ’s degree, the number of undirected network connections she has to everyone in $U$ .

Table 1.

Network Scale-up Studies That Have Been Completed

Hidden Population(s)	Location	Citation
Mortality in earthquake	Mexico City, Mexico	Bernard et al. (1989)
Rape victims	Mexico City, Mexico	Bernard et al. (1991)
HIV prevalence, rape, and homelessness	United States	Killworth, McCarty, et al. (1998)
Heroin use	14 U.S. cities	Kadushin et al. (2006)
Choking incidents in children	Italy	Snidero et al. (2007, 2009, 2012)
Groups most at risk for HIV/AIDS	Ukraine	Paniotto et al. (2009)
Heavy drug users	Curitiba, Brazil	Salganik, Fazito, et al. (2011)
Groups most at risk for HIV/AIDS	Kerman, Iran	Shokoohi, Baneshi, and Haghdoost (2012)
Men who have sex with men	Japan	Ezoe et al. (2012)
Groups most at risk for HIV/AIDS	Almaty, Kazakhstan	Scutelniciuc (2012a)
Groups most at risk for HIV/AIDS	Moldova	Scutelniciuc (2012b)
Groups most at risk for HIV/AIDS	Thailand	Aramrattan and Kanato (2012)
Groups most at risk for HIV/AIDS	Rwanda	Rwanda Biomedical Center (2012)
Groups most at risk for HIV/AIDS	Chongqing, China	Guo et al. (2013)
Groups most at risk for HIV/AIDS	Tabriz, Iran	Khounigh et al. (2014)
Men who have sex with men	Taiyuan, China	Jing et al. (2014)
Drug and alcohol users	Kerman, Iran	Sheikhzadeh et al. (2014)
Men who have sex with men	Shanghai, China	Wang et al. (2015)
Men who have sex with men	Tbilisi, Georgia	Sulaberidze et al. (2016)
Illicit drug users	Iran	Nikfarjam et al. (2016)

The basic scale-up model assumes that each person’s connections are formed independently, that reporting is perfect, and that visibility is perfect (Killworth, McCarty, et al. 1998). Together, these three assumptions lead to the probabilistic model:

y_{i}, A_{j} = d_{i}, A_{j} ~ Binomial (d_{i, U}, \frac{N_{Aj}}{N}),

for all $i$ in $U$ and for any group $A_{j}$ . In words, this model suggests that the number of connections from a person $i$ to members of a group $A_{j}$ is the result of a series of $d_{i, U}$ independent random draws, where the probability of each edge being connected to $A_{j}$ is $\frac{N_{A_{j}}}{N}$ .

The basic scale-up model leads to what we call the basic scale-up estimator:

{\hat{N}}_{H} = \frac{\sum_{i \in s_{F}} y_{i, H}}{\sum_{i \in s_{F}} {\hat{d}}_{i, U}} \times N,

where ${\hat{d}}_{i, U}$ is the estimated degree of respondent $i$ from the known population method (Killworth, Johnsen, et al. 1998). Killworth, McCarty, et al. (1998) showed that equation (12) is the maximum likelihood estimator for $N_{H}$ under the basic scale-up model, conditional on the additional assumption that $d_{i, U}$ is known for each $i \in s_{F}$ .

Given this background, we can now compare the basic and generalized scale-up approaches by comparing their estimands; that is, we compare the quantities that they produce in the case of a census with perfectly observed degrees. The basic scale-up estimand can be written

{\hat{N}}_{H} = \frac{y_{F, H}}{d_{F, U}} \times N = \frac{y_{F, H}}{{\bar{d}}_{U, F}},

where $d_{F, U} = \sum_{i \in F} d_{i, U}$ and ${\bar{d}}_{U, F} = d_{U, F} / N = d_{F, U} / N$ . Furthermore, as shown in Section 2, the generalized scale-up estimand is

{\hat{N}}_{H} = \frac{y_{F, H}}{{\bar{υ}}_{H, F}} .

Comparing equations (13) and (14) reveals that both estimands have the same numerator, but they have different denominators. The network reporting identity from Section 2 (total out-reports = total in-reports) shows that the appropriate way to adjust the out-reports is based on in-reports, as in the generalized scale-up approach. However, the basic scale-up approach instead adjusts out-reports with the degree of respondents. Although using the degree of respondents cleverly avoids any data collection from the hidden population, our results reveal that it will be correct only under a very specific special case ( ${\bar{d}}_{U, F} = {\bar{v}}_{H, F}$ ).

To further clarify the relationship between the basic and generalized scale-up approaches, we propose a decomposition that separates the difference between the two estimands into three measurable and substantively meaningful adjustment factors:

N_{H} = \underset{\binom{basic}{s c a l e - up}}{\underset{︸}{(\frac{y_{F, H}}{{\bar{d}}_{U, F}})}} \times \underset{adjustment factors}{\underset{︸}{\underset{\binom{frame ratio}{ϕ_{F}}}{\underset{︸}{\frac{1}{{\bar{d}}_{F, F} / {\bar{d}}_{U, F}}}} \times \underset{\binom{degree ratio}{δ_{F}}}{\underset{︸}{\frac{1}{{\bar{d}}_{H, F} / {\bar{d}}_{F, F}}}} \times \underset{\binom{true positive rate}{τ_{F}}}{\underset{︸}{\frac{1}{{\bar{v}}_{H, F} / {\bar{d}}_{H, F}}}}}} = \underset{\binom{generalized}{scale - up}}{\underset{︸}{(\frac{y_{F, H}}{{\bar{v}}_{H, F}})}} .

The decomposition shows that when the product of the adjustment factors is 1, the two estimands are both correct. However, when the product of the adjustment factors is not 1, then the generalized scale-up estimand is correct but the basic scale-up estimand is incorrect. We now describe each of the three adjustment factors in turn.

First, we define the frame ratio, $ϕ_{F}$ , to be

ϕ_{F} = \frac{avg # connections from a member of F to the rest of F}{avg # connections from a member of U to F} = \frac{{\bar{d}}_{F, F}}{{\bar{d}}_{U, F}} .

$ϕ_{F}$ can range from 0 to infinity, and in most practical situations we expect that $ϕ_{F}$ will be greater than 1. Result B.6 shows that we can make consistent and essentially unbiased estimates of $ϕ_{F}$ from a sample of $F$ .³

Next, we define the degree ratio $δ_{F}$ to be

δ_{F} = \frac{avg # connections from a member of H to F}{avg # connections from a member of F to the rest of F} = \frac{{\bar{d}}_{H, F}}{{\bar{d}}_{F, F}} .

$δ_{F}$ ranges from 0 to infinity, and it is less than 1 when the hidden population members have, on average, fewer connections to the frame population than frame population members. Result C.6 shows that we can make consistent and essentially unbiased estimates of $δ_{F}$ from samples of $F$ and $H$ .

Finally, we define the true positive rate, $τ_{F}$ , to be

τ_{F} = \frac{# in-reports to H from F}{# edges connecting H and F} = \frac{v_{H, F}}{d_{H, F}} = \frac{{\bar{v}}_{H, F}}{{\bar{d}}_{H, F}} .

$τ_{F}$ relates network degree to network reports.⁴ $τ_{F}$ ranges from $0$ , if none of the edges are correctly reported, to $1$ if all of the edges are reported. Substantively, the more stigmatized the hidden population, the closer we would expect $τ_{F}$ to be to 0. Result C.7 shows that we can make consistent and essentially unbiased estimates of $τ_{F}$ from a sample of $H$ .

Furthermore, the decomposition in equation (15) can be used to derive an expression for the bias in the basic scale-up estimator when we have a census and degrees are known:

bias ({\hat{N}}_{H}^{basic}) \equiv {\hat{N}}_{H}^{basic} - N_{H}

= {\hat{N}}_{H}^{basic} [1 - \frac{1}{ϕ_{F} δ_{F} τ_{F}}]

The comparison between the basic and generalized scale-up approaches leads to two main conclusions. First, the estimand of the basic scale-up approach is correct only in one particular situation: when the product of the three adjustment factors is 1. The estimand of generalized scale-up approach, in contrast, is correct more generally. Second, as equation 15 shows, if the adjustment factors are known (or have been estimated), then they can be used to improve basic scale-up estimates.

3.1. Illustrative Simulation

To illustrate our comparison between the basic and generalized scale-up approaches, we conducted a series of simulation studies. The simulations were not meant to be a realistic model of a scale-up study, but rather, they were designed to clearly illustrate our analytic results. More specifically, the simulation investigated the performance of the estimators as three important quantities vary: (1) the size of the frame population $F$ , relative to the size of the entire population $U$ ; (2) the extent to which people’s network connections are not formed completely at random; and (3) the accuracy of reporting, as captured by the true positive rate $τ_{F}$ (see equation 18).⁵

As described in detail in Appendix G in the online journal, we created populations of $5, 000$ people with different proportions of the population on the sampling frame ( $p_{F}$ ). Next, we connected the people with a social network created by a stochastic block model (Wasserman and Faust 1994; White, Boorman, and Breiger 1976) in which the randomness of the mixing was controlled by a parameter $ρ$ such that $ρ = 1$ is equivalent to random mixing (i.e., an Erdos-Reyni random graph) and the mixing becomes more nonrandom as $ρ \to 0$ . Then, for each combination of parameters, we drew 10 populations, and within each of these populations, we simulated 500 surveys. For each survey, we drew a probability sample of 500 people from the frame population, a relative probability sample of 30 people from the hidden population, and simulated responses with a specific level of reporting accuracy ( $τ_{F}$ ). Finally, we used these reports and the appropriate sampling weights to calculate the basic and generalized scale-up estimates.

Figure 2 shows that the simulations support our analytic results. First, the simulations show that the generalized scale-up estimator is unbiased even in the presence of incomplete sampling frames, nonrandom mixing, and imperfect reporting. Second, they show that the basic scale-up estimator is unbiased in a much smaller set of situations. More concretely, the basic scale-up estimator is unbiased in situations in which the basic scale-up model holds—when everyone is in the frame population ( $p_{F} = 1$ ), there is random mixing ( $ρ = 1$ ), and respondents’ reports are perfect ( $τ_{F} = 1$ ).⁶ Furthermore, Figure 3 illustrates that our analytic approach (equation 3) can correctly predict the bias of the basic scale-up estimator.

Figure 2.

Estimated size of the hidden population for the generalized and basic scale-up estimators. Each panel shows how the two estimators change as the amount of random mixing is varied from low ( $ρ = 0.1$ ; members of the hidden population are relatively unlikely to form contacts with nonmembers) to high ( $ρ = 1$ ; members of the hidden population form contacts independent of other people’s hidden population membership). The columns show results for different sizes of the frame population, from small (left column, $p_{F} = 0.1$ ), to large (right column, $p_{F} = 1$ ). The rows show results for different levels of reporting accuracy, from a small amount of true positives (top row, $τ_{F} = 0.1$ ), to perfect reporting (bottom row, $τ_{F} = 1$ ). For example, looking at the middle of the center panel, when $p_{F} = 0.5$ , $τ_{F} = 0.5$ , and $ρ = 0.5$ , we see that the average basic scale-up estimate is about 50, while the average generalized scale-up estimate is 150 (the true value). The generalized scale-up estimator is unbiased for all parameter combinations, while the basic scale-up estimator is only unbiased for certain special cases (e.g., when $ρ = 1$ , $τ_{F} = 1$ , and $p_{F} = 1$ ). Full details of the simulation are presented in Appendix G in the online journal.

Figure 3.

Bias (open circles and diamonds) and predicted bias (solid lines) in the basic scale-up estimates and generalized scale-up estimates for the same parameter configurations depicted in Figure 2. Our analytical results (equation 20) accurately predict the bias observed in our simulation study.

4. Recommendations for Practice

The results in Sections 2 and 3 lead us to recommend a major departure from current scale-up practice. In addition to collecting a sample from the frame population, we recommend that researchers consider collecting a sample from the hidden population so that they can use the generalized scale-up estimator. As our results clarify, researchers using the scale-up method face a decision: they can collect data from the hidden population or they can make assumptions about the adjustment factors described in Section 3. The appropriate decision depends on a number of factors, but we think that two are most important: (1) the difficulty of sampling from the hidden population and (2) the availability of high-quality estimates of the adjustment factors in Section 3. For example, if it is particularly difficult to sample from a specific hidden population and high-quality estimates of the adjustment factors are already available, then a basic scale-up estimator may be appropriate. If, however, it is possible to sample from the hidden population and there are no high-quality estimates of adjustment factors, then the generalized scale-up estimator may be appropriate. Many realistic situations will be somewhere between these two extremes, and the trade-offs must be weighed on a case-by-case basis.

To aid researchers deciding between basic and generalized scale-up approaches, we collected the conditions needed for consistent and essentially unbiased estimates into Table 2; formal proofs of these results are presented in Online Appendices B and C. We find it helpful to group these conditions into four broad categories: sampling, survey construction, network structure, and reporting behavior.

Table 2.

Summary of the Conditions Needed for the Generalized and Modified Basic Network Scale-up Estimators, and Their Components, to Produce Estimates That Are Consistent and Essentially Unbiased

Quantity	Conditions Required	Condition Type	Result
Reported connections to H $({\hat{y}}_{F, H})$	1. Probability sample from F	Sampling	B1
Average personal network size of F $({\hat{d}}_{F, F})$	1. Probability sample from F 2. Groups of known size total is accurate N_A 3. Probe alter condition $({\bar{d}}_{A, F} = {\bar{d}}_{F, F})$ 4. Accurate reporting condition $(y_{F, A} = d_{F, A})$	Sampling Survey construction Survey construction Reporting behavior	B3
Average visibility of H $({\hat{\bar{υ}}}_{H, F})$	1. Relative probability sample from H 2. Groups of known size total is accurate $N_{A_{H} \cap F}$ 3. Probe alter condition $(\frac{υ_{H, A \cap F}}{N_{A \cap F}} = \frac{υ_{H, F}}{N_{F}})$ 4. Accurate aggregate reports about visibility $({\bar{υ}}_{H, A_{H} \cap F} = υ_{H, A_{H} \cap F})$	Sampling Survey construction Survey construction Reporting behavior	C2
Generalized scale-up $({\hat{N}}_{H} = \frac{{\hat{y}}_{F, H}}{{\hat{\bar{υ}}}_{H, F}})$	1. Conditions needed for ${\hat{y}}_{F, H}$ 2. Conditions needed for ${\hat{\bar{υ}}}_{H, F}$ 3. No false positive reports about connections to H $(η_{F} = 1)$	Sampling Sampling, survey construction, Reporting behavior Reporting behavior	C8
Modified basic scale-up $({\hat{N}}_{H} = \frac{{\hat{y}}_{F, H}}{{\hat{\bar{d}}}_{F, F}})$	1. Conditions needed for ${\hat{y}}_{F, H}$ 2. Condition needed for ${\hat{\bar{d}}}_{F, F}$ 3. No false positive reports about connections to H $(η_{F} = 1)$ 4. >Members of H and members of F have same average personal network size $(δ_{F} = 1)$ 5. No false negative reports about connections to H $(τ_{F} = 1)$	Sampling Sampling, survey construction Reporting behavior Reporting behavior Network structure Reporting behavior	Sections 2 and 3

Note: This table uses the version of the basic scale-up estimator we recommend in Section 4.2.

A review of the conditions in Table 2 necessarily raises practical concerns. In situations in which researchers are trying to make estimates about real hidden populations, they probably will not know how close they are to meeting these conditions. Therefore, researchers may wonder how their estimates will be affected by violations of these assumptions, both individually (e.g., “How would my estimates be affected if there was a problem with the survey construction?”) and jointly (e.g., “How would my estimate be affected if there was a problem with my survey construction and reporting behavior?”). To address this concern, in Appendix D in the online journal, we develop a framework for sensitivity analysis that shows researchers exactly how estimates will be affected by violations of all assumptions, either individually or jointly. Table 3 summarizes the results of our sensitivity framework.

Table 3.

Analytical Expressions Researchers Can Use to Perform Sensitivity Analysis for Estimates Made Using Scale-up Estimators

Quantity	Conditions Required	Adjusted Estimand for Sensitivity Analysis
Generalized scale-up $({\hat{N}}_{H} = \frac{{\hat{y}}_{F, H}}{{\hat{\bar{υ}}}_{H, F}})$	1. Probability sample from F with accurate weights $(K_{F_{2}} = 0 and {\bar{\in}}_{F} = 1)$ 2. Relative probability sample from H with accurate weights $(K_{H} = 0)$ 3. Conditions needed for ${\hat{\bar{υ}}}_{H, F}$ $({\hat{\bar{υ}}}_{F, H} = k {\bar{υ}}_{F, H})$ 4. No false positive reports about connections to H $(η_{F} = 1)$	${\hat{N}}_{H} \cdot \frac{(1 + K_{H})}{{\bar{\in}}_{F} (1 + K_{F_{2}})} \cdot k \cdot η_{F} ⇝ N_{H}$
Modified basic scale-up $({\hat{N}}_{H} = \frac{{\hat{y}}_{F, H}}{{\hat{\bar{d}}}_{F, F}})$	1. Probability sample from F with accurate weights for $y_{F, H}$ $(K_{F_{2}} = 0)$ 2. Probability sample from F with accurate weights for $y_{F, A}$ $(K_{F_{1}} = 0)$ 3. Condition needed for ${\hat{\bar{d}}}_{F, F}$ $({\hat{\bar{d}}}_{F, F} = k {\bar{d}}_{, F})$ 4.No false positive reports about connections to H $(η_{F} = 1)$ 5. Members of H and members of F have same average personal network size $(δ_{F} = 1)$ 6. No false negative reports about connections to H $(τ_{F} = 1)$	${\hat{N}}_{H} \cdot \frac{(1 + K_{F_{1}})}{(1 + K_{F_{2}})} \cdot k \cdot \frac{η_{F}}{δ_{F} τ_{F}} ⇝ N_{H}$

Note: See Appendix D in the online journal for more details. $K_{F_{1}}$ , $K_{F_{2}}$ , and $K_{H}$ are indices that reflect how imperfect the sampling weights researchers use to make estimates are; when these $K$ values are 0, the weights are exactly correct; the farther they are from 0, the more imperfect the weights are. (We use the symbol ⇝ as a shorthand for “is consistent and essentially unbiased for.”)

Another problem researchers face in practice is putting appropriate confidence intervals around estimates. The procedure currently used in scale-up studies was proposed by Killworth, McCarty, et al. (1998), but it has a number of conceptual problems, and in practice, it produces intervals that are anticonservative (i.e., the actual coverage rate is lower than the desired coverage rate). Both of these problems—theoretical and empirical—do not seem to be widely appreciated in the scale-up literature. Therefore, instead of the current procedure, we recommend that researchers use the rescaled bootstrap procedure (Rao, Wu, and Yue 1992; Rao and Wu 1988; Rust and Rao 1996), which has strong theoretical foundations, does not depend on the basic scale-up model, can handle both simple and complex sample designs, and can be used for both the basic scale-up estimator and the generalized scale-up estimator. In Appendix F in the online journal, we review the current scale-up confidence interval procedure and the rescaled bootstrap, highlighting the conceptual advantages of the rescaled bootstrap. Furthermore, we show that the rescaled bootstrap produces slightly better confidence intervals in three real scale-up data sets: one collected via simple random sampling (McCarty et al. 2001) and two collected via complex sample designs (Salganik, Fazito, et al. 2011; Rwanda Biomedical Center 2012). Finally, and somewhat disappointingly, our results show that none of the confidence interval procedures work very well in an absolute sense, a finding that highlights an important problem for future research.

We now provide more specific guidance for researchers based on the data they decide to collect. In Section 4.1 we present recommendations for researchers who collect a sample from both the frame population, $F$ , and the hidden population, $H$ ; in Section 4.2, we present recommendations for researchers who only select a sample from the frame population.

4.1. Estimation with Samples from $F$ and $H$

We recommend that researchers who have samples from $F$ and $H$ use a generalized scale-up estimator to produce estimates of $N_{H}$ (see Section 2):

{\hat{N}}_{H} = \frac{{\hat{y}}_{F, H}}{{\hat{\bar{υ}}}_{H, F}}

For researchers using the generalized scale-up estimator, we have three specific recommendations. Of all the conditions needed for consistent and essentially unbiased estimation, the ones most under the control of the researcher are those related to survey construction, so we recommend that researchers focus on these during the study design phase. In particular, we recommend that the probe alters be designed so that the rate at which the hidden population is visible to the probe alters is the same as the rate at which the hidden population is visible to the frame population (see Result C.2 for a more formal statement, and see Section C5 for more advice about choosing probe alters). Second, when presenting estimates, we recommend that researchers use the results in Table 3 to also present sensitivity analyses highlighting how the estimates may be affefcted by assumptions that are particularly problematic in their setting. Finally, we recommend that researchers produce confidence intervals around their estimate using the rescaled bootstrap procedure, keeping in mind that this will likely produce intervals that are anticonservative.

We also have three additional recommendations that will facilitate the cumulation of knowledge about the scale-up method. First, although the generalized scale-up estimator does not require aggregate relational data from the frame population about groups of known size, we recommend that researchers collect these data so that the basic and generalized estimators can be compared. Second, we recommend that researchers publish estimates of $δ_{F}$ and $τ_{F}$ , although these quantities play no role in the generalized scale-up estimator (Figure 4). As a body of evidence about these adjustment factors accumulates (e.g., Salganik, Fazito, et al. 2011; Maghsoudi et al. 2014), studies that are not able to collect a sample from the hidden population will have an empirical foundation for adjusting basic scale-up estimates, either by borrowing values directly from the literature or by using published values as the basis for priors in a Bayesian model. Finally, we recommend that researchers design their data collections—both from the frame population and the hidden population—so that size estimates from the generalized scale-up method can be compared with estimates from other methods (e.g., see Salganik, Fazito, et al. 2011a). For example, if respondent-driven sampling is used to sample from the hidden population, then researchers could use methods that estimate the size of a hidden population from recruitment patterns in the respondent-driven sampling data (Berchenko, Rosenblatt, and Frost 2013; Crawford, Wu, and Heimer 2015; Handcock, Gile, and Mar 2014, 2015; Johnston et al. 2015; Wesson et al. 2015).

Figure 4.

Recommended schematic of inputs and outputs for a study using the generalized scale-up estimator. We recommend that researchers produce size estimates using the generalized scale-up estimator and that researchers produce estimates of the adjustment factors $δ_{F}$ and $τ_{F}$ to aid other researchers.

4.2. Estimation with Only a Sample from $F$

If researchers cannot collect a sample from the hidden population, we have three recommendations. First, we recommend two simple changes to the basic scale-up estimator that remove the need to adjust for the frame ratio, $ϕ_{F}$ . Recall, that the basic scale-up estimator that has been used in previous studies (see Section 3) is

{\hat{N}}_{H} = \frac{{\hat{y}}_{F, H}}{{\hat{d}}_{F, U}} \times N = \frac{{\hat{y}}_{F, H}}{{\hat{d}}_{F, U} / N} .

Instead of equation 22, we suggest a new estimator, called the modified basic scale-up estimator, that more directly deals with the fact that researchers sample from the frame population $F$ (typically adults), and not from the entire population $U$ (adults and children):

{\hat{N}}_{H} = \frac{{\hat{y}}_{F, H}}{{\hat{d}}_{F, F}} \times N_{F} = \frac{{\hat{y}}_{F, H}}{{\hat{d}}_{F, F} / N_{F}} .

There are two differences between the modified basic scale-up estimator (equation 23) and the basic scale-up estimator (equation 22). First, we recommend that researchers estimate ${\hat{d}}_{F, F}$ (i.e., the total number of connections between adults and adults) rather than ${\hat{d}}_{F, U}$ (i.e., the total number of connections between adults and everyone). To do so, researchers should design the probe alters for the frame population so that they have similar personal networks to the frame population; in Appendix B.4 in the online journal, we define this requirement formally, and in Section B.4.1 we provide guidance for choosing the probe alters. Second, we recommend that researchers use $N_{F}$ rather than $N$ .⁷ These two simple changes remove the need to adjust for the frame ratio $ϕ_{F}$ , and thereby eliminate an assumption about an unmeasured quantity. An improved version of the basic scale-up estimator would then be

{\hat{N}}_{H} = \frac{{\hat{y}}_{F, H}}{\underset{modified basic scale - up}{\underset{︸}{({\hat{d}}_{F, F} / N_{F})}}} \times \underset{adjustment factors}{\underset{︸}{\frac{1}{{\hat{δ}}_{F}} \times \frac{1}{{\hat{τ}}_{F}}}} .

Our second recommendation is that researchers using the modified basic scale-up estimator (equation 23) perform a sensitivity analysis using the results in Table 3. In particular, we think that researchers should be explicit about the values that they assume for the adjustment factors $δ_{F}$ and $τ_{F}$ . Our third recommendation is that researchers construct confidence intervals using the rescaled bootstrap procedure, while explicitly accounting for the fact that there is uncertainty around the assumed adjustment factors and bearing in mind that this procedure will likely produce intervals that are anticonservative.

5. Conclusion and Next Steps

In this paper, we developed the generalized network scale-up estimator. This new estimator improves upon earlier scale-up estimators in several ways: it enables researchers to use the scale-up method in populations with nonrandom social mixing and imperfect awareness about membership in the hidden population, and it accommodates data collection with complex sample designs and incomplete sampling frames. We also compared the generalized and basic scale-up estimators, leading us to introduce a framework that makes the design-based assumptions of the basic scale-up estimator precise. Finally, researchers who use either the basic or generalized scale-up estimator can use our results to assess the sensitivity of their size estimates to assumptions.

The approach we followed to derive the generalized scale-up estimator has three elements, and these elements may prove useful in other problems related to sampling in networks. First, we distinguished between the network of reports and the network of relationships. Second, using the network of reports, we derived a simple identity that permitted us to develop a design-based estimator free of any assumptions about the structure of the network of relationships. Third, we combined data from different types of samples. Together, these three elements may help other researchers in other situations derive relatively simple, design-based estimators that are an important complement to complex, model-based techniques.

Although the generalized scale-up estimator has many attractive features, it also requires that researchers obtain two different samples, one from the frame population and one from the hidden population. In cases in which studies of the hidden population are already planned (e.g., the behavioral surveillance studies of the groups most at risk for HIV/AIDS), the necessary information for the generalized scale-up estimator could be collected at little additional cost by appending a modest number of questions to existing questionnaires. In cases in which these studies are not already planned, researchers can either collect their own data from the hidden population, or they can use the modified basic scale-up estimator and borrow estimated adjustment factors from other published studies.

The generalized scale-up estimator, like all estimators, depends on a number of assumptions, and we think three of them will be most problematic in practice. First, the estimator depends on the assumption that there are no false positive reports, which is unlikely to be true in all situations. Although we have derived an estimator that works even in the presence of false positive reports (Appendix A in the online journal), we were not able to design a practical data collection procedure that would allow us to estimate one of the terms it requires. Second, the generalized scale-up estimator depends on the assumption that hidden population members have accurate aggregate awareness about visibility (equation 9). That is, researchers have to assume that hidden population respondents can accurately report whether or not their alters would report them, and we expect this assumption will be difficult to check in most situations. Third, the generalized scale-up estimator depends on having a relative probability sample from the hidden population. Unfortunately, we cannot eliminate any of these assumptions, but we have stated them clearly and we have derived the sensitivity of the estimates to violations of these assumptions, individually and jointly.

Our results and their limitations highlight several directions for further work, in terms of both of improved modeling and improved data collection. We think the most important direction for future modeling is developing estimators in a Bayesian framework, and a recent paper by Maltiel et al. (2015) offers some promising steps in this direction. We see two main advantages of the Bayesian approach in this setting. First, a Bayesian approach would allow researchers to propagate the uncertainty they have about the many assumptions involved in scale-up estimates, whereas our current approach captures only uncertainty introduced by sampling. Furthermore, as more empirical studies produce estimates of the adjustment factors ( $τ_{F}$ and $δ_{F}$ ), a Bayesian framework would permit researchers to borrow values from other studies in a principled way. In terms of future directions for data collection, researchers need practical techniques for estimating the rate of false positive reporting. These estimates, combined with the estimator in Appendix A in the online journal, would permit the relaxation of one of the most important remaining assumptions made by all scale-up studies to date. We hope that the framework introduced in this paper will provide a basis for these and other developments.

Footnotes

Acknowledgements

We thank Alexandre Abdo, Francisco Bastos, Russ Bernard, Neilane Bertoni, Dimitri Fazito, Sharad Goel, Wolfgang Hladik, Jake Hofman, Mike Hout, Karen Levy, Rob Lyerla, Mary Mahy, Chris McCarty, Maeve Mello, Tyler McCormick, Damon Phillips, Justin Rao, Adam Slez, and Tian Zheng for helpful discussions. Many of the methods described in this paper can be implemented using our accompanying open-source R package, called networkreporting, which is available on the Comprehensive R Archive Network. Replication materials for our analysis can be freely downloaded from the Harvard Dataverse ().

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the Joint United Nations Programme on HIV/AIDS, the National Science Foundation (CNS-0905086), and the Eunice Kennedy Shriver National Institute of Child Health and Human Development (R01-HD062366, R01-HD075666, and R24-HD047879). Some of this research was conducted while Dr. Salganik was an employee at Microsoft Research. The opinions expressed here represent the views of the authors and not the funding agencies.

Notes

Author Biographies

Dennis M. Feehan is an assistant professor of demography at the University of California, Berkeley. His research lies at the intersection of demography, social networks, and quantitative methodology.

Matthew J. Salganik is a professor of sociology at Princeton University, and he is affiliated with several of Princeton’s interdisciplinary research centers: the Office for Population Research, the Center for Information Technology Policy, the Center for Health and Wellbeing, and the Center for Statistics and Machine Learning. His research interests include social networks and computational social science.

References

Aramrattan

Kanato

2012. “Network Scale-up Method: Application in Thailand.” Presented at Consultation on Estimating Population Sizes through Household Surveys: Successes and Challenges, March 28–30, New York.

Bengtsson

Thorson

2010. “Global HIV Surveillance among MSM: Is Risk Behavior Seriously Underestimated?” AIDS 24(15):2301–303.

Berchenko

Rosenblatt

Frost

S.D.W.

2013. “Modeling and Analysing Respondent Driven Sampling as a Counting Process.” arXiv:1304.3505 [stat].

Bernard

H. R.

Hallett

Iovita

Johnsen

E. C.

Lyerla

McCarty

Mahy

Salganik

M. J.

Saliuk

Scutelniciuc

Shelley

G. A.

Sirinirund

Weir

Stroup

D. F.

2010. “Counting Hard-to-count Populations: The Network Scale-up Method for Public Health.” Sexually Transmitted Infections 86(Suppl. 2):ii11–15.

Bernard

H. R.

Johnsen

E. C.

Killworth

P. D.

Robinson

1989. “Estimating the Size of an Average Personal Network and of an Event Subpopulation.” Pp. 159–75 in The Small World, edited by Kochen

Norwood, NJ: Ablex.

Bernard

H. R.

Johnsen

E. C.

Killworth

Robinson

1991. “Estimating the Size of an Average Personal Network and of an Event Subpopulation: Some Empirical Results.” Social Science Research 20(2):109–21.

Crawford

F. W.

Heimer

2015. “Hidden Population Size Estimation from Respondent-driven Sampling: A Network Approach.” arXiv:1504.08349 [stat].

Csardi

Nepusz

2006. “The Igraph Software Package for Complex Network Research.” InterJournal, Complex Systems:1695.

Danenberg

2013. “Functional: Curry, Compose, and Other Higher-order Functions.” R package version0.4.

10.

Ezoe

Morooka

Noda

Sabin

M. L.

Koike

2012. “Population Size Estimation of Men Who Have Sex with Men through the Network Scale-up Method in Japan.” PLoS ONE 7(1):e31184.

11.

Feehan

D. M.

Salganik

M. J.

2014. “The Network Reporting Package.” R package version 0.1.1.

12.

Gile

K. J.

Handcock

M. S.

2010. “Respondent-driven Sampling: An Assessment of Current Methodology.'' Pp. 285–327 in Sociological Methodology, vol. 40, edited by Liao

Tim Futing

. Hoboken, NJ: Wiley-Blackwell.

13.

Gile

K. J.

Handcock

M. S.

2015. “Network Model-assisted Inference from Respondent-driven Sampling Data.” Journal of the Royal Statistical Society, Series A: Statistics in Society 178(3):619–39.

14.

Gile

K. J.

Johnston

L. G.

Salganik

M. J.

2015. “Diagnostics for Respondent-driven Sampling.” Journal of the Royal Statistical Society, Series A: Statistics in Society 178(1):241–69.

15.

Gilovich

Savitsky

Medvec

V. H.

1998. “The Illusion of Transparency: Biased Assessments of Others’ Ability to Read One’s Emotional States.” Journal of Personality and Social Psychology 75(2):332–46.

16.

Goel

Salganik

M. J.

2010. “Assessing Respondent-driven Sampling.” Proceedings of the National Academy of Science 107(15):6743–47.

17.

Guo

Bao

Lin

Zhang

Hladik

Abdul-Quader

Bulterys

Fuller

Wang

2013. “Estimating the Size of HIV Key Affected Populations in Chongqing, China, Using the Network Scale-up Method.” PLoS ONE 8(8):e71796.

18.

Handcock

M. S.

Gile

K. J.

Mar

C. M.

2014. “Estimating Hidden Population Size Using Respondent-driven Sampling Data.” Electronic Journal of Statistics 8(1):1491–1521.

19.

Handcock

M. S.

Gile

K. J.

Mar

C. M.

2015. “Estimating the Size of Populations at High Risk for HIV Using Respondent-driven Sampling Data.” Biometrics 71(1):258–66.

20.

Heckathorn

D. D.

1997. “Respondent-driven Sampling: A New Approach to the Study of Hidden Populations.” Social Problems 44(2):174–99.

21.

Heimer

2005. “Critical Issues and Further Questions about Respondent-driven Sampling: Comment on Ramirez-Valles, et al.” AIDS and Behavior 9(4):403–8.

22.

Jing

Wang

Cui

2014. “Estimating the Sizes of Populations at High Risk for HIV: A Comparison Study.” PLoS ONE 9(4):e95601.

23.

Johnston

L. G.

McLaughlin

K. R.

El Rhinila

Latifi

Toufik

Bennani

Alami

Elomari

Handcock

M. S.

2015. “Estimating the Size of Hidden Populations Using Respondent-driven Sampling Data: Case Examples from Morocco.” Epidemiology 26(6):846–52.

24.

Joint United Nations Programme on HIV/AIDS. 2010. “Guidelines on Estimating the Size of Populations Most at Risk to HIV.” Geneva, Switzerland: UNAIDS/WHO Working Group on Global HIV/AIDS and STI Surveillance.

25.

Kadushin

Killworth

P. D.

Bernard

H. R.

Beveridge

A. A.

2006. “Scale-up Methods as Applied to Estimates of Heroin Use.” Journal of Drug Issues 36(2):417–40.

26.

Karon

Wejnert

2012. “Statistical Methods for the Analysis of Time-location Sampling Data.” Journal of Urban Health 89(3):565–86.

27.

Khounigh

A. J.

Haghdoost

A. A.

SalariLak

Zeinalzadeh

A. H.

Yousefi-Farkhad

Mohammadzadeh

Holakouie-Naieni

2014. “Size Estimation of Most-at-risk Groups of HIV/AIDS Using Network Scale-up in Tabriz, Iran.” Journal of Clinical Research and Governance 3(1):21–26.

28.

Killworth

P. D.

Johnsen

E. C.

McCarty

Shelley

G. A.

Bernard

1998. “A Social Network Approach to Estimating Seroprevalence in the United States.” Social Networks 20(1):23–50.

29.

Killworth

P. D.

McCarty

Bernard

H. R.

Shelley

G. A.

Johnsen

E. C.

1998. “Estimation of Seroprevalence, Rape, and Homelessness in the United States Using a Social Network Approach.” Evaluation Review 22(2):289–308.

30.

Killworth

P. D.

McCarty

Bernard

H. R.

Johnsen

E. C.

Domini

Shelly

G. A.

2003. “Two Interpretations of Reports of Knowledge of Subpopulation Sizes.” Social Networks 25(2):141–60.

31.

Killworth

P. D.

McCarty

Johnsen

E. C.

Bernard

H. R.

Shelley

G. A.

2006. “Investigating the Variation of Personal Network Size under Unknown Error Conditions.” Sociological Methods and Research 35(1):84–112.

32.

Kurant

Markopoulou

Thiran

2011. “Towards Unbiased BFS Sampling.” IEEE Journal on Selected Areas in Communications 29(9):1799–1809.

33.

Lavallée

2007. Indirect Sampling. New York: Springer.

34.

Rohe

2015. “Central Limit Theorems for Network Driven Sampling.” arXiv:1509.04704 [math, stat].

35.

Maghsoudi

Baneshi

M. R.

Neydavoodi

Haghdoost

2014. “Network Scale-up Correction Factors for Population Size Estimation of People Who Inject Drugs and Female Sex Workers in Iran.” PLoS ONE 9(11):e110917.

36.

Maltiel

Raftery

A. E.

McCormick

T. H.

Baraff

2015. “Estimating Population Size Using the Network Scale Up Method.” Annals of Applied Statistics 9(3):1247–77.

37.

McCarty

Killworth

P. D.

Bernard

H. R.

Johnsen

Shelley

G. A.

2001. “Comparing Two Methods for Estimating Network Size.” Human Organization 60(1):28–39.

38.

McCormick

Kolaczyk

Zheng

2012. “Surveying Hard-to-reach Groups through Sampled Respondents in a Social Network.” Statistics in Biosciences 4(1):177–95.

39.

McCormick

T. H.

Zheng

2007. “Adjusting for Recall Bias in ‘How Many X’s Do You Know?’ Surveys.” Proceedings of the Joint Statistical Meetings, Salt Lake City, UT.

40.

McCreesh

Frost

Seeley

Katongole

Tarsh

M. N.

Ndunguse

Jichi

Lunel

N. L.

Maher

Johnston

L. G.

. 2012. “Evaluation of Respondent-driven Sampling.” Epidemiology 23(1):138.

41.

Mills

H. L.

Colijn

Vickerman

Leslie

Hope

Hickman

2012. “Respondent Driven Sampling and Community Structure in a Population of Injecting Drug Users, Bristol, UK.” Drug and Alcohol Dependence 126(3):324–32.

42.

Mouw

Verdery

A. M.

2012. “Network Sampling with Memory: A Proposal for More Efficient Sampling from Social Networks.” Pp. 206–56 in Sociological Methodology, vol. 42, edited by Liao

Tim Futing

. Thousand Oaks, CA: Sage.

43.

Nikfarjam

Shokoohi

Shahesmaeili

Haghdoost

A. A.

Baneshi

M. R.

Haji-Maghsoudi

Rastegari

Nasehi

A. A.

Memaryan

Tarjoman

2016. “National Population Size Estimation of Illicit Drug Users through the Network Scale-up Method in 2013 in Iran.” International Journal of Drug Policy 31:147–52.

44.

Paniotto

Petrenko

Kupriyanov

Pakhok

2009. “Estimating the Size of Populations with High Risk for HIV Using the Network Scale-up Method.” Technical Report. Kiev, Ukraine: Kiev Internation Institute of Sociology.

45.

R Core Team. 2014. “R: A Language and Environment for Statistical Computing.” Vienna, Austria: R Foundation for Statistical Computing.

46.

Rao

Yue

1992. “Some Recent Work on Resampling Methods for Complex Surveys.” Survey Methodology 18(2):209–17.

47.

Rao

J. N.

C.F.J.

1988. “Resampling Inference with Complex Survey Data.” Journal of the American Statistical Association 83(401):231–41.

48.

Rohe

2015. “Network Driven Sampling: A Critical Threshold for Design Effects.” arXiv:1505.05461 [math, stat].

49.

Rudolph

A. E.

Fuller

C. M.

Latkin

2013. “The Importance of Measuring and Accounting for Potential Biases in Respondent-driven Samples.” AIDS and Behavior 17(6):2244–52.

50.

Rust

Rao

1996. “Variance Estimation for Complex Surveys Using Replication Techniques.” Statistical Methods in Medical Research 5(3):283–310.

51.

Rwanda Biomedical Center. 2012. “Estimating the Size of Key Populations at Higher Risk of HIV through a Household Survey (ESPHS) Rwanda 2011.” Technical Report. Calverton, MD: RBC/IHDPC, SPF, UNAIDS, and ICF International.

52.

Salganik

M. J.

2012. “Commentary: Respondent-driven Sampling in the Real World.” Epidemiology 23(1):148–50.

53.

Salganik

M. J.

Fazito

Bertoni

Abdo

A. H.

Mello

M. B.

Bastos

F. I.

2011. “Assessing Network Scale-up Estimates for Groups Most at Risk of HIV/AIDS: Evidence from a Multiple-method Study of Heavy Drug Users in Curitiba, Brazil.” American Journal of Epidemiology 174(10):1190–96.

54.

Salganik

M. J.

Mello

M. B.

Abdo

A. H.

Bertoni

Fazito

Bastos

F. I.

2011. “The Game of Contacts: Estimating the Social Visibility of Groups.” Social Networks 33(1):70–78.

55.

Sarndal

C. E.

Swensson

Wretman

1992. Model Assisted Survey Sampling. New York: Springer.

56.

Scott

2008. “‘They Got Their Program, and I Got Mine’: A Cautionary Tale Concerning the Ethical Implications of Using Respondent-driven Sampling to Study Injection Drug Users.” International Journal of Drug Policy 19(1):42–51.

57.

Scutelniciuc

2012a. “Network Scale-up Method Experiences: Republic of Kazakhstan.” Presented at Consultation on Estimating Population Sizes through Household Surveys: Successes and Challenges, March 28–30, New York.

58.

Scutelniciuc

2012b. “Network Scale-up Method Experiences: Republic of Moldova.” Presented at Consultation on Estimating Population Sizes through Household Surveys: Successes and Challenges, March 28–30, New York.

59.

Sheikhzadeh

Baneshi

M. R.

Afshari

Haghdoost

A. A.

2014. “Comparing Direct, Network Scale-up, and Proxy Respondent Methods in Estimating Risky Behaviors among Collegians.” Journal of Substance Use 21(1):9–13.

60.

Shelley

G. A.

Bernard

H. R.

Killworth

Johnsen

McCarty

1995. “Who Knows Your HIV Status? What HIV+ Patients and Their Network Members Know about Each Other.” Social Networks 17(3–4):189–217.

61.

Shelley

G. A.

Killworth

P. D.

Bernard

H. R.

McCarty

Johnsen

E. C.

Rice

R. E.

2006. “Who Knows Your HIV Status II? Information Propagation within Social Networks of Seropositive People.” Human Organization 65(4):430–44.

62.

Shokoohi

Baneshi

M. R.

Haghdoost

A. A.

2012. “Size Estimation of Groups at High Risk of HIV/AIDS Using Network Scale Up in Kerman, Iran.” International Journal of Preventive Medicine 3(7):471–76.

63.

Sirken

M. G.

1970. “Household Surveys with Multiplicity.” Journal of the American Statistical Association 65(329):257–66.

64.

Snidero

Morra

Corradetti

Gregori

2007. “Use of the Scale-up Methods in Injury Prevention Research: An Empirical Assessment to the Case of Choking in Children.” Social Networks 29(4):527–38.

65.

Snidero

Soriani

Baldi

Zobec

Berchialla

Gregori

2012. “Scale-up Approach in CATI Surveys for Estimating the Number of Foreign Body Injuries in the Aero-digestive Tract in Children.” International Journal of Environmental Research and Public Health 9(11):4056–67.

66.

Snidero

Zobec

Berchialla

Corradetti

Gregori

2009. “Question Order and Interviewer Effects in CATI Scale-up Surveys.” Sociological Methods and Research 38(2):287–305.

67.

Sulaberidze

Mirzazadeh

Chikovani

Shengelia

Tsereteli

Gotsadze

2016. “Population Size Estimation of Men Who Have Sex with Men in Tbilisi, Georgia: Multiple Methods and Triangulation of Findings.” PLoS ONE 11(2):e0147413.

68.

Tillé

Matei

2015. “Sampling: Survey Sampling.” R package version2.7.

69.

Volz

Heckathorn

D. D.

2008. “Probability-based Estimation Theory for Respondent-driven Sampling.” Journal of Official Statistics 24(1):79–97.

70.

Wang

Yang

Zhao

Chen

Zhang

2015. “Application of Network Scale Up Method in the Estimation of Population Size for Men Who Have Sex with Men in Shanghai, China.” PLoS ONE 10(11):e0143118.

71.

Wasserman

Faust

1994. Social Network Analysis. New York: Cambridge University Press.

72.

Wesson

Handcock

M. S.

McFarland

Raymond

H. F.

2015. “If You Are Not Counted, You Don’t Count: Estimating the Number of African-American Men Who Have Sex with Men in San Francisco Using a Novel Bayesian Approach.” Journal of Urban Health 92(6):1052–64.

73.

White

H. C.

Boorman

S. A.

Breiger

R. L.

1976. “Social Structure from Multiple Networks. I. Blockmodels of Roles and Positions.” American Journal of Sociology 81(4):730–80.

74.

White

R. G.

Hakim

A. J.

Salganik

M. J.

Spiller

M. W.

Johnston

L. G.

Kerr

L. R.

Kendall

Drake

Wilson

Orroth

. 2015. “Strengthening the Reporting of Observational Studies in Epidemiology for Respondent-driven Sampling Studies: ‘STROBE-RDS’ Statement.” Journal of Clinical Epidemiology 68(12):1463–71.

75.

Wickham

2009. “ggplot2: Elegant Graphics for Data Analysis.” New York: Springer.

76.

Wickham

2011. “The Split-apply-combine Strategy for Data Analysis.” Journal of Statistical Software 40(1):1–29.

77.

Wickham

2012. “stringr: Make It Easier to Work with Strings.” R package version 0.6.2.

78.

Wickham

Chang

2013. “devtools: Tools to Make Developing R Code Easier.” R Package Version 1.4.1.

79.

Yamanis

T. J.

Merli

M. G.

Neely

W. W.

Tian

F. F.

Moody

Gao

2013. “An Empirical Analysis of the Impact of Recruitment Patterns on RDS Estimates among a Socially Ordered Population of Female Sex Workers in China.” Sociological Methods and Research 42(3):392–425.

80.

Zheng

Salganik

M. J.

Gelman

2006. “How Many People Do You Know in Prison? Using Overdispersion in Count Data to Estimate Social Structure in Networks.” Journal of the American Statistical Association 101(474):409–23.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

4.27 MB

Generalizing the Network Scale-up Method

Abstract

Keywords

1. Introduction

2. The Generalized Scale-Up Estimator

2.1. Estimating N H from Sampled Data

3. Comparison Between the Generalized and Basic Scale-Up Approaches

3.1. Illustrative Simulation

4. Recommendations for Practice

4.1. Estimation with Samples from F and H

4.2. Estimation with Only a Sample from F

5. Conclusion and Next Steps

Footnotes

Acknowledgements

Funding

Notes

Author Biographies

References

Supplementary Material

2.1. Estimating $N_{H}$ from Sampled Data

4.1. Estimation with Samples from $F$ and $H$

4.2. Estimation with Only a Sample from $F$