Methods for Improving Representativeness in a Web Survey on Sexual Assault Among College Students

Abstract

Many colleges and universities conduct web-based campus climate surveys to understand the prevalence and nature of sexual assault among their students. When designing and fielding a web survey to measure a sensitive topic like sexual assault, methodological decisions, including the length of the field period and the use or amount of an incentive, can affect the representativeness of the respondent sample leading to biased or imprecise estimates. This study uses data from the Campus Climate Survey Validation Study (CCSVS) to assess how the interaction between field period length and survey incentive amount affects nonresponse, sample representativeness, and the precision of survey estimates. Research suggests that using robust incentives gives potential survey respondents a reason to complete the survey beyond their intrinsic motivation to do so. Likewise, extending the field period gives more time to people who may be less intrinsically motivated to complete the survey. Both serve to increase sample size and representativeness, minimize bias, and improve estimate precision. Schools, however, sometimes lack the time and/or resources for both a robust incentive and a lengthy field period, and this study examines the extent to which the potential negative impacts of not using one can be mitigated by the presence of the other. Findings indicate that target response rates can be achieved using a smaller incentive if the field period is lengthy but, even with a lengthy field period, the use of a smaller incentive can result in biased estimates due to a lack of representativeness. Conversely, when a robust incentive is used and weights are developed to adjust for nonresponse, a shorter field period will not have a significant impact on point estimates, but the estimates will be less precise due to fewer respondents participating in the survey.

Keywords

web surveys college students sexual assault incentives field period topic salience bias nonresponse bias

Introduction

Web-based surveys can be an inexpensive and quick method to gather information from known populations such as college students (Mitra, Jain-Shukla, Robbins, Champion, & Durant, 2008). Because a web-based platform allows the respondent to take the survey in private at a convenient time (Sax, Gilmartin, & Bryant, 2003), web-based surveys are also ideal for measuring sensitive topics, such as sexual assault. For these reasons, colleges and universities across the country use web-based surveys to collect data from students on a variety of topics: most notably, the prevalence and nature of sexual assault among their students (see, for example, Axinn, Wagner, Couper, & Crawford, 2018; Cantor et al., 2015; Krebs et al., 2016).

However, the ease and other benefits of web-based surveys have resulted in college students being inundated with survey opportunities (Fosnacht, Sarraf, Howe, & Peck, 2017). Because of this over-surveying of the student population, some students only respond to the surveys in which the topic is most salient to them (Parsons & Manierre, 2014; Saleh & Bista, 2017). This salience effect leads to concern about biased samples and estimates, particularly for surveys on sensitive topics, such as sexual assault (Axinn et al., 2018; Cantor et al., 2015). Thus, to minimize the potential for bias in the sample and the resulting survey estimates, a successful web-based survey must ensure that respondents are representative of the population for which inferences will be made (Cook, Heath, & Thompson, 2000).

Two of the more common approaches used by survey designers to increase the representativeness of a sample and reduce the potential for topic salience bias are offering less responsive respondents plenty of time to complete the survey (i.e., using a lengthy field period) and providing a monetary incentive for participating. Although these two techniques have been examined individually in the literature (see, for example, Mitra et al., 2008, or Coopersmith, Vogel, Bruursema, & Feeney, 2016), the interaction between the two methods has not been explored. This is important because, whereas robust incentives and a long field period length may be ideal, schools are often faced with less than ideal time and resource constraints for conducting a campus climate survey. Therefore, it is important to understand whether one approach can be used to mitigate the impacts of not having the other.

This study examines the interaction and impact of survey incentive amount and field period length on response rates, sample representativeness, survey estimates, and standard errors in the context of a survey on sexual assault among college students. First, it examines how incentive amount affects response rates and sample representativeness, and whether a longer field period can mitigate the impact of using a lower survey incentive amount. Second, it assesses whether the potential bias and lack of precision that often result from a shorter field period can be ameliorated through using a robust incentive and by accounting for known characteristics of the population of interest. These objectives are accomplished through the use of an experiment with different survey incentive amounts, and a simulation of the impact of different field period lengths, among a large sample of college students who participated in a survey about their experiences with sexual assault.

Literature Review

Researchers working in the area of campus sexual assault have long recognized the potential impact that methodological decisions can have on survey estimates of the prevalence and incidence of victimization. Sexual assault prevalence rates can vary widely from one school, and one study, to the next, and the differences are often at least partly attributed to methodological differences in how the studies are designed and conducted (Fedina, Holmes, & Backes, 2018; Krebs et al., 2017; Mellins et al., 2017; Muehlenhard, Peterson, Humphreys, & Jozkowski, 2017; Rennison & Addington, 2014).

Part of the reason that methodological decisions matter so much in campus climate surveys, particularly web-based surveys where no interviewer is present, is because surveys on sensitive topics are particularly vulnerable to topic salience bias (Brown, Decker, & Connelly, 1989; Roster, Albaum, & Smith, 2017; Tourangeau, Groves, & Redline, 2010). Topic salience bias occurs when only those persons for whom the survey topic is most relevant respond to the survey. In the case of campus climate surveys, topic salience bias would typically mean that students who have some direct or indirect experience with sexual victimization are more likely to complete the survey than nonvictims, thus resulting in a sample that is not representative of the population of interest and inflated estimates of the prevalence of sexual assault (Axinn et al., 2018; Cantor et al., 2015). While it is possible that surveys on extremely sensitive or traumatic topics, such as sexual assault, may suppress victim reporting due to the traumatic nature of recalling the event, the studies to date which have examined this topic (e.g., Axinn et al., 2018; Cantor et al., 2015) have found the net impact of topic salience bias to increase estimates.

However, when considering how salience influences a person’s desire to respond to a survey, one must also consider the other possible factors of influence. This holistic approach is called leverage-saliency theory (Groves, Singer, & Corning, 2000). Under leverage-saliency theory, all potential factors affecting a person’s decision to participate in a survey are considered. Beyond interest in the topic itself, these factors can include a monetary incentive, a sense of civic duty, or marketing from a person of importance such as a school president (Fan & Yan, 2010).

The risk of topic salience bias in a survey of sexual assault makes one of the benefits of web-based surveys—the speed at which data can be collected—a potential liability. Mitra et al. (2008) found that among college students, at least 70% (and up to 90%) of surveys were completed within the first 2 weeks of a 2-month field period. If these “early responders” are more interested in the survey topic, and, therefore, respond differently than “late responders,” the survey estimates will be biased because there is a disproportionate number of “early responders.”

Several other studies have looked at the relationship between an individual’s interest in a survey topic and their patterns of response during the field period. Studies on teacher satisfaction (Green, 1991), patient satisfaction (Gadkari, McHorney, Pedan, & Gowda, 2011; Paganini-Hill, Hsu, Chao, & Ross, 1993; Yessis & Rathert, 2006), and task-based questions (Sauro, 2015) found that the more salient populations—satisfied teachers, satisfied patients, and persons who prefer certain tasks, respectively—were more likely to respond early. On the contrary, studies on job satisfaction (Borg et al., 2008) and grant application usage (Welch & Barlau, 2013) found no differences between early and late responders. None of these studies, however, assessed topic salience bias using a sensitive topic like sexual assault.

Over the past decade, the survey methodological field has moved away from relying solely on response rates as an indicator of whether a sample is biased toward using representativeness as a critical criterion (Cook et al., 2000). Representativeness, in terms of nonresponse bias, needs to be based on characteristics correlated to the outcome of interest (Groves, 2006; Groves & Peytcheva, 2008). Fosnacht et al. (2017) examined response rates across a set of web-based surveys of college students and found the number of respondents was a better indication of representativeness than response rate. As such, a higher response rate is needed in smaller schools than in larger schools only because the required number of respondents to have a representative sample is a larger percentage of the student body.

Incentives are often used to improve response rates and the representativeness of the sample and are believed to encourage respondents to participate even when the survey topic may not have salience for them (Saleh & Bista, 2017). Incentives can be administered in two ways: prepaid or postpaid. A prepaid incentive is given to all persons in the sample whether they ultimately respond or not. Prepaid incentives are also called “token” incentives because they are usually for a nominal amount (e.g., US$1 or US$2). A postpaid or “promised” incentive is given only to those persons in the sample who complete a survey. Prepaid incentives have been shown to increase response in mail surveys (Mercer, Caporaso, Cantor, & Townsend, 2015), but these findings have not translated to web-based surveys (Bosnjak & Tuten, 2003; Coopersmith et al., 2016; Goritz, 2006). In fact, Coopersmith et al. (2016) found that, among college students, prepaid incentives led to less representative samples. This may be because for web-based surveys, unlike mail surveys, sampled persons may not open or read all email invitations, thus missing the incentive (Goritz, 2006). Alternatively, Coopersmith et al. (2016) found postpaid incentives to be effective at obtaining a representative sample. It should be cautioned that incentives, especially nonmonetary ones, have been shown to sometimes have negative effects on bias. Merkle, Edelman, Dykeman, and Brogan (1998) found providing a pen as an incentive increased the response propensities of Democrats compared with Republicans leading to a less representative sample. Moreover, the impact of the postpaid incentive has not been found to be linearly associated with the amount of the incentive. Porter and Whitcomb (2003) found, in a lottery-based postpaid incentive among high school students, amounts ranging between US$50 and US$200 did not appreciably yield different response rates.

Research Questions

Prior research suggests that web-based surveys using a small or negligible incentive can lead to a less representative sample and less precise estimates due to topic salience bias and lower responses rates. Similarly, a short survey field period can have the same negative impacts. In the context of a campus climate survey to measure the prevalence of sexual assault, this study examines the interaction between survey incentive amount and field period length and begins to address the question of whether one can mitigate the impact of the other. In other words, the research questions are as follows:

Research Question 1: How does the interaction of field period and incentive amount affect response rates and bias?

Research Question 2: Can a robust incentive and proper weighting ameliorate the impact of a shorter field period in terms of sample representativeness and precision?

Method

Data Source

The data for this study are from the Campus Climate Survey Validation Study (CCSVS; Krebs et al., 2016), which was sponsored by the U.S. Bureau of Justice Statistics (BJS) and the Office on Violence against Women (OVW) and conducted by RTI International during Spring 2015. The goal of the CCSVS was to measure the rate of sexual assault during the current (2014-2015) academic year and students’ perceptions of the climate related to sexual assault. The CCSVS was administered at nine postsecondary institutions of varying sizes, levels of institutional control (public, private not-for-profit), and geographic locations. The target population within each school was all degree-seeking, undergraduate students enrolled in the Spring 2015 semester. The survey took about 15-min, on average, to complete and was administered through the web, accessible on a computer, tablet, or smartphone.

Schools provided a student roster which included the names and email addresses for all undergraduate students enrolled during the spring semester, as well as demographic and other characteristics (e.g., year of study, academic performance). The characteristics provided were chosen because they may be correlated with a student’s propensity to respond. Within each school, a target number of undergraduate male and female respondents was set to achieve the desired precision goals for the study, and a stratified (by sex) simple random sample of students was selected. Each sampled student was emailed an initial invitation to participate, as well as several follow-up emails during the field period. To minimize topic salience bias, a neutral field name was used for the study (the College Experiences Survey) and a very general description of the survey topics was included in the recruitment email and survey home page; more specific information about the nature of the questions was provided on the informed consent screen after students logged into the survey website. However, even with these methods, once the student learned of the survey topic, there is a risk that topic salience influenced their decision on whether to participate. Each email notification informed the students that, upon completion of the survey, they would receive a code for an electronic gift card (in the amount of US$10, US$25, or US$40, described in more detail below) from a choice of nine retailers or restaurants.

For each school, the field period lasted from the week after the school’s spring break to the start of spring semester finals. This led to field period lengths ranging from 42 to 60 days across the nine schools. As can be seen in Figure 1, eight of the nine schools achieved their target sample size within 14 to 26 days of data collection. However, data collection was kept open after the target sample size was achieved, enabling assessment of how late responders and early responders differed. After data collection, postsurvey adjustments for nonresponse and coverage were conducted using the auxiliary information provided by the school and available for all students invited to participate in the survey.

Figure 1.

Percentage of desired interviews among females completed by school and day of data collection (CCSVS, 2015).

Across the nine schools, approximately 23,000 responses were collected from 14,989 females and 8,034 males. This analysis focused on the female respondents because the outcome of interest is the prevalence of sexual assault and higher prevalence rates among females enables a better assessment of the research questions addressed in this study. Sexual assault was measured using behaviorally specific survey questions; after providing a detailed description of various types of sexual contact, students were asked whether, since the beginning of the 2014-2015 academic year, they had experienced sexual contact that they did not consent to and did not want to happen. The measure of sexual assault used in the CCSVS includes both rape (unwanted sexual contact involving penetration) and sexual battery (unwanted sexual contact that did not involve penetration, such as touching or grabbing of sexual body parts).

Analysis Plan

The first analysis uses the results of an experiment to look at the impact of different survey incentive amounts on response rates and survey estimates and explores whether a longer field period can mitigate the impact of using a lower survey incentive. The second analysis focuses on whether a robust survey incentive and proper weighting can mitigate the impact of having a shorter field period.

How does the interaction of field period and incentive amount affect response rates and bias?

To understand how survey incentive amount affects the potential for bias in the estimates, a randomized incentive experiment was conducted in four of the CCSVS schools.¹ In two schools, students were randomized to receive either a US$10 or US$25 incentive amount. In the other two schools, students were randomized to receive either a US$25 or US$40 incentive amount. The randomization was done so that half of sampled students were assigned to each incentive amount.

The impact of different incentives across a field period ranging from 0 to 60 days was evaluated through the cumulative completion rate and the sexual assault victimization rate over time. The cumulative completion rate was based on the target sample size for each of the schools, meaning that for each incentive group, the target was half the total target sample size for the school. The sexual assault victimization rate was calculated for each incentive group after each day of data collection and reflects the percentage of respondents who reported experiencing sexual assault victimization during the 2014-2015 academic year.

Can a robust incentive and proper weighting ameliorate the impact of a shorter field period on sample representativeness and estimate precision?

To determine whether the potential nonresponse and topic salience bias resulting from a short field period can be ameliorated with incentives and weighting, different survey field period scenarios were simulated among the seven schools where an incentive of US$25 or more was offered to all students. First, three survey field periods were created based on the respondent’s completion date: (a) 21 days or less from the start of data collection, (b) 28 days or less from the start of data collection, and (c) 60 days or less from the start of data collection.² These field period times were selected to mimic the field lengths of other campus climate surveys, up to the maximum period a campus climate study could reasonably be fielded. For example, Cantor et al. (2015) used a 21-day field period. Under each field period scenario, any student who had not completed the survey by the simulated last day of the field period was treated as a nonrespondent. Response rates for each scenario were computed to illustrate the improvement in participation due to the longer field period.

Next, survey weights accounting for nonresponse were created for each scenario. The weights were based on student characteristics from the school-provided rosters of enrolled undergraduates. Schools provided all or some of the student characteristics³:

• Sex	• Race
• Hispanicity	• Transfer status
• On-campus living status	• Grade point average (GPA)
• SAT/ACT score	• Year of study
• Age

Calibration was used to adjust the design-based survey weights (i.e., the inverse probability of selection) for differential nonresponse across the school-provided characteristics (Kott, 2006). For each scenario, the calibrated weights were used to calculate sexual assault victimization prevalence estimates and standard errors by school and averaged across all schools. This allowed for an evaluation of whether estimates and the precision of the estimates differed across each scenario.

Using the weighted estimates, relative standard errors (RSEs; i.e., the standard error as a percentage of the point estimate) were computed for each field period scenario, for each school, and across all schools. Changes in the RSEs across each field period scenario serve as a measure of the extent to which a robust incentive mitigates the precision concerns associated with a shorter field period length.

In addition, based on the calibrated weights for each scenario, the design effect due to the unequal weighting effect (UWE; Kish, 1987) was calculated. The UWE is defined as 1 + CV² where CV is the coefficient of variation among the nonresponse adjusted weights. Because the CCSVS utilized a simple random sample, the UWE will be 1 if the final respondent sample is perfectly representative of the starting sample. As the UWE increases, it indicates a divergence from the proportional distribution expected under a simple random sample. In other words, a larger UWE means that the respondent sample is less representative of the population than the starting sample (which does represent the population).

Results

How Does the Interaction of Field Period and Incentive Amount Affect Response Rates and Bias?

Figures 2 and 3 present the cumulative completion rate for the schools in the US$25 versus US$10 incentive experiment and the US$25 versus US$40 incentive experiment.⁴ The first figure shows that the difference in completion rates between the US$25 and US$10 incentive samples accelerated throughout the field period, with the rate for the US$25 sample continuing to climb above the rate for the US$10 sample. If the survey had been stopped after 3 weeks (21 days), the US$25 incentive sample would have been at the desired completion rate, whereas the US$10 incentive sample would have been considerably below, at about 86% of the desired completion rate. However, by the 4-week mark (28 days), the US$10 incentive sample also reached the desired completion rate. Ultimately, by the end of the field period, the US$25 incentive sample resulted in a completion rate that was statistically significant 25% higher than the US$10 incentive sample (162.2% of target for US$25 schools and 137.5% of target for US$10 schools). However, the findings show that with a lower incentive, the desired completion rate can still be met or exceeded if the survey is left in the field for a sufficient period of time.

Figure 2.

Cumulative completion rate among female students by time in field and US$25 and US$10 incentive amounts.

Figure 3.

Cumulative completion rate among female students by time in field and US$25 and US$40 incentive amounts.

Figure 3, showing the results from the US$25 versus US$40 incentive experiment, suggests that increasing the sample above US$25 does not result in a statistically significant difference in completion rate by field period length. The sample reached the desired completion rate in 20 days with the US$40 incentive, compared with 21 days with the US$25 incentive. At the end of the field period, there was no statistically significant difference in the cumulative completion rate for the two samples (124.3% of target for US$25 schools and 130.0% of target for US$40 schools).

Together, the two experiments suggest that a longer field period can help ensure that the desired survey completion rate is reached when a lower incentive is used. However, with larger incentives, a longer field period does not significantly affect the number of completed interviews.

Figures 4 and 5 present the sexual assault victimization prevalence rates over the field period for the US$25 versus US$10 incentive experiment and the US$25 versus US$40 incentive experiment. Figure 4 shows that in the first 7 days of the field period, the sexual assault victimization prevalence rates were very similar at 10.7% for the US$25 incentive sample and 11.6% for the US$10 incentive sample. This could indicate that victims who are interested in the topic initially respond at similar rates, regardless of the incentive amount. However, throughout the field period, the sexual assault victimization rate for US$25 incentive sample steadily declined, whereas the rate for the US$10 sample increased before flattening out. By the end of the field period, the difference between the two incentive groups was statistically significant, with the US$10 incentive respondents having a 9.4% prevalence rate compared with 7.7% for the US$25 incentive respondents. This higher prevalence rate among the lower incentive sample may be indicative of topic salience bias; namely, victims are more likely to be interested in the topic and respond to the survey than nonvictims. A higher incentive is seemingly more effective at encouraging completion among nonvictims and mitigating the salience effect, regardless of the length of the field period.

Figure 4.

Sexual assault prevalence rates among female students by time in field and US$25 incentive and US$10 incentive amount.

Figure 5.

Sexual assault prevalence rates among female students by time in field and US$25 incentive and US$40 incentive amount.

At schools with the US$25 versus US$40 incentive experiment, the sexual assault victimization prevalence rates were significantly different for the two samples throughout the field period (Figure 5). The prevalence rate remained relatively stable for the US$40 incentive sample throughout the period at about 9%, suggesting that even in the initial days of data collection, a US$40 incentive largely mitigates any topic salience bias. Among the US$25 incentive sample, the prevalence rate started at 6.8% but as the field period process progressed, the prevalence rate converged toward the US$40 incentive sample sexual assault victimization rate. Despite the convergence, the final rates remained statistically different (with higher victimization rates in the US$40 incentive condition than the US$25 incentive condition). This finding goes counter to the expectation that the lower incentive amount would yield a higher victimization rate but is in line with the literature (e.g., Porter and Whitcomb, 2003) which finds diminishing returns in the influence of a monetary incentive to increase participation among nonsalient persons when the incentive is more than US$25 or US$30.

Can a Robust Incentive and Proper Weighting Ameliorate the Impact of a Shorter Field Period on Sample Representativeness and Estimate Precision?

Figure 6 presents the response rates by field period length for females at the seven schools where a robust incentive of US$25 or more was offered to all students. The findings suggest that even when offering a robust incentive, extending the field period produces additional benefits in terms of response rates. Specifically, the combined school response rate increased as the field period increased, from 44.9% at 21 days to 49.1% at 28 days to 55.9% after the full field period (~60 days). This trend was found at each individual school as well. The number of respondents increased 24.6%, on average, from the 21-day field period to the full 60-day field period.

Figure 6.

Response rates among female respondents by school and field period scenario.

Figure 7 presents the sexual assault prevalence rates by field period length for female students at the seven schools where an incentive of US$25 or more was offered. The prevalence rates for each field period were weighted to represent the full population of undergraduate females at each school. The combined prevalence rate and the individual rates for the seven schools were fairly consistent across the three field periods. Similar to the findings from the first part of the analysis, this lack of difference suggests that with a robust incentive and proper weighting, it may be possible to keep the survey in the field for as little as 3 weeks without major concerns about the potential impact of topic salience bias on the victimization prevalence estimates.

Figure 7.

Prevalence rate (weighted) of sexual assault among female undergraduate students by school and field period scenario.

It is also important, however, to consider the impact of a shorter period on the precision of estimates. Table 1 shows the RSEs of the prevalence estimates by school and field period length. Although the estimates did not change significantly across the three field period scenarios, for all seven schools where the robust incentive was offered (US$25 or US$40), the precision of the estimates improved with the longer field period. Improved precision enables researchers to better detect differences in rates among subpopulations and across schools.

Table 1.

Relative Standard Errors for Sexual Assault Victimization Prevalence Rates Among Female Undergraduate Students by School and Field Period Scenario.

School	21 days (%)	28 days (%)	60 days (%)
All schools	3.3	3.2	2.8
1	5.9	5.4	3.7
2	16.2	15.0	14.4
3	7.5	7.4	6.9
5	5.2	4.4	3.8
6	6.7	6.2	5.5
7	5.4	5.0	4.5
9	8.1	7.1	6.4

Finally, Table 2 presents the school-level UWEs for each field period scenario. Across all schools except one,⁵ the UWE gets closer to 1 with each longer field period scenario. This suggests that, while the impact of a shorter field period on point estimates can be mitigated with weighting, bringing additional students into the survey through the longer field period improved the representativeness of the sample. The improvement in the sample representativeness did vary across the schools though. For instance, in School 1, a 60-day field period compared with a 21-day field period yielded a sample that was 32% less variable than the original population; however, School 5 only saw a 5% increase in sample representativeness in a 60-day field period compared with a 21-day field period.

Table 2.

Design Effects Due to Unequal Weighting by School and Field Period Length.

School	21 days	28 days	60 days
1	1.464	1.384	1.112
2	1.082	1.076	1.190
3	1.010	1.009	1.006
5	1.145	1.113	1.091
6	1.206	1.179	1.141
7	1.067	1.061	1.048
9	1.152	1.134	1.118

Discussion

Web-based surveys have several advantages over other survey modes, including ease of communication with the sample, lower cost, and flexibility in terms of when and how respondents take the survey (Saleh & Bista, 2017). Furthermore, web surveys have been shown to minimize social desirability bias which occurs when a respondent provides an answer more in line with what they perceive the social norm to be rather than their true opinion (Burkill et al., 2016; Heerwegh, 2009; Kreuter, Presser, & Tourangeau, 2008; Pew Research Center, 2015). The benefits of web-based surveys make them ideal for surveys on sensitive topics such as sexual assault. As such, many colleges and universities have administered web-based campus climate surveys with the goal of measuring the rate and nature of sexual assault among their student body (see Axinn et al., 2018; Cantor et al., 2015; Krebs et al., 2016). However, web-based surveys may be particularly vulnerable to salience bias whereby sampled persons are more likely to respond if the survey is of interest or relevance to them.

This study examined two methods used to mitigate the salience effect—a survey incentive for completing the survey and an extended field period. Unlike other studies on these topics, this study examined the interaction of these two methods to determine whether the use of one could mitigate the impact of not having the other or whether using the two methods in tandem is necessary for an optimal study design. Findings showed that with a lower incentive of US$10, the desired completion rate can still be met or exceeded if the survey is left in the field for a longer period of time (4 weeks or more). However, even with an extended field period, victimization rates were inflated among the lower incentive sample, suggesting a topic salience effect on the estimates. Increasing the incentive above US$25 did not have an appreciable impact on the fielding time needed to hit target completion rates but did result in more stable victimization estimates across the duration of the field period. These findings were consistent with Coopersmith et al. (2016), who found that a postpaid incentive can help ensure a more representative sample, and with Porter and Whitcomb (2003), who found that an excessive incentive did not appreciably improve the representation. However, our findings extend the existing literature by showing that not only does the amount of the incentive matter, up to a point, but also that a longer field period with a larger incentive is ideal for minimizing the potential for a salience effect. In addition, our findings found the US$40 incentive did produce a statistically larger victimization rate compared with the US$25 incentive—which is counter to the literature, but we did not find differences in the representativeness of the two samples.

Research Question 2 examined whether a robust incentive and proper weighting could ameliorate the salience bias associated with using a shorter field period. Although this was largely found to be true, a longer field period still resulted in more representative and precise estimates. This is likely due to the conclusion that other researchers have reached—that the number of respondents is more important than a particular response rate (Fosnacht et al., 2017). After about 60 days in the field, a representative sample was achieved, as evidenced by the UWEs near 1 for all schools. However, because multiple student population characteristics could be used in a nonresponse adjustment, the combination of using a robust survey incentive and proper weighting produced statistically similar estimates of sexual assault in both short (21-day) and long (60-day) field periods. This finding suggests that having the right covariates when weighting your respondent sample can help moderate a salience effect. To better inform schools about which covariates are most correlated with sexual assault, future analysis should assess how different combinations of student characteristics affect the prevalence rates.

Furthermore, though not explicitly stated, efforts to improve the representativeness of climate survey samples will ensure that the survey data reflect the entire student body at a given campus, including traditionally underrepresented subpopulations. This, in turn, will allow for subgroup estimates to be developed for vulnerable populations, which can be used to inform prevention and response efforts at the school. Schools are particularly interested in understanding how sexual assault rates and the experiences of victims differ among minority groups, such as minority racial and ethnic groups, LGBTQ+ persons, and disabled persons. The finding that using an incentive and longer field period length helped ensure that students in minority groups were properly represented in our sample will be of use and interest to schools embarking on similar sexual assault research.

Although these findings provide survey designers with important information on how to best coordinate the use of incentives and field period length, there are other methods not examined in this study that have also been found to improve sample representativeness and reduce bias. For example, Fan and Yan (2010) found that emails from persons of authority (e.g., university presidents) increased the likelihood of participation among persons who were not otherwise interested in the survey. Griggs et al. (2018) found that among college students, a personalized greeting (e.g., using the student’s first name) yielded a more representative respondent sample than a generic greeting (e.g., “Dear student”). Other research has found lotteries to be an effective form of postpaid incentives when individual incentives are cost-prohibitive (Bosnjak & Tuten, 2003; Goritz, 2006; Porter & Whitcomb, 2003). Future research should examine the potential interactions of these other design features for mitigating bias and improving sample representativeness. Additional future research can include a better understanding of how each student characteristics used to produce the survey weights affects the estimates for sexual assault. This will help schools better know which characteristics are critical for including in the frame building and sampling process.

Conclusion

Findings from this research suggest that representative samples and unbiased estimates can be achieved in web-based campus climate surveys when researchers consider the impact of field period length, incentives, and proper weighting and design their studies accordingly. Although a longer field period can mitigate the impact of a lower incentive and a robust incentive and proper weighting can mitigate the impact of a shorter field period, the combination of using a robust survey incentive (e.g., US$25) and long field period (45-60 days) provides the best protection against topic salience bias.

Footnotes

Acknowledgements

The authors would like to thank Bureau of Justice Statistics (BJS) and the Office on Violence against Women (OVW) for sponsoring this research. However, we would like to note that the views expressed in this presentation are those of the authors only and do not reflect the views or position of BJS or the Department of Justice.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding was paid by the Department of Justice, Office of Justice Programs through cooperative agreement 2011-NV-CX-K068.

ORCID iD

Marcus E. Berzofsky

Notes

Author Biographies

Marcus E. Berzofsky, DrPH, is a senior research statistician at RTI International. His research focuses on survey statistics and survey methodology emphasizing the need to minimize the different error sources which could bias survey estimates. He is interested in evaluating methods to determine the optimal approach to interviewing populations on sensitive topics such as sexual assault and crime victimization.

Lynn Langton, PhD, is a senior research criminologist at RTI International. Her research focuses broadly on victimization, victim services, sexual violence, hate crime, financial fraud and white-collar crime, police–community relations, and survey methodology. She is interested in evaluating the effectiveness of services for victims of crime, efforts to prevent victimization and repeat victimization and optimal approaches to collecting data on sensitive topics.

Christopher Krebs, PhD, is a chief scientist in the Division for Applied Justice Research at RTI International. He has extensive research experience in the areas of corrections, intimate partner violence and sexual violence, and criminal justice systems. He has employed both quantitative and qualitative methods in his research and has extensive experience designing studies, developing survey instruments, evaluating programs, analyzing data, and disseminating findings.

Christine Lindquist, PhD, is a program director in the Division for Applied Justice Research at RTI International. Her research interests and areas of expertise include sexual assault among university women and men, campus climate related to sexual misconduct, and workplace sexual harassment. She has played key roles on several federally funded campus climate surveys and the Office on Violence against Women (OVW)-funded Campus Grantee Training and Technical Assistance project.

Michael Planty, PhD, is the director for the Center for Community Safety and Crime Prevention at RTI International. He has led several projects for the U.S. Department of Justice and plays a key role in several studies related to school safety and national incident-based crime statistics.

References

Axinn

Wagner

Couper

Crawford

(2018). Campus climate surveys of sexual misconduct: Limiting the risk of nonresponse bias (Population Studies Center Research Report 18-887). Retrieved from https://www.psc.isr.umich.edu/pubs/abs/14592

Borg

Braun

Baumgartner

M. K.

(2008). Attitudes of demographic item non-respondents in employee surveys. International Journal of Manpower, 29(2), 146-160. doi:10.1108/01437720810872703

Bosnjak

Tuten

T. L.

(2003). Prepaid and promised incentives in web surveys. Social Science Computer Review, 21, 208-217.

Brown

T. L.

Decker

D. J.

Connelly

N. A.

(1989). Response to mail surveys on resource-based recreation topics: A behavioral model and an empirical analysis. Leisure Sciences, 11, 99-110. doi:10.1080/01490408909512210

Burkill

Copas

Couper

M. P.

Clifton

Prah

Datta

Erens

(2016). Using the web to collect data on sensitive behaviours: A study looking at mode effects on the British national survey of sexual lifestyles and attitudes. PLoS ONE, 11(2), e0147983. doi:10.1371/journal.pone.0147983

Cantor

Fisher

Chibnall

Townsend

Lee

Bruce

Thomas

(2015). Report on the AAU campus climate survey on sexual assault and sexual misconduct. Retrieved from https://www.aau.edu/uploadedFiles/AAU_Publications/AAU_Reports/Sexual_Assault_Campus_Survey/AAU_Campus_Climate_Survey_12_14_15.pdf

Cook

Heath

Thompson

R. L.

(2000). A meta-analysis of response rates in web- or internet-based surveys. Educational and Psychological Measurement, 60, 821-836. doi:10.1177/00131640021970934

Coopersmith

Vogel

L. K.

Bruursema

Feeney

(2016). Effects of survey amount and type of survey response rates. Survey Practice, 9, 1-13.

Fan

Yan

(2010). Factors affecting response rates of web surveys: A systematic review. Computers in Human Behavior, 26, 132-139.

10.

Fedina

Holmes

J. L.

Backes

B. L.

(2018). Campus sexual assault: A systematic review of prevalence research from 2000 to 2015. Trauma, Violence, & Abuse, 19, 76-93.

11.

Fosnacht

Sarraf

Howe

Peck

L. K.

(2017). How important are high response rates for college surveys? The Review of Higher Education, 40, 245-265. doi:10.1353/rhe.2017.0003

12.

Gadkari

McHorney

Pedan

Gowda

(2011). Non-response bias in a survey assessing patients’ medication and health beliefs. Value in Health, 14(3), A149.

13.

Goritz

A. S.

(2006). Incentives in web studies: Methodological issues and a review. International Journal of Internet Science, 1, 58-70.

14.

Green

K. E.

(1991). Reluctant respondents: Differences between early, late, and nonresponders to a mail survey. The Journal of Experimental Education, 59, 268-276.

15.

Griggs

Berzofsky

Shook-Sa

Lindquist

Enders

Krebs

. . . Langton

(2018). The impact of greeting personalization on prevalence estimates in a survey of sexual assault victimization. Public Opinion Quarterly, 82(2), 1-13. doi:10.1093/poq/nfy019

16.

Groves

R. M.

(2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70, 646-675.

17.

Groves

R. M.

Peytcheva

(2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72, 167-189.

18.

Groves

R. M.

Singer

Corning

(2000). Leverage-saliency theory of survey particiption: Descriptions and illustration. Public Opinion Quarterly, 64, 299-308.

19.

Heerwegh

(2009). Mode differences between face-to-face and web surveys: An experimental investigation of data quality and social desirability effects. International Journal of Public Opinion Research, 21, 111-121. doi:10.1093/ijpor/edn054

20.

Kish

(1987, June). Weighting in Deft2. The Survey Statistician, pp. 26-30.

21.

Kott

P. S.

(2006). Using calibration weighting to adjust for nonresponse and coverage errors. Survey Methodology, 32, 133-142.

22.

Krebs

Lindquist

Berzofsky

M. E.

Shook-Sa

B. E.

Peterson

K. C.

Planty

. . . Stroop

(2016). Campus climate validation survey: Final technical report (Bureau of Justice Statistics Research and Development Series, NCJ249545). Washington, DC: Department of Justice, Bureau of Justice Statistics. Retrieved from https://www.bjs.gov/content/pub/pdf/ccsvsftr.pdf

23.

Krebs

Lindquist

Planty

Langton

Berzofsky

M. E.

Asefnia

. . . Enders

(2017). Sensitivity of sexual victimization estimates to definitional and measurement decisions. Journal of Interpersonal Violence. Advance online publication. doi:10.1177/0886260517744190

24.

Kreuter

Presser

Tourangeau

(2008). Social desirability bias in CATI, IVR, and web surveys: The effects of mode and question sensitivity. Public Opinion Quarterly, 72, 847-865.

25.

Mellins

C. A.

Walsh

Sarvet

A. L.

Wall

Gilbert

Santelli

J. S.

. . . Hirsch

J. S.

(2017). Sexual assault incidents among college undergraduates: Prevalence and factors associated with risk. PLoS ONE, 12(11), e0186471. doi:10.1371/journal.pone.0186471

26.

Mercer

Caporaso

Cantor

Townsend

(2015). How much gets you how much? Monetary incentives and response rates in household surveys. Public Opinion Quarterly, 79, 105-129. doi:10.1093/poq/nfu059

27.

Merkle

D. M.

Edelman

Kathy

Brogan

(1998, May 14-17). An experimental study of ways to increase exit poll response rates and reduce survey error. Paper presented at the Annual Meeting of the American Association for Public Opinion Research, St. Louis, MO.

28.

Mitra

Jain-Shukla

Robbins

Champion

Durant

(2008). Differences in rate of responses to web-based surveys among college students. International Journal on E-learning, 7, 265-281.

29.

Muehlenhard

C. L.

Peterson

Z. D.

Humphreys

T. P.

Jozkowski

K. N.

(2017). Evaluating the one-in-five statistic: Women’s risk of sexual assault while in college. The Journal of Sex Research, 54, 549-576. doi:10.1080/00224499.2017.1295014

30.

Paganini-Hill

Hsu

Chao

Ross

R. K.

(1993). Comparison of early and late respondents to a Postal Health Survey Questionnaire. Epidemiology, 4, 375-379.

31.

Parsons

N. L.

Manierre

M. J.

(2014). Investigating the relationship among prepaid token incentives, response rates, and nonresponse bias in a web survey. Field Methods, 26, 191-204. doi:10.1177/1525822X13500120

32.

Pew Research Center. (2015). From telephone to web: The challenge of mode of interview effects in public opinion polls. Retrieved from https://www.pewresearch.org/methods/2015/05/13/from-telephone-to-the-web-the-challenge-of-mode-of-interview-effects-in-public-opinion-polls/

33.

Porter

S. R.

Whitcomb

M. E.

(2003). The impact of lottery incentives on student survey response rates. Research in Higher Education, 44, 389-407.

34.

Rennison

C. M.

Addington

L. A.

(2014). Violence against college women: A review to identify limitations in defining the problem and inform future research. Trauma, Violence, & Abuse, 15, 159-169. doi:10.1177/1524838014520724

35.

Roster

C. A.

Albaum

Smith

S. M.

(2017). Effect of topic sensitivity on online survey panelists’ motivation and data quality. Journal of Marketing Theory and Practice, 25(1), 1-16. doi:10.1080/10696679.2016.1205449

36.

Saleh

Bista

(2017). Examining factors impacting online survey response rates in educational research: Perceptions of graduate students. Journal of Multidisciplinary Evaluation, 13, 63-74.

37.

Sauro

(2015). Does response timing matter in online research? Retrieved from http://www.measuringu.com/blog/early-late-responders.php

38.

Sax

L. J.

Gilmartin

S. K.

Bryant

A. N.

(2003). Assessing response rates and nonresponse bias in web and paper surveys. Research in Higher Education, 44, 409-432.

39.

Tourangeau

Groves

R. M.

Redline

C. D.

(2010). Sensitive topics and reluctant respondents: Demonstrating a link between nonresponse bias and measurement error. Public Opinion Quarterly, 74, 413-432. doi:10.1093/poq/nfq004

40.

Welch

W. W.

Barlau

A. N.

(2013). Addressing survey nonresponse issues: Implications for ATE principal investigators, evaluators, and researchers. Retrieved from https://atecentral.net/r21514/addressing_survey_nonresponse_issues_implications_for_ate_principal_investigators_evaluators_and_researchers

41.

Yessis

Rathert

(2006). Initial versus prompted responders to patient satisfaction surveys: Implication for interpretation of patient feedback. Journal of Applied Management and Entrepreneurship, 11(4). Retrieved from https://www.questia.com/library/journal/1P3-1176774531/initial-versus-prompted-responders-to-patient-satisfaction