An Automated Refusal Conversion Strategy for Web Surveys

Abstract

This article presents results from an experiment conducted during the web-based 2017 Federal Employee Viewpoint Survey (FEVS) to evaluate an automated refusal conversion strategy whereby a sample of individuals was given the opportunity to opt out from the survey and stop receiving additional e-mail reminders. Before being added to the unsubscribe list, however, the individual was asked to cite the primary reason for choosing not to take the FEVS. A randomly assigned subset was given a last-moment appeal, tailored to the reason provided, at which point the individual could either confirm desiring to opt out or navigate to the start of the survey. Because the complementary subset did not receive the appeal, we are able to report on the efficacy of the strategy in convincing individuals who may not have been initially inclined to participate to do so.

Background

Few surveys conducted in practice are mandatory in nature. More commonly, contingent on making contact, the sampled individual (or entity) is asked to participate in the survey, but the decision to do so is ultimately a voluntary one. A profusion of evidence has emerged over the last two decades indicating refusal rates in surveys worldwide are increasing (Atrostic et al. 2001; Brick and Williams 2013; Curtin et al. 2005; de Leeuw and de Heer 2002; Dutwin et al. 2014; Groves and Couper 1998). A higher refusal rate decreases a survey’s response rate, which increases the risk of nonresponse bias (Groves 2006). One commonly adopted strategy to combat this trend is to attempt refusal conversions, whereby initially refusing individuals are persuaded to reconsider their decision. Typically, a distinction is first made between “hard” and “soft” refusals. Hard refusals are characterized by unequivocal directives to be removed from the survey list and no longer contacted. Soft refusals are characterized by reasons such as being temporary unavailable, unsure, or indifferent to the survey and its objectives. As Dutwin et al. (2014) note, survey organizations generally focus refusal conversion attempts on soft refusals exclusively.

Refusal conversion attempts are most often utilized in interviewer-administered surveys (e.g., Cominole et al. 2008; Groves and McGonagle 2001; Olson et al. 2011; Triplett 2002). If interviewers are trained to use a contact history instrument (Maitland et al. 2009), refusal report form (Lavrakas 1993), or something similar to capture, to the extent possible, reasons for the refusal, that information can be used to tailor a refusal conversion attempt later in the data collection period. Such attempts are certainly worthwhile, as they have been shown to result in conversion rates of 10–30% (Dutwin et al. 2014) and may recruit respondents who differ from those not requiring conversion with respect to key survey outcome variables (Kendrick et al. 2001; O’Neill et al. 1995). Granted, those requiring conversion have been found to be more prone to providing incomplete data (Yan et al. 2010) and to spending less time completing the survey (Dahlhamer et al. 2008; Miller and Wedekin 2003), which introduces data quality concerns.

By comparison, refusal conversion attempts are less frequently utilized in self-administered surveys. This is largely due to the ambiguity distinguishing a refusal from other forms of nonresponse. The vast majority of refusals are tacit, with the individual simply ignoring the survey request without providing feedback. Indeed, it is quite rare for individuals to contact the survey administration team to explicitly refuse. Providing a few numbers to put things into perspective, as reported in U.S. Office of Personnel Management (U.S. OPM 2017), only 94 of the 941, 425 individuals sampled as part of the web-based 2016 Federal Employee Viewpoint Survey (FEVS) contacted survey administrators to request to be removed from the survey roster. Of these, 26 were considered hard refusals and 58 were considered soft refusals. A conversion attempt was made for the soft refusals only; eight of those individuals (about 14%) ended up completing the survey.

For self-administered mail surveys, aside from cost savings that can be achieved, it has been argued in the literature that offering the respondent the opportunity to opt out has the potential to increase, not decrease, the likelihood of participating because it engenders trust and empathy with the researcher (Mullen et al. 1987; Sudman 1985). Similarly in spirit, Anderson (2015) recommends administrators of online panels abide by the statute in the Controlling the Assault of Non-Solicited Pornography and Marketing Act of 2003 requiring all unsolicited commercial e-mails to contain a visible unsubscribe link.

In this article, we present results from an experiment fielded to evaluate a novel technique for automating refusal conversions in a web-based survey of individuals by way of a link to opt out (i.e., unsubscribe) embedded in the survey invitation e-mail and reminders. Prior to granting the request to opt out, however, individuals were asked to first indicate the single most influential reason for choosing not to participate. In response to the reason cited, a last-moment appeal was presented in hopes that the individual would reconsider and decide to take the survey. A control group was not given the last-moment appeal, enabling us to assess the effectiveness of the strategy.

This article is structured as follows. The second section details the experimental design and provides background information about the survey on which it was implemented. The third section presents key findings, and the fourth section concludes with a discussion and ideas for further research.

Data and Experimental Design

Data presented in this article are drawn from the 2017 FEVS (www.opm.gov/fevs). First administered in 2002, the FEVS is an annual organizational climate survey administered by the U.S. OPM. The preponderantly attitudinal survey is designed to measure various facets of an employee’s overall satisfaction level such as level of enjoyment with the kind of work performed, perceptions of senior leadership within the agency, and opportunities for advancement. While the primary survey stakeholders are agency human resource managers seeking to identify aspects of the organization that are working well as well as those that may require intervention, Fernandez et al. (2015) summarize the ever-growing body of public administration literature using FEVS data to investigate other organizational phenomena such as diversity and performance management, recruitment and retention, and perceptions of equity and fairness.

With few exceptions, the sampling frame for the survey is produced from an extract of the Statistical Data Mart of the Enterprise Human Resources Integration (EHRI-SDM), a large-scale database of U.S. federal government personnel managed by OPM. The FEVS target population are full- or part-time, permanent (i.e., nonseasonal and nonpolitical) civilian personnel employed with their agency for at least six months before survey commencement. Based on the hierarchically stratified sample design described in U.S. OPM (2017), 1,139,882 employees from 85 distinct agencies were selected to participate in the 2017 FEVS fielded between May 2 and June 22, 2017. Sampled employees were sent an e-mail invitation to participate containing a personalized URL to access the survey, and five reminder e-mails were sent to individuals who had not yet completed the survey in weekly increments thereafter. Hence, despite the agencies having staggered start and end dates, each had a field period lasting precisely six weeks.

A total of 112,576 employees, or approximately 10% of the overall 2017 FEVS sample, were randomly designated for the opt-out experiment. As shown in Figure 1, the opportunity to opt out was offered in all e-mail solicitations via a link labeled “Click here if you are considering not participating in the FEVS.” This link was absent in e-mails sent to employees not designated to be part of the opt-out experiment.

Figure 1.

Example 2017 Federal Employee Viewpoint Survey e-mail body containing a link to the opt-out survey.

When the link to opt out was clicked, a short survey was launched. The screenshot in Figure 2 shows the initial landing page, which poses the question “Would you say that you are unsure about participating in the FEVS or that you do not wish to participate?” The purpose of this question was to gauge the nonresponse conviction level—in essence, to be able to make a proxy distinction between soft and hard refusals. Regardless of one’s answer (i.e., not restricting focus only to soft refusals as is often done in practice), the individual was subsequently presented with the question shown in Figure 3, which was crafted to determine the single most influential reason for not wanting to participate in the 2017 FEVS. Drawing on results from an open-ended question included in a nonrespondent follow-up study fielded following the 2004 administration of the survey (U.S. OPM 2006), six options were provided (e.g., being too busy, data confidentiality concerns), as was a write-in option for someone whose primary reason was not listed. Two research team members independently recoded 176 write-in responses into new or existing response categories, with 28 initially discordant recodes requiring reconciliation. Example reasons unaccounted for by the original set of response categories include a recent or pending employment status change, technical issues accessing the survey, and the belief that one had already completed the survey. In the end, 126 write-in responses were categorized, and the remaining 50 were classified as “Other.”

Figure 2.

Screenshot of opt-out survey landing page and first question gauging nonresponse conviction level.

Figure 3.

Screenshot of second opt-out survey question asking one’s primary reason for refusing to participate.

After answering the question shown in Figure 3, a randomly predetermined 75% of individuals received a last-moment appeal tailored to the response given. The appeal was succinct, taking the form of a bulleted list of three to four assurances and survey facts about which the individuals may not have been aware. As one example, Figure 4 shows the appeal for individuals who indicated being too busy to take the survey. Individuals choosing “Other” as the reason for not wanting to take the survey were presented with a generic appeal of four bullet points drawn verbatim from other tailored appeals. Three team members with over 20 years of combined FEVS experience shared in the responsibility of authoring the appeals. Each team member independently critiqued the other team members’ first draft, and the three convened to assimilate all feedback and finalize the wording. A comprehensive summary of the wording used for all appeals is provided in Appendix A.

Figure 4.

Screenshot of the tailored appeal presented to individuals who stated being too busy as their primary reason for choosing not to participate.

On all appeal pages, immediately below the bulleted list was a link labeled “I will take the survey now,” which would navigate the individual to the start of the survey. If instead the individual clicked on the button labeled “I do not want to take the survey,” a short message appeared on a new page indicating that the individual would no longer receive e-mail reminders to participate from the FEVS administration team but that the survey link would remain active in case the individual changed his or her mind before the end of the field period. Note that the complementary 25% of individuals were routed directly to this same page after answering the item shown in Figure 3. Because they were not given the last-moment appeal, they serve as a control group for quantifying the effectiveness of the automated refusal conversion strategy.

Results

Figure 5 is a flowchart summarizing the dispositions and associated counts for individuals selected to be part of the 2017 FEVS opt-out experiment. Of the original count of 112,576 individuals, 105,319 were deemed eligible to be included in the present analyses. We excluded from consideration individuals who had no chance to participate, such as being caused by the failure to acquire a valid e-mail address, EHRI-SDM indicating departure from one’s position between the time of sample selection and survey administration, or being on temporary assignment or an approved extended absence during the field period. We were surprised, and somewhat discouraged, to observe only 1,533 individuals (about 1.5% of those eligible) clicked on the link to launch the opt-out survey, especially considering the overall response rate to the 2017 FEVS was 45.5%. A small-scale pilot study conducted during the 2016 FEVS found similarly low click rates, but it was speculated at the time that the low rates were caused by introducing the opt-out link too late in the field period—at the second reminder, corresponding to the midway point of the field period—when the majority of responses have already been obtained and also that the link was placed too far toward the bottom of the e-mail body.

Figure 5.

Flowchart summarizing dispositions and counts for individuals designated to be part of the 2017 Federal Employee Viewpoint Survey opt-out experiment.

Of the 1,533 individuals clicking on the opt-out survey link, 831 or 54.2%, ended up completing the 2017 FEVS. This number includes 79 people who initially opted out but later took the survey. A total of 485 individuals opted out and never took the survey, while 217 viewed the page but neither opted out nor returned to the survey. The response rate was 9 percentage points lower for individuals who never clicked on the opt-out survey: 46,897 of 103,786 or 45.2%. This was a statistically significant difference (t = 7.03, p < .0001) and very similar to the response rate for the complementary portion of the FESV 2017 sample not designated to be part of the opt-out experiment: 386,439 of 842,960 or about 45.8%.

As expected, those who viewed the opt-out survey and received a tailored appeal were more likely to complete the survey than those who received no appeal. The difference was 646/1,168 = 55.3% for the former versus 185/365 = 50.7% for the latter. Of note, however, is that employees who viewed the opt-out survey but were given no appeal were still about 5 percentage points more likely to complete the survey than those who never viewed the opt-out survey.

Table 1 contrasts the distributions of seven demographic variables derived from EHRI-SDM for the original opt-out cohort, nonrespondents of the opt-out cohort for any reason, nonrespondents who opted out, specifically, and refusal conversions. We define a refusal conversion as someone who answered at least one of the two items in the opt-out survey yet was ultimately classified as a complete case using the same definition in U.S. OPM (2017), meaning he or she answered at least 21, or 25%, of the 84 core FEVS survey items. One comparison of particular interest to us is whether those who opted out resemble those who did not respond for any reason. If those two distributions are similar, it would further legitimize making inferences on the distribution of nonresponse reasons cited because the opters out would appear to serve as a proxy of sorts for the larger pool of nonrespondents.

Table 1.

Demographic Distributions for the Original Opt-out Cohort, Nonrespondents of the Opt-out Cohort, Nonrespondents Who Opted Out, and Refusal Conversions.

Demographic	Original Opt-out Cohort	Nonrespondents of the Opt-out Cohort	Nonrespondents Who Opted Out	Refusal Conversions
Work location
Headquarters	19.6	15.1	15.7	28.3
Field	80.4	84.9	84.3	71.7
Supervisory status
Nonsupervisor	85.8	88.1	90.7	82.8
Supervisor/executive	14.2	11.9	9.3	17.2
Gender
Female	46.4	43.8	43.9	54.0
Male	53.6	56.2	56.1	46.0
Minority status
Nonminority	63.7	59.9	63.5	68.8
Minority	35.9	39.6	36.3	31.1
Age
<30 years old	4.0	4.5	0.8	2.9
30–39 years old	20.6	22.1	8.9	14.7
40–49 years old	26.0	26.5	20.6	24.5
50–59 years old	33.8	32.2	43.1	38.1
60 and older	15.5	14.8	26.6	19.7
Federal tenure
<5 years	13.0	13.3	8.7	9.9
5–9 years	22.6	23.4	17.5	22.0
10–19 years	33.7	34.3	32.8	31.5
20 or more years	30.7	29.1	41.0	36.6
Income level (USD)
<$50,000	19.7	23.9	19.6	13.5
$50,000–$100,000	49.2	50.2	56.7	46.1
$100,000–$150,000	24.4	20.0	20.0	32.1
$150,000 or more	6.7	5.9	3.7	8.3

Results are somewhat mixed on that front. The distributions are nearly identical for work location (15.1% vs. 15.7% working at the headquarters office), gender (43.8% vs. 43.9% female), and moderately similar for minority status and income level. The distributions diverge somewhat with respect to age and tenure with the U.S. federal government. Opters out tend to be older (69.7% age 50 or older vs. 47%) and longer tenured (41% employed for 20 or more years vs. 29.1%). On the other hand, refusal conversions look markedly different from both the original opt-out cohort and nonrespondents. Comparatively speaking, refusal conversions are more likely to work at the headquarters office, to be female, nonminority, a supervisor or executive, and at a higher income level. Based on what we know from internal analyses of weighting procedures used to adjust FEVS data for unit nonresponse, these are characteristics that are already associated with higher response propensity. So, while the increased response rate observed is certainly a positive feature of our opt-out survey design, there is some evidence that the refusal conversions tend to align more closely with individuals who are already inclined to take the survey. Admittedly, it would be preferable if we could demonstrate converting refusals who more closely resembled nonrespondents.

Restricting the scope to only those individuals who viewed the opt-out survey, Table 2 presents conversion rates by nonresponse conviction level and the primary nonresponse reason cited. The most striking difference is that between individuals indicating they did not want to participate and individuals who were unsure about participating. For the former, only 20.3% of individuals ultimately completed the FEVS, whereas 62.8% of the latter completed the FEVS. Hence, the refusal conversion strategy was much less effective for individuals who entered with a stronger conviction level about not participating. The conversion rate also exhibits substantial variability among the various nonresponse reasons cited. Some of the reasons garnering a lower-than-average conversion rate were being too busy (22.9%), receiving too many survey requests (20%), and the sense that survey results would not be used to change anything in the workplace (25.2%). In contrast, some of the reasons garnering a higher conversion rate were those related to confidentiality concerns (47.3%) or a belief that participation was not supported by agency leadership (58.3%).

Table 2.

Opt-out Survey Conversion Rates by Nonresponse Conviction Level and Nonresponse Reason Cited.^a

Condition	Count	Conversion Rate
Nonresponse conviction level
Unsure about participating	325	62.8
Do not want to participate	551	20.3
Nonresponse reason
Participation not supported by agency leadership	12	58.3
Claim already completed survey	7	57.1
Confidentiality concerns	186	47.3
Other	50	40.0
Recent employment change	29	37.9
Dislike format/technical issues	34	32.4
Survey results never shared with employees	24	29.2
Survey results are not used to change anything	226	25.2
Too busy	118	22.9
Receive too many survey requests	80	20.0
Indifference	9	0.0

^aBecause each item on the opt-out survey was voluntary, counts for the conditions listed do not sum to 1,533, the number of individuals we reported as having clicked on the link to start the opt-out survey.

Table 3 takes a closer look at the effect of the last-moment appeal by cross-classifying the conversion rates by nonresponse conviction level and nonresponse reason. To limit the risk of reporting on findings that may be largely a function of point estimate instability due to small sample sizes, results are suppressed for the nonresponse reason if the count of individuals in either appeal condition was less than 20. Instead, these results are lumped together into a catch-all category labeled “All other responses.” Two particularly interesting findings emerge. First, to be expected, individuals stating uncertainty about participating in the 2017 FEVS were more likely to be persuaded to complete the survey via the last-moment appeal. The appeal produced a 13.9 percentage point increase, whereas the increase was only 8.5 percentage points for individuals stating a desire not to participate from the outset. Second, while the effect of the last-moment appeal was always positive, meaning it prompted more completes than when absent, the effect sizes varied substantively among the primary nonresponse reasons cited. The largest increase was observed for those citing confidentiality concerns, where the appeal reminded the respondent that all FEVS items are voluntary and assured the respondent that no survey results would be provided to superiors in such a way that individual responses. For these individuals, the tailored appeal drove the conversion rate up 33.6 percentage points, from 20.0% to 53.6%. The next highest increases were the appeals given for individuals claiming to be too busy or to receive too many survey requests, which were 17 and 14.2 percentage points, respectively. One of the weakest increases observed was the 8.8 percentage point change observed for individuals who felt the survey and its results would not be used to change anything in the workplace. Although the appeal page pointed to resources containing concrete examples of organizational changes that have been implemented as a result of the FEVS, aside from the “All other responses” category, this was the only increase summarized in Table 3 that was not statistically significant at the α = 0.05 level.

Table 3.

Opt-out Survey Conversion Rates by Appeal Condition Classified by Nonresponse Conviction Level and Nonresponse Reason Cited.^a

Conditions	No Appeal		Tailored Appeal		Difference (Tailored vs. No Appeal)
Conditions	Count of Individuals	Conversion Rate	Count of Individuals	Conversion Rate	Conversion Rate Increase	t	p Value^b
Nonresponse conviction level
Unsure about participating	91	52.7	234	66.7	13.9	2.60	.0055
Do not want to participate	124	13.7	427	22.2	8.5	2.72	.0038
Nonresponse reason
Confidentiality concerns	35	20.0	151	53.6	33.6	4.79	<.0001
Survey results are not used to change anything	54	18.5	172	27.3	8.8	1.62	.0561
Too busy	22	9.1	96	26.0	17.0	2.62	.0082
Receive too many survey requests	21	9.5	59	23.7	14.2	2.07	.0264
All other responses	39	30.8	126	38.1	7.3	0.95	.1739

^bThe p value reported here is based on a one-side alternative hypothesis that the conversion rate for the tailored appeal condition is greater than the conversion rate for the no appeal condition.

Discussion

The purpose of this article was to present results from an experiment fielded during the 2017 FEVS in which a portion of sampled individuals was given the opportunity to opt out from the web-based survey and terminate receipt of subsequent reminder e-mails. Prior to granting the request to opt out, however, the individual was asked to report his or her nonresponse conviction level as well as the primary reason for declining to participate. From there, individuals were randomly partitioned into two groups: one that received a last-moment appeal tailored to the reason provided and another that received no appeal.

Although we were surprised by the extremely low percentage of individuals who noticed and clicked on the link to launch the opt-out survey (about 1.5%), we found that doing so yielded promising results. These individuals were far more likely to complete the survey than opt out, and they were even more likely to complete the survey than those who never clicked on the link. Taken together, these two points suggest that the strategy is a net positive feature practitioners could consider incorporating into other web-based surveys to identify tacit refusals and help automate the conversion process. Of course, it may also serve as a tool for nonresponse bias analyses (Gorsak et al. 2018).

Further research could look into alternative methods to get more individuals to locate and click on the opt-out link embedded within the survey invitation and reminder e-mails. This is hardly a new problem, as Couper (2008:325) asserts “relatively little can be done with the e-mail invitation to ensure it is opened, read, and acted on. This remains our biggest challenge as web survey designers.” Rather than further manipulations to the body of e-mail invitations and reminders, a future study could examine the impact of offering the opportunity to opt out via a separate follow-up e-mail, perhaps tailored to certain subgroups in the spirit of work described in Lynn (2016).

Another key finding from this study was that the impact of the last-moment appeal was more pronounced on the conversion rate for those who indicated being unsure about their intent to participate as opposed to those indicating at the outset that they did not want to participate. This was to be expected, as we viewed the former and the latter as respective proxies of sorts for soft and hard refusals. As noted, however, effect sizes varied notably among the primary nonresponse reason cited, which may in turn be a function of varying levels of persuasion achieved by the tailored last-moment appeals. Further research could look into optimizing the appeal language with feedback, say, from focus groups comprised of FEVS-eligible employees.

Footnotes

Appendix A

Table A1.

Summary of Assurance and Survey Fact Wording Used for the Tailored Appeal.^a

Nonresponse Reason Given by Employee	Appeal Presented on Subsequent Page
I am too busy to take the survey	As federal employees ourselves, the FEVS team understands you are busy and that your time is valuable. The survey is sent to the minimum number of federal employees necessary to provide for a representative sample of the government-wide workforce. The survey should only take about 20 minutes to complete. You are allowed to complete the survey during regular work hours, but, if necessary, you can complete it during nonwork hours from any web browser.
I receive too many requests to take surveys	The FEVS team is aware that federal employees receive a lot of surveys these days. This is a very high profile survey, however, one used to identify the most (and least) desirable places to work in the federal government. For example, it is used in the Partnership for Public Services’ Best Places to Work® Rankings (http://bestplacestowork.org/BPTW/index.php). You are the only source of this information. There is no substitute for your perspectives about what is working well and what needs improvement within your agency.
Survey results are never shared with employees	The FEVS team produces over 30,000 reports each year, including a report summarizing responses for any agency or office within so long as it has at least 10 respondents. We work with representatives from each agency who serve as points of contact for disseminating these reports throughout the agency. If you feel survey results are not sufficiently distributed to employees in your agency or office, you should contact a representative within your agency’s human resources office. Any federal employee with a government e-mail address can access the Unlocking Federal Talent tool at www.unlocktalent.gov, which presents a wide range of FEVS results in a visual, interactive way.
I am concerned about the confidentiality of my responses	Your supervisor and agency’s leadership never get access to individual-level survey results that could be used to identify your responses. You do not have to answer all items on the survey. You can skip any item you do not want to answer. The FEVS maintains strict minimum respondent count thresholds when reporting aggregated survey results, masking your work unit and demographic categories where necessary to prevent others from identifying your responses. A rigorous statistical disclosure avoidance procedure is applied to the survey data prior to any release of individual-level survey responses.
Participation in the survey is not supported by leadership in my agency	While the survey is open, senior leadership at your agency regularly requests response rate updates from the FEVS team. In fact, to meet the growing demand for these requested updates, the FEVS team developed a website for your agency’s leadership to monitor response rates in real time and to benchmark them against the government-wide average. Low response rates can negatively impact the face validity of survey results and the representativeness of the sample. Believe us when we say that your agency’s leadership very much wants as many employees as possible to take the FEVS.
Survey results are not used to change anything in my workplace	Based on feedback the FEVS team has received, almost all agencies have an action planning committee composed of employees throughout the agency who are tasked with extracting insights from the FEVS and other sources to improve the organizational culture. In the past, these committees have used survey results to expand training and leadership development opportunities, increase telework utilization, and improve channels of communication between leadership and the workforce. Specific examples can be found on the Community of Practice page of the Unlocking Federal Talent tool at www.unlocktalent.gov, which any federal employee with a government e-mail address can access.
Other, please specify	Your supervisor and agency’s leadership never get access to individual-level survey results that could be used to identify your responses. You do not have to answer all items on the survey. You can skip any item you do not want to answer. You are the only source of this information. No one else can provide the unique perspectives you have about your agency. The survey should only take about 20 minutes, and you are allowed to do it during work hours.

^aEach of the bulleted lists above is preceded by a statement “Thank you for providing feedback about why you do not want to take the FEVS. You may already be aware, but let us remind you one last time of a few important points:”

Acknowledgments

We thank Department of Commerce Labor-Management Forum members Laurie Schwede, Jennifer Childs, Gerson Morales, Paul Beatty, and Andrea Contratto for their input and feedback in the development of the opt-out survey.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Anderson

2015. Why email bounces and opt-outs are good for your research panel. https://www.qualtrics.com/blog/why-email-bounces-and-opt-outs-are-good-for-your-research-panel/ (accessed February 26, 2018).

Atrostic

Bates

Burt

Silberstein

. 2001. Nonresponse in US government household surveys: Consistent measures, recent trends, and new insights. Journal of Official Statistics 17:209–26.

Brick

Williams

. 2013. Explaining rising nonresponse rates in cross-sectional surveys. Annals of the American Academy of Political and Social Science 645:36–59.

Cominole

Franklin

Siegel

. 2008. Combined strategies to increase early response rates. Proceedings of the Annual Conference of the American Association for Public Opinion Research, New Orleans, LA, May 15–18.

Couper

. 2008. Designing effective web surveys. New York: Cambridge University Press.

Curtin

Presser

Singer

. 2005. Changes in telephone survey nonresponse over the past quarter century. Public Opinion Quarterly 69:87–98.

Dahlhamer

Simile

Taylor

. 2008. Do you really mean what you say? Doorstep concerns and data quality in the National Health Interview Survey (NHIS). Proceedings of the Joint Statistical Meetings of the American Statistical Association, Denver, CO, August 3–7.

de Leeuw

de Heer

. 2002. Trends in household survey nonresponse: A longitudinal and international comparison. In Survey nonresponse, eds. Groves

Dillman

Eltinge

Little

, 41–54. New York: Wiley.

Dutwin

Loft

Darling

Holbrook

Johnson

Langley

Lavrakas

, et al. 2014. Current knowledge and considerations regarding survey refusals, report prepared for the American Association of Public Opinion Research Council by the Task Force on Survey Refusals. http://www.aapor.org/AAPOR_Main/media/MainSiteFiles/RefusalTF_FINAL090814.pdf (accessed February 26, 2018).

10.

Fernandez

Resh

Moldogaziev

Oberfield

. 2015. Assessing the past and promise of the Federal Employee Viewpoint Survey for public management research: A research synthesis. Public Administration Review 75:382–94.

11.

Gorsak

Lewis

Frank

. 2018. Understanding their apprehension: A look into the Federal Employee Viewpoint Survey. Proceedings of the Federal Committee on Statistical Methodology Research and Policy Conference, Washington, DC, March 7–9.

12.

Groves

2006. Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly 70:646–75.

13.

Groves

Couper

. 1998. Nonresponse in household interview surveys. New York: Wiley.

14.

Groves

McGonagle

. 2001. A theory-guided interviewer training protocol regarding survey participation. Journal of Official Statistics 17:249–65.

15.

Kendrick

Hapgood

Marsh

. 2001. Do safety practices differ between responders and non-responders to a safety questionnaire? Injury Prevention 7:100–03.

16.

Lavrakas

1993. Telephone survey methods: Sampling, selection, and supervision, 2nd ed. Thousand Oaks, CA: Sage.

17.

Lynn

2016. Targeted appeals for participation in letters to panel survey members. Public Opinion Quarterly 80:771–82.

18.

Maitland

Casas-Cordero

Kreuter

. 2009. An evaluation of nonresponse bias using paradata from a health survey. Proceedings of the Joint Statistical Meetings of the American Statistical Association, Washington, DC, August 1–6.

19.

Miller

Wedeking

. 2003. The harder we try, the worse it gets? Examining the impact of refusal conversions and high callback attempts on the quality of survey data. Proceedings of the Annual Conference of the American Association for Public Opinion Research, Nashville, TN, May 16.

20.

Mullen

Easling

Nixon

Koester

. 1987. The cost-effectiveness of randomized incentive and follow-up contacts in a national mail survey of family physicians. Evaluation & the Health Professions 10:232–45.

21.

Olson

Lepkowski

Garabrant

. 2011. An experimental examination of the content of persuasion letters on nonresponse rates and survey estimates in a nonresponse follow-up study. Survey Research Methods 5:21–26.

22.

O’Neill

Marsden

Silman

. 1995. Differences in the characteristics of responders and non-responders in a prevalence survey of vertebral osteoporosis. Osteoporosis International 5:327–34.

23.

Sudman

1985. Mail surveys of reluctant professionals. Evaluation Review 9:349–60.

24.

Triplett

. 2002. What is gained from additional call attempts & refusal conversion and what are the cost implications? Technical report of the Urban Institute.

25.

U.S. OPM (Office of Personnel Management). 2006. 2004 Federal human capital survey nonresponse study. Technical report.

26.

U.S. OPM (Office of Personnel Management). 2017. 2016 Federal employee viewpoint survey: Technical report. https://www.opm.gov/fevs/archive/2016FILES/2016_FEVS_Technical_Report.pdf (accessed February 26, 2018).

27.

Yan

Curtin

Jans

. 2010. Trends in income nonresponse over two decades. Journal of Official Statistics 26:145–64.