Abstract
Web surveys provide researchers a platform to gather data in an inclusive manner and from a global audience. However, web surveys typically suffer from low response rates. To combat this issue, numerous studies have investigated how invitation messaging can improve response rates. A few studies have investigated embedding the first survey item in the e-mail invitation message and how adding humor to a reminder message might improve response rates. Our study adopted a 2 (invitation message embedded item vs. not) × 2 (reminder message embedded item vs. not) × 2 (humorous reminder message vs. not) study design. Results showed that among participants who only received the initial invitation, embedding a survey item in that invitation did not enhance completion rates. However, completion and break-off rates, compared to nonresponse, increased when participants received a survey item embedded in their reminder. Further, participants who received a humorous reminder were more likely to complete the survey than to break off. No interaction effects between the embedded item in the reminder message and the humorous tone of the reminder message emerged. Implications for invitation design are discussed.
Web surveys are one of the most common tools used by social scientists to study a variety of social phenomena such as national elections, consumer behavior, career choices, and organizational behavior trends (Baruch & Holtom, 2008; Fan & Yan, 2010). Various features of web surveys make them an attractive and widely used tool. For instance, web surveys make it possible to survey individuals from a diverse population in a cost-effective and confidential manner which, in turn, encourages participants to respond honestly (Bachmann, Elfrink, & Vazzana, 1999). Furthermore, web surveys lend themselves to easy administration over multiple time points and to the efficiency of transferring data to statistical software for data analysis (Wyatt, 2000).
Unfortunately, to carry out research and make generalizable inferences using web surveys, researchers are at the mercy of participants’ willingness to respond to these surveys. As most web surveys are voluntary in nature, attaining an optimal response rate is typically a challenge. Such a goal is important because high response rates ensure that the inferences made are representative of the target population (Rogelberg & Stanton, 2007). Additionally, larger sample sizes allow researchers to unravel statistical effects and report those effects with greater confidence and precision (Baruch & Holtom, 2008). Hence, survey researchers have dedicated considerable time and effort to identifying factors that elicit high response rates among participants.
Extensive research has examined the impact that different elements of a survey invitation message have on response rates. For instance, personalizing survey invitations positively affects survey response rates and by some accounts (Sauermann & Roach, 2013) can increase the odds of participation by as much as 48%. Other research shows that survey invitations that explicitly provide a deadline to respondents tend to be associated with higher response rates (Porter & Whitcomb, 2003). Additionally, other empirical studies indicate that the gender of the survey sender has an impact on survey response rates. Precisely, a survey invitation from a female contact is likely to elicit a higher response rate than a male contact within a target group that is male dominated (Keusch, 2012).
Investigating the influence of survey invitation designs on response rate is important, as a survey invitation is one of the few platforms available for researchers to communicate the aim of the survey to their audience and the only opportunity to persuade them to participate. The current research, therefore, aims to build upon and extend past research by investigating the impact of two design features of web surveys on response rates. Specifically, we examine whether embedding a survey item in the survey invitation and sending respondents a humorous reminder will enhance survey response rates. Recent studies on web survey response rates revealed that participants who received an embedded item in their survey invitation were more likely to respond to the first question of the survey and were more likely to answer a greater proportion of the survey (Liu & Inchausti, 2017; Varghese, Moore, & Earnhart, 2018) relative to those participants who received a traditional survey URL. This might be because providing an embedded item gets the researcher’s foot in the door, so to speak, and when individuals comply with an initial request (e.g., start the survey), they are more likely to comply with the larger request (e.g., complete the survey; Petrova, Cialdini, & Sills, 2007). As an alternative explanation, empirical research (Vannette, 2016) has shown that when an invitation message reduced barriers to participation, survey completion rates were enhanced. Providing an embedded item reduces hurdles to survey participation.
Although embedding a survey item resulted in positive outcomes, researchers also witnessed a greater, but not statistically significant, break-off rate in the embedded condition relative to the traditional condition (Liu & Inchausti, 2017). Therefore, we hypothesize in our study the following:
Advertising and education research suggest that humor can be an effective method of persuasion (Heinecke, 1997; Wallinger, 1997). We draw from this literature to understand how a humorous e-mail reminder is likely to increase survey response rates. According to the elaboration likelihood model, individuals process information based on their level of involvement with the topic. When individuals find the topic to be salient, they are likely to adopt a central route to persuasion and process a message presented to them systematically. Depending on the extent to which engaged individuals perceive arguments/counterarguments as compelling, they either buy in or reject the message, respectively (Petty & Cacioppo, 1986).
When the issue is less salient, the peripheral route to persuasion is taken; people bank on heuristic/peripheral factors (e.g., situational elements and the credibility of the source) to process arguments. Scholars argue that this is the default route to processing information; individuals tend to opt for the central route only when they are immensely involved with or are passionate about the topic.
Source attractiveness is one heuristic factor, and humor specifically tends to enhance source attractiveness by generating a positive affect. Past research shows that people who experience a positive affect are more likely to comply with requests (Bless & Schwarz, 1999). Sending a humorous survey reminder has the potential to elicit a positive affect toward the survey and persuade a respondent to attempt, and eventually, to complete it. These suppositions have received some empirical support. Rath et al. (2017) conducted an experiment in which the survey respondents either received a standard reminder or a reminder that had a humorous tone. The results of the study revealed that humor had a positive impact on response rates (Rath et al., 2017). Therefore, in our study, we hypothesized the following:
Although some research has looked at the influences of humor and an embedded item separately on response rates, there is a paucity of studies replicating these effects. Moreover, to our knowledge, there have not been any studies that have examined the interactive effects of these two design features on survey response rates. Current research attempts to address this gap within the literature. Given that obtaining high response rates for web surveys is challenging and there are limitations associated with low response rates (such as low generalizability of conclusions to the relevant population), identifying factors that could augment response rates in an additive manner would be beneficial to the field of survey research as a whole. Although our efforts to test for the interactive efforts are exploratory, we anticipate that individuals who receive a humorous reminder and received an embedded item will be more likely to complete the survey than their counterparts who are exposed to only one or neither of these design features. We anticipate this response pattern because an embedded item takes participants a step closer to survey participation, and the positive affect induced by the humorous reminder makes them better candidates to comply with the request.
Method
An online survey was administered to a disproportionate stratified random sample of students (n = 56,896) who had registered for the February 2017 ACT test (N = 333,059). Students who self-reported as Asian, Hispanic, or African American were oversampled. Specifically, while Asian students account for 4.6% of the population, they were, 18% of the sample, sent an e-mail invitation. For Hispanic and African American students, this was 13.5% and 19.5%, respectively. White students (56.5% vs. 19.8%), those students who chose two or more races and those students who did not answer the race/ethnicity question (9.5% vs. 4.0%, respectfully), were undersampled. All American Indian/Alaska Native and Native Hawaiian/Other Pacific Islander students were included. This sampling method was utilized to ensure sufficient representation across all racial/ethnic groups for the applied research study. Weighting was applied to all analyses to adjust for the stratified random sampling procedure. Weights were calculated based on the racial distribution for all U.S. students who registered for the February 2017 ACT test. Students’ self-reported race was obtained from ACT’s database of students who registered for the ACT test.
All participants in the study were sent an e-mail acknowledging that they had recently registered to take the ACT test followed by a request for them to participate in a survey about their test-taking experience. The first survey item was a filter question where we asked students about their status as a test taker. This is a common item we administer in all of our student surveys. Although most e-mail addresses that were provided at the time of registration belonged to students, occasionally a school counselor or parent registered a student for the test and inadvertently included their own and not the student’s e-mail address. Based on their response to the filter question, students were redirected to the remaining items in the online survey, and all nonstudents were sent to the end of the survey. Nonstudents were classified as nonresponders in all analyses.
This study employed a 2 × 2 × 2 factorial experimental design. Prior to survey administration, students were randomly assigned to one of two conditions, namely, an e-mail invitation with an embedded item versus a traditional e-mail invitation. In Condition 1, the invitation instructed students to start the survey by clicking on their answer to the first survey question (Figure 1).
E-mail invitation for Condition 1: First question embedded.
Participants who were randomly assigned to Condition 2 received the traditional e-mail invitation. Here, the first question was not embedded in the invitation message. Instead, participants were asked to click on a survey URL (Figure 2). Once participants clicked on the URL, they were redirected to the online survey and presented with the filter question.
E-mail invitation for Condition 2: Traditional e-mail.
Within each of these two conditions (embedded item invitation vs. traditional invitation), students were randomly assigned, before initial invitations were sent out, to a reminder message experimental group using a 2 (reminder with an embedded item vs. traditional reminder) × 2 (reminder with a humorous narrative vs. no humorous narrative) research design. The humorous narrative used for this experiment is presented in Figure 3. The traditional reminder had the same language as the original invitation but included “This is a friendly reminder…” at the start of the message. Only those students who did not complete the survey within the first 5 business days of the initial invitation received a reminder message. The survey closed 7 days after the reminder message was sent. An identical survey was sent to participants in all experimental groups. Figure 4 summarizes the random assignment to conditions and the number of participants eligible to receive the reminder message.
Language used in the humorous reminder message.
Flowchart of eligibility and sample size by group.
Results
The first hypothesis stipulated that embedding a survey item in the e-mail messaging would be positively associated with completion rate. This hypothesis was tested in two different contexts; we separately examined the impact of embedding a survey item in the initial invitation and the impact of embedding a survey item within a reminder context using binary logistic and multinomial logistic regression analysis, respectively. The second hypothesis stipulated that using humorous message reminders would be positively associated with completion rate. This hypothesis was tested using multinomial logistic regression analysis.
In order to test the first context of the first hypothesis, we carried out a multinomial logistic regression analysis that included responses from participants who either broke off (answered between 1% and 79% of the survey) or completed (answered at least 80% of the survey) the survey before a reminder message was sent. Therefore, students who participated after the reminder were classified as nonresponders in this analysis. We thought it was prudent to ignore survey participation after the reminder message was disseminated because participation in this instance is likely to be influenced by both the manipulation of the initial invitation message and the reminder message. A total of 4,550 students were classified as participants (2,130 broke off and 2,420 completed) with all 56,896 people included in the analysis. Our dependent variable had three outcomes: completion, break off, and nonresponse (answered 0% of the survey).
Descriptive statistics for this analysis are presented in Table 1. There were slightly more students who received the embedded invitation and completed the survey (4.3%) than students who received the traditional invitation (4.2%). Break-off rates were higher for those students who received the embedded invitation (5.7%) relative to those students who received the traditional invitation (1.8%). Despite these patterns, more data overall were collected from the embedded item invitation group (n = 2,851) than from the traditional invitation group (n = 1,699).
Percentage of Students, by Experimental Group and Outcome.
Note. N = 4,550.
Results from the logistic regression analysis reflect the descriptive statistics. Embedding a survey item in the initial invitation did not have a statistical impact on survey completion rates, b = 0.04, p n.s., CI [0.95, 1.12]. Students who received the embedded invitation were more likely to break off than not attempt at all (b = 1.27, p < .001, CI for odds ratio (OR) = [3.20, 3.94]. Precisely, among participants who received the embedded invitation message, the odds of breaking off were 3.55 times higher than those students who received the traditional invitation message. Hence, among participants who only received the initial invitation, embedding a survey item in that invitation did not enhance completion rates but did enhance the overall volume of data, when combining both complete and incomplete data.
In the second context of the first hypothesis, we carried out a multinomial logistic regression analysis to examine whether embedding a survey item in the reminder enhanced completion rates. This analysis included everybody who received a reminder message (n = 52,346). We controlled for the effect of embedding a survey item in the initial survey invitation.
Descriptive statistics for data used in the multinomial model are presented in Table 2. More students who received the embedded item in the reminder message were more likely to complete (3.4%) and break off (3.7%) from the survey relative to those students who received the traditional reminder message (2.9% complete and 1.6% break off). Likewise, more students who received the humorous reminder message completed the survey (4.4%) and broke off (3.0%) relative to those students who did not receive the humorous reminder (1.9% complete and 2.3% break off).
Percentage of Students, by Reminder Message and Outcome Measure.
Note. N = 52,346.
The multinomial regression results revealed that the odds of participants completing the survey when they received a survey item embedded in their reminder was 1.15 times higher (b = 0.14, p = .02, OR CI [1.01, 1.28]), and the odds of breaking off from the survey was 2.49 times higher (b = 0.92, p < .001, OR CI [2.14, 2.91]) than not attempting the survey at all (see Table 3). However, the odds of participants breaking off from the survey than completing it were 2.18 times higher when they received an embedded item in their reminder (b = 0.78, p < .001 OR CI [1.80, 2.64]).
The Effects of Embedding a Survey Item in the Reminder and the Reminder’s Humorous Message on Survey Outcomes.
Note. SE = standard error.
aReference category is “did not attempt.” bReference category is “complete.”
*p < .01. **p < .001.
The two sets of multinomial logistic regression analyses provide mixed results and reveal that there is some support for Hypothesis 1, that is, when participants received an embedded item in the reminder message, they were more likely to complete or break off from the survey than to not attempt the survey at all. Contrary to our hypothesis, an embedded item in the invitation or the reminder did not dissuade them from breaking off from the survey instead of completing it. Although we were unable to attain higher rates of completion using the embedded item invitation format in every instance, it is important to note that those who received an embedded item in their survey invitation or reminder message were more likely to attempt the survey. Albeit incomplete, researchers are still garnering some data that might be valuable for research purposes.
In order to test our second hypothesis, we used the same multinomial regression model in Table 3 to further investigate whether adding humor to the reminder augmented completion rates. In this instance, we regressed the survey completion variable on the independent variable “humorous reminder” while holding the initial invitation message, the embedded reminder message, and any interaction terms constant. Results suggested that among participants who received a humorous reminder, the odds of survey completion or breaking off was higher than not attempting the survey at all. Precisely, the odds of participants who received the humorous reminder completing the survey was 2.23 times higher (b = 0.80, p < .001, OR CI [1.93, 2.58]), and the odds of breaking off from the survey was 1.40 (b = 0.33, p < .001, OR CI [1.23, 1.59]) times higher than not attempting the survey at all. Further, among students who received the humorous reminder message, the odds of completing the survey was 1.59 times higher than breaking off from the survey (b = −0.46, p < .001, OR CI [1.32, 1.93]). These results provide evidence to support Hypothesis 2; humor indeed elicits compliance from participants to participate in the survey thereby enhancing completion rates.
We carried out an exploratory analysis to examine whether either of the reminder message features (i.e., an embedded survey item or a humorous tone) modified the effect of the other feature on completion rates. To test this, we capitalized on the 2 × 2 research design mentioned earlier. The interaction was not statistically significant suggesting that although embedding a survey item in the invitation/reminder and adding humor to the reminder may positively influence completion rates separately, together these features may not provide researchers any added advantage.
Discussion
Researchers and other stakeholders (e.g., human resource personnel within organizations) administer web surveys to gather data to study social, psychological phenomena and to understand dominant attitudes about varying topics (e.g., new products in the market, organizational trends, political events). An optimal survey response rate increases the confidence in the research findings. Our study, using an experimental design, attempted to test the efficacy of two invitation messaging design factors in improving survey response rates.
Recent research in survey methodology makes a case for embedding survey items in survey invitations (Liu & Inchausti, 2017). Our study, however, shows that embedding a survey item in an invitation elicits a 2-fold increase in survey break off relative to survey completion. These results speak to past research findings. Liu and Inchausti (2017) also found higher break-off rates with the embedded item design in the invitation, even if these differences were not statistically significant. Our research in contrast observed a significant difference suggesting that we may be able to encourage participants to start the survey and provide some useful information, but this method may not necessarily encourage them to complete the entire survey, particularly for those respondents who never received a reminder message.
Our research also examined the effect of introducing embedded items at a later stage of survey administration, that is, in the reminder. There is indeed a silver lining associated with embedding a survey item in a reminder. Our findings suggest that doing so increases the likelihood of completion and break off substantially compared to nonresponse behaviors. Although the introduction of a survey item in the reminder does not produce the most ideal situation (e.g., completion rates being higher than break-off rates), the embedded item appears to be encouraging participants to engage with the survey at some level. In several situations, obtaining partial data compared to no data at all is advantageous. For example, when an analysis focuses on answers to specific survey items, obtaining as many responses as possible is needed. Nonetheless, there may be instances in which partial completion is as good as obtaining no data from the participant (e.g., when estimating prediction models). In other words, embedding a survey item in a survey invitation or reminder may be beneficial when research questions can be addressed by using data from partial to almost complete surveys. From this perspective, it is imperative that questions of most relevance to the study be placed at the beginning of the survey.
The findings from the current study also provide support for using humor to persuade survey respondents to participate in the survey. As suggested by the elaboration likelihood model, introducing humor in the survey reminders can persuade participants to participate in the survey by increasing the positive affect participants had toward the survey. Precisely, humor in the reminder was twice more likely to encourage completion than nonresponsive behavior among participants. Perhaps the humorous message put a smile on their faces or made them laugh. While our study did not explicitly measure positive affect, the results provide evidence that adding humor motivated students to participate. Future research might measure affect to provide direct evidence of this relationship. In addition to finding support for the persuasion theory within a survey research context, this study has replicated past research findings (Rath et al., 2017).
This study attempted to address an important research gap related to determining whether integrating survey design factors would amplify the positive effect they may separately have on survey response. Although an embedded item within a reminder had some positive effect in some instances, adding humor to the embedded item reminder did very little to improve the positive impact of the embedded item on response rates. Therefore, as per our study results, researchers are not likely to augment response rates by manipulating both these design factors relative to choosing just one to be included in the reminder.
Similar to other research endeavors, our project is not without limitations. Because our investigation of effects of the embedded invitation excluded those respondents who received the reminder message, our findings generalize to only those respondents who participated before a reminder message was sent. Future research might consider a research design that randomizes nonrespondents to the reminder message after initial respondents are excluded. The humor we added to our reminder was context-specific and catered to students. Hence, these results may not generalize to professionals who take organizational surveys. Future research is warranted to examine whether certain kinds of humor appeal to certain demographic groups.
Although research pertaining to survey methodology has identified various design features that tend to increase survey response rates, additional research is needed to replicate these findings in other contexts as well as to expand upon them to try to identify various design factors that work together to boost survey response rates. For instance, it would be beneficial to investigate whether adding humor at different levels (e.g., in the initial invitation and reminder) will elicit higher levels of participation or whether manipulating the humor message (e.g., testing different humorous messages) engages or appeals to certain populations versus others (e.g., adults vs. students).
Footnotes
Authors’ Note
All statements expressed in this article are the authors’ opinions and do not reflect the official opinions or policies of ACT. Please contact
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the Department of Research at ACT.