Abstract
Falling participation rates is one of the most significant challenges facing survey research today. To curb this negative trend, scholars have searched for factors that can increase and decrease citizens’ willingness to participate in surveys. In this article, we investigate the timing effects of survey invitation e-mails on participation rates in a university-based online panel with members of the Swedish public. Through two large-scale experimental studies, we examine whether the day of week (N = 11,294) and time of day (N = 47,279) for sending out survey invitations impact participation rates. We also ask respondents when they prefer to answer surveys. We find that the timing of survey invitations affects participation rates, however, the effects are small, short-lived, and even out within a week. We also find that the effects of timing vary by employment status and age. The results have implications for scholars and practitioners who utilize online panels for web surveys. When quick answers are important, there may be some limited gains of tailoring the timing of the survey invitation to different individuals. In surveys with more extended field periods, however, such efforts seem less warranted.
Over the past decades, web surveys have seen dramatic growth in popularity, and it is a common survey mode in various fields such as opinion polling, organizational research, and marketing (Brüggen & Dholakia, 2010, p. 239). At the same time, low participation rates are endemic in this type of surveys, which raises questions about the generalizability of the findings. In this article, we examine whether the timing of survey invitation e-mails to online panel members is important or not. The goal is to explore whether participation rates in online panel web surveys can be increased by optimizing the timing of the survey invitation to participants.
The web survey mode is convenient both for survey practitioners, as it provides a time and cost-saving method for collecting survey response (e.g., Dillman, 2011), and for participants, since survey response is more flexible (Keusch, 2015, pp. 190–191). Currently, most web surveys are conducted with panels of standing members (Brüggen & Dholakia, 2010, p. 239). The panels usually consist of individuals recruited to participate in surveys conducted online (e.g., Couper, 2000; Göritz, Reinhold, & Batinic, 2002), and they offer many advantages over cross-sectional web surveys. For example, it reduces the cost and time of recruiting participants and allows for shorter and more specific questionnaires once the respondents have provided information about sociodemographic characteristics such as age and sex (Göritz, 2004). Additionally, respondents’ answers can be cross-referenced and validated with previous panel waves (Göritz & Moser, 2000).
Despite the many advantages, online panel surveys suffer from low participation rates (Brüggen & Dholakia, 2010, p. 239). In online panels, this can occur at two levels. First, people can decline to join a panel when invited. Second, individuals who decide to join the panel can still decline participation in surveys conducted within the panel. In fact, many online panels have a membership base consisting of mostly nonresponding individuals (Couper, 2000; Wansink, 2001). If the nonresponding individuals have specific characteristics in common—that is, if the nonresponse is systematic—and these characteristics correlate with central outcome variables, this can bias the results and reduce their generalizability (e.g., Bethlehem, Cobben, & Schouten, 2011; Groves, 2006; Groves & Peytcheva, 2008).
According to Zheng (2011), web survey invitations are generally sent on weekdays between 5 a.m. and 4 p.m. However, if this is the optimal time for improving participation rates in web surveys with online panel members remains an open question. In this article, we study the effects of timing of invitation e-mails on participation rates through two survey experiments in a Swedish university–based online panel. We find that participation rates are the highest when the survey is dispatched on weekdays, but the effects are short-lived and disappear within a week. We also find that the results differ somewhat for individuals of different age and employment status. Furthermore, we find indications that matching the invitation timing with respondents’ self-expressed timing preferences can potentially increase participation rates somewhat in the short term.
Background
The web survey mode offers several advantages for survey practitioners when compared to traditional surveys conducted through, for example, mail or telephones (for overviews, see Sax, Gilman, & Bryant, 2003; Sheehan, 2001). Besides being a time and cost-saving method for data collection, web surveys can provide quicker responses (Göritz et al., 2002; Ilieva, Baron, & Healy, 2002) and improve response quality as people tend to give more extensive open-ended answers (Paolo, Bonaminio, Gibson, Partridge, & Kallail, 2000; Smyth, Dillman, Christian, & McBride, 2009). Self-administered surveys, such as web surveys, have also been found to provide more honest answers on controversial and sensitive issues than other survey modes (Chang & Krosnick, 2009; Kreuter, Presser, & Tourangeau, 2008). Given its many advantages, the falling participation rates and difficulties in recruiting individuals willing to participate in web surveys seem particularly unfortunate.
Participation Rates in Web Surveys
The trend of falling participation rates applies to all survey modes and across the world (Fan & Yan, 2010, p. 136). However, web surveys have markedly lower participation rates than other types of survey modes (e.g., Frippiat & Marquis, 2010; Rogelberg & Stanton, 2007), even though other modes are closing in (telephone surveys in particular).
The problem with low participation rates in web surveys is alarming for several reasons. First, low participation rates reduce sample sizes and statistical power and thereby the possibilities of identifying significant relationships between important variables (Cohen, 1992). Second, small sample sizes also limit the possibility of conducting subsample analyses; hence, it reduces the chances of detecting moderating and interaction effects, as well as effect heterogeneity in a population. Third, irrespective of sample size, lower participation rates increase the risk of nonresponse bias when the nonresponse is systematic, potentially leading to lower validity and representativeness of the findings (Bethlehem et al., 2011; Groves, 2006; Groves & Peytcheva, 2008).
Simply put, lower survey participation rates may impair the accuracy of inferences being drawn about a population, as well as its subsets. This emphasizes the need to understand which factors impact participation rates and individuals’ willingness to participate in surveys, with the aim to develop modes and research designs that can adjust for the nonresponse issues (e.g., Couper, 2000, p. 466; Deutskens, De Ruyter, Wetzels, & Oosterveld, 2004, p. 22; Luiten & Schoten, 2013; Schouten, Peytchev, & Wagner, 2017).
Prior studies have investigated factors such as survey length and design, contacts with respondents, the “source” of the survey (e.g., private actors, researchers, or public authorities), and different compensatory incentives for participation (such as monetary rewards; for overviews, see Fan & Yan, 2010; Keusch, 2015; Sheehan, 2001).
Another factor that could potentially affect the participation rate is the timing when respondents are invited to participate in a survey. Receiving an invitation at a convenient time may decrease the cost of participating, simply because the individual will have enough time to complete the survey. Receiving the invitation at an inconvenient time, on the other hand, could make individuals decline by inducing stress.
It is possible that efficient timing strategies may increase participation. Web surveys provide a useful context for exploring this possibility, as the invitation can be timed precisely, both with respect to the day of week and the time of day. Moreover, computer software used for web surveys usually provides information about undeliverable e-mails, and when the survey was opened and completed—information that may be used to improve the data collection procedures (e.g., Lewis & Hess, 2017; Paolo, et al., 2000, 84; Shinn, Baker, & Briers, 2007). Despite this, there has as yet been relatively little research published on the relationship between invitation timing and participation rate in web surveys—particularly in web surveys conducted with online panel members.
Participation Rates and Timing of the Survey Invitation
Research on timing effects for other types of survey modes than web surveys, such as face-to-face and landline telephone interviews, indicates that weekday evenings and weekend daytimes, on average, are the best times to contact participants in order to obtain an interview (see, e.g., Durrant, D’Arrigo, & Steele, 2011; Weeks, Jones, Folsom, & Benrud, 1980; Weeks, Kulka, & Pierson, 1987). 1 These findings seem reasonable; the likelihood of individuals being at home and available for answering a telephone interview, for example, is probably higher during weekday evenings and weekends.
In self-administered surveys, such as web surveys, however, it is more difficult to predict how critical the timing of the invitation is for the participation rate. With web surveys, the researcher has control over when the invitation is dispatched, but not when it arrives at the potential respondent, nor when the respondent sees the invitation. Self-administered surveys do not require the participant to be available immediately for responding to an invitation. Unlike telephone and face-to-face surveys, the response to an e-mail (or mail) invitation can be postponed to another, more convenient time (Keusch, 2015, pp. 190, 191). This suggests that invitation timing will matter less for participation rates in web surveys. On the other hand, individuals that receive a survey invitation at an inconvenient time can easily ignore the request or quickly forget that they have received it. When being approached directly by an interviewer over telephone or face-to-face, declining might be more difficult from a social point of view. This may increase the importance of invitation timing in web surveys.
Because web surveys differ from traditional survey modes—such as telephone interviews—in several respects, separate studies of timing effects in web surveys are warranted. Publications of such empirical studies are still relatively few; however, there are some examples that should be mentioned. These studies, which are all cross-sectional, and mainly American cases, have produced mixed findings.
In 2004, Faught, Whitten, and Green published findings from a study with 4,994 individuals sampled from e-mail lists of U.S. manufacturing firms, which showed that e-mail invitations sent on Wednesday mornings yield the highest participation rate. Another study, a meta-analysis of employee and customer surveys administered via SurveyMonkey, showed that e-mail invitations sent on Mondays yield the highest participation rate out of the five weekdays (Zheng, 2011). Yet another study, conducted with employees of the U.S. Department of Defense, showed that, compared with morning and afternoon times on Wednesdays and Thursdays, invitations sent on Tuesday mornings yield the highest participation rate (Lewis & Hess, 2017). In contrast to above studies, experiments conducted with scientists and engineers in the United States (Sauermann & Roach, 2013) 2 and agricultural education and extension journal authors (Shinn et al., 2007) 3 showed no effects of invitation timing, neither across the time of day nor the day of week. 4
While previous studies provide important empirical insights into the largely understudied role of timing effects in web surveys, there are several reasons why more research on this topic is needed. First, the results from previous studies are inconclusive regarding whether the timing of the invitation e-mails matter in web survey modes (Lewis & Hess, 2017, p. 354). Secondly, existing studies on timing effects in web surveys have mainly been conducted in an American context, with individuals from specific job sectors (where at least some can be suspected to work irregular hours and have continuous access to the Internet). This makes research in other contexts than the American, and with more diverse samples of individuals, warranted (Lewis & Hess, 2017, p. 361; Sauermann & Roach, 2013, p. 284). Thirdly, only the Lewis and Hess (2017), Sauermann and Roach (2013), and Zheng (2011) studies are recent enough to provide dependable information to today’s web survey practitioners, due to the significant, and rapid changes in online behavior the past decades.
Moreover, none of the previous studies on timing effects in web surveys were conducted with members of an online panel. Unlike cross-sectional surveys—to which participants are often recruited and invited to participate simultaneously—members in panels are, once they have accepted to join the panel, prepared for receiving repeated survey invitations (Evans & Mathur, 2005; Göritz, 2004). This could possibly affect the importance of timing for participation. With panel studies, which include information about the participants’ response time in prior waves, it is also possible to use this information to optimize the invitation timing in subsequent waves—something which has shown to be effective in panel surveys conducted over telephone and face-to-face (e.g., Kreuter & Muller, 2015; Lipps, 2012). 5
In addition to access to past response behavior, panel surveys have another advantage compared to cross-sectional surveys; they usually have extensive demographic background information about the respondents from previous waves. These variables can be used to predict the best invitation times for specific subsets of a sample, subsets which may be difficult to reach in certain time windows (Kreuter & Muller, 2015). A study by Bergmann and Scherpenzeel (2016) conducted face-to-face with members of a German survey panel showed that information about response patterns in previous waves could be used to tailor the contact timing for certain hard-to-reach groups, and thereby increase the participation rate among these specific individuals.
Our study, which is conducted with members of a Swedish online panel, contributes to previous research on timing effects in web surveys in several ways. First, it constitutes the (to our knowledge) first reported large-scale experimental study of timing effects in the web survey mode, combined with the use of prerecruited online panel members. Second, it is conducted with individuals of diverse labor market status and in another context than the American. Because of our large sample sizes, we are also able to test timing effects in different subgroups—something which should be of interest given that the optimal time to answer web surveys may differ between individuals with different sociodemographic characteristics.
Moderating Effects of Sociodemographic Characteristics
In the past decade, a growing number of scholars have emphasized the importance of thinking about survey representativeness in terms of nonresponse bias, not only in terms of response rate (e.g., Bethlehem et al., 2011; Groves, 2006; Groves & Peytcheva, 2008). This warrants studies that not only account for factors that can increase the total participation rate but also the heterogeneity of the survey sample. If certain subsets of a sample are hard to reach (and which may also have different response patterns on central survey questions), tailored survey designs that target subgroups differently should be of relevance. This has led to calls for more studies examining which groups are susceptible to which type of treatments (e.g., the timing of the survey invitation), to help reduce potential systematic nonresponse bias in surveys (Luiten & Schouten, 2013, p. 187; also see Durrant et al., 2011).
Most previous studies of timing effects in web surveys have focused on the average participation rate rather than sample heterogeneity (Sauermann & Roach, 2013, p. 274). Time availability and flexibility, however, differ across groups—something which is bound to have different effects on survey response. First, the amount of time and flexibility likely depends on external factors such as employment situation (e.g., if an individual work full-time, part-time, or is unemployed). In a post hoc analysis of six face-to-face UK Government surveys, for example, Durrant, D’Arrigo, and Steele (2011) found that contact calls were most successful during Saturday evenings in cases when the household included pensioners, and Bergmann and Scherpenzeel (2016) found that among employed, contact calls made during early evenings yield the highest participation rate.
The flexibility in time for answering surveys likely also depends on individual factors such as interest in the survey topic and age. Previous research has shown that interest in the survey topic can be predictive of participation (Groves, Presser & Dipko, 2004; Keusch, 2013). This, in turn, may indicate timing sensitivity; interested individuals may be more likely to prioritize the survey over other activities than those less interested in the topic. 6 When it comes to age, it is possible that younger individuals will be more flexible in terms of survey response than older. They (1) spend more time online (Bennett, Maton, & Kervin, 2008; Cheong, 2008), (2) are quicker in adopting new online technologies (Mannukka, 2007; Prensky, 2001), and (3) have better operational and formal skills in using various online technology (Prensky, 2001; van Deursen & van Dijk, 2010). 7 Previous studies of telephone and face-to-face surveys have indicated that, for individuals aged 15–44 (Wagner, 2013), and individuals aged 50–65 (Bergmann & Scherpenzeel, 2016), contact calls that were made in the early evenings yield the highest participation rates. However, these studies do not differentiate between younger individuals under, for example, the age of 30; hence, they do not allow for drawing inferences about timing effects among younger individuals compared to middle-aged and older individuals.
In this article, we focus on employment status and age as potential moderating variables of timing effects. We do not include interest in the survey topic (here political interest) because our panel is skewed toward highly interested, and ceiling effects make it difficult to accurately estimate timing effects for individuals with lower interest (for a breakdown of the samples on political interest, see Online Appendix A in the supplementing file accompanying this article). 8
Research Questions
Because of the overall difficulties of making theoretical predictions about effects of invitation timing in web surveys—and with the relatively sparse and inconsistent baseline empirical findings present—we use an exploratory approach, and formulate a set of research questions, rather than hypotheses, for our empirical analyses.
The first research question concerns the average effect of invitation timing on participation rates in web surveys. The question is split into (a) the effects of sending invitation e-mails at different days of the week and (b) the effects of sending invitation e-mails during different times of the day. More specifically, we ask: Q1: Can the [a: day of week] [b: time of day] for e-mail invitations to online panel surveys impact the participation rate?
These two questions resemble the questions asked previously by Faught et al. (2004) and Sauermann and Roach (2013). Are the findings of those studies generalizable to other contexts than the American, and does timing matter more in a more diverse sample? We ask the same questions, but this time in a web survey with prerecruited online panel members, on a sample of Swedish respondents, and with more diverse demographic backgrounds, spanning over a broader set of job sectors and employment status.
The following two questions concern timing effects of survey invitations on participation rate in different demographic groups. The distribution of available time for responding to surveys likely varies between individuals, in ways that can affect how critical the invitation timing is. As suggested earlier, the flexibility likely depends on both external factors, such as employment status, and internal factors, such as topic interest and age. In this study, we focus on employment status and age. Specifically, we ask: Q2: Do effects of [a: day of week] [b: time of day] for e-mail invitations on participation rate vary by employment status? Q3: Do effects of [a: day of week] [b: time of day] for e-mail invitations on participation rate vary by age?
Through the above questions, we explore whether there are reasons for survey practitioners to consider tailoring the timing of survey invitations to different individuals to increase participation. Another way of exploring these possibilities is to ask the panel members explicitly what day of week and at what time of day they prefer to answer surveys. As a complement to the experimental studies, we therefore ask the respondents at what time they prefer to answer surveys, and we investigate whether matching respondents’ preferences correlate with an increased participation rate within a week. Precisely, we ask: Q4: At what [a: day of week] [b: time of day] does online panel members prefer to answer web surveys? Q5: Does receiving an invitation e-mail at a preferred [a: day of week] [b: time of day] correlate with the likelihood of online panel members answering web surveys within a week?
Method and Data
To answer our research questions, we conducted two large-scale survey-embedded experiments in a Swedish online panel—the Swedish Citizen Panel (SCP). SCP is administered by the Laboratory of Opinion Research, a research infrastructure at the University of Gothenburg that collects web survey data. At the time of writing, SCP has approximately 55,000 active participants, including both nonprobability and probability-recruited members. Participation in the Citizen Panel is voluntary, and respondents do not receive payment for their participation (for more information, see www.lore.gu.se).
The samples in our studies consist of nonprobability recruited members of the panel, with overrepresentations of politically interested, highly educated middle-aged males (for breakdowns of the samples by demographic groups, see Online Appendix A).
Analyses and Results
Study 1: Day of Week
In the first experiment, we explored how different days of the week for the survey invitation influenced participation rate (Q1a). We also analyzed whether the effects differed by individuals’ employment status (Q2a) and age (Q3a), what day of week respondents preferred to answer surveys (Q4a), and whether receiving an invitation e-mail at the preferred day correlated positively with participation rate (Q5a).
A sample of 11,294 panel members participated in the experiment conducted between October 15 and November 19, 2014. Respondents were randomly assigned to one of the seven treatment groups, one for each day of the week. The first group received an invitation on a Wednesday (October 15, 2014, at 8 a.m.), and the last group received it on a Tuesday (October 21 at 8 a.m.). Reminders were sent out the same weekday a week later.
The dependent variable is the net participation rate in the survey (abbreviation “NPR” in tables and figures). 9 The NPR was calculated at two time points for each treatment group: 24 hr after the invitation and 6 days after the invitation. Employment status and age were included as interaction variables.
In addition to employment status and age, we included four control variables in the analyses, which could potentially influence at what time and on which days people are more likely to respond; type of e-mail address (private or nonprivate e-mail) and type of device that the participants commonly use when they answer the panel surveys, response propensity, and panel tenure. If the invitee uses a job e-mail address, for example, they may be more likely to respond if they receive the invitation during regular working hours and workdays; if the invitee uses a private e-mail, they may instead be more likely to respond to invitations received during evenings and weekends. Type of device may also impact at what time individuals are more likely to respond. A study of opening rates of e-mail newsletters, conducted by the Brafton Marketing Company, for example, showed that desktop and smartphone users most frequently opened e-mails during the midworkday, whereas tablet users more frequently opened e-mails during evenings (Kaye, 2013). Panel tenure is included because previous research has shown that panel tenure can take out effects of other variables on participation. Effects of topic interest, for example, seem to disappear when the panel members have been enrolled in the panel for 6 months or longer (see Keusch, 2013).
Did the day of week for the survey invitation affect participation rates?
The NPRs 24 hr after the invitation, and 6 days after the invitation, for each treatment group are presented in Table 1.
Participation Rate by Day of Week (Percent).
Note. Significant differences from Wednesday: †p < .1. *p < .05. **p < .01.
After 24 hr, the total mean participation rate was 35%. Monday to Thursday reached a participation rate between 1 and 4 percentage points above the mean, and Friday reached a participation rate equal to the mean. Saturday and Sunday yield the lowest participation rates, with 6 and 2 percentage points below the mean. Logistic regression analyses with Wednesday as the reference group (the group with the highest participation rate, 39%) showed a significantly higher participation rate for Wednesday compared to Tuesday, Friday, Saturday, and Sunday.
After 6 days, there were little differences in the participation rate between the treatment groups. The participation rate varied ±2 percentage points from the mean of 52%, and a χ2 test did not indicate significant differences. The analyses thus confirm timing effects on participation rate within 24 hr, but not in the long term, as the participation rate evens out within 6 days.
Did the effects of day of week for the survey invitation on participation rate vary by employment status and age?
To assess whether the effects of day of week for the survey invitation varied for individuals of different employment status and age, we conducted a multivariate logistic regression analysis. More precisely, we regressed participation rate on the day of the survey invitation (with Wednesday as the reference group), age, and employment status, and we included interaction terms for invitation day with age and employment status. Age was coded into three categories: 16–29, 30–64, and 65+, with 30–64 as the reference category. Employment status was coded as nonemployed (including students and pensioners), with employed as the reference category.
To account for variables that previous research has shown to be important for survey participation, we also included sex, level of education, type of e-mail address, response propensity, device, and panel tenure as controls. Results from the regression analyses are presented in Table 2.
Effects of Day of Week for Survey Invitation on Participation Rates by Employment Status and Age (Logistic Regression, Odds Ratios).
Note. Employment status was coded as employed (“full-time employment,” “part-time employment,” and “self-employed”) and nonemployed (“participating in labor market policy measures,” “unemployed,” “pensioner,” “illness/activity compensation,” “student,” and “other”). Level of education was coded as low (“not completed elementary school,” “elementary school,” “high school, less than 3 years,” and “high school 3 years or more”), middle (“postsecondary education (not college/university), less than 3 years,” “postsecondary education (not college/university), 3 years or more,” “university/college, less than 3 years”), and high (“university/college, 3 years or more,” “PhD”). Private e-mail addresses were derived from the e-mail servers the participants are utilizing in the panel (e.g., gmail.com, hotmail.com). Among the nonprivate e-mail server providers, there were a majority of easily identifiable workplace servers (a smaller number of observations represent unidentified specific servers, e.g., servers with proper names). Device was derived from user agent information and validated with screen resolution. Response propensity was calculated as the average response over every major wave of the Citizen Panel from 2011 to 2017 (17 waves in total; the average number of invites/panelist = 8, std. = 3.9). The correlation between response propensity up until the experiments in 2014 and the full period 2011–2017 is .77; 5,914 responses that lacked survey response device and 520 lacking employment information were imputed.
†p < .1. *p < .05. **p < .01. ***p < .001.
The analysis shows that when the effects of day of week on participation rates is tested while holding employment status and age, and other control variables constant, response propensity within 24 hr was significantly higher for invitations sent on a Wednesday than for invitations sent on a Friday, Saturday, and Sunday. After 6 days, however, there were no differences in participation between groups that received the invitation on a Wednesday compared to any of the other weekdays. The multivariate regression analyses thus confirm that there are some overall effects of the invitation day on participation rates after 24 hr, but which disappear as the field period progresses.
To explore the potential moderating effects of employment status and age, we estimated the predicted probabilities of responding by day of week for individuals in different age groups, and for employed and nonemployed individuals, while holding other variables at their means (predictions are based on the regression model presented in Table 2). The predicted probabilities of responding for individuals in different age groups are presented in Table 3.
Predicted Probabilities of Responding in Different Age Groups by Day of Week.
Note. For comparisons between all treatments pairwise, see Online Appendix B.
Significant differences from Wednesday: †p < .1. *p < .05. **p < .01. ***p < .001.
Table 3 reveals that in the short-term (within 24 hr), middle-aged individuals (30–64 years of age) are somewhat less prone to respond to surveys that are sent on a Friday, Saturday, or Sunday compared to Wednesday (our day of reference). Only Saturday yields consistently lower probabilities of responding compared to Wednesday when we control for differences after 6 days. For the younger age-group (16–29), and for older individuals (65+), there are no notable significant differences in predicted probabilities depending on dispatch day. However, it should be noted that the sample sizes are small, especially for the younger group (around 150 per treatment), which make these results more tentative. The predicted probabilities of responding for employed and nonemployed individuals are presented in Table 4.
Predicted Probabilities of Responding for Nonemployed and Employed by Day of Week.
Note. For comparisons between all treatments pairwise, see Online Appendix B.
Significant differences from Wednesday: †p < .1. *p < .05. **p < .01. ***p < .001.
Table 4 reveals that the probability of responding within 24 hr is the highest when invitations are dispatched on a Wednesday, both for nonemployed and for employed. For both groups, only Saturday and Sunday are significantly different from Wednesday. After 6 days, there are no notable differences in probabilities of responding between different dispatch days, neither for nonemployed nor for employed individuals.
In summary, Study 1 reveals that which day of the week a survey invitation is sent has no relevance for participation rates in the long term. In the short term, after 24 hr, weekends seem somewhat less convenient, but the differences between dispatch days disappear after 6 days. Subgroup analyses of individuals of different employment status simply confirm the general findings of invitation timing effects: Weekends are worse in the short run, however not after about a week and there are no notable differences between the subgroups. Analyses of conditional timing effects in different age groups indicate that middle-aged individuals are somewhat less flexible in responding to surveys that are dispatched on weekends in the short term (within 24 hr), but also these effects even out within a week.
What day of week does online panel members prefer to answer web surveys?
To explore what, if any, preferences the panelists have, we asked the following multiple-response question: “What day during the week would you like to respond to surveys from the Citizen Panel?” The participants’ day preferences for receiving a survey invitation are presented in Table 5. The respondents could choose as many of the 7 days during the week they preferred. Column 1 displays the preferences of all respondents, including those who stated a preference for all alternatives (i.e., a nonpreference), which was 28% of the responding sample. Columns 2 and 3 exclude respondents with three or more and two or more preferences, respectively. Two days seem to be more preferred than others across the three measures, Monday and Sunday.
Preferred Day of Week to Answer Surveys by Number of Stated Preferences (Percent).
Did receiving an invitation e-mail at a preferred day correlate positively with participation rate within a week?
Even though respondents’ preferred day was unknown before the study was conducted, a subsample was “unintentionally” matched, meaning that they received the invitation during the day they happened to prefer. This helps us assess whether there is a potential to optimize the day for e-mail invitations, by asking individuals what day they prefer to answer surveys.
We label the unintentionally matched group—52% of the total sample which had distinct preferences (two or fewer preferences)—as “optimized.” Table 6 reveals that the optimized group had a significantly higher participation rate (+26% higher) after 24 hr, compared to the “nonoptimized” respondents. After 6 days, the difference was still significant, but it had dropped to +7%. This indicates that matching respondents’ preferred response day with the day of the invitation e-mail can provide faster responses but that the effects decrease with time.
Participation Rate by Optimized Day of Invitation (Percent).
Note. The table excludes respondents with more than two preferences.
***p < .001.
To summarize, a nonnegligible number of individuals have stated preferences for specific days to answer web surveys, Mondays and Sundays in particular, and there are indications that matching respondents’ preferred day with the actual day for the survey invitation can increase participation rate, at least within a week. It needs to be noted here though that we asked respondents about their preferences in the same survey as we measured participation. This means that the correlations between the stated preferences and actual response time should be interpreted with caution, and not used to conclude causality (this would require a design with different time points for measurement). 10
Study 2: Time of Day
The second experiment was designed to test participation rate by what time during the day respondents received the survey invitation (Q1b). We also explored whether invitation timing matters differently depending on employment status (Q2b) and age (Q3b), what time of day respondents prefer to answer surveys (Q4b), and whether receiving an invitation e-mail at a preferred time correlates positively with participation within a week (Q5b).
A sample of 47,279 panel members received a survey invitation on a Thursday, November 27, 2014. To test Q1b, we randomly assigned respondents one of six different “time of day” conditions. The rationale of the chosen time treatments was the following: The day was split into seven periods, each representing a distinct part of the 24-hr cycle; early morning (6 a.m.–9 a.m.), late morning (9 a.m.–12 p.m.), lunchtime (12 p.m.–1 p.m.), afternoon (1 p.m.–5 p.m.), early evening (5 p.m.–8 p.m.), late evening (8 p.m.–11 p.m.), and nighttime (11 p.m.–6 a.m.). One third into each period was designated as the invitation time for each treatment group (e.g., a third into the first period 6 a.m. to 9 a.m. is 1 hr, that is, at 7 a.m.). The nighttime was excluded from the experiment, as we deemed sending e-mail invitations as late as 1:20 a.m. could be considered inappropriate.
Did the time of day for the survey invitation affect participation rates?
Table 7 shows the participation rate split by when respondents received the invitation. The total participation rate after 24 hr was 34%. Sending invitations at 12:20 p.m. resulted in the highest participation rate (35%); invitations dispatched at 9 p.m. yield the lowest (33%). Logistic regression analyses with 9 p.m. as the reference group (the group with the lowest participation rate) showed that the participation rate was significantly higher at 7 a.m., 10 a.m., 12:20 p.m., and 2:20 p.m. After 6 days, the total participation rate had increased to 48%, with a maximum variation between the treatment groups on 1%. A χ2 test showed no significant differences in participation rate between the treatment groups after 6 days. Similar to day of week, therefore, the findings indicate that invitation timing matters in the very short term, but the effects fade out as the field period progresses.
Participation Rates by Time of Day.
Note. Significant differences from 9 p.m.: *p < .05. **p < .01.
Did the effects of time of day for the survey invitation on participation rate vary by employment status and age?
Results from a multivariate logistic regression are presented in Table 8. The participation rate was regressed on the timing of the survey invitation (with 9 p.m. as the reference group), employment status, and age. Interaction effects between timing and age and between timing and employment status were included in the model. Further, sex, level of education, response propensity, device, and panel tenure were also included as control variables.
Effects of Time of Day for Survey Invitation on Participation Rates by Employment Status and Age (Logistic Regression, Odds Ratios).
Note. For coding of variables, see notes in Table 2.
†p < .1. *p < .05. **p < .01. ***p < .001.
First, the odds ratios above 1 in Table 8 indicate that, when employment status, age, and other control variables are held constant, it is more likely to get a response within 24 hr for invitations sent at 12:20 p.m., 2:20 p.m., and 6 p.m., compared to invitations sent at 9 p.m. After 6 days, there are no differences in participation rate between groups that received their survey invitation at 9 p.m., and at other times during the day. The multivariate analysis thus confirms some overall effects of invitation time on participation rate after 24 hr, but these effects are short termed only.
To explore the potential moderating effects of employment status and age, we estimated the predicted probabilities of responding by time of day for individuals in different age groups, and for employed and nonemployed individuals, while holding other variables at their means (predictions are based on the regression model presented in Table 8). The predicted probabilities of responding for individuals in different age groups are presented in Table 9.
Predicted Probabilities of Responding in Different Age Groups by Time of Day.
Note. For comparisons between all treatments pairwise, see Online Appendix D.
Significant differences from 9 p.m.: †p < .1. *p < .05. **p < .01.
Table 9 reveals that middle-aged individuals (30–64 years of age) and older individuals (65+) are somewhat less willing to respond to surveys that are dispatched at 9 p.m. than earlier during the day, but the differences are marginal and mostly nonsignificant. For the younger age-group (16–29), there are no indications of timing effects. The predicted probabilities of responding for employed and nonemployed individuals are presented in Table 10.
Predicted Probabilities of Responding for Nonemployed and Employed by Time of Day.
Note. For comparisons between all treatments pairwise, see Online Appendix D.
Significant differences from 9 p.m.: †p < .1. *p < .05. **p < .01.
Table 10 reveals that the probability of responding within 24 hr is lower for employed individuals when the invitation is dispatched at 9 p.m. compared with 12:20 p.m., 2:20 p.m., and 6 p.m. For nonemployed, 9 p.m. yield somewhat lower probabilities of responding in the short term (24 hr) compared with midday (12:20 p.m.). After 6 days, however, there are no differences in probabilities of responding between different dispatch times, neither for nonemployed nor for employed individuals.
To summarize, Study 2 reveals that at what time of the day a survey invitation is sent affected the participants in the short term, where 9 p.m. seem to be marginally less convenient than other times, but no differences could be discerned after 6 days. Closer analyses of subsample effects showed that timing mattered somewhat less for younger individuals and nonemployed, but they mainly confirmed the general findings. Since the overall timing effects are marginal in size, and that most subgroup differences disappear within a week, we conclude that there are no indications of substantial benefits of tailoring the time of day for the survey invitation to individuals of different employment status and age.
What time of day does online panel members prefer to answer web surveys?
To explore what time of day individuals prefer to answer web surveys, we asked the multiple-response question: “At what time during the day would you like to respond to surveys from the Citizen Panel?” The respondents were presented with seven different alternatives (see Table 11). The first column in Table 11 reports invitation preferences, including those who stated a preference for all alternatives (which was only 1%). Columns 2 and 3 exclude respondents with three or more and two or more preferences, respectively. The two evening periods (5 p.m. to 8 p.m. and 8 p.m. to 11 p.m.) seem to be preferable compared to other periods, irrespective of which measure we use.
Preferred Time of Day to Answer Surveys by Number of Stated Preferences (Percent).
Did receiving an invitation e-mail at a preferred time of day correlate positively with participation rate within a week?
Table 12 shows the participation rate for individuals who were unintentionally optimized (i.e., those who received their invitation at their preferred time) and the nonoptimized. The analysis yields no differences in participation rate between the nonoptimized and optimized, neither within 24 hr nor after 6 days. Hence, we conclude that there is a preference for answering web surveys during evening hours, but there are no indications that matching the timing of the invitation with respondents’ self-expressed preferred time to answer surveys would impact participation rate positively, neither in the short term nor after a week.
Net Participation Rate by Optimized Invitation Time During the Day, in Percent.
Note. The table excludes respondents with more than two preferences.
†p < .1.
Concluding Discussion
This article set out to explore whether the timing of survey invitations, in terms of day of week and time of day, can impact participation rates in web surveys with members of a standing online panel. The findings indicate that both the day of week and the time of day for the survey invitation can influence the participation rate somewhat in the very short term (within the first 24 hr), where the best time for sending invitations seem to be weekdays and between noon (12:20 p.m.) and early evening (6 p.m.). These initial effects are, however, short-lived, and they disappear as the field periods progress.
Closer analyses of the effects in different subsets of the samples show only a few effects of age and employment status, and these effects are marginal in size and generally even out after a week. Hence, we conclude that there are likely no substantial gains in tailoring the survey invitations to different subgroups, except maybe in specific cases when certain groups of individuals are targeted (e.g., middle-aged individuals in the middle of their career), and fast responses (within 24 hr) are important.
In addition to the findings regarding effects of the actual timing of the survey invitation on participation rates, analyses of respondents’ self-expressed preferences yield a somewhat greater preference to answer surveys during evenings and on Mondays and Sundays. Concerning day of week, the analyses also revealed a positive correlation between matching the day of week for the survey invitation with the respondents’ self-expressed day preference on the participation rate, when measured after a week. Concerning time of day, however, there were no indications that matching could affect participation rates even in the short term (within 24 hr).
The broader implications of the findings are that researchers and other practitioners using online panels for surveys cannot use the timing of invitation e-mails as a measure for improving the overall participation rate much. However, for those working with close deadlines that require fast responses, sending invitation e-mails during weekdays could potentially be used to increase, or speed up, the number of responses somewhat. In these cases, it is possible that matching the day of the invitation with individual panel members’ preferred day to answer surveys could increase participation to some extent; however, more research is needed to conclude causality here.
When using the findings of the present studies for drawing inferences about the best timing for survey invitations, we want to emphasize that there are a few shortcomings of the design that needs to be considered. First, we note that respondents may not receive the survey invitation at the same time as the survey company dispatches the invitation. Future research should thus explore invitation timing effects by measuring when the invitee receives and opens the invitation e-mail, for example, by using e-mail receipts.
We also acknowledge that respondents might be biased when they report the day and time they prefer to answer surveys. It is possible that respondents say that they prefer the time during which they responded in this survey, simply to rationalize why they are responding at that time—a so-called confirmation bias effect (see, e.g., Hahn & Harris, 2014; Taber & Lodge, 2006). If many respondents answered within 24 hr after receiving the invitation, this bias could have unjustly raised the number of “optimized” individuals in our sample, which would enhance the correlations seen in the study. For this reason, more studies are needed to conclude a causality here. In future experiments, invitation e-mails could be randomly sent to some individuals at their preferred time of day and preferred day of week and to some at a time and day that mismatches their preferred time to respond.
This article has provided insights into possibilities of improving (particularly short-term) participation rates in web surveys with prerecruited online panel members, by optimizing the timing of the survey invitations. However, more studies are needed to develop and confirm these initial results. Future studies should also explore things such as combinations of different times of the day and days of the week as there might be interaction effects that we are not able to investigate with the present research design. Studies of whether tailored invitation time to specific subgroups of the survey samples can reduce potential response biases in surveys should also be conducted, where effects on central outcome variables are compared between surveys with tailored invitation times and nontailored invitation times.
Supplemental Material
Supplemental Material, SSC810387_Supplemental - Invitation Timing and Participation Rates in Online Panels: Findings From Two Survey Experiments
Supplemental Material, SSC810387_Supplemental for Invitation Timing and Participation Rates in Online Panels: Findings From Two Survey Experiments by Elina Lindgren, Elias Markstedt, Johan Martinsson and Maria Andreasson in Social Science Computer Review
Footnotes
Authors’ Note
This article was presented in an earlier version at the 19th General Online Research Conference March 15–17, 2017, Berlin, Germany. The data for the studies can be obtained by e-mail, using the following e-mail address:
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
