Surveying Clinicians by Web

Advantages

A primary motivation for driving researchers to the web when conducting clinician surveys is the potential for reduced costs. Several studies have documented this advantage with experiments comparing costs across web and mail survey administrations (e.g., Raziano et al., 2001; VanDenKerkhof, Parlow, Goldstein, & Milne, 2004). Overall, studies have estimated the costs of conducting a web survey to be one third to one half of the costs of a comparable mail survey. For example, among dentists, both Schleyer and Forrest (2000) and Hardigan, Succar, and Fleisher (2012) reported that that web-based survey costs were roughly one third of mail survey costs. A similar differential was reported by Akl, Maroun, Klocke, Montori, and Schünemann (2005) in a survey administered to residents and medical faculty. We located one exception, Leece et al. (2004), who reported costs were higher for a web- versus mail-based administration of a membership survey of surgeons. In that study, the higher costs achieved by the web were due to expensive software fees, which may be a consideration for researchers without institutional access to free or inexpensive software.

Another advantage unique to web surveys is their potential for shorter field periods (Beebe, Locke, Barnes, Davern, & Anderson, 2007; Leece et al., 2004). Raziano et al. (2001) reported that the average response time for a web-based survey of geriatric chiefs was 18 days compared to 33 days for a conventional postal survey. VanDenKerkhof et al. (2004) compared response times from an experiment in which anesthesiologists were randomly assigned to complete a short web or mail survey. Results indicated that roughly half of the responses to the web survey (53%) were received in response to the initial mailing compared to only one third of responses (36%) submitted in response to the initial postal mailing for the mail group. Compared to mail, Akl et al. (2005) reported a significantly shorter response time of 3.8 days for residents and 8.4 days for faculty responding by web. Shorter periods for data collection may be a particularly important advantage of web surveys depending on the topic of the survey and the researcher’s need to obtain responses quickly (McMahon et al., 2003).

Because they rely on computer assistance in addition to self-administration, web-based surveys offer greater ease and adaptability in designing questionnaires than do paper surveys (Couper, 2008). Web surveys can accommodate a number of types of questions including radio buttons, list boxes, check boxes, and text fields for numeric and open-ended responses. Many of the commercially available products allow users to select the desired question type from a menu of options. Web surveys also offer more dynamic features such as the ability to play video, or use hyperlinks and pop-up windows. In addition, there are designs for which a web survey may be a more viable option, such as when (1) questions ask for sensitive information that might be provided less honestly to an interviewer; (2) researchers need to control the order in which questions are read and answered; (3) questionnaires include items with long lists of response options, or (4) questions contain conditional text in which information from sample files or previously administered questions is required and automatically filled.

While some studies that compare data quality in web versus other modes find no differences between the modes (e.g., Akl et al., 2005; Millar & Dillman, 2011), or that item-missing data is higher among respondents electing to complete the questionnaire online (e.g., Lusk, Delclos, Burau, Drawhorn, & Aday, 2007), the bulk of evidence indicates that increased flexibility in questionnaire design translates into gains in data quality as respondents often provide more complete data when responding online (e.g., Messer, Edwards, & Dillman, 2012). Schleyer and Forrest (2000) examined whether there were more unanswered items among dentists responding by web versus e-mail or fax, and found web responders completed significantly more fields on the questionnaire. McMahon et al. (2003) reported rates of unanswered questions were lowest among pediatricians receiving a web-based version of the survey (0.4%), followed by those who received a mail version (2.1%), and highest among those who got the survey via fax (2.8%).

An additional advantage to administering surveys by web is that they can be programmed to ensure that respondents are administered only those questions for which they are eligible, whereas paper surveys must rely solely on respondents to accurately follow instructions for skip patterns (i.e., items for which eligibility to answer a given question is contingent upon prior responses). Web questionnaires can also be programmed to reduce inaccurate, incomplete, and out-of-range responses. In a survey of pediatricians, McMahon et al. (2003) evaluated responses to questions that instructed respondents to “Select One” response. In contrast to a web-based administration of the questions for which respondents were prohibited from ever selecting more than one answer, 12% of respondents who received a mailed version of the questionnaire and 9% of respondents who received the survey by fax selected more than one answer. Web programming can also reduce inaccurate or item-missing data by providing a warning to respondents when they have left a question unanswered or somehow incomplete. For example, clinician surveys often ask respondents to provide distributions or tallies that sum to 100% (e.g., “In the last year, what percent of your professional time did you spend in the following: clinical practice, education, research, administration or other professional activities (Total 100%)?”) (Griffith, Cook, Guyatt, & Charles, 1999, p. 1002). A web-based application can provide feedback to respondents as they complete these items, verify that responses total 100%, and warn respondents when they do not.

In addition to lower costs, shorter field periods, and greater adaptability in designing questionnaires, web surveys offer other potential advantages in comparison to more traditional modes. Compared to mail surveys, responses to web surveys are compiled automatically, thereby eliminating the need for data entry and any errors due to manual data entry (Raziano et al., 2001; VandenKerkhof et al., 2004). When researchers have access to valid e-mail addresses, a web survey can be deployed to the sample instantaneously without constraints based on location or geography (Schleyer & Forrest, 2000). Finally, both the type and volume of paradata—data produced during the process of data collection, such as time respondents spend answering particular questions and other technical information—are more readily available with online surveys. These data can be used to evaluate the quality of the survey data collected and to make recommendations for improving the administration of the survey or design of the questionnaire (e.g., Couper, 1998).

Disadvantages

Web surveys are, however, associated with several disadvantages, including sample frame and coverage limitations, technology barriers, relatively lower response rates than those achieved using other modes, and the potential for nonresponse bias. When conducting a web survey with physicians or other clinicians, researchers will need to evaluate how representative their sample is of the population they wish to study. Coverage error—error that occurs as a result of a mismatch between the sample frame and the targeted population—may be “the biggest threat to inference in Web surveys” (Couper, 2000). While sample frames often contain clinicians’ mailing addresses and telephone numbers, the inclusion and accuracy of e-mail addresses is more variable (Dykema et al., 2011). If the missing contact information is correlated with other analytic variables, the representativeness of the survey data may be reduced. For example, in a study of Canadian anesthesiologists, VanDenKerkhof et al. (2004) found differences in age, years of experience, job setting (academic or nonacademic), and Internet use between those who provided e-mail addresses and those who did not. Further difficulties may arise when researchers attempt to locate e-mail addresses, which may not be published as widely or consistently as mailing addresses and phone numbers (Braithwaite, Emery, de Lusignan, & Sutton, 2003), and when researchers attempt to identify the “best” e-mail address to use among health professionals with multiple listings (Klabunde et al., 2012).

When conducting web surveys with clinicians, coverage error can be exacerbated by patterns of technology use and by disparities in the technology itself. Unique to web surveys is the need to manage and accommodate different usage patterns and levels of computer literacy among clinicians, in addition to variation across operating systems (Ahern, 2005; Schleyer & Forrest, 2000; Whitehead, 2007). In general, technical inconsistencies or problems, such as the lack of compatibility across web browsers used by respondents, can be barriers to accessing, viewing, completing, or submitting a web questionnaire (Braithwaite et al., 2003; Guise, Chambers, Välimäki, & Makkonen, 2010; Leece et al., 2004). In addition to choice of web browser, variation in computer performance can occur. Some of the aforementioned advantages of web surveys may have negative side effects purely through the inclusion of these technologies. For example, advanced web programming features, such as Java and video, may slow down and lock up computers leading to reduced participation (Schleyer & Forrest, 2000; Swartz & Hancock, 2002), and security settings may block access.

Advantages of web surveys may also be reduced by response rates among clinicians that are frequently lower than for mail, phone, or mixed-mode (e.g., mail/web) designs (McLeod, Klabunde, Willis, & Stark, 2013; VanGeest & Johnson, 2011; VanGeest, Johnson, & Welch, 2007). While there is substantial variability in the range of response rates reported in Internet surveys of health professionals (e.g., Braithwaite et al., 2003), rates of under 20%, particularly for physician surveys, are not uncommon (Golnik, Ireland, & Borowsky, 2009; Rodriguez et al., 2006; Yusuf & Baron, 2006). Higher response rates (e.g., in excess of 50%) tend to occur in studies that use additional techniques for securing participation, such as sending nonresponders a paper copy of the questionnaire (e.g., Beebe et al., 2007), or attaching a copy of the questionnaire to the e-mail invitation (e.g., Raziano et al., 2001) or in studies in which the sample includes clinicians with prior and demonstrated experience using the Internet (e.g., Potts & Wyatt, 2002).

Several studies make direct comparisons between a web-only or web-also ¹ mode of administration versus administration using a different mode. The vast majority of studies report lower response rates for web-based clinician surveys (see Table 1). ² These studies of clinicians are consistent with meta-analyses comparing web to other modes across varied populations which show that, on average, response rates for surveys administered by web are around 10% lower (Manfreda, Bosnjak, Berzelak, Haas, & Vehovar, 2008; Shih & Fan, 2008). Similar findings are reported in the meta-analysis of clinician surveys reported by Cho, Johnson, and VanGeest (2013).

OPEN IN VIEWER

Table 1.

Comparison of Response Rates Between Web-Only/Web-Also Surveys Versus Mail Surveys.

Source	Sample members			Response rates
		N	Number of contacts	Web-only/web-also (%)	Mail (%)	Difference
Raziano, Jayadevappa, Valenzula, Weiner, and Lavizzo-Mourey (2001)	Geriatric medicine division chiefs	114	3	58	77	−19
McMahon et al. (2003)	Pediatricians	300	2	26	41	−15
Leece et al. (2004)	Surgeons	442	3	45	58	−13
Losch, Thompson, and Lutz (2004)	Primary care physicians	750	2	13	23	−10
VanDenKerkhof, Parlow, Goldstein, and Milne (2004)	Anesthesiologists	1,333	4	35	69	−34
Akl, Maroun, Klocke, Montori, and Schünemann (2005)	Residents	119	3	63	80	−17
	Medical faculty	83	3	85	81	+4
Beebe, Locke, Barnes, Davern, and Anderson (2007)	Physicians with an appointment at Mayo	500	2	38	37	+1
			3	47	57	−10

For example, in a survey of geriatric medicine division chiefs, Raziano et al. (2001) assigned sample members to a postal mailing group or an e-mail group that could respond online or by completing an e-mail attachment. The resulting response rate for the e-mail group was 58%, significantly lower than the 77% rate achieved by the mail group. McMahon et al. (2003) compared postal, fax, and web-based methods in a survey of pediatricians. The response rate after two contact attempts was 41% for the postal, 47% for the fax, and 26% for the web-based group. Leece et al. (2004) reported that surveying surgeons by web yielded a significantly lower response rate after four contact attempts of 45% versus 58% by mail. Losch, Thompson, and Lutz (2004) compared response rates for a state-based sample of primary care physicians in which sample members were mailed a copy of the questionnaire or instructions for completing the survey online. After the initial mailing and one reminder, response rates for the paper condition were nearly twice those of the web condition, 23% versus 13%. VanDenKerkhof et al. (2004) compared response rates from a randomized trial in which anesthesiologists were randomly assigned to complete a short questionnaire online or on paper. Results indicated that participants in the web survey group were roughly half as likely to participate as the mail survey group, 35% versus 69%.

Findings from Akl et al. (2005) and Beebe et al. (2007) suggest that variation across types of sample members and design features may contribute to variability in response rates for clinician web surveys. For example, Akl et al. reported that a web survey resulted in a significantly lower response rate than a mail survey among residents (63% by web vs. 80% by mail), but that response rates among faculty did not differ significantly between the two modes (85% by web versus 81% by mail). The authors speculated that the lower response rate for residents responding online may have been due to the inclusion of institutional e-mails addresses that were not checked regularly. Beebe et al. randomly assigned physicians at the Mayo Clinic to receive a questionnaire through interoffice mail or by clicking on a link in an e-mail invitation. After the initial invitation to participate in the study, response rates for the web and mail groups were similar, 38% versus 37%. However, following a second contact in which sample members were reminded to complete the survey if they had not already done so, a 10-point difference in response rates emerged such that the response rate in the mail group rose to 57% compared to only 47% in the web group. It is instructive to note that in order to compare response rates between the two conditions, researchers did not send an additional copy of the survey with the postcard reminder, and the web-based reminder did not include a link to the web survey. Excluding the link may have artificially reduced response for the web as best practices would recommend including the link with all e-mail reminders.

Response rates, however, are just one indicator of overall data quality and a “low” response rate does not necessarily indicate inferior data. Even with a high response rate, nonresponse bias may be large if those surveyed differ substantially from those who are not. Conversely, nonresponse bias may be small, even with a low response rate, if respondents are similar to nonparticipants on the characteristics of interest (Biemer & Lyberg, 2004). In their review of methods to increase participation in physician surveys, Flanigan, McFarlane, and Cook (2008) report that, in general, nonresponse bias may be lower in surveys of physicians than in surveys of other populations, possibly because physicians are relatively homogeneous (Kellerman & Herold, 2001). When documented using traditional mail survey methods, nonresponse bias has been suggested in studies that find women, younger physicians, nonspecialists, and recently licensed physicians more likely to respond (Barclay, Todd, Finlay, Grande, & Wyatt, 2002; Cull, O’Connor, Sharp, & Tang, 2005). Little research, however, has examined nonresponse bias in web-based provider surveys. Given the consistently lower response rates for web surveys among studies that have directly compared mail to web, there is reason to be cautious.

While few studies assess nonresponse bias directly, several studies have examined demographic representativeness by comparing the distribution of select respondent characteristics from survey data collected by web versus mail to the distribution of these characteristics in administrative data. Beebe et al. (2007) reported that regardless of whether they responded by web or mail, the distribution of age, gender, and tenure among physicians in the survey data mirrored the administrative data. In contrast, specialists were significantly underrepresented when the survey was administered by web only, a differential that was reduced by following the web administration with a postal mailing in a mixed-mode design. Geisen, Murphy, Olmsted, and Severance (2010) examined outcomes among physicians randomly assigned to one of the three groups with: a “web-always” group in which sample members had the option of responding by web in all mailed contacts, a “mail/web” group in which sample members who did not initially respond by mail were later provided with a web option, and a “mail-only” group in which sample members could only respond by mail. They reported no differences between responders and nonresponders by mode for age, gender, or practice type.

Studies have also tried to address the impact of mode on the propensity to respond by comparing characteristics and responses to survey questions among clinicians responding by web with those participating in another mode, such as mail or fax. Drawing generalizations across these studies is complicated by several factors including: (1) variability among the studies with regard to whether or not sample members are randomly assigned to the mode comparisons versus simply being provided with the choice to complete the survey using different modes; (2) the confounding of mode with early versus late responding when modes are introduced sequentially (e.g., Ahlers-Schmidt et al., 2010); and (3) small sample sizes for the comparisons, which make it more difficult to detect effects (e.g., Akl et al., 2005). Given these caveats, studies of clinicians have, in general, found few significant differences between responders completing by web versus another mode in terms of gender, age, specialization, years in practice or since graduation, practice type, geographic location, and in response to select, topically-based survey questions (e.g., Ahlers-Schmidt et al., 2010; Akl et al., 2005; Beebe et al., 2007; Geisen et al., 2010; Guise et al., 2010; Leece et al., 2004). This is not to say that there are never differences. For example, Beebe et al. (2007) found physicians who reported spending more than half a day on the Electronic Medical Record were “significantly overrepresented when the survey was administered by web-only.” Lusk et al. (2007) employed a mixed-mode design in which sample members were contacted by mail at every point but given the option of completing by web. They found some evidence to indicate that those who elected to complete by web were more likely to be male and younger. McMahon et al. (2003) compared outcomes for sample members randomly assigned to complete the survey by mail, fax, or web. They reported no differences among groups with regard to practice setting or location, Medicaid acceptance, or having a computer at work, Internet at work, or Internet at home. Sample members responding in the web group, however, reported graduating more recently and were more likely to report checking e-mail daily and downloading information.

Incentives

Researchers often make use of incentives to increase survey participation. Incentives may also have other beneficial effects, such as decreasing costs and item nonresponse, and increasing sample representativeness, data quality, and the likelihood that sample members less inclined to participate (e.g., those for whom the survey’s topic is less interesting) return a completed survey (Groves, Singer, & Corning, 2000; Singer, 2002; Singer & Kulka, 2002). When contemplating the inclusion of an incentive in a survey, researchers are well advised to consider the major dimensions along which the type and administration of the incentive can vary as these may determine the incentive’s effectiveness (Church, 1993; Edwards et al., 2009; Klabunde et al., 2012; McLeod et al., 2013; Singer & Ye, 2013). The first dimension concerns how liquid the incentive is, that is, whether the incentive is monetary (e.g., cash or a check), nonmonetary (e.g., a gift or the results of the study), or something in between (e.g., a gift card to a specific store or website, or a charitable donation made on behalf of the clinician). Monetary incentives are typically more effective than nonmonetary incentives with cash often affecting participation more than other kinds of payments. A second dimension includes the timing of the incentive and whether it is “prepaid” and given to sample members in advance of completing the survey versus “promised” and offered contingent upon completion of the survey. Experimental results from mail and electronic surveys indicate that prepaid incentives are more effective than promised incentives in increasing survey participation (Church, 1993; Edwards et al., 2009). A third consideration is the actual amount or value of the incentive. While studies indicate that larger amounts are often associated with higher response rates (Church, 1993; Singer, Van Hoewyk, Gebler, Raghunathan, & McGonagle, 1999), the relationship between these variables is likely nonlinear. The mode of administration for the survey (and incentive) encompasses the fourth dimension. Whether the survey is delivered by web, mail, phone, fax, in person, or using some other method may create barriers for administering certain types of incentives.

Summarizing results from studies that test the impact of incentives in surveys of physicians and nurses, researchers report that small, prepaid monetary incentives are particularly effective in increasing response rates when compared to groups that are not offered incentives (see reviews in Flanigan et al., 2008; Kellerman & Herold, 2001; VanGeest & Johnson, 2011; VanGeest et al., 2007). Less effective (and often more costly) are monetary incentives that are promised, nonmonetary incentives, and sweepstakes or lotteries. Findings for the effectiveness of small, prepaid incentives are consistent with the theory of social exchange, which posits that individuals are more likely to respond positively to a request when they trust the originator of the request and perceive the ratio of rewards to costs to be personally acceptable (Dillman, 2007). In comparison to not offering an incentive, a small, prepaid amount is effective because it is viewed as a “token of appreciation” that motivates the recipient to respond in a reciprocal manner.

The vast majority of incentive-based experimental studies conducted among clinicians have used mail as the mode of administration. As of yet, few studies examine the influence of incentives in provider surveys that rely primarily or exclusively on the Internet as the sole mode of administration. When data collection moves from a postal to a web administration, significant constraints to delivering incentives are encountered. Prepaid incentives are difficult to deliver effectively in web surveys that use e-mail to recruit participants. For example, it is not possible to administer cash using e-mail and alternatives, such as electronic gift certificates or online banking (e.g., PayPal), may not be as effective (Birnholtz, Horn, Finholt, & Bae, 2004; Bosnjak & Tuten, 2003). Further, methods found to be successful with mail surveys may not directly transfer to web surveys. For instance, Dykema et al. (2011) included a $2 bill in a postal letter sent to physicians notifying them that they would be sent an e-mail invitation to complete a web survey. Administering the incentive in this manner had no effect on response rates.

For web surveys, this complexity in incentive administration may require an increase in the amount of the incentive in order to have a strong influence on participation. As amounts increase, they may invoke sentiments of economic exchange and be viewed as providing payment for the request to complete the survey (Sauermann & Roach, 2013). This may be especially true for physicians who are often paid amounts in excess of $100 for participating in research and who perceive there to be substantial demands upon their time to participate in surveys (Klabunde et al., 2012).

Recommendations about the optimal amount for a monetary incentive are not possible as the research available is insufficient to draw definitive conclusions (Klabunde et al., 2012; McLeod et al., 2013). While larger amounts often increase response rates, no logical cap has been identified, and studies, particularly among physicians, are contradictory. For example, Asch, Christakis, and Ubel (1998) found $5 yielded a higher response rate than $2, but Mizes, Fleece, and Roos (1984) reported that $5 was not better than $1. While Halpern, Ubel, Berlin, and Asch (2002) showed $10 resulted in significant improvement in response rates over $5, VanGeest, Wynia, Cummins, and Wilson (2001) reported that there were no significant differences among physicians receiving prepaid amounts of $5, $10, and $20, and further, response rates were highest in the $10 group. Similarly, Kasprzyk, Montaño, St. Lawrence, and Phillips (2001) reported that a prepaid $25 check was not significantly more effective than $15. In studies that test larger amounts, however, Gunn and Rhodes (1981) reported a marginally significant higher response rate among physicians promised $50 versus $25, and Keating, Zaslavsky, Goldstein, West, and Ayanian (2008) reported a significantly higher response rate among physicians mailed a prepaid check for $50 versus $20. In a survey of general internists randomly sampled from the AMA Masterfile, Dykema et al. (2011) reported that the response rate was significantly higher among physicians promised $100 versus $50. With the exception of Dykema et al., the preponderance of studies have been conducted by mail using prepaid incentives.

Researchers are likely to consider employing a prize draw or sweepstake ³ as an incentive in a web survey (Göritz, 2010). These are frequently used with web surveys because they offer a relatively inexpensive and fixed-cost alternative (i.e., the amount spent on a sweepstake does not increase with larger sample sizes). Further, they are easy to administer effectively with a web survey; regardless of whether sample members are recruited using an e-mailed or mailed invitation, a statement alerting potential respondents to the incentive is effortless to insert. The major drawback to prize draws is that their actual influence on survey participation tends to be small and often equivalent to not offering an incentive (Göritz, 2006; Singer & Ye, 2013). This is likely due to the fact that in addition to being a type of promised or contingent incentive, which are typically less effective than prepaid incentives, the provision of a reward through a sweepstake is not guaranteed (i.e., not all sample members who complete the survey will receive an incentive). Dykema et al. (2011) found that a $200 prize drawing when used as an incentive in a web survey had no impact on response rates among physicians over a control group that was not offered a promised incentive. For physicians and other clinicians, the value of the incentive employed in a sweepstake may need to be large in order to be effective. Consistent with this assertion, Sauermann and Roach (2013) recommend that for a fixed value, a prize draw may be more effective if it is divided into a few larger amounts than several smaller amounts.

Contacts: Prenotification, Invitation, and Reminders

Researchers will need to make several decisions about the type, timing, number, and nature of the requests they make of clinicians to participate in a web survey including: (1) whether to provide sample members with advance notice—such as by e-mail, mail, or phone—of the upcoming web survey invitation; (2) what mode or combinations of modes to use for prenotification, invitation, and reminders; and (3) what information these various contacts should include and how to format this information. These decisions will likely be based in part on the researcher’s budget and on the type and quality of information available in the sample frame.

With regard to prenotification, the bulk of evidence from population- and provider-based surveys suggests that contacting sample members beforehand about an upcoming survey is most effective when the mode for prenotification is different from the mode of survey administration (de Leeuw, Callegaro, Hox, Korendijk, & Lensvelt-Mulders, 2007; VanGeest & Johnson, 2011; VanGeest et al., 2007). For example, prenotification of respondents in a mode other than the web, such as using a postal letter, has been shown to increase response rates in web-based surveys of students (Crawford et al., 2004), grant applicants (Harmon, Westin, & Levin, 2005), and university faculty (Dykema, Stevenson, Klein, Kim, & Day, 2013). While less is known about the effectiveness of precontacting clinicians by mail for a web survey, Dykema et al. (2011) found that the addition of a postal prenotification letter significantly increased response rates in a web survey of physicians.

The two primary options for inviting sample members to participate in a web survey are e-mailed and mailed invitations (Dykema et al., 2013). E-mailed invitations, in which a hot-linked URL is embedded in an e-mail’s text, are most commonly used due to their ease of sending, ease of responding, and low costs. Mailed invitations, in which an URL is included on a postcard or in a letter that respondents must manually type into a browser, are less common because they require a mailing address, require more effort by sample members in order to respond, and are more costly.

When sample frames include both e-mail and mailing addresses and budgets permit, we recommend first sending a postal letter, but also including the URL in the letter. This hybridization of prenotification and invitation serves the dual functions of alerting providers to the forthcoming e-mail invitation and of offering motivated sample members the opportunity to participate immediately (though admittedly with some additional effort). This hybrid prenotification/invitation technique may offer several advantages. The mailed letter is likely to both highlight the upcoming web survey and lend added legitimacy to the survey effort overall. Clinicians may be less likely to regard the subsequent e-mail invitation as spam because of the letter. Preceding the e-mail invitation with the postal mailing may increase participation if the two different modes of contact differentially affect the likelihood of the providers’ choice to participate. Inviting sample members using a mailed invitation with an URL allows researchers to couple small, cash incentives that have been so effective in postal surveys with a method for accessing and completing the survey. For example, in a study of the general population, Messer and Dillman (2011) reported that including a $5 incentive in a mailed invitation to complete a web survey significantly increased the rate of participation by 18 percentage points over not including an incentive (see too Dykema et al., 2013).

The use of the hybrid prenotification/invitation technique also offers advantages related to practical issues when administering web surveys with health providers. Depending on the clinical population being studied and the source of sample, the quality of e-mail addresses may be highly variable. When available e-mail addresses are not accurate or missing, the addition of a mailed prenotification/invitation could increase response rates by increasing the likelihood that the selected sample members receive a method for participating. Similarly, mailed invitations could also increase response rates by offsetting e-mail notifications that are sent to inaccurate e-mail addresses, screened by staff, left unread, or diverted by spam filters.

Similar to studies of the general population, studies of clinicians provide strong evidence that multiple contacts ultimately increase response rates (e.g., Klabunde et al., 2012). Sending e-mailed follow-up reminders to sample members who fail to complete a web survey after the initial invitation is a cost-effective strategy to improve survey participation. In a survey of dentists, Schleyer and Forrest (2000) reported a response rate of 33% after the initial e-mail invitation. The response rate increased to 50% following the first reminder, to 57% after the second reminder, and to 64% after the third reminder. Increased response rates for web surveys following e-mailed reminders have also been reported for many groups of providers including physicians (e.g., Beebe et al., 2007) and nurses (e.g., Guise et al., 2010).

Another effective, but usually more costly follow-up technique, includes varying the mode in which the reminder contact is delivered, such as requesting participation from nonresponders by sending them a mailed reminder or by placing a reminder phone call (Dillman, 2007; McLeod et al., 2013). For example, even after sending nonresponding physicians four e-mailed reminders, Dykema et al. (2011) substantially increased their overall response rate by sending nonresponding physicians a final postal letter that included a printed URL, which sample members manually typed into a browser in order to access the web survey. Researchers should also vary the type of appeal made across contact attempts in an effort to bring in sample members with different motivations for participating (Dillman et al., 2009; Groves, Cialdini, & Couper, 1992; Schaeffer, Dykema, & Maynard, 2010). By combining, for example, an initial contact attempt that emphasizes the utility and value of the research with a subsequent contact that emphasizes a personal appeal for help, researchers may both increase response rates and the breadth of sample members that choose to participate.

Research indicates that the content, format, and design of e-mail and postal contacts can have an effect on survey participation (e.g., Kaplowitz, Lupi, Couper, & Thorp, 2012). We make several general recommendations (see too Dillman et al., 2009; Schleyer & Forrest, 2000; Sprague, Quigley, & Bhandari, 2009): (1) personalize salutations for invitations and follow-up requests; (2) state the purpose and utility of the study clearly and briefly; (3) provide information about the sponsor of the research project and who to contact if more information is desired; (4) ensure confidentiality explicitly (or anonymity if appropriate); (5) include a hot-linked URL in e-mail correspondence and a manual-entry URL in mailed correspondence; (6) avoid terms used by spammers in the subject line and in body of e-mail (e.g., “opportunity” and “click”); (7) avoid sending e-mail messages from a generic account; and (8) include an inviting and appropriate subject line for e-mail messages.

Contacts: Mixed-Mode Approaches to Collecting Survey Responses

Many provider surveys incorporate a mixed-mode approach in which clinicians are given the opportunity to complete the survey in more than one mode, such as supplementing a web survey with the option to complete the survey by mail, fax, e-mail, phone, or in person. The overarching rationale for implementing a mixed-mode design “is that the drawbacks of one mode can be compensated for by the strengths of another” (Couper, 2011, p. 898). Mixed-modes designs may be implemented in an effort to increase response rates and sample representativeness, decrease nonresponse bias, increase coverage of the target population or subgroups, reduce measurement error, and reduce costs (Beebe et al., 2007; de Leeuw, 2005). Mixed-mode designs may be particularly beneficial for web-based provider surveys because there may be systemic barriers to contacting providers using e-mail, providers may encounter technical problems accessing and completing a web survey, providers may have strong preferences for mail or some other mode, and modes other than web may be more effective at overcoming gatekeepers (e.g., Raziano et al., 2001).

Web-based, mixed-mode designs can take many forms because researchers can offer sample members the opportunity to respond online in several different ways. There are two important dimensions to consider when mixing different modes. The first dimension concerns the order in which modes are offered, either sequentially (one after another) or concurrently (at the same time). The second dimension encompasses the amount of involvement sample members are given in selecting a mode and whether modes are assigned to sample members randomly, offered as a choice among available options, or offered as a choice based on previously stated preferences (Couper, 2011; Olson, Smyth, & Wood, 2012). Here, we summarize findings for three of the dominant approaches found in the provider-based literature.

In the sequential, web-other (e.g., web–mail, web–fax) approach, the survey is initially administered using predominantly web-based methods with mail or another mode offered later in an effort to convert nonresponders. Several provider studies demonstrate that this is a particularly effective strategy for increasing response rates over single-mode web surveys. For example, after three e-mailed requests to participate in a web survey, Raziano et al. (2001) obtained a response rate of 58% among geriatric chiefs. Following these requests with a conventional postal mailing raised the overall response rate to 83%, an increase of 25 percentage points. McMahon et al. (2003) increased their response rate among pediatricians by 21 points—from 26% to 47%—by following two e-mailed requests with a third contact that used a combination of techniques including a postal fax-based mailing. Leece et al. (2004) reported a response rate of 45% among surgeons after three e-mailed contacts. They increased their response rate by 19 percentage points (to 64%) after sending nonresponders a paper survey. Beebe et al. (2007) noted a 47% response rate in a web survey of physicians after two contact attempts. Sending nonresponders a paper questionnaire resulted in an increase of 16 percentage points to 63%. After three e-mailed contacts, Ahlers-Schmidt et al. (2010) reported a response rate of 33% among physicians. Follow-up to nonresponders by fax increased the response rate to 50%, and a final contact by phone led to an overall response rate of 69%. Guise et al. (2010) reported that the web survey response rate for a sample of nurses was only 20% following several contacts. Contacting nonresponders using a mixed-mode approach that included a conventional postal mailing and additional e-mail notifications raised the final rate to 55%.

In the sequential, other-web (e.g., mail–web) approach, the survey is initially administered in another mode, such as mail, with web offered later in an effort to convert nonresponders. Findings from studies that combine modes using this approach indicate that increases in response rates due to the web option are likely to be small. For instance, Raziano et al. (2001) reported only a slight increase in response (from 77% to 81%) when geriatric chiefs who had received three requests to participate using conventional mail survey methods were asked to respond by e-mail. While Beebe et al. (2007) reported that following a mailed contact with a request to complete an online version of the survey increased their response rate from 57% to 71%, a quarter of the completed surveys were not completed online, but instead filled out using the original paper questionnaire. Note that if the primary goal is to maximize rates, research indicates that a mail–web design is likely to yield a higher response rate than a web–mail design (Beebe et al., 2007; Converse, Wolfe, Xiaoting, & Oswald, 2008; Stevenson, Dykema, Kniss, Black, & Moberg, 2011). With both the web-other and other-web approaches, it is often unclear if the boost in response rates was due to the change in mode or simply because respondents received an additional request to participate.

The first two approaches described, the web-other and the other-web approach, involve a sequential implementation of modes. In a third approach found in the literature, the option to complete the survey online is offered simultaneous to another mode (or modes) as a choice. Losch et al. (2004) compared response rates among primary care physicians in which sample members were assigned to a “web” condition that included mailed instructions for completing the survey online, a “paper” condition that included a mailed copy of the questionnaire, or a “choice” condition in which they were sent a paper questionnaire and written instructions for the option of completing the survey online. While the response rate for the web condition (13%) was substantially lower than the choice (24%) or paper (23%) conditions, the choice option did not increase participation over the paper option (see too Grava-Gubins & Scott, 2008). Similar results were reported by Scott et al. (2011) who reported that an online approach resulted in a lower response rate (13%) than either a web/mail choice approach (20%) or a web–mail sequential approach (21%), but response rates for the latter two approaches were similar. Geisen et al. (2010) mailed physicians a paper questionnaire and allowed them to return the survey by mail or fax (mail-only) versus also including written instructions for completing the survey online (web-always) versus only including the online option in subsequent mailings sent to nonresponders (mail/web). Findings indicated that providing a web-based option had no impact on response rates, which were 52%, 50%, and 51%, respectively. In a recent meta-analysis conducted with a range of populations, Medway and Fulton (2012) found response rates were significantly higher for single-mode mail surveys than for mail surveys that offered sample members the option of completing the survey online.

These findings are consistent with research involving mode preferences, which suggest that physicians favor mail over other modes (e.g., Olson, Srinath, Burich, & Klabunde, 2000). These results also highlight an important takeaway from research examining mode choice among health professionals. When simultaneously provided with a paper questionnaire and the option to complete an online instrument, clinicians overwhelmingly select the paper option (e.g., Keating et al., 2008; Losch, Thompson, & Lutz, 2004; Lusk et al., 2007; Recklitis, Campbell, Kutner, & Bober, 2009).

Questionnaire Length

In their systematic review of methods to increase response rates in surveys of physicians, VanGeest, Johnson, and Welch (2007) found a positive association between shorter questionnaire lengths and survey participation. None of the studies included in their review were web based, and we were not able to locate any studies that experimentally manipulated the length of a single-mode web survey among clinicians. However, consistent with findings from VanGeest et al. are studies that examine questionnaire length for online surveys administered in nonclinical settings, which find that respondents are less likely to start and complete longer (e.g., 30 min or more) versus shorter questionnaires (e.g., 10 min or less; Crawford, Couper, & Lamias, 2001; Deutskens, De Ruyter, Wetzels, & Oosterveld, 2004; Galesic & Bosnjak, 2009; Marcus, Bosnjak, Lindner, Pilischenko, & Schütz, 2007). The process by which sample members estimate length is different for web surveys than other modes of survey administration. With a mailed survey, sample members can examine the printed version. In contrast, the length of a web survey can be conveyed by including an explicit statement about the survey’s length, through the use of progress indicators, by scrolling or paging through the survey, or by completing the survey. In order to maximize participation—both encouraging sample members to start the web survey and ensuring that they complete it—we recommend that researchers keep their questionnaire as short as possible (Couper, 2008). In addition, researchers should provide sample members with an accurate estimate of the time it will take to complete the survey at the outset (Kaplowitz et al., 2012).