Adding Postal Follow-Up to a Web-Based Survey of Primary Care and Gastroenterology Clinic Physician Chiefs Improved Response Rates but not Response Quality or Representativeness

Abstract

This study assessed whether postal follow-up to a web-based physician survey improves response rates, response quality, and representativeness. We recruited primary care and gastroenterology chiefs at 125 Veterans Affairs medical facilities to complete a 10-min web-based survey on colorectal cancer screening and diagnostic practices in 2010. We compared response rates, response errors, and representativeness in the primary care and gastroenterology samples before and after adding postal follow-up. Adding postal follow-up increased response rates by 20–25 percentage points; markedly greater increases than predicted from a third e-mail reminder. In the gastroenterology sample, the mean number of response errors made by web responders (0.25) was significantly smaller than the mean number made by postal responders (2.18), and web responders provided significantly longer responses to open-ended questions. There were no significant differences in these outcomes in the primary care sample. Adequate representativeness was achieved before postal follow-up in both samples, as indicated by the lack of significant differences between web responders and the recruitment population on facility characteristics. We conclude adding postal follow-up to this web-based physician leader survey improved response rates but not response quality or representativeness.

Keywords

physicians surveys survey methods respondents data quality organizational structure colorectal neoplasms

Introduction

Previous research has documented that response rates for physician surveys are typically lower than response rates for general population surveys (Asch, Jedrziewski, & Christakis, 1997; Cummings, Savitz, & Konrad, 2001). The highest response rates for physician surveys have been derived from postal and telephone administration methods (McLeod, Klabunde, Willis, & Stark, 2013; VanGeest, Johnson, & Welch, 2007). In both physician and general population samples, however, web-based surveys are increasingly employed due to several advantages they hold for both researchers and respondents over postal and telephone surveys. For researchers, advantages of web surveys include lower cost (Cobanoglu, 2001; Couper, 2000; Dillman, 2000; Dykema, Jones, Piche, & Stevenson, 2013; Raziano, Jayadevappa, Valenzula, Weiner, & Lavizzo-Mourey, 2001; Schleyer & Forrest, 2000), faster response time (Bates, 2001; Beebe, Locke, Barnes, Davern, & Anderson, 2007; Dykema et al., 2013; Raziano et al., 2001; Yun & Trumbo, 2000), immediate storage of data in electronic format (Dykema et al., 2013; Schleyer & Forrest, 2000), and the ability to employ programming to minimize response patterns that can compromise data quality (such as response fatigue, item nonresponse, improper adherence to skip patterns, ambiguous or unclear response selections, and data entry errors; Birnbaum, 2004; Dykema et al., 2013). For physician respondents, these latter two advantages may help address common barriers to participation such as time and effort required to complete the survey (Klabunde, Willis, & Casalino, 2013) and concerns about confidentiality (Klabunde et al., 2013). Specifically, the availability of design features such as automated skip patterns and drop-down menus can make the response task simpler and less time consuming for participants than in-person, mail, or telephone surveys, and the lack of an interviewer or need to send responses through the mail may reduce concerns about confidentiality.

Prior studies suggest, however, that web-based physician surveys may produce lower response rates than postal surveys (Akl, Maroun, Klocke, Montori, & Schunemann, 2005; Leece et al., 2004; Losch, Thompson, & Lutz, 2004; McMahon et al., 2003; Raziano et al., 2001). The response rate disadvantage associated with web-based surveys may be particularly problematic when surveying physicians, given their already lower response rates relative to the general population. Maximizing response rates is particularly critical for studies of organizational behavior targeting physician leaders, as statistical power in such studies is highly dependent on the number or organizations represented in the sample. Because the number of eligible organizations in such studies is typically small, the loss of a significant fraction of organizations to nonparticipation can seriously compromise the external and internal validity of study findings.

Although achieving a high response rate may be necessary for accruing an adequate sample size, response rates are not in themselves sufficient or reliable indicators of survey quality (Johnson & Wislar, 2012). Indeed, at least two prior studies have documented that higher response rates are not associated with lower nonresponse bias in physician surveys (McFarlane, Olmsted, Murphy, & Hill, 2007; Thomsen, 2000). Therefore, assessments of survey quality must consider not only response rates but also response bias or representativeness. While prior studies of general population surveys suggest web-based administration produces less representative samples than postal administration (Akl et al., 2005; Leece et al., 2004; Losch et al., 2004; McMahon et al., 2003; Raziano et al., 2001), response bias is less pronounced in web-based surveys of physicians (Beebe et al., 2007; Kellerman & Herold, 2001; Scott et al., 2011).

Mixed-mode survey administration (whereby more than one mode of survey administration is offered to study participants) has been employed in several prior survey research studies as a strategy for maximizing survey quality while controlling costs of survey administration (Beebe, Davern, McAlpine, Call, & Rockwood, 2005; Beebe et al., 2007; de Leeuw, 2005; Dillman et al., 2009; Dillman & Smyth, 2007; Dykema et al., 2013; Griffin & Obenski, 2002; Scott et al., 2011). Mixing web and postal administration strategies may represent an opportunity to harness the advantages of web survey administration while still benefiting from the higher response rates typical of postal surveys (Beebe et al., 2007; Dillman, 2000). However, one concern with mixed-mode survey administration is the potential for mode effects, whereby the quality of the data collected varies depending on what mode is used (de Leeuw, 2005, 2010; Dillman, 2000). For survey quality measures such as item nonresponse and the amount of detail provided in open-ended questions, one might expect web administration to perform better due to the availability of interactive features (such as automatic prompts for responses to any items skipped) and conveniences (such as access to a keyboard, which can make it easier to provide detailed responses to open-ended items).

Few prior physician survey research studies have targeted physician leaders or health administrators as responders. Indeed, a recent review of health care provider survey studies published between 2000 and 2010 identified only four prior studies targeting health administrators (McLeod et al., 2013). We identified four prior studies that specifically targeted physician leaders (Duke, 2007; Nelson et al., 2006; Robinson et al., 2009; Seeff et al., 2004). Only one of these studies involved web administration of the survey (Duke, 2007), and this study, which achieved a 19% response rate, did not examine response quality or representativeness. None of the surveys targeting physician (as opposed to nurse) leaders in these prior studies achieved a response rate greater than 32% without use of incentives (Duke, 2007; Nelson et al., 2006; Robinson et al., 2009; Seeff et al., 2004). Those employing incentives achieved response rates of 60% (Nelson et al., 2006) to 74% (Seeff et al., 2004). Additional research is needed to identify effective approaches to improving response rates, data quality, and representativeness in surveys targeting physician leaders.

The current study adds to the literature on physician survey methods by examining whether adding postal follow-up to a web-based survey about colorectal cancer screening and diagnostic practices, administered to a sample of primary care and gastroenterology physician leaders without an incentive, improves response rates, response quality, and representativeness.

Method

Population

The survey was administered to the physician chiefs of primary care and gastroenterology at Veterans Health Administration (VHA) medical facilities. The largest integrated health care system in the United States, the VHA includes 152 medical facilities (U.S. Department of Veterans Affairs, 2013). The primary purpose of the survey was to collect information on primary care and gastroenterology clinic structures and processes related to timely diagnostic follow-up of positive fecal occult blood tests (FOBTs) conducted for colorectal cancer screening purposes. To ensure surveyed facilities were engaged in some FOBT-based colorectal cancer screening, the survey sample was restricted to VHA facilities that conducted at least 1,400 FOBTs in 2009 (n = 125).

Survey Recruitment and Administration

The survey was conducted between August and December 2010. Survey recruitment involved the following steps (summarized in Figure 1). An initial e-mail message was sent to all regional VHA Directors from the VHA Office of the Deputy Under Secretary for Health, Operations and Management in the VHA requesting that the Director forward an attached approval request e-mail to their facility directors and chiefs of staff (COS). The forwarded COS e-mail asked COS to respond to indicate whether they approve or decline their facility’s participation in the survey. They were instructed to communicate their preferences by clicking on links in the e-mail to the survey application (one indicating approval to have their facility participate and another indicating they prefer not to have their facility participate). Once connected to the survey application, the COS were asked to verify their facility and provide the contact information for their chiefs of primary care and gastroenterology, if applicable. A reminder e-mail was sent after 1 week to any COS not responding to the initial COS e-mail. Two weeks after the initial COS e-mail, the project coordinator attempted to reach by phone all COS who had not responded to determine if they supported their facility participating in the survey. At that time, the coordinator obtained their approval or refusal, as well as the names of primary care and gastroenterology chiefs (if approval was received) over the phone. As a final contact, all nonresponding COS were mailed a letter asking them to indicate their site’s participation preference, provide the contact information for their chiefs of primary care and gastroenterology (if applicable), and return their response in the self-addressed, stamped envelope provided.

Figure 1.

Survey recruitment steps.

Once approval was received from the local COS, the study team sent recruitment e-mails to primary care and gastroenterology chiefs, which were cosigned by medical leadership. The recruitment e-mails included a link to the survey website and informed potential participants that their COS had approved their facility’s participation in the survey. Reminder e-mails were sent to all primary care and gastroenterology chiefs who had not yet responded 1 and 2 weeks after the initial recruitment e-mail. Approximately 3–5 weeks after the initial recruitment e-mail, all nonresponding chiefs were sent a pen and paper version of the survey by standard U.S. Postal Service mail that could be completed and returned in the self-addressed, stamped envelope provided. The survey materials also included the URL addresses for the online versions of the survey and potential respondents were instructed that they could still complete the survey online if they preferred. Per VHA policy, participants were not offered an incentive for participating in the survey.

Content and Features to Reduce Response Errors

The survey asked about organizational structures and clinic processes related to colorectal cancer screening and diagnostic follow-up present at the respondent’s medical center. Both surveys were pretested on two primary care and two gastroenterology physicians, as well as members of the research team, before finalizing. Features used to minimize response errors on the web survey included: ensuring the survey took less than 10 min to complete, providing instructions on the number of responses requested for each question, automating skip patterns, and providing a prompt about any missing responses (which would display once, before allowing the responder to advance to the next page). Additionally, menus, command buttons, check boxes, radio buttons, and pop down lists were employed to reduce the need for typing and reduce errors associated with typing skill level. In the postal survey, features used to minimize response errors included: ensuring the survey took less than 10 min to complete, providing instructions on skip patterns and the number of responses requested to each question, and employing check box formatting.

Protection of Human Subjects

The Institutional Review Boards at the Minneapolis and Boston Veterans Affairs Medical Centers approved the study protocol.

Dependent Measures

Response Rates

Survey response rates for the primary care and gastroenterology samples were estimated using the American Association for Public Opinion Research Response Rate 1 calculation method, which divides the number of completed surveys by the total number of individuals in the recruitment sample (The American Association for Public Opinion Research, 2011).

Data Quality

The data quality measures examined included (1) the mean number of missing items for questions asked of all respondents (possible range 0–12 for the primary care sample; 0–22 for the gastroenterology sample); (2) the mean number of items for which more than one response was indicated for a “select one” response instruction, which we refer to hereafter as “response errors” (possible range 0–6 for both samples); (3) the mean number of total errors, calculated by summing the above two types of errors (possible range 0–18 for the primary care sample; 0–28 for the gastroenterology sample); and (4) the mean character count for an open-ended question at the end of the survey, which asked “Do you have any other thoughts or concerns about the follow-up of positive FOBT results at your facility you would like to share with us?”

Independent Measures

Response Status

Our analyses compare three groups of responders, classified according to response status on December 20, 2011: (1) web responders, (2) postal responders, and (3) all responders.

Time of Response

Participants were also classified according to when they responded (1) before the e-mail reminders (labeled “invitation”), (2) after the first e-mail reminder but before the second reminder (labeled “e-mail 1”), (3) after the second e-mail reminder but before postal administration (labeled “e-mail 2”), and (4) after postal administration (labeled “postal”).

Facility characteristics used to assess representativeness included the main outcome of the larger study (the proportion of positive FOBT screening results followed by diagnostic colonoscopy within 6 months) and the following characteristics suspected or demonstrated in prior research to be associated with this outcome: facility region (northeast, midwest, west, south); facility location (urban, rural); the proportion of patients age 50–75 receiving care at the facility who are adherent to colorectal cancer screening recommendations (i.e., received on FOBT in the past year, a colonoscopy in the past 10 years, or a sigmoidoscopy in the past 5 years); the dominant colorectal cancer screening modality employed at the facility (FOBT, mixed, or colonoscopy); the primary care and gastroenterology missed opportunity rates (defined as the proportion of appointments scheduled in primary care and gastroenterology clinics that were either cancelled less than 2 weeks in advance or were not attended); and the facility complexity score (Stefos, LaVellee, & Holden, 1992), a measure summarizing facility workload, patient risk level, complexity of clinical services, and teaching and research activity, which has been found to be associated with colorectal cancer screening program quality in prior studies (Yano, Soban, Parkerton, & Etzioni, 2007). Scores on this measure range from −1.41 to +1.67, with higher scores indicating greater complexity.

Analyses

Response Rates

We estimated survey response rates by sample (primary care, gastroenterology) and timing. Further, we used logarithmic regression techniques to examine the incremental impact of switching the final contact to postal administration. Specifically, we estimated a trend line that fit the observed response rates after the first three survey contacts (initial e-mail invitation and two e-mail reminders). We then used this fitted regression equation to estimate the response rate that would be expected from a hypothetical fourth e-mail reminder, and compared this estimate to the observed response rate achieved when the fourth contact involved postal administration.

Representativeness

To examine the impact of adding the final postal administration contact on sample representativeness, we compared the percentage distribution of facility characteristics in the recruitment population (all physician leaders from eligible facilities) to the distribution before (web responders) and after adding the postal follow-up (all responders) in each sample (primary care and gastroenterology). One sample chi-square and t-tests, treating the reference population as fixed, were used to identify significant differences between each response group and the recruitment population.

Data Quality

To assess the impact on data quality of adding postal administration, we compared the quality measures defined above across web and postal response groups and used negative binomial regression models to identify significant differences.

Results

Response Rates

Figures 2 and 3 provide summaries for the primary care (Figure 2) and gastroenterology (Figure 3) samples of observed response rates by timing (bars), and the predicted response rates had the last contact been an e-mail reminder (last point on the lines). Response rates were 32.7% and 34.7% in the primary care and gastroenterology samples, respectively, before the first reminder; 46.7% and 46.5% after the first reminder; and 53.3% and 56.4% after the second reminder. Adding the postal contact increased response rates by 19.6 percentage points in the primary care sample, and by 24.8 percentage points in the gastroenterology sample, resulting in total response rates of 72.9% in the primary care sample and 81.2% in the gastroenterology sample. The predicted response rate for an additional e-mail contact was 13 percentage points lower than the observed response rate following the postal contact in the primary care sample (60 vs. 73%), and 20 percentage points lower than the observed response rate in the gastroenterology sample (61 vs. 81%).

Figure 2.

Predicted and observed response rates by response timing, primary care sample.

Figure 3.

Predicted and observed response rates by response timing, gastroenterology sample.

Representativeness

Tables 1 and 2 present the frequency distributions for facility characteristics by response group in the primary care and gastroenterology samples, respectively. Before postal responders were included, there were small but statistically insignificant differences between respondents and the population on region, location, complexity, and dominant screening mode in both the primary care and the gastroenterology samples. With the exception of region in the primary care sample, any existing small differences were further reduced after adding postal responders. There were no differences between the respondents and the population on screening rates, follow-up rates, or missed opportunity rates in either sample. Because there were no statistically significant differences between responders and the population on any characteristics before postal administration, our analysis did not demonstrate significant improvement in representativeness by adding postal follow-up in either the primary care or gastroenterology samples.

Table 1.

Percentage Distribution of Facility Characteristics in the Recruitment Population and Each Response Group (Web, Postal, All Responders), Primary Care Sample (N = 107).

Characteristic	Recruitment Population % (n = 107)	Web Responder % (n = 61)	p Value* (Web vs. Population)	Postal Responder % (n = 17)	p Value* (Postal vs. Population)	All Responders% (n = 78)	p Value* (Web + Postal vs. Population)
Region
Northeast	15.9	19.7	.8178	5.9	.5818	16.7	.8589
Midwest	25.23	26.2		35.3		28.2
South	39.25	34.4		35.3		34.6
West	19.63	19.7		23.5		20.5
Location
Urban	79.44	82.0	.6253	70.6	.3665	79.5	.9918
Rural	20.56	18.0		29.4		20.5
Mean complexity score	0.17	0.24	.4985	−0.02	.3496	0.18	.8924
CRC screening rate	82.2	81.7	.3596	82.6	.6671	81.9	.4802
Proportion FOBT + with colonoscopy ≤ 6 months	0.50	0.51	.5531	0.50	.8368	0.50	.6876
Dominant screening mode
FOBT	16.8	16.4		17.7		16.7
Mixed	34.6	31.2		35.3		32.1
Colonoscopy	48.6	52.5	.8175	47.1	.9911	51.3	.8769
PC missed opportunity rate	0.11	0.11	.7599	0.10	.2451	0.11	.5402
Total	100	57.0		15.9		72.9

Note. CRC = colorectal cancer; FOBT = fecal occult blood test; PC = Primary Care.

*p Values calculated using one-sample chi-square tests for categorical variables and one-sample t-tests for continuous variables.

Table 2.

Percentage Distribution of Facility Characteristics in the Recruitment Population and Each Response Group (Web, Postal, All Responders), Gastroenterology Sample (N = 101).

Characteristic	Recruitment Population % (n = 101)	Web Responder % (n = 60)	p Value* (Web vs. Population)	Postal Responder % (n = 22)	p Value* (Postal vs. Population)	All Responders% (n = 82)	p Value* (All vs. Population)
Region
Northeast	16.8	18.3		18.2		18.3	.8972
Midwest	24.8	25.0		18.2		23.2
South	37.6	41.7		36.4		40.2
West	20.8	15.0	.7246	22.2	.8337	18.3
Location
Urban	80.2	86.7		68.2		81.7
Rural	19.8	13.3	.2088	31.8	.1572	18.3	.7320
Mean complexity score	0.18	0.33	.1492	0.18	.9805	0.29	.2311
CRC screening rate	82.2	82.2	.9575	82.0	.6333	82.1	.7466
Proportion FOBT + with colonoscopy ≤ 6 months	0.50	0.50	.6832	0.50	.8643	0.50	.6586
Dominant screening mode
FOBT	15.8	18.3	.8476	4.6		14.6
Mixed	35.6	33.3		50.0	.2136	37.8
Colonoscopy	48.5	48.3		45.5		47.6	.9060
GI missed opportunity rate	0.10	0.10	.8264	0.10	.8662	0.10	.7853
Total	100	59.4		21.8		81.2

Note. CRC = colorectal cancer; FOBT = fecal occult blood test; GI = Gastrointestinal Clinic.

*p Values calculated using one-sample chi-square tests for categorical variables and one-sample t-tests for continuous variables.

Data Quality

Table 3 provides data quality comparisons for web and postal responders by physician type. In the primary care sample, the differences in mean number of missing items, mean number of response errors, and total errors across response groups were not statistically significant. In the gastroenterology sample, although the number of missing items was small in both response groups (mean <2 in both cases), the difference in number of missing items for web (0.05) and postal responders (1.55) was statistically significant. Likewise, the mean number of response errors for web (0.20) was significantly lower than the mean number for postal responders (0.63) in the gastroenterology sample; as was the mean total number of errors (0.25 vs. 2.18). Web responders in the gastroenterology sample also provided more information on the open-ended question, as indicated by the significantly higher character count for this question among web responders.

Table 3.

Data Quality Measures by Response Group and Sample.

Characteristic	All Responders M (SD)	Web Responder M (SD)	Postal Responder M (SD)	p Value* for Web vs. Mail
Primary care sample (n = 107)
Mean number of missing items	0.31 (2.49)	0.36 (2.82)	0.12 (.33)	.6648
Mean number of improper check all	0.09 (.29)	0.07 (.25)	0.18 (.39)	.1949
Mean total error count	0.40 (2.50)	0.43 (2.82)	0.29 (.59)	.7651
Mean character count for comments	78.8 (149.31)	95.5 (162.78)	18.94 (55.31)	.0782
Gastroenterology sample (n = 101)
Mean number of missing items	0.45 (1.69)	0.05 (.22)	1.55 (3.04)	<.0001
Mean number of improper check all	0.32 (.54)	0.20 (.40)	0.63 (.73)	.0033
Mean total error count	0.77 (1.87)	0.25 (.47)	2.18 (3.16)	<.0001
Mean character count for comments	141.3 (243.08)	178.4 (271.27)	40.41 (81.11)	.0394

*p Values obtained from negative binomial regressions.

Discussion

In this study, postal follow-up to a web-based physician leader survey markedly increased response rates but not response quality or representativeness. The finding that mixed-mode administration markedly enhanced response rates is consistent with several prior physician survey studies demonstrating that switching to postal administration in the final contact for initial web-based surveys increased response rates by 10–24 percentage points (Beebe et al., 2007; Leece et al., 2004; Matteson et al., 2011; McMahon et al., 2003; Raziano et al., 2001; Scott et al., 2011). We observed a more sizable response rate boost from switching to postal administration in our specialist sample of gastroenterologists (25 percentage point difference from response rate after the last e-mail reminder) than in our primary care sample (20 percentage point difference). The reasons for this difference are not clear but at least two prior studies similarly found postal follow-up disproportionately increased participation among specialists in the sampling frame (Beebe et al., 2007; Scott et al., 2011).

The relatively high response rates achieved in our physician leader survey (73–81%) were achieved without use of an incentive; an encouraging finding given that none of the prior published studies targeting physician leaders identified in a recent review of physician surveys achieved response rates this high without use of an incentive (McLeod et al., 2013). Our ability to achieve higher response rates than observed in these prior studies may be attributable to a variety of factors, including the use of mixed-mode administration; the short length of the survey; the fact that survey recruitment was facilitated by medical leadership; and the fact that the VHA was an early adopter of electronic communication, training, and medical records documentation, which may indicate a greater receptiveness to web-based administration in this environment compared to other settings. Additional features of the VHA system that might enhance the ability to achieve high response rates and limit generalizability to other settings include the availability of centralized data repositories containing up-to-date contact information on employees (essentially eliminating the coverage error problems common in other physician surveys), and the presence of uniform standards for electronic communication and web browsers (minimizing problems of compatibility that can limit response rates to web-based surveys in national physician sample and less well-integrated health care systems). The findings suggest that high response rates can be achieved in physician leader surveys administered with web as the primary mode in settings, like the VHA, where the target population is accustomed to electronic communication, and where coverage errors and compatibility concerns are likely to be minimal. In the era of increasing reliance on electronic forms of communication and documentation in medical settings, it may be more feasible to achieve high response rates from web-based surveys than in the past, and therefore web-based physician survey administration methods warrant further study.

Three prior physician survey studies examined the impact of mixed-mode administration on sample representativeness and found significant differences between web and postal responders on key characteristics or outcomes (Beebe et al., 2007; Matteson et al., 2011; McMahon et al., 2003). We found, however, that mixed-mode administration did not markedly enhance sample representativeness. This finding in part reflects the fact that the sample of initial web responders in our study was already fairly representative of the sampling frame. This finding is consistent with prior studies finding minimal nonresponse bias in physician surveys generally (Barclay, Todd, Finlay, Grande, & Wyatt, 2002; Bjertnaes, Garratt, & Botten, 2008; Cull, O’Connor, Sharp, & Tang, 2005; McFarlane et al., 2007; Menachemi, Hikmet, Stutzman, & Brooks, 2006) and physician web-based surveys specifically (Scott et al., 2011), and documenting that higher response rates are not associated with lower nonresponse bias in physician surveys (Barton et al., 1980; McFarlane et al., 2007; Thomsen, 2000).

In this study, we found high response quality for both web and postal responders, and comparable response quality across modes in the primary care sample. However, in the gastroenterology sample, web responders were significantly less likely to skip questions or make response errors and had significantly higher character counts for the open-ended question. These findings of higher response quality among web responders contrast with the findings from one prior physician survey study that Internet respondents were more likely to skip or not answer certain questions on demographics or lifestyle habits than postal responders (Lusk, Delclos, Burau, Drawhorn, & Aday, 2007), and another documenting that web-only respondents had lower item completion rates than respondents offered either a simultaneous or sequential mixed web and postal modality (Scott et al., 2011). However, our findings are consistent with several nonphysician survey studies (Bech & Kristensen, 2009; Borkan, 2010; Denscombe, 2009; Shin, Johnson, & Rao, 2012) and two physician survey studies (Matteson et al., 2011; McMahon et al., 2003) finding superior or comparable response quality among web responders compared to postal responders. While it is plausible that the differences in response quality we observed were due to differences in the characteristics of early and late responders rather than mode, we found no significant differences in the characteristics of web and postal responders in our study (results not shown). We therefore suspect the superior data quality observed among web responders in our study is likely attributable to design features employed in the web version of the instrument (e.g., prompting for a response when none was initially entered) that were not possible on the postal version. Additionally, the availability of a keyboard may have facilitated more lengthy responses to the open-ended question in the web administration mode. Finally, the targeted sample of VHA physician may be particularly comfortable with web administration, as the VHA has emphasized electronic communication and training for more than 20 years.

The findings from this study should be qualified by the following limitations. First, because the survey targeted section chiefs in the VHA, our findings may not generalize to other samples of physicians with less administrative responsibility or who work in non-VA settings, especially outside of integrated delivery systems. Second, because survey invitations came from medical leadership rather than research staff, our findings may not generalize to physician surveys administered solely by research staff. Third, we did not have adequate statistical power to detect small but potentially meaningful changes in response quality and representativeness due to the relatively small population size of clinic chiefs. Finally, because the study was observational and did not employ a randomized design, survey mode is confounded with response timing. For this reason, it is difficult to determine whether differences in data quality across modes are due to differences in survey mode or differences in early versus late respondents. However, we believe the findings are still informative because the sequential multimode protocol that we used is common. Understanding whether this protocol can affect data quality and change results is informative even if the causes for the change cannot be teased apart.

To summarize, we found that web administration of a physician survey produced adequate representativeness and that adding a final contact administered by postal mail significantly enhanced response rates without marked detriment to response quality. These findings contrast with prior studies demonstrating low physician response rates and provide reassurance that high response rates, representativeness, and data quality can be achieved in web-based physician leader surveys administered without an incentive.

Footnotes

Authors’ Note

The views expressed in this article are those of the authors and do not necessarily represent the position or policy of the Department of Veterans Affairs.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the following grants from the Department of Veterans Affairs: Health Services Research & Development (HSR&D) grant # IIR 08-334-2 (Partin), HSR&D Research Career Scientist award #RCS 10-185 (Partin), HSR&D Career Development Awards #CDA 07-016 (Haggstrom), and #CDA 08-024 (Powell), and Clinical Science Research & Development Career Development Award CDA-2 (Shaukat).

References

Akl

E. A.

Maroun

Klocke

R. A.

Montori

Schunemann

H. J.

(2005). Electronic mail was not better than postal mail for surveying residents and faculty. Journal of Clinical Epidemiology, 58, 425–429.

Asch

D. A.

Jedrziewski

M. K.

Christakis

N. A.

(1997). Response rates to mail surveys published in medical journals. Journal of Clinical Epidemiology, 50, 1129–1136.

Barclay

Todd

Finlay

Grande

Wyatt

(2002). Not another questionnaire! Maximizing the response rate, predicting non-response and assessing non-response bias in postal questionnaire studies of GPs. Family Practice, 19, 105–111.

Barton

Bain

Hennekens

C. H.

Rosner

Belanger

Roth

Speizer

F. E.

(1980). Characteristics of respondents and non-respondents to a mailed questionnaire. American Journal of Public Health, 70, 823–825.

Bates

. (2001). Internet versus mail as data collection methodology from a high coverage population. Proceedings from the American Statistical Association, Section on Survey Research Methods, August 5-9, 2001.

Bech

Kristensen

M. B.

(2009). Differential response rates in postal and web-based surveys among older respondents. Survey Research Methods, 3, 1–6.

Beebe

T. J.

Davern

M. E.

McAlpine

D. D.

Call

K. T.

Rockwood

T. H.

(2005). Increasing response rates in a survey of Medicaid enrollees: The effect of a prepaid monetary incentive and mixed modes (mail and telephone). Medical Care, 43, 411–414.

Beebe

T. J.

Locke

G. R.

Barnes

S. A.

Davern

M. E.

Anderson

K. J.

(2007). Mixing web and mail methods in a survey of physicians. Health Services Research, 42, 1219–1234.

Birnbaum

M. H.

(2004). Human research and data collection via the internet. Annual Review of Psychology, 55, 803–832.

10.

Bjertnaes

O. A.

Garratt

Botten

(2008). Nonresponse bias and cost-effectiveness in a Norwegian survey of family physicians. Evaluation & the Health Professions, 31, 65–80.

11.

Borkan

(2010). The mode effect in mixed-mode surveys: Mail and web surveys. Social Science Computer Review, 28, 371–380.

12.

Cobanoglu

(2001). A comparison of mail, fax and web-based survey methods. International Journal of Market Research, 43, 441–442.

13.

Couper

(2000). Web surveys: A review of issues and approaches. Public Opinion Quarterly, 64, 464–494.

14.

Cull

W. L.

O’Connor

K. G.

Sharp

Tang

S. F.

(2005). Response rates and response bias for 50 surveys of pediatricians. Health Services Research, 40, 213–226.

15.

Cummings

S. M.

Savitz

L. A.

Konrad

T. R.

(2001). Reported response rates to mailed physician questionnaires. Health Services Research, 35, 1347–1355.

16.

de Leeuw

E. D.

(2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics, 21, 233–255.

17.

de Leeuw

E. D

. (2010). Mixed-mode surveys and the internet. Survey Practice, 3. Retrieved from http://www.surveypractice.org/index.php/SurveyPractice/article/view/150/html

18.

Denscombe

(2009). Item non-response rates: A comparison of online and paper questionnaires. International Journal of Social Research Methodology, 12, 281–291.

19.

Dillman

D. A.

(2000). Mail and internet surveys: The tailored design method. New York, NY: John Wiley.

20.

Dillman

D. A.

Phelps

Tortora

Swift

Kohrell

Berck

Messer

B. L.

(2009). Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the internet. Social Science Research, 38, 1–18.

21.

Dillman

D. A.

Smyth

J. D.

(2007). Design effects in the transition to web-based surveys. American Journal of Preventive Medicine, 32, S90–S96.

22.

Duke

E. M

. (2007). The Pediatric Rheumatology Workforce: A study of the Supply and Demand for Pediatric Rheumatology. Report to Congress. Bureau of Health Professions. Health Resources & Services Administration. US Department of Health & Human Services.

23.

Dykema

Jones

N. R.

Piche

Stevenson

(2013). Surveying clinicians by web: Current issues in design and administration. Evaluation & the Health Professions, 36, 352–381.

24.

Griffin

D. H.

Obenski

S. M.

(2002). Meeting 21st century demographic needs implementing the American community survey (Report No. 2). Washington, DC: U.S. Department of Commerce, Economics and Statistics Administration; U.S. Census Bureau.

25.

Johnson

T. P.

Wislar

J. S.

(2012). Response rates and nonresponse errors in surveys. Journal of the American Medical Association, 307, 1805–1806.

26.

Kellerman

S. E.

Herold

(2001). Physician response to surveys. A review of the literature. American Journal of Preventive Medicine, 20, 61–67.

27.

Klabunde

C. N.

Willis

G. B.

Casalino

L. P.

(2013). Facilitators and barriers to survey participation by physicians: A call to action for researchers. Evaluation & the Health Professions, 36, 279–295.

28.

Leece

Bhandari

Sprague

Swiontkowski

M. F.

Schemitsch

E. H.

Tornetta

Guyatt

G. H.

(2004). Internet versus mailed questionnaires: A controlled comparison. Journal of Medical Internet Research, 6, e39.

29.

Losch

M. E.

Thompson

Lutz

. (2004). The effect of mode on response rates and data quality in a survey of physicians. Proceedings from the 59th Annual Conference of the American Association for Public Opinion Research. Phoenix, AZ.

30.

Lusk

Delclos

G. L.

Burau

Drawhorn

D. D.

Aday

L. A.

(2007). Mail versus internet surveys: Determinants of method of response preferences among health professionals. Evaluation & the Health Professions, 30, 186–201.

31.

Matteson

K. A.

Anderson

B. L.

Pinto

S. B.

Lopes

Schulkin

Clark

M. A.

(2011). Surveying ourselves: Examining the use of a web-based approach for a physician survey. Evaluation & the Health Professions, 34, 448–463.

32.

McFarlane

Olmsted

M. G.

Murphy

Hill

C. A.

(2007). Nonresponse bias in a mail survey of physicians. Evaluation & the Health Professions, 30, 170–185.

33.

McLeod

C. C.

Klabunde

C. N.

Willis

G. B.

Stark

(2013). Health care provider surveys in the United States, 2000-2010: A review. Evaluation & the Health Professions, 36, 106–126.

34.

McMahon

S. R.

Iwamoto

Massoudi

M. S.

Yusuf

H. R.

Stevenson

J. M.

David

Pickering

L. K.

(2003). Comparison of e-mail, fax, and postal surveys of pediatricians. Pediatrics, 111, e299–e303.

35.

Menachemi

Hikmet

Stutzman

Brooks

R. G.

(2006). Investigating response bias in an information technology survey of physicians. Journal of Medical Systems, 30, 277–282.

36.

Nelson

J. E.

Angus

D. C.

Weissfeld

L. A.

Puntillo

K. A.

Danis

Deal

Cook

D. J.

(2006). End-of-life care for the critically ill: A national intensive care unit survey. Critical Care Medicine, 34, 2547–2553.

37.

Raziano

D. B.

Jayadevappa

Valenzula

Weiner

Lavizzo-Mourey

(2001). E-mail versus conventional postal mail survey of geriatric chiefs. Gerontologist, 41, 799–804.

38.

Robinson

J. C.

Casalino

L. P.

Gillies

R. R.

Rittenhouse

D. R.

Shortell

S. S.

Fernandes-Taylor

(2009). Financial incentives, quality improvement programs, and the adoption of clinical information technology. Medical Care, 47, 411–417.

39.

Schleyer

T. K.

Forrest

J. L.

(2000). Methods for the design and administration of web-based surveys. Journal of the American Medical Informatics Association, 7, 416–425.

40.

Scott

Jeon

S. H.

Joyce

C. M.

Humphreys

J. S.

Kalb

Witt

Leahy

(2011). A randomised trial and economic evaluation of the effect of response mode on response rate, response bias, and item non-response in a survey of doctors. BMC Medical Research Methodology, 11, 126.

41.

Seeff

L. C.

Richards

T. B.

Shapiro

J. A.

Nadel

M. R.

Manninen

D. L.

Given

L. S.

… McKenna

M. T

. (2004). How many endoscopies are performed for colorectal cancer screening? Results from CDC’s survey of endoscopic capacity. Gastroenterology, 127, 1670–1677.

42.

Shin

Johnson

T. P.

Rao

(2012). Survey mode effects on data quality: Comparison of web and mail modes in a U.S. national panel survey. Social Science Computer Review, 30, 212–228.

43.

Stefos

LaVellee

Holden

(1992). Fairness in prospective payment: A clustering approach. Health Services Research, 27, 239–261.

44.

The American Association for Public Opinion Research. (2011). Standard definitions: Final dispositions of case codes and outcome rates for surveys (7th ed.). Retrieved from http://aapor.org/Content/NavigationMenu/AboutAAPOR/StandardsampEthics/StandardDefinitions/StandardDefinitions2011.pdf

45.

Thomsen

(2000). An examination of nonresponse in a work environment questionnaire mailed to psychiatric health care personnel. Journal of Occupational Health Psychology, 5, 204–210.

46.

U.S. Department of Veterans Affairs. (2013). Veterans health administration: Where do I get the care I need? Retrieved from http://www.va.gov/health/findcare.asp

47.

VanGeest

J. B.

Johnson

T. P.

Welch

V. L.

(2007). Methodologies for improving response rates in surveys of physicians: A systematic review. Evaluation & the Health Professions, 30, 303–321.

48.

Yano

E. M.

Soban

L. M.

Parkerton

P. H.

Etzioni

D. A.

(2007). Primary care practice organization influences colorectal cancer screening performance. Health Services Research, 42, 1130–1149.

49.

Yun

G. W.

Trumbo

C. W

. (2000). Comparative response to a survey executed by post, e-mail, & Web form. Journal of Computer-Mediated Communication, 1. Retrieved from http://jcmc.indiana.edu/vol6/issue1/yun.html