Abstract
Surveys of health care providers (e.g., physicians and other health care professionals) are an important tool for assessing health care practices and the settings in which care is delivered. Although multiple methods are used to increase survey data quality, little is known about which methods are most commonly implemented. We reviewed 117 large surveys described in literature published between 2000 and 2010, examining descriptions of survey design features, survey implementation, and response rates. Despite wide variation, the typical provider survey selected practicing physicians as respondents, used the American Medical Association Masterfile as sample frame, included mail as both mode of initial contact and questionnaire administration mode, and offered monetary incentives to respondents. Our review revealed inconsistency of documentation concerning procedures used, and a variety of response rate calculation methods, such that it was difficult to determine practices that maximize response rate. We recommend that reports provide more comprehensive documentation concerning key methodological features to improve assessment of survey data quality.
Introduction
Over the past several decades, the quality of health care has received increased attention, especially the development and dissemination of best practices into highly variable, real-world practice settings. Surveys of physicians, nurses, and other individuals who provide health care are an important tool for determining which care practices are actually implemented, as well as the underlying circumstances that facilitate or hinder the adoption of best practices. Provider surveys have been used to track clinician knowledge, attitudes, and beliefs, as well as financial and operational aspects of health care systems. While researchers are motivated to employ methods that maximize the quality of the data collected, evidence suggests that response rates in surveys of providers—particularly of physicians—are relatively low and may be declining (Cook, Dickinson, & Eccles, 2009).
High-quality surveys are generally characterized as minimizing total survey error (Groves & Lyberg, 2010), and producing data that are representative of a targeted population, with minimal bias related to: (1) population coverage; (2) errors of nonobservation, such as those due to unit and item nonresponse (Groves, Fowler, Couper, Lepkowski, & Singer, 2009); and (3) errors of observation, particularly response error. Investigators seek to minimize error through optimizing survey design and methodology, which involves decisions about factors such as sampling frame, mode of initial contact, respondent incentives, data collection mode, instrument design, questionnaire length, and follow-up procedures (Burns et al., 2008; Dillman & Christian, 2009). Understanding how these design choices affect data quality is of vital importance for guiding the conduct, as well as the appraisal, of provider surveys.
In this article, we describe a literature review conducted to inform a discussion of current methodologies in designing and fielding large-scale surveys of health care providers that took place during a workshop convened by the National Cancer Institute (NCI) in Rockville, MD, in November 2010. The goals of the NCI workshop were to describe and discuss current as well as emerging methods for fielding large-scale provider surveys and identify opportunities for future methods research that could enhance the quality of these surveys. For purposes of the workshop and this review, a large-scale survey was defined as having 500 or more respondents. Our review identified 117 large health care provider surveys described in literature published between 2000 and 2010. First, we summarize the procedural and methodological characteristics of these surveys. Then, we examine the methods reported for calculating response rate. Finally, we discuss how survey features may be related to response rates obtained, and make conclusions concerning the degree to which best practices for provider surveys can be identified. We also suggest further research avenues that may be fruitful to pursue, discussed more fully in a separate paper (Klabunde et al., 2012). Prior reviews have focused exclusively on surveys of physicians (Cummings, Savitz, & Konrad, 2001; Field et al., 2002; Kellerman & Herold, 2001; VanGeest, Johnson, & Welch, 2007) and methods for attaining higher response rates in surveys of physicians (Field et al., 2002; Kellerman & Herold, 2001; VanGeest et al., 2007) and nurses (VanGeest & Johnson, 2011). The present review makes a unique contribution to the existing literature by including health care providers in addition to physicians, and considering not only response rate, but a range of features that may influence survey quality.
Method
We worked to identify relatively large surveys (those with 500 or more responses from health care providers), as documented in literature published between 2000 and 2010. We searched published literature as well as publicly available reports and papers found in sources other than peer-reviewed journals (i.e., “gray” literature) to determine the characteristics of these recently fielded surveys. We conducted an online search of National Library of Medicine sources for articles published between 2000 and 2010 using the search terms health care provider survey, physician survey, nurse survey, or physician assistant survey, with the words “national,” “state,” “methods and incentive,” or “methods and response rate.” The search yielded about 200 citations. We then reviewed each of these articles to confirm that it described findings from a survey with responses from at least 500 providers. A follow-up review of literature cited in these articles also served to identify other large provider surveys.
We also examined “gray” literature by conducting an online search using Google and the search terms previously mentioned, as well as by scrutinizing websites of organizations known to review or implement health care provider surveys. Finally, we examined proceedings from American Statistical Association and American Association for Public Opinion Research (AAPOR) conferences and reviewed government reports from the Health Resources and Services Administration, the Centers for Disease Control and Prevention (CDC), the Agency for Healthcare Research and Quality (AHRQ), and the Department of Veterans Affairs.
Once we had obtained articles and documents describing specific surveys, we abstracted, to the extent possible, the following information about each survey: year fielded, sponsor, type of provider surveyed, sampling frame, initial sample size, mode of initial respondent contact, data collection mode, survey length, respondent incentives, follow-up strategy, response rate, response rate calculation, first author, year of publication, and associated website. Finally, we attempted to determine the characteristics of surveys achieving a response rate of at least 60%, which is a level set as a minimum requirement for publication by some medical and public health journals ( Journal of the American Medical Association, 2011), and was the median response rate reported for the surveys included in the review. We excluded from this assessment two surveys that reported neither initial sample size nor response rate and two intercept surveys that did not have a fixed initial sample and for which a response rate therefore could not be calculated. When both a cooperation rate and a response rate were reported, we used the response rate in our analysis. Beyond this, we did not attempt to standardize, or correct, response rates across surveys, as the information provided in survey reports that we reviewed was often incomplete. We conducted sensitivity analyses of different response rate cut points, including groupings of “low,” “medium,” and “high,” but determined that this considerably complicated the reporting of findings without yielding insights beyond our 60% threshold approach.
Results
The search yielded reports of 117 unique surveys fielded between 1998 and 2009, and published between 2000 and 2010 (see Appendix; available at: http://appliedresearch.cancer.gov/surveys/physician/appendix.html). Thirty-seven of these surveys were components of survey projects that were fielded annually or at some other periodicity; these included the National Ambulatory Medical Care Survey (NAMCS; 11 surveys), the Medical Expenditure Panel Survey (MEPS; 11 surveys), and the American College of Radiology Survey of Radiologists (three surveys). We counted each repetition or cycle of these survey projects as a separate survey because of differences over time in methodology, instrumentation, and response rates obtained, and because a new sample or cohort was selected for each fielding. Thus, for purposes of this review, a survey is defined as a single, time-limited fielding of a questionnaire instrument. In the sections that follow, we present results based on the characteristics of the 117 individual surveys, and describe:
Survey characteristics, including year fielded, sponsorship, and provider type.
Survey design, including sampling frame, sample size, data collection mode, and burden.
Survey implementation, including modes of initial contact, incentives, and follow-up strategies.
Response rates.
Features of surveys for which response rates of 60% or greater were reported.
Survey Characteristics
Year Fielded
The number of surveys fielded per year showed no discernible pattern for the 117 surveys reviewed. From 1998 to 2009, the number varied from 4 to 12 per year, with a median of 10. The date the survey was fielded was missing from documentation for 13% (n = 15) of the surveys.
Sponsorship
Government agencies were the primary sponsors of the 117 surveys: 44% (n = 52) were sponsored by government agencies such as AHRQ, the National Institutes of Health, and the CDC. Notably, two major Federal efforts fielded annually to quantify aspects of ambulatory health care delivery—the NAMCS and the MEPS—each fielded 11 surveys during the review period. These differ from other provider surveys in terms of the resources allocated to them, and the fact that they resemble establishment surveys, as they include a range of respondents within the same sampled unit. Because of fundamental differences in approach, and so that their multiple survey fieldings do not overwhelm our computed frequencies, they are listed separately in the tables. Relatively fewer (9%) surveys were sponsored through partnerships between government agencies and nonprofit organizations or medical associations. For example, in 2000, the National Institute of Diabetes and Digestive and Kidney Diseases, the Commonwealth Fund, and the Robert Wood Johnson Foundation cosponsored a survey on referral preferences of primary care physicians (PCPs; Kinchen, Cooper, Levine, Wang, & Powe, 2004). Nonprofit organizations and universities sponsored 19% (n = 22) and 14% (n = 17) of surveys, respectively, and medical associations sponsored 6% (n = 7). Few surveys were sponsored by pharmaceutical and medical companies (n = 5) or health care organizations (n = 2). Mention of sponsorship was missing from documentation for two surveys.
Provider Type
Physicians were the predominant type of health care provider surveyed. PCPs were the respondents for 15% (n = 18) of the 117 surveys; specialty physicians for 24% (n = 28); and both PCPs and specialty physicians for 31% (n = 36). Administrative staff, along with PCPs and specialty physicians, were included as respondents in the NAMCS and MEPS, which represented 19% (n = 22) of the 117 surveys. Only one other survey targeted both administrative staff and providers. Nurses and nurse practitioners were targeted for 7% (n = 8) of the surveys, with the remaining 3% (n = 4) targeting clinical administrators with nursing or medical backgrounds.
Survey Design
Sampling Frame
Sampling frames are used to identify all members of the target population about which information is desired. The sampling frame most frequently used in the 117 surveys was the American Medical Association (AMA) Physician Masterfile (47%; n = 55); 11 of these surveys were from NAMCS. However, 23% (n = 27) of the surveys used mailing lists of professional associations, while other frame sources consisted of mailing lists from licensure and specialty boards (9%; n = 11), and from health plans or pharmacy benefit companies (4%; n = 5). MEPS, which derives its sample from providers named by a random, stratified sample of patients, dominated the frame category “physicians identified by sampled patients” (10%; n = 12).
A sampling frame can create bias in findings if coverage is incomplete (i.e., if some members of the target population are missing or cannot be contacted due to outdated information) or if the categorizing information that it provides, such as specialty or part-time status, is inaccurate or incomplete. Researchers may seek to maximize frame coverage by linkage to other, more accurate data sources, but this can add substantial cost to survey implementation. In our review, we rarely found documentation describing information related to sampling frame coverage, which would allow the reader to assess sampling frame quality. Furthermore, very few studies reported linking data sources.
Sample Fielded
The sample fielded represents the maximum number of cases available for analysis and is associated with the level of effort expended by researchers to obtain the necessary data. For many of the 117 surveys, it was difficult to directly identify the number of cases targeted as recipients of the survey, although the size of the sample fielded could often be estimated from information provided about the number of completed surveys, the reported response rate, and information about how the response rate was calculated. The size of the fielded samples varied widely; about one third (32%; n = 38) of the surveys involved fewer than 2,000 providers. Including NAMCS, about another third (32%; n = 38) involved 2,000–4,000 providers. Fewer surveys (22%; n = 25) had an initial sample size of 4,000 or greater. The MEPS in particular tended to field a large sample (between 7,000 and 20,000, depending on administration cycle).
Data Collection Mode
Despite dramatic increases in the use of information technologies that have occurred over the past decade, the majority of provider surveys (54%; n = 63) relied upon the well-established survey technique of mailing paper questionnaires. Providers may prefer this method for several reasons. Paper is a highly mobile and convenient medium; questionnaires can be completed in a range of locations with readily available tools (e.g., a pencil); the process is easily resumed following interruptions; and returning the questionnaire is simple because mailboxes are ubiquitous in business and residential settings. Only a few surveys attempted methods other than mail as the sole mode of data collection. Five percent (n = 6) employed the Internet, 4% (n = 5) used telephone, and 3% (n = 3) asked respondents to return paper questionnaires in person. Fax and e-mail were not reported as the primary mode of response for any reviewed survey, although these were mentioned as optional data collection strategies in a handful of cases.
Data collection using mixed modes was common. NAMCS and MEPS both used a complex, multi-instrument, multistage combination of electronic, telephone, and paper data collection strategies directed toward different staff members within providers’ offices. Simpler mixed-mode strategies were implemented in 15% (n = 18) of the 117 surveys, allowing a single identified respondent to complete the questionnaire using one of the several modes, usually mail or Internet. These 18 surveys used two mixed-mode designs: choice and sequential presentation. In 7 of the 18 mixed-mode surveys, potential respondents were initially and repeatedly offered a choice of response modes, while in the remaining 11, potential respondents were first asked to respond using one mode (often mail) and then nonrespondents were asked to respond by an alternate mode (often Internet or telephone).
Survey Burden
Burden is a term generally used to describe the effort required by the respondent to complete the survey, and is related to the format of the instrument, the number of items presented, the terminology used in the survey items, and the complexity of the constructs measured. Information related to a survey’s burden can sometimes shed light on the interpretation of the response rate reported. In our review, 68% (n = 80) of the 117 surveys reported any measure of burden, and this was reported in several different forms (e.g., the typical time required to complete the survey, the number of pages in the survey instrument, or the number of items in the instrument), making comparisons or overall conclusions about survey burden difficult. Of the 10 surveys reporting burden in time required to complete the survey, the modal duration was 20 min, and the range was 5–60 min.
Survey Implementation
Mode of Initial Contact
Researchers used mail most frequently to contact potential respondents initially, regardless of the mode they later used to collect survey data. Seventy-four percent (n = 86) of the reviewed surveys used letters or postcards to make contact with potential respondents. The only survey that documented use of telephone for initial contact was MEPS (n = 11). Initial contact by e-mail was reported infrequently, occurring in about 10% (n = 12) of surveys. Use of e-mail as an initial contact occurred most frequently when the survey was administered through the Internet and respondents were able to access the survey through an e-mail website address link.
Incentives
The use of incentives was frequently reported for the 117 provider surveys; major exceptions were the 22 NAMCS and MEPS surveys. Of the remaining 95 surveys, 49% (n = 47) reported using incentives. Incentives differed in terms of:
Whether they were offered before completion (prepaid) or after completion (contingent);
Their type (cash, gift cards, lottery chances, or material goods); and
Their target (entire sample vs. nonrespondents).
Of the 47 surveys offering incentives, 53% (n = 25) used prepaid monetary incentives of cash or checks provided to the entire sample. Prepaid incentive levels varied from $1 to $75, with a median of $10 and a mean of $20. In contrast, contingent monetary incentives were offered in eight surveys, and tended to be larger, with a range of $5–$150, a median of $25, and a mean of $52. The remaining 14 surveys used a variety of incentive strategies, including a prepaid/contingent combination, nonresponder cash, and lottery. Further, the use of incentives appears to have increased over time. Excluding NAMCS and MEPS, between 1998 and 2000, 43% (12 of the 28) of fielded surveys offered an incentive; between 2002 and 2005, 48% (15 of the 31) offered incentives; and from 2006 to 2009, 71% (15 of the 21) did so. For 15 surveys, no information was available regarding the year fielded, so these were excluded from analyses concerning time trends.
Follow-Up Strategies to Convert Nonrespondents
The level of follow-up implemented in each reviewed survey was difficult to determine because documentation often lacked sufficient detail. Common strategies included:
Extending the field period to provide more time for initial nonrespondents to complete the survey.
Increasing the number of recontact using a single contact mode (e.g., additional mailings of a paper questionnaire).
Switching contact mode to encourage response to a single data collection mode (e.g., following up a letter with telephone calls to encourage a mailed survey response).
Offering multiple sequential response modes, such as a mailed survey after an initial Internet survey.
Tailoring recontact efforts based on information obtained in follow-up communications (e.g., faxing surveys when fax numbers could be obtained from office staff, or locating and then mailing to providers’ home addresses).
Offering incentives to nonrespondents at follow-up, as discussed above.
Of the 117 surveys, 82% (n = 96) made some mention of follow-up, whereas follow-up was not addressed in 18% (n = 21). For the surveys describing follow-up efforts, the most common strategies were (a) multiple additional mailings for 32% (n = 31) and (b) multiple mailings and an additional contact mode (usually telephone reminders) for 28% (n = 27). NAMCS and MEPS (n = 22) both employed multiple, intensive follow-up strategies. For the remaining 17% of surveys describing follow-up (n = 16), efforts were limited to one telephone call, e-mail, or mailing.
Response Rates
Documentation
Within the reports of the surveys reviewed, response rates were frequently difficult to locate and, even when identified, tended to be reported inconsistently, particularly within journal articles. In several articles, no single rate was directly reported, but could be computed based on raw frequencies (e.g., if 50 surveys were returned of the 100 eligible cases, a value of 50% could be assigned). Two surveys reported neither initial sample size nor response rate. Finally, two reports described intercept surveys, which establish no fixed initial sample, and therefore preclude computation of a response rate. Figure 1 displays the distribution of reported response rates for the 117 surveys, and illustrates that values of 60–79% were most prevalent.
Distribution of response rates reported in health care provider surveys published 2000–2010 (N = 117 surveys). Note. Figure is available in full color in the online version at ehp.sagepub.com.
Calculation Method
Among the 117 surveys reviewed, we found considerable variation in the method used to calculate response rate. In 9% (n = 10), the method used was unidentifiable. Journal articles infrequently differentiated between partially and fully completed surveys, whereas the gray literature tended to be clearer in excluding cases that did not meet specific criteria for defining a completed questionnaire. Eligibility status also was poorly documented, particularly with respect to the status of noncontacts (i.e., persons who could not be contacted because of missing or outdated information). Frequently, eligibility rules could be determined only indirectly, through the reader’s calculations using the numbers provided. Overall, authors differed substantially in the way in which they handled ineligible and noncontact cases when calculating response rate.
The most conservative form of calculating a response rate involves dividing the number of completed surveys by the total number fielded, equivalent to AAPOR Response Rate 1 (or RR1; American Association of Public Opinion Research, 2011). This method was used in 25% (n = 29) of the 117 surveys; 11 of these were fielded by MEPS. An alternative method removes noninterviewed cases that are known to be ineligible (AAPOR RR3), or else estimates the number ineligible (RR4). Because of this adjustment, either method will normally result in a higher computed value than RR1. In the 21% of surveys (n = 25), researchers divided the number of completed surveys by a denominator that adjusted for ineligible cases. In 18 of these surveys, the ineligible cases were known as such (RR3); in another 7, the number of ineligible cases among nonrespondents was estimated (RR4).
Ten percent (n = 12) of surveys (including the 11 iterations of NAMCS) screened potential respondents, then fielded a survey to all eligible cases. These surveys reported the number of surveys completed divided by the number of screened, eligible cases. However, the extent of noncontact to the initial screener was unaccounted for in response rate calculation, so the reported rate fails to take into account a screener response rate. Assuming a screening response rate of less than 100%, doing so would have resulted in lower computed values of overall response rate than those reported.
In 29% (n = 34) of surveys, response rate calculation involved dividing the number of completed surveys by a denominator that excluded all cases that were unable to be contacted. This is a more liberal approach that results in a relatively higher reported rate. Nine surveys excluded all noncontacts from the total cases fielded, and in 25 surveys, the denominator was adjusted for known ineligible cases, but all noncontact cases were excluded. Either method represents a cooperation rate, as opposed to a true response rate, according to AAPOR definitions (e.g., AAPOR RR5). However, very few authors used the term cooperation rate or specifically stated that noncontact cases were assumed to be ineligible when reporting this rate. Finally, five surveys (4%) relied on nonstandard and idiosyncratic methods, employing various combinations of subtracting ineligible cases from the numerator or subtracting refusals and partial completes from the denominator.
It appears that over time, fewer authors reported response rates based on the most conservative computational formula, and instead began to adopt the more liberal cooperation rate. Excluding the 22 surveys from NAMCS and MEPS, and the 4 surveys with missing response rate information, the rate of usage of the conservative RR1 formula was 22% (n = 4) between 1998 and 2000, 28% (n = 8) between 2001 and 2004, but only 4% (n = 1) between 2005 and 2008. In contrast, the usage of a cooperation rate increased, from no surveys between 1998 and 2000, to 3% (n = 1) between 2001 and 2004, and 24% (n = 6) between 2005 and 2008.
Changes in Response Rates Over Time
Despite changes in response rate computation that might be expected to result in higher reported rates over time, we found evidence of a modest downward trend in response rates from 1998 to 2008, especially after excluding NAMCS and MEPS. Figure 2 displays the percentage of surveys with response rates of 60% or greater by 3-year interval fielded, for 91 surveys for which information on response rates was provided. From 1998 to 2000, 61% (11 of the 18 surveys) reported response rates of 60% or greater; from 2001 to 2004, 55% (16 of the 29) attained this level; and from 2005 to 2008, 36% (9 of the 25) did so. None of the four surveys fielded in 2009 reported response rates above 60%, but it is likely that surveys ultimately achieving this level maintained longer field periods, and were not published at the time of our review.
Percent of health care provider surveys published 2000–2010 with response rates of 60% or higher, by year fielded. N = 91 surveys; National Ambulatory Medical Care Survey (NAMCS), Medical Expenditures Panel Survey (MEPS), and four surveys with missing response rate information are excluded. Note. No surveys with a 2009 field period reported a response rate of 60% or higher; however, these may not have been published at the time of the literature review. Figure is available in full color in the online version at ehp.sagepub.com.
Features of Surveys With Relatively High-Response Rates
It is of considerable interest to determine whether particular practices, such as providing a respondent incentive, have the result of increasing survey response rates. However, we found it difficult to make such determinations in a straightforward manner, given that, as described above, computation of response rates varied considerably between surveys, making direct comparisons difficult. Therefore, we briefly summarize the overall trends we observed concerning the relationship between survey design and administration characteristics, on one hand, and degree to which these appear to be related to the likelihood of attaining a reasonably high-response rate (we chose a level of 60% or higher, which was reached for 53% of the 113 surveys). We make only limited attempts to strictly quantify these differences, again due to inconsistencies in calculation of response rates. We do note where results may be especially problematic due to such calculation inconsistency.
Sponsorship
Government-sponsored surveys generally reported reaching a 60% response rate more frequently than did those sponsored by other types of organizations (81% vs. 30%). Although Federal surveys tended to use more conservative measures, this difference did not appear to be a product of differential response rate calculation.
Provider Type
As shown in Table 1, surveys of nurses reported achieving a 60% response rate more frequently than did surveys of other providers. Surveys of primary care providers generally achieved this response rate more frequently than did those of specialty physicians. However, when surveys using similar response rate calculation formulas are compared, little evidence suggests that physician specialty strongly affected response rate.
Type of Provider Surveyed and Percent Reporting Response Rate ≥60%, for Health Care Provider Surveys Published 2000–2010. a
Note. NAMCS = National Ambulatory Medical Care Survey; MEPS = Medical Expenditures Panel Survey.
aResponse rate calculation differed systematically across studies, so values of percent reporting response rates ≥60% are not directly comparable across provider type (see text).
bNAMCS and MEPS are listed separately because each consists of multiple cycles of essentially the same survey (see text).
Data Collection Mode
The proportion of surveys reaching a 60% level of response varied by mode, but produced a complex pattern, with generally higher response rates for telephone, mail, and mixed-mode surveys, as a group, than for Internet surveys.
Initial Contact
For surveys reporting at least a 60% response rate, the use of mail for initial contact was more common than was contact by e-mail.
Incentives
As shown in Table 2, after excluding NAMCS and MEPS, which confer no incentives, surveys achieving 60% were likely to be those offering a monetary incentive. We observed little difference in response rates related to the use of prepaid and contingent monetary incentives. However, it is likely that the higher value of contingent incentives relative to prepaid incentives was a factor in producing this result. We did find that the level of prepaid incentive was related to likelihood of attaining a 60% response rate, with a higher proportion of surveys conveying more than a $30 incentive achieving 60% relative to those offering lesser or no incentive (Figure 3).
Incentive Strategy and Percent Reporting Response Rate ≥60%, for Health Care Provider Surveys Published 2000–2010.
Note. NAMCS = National Ambulatory Medical Care Survey; MEPS = Medical Expenditures Panel Survey.
Prepaid incentive level and percent reporting response rate ≥60%, for health care provider surveys published 2000–2010. Note. Figure is available in full color in the online version at ehp.sagepub.com.
Follow-Up
Response rate appeared to be related to the intensity of follow-up, with only 2 of the 13 surveys that reported no follow-up attaining a response rate of 60%. Our review revealed little evidence that researchers increased the intensity of their follow-up to address decreasing response rates over time. Table 3 summarizes the follow-up strategies used. NAMCS and MEPS conducted extensive multimode follow-up in all 22 of their survey fieldings. Although it was a common follow-up strategy, the use of multiple additional mailings alone was associated with relatively lower response rates, relative to the use of mixed approaches to follow-up.
Follow-Up Strategies Used and Percent Reporting Response Rate ≥60%, for Health Care Provider Surveys Published 2000–2010.
Note. NAMCS = National Ambulatory Medical Care Survey; MEPS = Medical Expenditures Panel Survey.
Discussion
Our review of the 117 large health care provider surveys described in literature published between 2000 and 2010 revealed wide variation in survey characteristics, but several features dominate. Physicians were the target population of interest in the great majority of surveys, likely a reflection of the perceived central role of physicians in clinical decision making. Provider surveys relied most often on the AMA Masterfile as a sampling frame, mainly because it contains the most readily available listing of physicians. Another major trend was to rely on mail as the major mode for both initial contact and questionnaire administration, but with mixed modes also favored. Finally, response incentives were often used, mainly in small amounts that were delivered to the entire sample in an initial contact, as opposed to a larger amount delivered contingent on survey completion. Limitations to our approach included restriction to studies published since 2000, and the relatively high lower limit on sample size (n = 500), which excluded smaller surveys that may exhibit unique characteristics as well as surveys with large numbers of fielded cases that failed to obtain at least 500 completes, potentially omitting surveys with very low response rates from the review. Future work could focus on the procedures and methods used in provider surveys with smaller samples (n < 500). There is also a need for periodic assessment of the procedures and methods used in provider surveys; a subsequent search of Pubmed and the web for any new, large-scale surveys that were published in 2011 and 2012 and not captured in our review did not identify additional surveys. However, with time, new survey reports will be published and their inclusion in future reviews will be important.
Many of the procedures implemented in the provider surveys had the explicit objective of increasing response rates. Response rate is the most common, single standard for measuring the quality of provider surveys (Cull, O’Connor, Sharp, & Tang, 2005), and some high-impact medical journals require a response rate of 60% or higher as a condition for considering manuscripts that are based on survey data ( Journal of the American Medical Association, 2011), even though it has been argued that assessment of nonresponse bias may be more informative for understanding survey limitations than focusing on a response rate threshold (Johnson & Wislar, 2012). Fifty-three percent of the surveys reported relatively high-response rates of 60% or greater, and there was the suggestion of a decline in response rates over the decade. The decline occurred in spite of increased use of incentives. Our findings support those of two reviews that examined surveys conducted from 1994 to 2005 (Cook et al., 2009; Cull et al., 2005).
However, a strong caveat to our analysis of response rates is that we also found incomplete documentation and inconsistent computation of response rates, and our attempts to determine survey features that may be related to higher response rates were complicated by the possibility of systematic variation in response rate computation across qualitatively different types of studies. After accounting for these differences as much as possible, we identified several overall trends: Surveys achieving a response rate of at least 60% tended to be those that (a) were sponsored by the Federal government; (b) selected nurses rather than physicians as respondents (and there was limited evidence that surveys of PCPs produced higher rates than did those of specialty physicians); (c) relied on mail, phone, or mixed mode more than Internet-only data collection; (d) mailed rather than e-mailed providers at the initial point of contact; and (e) provided some type of monetary incentive, as opposed to no incentive or a nonmonetary incentive. To better assess the success of surveys in obtaining reasonable response rates, and to facilitate identification of factors that lead to higher rates, we encourage journal editors to require that manuscript authors employ AAPOR standard definitions for reporting response rates, and that they explicitly state their assumptions concerning eligibility of noncontact cases.
More generally, and consistent with an earlier review of physician surveys fielded between 1986 and 1996 by Cummings, Savitz, and Konrad (2001), we found that information about survey methodology is generally lacking in the documentation of many health care provider surveys. Clear descriptions of the sampling frame, size of the sample fielded, survey burden, use of incentives, the type and intensity of follow-up of nonrespondents, and features of instrument design are important process measures for assessing survey quality, but are often missing, particularly in journal articles. However, with the widespread use of web technology and increasing use by many journals of websites for online publishing of supplemental material, researchers can post survey details that they may be unable to include in print journal articles due to word count limits.
Finally, even where survey procedures used were specified, we found that techniques noted in the methodological literature to improve data quality, and response rates, were not always implemented. These include prepaid monetary incentives (Berry & Kanouse, 1987; Delnevo, Abatemarco, & Steinberg, 2004; Leung et al., 2004); mail as opposed to Internet mode of administration (Leese et al., 2004; McMahon et al., 2003); initial contact by mail (Burns et al., 2008; Field et al., 2002; Shiono & Klebanoff, 1991); and intensive follow-up (Dillman & Christian, 2009). Overall, however, no single design factor, or even set of factors, appears to guarantee high-response rates, and optimal levels of some of these factors have not been determined. In particular, more information is needed about the cost-effectiveness of the use of larger incentive amounts, and particularly whether larger incentives substantially increase the representativeness of data through production of higher response rates.
Based on this review, we support Cummings et al. (2001), who over a decade ago advocated the development of a set of reporting standards that describe the full complement of relevant health care provider survey features. Our findings and recommendations also parallel a recent review of guidelines and reporting practices for health surveys in general, which identified limited guidance and poor documentation in peer-reviewed journals as issues (Bennett et al., 2011). Procedural and methodological features of health care provider surveys that should be described in all survey reports include survey characteristics (year fielded, sponsorship, provider type surveyed), survey design (sampling frame, sample size, data collection mode, burden), survey implementation (mode of initial contact, incentives, follow-up strategy), and response rate (including relevant AAPOR formulas used in response rate calculations). Implementation of reporting standards for provider surveys would not only provide a basis for survey replication but also convey critical information needed by readers to evaluate the quality of the survey methodology and the robustness and representativeness of survey findings. Ultimately, reporting standards for health care provider surveys might lead to the establishment of a set of best practices for survey design, execution, and analysis, and allow for more accurate interpretation of findings, as well as improvement in the quality of future health care provider surveys fielded.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.