Abstract
Surveys of physicians are an important tool to assess opinions and self-reported behaviors of this policy-relevant population. However, this population is notoriously difficult to survey and plagued with low and falling response rates. In order to evaluate the potential import of response rate, we examine the presence of nonresponse bias in a survey of physicians providing diabetes care that achieved a 36% response rate. Unlike other studies examining differences in individual characteristics for responding and nonresponding physicians, we also assess differences with respect to aggregate patient demographic, clinical, and behavioral characteristics. We are unable to demonstrate nonresponse bias, even with what could be construed as a relative low response rate. Nonetheless as the threat of nonresponse bias can never be completely assuaged, we believe that it should be monitored as a matter of course in physician surveys and offer a new dimension by which it can be evaluated.
Background
Physician surveys are an important source of information about practice patterns, medical knowledge, and beliefs about the health care delivery system. However, the ability to produce generalizable survey estimates from a physician survey can be limited if survey responders are systematically different from nonresponders. The magnitude of the bias introduced by potential differences can be a serious threat to the validity of survey estimates when coupled with low response rates (Barton et al., 1980; Sackett, 1979). While the relationship between response rate and bias has been shown to be far less than absolute in the general population (Groves, 2006; Groves & Peytcheva, 2008), there is an identified need to evaluate the same in a provider population (National Cancer Institute executive summary). As the last few decades have seen a downward trend in response rates in the physician population, the need to systematically evaluate nonresponse bias becomes increasingly important (Cull, O’Connor, Sharp, & Tang, 2005; McLeod, 2010).
Historically, there is a dearth of studies that systematically evaluate the differences between physician responders and nonresponders as a measure of potential nonresponse bias (Asch, Jedrziewski, & Christakis, 1997; Cummings, Savitz, & Konrad, 2001). In our own work, we saw no differences between responders and nonresponders by age, gender, or tenure. We did see differences between specialists and primary care physicians, with specialists being underrepresented in the responding population (Beebe, Locke, Barnes, Davern, & Anderson, 2007). An earlier review of 50 survey studies of pediatricians also found small response biases with women, younger and nonspecialty physicians being more likely to respond than their counterparts (Cull et al., 2005).
While it is important to consider, as prior work has done, potential differences in the type of physicians who respond to surveys compared to those who do not respond, this approach is less valuable with physician populations than it is with more heterogeneous populations (VanGeest, Johnson, & Welch, 2007). Thus, it may be more informative to consider how physician responders differ from nonresponders with respect to the demographic and clinical characteristics of the patients they see. If, for example, physicians with more adherent, compliant, or healthier patients (i.e., nonsmokers) or those with better controlled diabetes (Barton et al., 1980; Sackett, 1979) are more likely to respond to a survey, then estimates of practice patterns and guideline adherence may be skewed. Given the growing importance of these latter domains as an important aspect of quality measurement and reporting, considering physician nonresponse bias with respect to patient characteristics is a logical next step in evaluating the generalizability of physician surveys. To our knowledge, nonresponse bias with respect to patient panel characteristics has not been systematically evaluated to date. We add to the scant literature addressing physician survey nonresponse bias by characterizing responding and nonresponding physicians in terms of the clinical and demographic characteristics of the patients, number of patients with diabetes in their panel and their degree (MD, DO, NP, PA, or CNM), specialty, and years in practice.
Method
Providers caring for patients with known diabetes as identified in a diabetes registry within the Mayo Health System (MHS) comprise the study population. The MHS is a network of 12 organizations comprising 34 clinics providing health care in more than 65 communities in Minnesota, Wisconsin, and Iowa. Each organization is affiliated with multiple regional hospitals, clinics, and/or other health care facilities. Located primarily in rural areas, MHS sites are independently managed physician-led community-centered facilities. As a whole, MHS employs 780 physicians and conducts approximately 2.5 million patient visits annually. Primary care specialties such as pediatrics, family medicine, internal medicine, and integrative medicine account for 40% of MHS physicians.
Providers were identified through a diabetes registry of patients receiving primary care through MHS. Patients with diabetes are automatically added to the registry triggered by a diabetes-related billing code and then are manually verified. The provider who first bills for a diabetes-related condition is the provider listed for the patient, however this can be manually changed if someone else is later identified as the primary care provider. All but one clinic within the MHS participates in the registry. Diabetes providers at this clinic as well as residents are excluded from the provider population. A total of 304 providers were identified in the diabetes registry and comprise the study population. These providers caring for patients with diabetes were sent a web survey in November of 2009. The 55-item survey was designed to capture provider sentiment about diabetes care teams, current diabetes practice tasks, delivery, and performance. The Mayo Clinic Survey Research Center sent the survey to providers' professional e-mail addresses. The body of the e-mail included information about the research project, its importance, and the estimated time it would take to complete and was signed by the MD investigator of the study. After 2 weeks, nonrespondents were sent a reminder e-mail and another link to the survey. A second reminder and link was sent after an additional 2 weeks to remaining nonrespondents.
In order to determine the potential for nonresponse bias, we compare responder and nonresponders with respect to both physician characteristics and aggregate patient characteristics of those patients with diabetes assigned to each physician as identified in the diabetes registry. The registry through which the provider population was identified contained a number of patient characteristics including age, gender, and the five components of the D5 as recommended by Minnesota Community Measurement (2011; http://www.thed5.org), including hemoglobin A1c (HbA1c) level, blood pressure, low-density lipoprotein (LDL) cholesterol level, daily aspirin use, and tobacco status (tobacco free or not). Patient characteristics are summarized at the provider level as the percentage of patients meeting the criteria (HbA1c less than 7, blood pressure less than 130/80, LDL less than 100, daily aspirin use, and tobacco free). The patient characteristics are further summarized as the percentage of patients meeting all five criteria and those meeting the three intermediate outcome measures (i.e., LDL, HbA1c, and blood pressure). Additional frame data available for all providers include the number of patients with diabetes on their panel, degree (MD, DO, NP, PA, or CNM), specialty, and years in practice.
Physician responders are compared to physician nonresponders by summarized patient characteristics and provider characteristics. Chi-square statistics are used for categorical variables and two sample t tests for continuous variables to detect significant differences by response status. Multivariate logistic regression is used to determine which factors, if any, predict response status, controlling for both physician and assigned patient characteristics. Analysis was done in SAS software version 9.1.3 (SAS Institute Inc., 1999). Reported differences are significant at p < .05 unless stated otherwise.
Results
Just over one third of physicians (35.5%, n = 108) responded to the survey. Responders did not significantly differ from nonresponders with respect to degree, specialty, or number of years in practice (Table 1). The mean number of patients with diabetes also did not differ significantly by response status (67 among responders and 74 among nonresponders). Further, physician responders did not differ from nonresponders with respect to any of the patient characteristics examined, including mean age, gender, and all five inputs into the D5. The summarized D3 and D5 also did not differ by response status (Table 1).
Physician and Patient Characteristics of Responding and Nonresponding Physicians
Notes: MD = Medical Doctor; DO = Doctor of Osteopathy; NP= Nurse Practitioner; PA = Physician Assistant: CNM = Certified Nurse-Midwife; SD = standard deviation; HbA1c = hemoglobin A1c; BP = blood pressure; LDL = low-density lipoprotein
When considering all of the available provider and patient characteristics simultaneously in the multivariate analysis, no single factor was independently associated with a higher likelihood of response (Table 2). These results held regardless of whether the patient performance measures were considered individually or rolled up into either the D3 or the D5.
Likelihood of Physician Survey Response
Notes: MD = Medical Doctor; DO = Doctor of Osteopathy; NP=Nurse Practitioner; PA = Physician Assistant: CNM = Certified Nurse-Midwife; SD = standard deviation; HbA1c = hemoglobin A1c; BP = blood pressure; LDL= low-density lipoprotein.
Discussion
In a survey of physicians providing primary care to patients with diabetes about diabetes care we saw no evidence of nonresponse bias with respect to provider or patient demographic characteristics. The fact that we do not see differences with respect to physician characteristics is somewhat at odds with other literature to date (Beebe et al., 2007; Cull et al., 2005). However, our sample size could be too small to detect differences. Of note, our results are suggestive of differences in likelihood of response with providers with larger panels of patients with diabetes and physicians being less likely to respond than those with smaller panels or mid-level providers.
More importantly, we expand our consideration of nonresponse bias beyond the common metrics of age and gender typified by past investigations to measures of patient health status and performance characteristics related to diabetes care. For the measures that we had in these domains, we saw no differences between physicians that responded and those that did not respond. Our findings suggest that our survey did not capture a biased physician sample with respect to the survey topic, and it is unlikely in this case that practice patterns or opinions on diabetes care would be skewed toward those that are considered better performers. Moreover, the similarity of panel size of patients with diabetes indicates that the survey was not just capturing a population for whom the topic may be more salient as relevant theoretical frameworks utilized in the survey research arena such as leverage-saliency theory (Groves, Presser, & Dipko, 2004; Groves, Singer, & Corning, 2000) and social exchange theory (Dillman, 1978; Dillman, 2000) would posit. While our results are consistent with the emerging observation in the general population survey research field suggesting that a low response rate does not necessarily portend bias, this concern can never completely be assuaged.
While a lower than expected response rate may introduce a concomitant attenuation of statistical power and precision due to a smaller than expected sample size, its relationship with response bias is possibly much weaker than was long assumed (Groves, 2006; Groves & Peytcheva, 2008). While in large populations more sample can be released to achieve enough respondent sample for adequate statistical power (even with low response rates), in smaller populations it is still necessary to achieve high response rates for adequate power. Hence, there is further need for investigation into mechanisms for increasing response rates in this population such as monetary incentives and mixed-mode designs (see VanGeest et al., 2007).
There are some potential limitations to the current study that must be recognized. We were limited to patient information about the portion of the physician panel that was listed in the diabetes registry. We do not know anything about the patients who do not have diabetes nor do we know about patients who are not listed in the registry. However, it is likely that most patients with diabetes are picked up as the initial population of the registry is based on payment data. It is also possible that patients may be attached to a physician in name only. This concern is assuaged somewhat by the observation that in a related survey of patients in the registry, 94% indicated that they have a primary care provider for their diabetes care (unpublished results). Our findings may not be generalizable to other physician populations as the study was conducted in one Midwestern Health System. However, there is much diversity within the health system; it spans 3 states, many communities, 12 organizations, and is affiliated with many different hospitals, thus representing many different types of providers with potentially different responding behaviors. Nonetheless, the study design should be replicated in other populations of physicians.
Conclusion
We found no evidence of nonresponse bias in a survey of physicians in a Midwestern Health System using metrics of bias atypically utilized in past investigations of this topic (viz. patient characteristics). From these findings, we cannot conclude that bias is never a problem; rather that it was not in this context for the domains that we had available. Our findings add to the mounting evidence against the once assumed relationship between response rate and nonresponse bias, calling into question guidelines that rely solely on response rate as a measure of data quality. Nonetheless nonresponse bias is always a threat that should be monitored. Our study provides a broadened approach by which this potential threat can be monitored with the goal of producing generalizable estimates of physician sentiment and self-reported behavior.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded in part by support from the Mayo Health System Practice-Based Research Network.