Abstract
Surveys are an important tool for assessing physician and nursing professionals’ practice patterns and guideline adherence. Obtaining quality survey data consisting of low item and unit nonresponse remains a persistent challenge in these populations. We tested the relative impact of two envelope types (padded vs. priority mail) on unit and item nonresponse in a survey of Minnesota health care workers. Respondents were randomized to receive a survey in one of two envelope types: a padded 8.5′′ × 11′′ envelope or a similarly sized priority mail envelope. After the first mailing, the response rate was 53.9% and did not differ across envelope conditions. Females and RNs were more likely to respond to the priority envelope than the padded envelope, but this finding did not hold in multivariate analysis. There was no difference in item nonresponse across the two envelope conditions. It may be that our two approaches were not enough to permeate the semi-porous membrane of gatekeeping that has been posited as a driver of low physician survey response rates relative to those observed in the general population. Nonetheless, our findings suggest that packaging may matter for some populations and not others.
Introduction
Surveys are used to obtain estimates of the views or practices of the population being surveyed. The representation of a resultant estimate can be compromised by nonresponse bias when there is less than full participation. While full participation has never been realized, response rates have been dropping in recent years (Berk et al. 2007; Curtin et al. 2005; de Leeuw and de Heer 2002; Hartge 1999; Hox and de Leeuw 1994; Morton et al. 2006; Steeh et al. 2001; Tickle et al. 2003). The challenges involved in obtaining high response rates in survey research are particularly pronounced in busy and potentially survey-fatigued populations like health care professionals (Keating et al. 2008; Sudman 1985). Yet accurate assessment of practice patterns, degree of guideline adherence, and general opinions and perspectives of broad samples of physicians and nurses are dependent on this method of data collection. The importance of these lines of inquiry and thus this method of data collection is increasing, given the recently passed Patient Protection and Affordable Care Act of 2010, which calls for increased ascertainment of provider sentiment.
To adequately meet these needs, it is important to obtain quality survey data reflected by both low item and unit nonresponse. While many studies have examined the effectiveness of a myriad of different modes and means of improving response rates among postal surveys of the general population (Edwards et al. 2009), fewer have considered physician populations (VanGeest et al. 2007) and even fewer have considered nurse populations (VanGeest and Johnson 2011). Strategies identified to increase the likelihood of response in general and physician populations specifically include providing both monetary and nonmonetary incentives and enhancing elements of questionnaire design. This line of inquiry is likely even more important in the population of health care providers compared to the general population, given the quantity of mailings directed to this group and the potential role of gatekeepers or individuals who screen providers’ mail, only passing on those deemed worthwhile.
One strategy identified as a means of potentially getting through gatekeepers focuses not on the questionnaires, incentives, or other elements observed inside the envelope, but on external elements designed to get the potential gatekeeper or respondent to open the envelope: a necessary precursor to survey completion. One example of this type of strategy is our prior work assessing the impact of adding a brightly colored “$25 incentive” sticker to the outside of an envelope on response rates and nonresponse bias in a survey of physicians (Ziegenfuss et al. 2012). Also included in this line of inquiry are experiments varying elements of the mechanism of the mailing, such as envelope color or whether the package was sent via priority mail, certified mail, or first-class mail, which may vary the perceived credibility of the request. It is this latter aspect that constitutes the focus of the present investigation.
A study on envelope manipulation by Asch and colleagues (1997) found that in a survey of academic physicians, Veterans Administration envelopes resulted in higher response rates than did university envelopes. The authors recognized the inability to extend these findings to other studies and concluded that the important finding was that packaging can impact response rates. Since then, few studies have examined the impact of the delivery or packaging of an initial mailing on physician survey response rates. Those that have been conducted suggest that registered (Pedrana et al. 2008; Thorpe et al. 2008), FedEx (Kasprzyk et al. 2001), and certified mail (Del Valle et al. 1997) generally yield higher rates of response than regular or first-class mail delivery. One study examining the influence of using priority versus first-class mail in a survey of physicians did not find any significant difference in response rates between the two groups (Brems et al. 2006). This study was limited by a relatively small n of 260.
Given the scant and equivocal nature of the current research in this area, coupled with the increasing interest in ascertaining the sentiments of health care providers such as physicians and nurses, we endeavored to examine the specific impact of envelope type (padded vs. priority mail) on unit and item nonresponse in a survey of these types of health providers in Minnesota.
Method
Sample and Procedures
This study was embedded in a larger study designed to ascertain practices and perspectives on H1N1 pandemic influenza vaccination among practicing registered nurses and physicians in Minnesota. The methodological details of this larger study have been described in detail elsewhere (Henriksen-Hellyer et al. 2011). In February 2010, we mailed a seven-page, self-administered questionnaire to practicing Minnesota registered nurses (n = 800) and physicians (n = 800) in specialties providing preventive, primary, and acute inpatient services. Our random sample was selected utilizing lists from Minnesota state licensing organizations for medicine and nursing.
In the initial mailing of the survey, we randomized prospective respondents to receive the survey in one of two types of envelopes: a padded 8.5′′ × 11′′ envelope or a similarly sized priority mail envelope. The randomization was done by a computer and was administered by someone who did not know the study goals. Potential respondents were not aware that there was an experimental manipulation of envelope type. The padded envelope was thicker, but the priority envelope may have demanded more attention as it would communicate that more value was given to the correspondence. The initial mailing of the survey included a cover letter, an endorsement letter from the Minnesota Department of Health, the seven-page questionnaire, a postage-paid return envelope, and a laser pointer pen as a thank-you gift. Physicians and nurses who did not respond to the first mailing were sent up to two subsequent mailings. To isolate the impact of the experimental manipulation and gauge the relative impact on needing to recontact individuals, this analysis focuses on the results of the first wave of mailings. This study was approved by the Mayo Clinic Institutional Review Board.
Analysis
Response rates are calculated after the initial mailing. The impact of the envelope type was assessed by comparing overall response rates between groups within experimental condition (profession, gender, and geographic region within the state) using χ2 tests of independence. Response rates were also compared across experimental conditions within groups (i.e., within females). Item nonresponse was summarized by calculating the average number of missing items in the survey and was compared between envelope type groups using a t-test. A multivariable logistic regression model was estimated to understand the independent impact of envelope type on the likelihood of response, controlling for profession, gender, and geographic region within the state. Interactions with envelope type were assessed to determine if the effect of gender, profession, or geographic region differed between experimental conditions. Two-sided p values less than .05 were considered statistically significant. All analyses were conducted using SAS version 9 (Cary, NC).
Results
Overall, nurses were more likely to respond (59.6% compared to 48.3% of physicians, p < .001, results not shown). After the first mailing, the response rate was 53.9% and did not differ across envelope conditions overall (55.6% in priority mail envelopes compared to 52.2% in padded envelopes, p = .18) or for physicians, but did for nurses. Specifically, 63.5% of nurses randomized to the priority mail envelope responded compared to 55.8% of those randomized to the padded envelope (p = .03). With regard to other subgroups, the envelope used did not impact males, but did impact females. A total of 59.2% of females randomized to priority mail responded compared to 52.5% randomized to the padded envelope (p = .03). There were no observed differences with respect to envelope type and metropolitan area or with respect to certification among the MDs (Table 1).
Response Rates to a Survey by Envelope Type, Overall, and by Professional Characteristics of Doctors and Nurses in Minnesota.
Note: p Values based on χ2.
There was no difference in item nonresponse across the two envelope conditions. Within the priority mail condition, 2.14 items were missing on average compared to 2.10 in the padded envelope condition (p = .74, results not shown).
In multivariable analysis, the effects of profession and geographic region remained statistically significant. After adjusting for gender and envelope type, MDs were less likely to respond as compared to nurses (adjusted odds ratio 0.62, 95% CI: 0.48–0.79, p < .001), and those in the Minneapolis/St. Paul (MSP) area were least likely to respond (nonmetro vs. MSP adjusted OR: 1.69, 95% CI: 1.30–2.20, p < .001; other metro vs. MSP-adjusted OR: 1.58, 95% CI: 1.23–2.04, p < .001). The effect of gender was no longer significant, explained in part due to the association between gender and profession (92% of nurses were female). None of these effects differed significantly across envelope types (nonsignificant interactions with envelope type).
Discussion
This study offers only partial support for the use of priority envelopes in surveys of health professionals. Specifically, nurses were more likely to respond to the priority envelope than the padded envelope in bivariate analyses. However, similar results were not observed among physicians and the finding did not hold in multivariate analysis, likely due to the impact of gender. When considering the added cost of priority mail—something not evaluated in the present study—inclusion of that mailing mechanism may be unwarranted. Further investigation of these manipulations, as well as others (e.g., certified mail), is encouraged. It may be that our two approaches were not enough to permeate the semi-porous membrane of gatekeeping that has been hypothesized by some to be a driver of low physician survey response rates relative to those observed in the general population (Beebe et al. 2007; Heywood et al. 1995; Parsons et al. 1991; VanGeest et al. 2007). Moreover, it is possible that the impact of envelope type mattered more or less when gatekeeping was present. As such, the importance of documenting the extent of gatekeeping as called on by others is further underscored (VanGeest et al. 2007). Nonetheless, our findings suggest that packaging may matter for some populations and not others, something suggested by Asch et al. (1997). Finally, we observed no impact of envelope type on item nonresponse.
Our incidental finding that nurses are significantly more likely to respond to the survey than their physician counterparts is important and runs counter to recent conjecture by VanGeest and Johnson (2011) that a more limited work-related autonomy and willingness to engage in research may suppress nurse survey response vis-à-vis physicians. Investigators considering the inclusion of these two types of health care professionals should account for this potential differential nonresponse and/or craft methods to minimize differences. One specific line of inquiry ought to consider the presence or absence of gatekeepers in nurse populations as a key contributor to their heightened response. This may help uncover whether the impact of priority mail in this population was because nurses directly observed this manipulation or not.
This study has a couple of key potential limitations. First, the sample was composed of health care professionals in Minnesota and therefore may not be representative of all care providers. Second, the overall response rate was relatively high, especially for a single mailing (53%), suggesting that the topic (perspectives on the H1N1 influenza pandemic) was quite salient to prospective respondents. As such, topic salience may have been the great equalizer in this instance, as it may have overridden differences in gatekeeping and/or envelope-opening. It is also possible that the results reported here are unique to those who are early responders and should not be generalized to later mailings. Another limitation could be sample size. Even with a sample of 1,600, we may have had inadequate power to detect overall differences. However, this is a necessary limitation to methodological work that takes advantage of existing studies that are powered to meet content-related objectives. This should not keep researchers from embedding experiments like ours that can build a body of evidence to bear on questions such as those addressed here. Finally, a full cost analysis is beyond the scope of this article, but should be pursued in future work to determine if any increased response rate to various mailing manipulations such as those described here are worth the additional costs.
In conclusion, although neither response rates nor nonresponse bias were impacted by the envelope types manipulated in this study, it is important that health survey methodologists continue to respond to the calls of Cull et al. (2005), Kellerman and Herold (2001), McMahon et al. (2003), and VanGeest and Johnson (2011) by continuing to further test methods of enhancing survey participation among elite populations such as physicians. Otherwise, the perspectives of the latter may not be adequately represented in debates and issues germane to the practice of medicine or to the realm of health care reform.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.