Abstract
Background: Surveys are a prominent tool in gerontological research but are hindered by low response rates. This study examined methods to improve response rate among social workers, an understudied cohort in survey methodology literature. Methods: In this brief report, we analyzed factors that influence response rate using mixed-mode survey design (mail vs. online response) and randomized monetary ($10 gift card) and in-kind (continuing education coupon) noncontingent incentives with three reminder postcards among social workers (N = 307). Results: Response rate was 34.1% with more surveys returned via mail than online. Participants who received a gift card responded at a higher rate (46.33%) than those who received continuing education coupons (28.0%). The results add to the literature of survey methodology to improve response rates among social workers and support noncontingent gift cards and mixed-mode survey return procedures as best practices for reducing bias and improving the quality of gerontological research.
• Social workers are key providers of mental health and case management services for older adults; their perspectives are vital. Ensuring sufficiently high responses to clinician surveys is important for reliable and valid interpretations of findings. • Survey response rates are improved with the use of multi-modal response methods; professionals benefit from the option to mail in their completed surveys. • Noncontingent monetary incentives, such as gift cards, yield higher response rates among social workers compared to relevant in-kind incentives such as continuing education credits.
• Improved survey methodologies, especially when targeting social workers and other behavioral healthcare providers, are important means of reducing potential bias in research. • Resources to support evidence-based survey practices enhance response rates and improve the reliability and validity of data collected for gerontological research.What this paper adds
Applications of study findings
Surveys of behavioral health and social service professionals are a prominent tool to assess a variety of relevant gerontological research questions. Unfortunately, surveys often yield low response rates resulting in underpowered studies and insufficient sample sizes (Cho et al., 2013). Reduced sample size and nonresponse biases are possible consequences of poor response rates, jeopardizing the generalizability of findings (Cho et al., 2013). Investigation of survey methodology to enhance response rates is necessary to improve the quality of data.
A meta-analysis that compared web versus other survey modes found that web-based surveys yielded lower response rates and response rate improved when multiple response modalities (i.e., web and mail) were offered (Daikeler et al., 2020). Poor design regarding web survey recruitment and implementation may be a driving factor behind lower response rates (Fan & Yan, 2010). Another meta-analysis that investigated various survey methodology techniques found that recruitment by mail generates a greater response rate compared to recruitment by email (Cho et al., 2013). Two follow-up reminders also improve response rate (Cho et al., 2013).
Noncontingent incentives (i.e., incentives provided regardless of intent to participate) increase response rate because human psychology suggests that people are more likely to return a favor when something is first offered to them, but form matters (Cho et al., 2013). Cash and gift card incentives improve response rates beyond that of in-kind incentives (i.e., non-monetary–based incentives); an in-kind incentive is largely ineffective unless it is relevant to the participants of the study (Hawley et al., 2009). In contrast to monetary incentives, in-kind incentives require additional effort to obtain the reward. Because they are more distal reinforcers, they are less likely to change behavior.
Most survey methodology research has been conducted on medical professionals (i.e., physicians, nurses; Cho et al., 2013; Daikeler et al., 2020). There are important differences between medical professionals and social service professionals including work type, patient experience, and salary that may impact the generalizability of methodology practices from medical professionals to behavioral health and social service professionals. Hawley and colleagues (2009) identified the dearth of research in survey recruitment methodology for behavioral health professionals and examined the generalizability of survey recruitment methodology to a heterogeneous cohort of behavioral health professionals including counselors, marriage and family therapists, psychiatrists, psychologists, and social workers. They found that noncontingent monetary incentives improved response rate above that of non-monetary incentives and no incentive. Limited research has been conducted on other survey methodology techniques among behavioral health professionals. Schmidt and colleagues (2022) found that offering only a web-based survey response option generated a 13% response rate among counselors despite the use of three follow-up reminders.
The current study was conducted as part of a larger survey project on clinical knowledge of aging. This study utilized a randomized design to investigate the generalizability of survey methodology practices to licensed social workers. The focus on a homogenous group improves the understanding of methodology that is specific to a group with unique characteristics as opposed to investigating a heterogenous group of mental health professionals (Hawley et al., 2009). Social workers are unique compared to other mental health professionals in that they provide a variety of services including clinical services, but also care coordination, advocacy, occupational and employee assistance, and more. Social workers also fluctuate in the time spent face-to-face with clients and level of education. This makes them a sought after and compelling population to investigate for a variety of important gerontological research topics. One aim of the study was to examine the impact of three different noncontingent research incentives. An additional aim was to investigate differential response rates when participants are given the opportunity to complete and return their completed survey via mail or using a web-based portal, following initial recruitment by mail with the use of multiple mailed postcard reminders.
Method
Procedures
Nine-hundred licensed social workers were randomly selected using a random number generator from a publicly available state licensure database (i.e., Division of Professional Registration, Missouri). Conducted within the context of a larger study of professional training and knowledge of aging, and approved by the University Institutional Review Board, a packet was mailed to the address used for participants’ licensure renewal and included a cover letter inviting participation, an informed consent, the eight-page survey tool (i.e., four pages double-sided), an incentive, and an addressed #9 business reply envelope. The three questionnaires used in this study all surrounded the goal of better understanding the psychometric components of a newly developed assessment for clinical knowledge of aging. Participants were given the option of completing the survey online by going to a link provided or by via paper-and-pencil and returning via United States Postal Service.
Participants were randomly assigned to one of three noncontingent incentive groups (n = 300) and were blind to the existence of the other incentives. Incentive group 1 received a $10 Starbucks gift card, group 2 received a coupon for a free online professional training of up to 2 contact hours of continued education valued between $25 and $39, and group 3 received a coupon for up to 3 contact hours of continued education valued between $39 and $69. Gift cards were selected to investigate the effects of a monetary-type incentive on response rate as research suggests that monetary incentives improve response rate compared to non-monetary forms (Cho et al., 2013). The two incentive groups provided CE credits of differing values were used to determine if the value and amount of continuing education hours impacted response rate. Participants were provided with a coupon code to redeem for a CE course of their choosing (i.e., ethics, motivational interviewing, trauma, aging) to increase the relevancy of the courses to their practice. Continuing education was examined as a non-monetary incentive for social workers because most states require participation in CE to maintain licensure and many agencies do not compensate their employees for participation. These credits may be an important and unique non-monetary incentive opportunity to improve survey response among social workers.
After the initial mailing of the packet, three reminder 6” × 9” postcards were sent to all participants spaced 2 weeks apart, regardless of whether they had already completed the survey due to anonymity of responses. The postcards were unique to each incentive group and included reminders of the study, the specific incentive received, and the link to participate in the online survey. This resulted in a total of four rounds of contact with prospective participants.
Results
Of the 900 surveys mailed to participants, 311 responses were received, with 4 online survey responses removed that were entirely blank. This results in a final sample size of 307 (34.1% completion rate). Most respondents were White (85.0%), identified as female (89.2%), and held a Master’s degree (95.4%). Of the 307 responses used for this research, more completed surveys were returned via mail (n = 170; 55.37%) compared to online (n = 137; 44.63%). Response rates for the noncontingent incentive groups of 300 social workers each varied; those receiving the $10 Starbucks gift card (n = 139; 46.33%) were more likely to participate than those receiving a coupon for either 2 (n = 83; 27.67%) or 3 hours (n = 85; 28.33%) of free CE training and credits for licensure renewal (χ2 (2, N = 900) = 29.94; p < .001). The number of participants who utilized the coupon for 2 or 3 CE hours was 16 and 12, respectively. In comparison to those in the other two incentive conditions, participants who received the gift card as a noncontingent incentive had increased likelihood of completing a printed survey and returning by mail (χ2 (2, N = 307) = 6.69; p = .035) rather than completing online.
Pattern of Survey Response by Noncontingent Incentive and Response Modality.
aBased on a total 900 solicitations; n = 300 per incentive type.
CE stands for continuing education.
The relationship between date of completion and mode of completion was examined to determine the impact of reminder postcards on the mode of completion for participants. The pattern of mode of completion did not vary by when surveys were completed/returned (χ2 (3, N = 307) = 7.37, p = .06). The relationships among date of completion and mode of completion by noncontingent incentive group were examined. Among participants who received the gift card, the pattern of how surveys were completed varied by when surveys were completed and returned (χ2 (3, N = 139) = 16.56, p < .001). In this incentive condition, respondents were more likely to return their survey by mail in the initial (n = 32, 58.2% of Wave 1 responses), first reminder (n = 37, 72.5% of Wave 2 responses), and second reminder (n = 17, 81.0% of Wave 3 responses) conditions but were more likely to respond online after the third reminder (n = 10, 83.3% of Wave 4 responses). Among participants who received 2 hours of CE, the mode of survey response also varied by when surveys were completed/returned (χ2 (3, N = 139) = 8.96, p = .029). In this incentive condition, respondents were more likely to return their survey by mail following the initial invitation (n = 18, 58.1% of Wave 1 responses) and after the third postcard reminder (n = 7, 53.8% of Wave 4 responses) but were more likely to return their survey online after the first (n = 14, n = 51.9% of Wave 2 responses) and second (n = 11, 91.7% of Wave 3 responses) postcard reminders. There was no relationship between survey response mode and survey response completion in the 3-h CE incentive group (χ2 (3, N = 83) = 6.25, p = .10).
Discussion
Our study of aging-related professional experiences and knowledge in licensed social workers had a final return rate of 34.10% of the 900 mailed invitations and follow-up reminders. Across the 307 completed responses, mailed-in returns (55.4%) were higher than web-based survey completions (44.6%). Multiple contacts with participants appear to increase response rate, but modality of response appears to be especially important. Previous research demonstrated that mail-in responses were the favored return modality among highly trained healthcare professionals (Taylor & Scott, 2019), and this finding appears to be consistent with social workers as well. Beebe et al. (2018) found that mail-only responses increased to 32.1% after three contacts, whereas Schmidt et al. (2022) had a response rate of 13.9% when using web-only responses and three contacts. Interestingly, one study found a response rate of 59% when using web-only recruitment and survey response modality (Annear, 2020). This high response rate contrasts with most survey methodology studies suggesting that mail recruitment and response modality yields the most responses from healthcare professionals (Daikeler et al., 2020). When study resources allow, it remains important to provide dual methods for survey completion (i.e., return mail vs. web-based) and multiple recruitment contacts.
Consistent with previous research, providing a noncontingent incentive with recruitment material increased the response rate among social workers in this study. The $10 gift card incentive (46.33% response) was significantly more successful than two other randomized incentives (coupons for 2 and 3 CEs; 28.0% pooled response). Only 16 responders who received the 2-h CE coupon and 12 responders who received the 3-h CE coupon redeemed the coupon suggesting it is not a strong incentive. One possible explanation for this is that healthcare professionals are unlikely to participate in CEs beyond the required hours (Hébert et al., 2022). Further, consistent with findings that the value of incentive does not change response rate (Cho et al., 2013), there was also no significant difference in response between the CE groups.
There are several limitations to this study. Due to anonymity, we are unable to confirm if all participants who were mailed study materials received them. Our estimates of actual response rates to received invitations may be slightly conservative. The use of a homogenous group enables us to draw conclusions regarding social workers but limits the generalizability of findings. Future research should investigate other populations of behavioral health professionals to determine if results generalize to other professions. Further, nonresponse bias was not examined due to limitations in availability of necessary data.
The results of the present study support the use of mixed modality, noncontingent incentives, and reminder contacts to increase survey response among social workers. This brief report adds to the gerontological survey literature by investigating mechanisms to increase survey participation among social workers, a vastly understudied group in the design literature. Surveys are a prominent method for examining assessment and treatment practices, clinician attitudes, and effectiveness of dissemination and implementation strategies (Hawley et al., 2009). As the older adult population continues to grow, surveys of behavioral health and other social service professionals will increasingly be relied upon to assess the climate of the field. This is especially true as many generalist clinicians will increasingly see older adults in their practice (Hoge et al., 2015). Methodology to improve response rate among this cohort is imperative for enhancing the science of gerontology.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Ethical Approval
This study was approved by the University IRB (IRB Protocol #2051522).
