Abstract
The National Health Service Staff Survey is an annual, England-wide survey of staff employed by National Health Service organisations. It is administered using online and paper questionnaires. Response rates have fallen over time, while the proportion of staff receiving the online version – which historically has a lower response rate – has increased. Three interventions to increase online response were tested using an experimental design: more reminders; change of signatory to the invitation to participate and concise messaging in the invitation. Thirteen thousand staff members in five National Health Service trusts received the online survey. Results were analysed using contingency tables and logistic regression to determine the effect of interventions independently and in combination. More reminders improved response by six percentage points (odds ratio 1.33, 95% CI 1.23–1.43) and concise messaging by two percentage points (odds ratio 1.08, 95% CI 1.00–1.17). Change of signatory had no statistically significant effect and neither did any combination of the interventions. Given that costs for the successful interventions were minimal, more reminders in combination with concise messaging are recommended as a strategy to improve online response to the survey.
Introduction
Launched in 2003, the National Health Service Staff Survey is an annual, England-wide survey of staff employed by National Health Service (NHS) organisations and is commissioned by NHS England. The most recent iteration of the survey involved 309 NHS organisations, and nearly 1.1 million staff were invited to participate. Full- and part-time staff who are directly employed by a participating organisation on 1 September of each survey year are eligible to take part. All NHS trusts (foundation trusts, acute and specialist hospital trusts, ambulance service trusts, mental health and learning disability trusts, and community trusts) are required to participate in the survey. Other NHS organisations – such as clinical commissioning groups, social enterprises and commissioning support units – take part on a voluntary basis.
Data are collected through a self-completion questionnaire. The questionnaire addresses key aspects of staff experience, including health and wellbeing, personal development, harassment and bullying, engagement and satisfaction, and support from managers. All participating organisations must use the standard core questionnaire but may also choose to add additional question modules and/or local questions. Ahead of each new survey year, the core questionnaire is reviewed by the NHS Staff Survey Advisory Group which is composed of various stakeholders, and questions may be added, removed or amended following this review. Questionnaires are sent out between late September and early December each year.
An online component was first introduced to the Staff Survey in 2013. Organisations can choose one of three methods to administer the survey – online, postal or a mixed methods approach (online and postal), and the percentage of staff receiving an online survey has consistently increased each year, from 19% in 2013 to 56% in 2015. Organisations opting for a mixed methods approach aim to target the postal surveys towards specific groups of staff that are thought to be less likely to respond to online surveys or who do not have regular access to a computer or staff email address.
In order to maintain confidentiality, the survey is administered by independent contractors appointed by each organisation. This ensures that organisations are not able to view individual staff members’ responses or to tell whether a staff member has completed the survey. At a minimum, contractors are responsible for distributing questionnaires, receiving completed questionnaires, entering and storing data, and sending reminders to non-responders. The survey is coordinated nationally by the Survey Coordination Centre, overseen by NHS England.
Results from the Staff Survey are primarily intended to be used by NHS organisations to help them review and improve staff experience, monitor change over time and identify variations between different staff groups to enable targeted action. The Care Quality Commission (CQC) also uses the results to monitor ongoing compliance with essential standards of quality and safety. In addition, the survey supports accountability of the Secretary of State for Health to parliament for delivery of the NHS Constitution.
Ensuring the NHS Staff Survey achieves a good response rate is important for maintaining both the quality and reliability of the data and its usefulness to stakeholders, including participating organisations, NHS England, CQC and the other organisations and researchers who use the data. Higher response rates provide greater precision to results and engender more confidence amongst data users. Low response rates risk bias, and there is evidence that non-responders tend to be less committed to the organisation. 1 To ensure that initiatives to improve staff experience, management and, ultimately, organisational effectiveness are on the soundest possible basis, it is vital that good response rates are achieved.
Several actions are taken each survey year in order to help maximise response rates, such as the following:
Non-responders are sent a number of reminders encouraging them to complete the survey. Currently, two reminders are sent to staff receiving a postal survey and six reminders are sent to online recipients. Organisations are advised to provide assurances of the survey’s confidentiality in their internal communications with staff. Organisations are able to use incentive schemes to encourage participation, such as rewarding departments with the highest response rate. Any prize draws must be handled by the organisation’s independent contractor in order to maintain staff confidentiality. If an organisation is using an online survey mode, they are strongly advised to ensure that their staff email addresses are accurate and up to date so that the survey emails are deliverable. Additionally, as of 2017, any organisation wanting to run an online or mixed mode survey must send written confirmation to their contractor confirming that they meet a defined set of eligibility criteria for running the survey online. If an organisation does not meet all of the criteria, they are required to run a postal survey. A mandatory minimum fieldwork period of eight weeks was introduced in 2017 to ensure that staff groups that typically take longer to respond have enough time to participate. Organisations are able to use a pre-approach letter/email to notify staff of the upcoming survey. This is sent out one to two weeks before the first mailing.
Despite such actions, response rates for the NHS Staff Survey have shown a general decline over the years. The response rate reached its lowest point in 2015 (41%), down from a peak of 60% in 2004, a year after the survey was first introduced. This decline is consistent with industry-wide trends in social and market research.1,2 Response rates for online surveys tend to be lower than for paper. 3 This trend was evident in the Staff Survey between 2013 and 2015; in 2013, the online response rate was 43%, compared to 50% for paper. Improvements to online response rates could therefore have a substantial impact on the overall response rate for the NHS Staff Survey. These considerations led to the decision to test a selection of interventions intended to improve online response rates in the 2015 survey.
Research has identified several strategies to increase survey response rates, which can broadly be classified as incentive-based (offering a tangible reward for participation) or design-based (focused on the survey materials, their content and style). 4 Social exchange theories provide a framework for understanding how people weigh the rewards and costs (both material or otherwise) of complying with a request to participate in a survey. 5 Positive design-based strategies to increase perceived benefit to respondents relevant to online surveys include specifying how results will be useful to the sponsoring organisation, emphasising sponsorship by a legitimate authority, and normative-social messaging conveying that others have responded. Incentive-based strategies include monetary gifts, vouchers and lottery draws. Such strategies need to be ‘tailored’ in ways appropriate to the survey population, survey mode, sponsorship and other factors. 5
Strategies to increase response rates have been the subject of several systematic reviews.6–8 These found that incentives, shorter questionnaires, use of coloured ink, increased number of contacts, pre-notification of the survey and making questionnaires and covering letters more personal were particularly effective at increasing response rates. For online surveys specifically, strategies noted to increase response rates were including a statement in survey correspondence indicating that other people had already responded to the survey and keeping headers simple while avoiding the term ‘survey’. 8 Invitations to respond are more effective when simply worded. 9 The effect of different types of appeal to the sample member has been investigated, specifically two types of invitation email content: appeal to altruistic motivation and appeal to egotistical motivation. 10 The results show that encouraging response in order to produce either a public good (altruistic motivation) or personal satisfaction (egotistical motivation) can improve uptake. This in turn suggests that effective normative-social strategies could include indicating the value of a survey to the health service and/or triggering a sense of pride in participation. Emphasising appropriate ‘official’ survey sponsorship can have a positive effect,11 but to avoid response bias the sponsor should not be seen as someone who might personally view the responses. 6 One study used pre-approach letters to alert participants of an upcoming telephone survey and found that this increased response rates by 5% overall. 12
Another strategy commonly found to improve response rates is reminders. Telephone reminders for a health-related postal survey were effective in increasing response rates. 13 The number of reminders is important: the response rate to an epidemiological online survey increased when 4–5 email reminders were sent and increased further when 11 reminders were sent. 14 Sending multiple reminders increased response rates to a mixed mode national health survey. 2 A review study modelled the impact of various interventions to increase response and found that the number of reminders sent was significantly related to the response rate. 15
Aside from offering tangible rewards and sending pre-notification of the survey (both existing options for organisations participating in the NHS Staff Survey), the literature therefore supports a combination of simplified and personalised messaging, normative-social messaging, emphasising official sponsorship and increased reminders as effective strategies to encourage response to online surveys.
Study design
A pilot study was conducted during the 2015 NHS Staff Survey to test a variety of interventions designed to increase response rates for the online component of the survey. Based on the literature, appropriate tailored interventions were discussed and agreed with NHS England and the survey advisory group. The following interventions were tested:
Intervention A: Increasing the number of email reminders from the standard three to six. Depending on the initial mailing date, reminders were sent approximately every 1.5 weeks. Reminders were sent only to those staff who had not already completed or opted out of the survey. Intervention B: Changing the signatory of the email from that of the Director of Research and Policy for the Staff Survey Coordination Centre to that of the Chief Nursing Officer for England, with the aim of drawing on people’s commitment to the NHS and showing that the survey is not just an administrative exercise, while avoiding the risks of having a member of staff too close to respondents which could cause confidentiality concerns. Intervention C: Using more concise messages including socio-normative messaging (such as ‘Lots of your colleagues have already responded…’). The last email also noted that this was the final reminder.
Research design
The study was designed as a true experiment with simple random allocation from the whole sample to the intervention conditions. The factorial design was structured to test combinations of interventions likely to be effective and practical in a future survey. The actual combination of interventions and the balance of sample numbers were carefully selected in discussion with the survey commissioners to provide detailed information on a limited set of potential solutions that could be trialled within the numbers likely to be available at trusts with a high uptake of the online survey option. The design was powered to test interventions against a two percentage-point difference on an anticipated baseline response rate, with the primary focus on pairs of interventions but with additional single-intervention conditions to allow testing of the independent effect of these. The target population comprised a subset of NHS trusts, all participating in the main staff survey for 2015 using an online methodology. The sample plan was to allocate intervention conditions as follows:
Group 1: Control (n = 2392)
Group 2: Intervention A (n = 1195)
Group 3: Intervention B (n = 1195)
Group 4: Intervention C (n = 1195)
Group 5: Interventions A + B combined (n = 2391)
Group 6: Interventions A + C combined (n = 2391)
Group 7: Interventions B + C combined (n = 2391)
Sample and sample selection
Five NHS trusts, with approximately 13,000 staff members receiving the online survey, were selected to take part in the pilot. These were purposively selected to represent a variety of trust types: one each of community trusts, combined acute and community trusts, ambulance trusts, acute hospital trusts and mental health/learning disability trusts.
Analysis methods
Response rates were initially examined using frequencies and bar charts of the response for each intervention condition and control. Further analysis was conducted using nested logistic regression models with a binary indicator (responded/did not respond) as the target variable. The case data were coded so that each intervention received was indicated using a 0/1 dummy variable, allowing main effects and their interactions to be modelled for the whole sample. First, main effects were entered and tested, corresponding to interventions A, B and C. Second, interaction effects were added to the previous model, corresponding to the combinations A + B, A + C and B + C, in order to test whether the effect of the interventions differed when used in combination. The regressions were run using GENLIN in SPSS v23. Model coefficients were used to compute the predicted response rates corresponding to each intervention and combination, and the odds ratio effect sizes based on these.
The analyses were treated as planned comparisons and unadjusted p-values were adopted. For reference, benchmark p-values adjusted for multiple comparisons are provided under the results tables.
Results
The achieved sample comprised 12,723 staff invited to respond to the survey, allocated to intervention conditions and responding as in Table 1. All intervention groups except the group receiving only Intervention B (Changed signatory) had a higher response rate than the control group. The highest response rates (in ascending order) were for intervention conditions A (More reminders), A + B (More reminders and Changed signatory) and A + C (More reminders and Concise messaging).
Frequencies for intervention and control groups (final response rates).
The first logistic regression model focused on only the main effects of the three interventions. The inclusion of main effects improved model fit compared to the null model (p<.001). However, the main effects predicted a limited amount of variability in response (Cox and Snell R2 = .004). The results for this model are shown in Table 2. These indicated that both More reminders and Concise messaging had statistically significant positive main effects. More reminders improved response rate by six percentage points (odds ratio 1.33, 95% CI 1.23–1.43), while Concise messaging improved response by two percentage points (odds ratio 1.08, 95% CI 1.00–1.17). Changed signatory had effectively no impact on response.
Logistic regression results for main effects (final response rate).
Note: N = 12,723. p-Values are unadjusted. Sidak adjusted p-value to maintain familywise error rate at 0.05 = 0.017.
Adding interaction terms to the main effects to account for combinations of interventions did not improve model fit significantly (p= .51) and increased prediction of response only marginally (Cox and Snell R2 = .005). The results for this model are shown in Table 3. The p-values for the interactions indicate whether there is any significant additional impact from combining the interventions, while the predicted response rates take account of this additional impact together with that of the component interventions. The results showed that the impact of Concise messaging on its own was similar to that in the first model but was not statistically significant. Although the response rates for the combined interventions were slightly higher compared to the single interventions, none of the interaction effects was statistically significant, suggesting that the interventions were largely independent and that combinations of interventions were no more or less effective than the interventions on their own.
Logistic regression results for main effects and interactions (final response rate).
Note: N = 12,723. p-Values are unadjusted. Sidak adjusted p-value to maintain familywise error rate at 0.05 = 0.009.
Discussion
There was clear evidence from this study for a positive effect from More reminders (Intervention A). This had a positive impact on response rate both as a stand-alone and in combination with other interventions tested.
There was evidence for a positive, but smaller, effect from Concise messaging (Intervention C). Although the combined interventions also produced increased response, this appears to have been due to the inclusion of effective main interventions and is not an additional impact from their use in tandem.
All of this evidence points to More reminders as being the most effective intervention. The measured improvement due to More reminders was substantial: six percentage points over a baseline response rate of 30% represents a very worthwhile improvement. This intervention has limited associated cost, as contractors administering the survey have automated systems for sending reminders and it is only necessary to program in the additional reminders. Concise messaging also has minimal cost, associated with editing and approving the letters to staff.
Following this pilot, both of these interventions were implemented survey-wide for the 2016 staff survey. The online response rate for the 2016 survey was an encouraging 45% compared to 40% in 2015. This increase is in line with the predicted impact of the interventions tested in the pilot study.
Conclusions
Issuing an increased number of reminders had a positive effect on response to the NHS Staff Survey. This was the most effective intervention tested and has a worthwhile impact on response. More concise messaging on the covering and reminder letters also has a positive effect; as this has negligible cost, there is no reason not to recommend this intervention in combination with more reminders as a way to improve survey response. Response rates should be monitored over future iterations of the survey to evaluate whether the increase is sustained.
Footnotes
Acknowledgements
We are grateful to the NHS trusts that took part in the pilot: Central London Community Healthcare NHS Trust; Lewisham and Greenwich NHS Trust; East of England Ambulance Service NHS Trust; King’s College Hospital NHS Foundation Trust; Calderstones Partnership NHS Foundation Trust. We are also grateful to Eliza Swinn and the staff of the Survey Co-ordination Centre who administer the survey.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The work was supported by NHS England as part of the staff survey coordination contract.