Developing the next generation of quality improvers: An evaluation of a region wide structured support quality improvement training programme

Abstract

Introduction

Despite an increasing and widespread understanding of the importance of quality improvement in healthcare, medical students and junior doctors receive variable training in improvement methods. The South West Foundation Doctor quality improvement programme attempted to equip junior doctors with the skills to develop and improve healthcare services.

Methods

A questionnaire-based evaluation was undertaken of a cohort of first-year doctors who took part in a 9-month quality improvement educational intervention. The study was conducted across a whole training region with some of the hospitals running the quality improvement educational intervention and one acting as the natural control cohort. A previously validated questionnaire was sent out at the start and end of their first year.

Results

In questions related to attitudes to all domains of quality improvement first-year doctors consider it an important part of their role as doctors to improve the quality of healthcare. However, they do not think they are equipped with the skills to do this improvement. During their first year as doctors, these self-reported skills increased dramatically in the hospitals that took part in the programme but not in the single hospital that did not run the programme.

Conclusion

First-year doctors consider improving the quality of healthcare to be an important part of their job. This can be contrasted to their self-reported skills related to quality improvement when they start their first year. In hospitals that run a structured, supported experiential quality improvement programme, these self-reported skills increase statistically significantly.

Keywords

Quality improvement evaluation education intervention questionnaire

Introduction

Despite over two decades of efforts to improve the quality and safety of healthcare using quality improvement methods, medical students and junior doctors receive variable training in this field. Most medical student undergraduate curricula do not specifically include quality improvement (QI) training.¹ Yet doctors (and particularity newly qualified doctors called foundation doctors in the United Kingdom) encounter many of the problems that can manifest as, or contribute to, poor quality health care experienced by patients. Their enthusiasm and number make them a potentially huge enabler for driving a change in the culture needed for improvement in healthcare. This is not the first time this has been said.^2–4 Several educational initiatives have been undertaken to involve junior doctors in QI with reported success.^5–7

The South West Foundation Doctor Quality Improvement Programme is an educational programme for first-year doctors (F1s), which capitalises on developing tacit knowledge and experiential learning by undertaking a structured, near-peer supported QI project throughout their first year of professional work.² Anecdotally, the programme was deemed very successful and spread beyond the initial four hospitals where it was being piloted. However, no objective evaluation of the impact of the programme has been undertaken.

How to best evaluate QI interventions has raised considerable discussion within the QI academic community, not least how to best evaluate educational interventions.^8–11 Evaluative studies assessing the organisational impact of improvement interventions have delivered conflicting results, highlighting the difficulty of disentangling the diverse contextual components influences on organisational performance.^12,13 There are, however, many studies that demonstrate the influence of improvement interventions on behaviour change of healthcare professionals in practice.¹⁴ In addition, there is some evidence that medical students' attitudes to medicine and their idealism degrade through their time at medical school and when they start as doctors.¹⁵

Methods

Questionnaires have been used and validated regularly in healthcare to assess the safety culture.¹⁶ The process of validation involves designing a questionnaire with potential responders fully involved to ensure that the questions are relevant and understandable and then after the survey to look for associations between the question types. This is called internal validity of the survey tool and shows whether the different questions are answering the same thing.¹⁷ As part of the evaluation of this programme, a questionnaire was designed and validated, as no other questionnaires have been used to assess the impact of QI educational interventions.¹ Other studies have looked at patient safety alone but these did not include all of the domains of QI.¹⁸

Study design

A questionnaire-based evaluation of a cohort of first-year junior doctors enrolled in a 9-month QI educational initiative. Attitudes and self-reported skills relating to QI were assessed before and after an experiential training programme. A previously designed and validated questionnaire was used.¹

Settings and participants

The experiential training programme was conducted in the Severn Deanery in the South West of England covering a population of around three million people. The Deanery contains eight acute hospitals with 280 first-year doctors serving across them, with between 20 and 55 doctors working in each hospital. The programme (described in more detail below) had been piloted and adapted in two of the hospitals in the preceding 4 years. The study period was over a 12-month period from August 2013.

We had originally intended to run the programme in the originating two hospitals and expand the programme to include a further two hospitals. This would have naturally enabled a control group of doctors in the remaining four hospitals without the intervention. However, three of the hospitals heard about the programme and ran it as well during the study period. Only one hospital did not participate.

The programme

The full description of the programme has been previously published.² To summarise, the programme consists of structured and supported QI projects undertaken by first-year doctors (F1s) during their first year in practice. The programme is available to all F1s and about 80% will complete a QI project using methods and tools like the Model for Improvement, driver diagrams and process mapping methods. The F1s identify problems they have observed and through facilitated discussions by the QI faculty their ideas form the basis of feasible projects that are pursued jointly in groups of between 6 and 10 doctors. Examples of completed projects include improving the timeliness and content of discharge summaries, minimising missed important information about critically ill patients during weekend handover, implementing ‘common clinical equipment’ boxes on wards to minimise time wasted spent looking for the same equipment, timely phlebotomy services, extended nil-by-mouth times and venous thromboembolism. The junior doctors develop measurement strategies of the system they are trying to improve over the first few months before planning and testing their interventions through repeated PDSA cycles. They then present their work to the hospital executive board at the end of the year and many have gone on to publish their work.^19–22

Questionnaire

To assess the impact of this programme, an 18 question survey was develop and validated.¹ Questions were related to three areas: attitudes to patient safety, self-reported improvement skills and psychological safety (their belief that they are able to speak up when they have patient safety concerns). Responses were recorded as a 5-point Likert scale. Several methods were used to validate the survey. An initial modified Delphi technique was used to develop the questions, and then these were trialled and adapted in consultation with potential respondents. An external statistician examined the internal consistency of the questions. The internal consistency of the two main question types, ones relating to attitudes and ones relating to self-reported skills had Cronbach’s alpha of 0.74 and 0.7, respectively. This level is similar to other widely used surveys to assess patient safety culture suggesting the questionnaire is a suitable measure to assess both first-year doctor’s attitudes and self-reported skills.²³ The Cronbach’s alpha for the questions relating to psychological safety was 0.47, which is low and suggests the questions are not accurate, as such these questions were not analysed further and are not included in this article. Student’s t-test was used to assess for a statistically significant change in the questionnaire scores. The Likert response score for one of the questions was reversed since it was purposefully negatively worded.

Results

Participants

All 280 F1s were sent the on-line survey in an electronic form with two reminders. The survey was sent at two separate time periods; within the first month of starting work (August 2013) and in their final month of work (July 2014). Ninety-three (33%) filled out the questionnaire at the beginning and 80 (28%) at the end of the period. This percentage is broadly equivalent to other published studies.^17,24,25 There was a roughly even distribution across all the hospitals. In the smaller cohort who did not take part in the programme, there were 15 responses before the survey and 8 afterwards. This is a very low figure and although this result was analysed statistically these results should be reviewed with caution.

Tables 1 and 2 show the questions relating to attitudes and self-reported skills, respectively. Across both cohorts first-year doctors felt that QI was very much part of their role and this was very similar at the start and the end of their year. The only exception was ‘reducing the costs of healthcare is part of my professional responsibility,’ which was agreed with in 57–86% of participants. All the other questions on attitudes had agreement between 84 and 100%.

Table 1.

Survey questions relating to attitudes.

Question	Study cohort before	Study cohort after	t-test p-value	Control cohort before	Control cohort after	t-test p-value
Contributing to improving patient safety is part of my professional responsibility	96% (4.60)	98% (4.52)	0.32	93% (4.29)	85% (4.42)	0.75
Listening to patients’ experiences can help us improve our services	100% (4.50)	97% (4.40)	0.32	100% (4.43)	100% (4.29)	0.54
It is part of my responsibility as a doctor to understand how I can improve patient safety	96% (4.37)	98% (4.32)	0.63	100% (4.43)	100% (4.29)	0.54
I consider improving patient experience part of my continuing professional development	96% (4.29)	95% (4.24)	0.64	100% (4.43)	100% (4.29)	1
I have a role in contributing to the scientific evidence behind patient care	91% (4.04)	84% (3.95)	0.39	93% (4.21)	85% (3.86)	0.16
The way in which I conduct my work impacts on the trust’s financial status	84% (3.92)	91% (4.09)	0.19	86% (3.86)	85% (4.00)	0.49
Reducing the costs of healthcare is part of my professional responsibility	77% (3.73)	69% (3.75)	0.96	86% (4.00)	57% (3.57)	0.10
At this stage in my career, it is important to have a basic training in quality improvement	83% (3.94)	88% (4.09)	0.23	86% (3.92)	100% (4.29)	0.13
Average scores of attitudes	90% (4.19)	90% (4.17)	0.80	93% (4.18)	89% (3.92)	0.79

Percentages are number of participants responding ‘agree’ or ‘strongly agree’. Number in parenthesis is the average score used for statistical comparison.

Table 2.

Questions related to self-reported skills.

Question	Study cohort before	Study cohort after	t-test p-value	Control cohort before	Control cohort after	t-test p-value
I have the skills to improve patient experience	84% (3.97)	79% (0.43)	0.44	86% (4.00)	85% (3.71)	0.46
My medical experiences so far have shown me how to improve work efficiency	72% (3.72)	97% (4.13)	0.0007	71% (3.64)	100% (4.00)	0.16
In my current role, I feel I can contribute to making clinical care more efficient	70% (3.71)	81% (3.86)	0.21	36% (3.21)	100% (4.00)	0.03
I feel confident in my ability to improve patient safety	58% (3.53)	68% (3.67)	0.311	21% (3.50)	50% (3.38)	0.71
I would only feel confident improving patient care with senior help (note this is a negatively worded question)	57% (3.24)	50% (3.10)	0.44	31% (2.77)	43% (3.39)	0.31
If I had an improvement idea, I would have the skills to execute the project	31% (3.04)	72% (3.64)	<0.0001	29% (2.84)	43% (3.14)	0.54
If I had an improvement idea, I would know who to approach in my trust	41% (2.96)	79% (3.78)	<0.0001	14% (2.31)	29% (3.00)	0.16
Average responses to skills questions	59% (3.37)	75% (3.68)	0.0002	41% (3.26)	64% (3.38)	0.65

Percentages are number of participants responding ‘agree’ or ‘strongly agree’. Number in parenthesis is the average score used for statistical comparison.

The results for self-reported skills contrast starkly to the attitudinal ones. When questioned near the start of their first year, only 31% in the study group felt they had the skills to run an improvement project and 70% felt that they could contribute to improving efficiency at work. There was a statistically significant increase in these scores for the study group (who had taken part in the programme), which was not seen in the control group; however, this result needs to be taken with caution as the control group ended up being small. Relating to work efficiency, taking part in the programme increased doctors’ confidence relating to this from 72 to 97% and similar results were seen for reported skills relating to improvement ideas. Overall in the study group, there was a statistically significant increase on the average scores relating to self-reported skills from 59% to 75%.

Discussion

Kirkpatrick described a framework to evaluate educational interventions: Level 1 assesses the participants' experience, Level 2 their learning, Level 3 assesses any modification to behaviour and Level 4 whether the organisation improved its performance.⁹ Because of the complex nature of organisational performance, it is not possible to assess the impact of this programme at the organisational level. The questionnaire that is described below can assess up to Level 2, the participants learning and self-reported skills relating to QI.

The programme described above as well as the questionnaire evaluation demonstrates some important lessons relating to organic spread of programmes; the attitudes junior doctors have to QI, the effect of experiential programmes as well as the difficulty in evaluation in a quantitative manner.

The programme was initially piloted in one hospital before being intentionally spread to another hospital. Two further hospitals then ran the programme, so after a 4-year period half the hospitals in a training region in the South West of England were running the programme. This seemed a pragmatic time to formally evaluate the programme as there was a natural cohort that roughly divided the first-year doctors in half. However, three of the other hospitals in the region decided to run versions of the training programme, so the two cohorts became very unequal and the ‘control’ group very small. Organic spread of programmes is a very desirable outcome as it strongly suggests that the programme is at least perceived as successful; although it certainly does not prove the effectiveness. However, in this case, the dilution creates a significant evaluation problem. During their first year of training, first-year doctors reported significant changes in their skill levels related to QI. The average response to positively agreeing (‘agree’ and ‘strongly agree’) with questions relating to skills at QI went from 59% to 75%, which was a statistically significant result. No statistical change was seen in the control group but as mentioned the numbers were very small and so this result should be treated with caution. It is possible that the programme does not have any actual affect and first-year doctors in areas that do not run this programme also feel they acquire these skills. This seems unlikely as most other areas do not have specific programmes training junior doctors in QI.

The responses to the attitudinal questions are also interesting. First-year doctors have strongly held opinions to the importance of QI at the start of their professional career and this does not change whether they take part in a training programme or not. In relation to most of the domains of quality (patient experience, effectiveness and safety), first-year doctors agree (again the sum of ‘strongly agree’ and ‘agree’) about the significance of their role in improving the quality of care for their patients. Even in areas related to finance (not traditionally thought of as a responsibility of junior doctors), there is still strong agreement, between 77% and 84%. This is perhaps a surprise but is certainly a very positive finding for the future. The next generation of clinical leaders understand the importance and their key role in improving the care their patients receive. When they start as doctors, this positive attitudinal finding is not met by their self-reported skills at improving, averaging 59% agreement.

This study has significant limitations. As stated above, the small numbers in the control group mean that any results from inter-group comparison should be treated with caution. The low response rate is also a weakness. Overall, only 30% of the potential respondents filled out the survey, which is less than desirable but consistent with other published studies.^17,24,25 Additionally, a questionnaire-based survey can only assess the first two Kirkpatrick levels and cannot look to see whether the behaviour of the doctors had changed which is the ultimate aim of a programme such as this.

Conclusion

An experiential, structured, supported QI training programme for first-year doctors resulted in a significant improvement in self-reported skills relating to aspects of QI. This effect was not seen in the control group; however, small numbers in this group mean that this result should be treated with caution. First-year doctors understand the importance of improving quality in healthcare and believe that they have a significant role in achieving this.

Main messages

A structured, supported quality improvement training programme results in a significant improvement in self-reported skills related to quality improvement

First-year doctors understand the importance of improving quality in healthcare

First-year doctors believe they have a significant role to play in improving the quality of healthcare

Research questions

Does running a quality improvement training programme benefit participants beyond the planned period of the programme?

What are the most effective methods of developing future quality improvers?

How do we train senior clinicians in quality improvement?

Footnotes

Authors' contributions

All authors are involved in running the programme. They all took part in drawing the questionnaire together and the final writing of the report. RB and MD administered the survey and collated the results.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Dahill M, Bethune R, Carson-Stevens A, et al. First-year doctors' attitudes and beliefs relating to quality and safety improvement and patient safety. Clin Risk 2015; 21: 47–49.

Bethune

Soo

Woodhead

. Engaging all doctors in continuous quality improvement: a structured, supported programme for first-year doctors across a training deanery in England. BMJ Qual Saf 2013; 22: 613–617.

Neale

Vincent

Darzi

. The problem of engaging hospital doctors in promoting safety and quality in clinical care. J R Soc Promot Health 2007; 127: 87–94.

Lemer

Moss

. Patient safety and junior doctors: are we missing the obvious? BMJ Qual Saf 2013; 22: 8–10.

Wong

Etchells

Kuper

. Teaching quality improvement and patient safety to trainees: a systematic review. Acad Med 2010; 85: 1425–1439.

Ahmed

Arora

Tiew

. Building a safer foundation: the Lessons Learnt patient safety training programme. BMJ Qual Saf 2014; 23: 78–86.

Coltart

Cheung

Ardolino

. Leadership development for early career doctors. Lancet 2012; 379: 1847–1849.

Wears

. Improvement and evaluation. BMJ Qual Saf 2015; 24: 92–94.

Parry

Carson-Stevens

Luff

. Recommendations for evaluation of health care improvement initiatives. Acad Pediatr 2013; 13: S23–S30.

10.

Marshall

Pronovost

Dixon-Woods

. Promotion of improvement as a science. Lancet 2013; 381: 419–421.

11.

Portela

Pronovost

Woodcock

. How to study improvement interventions: a brief overview of possible study types. BMJ Qual Saf 2015; 24: 325–336.

12.

Osborne

Nicholas

Ryan

. Association of hospital participation in a quality reporting program with surgical outcomes and expenditures for Medicare beneficiaries. JAMA 2015; 313: 496–504.

13.

Cohen

Liu

. Improved surgical outcomes for ACS NSQIP hospitals over time: Evaluation of hospital cohorts with up to 8 years of participation. Ann Surg 2016; 263: 267–73.

14.

Velez

Becker

Davidson

. A quality improvement intervention to address provider behaviour as it relates to utilisation of CA-MRSA guidelines. J Clin Nurs 2015; 24: 556–562.

15.

Woloschuk

Harasym

Temple

. Attitude change during medical school: a cohort study. Med Educ 2004; 38: 522–534.

16.

Hutchinson

Cooper

Dean

. Use of a safety climate questionnaire in UK health care: factor structure, reliability and usability. Qual Saf Health Care 2006; 15: 347–353.

17.

Waterson

Griffiths

Stride

. Psychometric properties of the Hospital Survey on Patient Safety Culture: findings from the UK. Qual Saf Health Care 2010; 19: e2–e2.

18.

Madigosky

Headrick

Nelson

. Changing and sustaining medical students' knowledge, skills, and attitudes about patient safety and medical fallibility. Acad Med 2006; 81: 94–101.

19.

Ward R, Spencer R, Soo E, et al. Standardising the organisation of clinical equipment on surgical wards at North Bristol NHS Trust: a quality improvement initiative. BMJ Qual Improv Report 2015; 4: doi: 10.1136/bmjquality.u208308.w3441.

20.

Watson

Kendrick

. Management of anaemia and blood transfusion in critical care - implementing national guidelines in ICU. BMJ Qual Improv Report 2014; 3: doi: 10.1136/bmjquality.u202106.w1109.

21.

Dyar

Hall

McIntyre

. Warfarin prescription and administration: reducing the delay, improving the safety. BMJ Qual Improv Report 2015; 4: doi: 10.1136/bmjquality.u204509.w1983.

22.

Stockdale

Trivdei

Jerome

. Implementation of a combined cardiopulmonary resuscitation and treatment escalation plan document in a district general hospital. BMJ Qual Improv Report 2013; 2: doi: 10.1136/bmjquality.u202653.w1236.

23.

Verbakel

Zwart

Langelaan

. Measuring safety culture in Dutch primary care: psychometric characteristics of the SCOPE-PC questionnaire. BMC Health Serv Res 2013; 13: 354–354.

24.

Bos

Sizmur

Graham

. The accident and emergency department questionnaire: a measure for patients' experiences in the accident and emergency department–reply. BMJ Qual Saf 2013; 22: 787–788.

25.

Sarac

Flin

Mearns

. Hospital survey on patient safety culture: psychometric analysis on a Scottish sample. BMJ Qual Saf 2011; 20: 842–848.