Quality improvement training for core medical and general practice trainees: a pilot study of project participation,completion and journal publication

Design of project intervention

Design of project evaluation

A mixed methods evaluation (informed by a Logic model and the New World Kirkpatrick model ²¹ ) was undertaken incorporating quantitative analysis of survey responses, assessment of project proposals and completed reports and thematic analysis of one-to-one communications (see online Appendices 1 and 2).

Study participants, training attendance and project completion

A total of 23 trainees attended the training day (23/25, 92.0%), with 12 based in general practice training (52%) and 11 undertaking core medical training (48%). Mean age was 30.7 years (range: 25–50 years) and 16 were male (70.0%). Three individuals withdrew (12.0%) following the training day citing lack of time due to other career priorities (e.g. repeating examinations). In total, 20 participants submitted QI project proposals (87.0%) and 10 completed their project (43%) within the specified timeframe (Table 1) – lack of available time because of competing work and educational priorities, poor preparation and planning at the initial project design stage and limited project buy-in from colleagues were cited as reasons for non-completion by participants.

Level 1 – satisfaction with training interventions

All 23 participants completed the pre-course knowledge test and confidence assessment, with 14 completing the end of project evaluation (61%). The training day was rated as ‘very good’ or ‘excellent’ by all 23 participants. A majority agreed that the training had been effective in teaching them how to perform a QI project in their place of work and had changed how they would perform a QI project in future (10/14, 71.4%). The most useful training activities were reported as: ‘explanation of the QI tools’ (n = 9, 39%), ‘open discussion of previous QI projects including critical analysis of a published report’ (n = 7, 30%) and the group exercises on developing appropriate improvement measures (n = 7, 30%).

Four respondents suggested that greater integration of leadership, teamwork and systems-thinking theory into real-life practical scenarios would have improved the training. The majority (12/14, 85.7%) reported that the initial feedback received from their mentor on their QI proposal was helpful in planning their project, with seven indicating that having a mentor was helpful to them when undertaking their project (7/14, 50.0%). Eleven respondents reported using BMJ Quality (11/14, 78.6%) and of these eight found it a useful resource (8/11, 73%).

Levels 2 and 3 – confidence assessment and behaviour

Confidence in aspects of QI knowledge and skills was very low overall, improved after training and was largely maintained at four months. For example, in ‘leading a QI project’ (P < 0.05), ‘using the Model for Improvement/PDSA cycles’ (P < 0.05) and the understanding of ‘human error theory’ (P < 0.05) (Tables 1, 2). Although an increase in QIKAT score was observed (46% vs 54%) it was not statistically significant (P = 0.17).

In terms of project plans, the mentors (authors) judged that the majority (Table 3) provided a clear description of proposed project aims (15/20, 75%); were practicably feasible within timescale (15/20, 75%); contained acceptable improvement measures (16/20; 80%) – with some given additional feedback to further strengthen the approach outlined. In four projects the ‘measures’ were not specific or precise enough to be adequately quantifiable and suggestions were given for further improvement (4/20, 20%). The vast majority described the appropriate use of QI tools for the projects aims and approach (19/20, 95%), while evidence of necessary planning and improvement underpinned by systems thinking (15/20, 75%), team working (17/20, 85%) and leadership (15/20, 75%) was also present.

Level 4 – expert review and publication outcomes

Ten completed QI project reports on a variety of clinical and service topics using a range of QI methods were submitted (10/23, 43%) by participants (Table 4). Five were received from GPST3s (50%) and five from CMTs (50%), with eight projects in total being judged as satisfactory (8/10, 80%). Of those deemed unsatisfactory, one was viewed as research rather than QI, while the other was incomplete. Four projects were successfully published in BMJ Quality Improvement Reports (4/23, 17%).

Strengths and limitations

Study participants representing primary and secondary care settings was a particular strength. The small sample size was a limitation which was compounded by study withdrawals and incomplete QI projects. The end of project evaluation response rate was lower than expected which may have biased results. The project completion timescale was tight and possibly over-ambitious for some which may have influenced completion rates. The peer review tool used was validated for criterion-based audit projects rather than generic QI projects, but still proved to be of value.

A lack of generic QI skills, knowledge and experience among clinicians is one of the main barriers to improving care. Similar to our findings, previous studies have reported that confidence in QI understanding and application for both trainees and qualified specialists is low.^6,27 Although the provision of QI training is increasing, curriculum exposure to QI at undergraduate and postgraduate level is variable and a clear learning need for related training is still evident for most. ¹³ Confidence increased significantly following the training day and was partially maintained after four months. This may reflect that the “real life” experience of designing and leading a QI project uncovered hidden challenges and may be an important factor to consider in the planning of future training interventions and in the development of future mentors. The study would also have been strengthened by assessing and reporting the quality of improvement cycles undertaken by participants. For example, much of the feedback from mentors to participants was around application of the PDSA method (a problematic issue identified in a recent systematic review²⁷). It would have been interesting to determine the association between levels of confidence in this area and the quality of PDSA cycles undertaken and also publication (albeit numbers are small from a statistical inference perspective).

In previous evaluations of QI training, improvements in knowledge and confidence were often demonstrated. ¹³ Training has been found to be most effective when content mirrors real-life workplace situations, and delivery is by interactive seminars and small group work, as these were found to be more likely to influence behaviour as well as improve knowledge. The inclusion of a practical project has also often led to change in behaviour and care processes. ¹³ This mirrors the feedback received from participants on our training package and further integration of teaching on systems thinking, teamwork and leadership into real life scenarios may improve effectiveness.

Several previously published evaluations of QI training interventions have used a longer-term collaborative approach to learning,^6,29 which was not feasible in our small study. There is evidence that QI collaboratives are moderately effective, although they have often been used in conjunction with other interventions and it is not clear which aspects of collaborative learning determines success. ³⁰

The experience of mentoring provision was variable. A systematic review in 2010 identified that a lack of faculty was a barrier for effective completion of QI projects ¹⁵ and a need for further training of potential mentors has previously been identified. ³¹ Published evaluations have shown that the use of mentoring can be very effective but is dependent on the student-mentor relationship. ³²

A minority of participants was able to successfully publish their projects in a peer reviewed journal. The potential to spread findings and innovations reported in projects undertaken by doctors-in-training (and others) is clear. Some may feel that the goal of publication may be setting the bar too high; however, we have shown that publication is possible. Encouraging the peer review and publication of QI projects may result in feedback for trainees to improve the standard of future QI work, while also acting as a valuable resource for those planning projects by describing successful (or otherwise) approaches within the training environment. Additionally encouraging publication would enhance the professional development and career prospects of trainees, while also making an important knowledge contribution to the evolving science of QI.

Given its very modest impact our study project has shed light on a number of important concerns. If up-skilling the healthcare workforce and those in training in QI is to be taken seriously then the following issues may need to be considered by professional bodies and healthcare policymakers. From a QI perspective, the educational principles underpinning the teaching of related knowledge, the application of that knowledge in the workplace, the production of a related written report and the formative assessment of the standard of that work needs to be adhered to if true capacity and capability is to be built at the scale envisaged. However, putting aside those in training, the reality is that most clinical leads and others with QI leadership responsibilities are unlikely to have ever been subjected to these requirements or to this level of educational scrutiny. Participation in QI (or clinical audit) is a professional and regulatory obligation in the UK and a requirement for doctors-in-training. Most clinical and educational supervisors should have some experience of audit. However, given the myriad of available approaches to QI they are likely to lack knowledge of many related methods and their suitability, and on how to provide formative feedback in these areas. Similar to this study, the use of a peer review tool may help educational supervisors to formatively assess QI proposals and completed projects and provide developmental feedback to trainees.