Reducing the blame culture through clinical audit in nuclear medicine: a mixed methods study

Abstract

Objectives

To identify the barriers and facilitators of doctors’ engagement with clinical audit and to explore how and why these factors influenced doctors’ decisions to engage with the NHS National Clinical Audit Programme.

Design

A single-embedded case study. Mixed methods sequential approach with explorative pilot study and follow-up survey. Pilot study comprised 13 semi-structured interviews with purposefully selected consultant doctors over a six-month period. Interview data coded and analysed using directed thematic content analysis with themes compared against the study’s propositions. Themes derived from the pilot study informed the online survey question items. Exploratory factor analysis using STATA and descriptive statistical methods applied to summarise findings. Data triangulation techniques used to corroborate and validate findings across the different methodological techniques.

Setting

NHS National PET-CT Clinical Audit Programme.

Participants

Doctors reporting on the Audit Programme.

Main Outcome measures

Extent of engagement with clinical audit, factors that influence engagement with clinical audit.

Results

Online survey: 58/59 doctors responded (98.3%). Audit was found to be initially threatening (79%); audit was reassuring (85%); audit helped validate professional competence (93%); participation in audit improved reporting skills (76%). Three key factors accounted for 97.6% of the variance in survey responses: (1) perception of audit’s usefulness, (2) a common purpose, (3) a supportive blame free culture of trust. Factor 1 influenced medical engagement most.

Conclusions

The study documents performance feedback as a key facilitator of medical engagement with clinical audit. It found that medical engagement with clinical audit was associated with reduced levels of professional anxiety and higher levels of perceived self-efficacy.

Keywords

clinical audit medical engagement performance PET-CT

Introduction

A lack of medical engagement is known to represent a significant barrier to quality improvement within NHS England.¹ In the context of clinical audit, securing medical engagement is critical to its long-term success because it helps to facilitate organisational learning so that the same errors are not subsequently repeated by others. By fostering open cultures medical engagement can help doctors to re-frame error as a learning opportunity.^2–4 By engaging doctors in this process, clinical audit goes beyond being a tool of quality control by providing a vehicle for continuous improvement in standards of diagnostic reporting.^5,6

The NHS National PET-CT Programme was first implemented in 2008 as part of a wider Department of Health initiative to increase access of PET-CT imaging to cancer patients in England. The programme currently provides over 30,000 PET-CTs each year to NHS patients in England. Incorporated into the initiative is the process of National PET-CT Audit which requires that 10% of all clinical reports of PET-CT scans on the programme are independently and externally audited (Figure 1(a) and (b)). The implementation of clinical audit represents a process innovation as clinical audit has not previously been undertaken in this imaging modality within NHS England.

Figure 1.

(a) The National NHS PET CT Audit Programme process. (b) The National NHS PET CT Audit Programme categorisation of discrepancies.

Notwithstanding its importance, there is no universal definition of medical engagement. Our study adopts a definition used by Spurgeon et al.⁷ which defines medical engagement as a two-tier concept whereby (1) individual doctors are motivated to perform well and are willing to implement ideas for improvement and (2) such commitment is both recognised and supported by the organisation.

Purpose

We undertook a study of consultant nuclear physicians’ and radiologists’ (reporters) attitudes and motivation with regard to participation in the NHS National PET-CT Audit Programme. The objective was to identify and explore the key context-specific factors to influence reporters’ engagement with the clinical audit programme. A review of the literature pertaining to clinical audit, medical engagement and process innovation was undertaken. Seven variables thought likely to influence medical engagement were drawn from the general literature and underpinned the study’s theoretical frame:

usefulness of audit;

opportunity for sense-making;

good professional relationships;

non-judgemental culture of trust;

supportive infrastructure;

leadership;

shared purpose/vision.

The study sought to (1) identify whether these variables could be applied to a clinical audit context and (2) explore how they influenced reporters’ decisions to engage with clinical audit.

Method

A single-embedded case study of the NHS National PET-CT Audit Programme was undertaken. The case study design was influenced by Yin.⁸ A mixed methods approach was adopted so as to enable the study to ‘simultaneously answer confirmatory and exploratory questions, and therefore verify and generate theory in the same study’.^9,10

The study was comprised of two parts: (1) a pilot study comprising in-depth interviews to explore the factors likely to influence medical engagement and (2) a confirmatory online survey sent to all reporters on the clinical audit programme.

Phase one: pilot study

Thirteen semi-structured interviews were purposefully sampled from participating hospitals to ensure a range of perspectives were represented. The interview questions were informed by the study’s theoretical frame and were developed in conjunction with a critical reference group comprising two NHS managers, a social science researcher, one auditor and one doctor reporting on the programme. Some question items were taken from previous studies and adapted so as to accommodate the qualitative nature of the interview method.^11,12 The interviews were audio-recorded with the permission of the participant and the content of the recordings was transcribed verbatim and verified by the respondent to clarify meaning. The interview data were coded, and thematic content analysis techniques were used whereby themes were drawn from the data and matched against the study’s theoretical frame to facilitate generalisation of the data to the theory.¹³ Reliability of the data collection process was maintained by the adoption of a standardised interview protocol.

Phase two: online survey

The themes drawn from the pilot study interviews informed the survey. The survey comprised 35, five-point Likert scale questions ranging from strongly disagree to strongly agree. In addition, three open text questions invited respondents to comment on their attitude toward the audit programme as well as any changes they would like to see implemented. The anonymous survey was sent out electronically to all 59 reporters on the scheme. Fifty-eight responses were received. The data were transferred to an Excel spreadsheet, and explorative factor analysis was applied and descriptive graphs produced to summarise the results.

Results

The pilot study interview data provided a thick description of how each variable influenced reporter attitude and motivation. Nine themes emerged from the interview data which when mapped against the study’s theoretical propositions confirmed the study’s theoretical frame. The pilot study results can be accessed in Nuclear Medicine Communications.¹⁴

The phase two survey data provided quantitative data and confirmed the pilot study findings. Explorative factor analysis was applied to the survey data using STATA software as a means of achieving data reduction.¹⁵ To aid interpretation, orthogonal rotation was performed. The rotated factor analysis correlated the survey question items with one another and summarised the strength of their correlation as factor loadings as presented in Figure 2(a). Only items with factor loadings > 0.40 or higher were considered. The technique reduced the nine themes identified by the pilot study interviews to three underlying factors. Each of the three factors had an eigenvalue > 1.0 and a Cronbach alpha > 0.70 and together accounted for 97.6% of the variation in the survey results. The three factors were: (1) reporter perception of audit’s usefulness (USEFUL1), (2) a common purpose (PURPOSE2) and (3) a supportive and non-judgemental culture of trust (TRUST3). They accounted for 69.6%, 15.2% and 12.8% of the variance in the model, respectively (Figure 2(b)).

Figure 2.

(a) Rotated factor loadings for survey question items. (b) Three key factors facilitating medical engagement with clinical audit.

The survey results are grouped around these three key underlying factors together with their corresponding themes.

Factor 1: reporter perception of audit’s usefulness

Theme 1: audit makes me a better reporter

76% of reporters agreed or strongly agreed that participation in the audit programme improved their reporting skills, 15% disagreed or strongly disagreed and 9% were unsure (Figure 3(a)). Interestingly however, when the same question was asked specifically about their own reporting, only 48% of reporters agreed or strongly agreed, 24% disagreed or strongly disagreed and 28% of reporters were unsure.

Figure 3.

(a) Participation in audit enables reporters to improve their reporting skills. (b) My participation in the audit programme helps me to validate my professional competence. (c) Participation in the audit programme is reassuring for me as a reporter.

Theme 2: involvement in audit gives reporters credibility

Asked whether participation in the audit programme validated professional competence, 93% of reporters agreed or strongly agreed and 7% disagreed and 2% were unsure (Figure 3(b)).

Theme 3: audit is reassuring

Asked if participation in the clinical audit programme felt reassuring, 85% of reporters agreed or strongly agreed, 7% disagreed and 8% were unsure (Figure 3(c)).

Factor 2: communication and common purpose

Theme 4: audit initially lacked clarity of process and purpose

At the start of the audit programme, 50% of reporters agreed or strongly agreed that the purpose of audit was clear, 34% disagreed or strongly disagreed, 16% were unsure (Figure 4(a)). However, when the same question was repeated in the present tense, 81% of reporters agreed or strongly agreed that the purpose of audit was clear, 5% disagreed and 14% were unsure (Figure 4(b)).

Figure 4.

(a) The purpose of the audit process was clear at the outset. (b) I am clear about the purpose of the audit programme. (c) Auditor feedback is more meaningful when reporters have the opportunity to discuss findings with the auditors.

Theme 5: doves achieve more than hawks

The metaphor of a dove had emerged from the interview data to distinguish auditors that gave reporters the opportunity to discuss and make sense of their performance feedback. When asked if auditor feedback was more meaningful when they had the opportunity to discuss audit findings, 93% of reporters strongly agreed or agreed, 2% disagreed or strongly disagreed and 5% were unsure (Figure 4(c)).

Factor 3: a supportive non-judgemental culture of trust

Theme 6: audit felt threatening at the start

Asked if audit felt threatening at the start, 79% of reporters agreed or strongly agreed, 12% disagreed and 9% were unsure (Figure 5(a)). When the same question was repeated in the present tense, the percentage of reporters who agreed or strongly agreed dropped to 27% with 54% disagreeing or strongly disagreeing and 19% unsure.

Figure 5.

(a) The audit programme felt threatening at the start. (b) The audit process now provides a culture within which knowledge can be shared. (c) The audit process is a collaborative process between auditors, managers and reporters. (d) The audit process supports my needs as a reporter.

Theme 7: personalities and behaviour matter

Asked if they were happy with the way audit feedback was given, 41% of reporters agreed or strongly agreed, 41% disagreed or strongly disagreed and 18% were unsure.

Theme 8: from ‘done unto’ to collaborative learning

Asked if the audit programme had always provided a culture within which knowledge could be shared, 23% agreed or strongly agreed, 50% disagreed or strongly disagreed and 27% were unsure. When the same question was put in the present tense, 50% agreed or strongly agreed, 21% disagreed or strongly disagreed and 29% said they were unsure (Figure 5(b)).

Theme 9: audit is a collaborative and supportive process

Asked if the audit programme was a collaborative process between auditors, managers and reporters, 69% agreed or strongly agreed, 19% disagreed or strongly disagreed and 12% were unsure (Figure 5(c)). Further, when asked whether the audit process supported their needs, 54% agreed or strongly agreed, 30% disagreed or strongly disagreed and 16% of reporters were unsure (Figure 5(d)). This represented an improvement as only 28% of reporters agreed or strongly agreed that they were adequately supported at the start of the programme.

Overall the results indicated that levels of medical engagement increased over time with 66% of reporters more positive about the value of audit compared to when they first joined the clinical audit programme (Figure 6(a)). Further, reporters felt more supported compared to when they first joined the programme and 46% of reporters stated that participation in the audit programme had increased their job satisfaction. When asked whether they would wish to continue to participate in the clinical audit programme were it not mandatory, 62% of reporters agreed or strongly agreed that they would wish to continue participating in the audit programme with 9% disagreeing and 29% unsure (Figure 6(b)).

Figure 6.

(a) I am more positive about the value of audit in PET- CT compared to when I first joined the audit programme. (b) I would wish to participate in the National NHS Audit Programme even if it was not mandatory.

Discussion

In general, we found the variables influencing engagement described in the general literature also held when transferred to a clinical audit context. For example; a non-judgemental culture of trust, usefulness, shared purpose, good professional relationships and a supportive infrastructure.¹ However, the context-specific nature of the study enabled us to better evaluate which of these variables were more important for clinical audit. Previous research has tended to focus on identifying the characteristics of organisations with high levels of medical engagement, whereas our study was set at the individual/team level as we were interested in understanding the context-specific factors that influenced doctors’ engagement with the process of clinical audit. This study adds to the medical engagement literature by undertaking quantitative and qualitative research methods to identify and explore the key contextual factors to influence doctors’ engagement with the NHS National Clinical Audit Programme. These were identified as (1) reporter perception of audit’s usefulness, (2) good communication and a common sense of purpose and (3) a supportive and non-judgemental culture of trust.

First, with regard to reporter perception of audit’s usefulness, our study highlights the key contextual influence of reporter performance feedback. To the best of our knowledge, the influence of performance feedback on medical engagement has not been previously documented within the context of a clinical audit programme. Our results showed that performance feedback had a significant influence on reporter perception of audit’s usefulness because of its ability to reassure reporters. Performance feedback allowed reporters to compare their performance against the anonymised performance of other reporters. This reassured reporters that any errors they made were most likely a product of the normal variance that affected all reporters over time. The study therefore showed that over time reporters valued audit feedback as they associated it with a process of professional validation whereby their performance was formally recorded and recognised.

In addition, the dissemination of reporter performance data highlighted the contested nature of imaging data and the extent to which diagnostic imaging as a science was dependent on interpretation, professional judgement and consensus building. Awareness of this emphasised the importance of actively including reporters in an open discussion of their performance so as to ‘better agree on what should be called discrepancies…’ (reporter survey response). Overall a change in attitude toward the usefulness of audit occurred because clinical audit was shown to provide reporters with ‘feedback on reporting ability and reassurance that the reports [were] of a good standard’ (reporter survey response).

Second, our findings emphasised the importance of good communication and a common sense of purpose. The pilot study had indicated that because clinical audit was a new process reporters needed time to make sense of its purpose. According to sensemaking theory, people organise to make sense of equivocal inputs by categorising events so that they fit with their own local context.¹⁶ The survey data lent support for this by showing how over time the purpose of audit became clearer and more aligned between reporters, managers and auditors:

At the beginning I had reservations because of the way it [audit] was translated into practice – not being able to discuss issues and the grading. I developed a more positive attitude towards it through my own experience and the annual national PET-CT audit days. (Reporter survey response)

This finding is consistent with Johnston et al. who undertook a review of the literature of clinical audit and concluded that doctors needed the space to ‘control the audit process using their own values and attitudes’.¹⁷

Third, our results underscored trust as a key facilitator of medical engagement with clinical audit. Often referred to in the literature as psychological safety, the concept is defined as a ‘sense that the team will not embarrass, reject, or punish someone for speaking up’.¹⁸ Its role in facilitating a positive team climate is recorded in the literature by Edmonson et al.^19–21 who showed how psychologically safe environments promoted learning between healthcare professionals in hospitals. Related to this our study demonstrated how a culture of blame initially eroded medical engagement with clinical audit. While the phenomenon of a ‘blame culture’ in medicine is described in the general literature, its impact on diagnostic reporting has not been previously researched within a clinical audit context.²² The findings of this study show how a culture of blame which existed at the start of the programme was associated with high levels of anxiety whereby the willingness of reporters to share clinical audit experience was curtailed. As a result, defensive reporting styles were adopted so that diagnostic reports became ‘much wordier reports to cover every little thing even if not relevant/already known’ (reporter survey response). In contrast, the study’s findings show how a more collaborative approach to clinical audit was associated with reduced levels of anxiety, increased levels of trust and a greater willingness to share knowledge. For example, the choice of telephone communication in place of email and the use of a neutral term such as ‘discrepancy’ as opposed to ‘error’ made reporters feel psychologically safer because these words and deeds were perceived to be symbolic of a more inclusive approach to clinical audit. The survey data confirmed that a change in reporter perception of audit from being thought of as threatening to something reassuring occurred: ‘Yes the audit process has lost the aggressive edge it had at the start. I view it as a supportive process rather than an adversarial one’ (reporter survey response). This fits with the conclusions of Hogan et al.²³ who found the most effective clinical audits to be those that involve the whole team. It also supports the findings of Vogus et al.²⁴ which stressed the need for organisations to create environments within which practitioners feel safe to speak up. The study’s findings therefore indicate that by supporting non-judgemental cultures of trust and guiding sensemaking, healthcare organisations can better facilitate medical engagement with clinical audit.

There were three main limitations of the study. First, the wider organisational and policy factors that may have affected reporter engagement were not taken account of. Second, the data were partially dependent on respondent recall so factors perceived as insignificant to the respondent are less likely to have been aired. Third, the study was undertaken within a mandated NHS environment whereby reporters were required to participate in clinical audit. The extent to which the study’s findings transfer to a voluntary audit environment is unclear and therefore represents an area for future research. The advantage of a mandated environment however was that there was no scope for self-selection out of or into the population being studied which meant the likelihood of sample selection bias within the population was limited.

Conclusion

Overall our survey findings confirm the general medical engagement literature and extend our understanding to the clinical audit context.^25–28 The study demonstrates how a supportive clinical audit environment can help doctors to more fully engage with clinical audit by reducing levels of anxiety and inducing a non-judgemental culture of trust within which a common sense of purpose and empathy for clinical colleagues can be promoted.^29,30 In contrast, the study’s data showed how an unsupportive environment constrained medical engagement; evidenced in lower levels of dialogue between doctors, reduced levels of trust, increased anxiety and a perceived culture of blame in which doctors adopted defensive reporting behaviours. The study also demonstrates the contextual nature of the facilitators of medical engagement. Three underlying factors were found to be most important in facilitating medical engagement with NHS National Clinical Audit Programme: (1) awareness of audit’s usefulness and the related influence of performance feedback, (2) good communication and a common purpose and (3) a non-judgemental culture of trust.¹²

Footnotes

Declarations

Acknowledgements

We thank Dr Marion Frenz (Senior Lecturer in Management at Birkbeck University of London) for assistance with the study.

Provenance

Not commissioned; peer-reviewed by Roland Hustinx

References

Atkinson

Spurgeon

Clark

Armit

. Engaging doctors: what can we learn from trusts with high levels of medical engagement?, UK: NHS Institute for Innovation and Improvement and Academy of Medical Royal Colleges, 2011.

Davies

Nultey

. Developing learning organisations in the new NHS. Br Med J 2000; 320: 998–1001.

Barwell

. New principles of best practice in clinical audit, London: Radcliff Publishing, 2011.

Senge

. The fifth discipline: the art and practice of the learning organisation, New York: Currency Doubleday, 1994.

Mirzaei

Maffioli

Hilson

. Clinical Audit in nuclear medicine. Eur J Nucl Med Mol Imag 2011; 38: 3–4.

Peters

Bomanji

Ell

Gordon

Hilson

Murrain

. Clinical governance improves the quality of nuclear medicine reporting. Nucl Med Commun 2008; 29: 999–1001.

Spurgeon

Barwell

Mazelan

. Developing a medical engagement scale (MES). Int J Clin Leadership 2008; 16: 213–223.

Yin

. Case study research, design and methods, California: Sage Publications, 2009.

Tashakkori

Teddlie

. Mixed methods in social and behavioural research projects, 2nd ed. London: Sage Publications Inc, 2010.

10.

Pope

Mays

. Qualitative research in health care, 3rd ed. Malden, MA: Blackwell Publishing Inc, 2006.

11.

Greenhalge

Robert

Bate

Macfarlane

Kyriakidou

. Diffusion of innovation in health service organisations, Oxford: Blackwell Publishing, 2005.

12.

Clark

. Medical engagement: too important to be left to chance, London: The King’s Fund, 2012, pp. 1–17.

13.

Eisenhardt

. Building theories from case study research. Acad Manag Rev 1989; 14: 4–4.

14.

Ross

Hubert

Saunders

Wong

. The importance of a supportive environment in clinical audit: a pilot study of doctors’ engagement with the NHS National PET-CT audit programme. Nucl Med Commun 2014; 35: 1052–1057.

15.

Frenz M and Lambert R. Mixed modes of innovation: an empiric approach to capturing firm’s innovation behaviour. OECD Science, Technology and Industry Working Papers, DSTI/DOC(2012)6. Paris, France: OECD Publishing, 2012.

16.

Weick

Sutcliffe

Obstfeld

. Organizing and the process of sensemaking. Org Sci 2005; 16: 409–421.

17.

Johnston

Crombie

Davies

Alder

Millard

. Reviewing audit: barriers and facilitating factors for effective clinical audit. Qual Health Care 2000; 9: 23–36.

18.

Edmonson

. Psychological safety and learning behaviour in work teams. Admin Sci Q 1999; 44: 350–383.

19.

Edmonson

Bohmer

Pissano

. Disrupted routines: team learning and new technology implementation in hospitals. Admin Sci Q 2001; 46: 685–716.

20.

Firth-Cozens

. Cultures for improving patient safety through learning: the role of teamwork. Qual Health Care 2001; 10: 26–31.

21.

Dopson

Fitzgerald

Ferlie

Gabbay

Locock

. No magic targets! Changing clinical practice to become more evidence based. Health Care Manag Rev 2002; 27: 35–47.

22.

Waring

. Beyond blame: cultural barriers to medical incident reporting. Soc Sci Med 2005; 60: 1927–1935.

23.

Hogan

Basnett

Mckee

. Consultants’ attitudes to clinical governance: barriers and incentives to engagement. Publ Health 2007; 121: 614–622.

24.

Vogus

Sutcliffe

Weick

. Doing no harm: enabling, enacting, and elaborating a culture of safety in health care. Acad Manag Perspect 2010; 24: 61–77.

25.

Dickinson H, Ham C, Snelling I and Spurgeon P. Are we there yet? Models of leadership and their effectiveness: an exploratory study. London: NHS National Institute for Health Research, 2013.

26.

MacLeod D and Clarke N. Engaging for success: enhancing performance through employee engagement. UK: Department for Business Innovation and Skills, 2009.

27.

Gagné

. The Oxford handbook of work engagement, motivation and self-determination theory, New York: Oxford University Press, 2014.

28.

Bailey

Madden

Alfes

Fletcher

Robinson

Holmes

. Evaluating the evidence on employee engagement and its potential benefits to NHS staff: a narrative synthesis of the literature. Health Serv Deliv Res 2015; 3: 26, doi:10.3310/hsdr03260.

29.

Von Krough

. Care in knowledge creation. California Manag Rev 1998; 40: 133–153.

30.

Zárraga

Bonache

. The impact of team atmosphere on knowledge outcomes in self-managed teams. Organ Stud 2005; 26: 661–681.