How is feedback from national clinical audits used? Views from English National Health Service trust audit leads

Abstract

Objective

To explore how the output of national clinical audits in England is used by professionals and whether and how their impact could be enhanced.

Methods

A mixed-methods study with the primary recipients of four national clinical audits of cancer care of 607 local audit leads, 274 (45%) completed a questionnaire and 32 participated in an interview. Our questions focused on how the audits were used and whether barriers existed to using the audits for local service improvement. We described variation in questionnaire responses between the audits using chi-squared tests. Results are reported as percentages with their 95% confidence intervals. Qualitative data were analysed using Framework analysis.

Results

More than 90% of survey respondents believed that the audit findings were relevant to their clinical work, and interviewees described how they used the audits for a range of purposes. Forty-two percent of survey respondents said they had changed their clinical practice, and 56% had implemented service improvements in response to the audits. The degree of change differed between the four audits, evident in both the questionnaire and the interview data. In the interviews, two recurring barriers emerged: (1) the importance of data quality, which, in turn, influenced the perceived relevance and validity of the audit data and therefore the ability to make changes based on it and (2) the need for clear presentation of benchmarked local performance data. The perceived authority and credibility of the professional bodies supporting the audits was a key factor underpinning the use of the audit findings.

Conclusion

National cancer audit and feedback is used to improve services, but their impact could be enhanced by improving the data quality and relevance of feedback.

Keywords

effectiveness hospitals national clinical audit quality improvement

Introduction

National clinical audits (NCAs) are well established in the National Health Service (NHS) in England as a means of assessing the quality of care with the aim of stimulating quality improvement. There are currently about 50 NCAs covering many areas of health care.¹ By measuring local performance, assessing compliance on the process and delivery of care against evidence-based standards and providing feedback, it is expected that the feedback will prompt local service improvements. Whether improvements are made by the professionals who read them remains uncertain, and it is not known how best to design NCAs to achieve this aim.

Kluger and DeNisi² proposed a model for understanding audit feedback, called Feedback Intervention Theory (FIT). It suggests there are three factors that determine whether people are likely to turn their attention to the feedback that they receive: characteristics of the feedback (content, format and frequency); the nature of the task performed, and situational or personality variables. The focus of this study is on the first of these as it is the component that can be influenced by the provider of an NCA.

NCAs include: benchmarking of local practice and performance; detailed analysis and feedback from a professional body or government; publically available findings and national monitoring or regulation, all of which are designed to prompt behavioural and organizational change.^1,3

Several national audits have documented improvements in patients’ care since their introduction,^4–7 but a systematic review found that results were heterogeneous.⁸ It found that multifaceted approaches to quality improvement, such as educational meetings on the interpretation of the feedback, were more successful than single approaches, and process measures tended to have greater influence than outcome measures. However, there is still a lack of evidence on the mechanisms of improvement and on local responses to national audit.

This study sought to understand perceptions of whether NCAs can be effective and how their impact can be enhanced. Key interests were to gather information on the scope and content of audit reports, their periodicity and the format of the audit output, gaining an understanding of the pathways for their local dissemination, and insight into the relevance of audit recommendations and potential barriers to their implementation. Specifically, we pursued three research questions: what is the perceived potential usefulness of NCAs and their feedback? How is published feedback actually used to improve local services? What are the main perceived barriers to acting on feedback for improvement?

Methods

Study design

We used a mixed-methods explanatory sequential design, comprising quantitative and qualitative components.⁹ The study was conducted between April and September 2014 and focused on the National Oesophago-Gastric Cancer Audit (NOGCA),¹⁰ National Bowel Cancer Audit (NBOCAP),¹¹ National Head and Neck Cancer Audit (DAHNO)¹² and the National Lung Cancer Audit (NLCA).¹³ These were chosen because they were well established and shared a common focus on cancer, while at the same time, they differed in scope, output and maturity (Table 1). They report national outcomes by NHS trusts (clusters of hospitals with shared management)¹⁴ and by Cancer Network. The four audits covered more than a third of all deaths from cancer in the UK. NOGCA, NBOCAP and DAHNO introduced consultant-level reporting at the time of the study.

Table 1.

Description of the national clinical audits at time of data collection (NOGCA, NBOCAP, NLCA all 2013; DAHNO, 2012).

	NOGCA	NBOCAP	DAHNO	NLCA
Year of publication of first national report	2008	2005	2006	2005
NHS trust participation	99%	99%	100%	100%
Number of patients included in report (annual)	11,516	29,445	8209	33,385
Estimated case ascertainment rate	83%	86%	96%	98%
Number of process of care indicators^a, reported by:
NHS trust	0	10	1	14
Cancer Network	8	4	18	22
Number of patient outcome indicators^a, reported by:
NHS trust	4	4	0	0
Cancer Network	0	1	2	1

Indicators are counted if NHS trusts or Cancer Networks are specifically named in tables or figures in the most recent annual report.

NOGCA: National Oesophago-Gastric Cancer Audit; NBOCAP: National Bowel Cancer Audit; NLCA: National Lung Cancer Audit; DAHNO: National Head and Neck Cancer Audit; NHS: National Health Service.

Survey

The study questionnaire was developed based on previous research on audit and feedback.^2,7,15–19 It focused on views on the content of the audit feedback; and the purpose, impact and barriers to using the feedback. Items were presented as five-point Likert scales, with ‘strongly agree’ to ‘strongly disagree’, and ‘very important’ to ‘of little importance’, including a ‘not applicable’ where appropriate.

Total population sampling was used for the questionnaire, and the same questionnaire was sent to contacts from all four audits. The Health and Social Care Information Centre (HSCIC) maintains lists of audit contact leads for each of the four national cancer audits, with a contact lead for each English NHS trust and foundation trust. Contact leads include clinical and non-clinical professionals, as described in Table 2. Each lead was sent the electronic questionnaire (n = 144 for NOGCA, n = 144 for NBOCAP, n = 145 for DAHNO and n = 174 for NLCA) by HSCIC administrators via an email link in April 2014. Data collection was completed five weeks later. In order to maximize response rate, four reminder emails were sent. The questionnaire was anonymous, but survey respondents could volunteer to be interviewed.

Table 2.

Participant characteristics.

Audit	NOGCA	NBOCAP	DAHNO	NLCA	Overall
Questionnaire
Number of participants	75	83	53	63	274
Seniority (median years since graduation (IQR))	16 (9, 20)	17 (14, 24)	14 (9, 17)	12 (10, 17)	15 (11, 20)
Profession
Surgeon	27	77	36	1	141
Oncologist and endoscopist	13	1	7	5	26
Respiratory physician	–	–	–	52	52
Non-clinical	17	1	1	1	20
Clinical nurse specialist	14	2	5	3	24
Interview
Number	8	8	5	11	32
Sex (number of males)	5	4	5	8	22
Seniority (median years since graduation (IQR))	29.5 (25, 34.5)	23.5 (17.8, 26.5)	20 (14, 25)	20 (17, 23)	23 (18.5, 26.5)
Profession
Surgeon	5	8	3	1	17
Oncologist and endoscopist	2	–	1	1	4
Respiratory physician	–	–	–	9	9
Clinical nurse specialist	1	–	1	–	2

Figures are actual numbers unless otherwise specified.

NOGCA: National Oesophago-Gastric Cancer Audit; NBOCAP: National Bowel Cancer Audit; DAHNO: National Head and Neck Cancer Audit; NLCA: National Lung Cancer Audit; IQR: interquartile range.

Interviews

A convenience sample of all volunteers was interviewed to probe further into the general survey responses and better understand why the survey responses were selected. An interview guide was developed, focusing on two key areas, based on the questionnaire: probing views on the design of the audit output and use of the audit (Appendix 1). Questionnaire responses were studied prior to the interviews to allow for specific probing. Interviews were conducted via telephone from private offices and were recorded. Interviews lasted on average 30 min. There were three interviewers, two of whom were trained over the course of the day on the topic and interview guide by the main interviewer. The main interviewer checked the other interviewers’ transcripts throughout to ensure a consistent approach.

Analysis

Questionnaire response data were dichotomized into positive responses (‘agree/strongly agree’ and ‘important/very important’) and neutral or negative responses.²⁰ These were used to calculate proportions of positive responses. We compared the four NCAs by clinical speciality, content, format, delivery of feedback and how established they are. We assessed variation in responses across the audits using chi-squared tests. Results are reported as percentages with their 95% confidence intervals. Quantitative data were analysed using Stata 12.

Following analysis of the quantitative data, interviews were transcribed ‘smooth verbatim’ and sent to each interviewee for member checking, with corrections limited to amending mistyped words. Qualitative data were analysed using Framework analysis, a social policy approach.²¹ The transcripts were read repeatedly to gain familiarity and were annotated in the margins. Main themes and sub-themes were identified iteratively and noted on the transcripts, at the same times as producing a coding tree. Themes were derived from the data and were charted by comparing responses from each interviewee for each theme and sub-theme. Data were mapped and interpreted by drawing conceptual diagrams and returning to the data to check for consensus and deviant cases. The saturation point in the thematic analysis was achieved across the four audits.

Ethics

We provided all interviewees with an information sheet, which included a brief statement on aims, use of data, access to findings and assurance of confidentiality. Questionnaire respondents were also given this information. The study was covered by existing ethics approvals of the Confidentiality Advisory Group, NHS Health Research Authority. All participants were provided with a summary of the findings.

Results

Survey findings

Out of 607 local audit leads, 274 (45%) responded to the questionnaire: NOGCA (52%), NBOCAP (58%), DAHNO (37%) and NLCA (35%). Respondents were typically experienced consultant doctors (Table 2).

Views on usefulness of audit for stimulating quality improvement

Overall, respondents felt that the national audits provide a unique source of information relevant to quality improvement. Across the four audits, 83% respondents agreed with the statement, ‘The audit provides access to information on clinical processes and outcomes that are not available from other sources’.

A high proportion of respondents (88%) thought that the audit reports are important to identify opportunities for quality improvement, whilst 86% believed that the reports are important for facilitating team discussions on quality issues (Table 3). These findings did not differ across audits.

Table 3.

Questionnaire responses on purpose of using audit report, impact of audit and barriers to using the report, showing % agreement with listed statements.

	% important/very important
	Overall (95% confidence interval)	NOGCA	NBOCAP	DAHNO	NLCA	p value
‘In your opinion, how important are the audit findings for the following purposes?’
To identify opportunities for quality improvement (n = 254)	88 (84–92)	92	88	85	87	0.75
To facilitate team discussions regarding quality and safety issues (n = 254)	86 (81–90)	87	84	83	89	0.80
To benchmark outcomes at my NHS trust with that of my peers (n = 254)	84 (79–89)	89	87	75	84	0.22
To facilitate discussions regarding service reorganization (n = 254)	73 (68–80)	76	70	65	82	0.18
To compare the volume of care at my NHS trust with that of my peers (n = 254)	70 (64–75)	72	70	71	66	0.88
To use the data for revalidation (n = 254)	63 (56–69)	66	69	60	53	0.22
To use the data for appraisal (n = 254)	61 (55–67)	63	70	54	53	0.13
‘In response to the audit…’	% agree/strongly agree
I have become more aware of levels of performance amongst my peers (n = 246)	65 (59–71)	69	69	51	66	0.19
I have become more aware of patterns of practice amongst my peers (n = 245)	64 (58–70)	67	66	62	60	0.84
I have become more aware of levels of performance in my Trust (n = 244)	60 (54–64)	52	60	54	75	0.06
I have implemented service improvements (n = 236)	56 (50–63)	54	40	61	75	0.00
I have become more aware of patterns of practice in my Trust (n = 242)	54 (48–60)	39	54	52	72	0.00
I have changed aspects of my clinical practice (n = 229)	42 (36–49)	38	25	48	64	<0.00
I have used audit findings to make a business case for an investment in some aspect of service improvement (n = 217)	38 (31–45)	30	31	33	58	0.00
I have been asked to investigate potential quality of care issues in my Trust (n = 208)	37 (30–44)	27	35	22	62	<0.00
I have received rewards for good outcomes (n = 218)	11 (7–15)	7	10	11	14	0.70
‘What are the main reasons that limit you using the audit findings?’	% agree/strongly agree
There are not enough resources (n = 243)	71 (65–76)	69	69	74	72	0.95
There is not enough support within my hospital/Trust/SCN (n = 243)	57 (51–64)	56	55	74	48	0.06
There is insufficient time to interpret the data presented in the report (n = 243)	36 (30–42)	38	33	37	36	0.95
The findings do not translate into clear actions for improvements (n = 243)	34 (28–40)	31	34	46	29	0.29
The report is too long (n = 246)	28 (22–34)	30	28	41	13	0.01
The audit report is too difficult to understand (n = 245)	17 (13–22)	14	20	22	13	0.47
Its findings are not relevant to my clinical work (n = 239)	9 (5–13)	10	12	11	1	0.16

The p value indicates statistical significance of the difference between the four audits.

NOGCA: National Oesophago-Gastric Cancer Audit; NBOCAP: National Bowel Cancer Audit; DAHNO: National Head and Neck Cancer Audit; NLCA: National Lung Cancer Audit; NHS: National Health Service; SCN: Strategic Clinical Networks.

How audit feedback is used for improvement

The audits had been used successfully by some respondents (Table 3). For example, 56% had implemented service improvements, and 38% had used the audit output to make a business case. However, there was variation in the reported impact between the four audits, with a higher proportion of NLCA respondents reporting an impact compared to respondents from the other three audits.

Barriers to using audit feedback for improvement

The reports were generally not viewed as too difficult to understand or as too long (with length of reports being typically around 60 pages), and 91% agreed that the audit findings are relevant to their clinical work. However, a third of respondents thought that the findings do not translate into clear actions for improvement (Table 3).

With respect to local organizational support, 71% of respondents stated that a lack of resources had restricted them from using feedback, and more than half said that a lack of support within their hospital, trust or Strategic Clinical Networks had limited their ability to use the audit output. Based on the logistic regression analysis, there did not appear to be relationships between support and resource barriers and implementation of service improvements, business cases and clinical practice changes.

Views on content of audit feedback

All four audit groups were probed on the same features, regardless of whether they were included in the audit output. The majority of respondents scored the content features favourably (Table 4). Ninety-four percent of respondents thought that the report summary is important, and 87% believed that comparative Trust outcomes are an important feature. In contrast, only half of respondents agreed that comparative consultant outcomes were important. These responses were relatively consistent across the four audits (except for consultant outcomes).

Table 4.

Questionnaire responses on content of report, showing % agreement with listed statements.

	% important/very important
‘In your opinion, how important are the following audit features?’	Overall (95% confidence interval	NOGCA	NBOCAP	DAHNO	NLCA	p value
Summary of audit findings (n = 256)	94 (91–97)	96	97	90	92	0.26
Comparative Trust-level outcomes (n = 255)	87 (82–91)	92	83	85	86	0.42
Recommendations based on the audit findings (n = 256)	84 (79–86)	95	83	74	84	0.02
Summaries of current guidelines and policies (n = 255)	84 (79–86)	85	81	92	82	0.40
Local action plans (n = 256)	69 (63–74)	79	64	63	67	0.18
Expert commentaries (n = 255)	57 (50–63)	61	48	61	59	0.35
Patient testimonials (n = 254)	50 (43–56)	61	37	49	53	0.04
Comparative consultant outcomes (n = 256)	49 (42–55)	60	52	53	28	0.00

The p value indicates statistical significance of the difference between the 4 audits.

NOGCA: National Oesophago-Gastric Cancer Audit; NBOCAP: National Bowel Cancer Audit; DAHNO: National Head and Neck Cancer Audit; NLCA: National Lung Cancer Audit

Interviews

In total, 32 participants were interviewed (Table 1). The themes selected are grounded in the data and emerged as significant topics from the mapping and interpretation analytical stage (Appendix 2).

The value of NCAs

The audits were universally viewed as important and valuable. Some interviewees expressed support for clinical audits being national. This support was often coupled with statements about how the audit reports are a valuable source of authority on the specific care pathways. According to interviewees, this is due to the credibility of the established professional societies and authoritative bodies associated with the reports, rather than the content of the reports themselves: ‘I don’t think DAHNO has ever told us anything we didn’t already know, it’s just given us data that’s authoritative […]. It’s providing data that makes people realise that we’re telling the truth’.

This authority appeared to be particularly important for some interviewees in making business cases or using audit data as evidence to use with managers or commissioners.

Additionally, most interviewees specifically detailed that they are useful for presenting an overview of current clinical practice, enabling local teams to identify areas for improvement, providing reassurance about current practice or reinforce what is already known, and for use as evidence for service planning and for making business cases:

It’s easier to identify [levels of performance] and show in something like a national audit rather than individually here in the hospital. NBOCAP

It’s very useful to have the data, to be able to say [to management], ‘This is what we’re good at and this is what we’re bad at and this is what we need more investment in. NLCA

However, there was variation in the extent to which the audits were described as providing this feedback: some interviewees described the potential purpose of the audit reports, while others spoke about how the reports are currently being used:

I think it allows reflection on the service and working out what could be improved. NLCA

I think the purpose ought to demonstrate, first of all, the quality of what’s happening as a sort of pen picture. NOGCA

While the audit reports are uniformly perceived as important and are valued, not all were perceived as achieving maximum value to NHS trusts.

Use of national audit feedback

All interviewees accessed the audit output, but the degree of use varied. Most described flicking quickly through to particular sections. Many interviewees were responsible for circulating the audit output within their team. The majority described discussing the audit output in meetings (both formal and informal). At meetings, the audit reports were analysed and presented in a number of ways, and this appeared to be dependent, to some extent, on the data included in the audit output. This included simply looking at national figures contained within the report, comparing their own additional calculations with national figures from the report, and finding how they compared with other NHS trusts. Many seemed to use the meetings to identify whether improvements were required. Some interviewees made plans to change, while others simply discussed the results. Some interviewees described using the audit report to make business cases or to present evidence of their current practice and service. A small number of interviewees had not made changes because they felt that they do not need to, either because they had been identified as having good or excellent performance by the audits, or because they believed they were already performing sufficiently well. A number of interviewees had not been able to make any changes, mainly because the audit report had not presented the relevant data in the right way, or because of internal issues. For others, the audits had contributed to small or significant changes:

I wouldn’t say the national audit was the only thing that contributed to that [change], but it certainly helped us with our funding for PET-CT. Because we were able to show, you know, what percentage of other centres were doing it and we weren’t, so that did help us with the funding for that but, you know, I wouldn’t say that the national audit was the only source of information. NOGCA

I think overall things have improved and our surgery waiting times and chemotherapy waiting times are much shorter than they used to be five years ago, and I think the audit data has probably helped with that by focusing how we’re doing there. NLCA

A few interviewees spoke about how they planned to use the audit report:

This time I will have that headline figure from DAHNO and, as I say, we don’t like shouting about it but we’re hopefully going to shout about it in a useful way and say, ‘Look, the national audit says we’re the worst in the country, are you happy with that or are you going to invest in it to change it?’ DAHNO

Interviewees offered numerous perceived impacts of the audits. For some, the effect has been to raise awareness; for others, it has been to acquire new resources, recruit more staff and to change clinical practice and change service organization. This variation was clear between the four audits, with NLCA interviewees speaking most often about the changes they have made.

Barriers to use of national audit feedback

Interviewees described numerous changes that should be made to increase the impact of the audits. Two themes were repeated continually by interviewees: the importance of data quality and the need for presentation of clear comparative local results.

The importance of data quality was a major barrier that emerged from the interviews. Although audit data collection and submission were not a study focus, it frequently came up as an issue affecting the quality of the data presented in the audit. For many, poor data quality compromised the ability to use the reports to effect changes.

The data has to be right in order for the commissioners to be able to act on it. NOGCA

If you can’t get a fair data, then you can’t do a fair analysis. DAHNO

Here, ‘fair’ data referred to data quality and suggested that data are used to judge individuals and organizations. Interviewees also frequently described how different hospital and audit data systems ‘don’t talk to each other’, resulting in duplication of data collection and storage, loss of time and opportunity to use data effectively. Some interviewees stated that their NHS trust simply does not have enough funds to support audit data collection and submission, while others said that their NHS trust (managers) do not prioritize the audit and its data collection and submission.

The [NHS] trust is a sort of huge organisation that has achieved an awful lot, and it doesn’t appear to recognise the need for quality of clinical data to be collected accurately, because I think it isn’t a national target that they are immediately judged upon […].They’re busy measuring themselves against other things like hand washing and all the other things that are considered to be important for national targets. NOGCA

A small number of interviewees felt that data quality might be slowly improving with increased reporting of consultant outcomes.

The second barrier strongly emphasized was the need for comparative local data. The reported availability of this information differed between audits. It seemed to be a very useful source of evidence to take to managers and commissioners, as well as important to help people identify whether they were performing within an acceptable range. Many spoke about how their own figures are the most important information in the audit output. When asked what they first look at when they open the audit report:

Usually it’s to look at the data for my hospital and make sure it looks accurate, then look at the data for our region, and then for the surrounding parts of London to see how we compare. And sometimes I do look at the national summaries, but it’s usually more about comparing with our regional neighbouring [NHS] trusts. NLCA

Some described how comparative Trust results motivated them to improve, driven by a competitive spirit. Where interviewees reported having made changes, they tended to refer to looking specifically at how they performed locally in comparison to others in the audit report. In contrast, where interviewees reported not having made changes, they tended to speak about how information on their local practice is not available to them:

At the moment we can’t go to the commissioners and say we need some more money to invest into our upper GI cancer service provision […]. We haven’t that evidence yet. […] There is nothing specific that I can use from the report. NOGCA

Discussion

Main findings

NCAs are valued as an authority on cancer services. More than half of survey respondents had implemented service improvements in response to audits, confirming that the audit cycle had been completed. However, the use of audit feedback varied between teams and across the four audits, and it is evident that some NHS trust teams are currently not using the audit feedback for improvement. Barriers include inadequate benchmarking of local results in audit feedback and a lack of high quality data. These findings are consistent with previous literature on the effectiveness of NCAs.^4–7 Our study adds to this literature by providing evidence on the mechanism of NCA use and the relationship between audit output and its use by local teams.

Implications

Interviewees expressed strong support for NCAs, describing them as having authority, credibility and power. This buy-in and value places the audits in a strong position to continue their efforts and to ultimately improve services, as compared to local audit.²² The legitimization of NCAs by ‘a suitable professional body’ is also underlined by the funding body, NHS England.¹⁴ The buy-in of the lead clinical audit contacts, senior consultants and other healthcare professionals through producing authoritative, valuable and unique feedback is an essential prerequisite to closing the audit cycle.

The two main barriers to using audit feedback both related to its content, a feedback characteristic of the FIT model.² Poor data quality may be an important explanation for not using audit feedback to improve services. Some interviewees were very aware of their own poor data quality, while others reported that they did not trust the quality supplied by other NHS trusts. NCAs rely on full participation, consistent data quality across the country with good case ascertainment and data completeness. The importance of trust in the audit was found to be essential in a systematic review.⁸ Additionally, poor information systems and the lack of ability to share data between different systems was frequently mentioned as a problem resulting in duplication of efforts, also found elsewhere by a study focusing on quality in the NHS.²³ While half of survey respondents did not view consultant-level reporting as important, a small number of interviewees felt that it might be helping to improve data quality more generally, likely via mechanisms of public reporting. Providing support for data collection and submission also requires effort from the NCA providers, as suggested by some interviewees and others,³ as well as making audit reports more manager/commissioner friendly to get more buy-in and support organizational decision making.²⁴

An additional reason for not using audit feedback was the inadequate presentation of benchmarked results in the audit output. Local comparative data were perceived as important for identifying areas for improvement and for use as evidence with managers and commissioners. Questionnaire respondents who were more aware of their NHS trust patterns of practice were more likely to report making changes to their practice. It has previously been reported that benchmarking is useful in making local data feedback more meaningful.²² NCA has the added potential for presenting comparative results, as well as benchmarking against a national average. However, based on participants’ reports, it is apparent that this information is not always adequately provided or presented in audit feedback.

Our findings have implications for improving the design of NCA, with a stronger focus on feedback mechanisms to support local use of report findings by clinical leaders.²⁴ The findings also point to refinements of the FIT. While the interviews corroborated the importance of feedback characteristics in line with FIT, they also indicate that the effectiveness of feedback is contingent not only of the individual, but on team receptivity to the feedback, as is evident from numerous references to ‘we’, ‘the service’ and ‘my hospital’ in the interviews. These multiple interactions between individual and team-level goals, beliefs and commitment which may determine whether a feedback report leads to an improvement action should be addressed in further research.

Both the questionnaire and interview responses identified variation between the four audits in the provision of comparative data. However, there may be differences between the clinical specialties represented. Furthermore, the NLCA teams, who reported higher uses of audit data for quality improvement, were supported in 2010 through the Improving Lung Cancer Outcomes Project,^25,26 which led to improved outcomes. Such multifaceted approaches have been found to increase the effectiveness of feedback.⁸ Additionally, the NLCA reported more outcome measures than the other audits, which has previously been found to increase their effectiveness. It is also worth considering that not all NHS trusts and specialties will have the same baseline performance, and the required improvements will therefore differ. While our study focused on the four established NCAs in cancer care, most of our findings will be relevant to other clinical areas.

Strengths and limitations

Previous studies have suggested that NCAs are effective,^4–6 but we were able to suggest the mechanisms of their impact and contextual factors that mediate effectiveness, triangulated by both questionnaire and interviews. Assessing the use of NCAs through self-report is open to subjectivity. However, self-report allows for exploration of the mechanism of change. Interviewees expressed a range of positive and negative perceptions of NCAs. Differences in the response rates between the four audits could also bias estimates of variation in impact between audits. There were three interviewers, each covering separate audits, and it is possible that different responses were elicited, although efforts were made to minimize this by having an interview training workshop, frequent meetings to discuss the interviews and mutual checking of interview audio recordings.

Conclusion

The majority of respondents from four NCAs expressed support for audit and reported that they made some changes in response, made possible by the buy-in and high regard that local audit recipients have for NCA. There are opportunities for increasing the relevance and quality of audit data, which could, in turn, increase its use amongst those currently not engaged. In particular, presentation of comparative local data and adequate resources for local data collection and submission could lead to greater use of NCAs for quality improvement.

Footnotes

Acknowledgements

We thank all of the study participants; Anne Cerchione and Eleanor Bunn for conducting the NLCA and DAHNO and NBOCAP interviews, respectively; Kimberley Greenaway; the clinical leads of the audits and the project teams; and Angela Kuryba.

Conflict of interest

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: All authors are involved in implementing national clinical audit.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The work was partly funded by NHS England through contracts to implement national clinical audits.

References

Healthcare Quality Improvement Partnership (HQIP). Programme list for the national clinical audit and patient outcome programme (NCAPOP), http://www.hqip.org.uk/ncapop-information-contact-and-publications-details-for-each-project/ (accessed 15 September 2015).

Kluger

DeNisi

. The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychol Bull 1996; 119: 254–284.

Healthcare Quality Improvement Partnership. Principles of quality in national clinical audit. London: Healthcare Quality Improvement Partnership, 2011, http://hqip.org.uk/assets/Guidance/HQIP-Principles-of-Quality-in-NCA-FINAL-July-2012new.pdf (accessed 3 October 2014).

Larsson

Lawyer

Garellick

. Use of 13 disease registries in 5 countries demonstrates the potential to use outcome data to improve health care’s value. Health Aff 2012; 31: 220–227.

Torjesen

. National audit shows improvements in hospital care of hip fracture patients. BMJ 2011; 343.

Bridgewater

Grayson

Brooks

. Has the publication of cardiac surgery outcome data been associated with changes in practice in northwest England: an analysis of 25,730 patients undergoing CABG surgery under 30 surgeons over eight years. Heart 2007; 93: 744–748.

Birkhead

Walker

Pearson

. Improving care for patients with acute coronary syndromes: initial results from the National Audit of Myocardial Infarction Project (MINAP). Heart 2004; 90: 1004–1009.

van der Veer

de Keizer

Ravelli

. Improving quality of care. A systematic review on how medical registries provide information feedback to health care providers. Int J Med Inform 2010; 79: 305–323.

Ivankova

Creswell

Stick

. Using mixed-methods sequential explanatory design: from theory to practice. Field Methods 2006; 18: 3–20.

10.

Chadwick G, Groene O, Cromwell D, et al. National oesophago-gastric cancer audit London: The Royal College of Surgeons of England, http://www.hscic.gov.uk/catalogue/PUB11093/clin-audi-supp-prog-oeso-gast-2013-rep.pdf (2013, accessed 3 October 2014).

11.

Scott N, Hill J, Smith J, et al. National bowel cancer audit annual report. Leeds: Health and Social Care Information Centre, http://www.hscic.gov.uk/catalogue/PUB11105/nati-clin-audi-supp-prog-bowe-canc-2013-rep1.pdf (2013, accessed 11 October 2015).

12.

Health and Social Care Information Centre. National head and neck cancer audit. Leeds: Health and Social Care Information Centre, http://www.hscic.gov.uk/catalogue/PUB11015/clin-audi-supp-prog-head-neck-dahn-11-12-rep0.pdf (2012, accessed 3 October 2014).

13.

Health and Social Care Information Centre. National lung cancer audit report. Leeds: Health and Social Care Information Centre, http://www.hscic.gov.uk/catalogue/PUB12719/clin-audi-supp-prog-lung-nlca-2013-rep.pdf (2013, accessed 11 October 2015).

14.

NHS Choices. The NHS in England, http://www.nhs.uk/NHSEngland/thenhs/about/Pages/authoritiesandtrusts.aspx (2014, accessed 3 October 2014).

15.

Ivers

Jamtvedt

Flottorp

. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012; 13: CD000259–CD000259.

16.

Hysong

. Meta-analysis: audit and feedback features impact effectiveness on care quality. Med Care 2009; 47: 356–363.

17.

Retegan

Russell

Harris

. Evaluating the value and impact of the Victorian Audit of Surgical Mortality. ANZ J Surg 2013; 83: 724–728.

18.

John

Paskins

Hassell

. Eight years’ experience of regional audit: an assessment of its value as a clinical governance tool. Clin Med 2010; 10: 20–25.

19.

The Royal College of Physicians. The national audit of the management of familial hypercholesterolaemia: national report. London: The Royal College of Physicians, https://www.rcplondon.ac.uk/sites/default/files/fh-full-audit-report-2011.pdf (2010, accessed 3 October 2014).

20.

Sizmur S, Picker Institute Europe. The reliability of trust-level survey scores: a comparison of three different scoring models. Oxford: Picker Institute Europe, http://www.pickereurope.org/wp-content/uploads/2014/10/The-reliability-of-trust-level-survey-scores-....pdf (2012).

21.

Ritchie

Spencer

Qualitative data analysis for applied policy research. In: Bryman

Burgess

(eds). Analysing qualitative data, London: Routledge, 1993, pp. 305–329.

22.

Bradley

Holmboe

Mattera

. Data feedback efforts in quality improvement: lessons learned from US hospitals. Qual Saf Health Care 2004; 13: 26–31.

23.

Dixon-Woods M, Baker R, Charles K, et al. Culture and behaviour in the English National Health Service: overview of lessons from a large multimethod study. BMJ Quality & Safety 2013; 23: 106--115.

24.

Berwick

James

Coye

. Connections between quality measurement and improvement. Med Care 2003; 41(Suppl 1): I30–I38.

25.

Russell

Jimenez

Martin

. A multicentre randomised controlled trial of reciprocal lung cancer peer review and supported quality improvement: results from the improving lung cancer outcomes project. Br J Cancer 2014; 110: 1936–1942.

26.

The Health Foundation. Final report for award holders – closing the gap through clinical communities. London: The Health Foundation, http://www.health.org.uk/media_manager/public/75/programme_library_docs/CtGtCC%20ILCOP%20final%20report.pdf (2012, accessed 3 October 2014).