Incident Reporting: A Curate's Egg?

Introduction

Incident reporting could be described as one of the cornerstones of patient safety, but this should not mean it is immune from critical appraisal. The World Alliance for Patient Safety, formed just over three years ago, includes ‘Reporting and Learning’ as one of six principle action areas. ² Two foundation policy papers issued by the American Institute of Medicine ³ and the British Department of Health ⁴ have strongly advocated incident reporting and laid the foundation for the establishment of national reporting systems. In Britain, the National Patient Safety Agency (NPSA) was formed in 2001 and has received public funding, estimated at £35 million during the financial year 2005–06. ⁵ The NPSA was tasked with establishing a national reporting and learning system (NRLS), whereby local incident reports could be fed into a national database to facilitate widespread learning and improvement in patient safety. Prior to implementing a national reporting system, a pilot data audit was commissioned and carried out by the Agency to ‘identify areas [of concern] so that action can be taken rather than just to quantify the extent of the issues precisely’. ⁶ The Australian Incident Monitoring Study or AIMS (funded by the Australian Government) took a somewhat different approach, carrying out a sizeable, linked series of formal studies (see Runciman ⁷ ) to develop a bespoke reporting process in Western Australia. Runciman has published extensively on incident reporting and highlighted a number of principles to be followed in reporting systems which include an anonymous option for reporters and the opportunity for one hospital to compare its profile and reporting rate with another, underpinned by the belief that ‘whoever provides the information should own it’ (Runciman, ⁷ p. 248).

Building a reporting system on an empirical base is commendable, but in a mapping of research literature on patient safety, ⁸ it was found that just one study had described the entire process of development, implementation and evaluation. ⁹ In their discussion on reporting systems, Westwood et al. conclude that an effective reporting system should produce data that can develop strategies for error reduction. The recent Public Accounts Committee Report, Learning to Improve Patient Safety, ⁵ asserted that reporting in the NHS has become very complex, and previously identified and documented barriers to reporting remain. A question should then be posed. In its predominant form at local level, and in relation to the NRLS for England and Wales, does the investment match the benefit?

Nature of Report Data

Although there is optimism about the qualitative information yielded by incident reporting data, ¹⁰ the usefulness of this information is likely to be dependent on a well-designed reporting form, completed by staff who have sufficient training both in identifying incidents and contributory factors, and who share a confidence that action to improve patient safety will ensue from a ‘fair blame’ organization.

However, the free text which provides the qualitative data or context is likely to be fragmented by memory and other cognitive limitations; ¹¹ the ever-present risk of hindsight bias, although downplayed in a meta-analysis, ¹² is a further complication which can also be accompanied by fundamental attribution error ¹³ and confirmation bias. ¹⁴

Incident reports also provide quantitative data, but this has little epidemiological value. The chronic issue of under-reporting results in an underestimated numerator, and an unknown denominator. ¹⁵ If patient safety is to improve as a result of incident reporting, it is imperative that health-care providers guard against false assurances based on volume of incidents reported and focus on incident analysis and contributory factors. A quantitative question is nevertheless worth considering: how many reports are needed to provide a meaningful database – that is, one which can inform action?

National Picture: the National Reporting and Learning System

The NHS's NRLS is essentially an external, industry-level model, which should feed back the results of data analysis to each provider organization. ¹⁶ However, in other industries, these systems – especially outside health care – are mostly confidential and sometimes incentivized, and, furthermore, they emanate from and contribute to a cultural make up significantly different from that of the NHS.

There has been considerable optimism as to the potential of the NRLS to be a contributor to improved patient safety. Shaw et al. ¹⁷ predicted that with strong information technology and cultural change, progress will be made. There have, of course, been hurdles along the way, for example, developing and migrating each Trust's local taxonomy for categorization of incidents, together with the technical challenge posed by linking up to each Trust's local system.

The NRLS currently receives around 3000 incident and near-miss reports each day, but when the reports are categorized for seriousness, the vast majority are in the minor, or less serious, bracket and as such prompt little local action. Such incidents, particularly near misses, may provide very useful learning, ¹¹ but not necessarily in the eye of the reporter, ¹⁸ or the local risk manager. It has also been demonstrated that serious and catastrophic incidents tend not to be reported.^5,19

The effectiveness of such a system must, in our view, be judged in terms of its clinical outcomes rather than process-centred measures such as number of reports received. In terms of volume, by January 2006, the NRLS held around 1 million incidents in its data bank, with 900,000 being added in the first nine months of 2006. When considering outcomes from this vast potential knowledge bank, the evidence could be better. Since its establishment in 2001, the NPSA have issued only around a dozen patient safety alerts and notices to the NHS, but appear not to have reported any data to demonstrate evidence of follow up, or audit of implemented guidance or provided evidence that the reported rate of related incidents has fallen since the issue of the notice. Indeed this situation was put forward as one of the reasons why Trusts are questioning the value of the NRLS. ⁵

The Local Picture

A number of generic barriers to incident reporting exist at a local level, the National Audit Office ²⁰ have identified a top six:

Findings from the Institute for Safe Medication Practices (ISMP) medication safety self-assessment for 1435 North American hospitals demonstrated that just 23% of staff felt they could report openly without reprisal. ²¹ Identifying vulnerabilities in the system is a principle reason for reporting, ²² but poor report design militates against this. However, it is notable that among those initiatives built to advance the effectiveness of reporting, ²² user understanding may not be commonly and explicitly sought. Tangential to this lack of understanding is that staff need to know what an incident (or error) is, so as to recognize its reporting value.^23,24 Taxonomies may help or alternatively conspire to create a muddled picture. ²⁵

There is a vast variation in the volume of incidents reported across the NHS (from 50 to 5000 + per Trust per annum), yet it has been recognized that most Trusts need to improve their local feedback processes. In many units, it is suggested that staff have barely enough time and energy to cope with daily problems, ²⁶ let alone those processes judged as somewhat distal to the momentum of their specific unit or ward. In simple terms, staff can feel that sending a report off to management will not result in any action. Therefore, they may not bother, except in situations that could directly affect them (slips, trips and falls of patients are the most widely reported incidents).

Action following reporting varies between organizations. Many rely on data entry clerks who have minimal training, sometimes inputting to voluminous databases which may limit the effectiveness of analysis but not the related cost. ²⁷ A system that holds hundreds of (redundant) reports focused on either trivia or important events but without detail will obscure effective assessment and, consequently, the issuing of critical warnings. ²⁸ Unless valid sense-making is carried out by those with the time and ability to make sense of the data, the process is relatively weak in resolving threats to patient safety. ²⁹

Incident Reports and Record Review: Comparative Safety-Information Capacity

Two recent studies conducted in acute hospital settings have identified the problem of under-reporting of incidents, when compared with incidents that are identified by direct analysis of the patient's record. Olsen et al. ³⁰ undertook a record review of a total of 280 consecutive medical and surgical patients in an acute hospital for evidence of adverse events and near misses. The research team also reviewed the incident database for reports on the same sample of patients. A total of 26 adverse events and 40 critical incidents (near misses) were identified from the records, i.e. an incident/near miss rate of 23%; yet for the same group of patients, just four clinical incidents and no near misses were reported. Hogan et al. ³¹ reviewed a sample of 220 randomly drawn records for medical and surgical patients treated in a large hospital within the year 2004–05, and compared the number of incidents/near misses identified from this source with other sources of data held by the study hospital. This included International Coding for Diseases (ICD) coding for complications and misadventure, complaints, claims incidents and inquest databases. Hogan's team identified an incident/near miss rate of 23% from record review, but a rate of reported incidents (from the clinical and health and safety database) of just 2.5%. Furthermore, Sari et al. ³² conducted a retrospective patient record review of 1006 random admissions in another large NHS hospital and found that of the 324 patient safety incidents identified, 270 (83%) were picked up by record review, but just 21 (7%) by routine reporting.

While these studies support the view that the richest data is to be derived from direct case note review, Wald and Shojania ¹⁵ caution that record analysis is not ideal for identifying near misses and can be costly. Furthermore, review of records is unlikely to elicit contributory factors, ³³ nevertheless these studies may highlight the current weaknesses in incident reporting.

Reluctant Reporters: the Enduring Case of Doctors

Kingston et al. ³⁴ have aptly described nurses as reporting more ‘habitually’ than doctors; that doctors are less likely to report incidents than their nursing or midwifery colleagues is not an isolated finding.^24,35–37 Figures of reports received from the first year of the NRLS did not suggest the trend had changed, just 8% of reporters identified as doctors. ³⁸

Uribe et al. ³⁹ have examined whether the barriers to reporting are different for doctors compared to nurses. Although both groups shared a common dislike for the time and extra effort required to report, their survey showed that doctors questioned the usefulness of reporting and its contribution to quality improvement, which was not echoed by the nurses who were far more concerned about ‘telling on someone else’ and legal action. Further analysis suggests doctors often see patient incidents as ‘complications’ of treatment and do not consider them reportable. This reluctance may be exacerbated by the subject matter of reports being perceived as trivia. ¹⁸ After an adverse event, established loyalties can lead to a confidential ‘chat’, where some individual learning may take place, but organizational learning is compromised. This is a culture whose members are unlikely to glorify in secrecy, but alternatively may not feel confident enough to share errors across the greater corpus without threatening relationships. A clear example of doctor's reluctance to report incidents is in the case of needle-stick injuries. Surgeons are most likely to suffer a needle-stick or sharp instrument injury, but are the least likely to report them among health professionals. ⁴⁰ It seems doctors may have a tendency to eschew organization-wide procedural matters, ⁴¹ but as effective incident reporting is best played as a team game, the lengthy process of cultural change visualized by Giles ⁴² to achieve effective levels of multidisciplinary reporting is probably inevitable. Moreover, unless the information-generating capacity of reporting improves, convincing medical staff to report may prove futile.

Reporting: Concluding Comments

The health-care community faces some choices concerning reporting, mindful of its limited popularity. The first choice is that practitioners will [hopefully] spot incidents and, having hurdled over the common barriers, actually report them. The alternative is to ignore them and only respond when the press or the legal system seeks to investigate. A third way would be to seek to focus on reporting and analysing serious incidents that could prompt effective system change and patient safety improvement. This could be coupled with a risk assessment of patient care pathways to reduce the chance of incidents happening in the first place. In such a comprehensive risk managed model of health-care, incident reporting becomes an integral part but not the lynchpin. This third option recognizes both the strengths and the limitations of reporting, which is crucial to improve the performance of this method of collecting incident data and, consequently, promoting effective risk management.

We now return to the questions posed earlier and propose a response. Organizations at both local and national levels may have plenty of data, but how much information? An answer might lie in studying the reporting process, but not by simply counting the number of reports or other quantitative data. A careful examination of the resultant action taken following incident analysis would, in our view, be more appropriate. This would have the potential of being directly beneficial to patients and staff, by providing information on both risk problems and risk solutions; it may also show to reluctant reporters, such as doctors, that a focus on clinical outcomes is possible. This could also help emolliate those critics who understandably ask the equally important question about the cost-effectiveness of reporting and how many reports are needed to build a confident picture of the scale and nature of incidents in the NHS.

Looking at how and what action is taken in response to the analysis of the reasons for incidents is not simply a matter for local organizations. Considering the recent pressure exerted on the NPSA by the House of Commons PAC, ⁵ there might also be some credit gained from this change of focus at a national level. Furthermore, knowledge of this type might at least help to build the bridge that could ultimately close the theoretical gap between the impact of reporting and clinical outcomes, as well as giving risk managers some insight into the best way to prioritize in an ever-increasing database.

Reporting alone does not change practice or improve quality. Only by investigation and analysis, using a range of methods, can we understand why something happened and take steps to prevent a repeat. Like the curate's egg, we can currently only judge incident reporting to be ‘good in parts’.