Medical device error and failure reporting: Learning from the car industry

Abstract

Background

Improving the design of technology relies in part, on the reporting of performance failures in existing devices. Healthcare has low levels of formal reporting of performance and failure of medical equipment. This paper examines methods of reporting in the car industry and healthcare and aims to understand differences and identify opportunities for improvement within healthcare.

Methods

A literature search was carried out in Pubmed, Medline, Embase, Engineering Village, Scopus. NHS England and MHRA publications and guidelines were also reviewed. Focus was placed on the current system of reporting in both industries, known degree of patient harm, initiating factors, barriers, quality and methods of incident investigation and their validity. The findings were used to compare error reporting system in the two industries.

Results

Derivation of healthcare incident data from different sources means the full extent of patient harm is not known. For example, in 2012 there were 13,549 and 38,395 incidents reported by MHRA and NRLS (National Reporting and Learning System) respectively leading to uncertainties on the extent of the problem. The car industry emphasises the role of reporting source in ensuring data quality. Utilising some aspects of this approach might benefit healthcare reporting. These include a specific reporting system that stresses the importance of organisational learning in improving safety and recognises the limitations of root cause analysis.

Conclusions

Learning from reporting systems within the car industry may help the healthcare sector improve its own reporting, aiding healthcare performance.

Keywords

Adverse event reporting medical device performance reporting patient safety medical device safety

Background

The scope of the problem

The Medicines and Healthcare products Regulatory Agency (MHRA) defines a reportable adverse incident as ‘any malfunction or deterioration in the characteristics and/or performance of a device, as well as any inadequacy in the labelling or instructions for use which, directly or indirectly, might lead to or might have led to the death of a patient, or user or of other persons or to a serious deterioration in their state of health.¹ NHS Improvement and Department of Health (DOH) publications have reported significant problems relating to error recognition and reporting in relation to medical devices (MD).¹

Majority of the reporting to MHRA occurs through manufacturers (53%) with NHS services officially reporting 31% of cases. The publicly available MHRA report in 2012, states that of 13,549 reports received, 2.2% resulted in death and 33.2% resulted in serious injury.¹

Data from different sources (NRLS and MHRA) varies considerably leading to lack of accurate data on device malfunction. For example, in 2012, NHS England reported (through NRLS) 38,395 incidents whilst MHRA received 13,549 for the same year. Incident causality is related to the healthcare organisation/user (25%), manufacturer (28%) and no established cause (47%).¹

The issues surrounding reporting are complex and include the equipment used and maintenance contracts in the NHS. A hospital audit carried out by Association of Laparoscopic Surgeons of Great Britain and Ireland in 2011 found that only 4 out of 10 hospitals used gold standard equipment and 1 in 4 did not have a maintenance contract to replace broken equipment.²

The current data on MD performance from healthcare relies on incident reporting which in itself is a poor source of information to manage the public health risk in relation to MDs.³ This system requires rethinking in order to benefit clinicians, healthcare trusts, manufacturers and ultimately patients.

Experience in other sectors

The issue of safety and error underreporting is not unique to healthcare. We considered a number of sectors and sought an area where both public and professionals used technology with a potential for safety critical failures. We selected the car industry as it offered parallels with healthcare and has fewer restrictions than other sectors (e.g. aviation) and thus a broader base for knowledge transfer. Like healthcare, the automotive industry has experienced much recent change with advancements in technology, changing market trends, legislation and ongoing requirements for high quality products.⁴

Whilst the car industry differs from healthcare in the type of product it provides, it still has to contend with the challenges of rapid technological advances and their application across a wide user base. It provides insights of the human factors, individual and organisational behaviour that may affect reporting.

It was considered that research in the car industry might provide a better understanding of the behaviour of individuals, manufacturers and government agencies based on the type of reporting involved.

MDs are often more complicated than other technologies in terms of their operating environment, level of integration with other systems components, training required in their use and potential cognitive and physical demands on the users.⁵ By utilising the knowledge gained by the car industry, we aim to better understand the causes of current underreporting issues in healthcare as well as factors that affect the initiation and quality of reporting.

This research paper aims to: i) describe the current reporting system for MDs in the UK healthcare system ii) present the barriers to reporting in healthcare, iii) identify factors that affect the initiation and quality of reporting in both industries and iv) and identify how and where successful lessons from the car industry might be applied to healthcare.

The terms errors and malfunctions are used interchangeably in this paper to indicate that malfunctions are themselves system errors with a potential to lead to harm.

Methods

We conducted a literature review in Embase, Medline and Pubmed for “incident reporting”,” error reporting”, “reporting errors” “medical device errors”, “reporting errors of medical devices” and “patient harm AND medical devices”. Additional information was accessed from NHS England, MHRA reports (2014) and DOH websites.

Literature from the car industry was searched in Engineering Village and Scopus using keywords “car withdrawal”, “automobile withdrawal” and “error reporting AND car industry”. Papers describing methods of reporting and their outcomes, organisational learning, individual and organisational behaviour to reporting errors and malfunctions were included. The SQUIRE (Standards for Quality Improvement Reporting Excellence) guidelines were used to present the results of this study as a recognised framework.

Focus was then placed on six main topics in both industries including 1) the current systems of reporting, 2) known degree of patient harm related to MDs, 3) barriers to reporting, 4) factors affecting the initiation of reporting 5) quality and effect of sources of reporting and 6) the methods of incident investigation and their effect on reporting.

Current systems of reporting in healthcare and the car industry

Current system of reporting in the car industry

Vehicle fault reporting occurs through government regulatory agencies and directly from manufacturers. Most recalls are voluntary by manufacturers⁶ similar to healthcare.

The concept of ‘acceptable level of error’ exists across all industries. In a study within the Swedish car industry,⁴ the concept of a ‘fault tolerant system’ is considered. Its aim is to handle the error without affecting the delivered service. Whilst multiple approaches have been researched on this, there is no clear evidence of how many have been tested or implemented in practice due to lack of tools, processes and often a disconnect between academia and industry.⁴

Maintenance of a “digital logbook” in the manufacturing phase is also suggested which would reduce the risks of human misinterpretation, misjudgement and unwanted decisions. This aims to help with error diagnosis, facilitation of training and use in system restart situations. However, such systems rarely enable ready diagnosis of use-errors arising from the human operator.⁴ Presently, a digital logbook system is not practiced in healthcare and may, in any event, only be applicable to some devices with associated issues of patient confidentiality.

Current system of medical device reporting in UK

Currently, incidents relating to MDs in UK are reported through the MHRA and the NRLS (National Reporting and Learning System) operated by NHS England (soon to be replaced by Patient Safety Incident Management System).¹ The timeline for submissions to MHRA is 2 days for public health threats, 10 days for death and serious injury and 15 days for potential injury.⁷

Adverse event reports submitted to MHRA are reviewed by a MD specialist and clinical advisers to allow for urgent assessment of cases which pose the greatest risk to patients. All reports are recorded, risk assessed and reviewed. Depending on the type of event, the MHRA may decide to i) intervene directly, ii) request the manufacturer to investigate or iii) log the incident in their database for future assessment.⁷ The MHRA recognises that there is a need to simplify reporting, improve learning and improve the quality and volume of reports. (Figure 1)¹

Approximately 1% of all patient safety incidents are reported via NRLS.⁸ Approximately 3% of all reported errors relate to MDs.⁵ It is recognised that patient safety incident data are prone to reporting error and bias affecting its analysis and interpretation. There is a recognised bias in the types of incidents reported, the extent of which is difficult to estimate.⁸

Conventional post market usability studies tend to focus on factors of speed and user satisfaction rather than safety and error rates. Additionally, studying error rates in laboratory based studies is not effective as these rates are too low compared to actual use.⁹

A study of adverse events in surgical specialities found that they affect up to 16% of patients admitted to hospital, approximately half of which are linked to failure of equipment during surgery.⁹ Another study of 66 complex vascular and endovascular procedures showed that adverse events were related to medical technology in up to 24% of cases.¹⁰

Barriers to effective error reporting in healthcare

Barriers in reporting (Table 1), recognition and resolution of MD errors influence the effectiveness of the system. These factors include conflict of interest, education and training, reporting compliance, impact on patient care, feedback on how information is used, device performance, response from manufacturers and complicated and time consuming system of reporting.¹¹

Table 1.

Main reported barriers to effective reporting of MD malfunction.

Barriers to effective reporting
1. Time consuming process
2. Complicated system of reporting
3. Lack of Education and Training
4. Lack of Feedback
5. Lack of information on how reporting information is used
6. Not knowing what to report
7. Not knowing who to report to
8. MDs failure considered differently to other events in healthcare
9. Manufacturer response (lack of)
10. Poor recognition of association between patient outcome and MD malfunction
11. Limited knowledge on new devices & their complications
12. Uncertainty on reporting responsibility
13. Negative opinion on institution’s ability to create true change
14. Fear of blame
15. Conflict of interest

Limited knowledge on new devices and their complications leads to difficulty in recognising the association between adverse outcomes and MD use.¹² Other authors¹³ support these findings and add that uncertainties regarding reporting responsibilities and negative opinions on the institution’s ability to create a positive change are important barriers. In particular, lack of feedback was considered a highly significant barrier (85%). In this study, 72% of physicians and 60% of nurses believed the reporting system should be made specific to the problem being reported.¹³

Whilst specificity is not a hindrance in the car industry, external recalls, initiated by end users are linked to poorer and reduced likelihood of reporting. They, together with design related problems, are associated with longer recall time.¹⁴ As in healthcare, it is suggested that often error causation is linked to factors such as design defects, training, maintenance or management rather than simply the end-user.¹⁴

A systematic review of studies between 2004 and 2013 regarding error reporting in healthcare concluded that errors would be reported more frequently if the reporting system was “easier”, if the users knew what to report and ‘how’ and ‘if’ timely feedback was received on the actions taken.¹²

In one study,¹² whilst 70% of clinicians believed that a cause for the incident would be found, only 40% believed this would bring a positive change. Whilst their results were not specific to MDs, similar events are likely to be observed.¹³ The authors believed that whilst errors in healthcare are a result of a series of events, the reporting system focuses on the type of error which in itself focuses on who is at fault. This does not allow for appropriate use of the expertise of the clinician reporting the event.¹³

Fear of blame is repeated in a number of studies.^13,15 In one study, 40% of physicians and 30% of nurses were concerned the report would be used against them rather than to address potential causes of events (US data).¹⁵

Figure 2 shows the current pathway of reporting for MDs whilst Figure 3 shows reporting pathway to manufacturers.

Factors affecting initiation of reporting in both industries

Rupp (2012) analysed the factors and bodies (regulatory, manufacturer and user) that initiate recall in the car industry (19 years data). The recall initiator had no significant effect on the response rate of the car owner. Additionally, accident and injury provided little indication of the repair rates carried out.¹⁵

The government was more likely to initiate a recall in larger and less hazardous cases involving older models, financially weak firms and reported injuries. Manufacturers were more likely to initiate a recall when the benefits (reduced expected liability) exceeded the expected cost of repair. The car owner on the other hand was more likely to respond to a widely publicised campaign.¹⁵

The initiation of reporting in healthcare is affected by the barriers presented above. Additionally, senior management may also lack awareness of the importance of reporting and learning systems.¹ The process of reporting involves first recognising the error followed by reporting. Every step is affected by a number of factors. The recognition step is influenced by education, training, knowledge, experience, performance of device and warning signals.¹¹ The reporting step is affected by the reporting system, processes and feedback on how information is used.^11,16,17

Quality and effect of sources of reporting in both industries

Factors that affect the quality of reporting have been researched by the car industry. One such factor is the source of reporting (internal vs external). External reporting is associated with difficulties in interpreting information and lack of symmetry in the information presented. These result in a lack of sufficient information presented to the manufacturer about actual field conditions.¹⁴ It is thought that these representation gaps in information are due to different cognitive structures between external parties and the specific automotive company.¹⁸

The media reporting of MDs has particular importance in patient and public reporting. In cases where incidents with devices occur, the framing of the investigation and their reporting often lacks appropriate emphasis on the use, design and potential for improvements. A better understanding of the culture of errors by MD designers, may allow for design changes thus promoting design innovations.¹⁹ Researchers from the car industry believe that design defects are complex and often ill-structured compared to manufacturing defects.¹⁹ Such defects are more difficult to identify and address and may not present at initial stages needing investigation of both design and manufacturing processes.¹⁴

There is great variability in the process of reporting.^11,16,17 Reports on device malfunction/failure are often sent directly to the manufacturer. Following this, the healthcare trust (in UK) is either reimbursed financially or the product is replaced. Variability is present not only between different hospitals but also within the same trust on the process of reporting and record keeping.¹¹

In 2012, NHS England analysed NRLS data on MD incidents and found major issues with the quality of reporting. They reported delays in report submission (49.7%), device name not recorded (65%), manufacturer not recorded (82.3%), no record on actions taken to prevent recurrence (47.7%), no apparent causes recorded (68.1%) and clinical outcome codes indicating death or severe harm miscoded (40%).¹ Incident reports are not always reviewed locally by NHS staff with MD safety expertise. Thus, opportunities to initiate an action plan at early stages are lost.¹

These factors influence the quality of reporting and organisational learning from incidents. Figure 1 shows the current reporting route (NHS England).¹ Other authors²⁰ address the importance of assessing the records of devices reported but sent back with a ‘No Fault Found” (NFF) summary which may uncover design flaws. In these cases, there is often a mismatch between the user needs, cognition, behaviour and environmental faults and engineering perspective.²⁰

Figure 1.

Current reporting route and feedback systems (adopted from NHS England/MHRA 2014).²

Figure 2.

Pathway to reporting.

Figure 3.

Pathway of errors/malfunctions reported directly to manufacturer.

Incident investigation methods (both industries) and their consequences on incident reporting

In healthcare, root cause analysis (RCA) is the main method used to investigate incidents, (including MDs) with a causal chain of events often presented. This in contrary to advice NHS Improvement in using a more systems-based approach to incident investigation.¹ Literature from the car industry questions the validity of a single chain of events model.²¹

Sheridan (2008) raised the point of effectiveness of RCA in studying the cause of failure in the car industry. In their opinion error causation is much more complex. It is important to consider both the conditions preceding the event (necessity) and the sufficient conditions for the target event to occur (sufficiency). Based on this logic, a fault tree model is presented as the most responsible technique in modelling incident causation.²²

In the car industry, a number of questions have been raised including whether errors are caused, and their relationship to an accident. They raise the notion of whether elimination of all errors is desirable given that we learn from them. Additionally, they raise the question of whether we expect automation to compensate for or exacerbate human errors.²² Human error in healthcare is complex due to the multiple contributing factors leading to the actual event and it can be argued whether automation would exacerbate the issue. Human error can be minimised but not eliminated. Table 2 shows suggested approaches to the issues faced by healthcare on error reporting.

Table 2.

Summary table of main issues on healthcare reporting and suggestions from the car industry.

Recognised issues in healthcare	Car industry approaches applicable to healthcare
Poor quality reporting and methods
1. Majority of reporting occurring unofficially 2. Lack of knowledge on what to report and how 3. Multiple sources of poor quality data which are not statistically comparable 4. Reporting methods not specific and concentrating on ‘incident reporting’ only 5. Recognised bias in the type of information reported	“Specific” and targeted reporting system for the industryRecognised differences between different sources of reporting on interpreting informationMaintenance of a digital logbook
Poor recognition
1. Poor recognition of medical device errors 2. Reporting too complex	1. Simulation training,2. Simplify existing methods 3. Increase specificity of reporting system for Healthcare-MedTech industries to reduce recognition issues
Inadequate testing methods
1. Lack of emphasis on human-machine interface/interaction 2. Root cause analysis (RCA) used as the main method of event analysis	1. Different approach to RCA advised, 2. Human factors and usability studies embedded in analysis, 3. Need for long term data
Organisational
1. Senior management lacks awareness on issues related to reporting 2. Lack of organisational learning	Education and training

Conclusion

The current level of MD reporting is known to be poor with the existing system lacking the ability to capture the required information that would be useful to clinicians, manufacturers and regulatory bodies.

This research has presented a number of factors contributing to poor reporting in healthcare as well as suggestions for system improvement drawn from the car industry. Amongst these, an increase in specificity of reporting method, need for long term data and recognition of behavioural differences between different sources could lead to better reporting methods and potentially reduce existing levels of underreporting. We also consider that bringing device performance reporting outside the context of incidents only would lead to improved knowledge and learning for all stakeholders.

Thus, a different and targeted approach to malfunction reporting of MDs is required with an emphasis on learning, effective communication and device improvement to ensure patient safety and healthcare performance.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Arkeliana Tase

Peter Buckle

References

MHRA. Stage three: directive. Improving medical device incident reporting and learning. Patient safety alert, 20 March 2014. https://www.england.nhs.uk/wp-content/uploads/2014/04/psa-med-dev-0414.pdf

Panesar

Salvilla

Patel

, et al. Laparoscopic cholecystectomy: device-related errors revealed through a national database. Expert Rev Med Devices 2011; 8: 555–560.

Wilkinson

How master data and global collaboration can lead to better outcomes at lower cost. UK: MedTech Innovation, RSS Print, 2020.

Farooqui A, Falkman P, and Fabian M. Error handling within highly automated automotive industry: current practice and research needs. In: 21st IEEE international conference in emerging technologies and factory automation, 6 September 2016. Berlin, Germany.

Improvement

NHS.

NaPSIR October to December 2018 – England, 2019.

Haunschild

Rhee

The role of volition in organisational learning: the case of automotive product recalls. Manage Sci 2004; 50: 1545–1560.

Blake

In: MHRA meeting, Personal Communication (ed A Tase),11 July 2019.

NHS Improvement. NRLS official statistics publications: data quality statement. March 2018. Availabe at: https://improvement.nhs.uk/documents/2552/NRLS_Data_quality_statement_March_2018.pdf.

Weerakkody

Cheshire

Riga

, et al. Surgical technology and operating room safety failures: a systematic review of quantitative studies. BMJ Qual Saf 2013; 22: 710–719.

10.

Albayati

Gohel

Patel

, et al. Identification of patient safety improvement targets in successful vascular and endovascular procedures: analysis of 251 hours of complex arterial surgery. Eur J Vasc Endovasc Surg 2011; 41: 795–802.

11.

Polisena

Gagliardi

Clifford

How can we improve the recognition, reporting and resolution of medical device-related incidents in hospitals? A qualitative study of physicians and registered nurses. BMC Health Serv Res 2015; 15: 220.

12.

Craig

Omp

The need for greater reporting of medical device incidents. EMJ Innov 2019; 54: 71–95.

13.

Harper ML, Helmreich RL, Henriksen K, et al. Identifying barriers to the success of a reporting system. USA: AfHRaQ, 2005.

14.

Ni J, Huang X. Discovery to recall in the automotive industry: a problem solving problem perspective on investigation of quality failures. J Supply Chain Manage 2017; 54:71–95.

15.

Rupp NG and Taylor CR. Who initiates recalls and who cares? Evidence from the automobile industry. J Ind Econ 2002; 50; 2:123–149.

16.

Karen Yunqiu

Sxd

Liping

Paolo

, et al. MediCHI: safer interaction in medical devices. In: CHI EA 2013 extended abstracts on human factors in computing systems: Changing perspectives 2013, pp.3267–3270.

17.

Polisena

Castaldo

Ciani

, et al. Health technology assessment methods guidelines for medical devices: how can we address the gaps? The international federation of medical and biological engineering perspective. Int J Technol Assess Health Care 2018; 34: 276–289.

18.

Cronin

Weingart

LR.

Representational gaps, information processing, and conflict in functionally diverse teams. Acad Manage Rev 2007; 32: 761–773.

19.

Myketiak

Paul

SC.

Transforming healthcare through innovations in mobile and wireless technologies. New/s design: informing future design processes by understanding media reporting of medical errors with medical devices. In: 5th EAI international conference on wireless mobile communication and healthcare, MOBIHEALTH 2015 ICST, October 14–16, 2015. London, GB

20.

Flewwelling

Easty

Vicente

, et al. The use of fault reporting of medical equipment to identify latent design flaws. J Biomed Inform 2014; 51: 80–85.

21.

Reason

J, HollnagelE,

and Paries

Revisiting the “Swiss cheese”. Model of accidents. EUROCONTROL, Eurocontrol Experimental Centre, 2006.

22.

Sheridan TB. Risk, human error and systems resilience: fundamental ideas. Hum Factors 2008; 50: 418–426.