Development of a Classification Scheme for Examining Adverse Events Associated with Medical Devices,Specifically the DaVinci Surgical System as Reported in the FDA MAUDE Database

Introduction

Introduced in 2001, robot-assisted surgery has rapidly gained popularity in urology and other surgical fields. It is estimated that between 2009 and 2012, ∼1,057,000 DaVinci procedures were performed in the United States, ¹ an increase from an estimated 15,000 procedures performed in 2004. ² Between the years 2010 and 2012, the number of procedures increased by almost 100,000 cases each year. ¹ The wide adoption of robot-assisted surgery has largely been due to decreased blood loss, smaller incisions, and quicker recovery time. ^{3 –5}

As with any new device, malfunctions and adverse events are closely monitored to ensure safe and effective implementation of this technology. Although individual institutions have published their experience with the DaVinci system, little is known about the national experience. Andonian and colleagues attempted to capture this issue in their analysis of the Manufacturer and User Facility Device Experience Database (MAUDE) managed by the United States Food and Drug Administration (FDA). ⁶ This analysis included data through 2007. Since that time, the applicability of this technology has continued to grow and it has now become a part of routine practice. The MAUDE database is unique as the only publicly available data source regarding adverse events related to medical devices throughout the United States.

In the current study, we reexamined the MAUDE database in an attempt to capture adverse events experienced with the DaVinci Surgical System. We also sought to design a standardized classification system to categorize the complications/machine failures. Currently, there is no method to ascribe causality for device-related events in any surgical field. A standardized classifier, similar to one for pharmaceutical agents and surgical complications (e.g., CTC or Clavien-Dindo), would allow for an easier comparison of events across disciplines and institutions, and it would allow us to standardize reporting. ⁷ This will, in turn, allow for better counseling of patients regarding potential complications before surgery and provide continued education for the medical community regarding potential pitfalls of this technology and how to prevent untoward events.

Methods

The MAUDE database of the United States' FDA was used for this study. Mandatory and voluntary reports are submitted on events associated with medical devices to the FDA. The database is public, and access is unrestricted. As per the Medical Device Reporting regulation, manufacturers are required to report to the FDA when they learn that any of their devices may have caused or contributed to death or serious injury. They are also required to report to the FDA when the device has malfunctioned and would be likely to cause or contribute to death. The same applies to device importers. User facilities are not required to report malfunctions where the malfunction would likely cause or contribute to death or serious injury. Therefore, these reports are completely voluntary.

The MAUDE database was queried by using the terms “robot,” “DaVinci,” and “Intuitive Surgical” for the years 2009–2012. A total of 2930 events were identified. Clearly duplicate entries were removed, leaving 2837 events for analysis. Each entry in the MAUDE database includes the event date, the date the report was received by the FDA, a report number, the event type, and a description of the event that occurred. The database reviewers then went through each event description and for each event identified the category of surgery performed (urology, obstetrics and gynecology [Ob/gyn], other, not specified) and device used (energy instrument, instrument accessory, monopolar scissors tip cover, surgical system, other).

A classification system was developed to associate a level of severity with each event and its relationship to the DaVinci Surgical System. To create this classification system, we built on the existing system to categorize surgical complications developed by Clavien-Dindo and adapted it to a grading system for device-related adverse events. ⁷ Event severity was categorized as mild, moderate, severe, and life threatening/death. Device-related causality was ascribed according to the following categories: not related, possibly, and definitely related (Table 1). This classification system was then tested among 14 surgeons with significant robotic experience from different specialties and institutions by administering a survey of 15 real-case scenarios drawn from the MAUDE database and asking them to classify the scenarios by using our classification system.

Using the externally validated classification system, two reviewers then separately classified the events according to severity and relationship to the device. The two reviewers independently analyzed and categorized the 2837 cases from the database. They were senior residents who were also involved in reviewing and rating the case scenarios.

Results

A total of 2837 events were examined. Between the years 2009 and 2012, ∼1,057,000 DaVinci procedures were performed, yielding a device error rate based on the MAUDE database of 0.27%. Of the events in our study, 34% were Ob/Gyn procedures; 19%, urology; 11%, other; and 36%, not specified. Eleven percent of the events were associated with instrument accessories, 39% with energy instruments, 19% with the DaVinci surgical system, and 31% were other components. Using our complication severity classification system, we identified 75% of the events as Level 1, 18% as Level 2, 4% as Level 3, and 3% as Level 4. Seventy seven percent of the events were classified as definitely related to the device, 15% as possibly related, and 8% as not related (Table 2).

Our classification system showed moderate interrater agreement with a Kappa score of 0.52. This was based on the responses of our 14 reviewers. The interobserver reliability between the 2 reviewers, based on the same-case scenarios provided to all the other 14 reviewers, was calculated to be 0.80.

After performing our analysis, we noted that severity of events differed between the category of surgery (p < 0.0001), device category (p < 0.0001), and relationship to the device (p < 0.0001) (Table 3). Our subset analysis demonstrated that urology procedures compared with Ob/Gyn were associated with more severe events (38% vs 26%, p < 0.0001). Energy instruments were associated with less severe events compared with the surgical system (8% vs 87%, p < 0.0001). Events that were definitely associated with the device tended to be less severe (81% vs 19%, p < 0.0001) (Table 4).

Discussion

In this study, we analyzed the malfunctions associated with the DaVinci Surgical System and instruments that were reported to the FDA MAUDE database from the years 2009 to 2012. Our goal was to develop a classification system by which complications associated with surgical systems can be attributed to device failure.

Previous studies have described earlier entries in the MAUDE database, ^2,6,8 whereas our classification system is the first to include an axis for device malfunction when describing adverse events associated with a complex medical device or surgical system. As complex medical devices become more ubiquitous in the operating room, it is important to characterize complications associated with these devices as well as to use a standardized method to ascribe causality. Although the Clavien-Dindo scale is the most widely accepted classification of surgical complications, it is not adequate to fully describe complications that are associated with complex medical devices.

Our classification system incorporates a simplified severity scale with an axis, which assigns a degree of causality to the medical device. Our system was validated by administering a survey of case scenarios to a multidisciplinary group of robotic surgeons. The kappa for IRR was noted to be 0.52, which indicates moderate inter-rater agreement. ⁹ Our relatively low kappa may be an underestimate due to a paucity of operative details in the vast majority of MAUDE database entries. If more clinical details were available (such as operative reports), this would result in a significantly higher degree of IRR. Despite this limitation, a kappa of 0.52 is significant and demonstrates validity of our classification system, even when reviewing events such as those in the MAUDE database.

In addition to validating our classification system, our data build on previously published studies, which describe earlier data in the MAUDE database pertaining to the DaVinci Surgical system. Lucas and colleagues report a total of 1914 incidents reported over a 6-year period. ² The focus of their study was a comparison between the DaVinci S system (dVS) and DaVinci surgical system (dV), and they found decreased console and side cart errors as well as open conversions with the newer dVS system. Of note, they found no difference in patient injury between the two systems. We found a higher number of reports during our study period (n = 2837). We attribute this to increased use of the DaVinci Surgical system as well as to possible increased awareness about reporting to the MAUDE database during our study period rather than increased rate of complications.

We found the majority of events to be classified as Level 1 according to our scale (75%). The majority of events were found to be definitely associated with the DaVinci Surgical System (77.1%), with the highest percentage associated with energy devices (39.3%). Our univariate analysis demonstrated that a significant difference in the severity of events was present for the category of surgery, type of device, and relationship to the DaVinci Surgical System than would be expected by chance. Given the limited nature of the data in the MAUDE database, we cannot further judge differences in severity of the events but are able to demonstrate that components of the DaVinci system are associated with adverse events.

Our bivariate analysis aimed at better delineating the relationship between severity and the various categories. We found the severity of events to be higher for urology compared with Ob/Gyn (38% vs 26%, p < 0.0001). This finding may be related to the greater variation in the types of procedures performed by urologists. Further study into this difference is warranted. System events were more severe in their impact on patient outcome requiring minor to major intervention due to patient or staff harm, suggesting that additional training in avoiding and trouble-shooting system errors may lead to improved patient outcomes.

In addition to descriptions of the MAUDE database, numerous studies have reported single surgeon experience, which varies significantly. Most importantly, the reported incidence of device failure associated with the DaVinci surgical system varies significantly between individual studies, from 0.4% to 10.9%. ^10,11 This wide range suggests that there is little consistency in what is considered a device failure. In addition, the reported rates from individual institutions are significantly higher than the estimated rate of events reported in the MAUDE database during our study period (0.27%). This suggests that only a fraction of device failures are reported to the MAUDE database or that there is a variation in the understanding of what constitutes a device failure. These findings argue for the need for a stand-alone standardized reporting system that is transparent and not proprietary for medical device failures. This will allow us to track the national experience and to better educate patients and providers in regards to the risks involved in utilizing complex medical devices.

The limitations of our study include those that have previously been described in regards to the MAUDE database. ^12,13 Since reporting of complications is voluntarily initiated by the institute, this could additionally underestimate the number of complications that occur. Further, institutions may be more likely to report complications that are deemed to be device related by their assessment rather than those believed to be surgeon or facility related but occurring with a robotic procedure. In addition, we note only moderate IRR for our classification system; however, as noted earlier, we feel that this is an artifact due to the scenarios available in the database and that the IRR would be higher if more clinical data were available for review.

Conclusions

We describe a novel classification system for surgical complications and device malfunction in regards to the DaVinci surgical system. We also report the complications associated with device malfunction as reported to the FDA MAUDE database from 2009 to 2012. Despite known limitations of the MAUDE database, our classification system is validated and will be a useful adjunct to future studies examining complex medical device failures.