Machine learning for improved medical device management: A focus on defibrillator performance

Abstract

Background

Poorly regulated and insufficiently maintained medical devices (MDs) carry high risk on safety and performance parameters impacting the clinical effectiveness and efficiency of patient diagnosis and treatment. After the MD directive (MDD) had been in force for 25 years, in 2017 the new MD Regulation (MDR) was introduced. One of the more stringent requirement is a need for better control of MD safety and performance post-market surveillance mechanisms.

Objective

To address this, we have developed an automated system for management of MDs, based on their safety and performance measurement parameters, that use machine learning algorithm as a core of its functioning.

Methods

In total, 1997 samples were collected during the inspection process of defibrillator inspections performed by an ISO 17020 accredited laboratory at various healthcare institutions in Bosnia and Herzegovina. This paper presents solution developed for defibrillators, but proposed system is scalable to any other type of MDs, both diagnostic and therapeutic.

Results

Various machine learning algorithms were considered, including Decision Tree (DT), Random Forest (RF), Naïve Bayes (NB) and Logistic Regression (LR). In addition, random forest regressor and XG Boost algorithms were tested for their predictive capabilities in the field of defibrillator output error prediction. These algorithms were selected because of their ability to handle large datasets and their potential for achieving high prediction accuracy. The highest accuracy achieved on this dataset was 94.8% using the Naive Bayes algorithm. The XGBoost Regressor with its r² of 0.99 emerged as a powerful tool, showcasing exceptional predictive accuracy and the ability to capture a substantial portion of the dataset's variability.

Conclusion

The results of this study demonstrate that clinical engineering (CE) and health technology management (HTM) departments in healthcare institutions can benefit from proposed automatization of defibrillator maintenance scheduling in terms of increased safety and treatment of patients, on one side, and cost optimization in MD management departments, on the other side.

Keywords

medical device performance assessment defibrillator post-market surveillance artificial intelligence

1 Introduction

Medical staff are nowadays more confident while performing diagnosis and treatment due to sophistication of Medical Devices (MDs) which allows them a better data analysis and control over diagnosis or treatments. Despite stringent international regulations governing the life-cycle of these devices, instances of malfunctions persist, leading to reported incidents of patient injuries and fatalities.^1–8 Various incidents involving patient injuries and incidents with death outcomes caused by medical devices are reported every year by users, healthcare professional or manufacturers. Some of the world's the most prominent databases containing this data are the FDA Manufacturer's Facility Device Experience (MAUDE) database⁹ and European Database on Medical Devices (EUDAMED) database.¹⁰

The number of these incidents is alarming, and suggests that medical device post-market surveillance, supervision mechanisms and maintenance strategies are not implemented efficiently to ensure patient safety and quality of healthcare.¹⁰

Despite the existence of self-test protocols that are usually built in the MDs software, often medical professionals cannot recognize performance malfunction which directly affects patient diagnosis or treatment.^{1,2,4,5,11–14} Such malfunctions are seen as huge deviations of MDs patient related output parameters. For a defibrillator, this means that critically ill patient will be either treated with higher or lower energy level than needed which results in failed resuscitation or burned patients.^15,16 Events like this are efficiently prevented in those healthcare institutions that implemented adequate supervision mechanisms, such as periodical parameter measurement of safety and performance characteristics.^15,16

This has been addressed with the new MDR 2017/745 where the need for better MD safety and performance post-market surveillance mechanisms is emphasized. Post-market surveillance strategies are implemented and enforced differently within the legal framework of each country in the world,¹¹ and even though, the high rates of MD malfunctions which are resulting in patient injuries suggest that lacking approaches for MD management should be redefined and improved. For instance, in Bosnia and Herzegovina, the post market surveillance mechanism and MD quality assessment are defined through an independent safety and performance inspections for 11 different MD types with measuring functions.^{2,5,14,17–23} This has been regulated following the rules of Legal Metrology. According to this framework, defibrillators are periodically tested by legally appointed, ISO 17020 accredited inspection laboratory.²⁴ All safety and performance measurements and defibrillator information, such as serial number, type, manufacturer and location are stored in developed database.²⁴

Acknowledging the power of data analysis, this study explores the potential of collected data to predict MD performance and anticipate future failures. The research focuses specifically on defibrillators, critical medical devices used in healthcare institutions. The hypothesis driving this investigation is that machine learning algorithms can provide accurate predictions about defibrillator performance and potential failures, thereby optimizing current medical device management strategies.

According to Taghipour²⁵ annual medical device maintenance and management cost in healthcare institution is approximately 1% of the total budget. Healthcare institutions, unfortunately cut these costs, so the usage of MDs as a consequence results in higher rate of incidents with serious injuries or deaths of the patients. Also, they state that numerous optimization models for medical device maintenance have been developed, but healthcare institutions still do not benet from these methods as other industries do. Due to increased complexity of healthcare institution environment and increased technological complexity of MDs, performing maintenance strategies in traditional manner is causing a lot of difficulties. Traditional MD management is based on software programs²⁶ that provide continuous updates, increase inventory accuracy, documents maintenance history and data analysis/reports. By introducing machine learning into healthcare, in terms of MD management strategies, presented challenges can be resolved since raw data can be transformed into useful information and analysis of the big data collections would improve outcomes and reduce the burden on the healthcare system. Machine learning algorithms can learn from experience with respect to some task and performance measurements.²⁷

2 Materials and methods

Data mining in healthcare is an emerging field of high importance for providing prognosis and a deeper understanding of medical data. Data mining applications in healthcare include analysis of health care centers for better health policymaking and prevention of hospital errors, early detection, prevention of diseases and preventable hospital deaths, more value for money and cost savings, and detection of fraudulent insurance claim. Researchers are using data mining techniques in the medical diagnosis of several diseases such as diabetes, stroke, cancer and heart disease.

This study incorporates a diverse dataset detailing defibrillator models, external inspections, energy levels, measured values, and associated error metrics. Data preprocessing ensures uniformity through handling missing values, standardizing energy levels, and categorizing error measurement. Datasets are split into training and validation sets for algorithm exposure and evaluation. Standard classification metrics gauge each algorithm's effectiveness in predicting defibrillator performance and detecting potential inaccuracies. Descriptive statistics and statistical tests inform our understanding of feature distributions and discern significant patterns. This methodology provides the foundation for exploring machine learning algorithms, aiming to identify effective approaches for predicting defibrillator performance and addressing potential inaccuracies.

This study was conducted on the basis of defibrillator inspections performed by an ISO 17020 accredited laboratory at various healthcare institutions in Bosnia and Herzegovina. In total, 1997 samples were collected during the inspection process. Of these, 1311 samples were collected between 2017 and 2020 and were used to train the model for performance prediction. The remaining 686 samples, collected between 2020 and 2022, were used to feed the created predictive model and to validate the method. Among the key features within the dataset are parameters such as ‘Verification is correct’, ‘External inspection’, and specific error measurements like ‘Measurement error’ for our puproses. These features are pivotal in capturing essential nuances related to defibrillator functionality, allowing for a nuanced examination of their performance characteristics. The dataset has undergone a preprocessing phase to handle missing values, ensuring its compatibility with machine learning algorithms for accurate and reliable analysis. This preprocessing step is crucial for maintaining the integrity of the dataset, enabling robust and meaningful insights to be derived. The primary purpose of this curated dataset is to facilitate evidence-based clinical engineering practices. The visualization illustrates the distribution of data between the training and testing sets following a train-test split. The blue bar represents the training set, comprising approximately 79.97% of the entire dataset. Conversely, the orange bar represents the testing set, accounting for 20.03% of the dataset. Notably, approximately 79.97% of the dataset is allocated to the training set, while the remaining portion, constituting 20.03%, is designated for the testing set. This visualization provides a clear depiction of how the dataset is partitioned for training and evaluation purposes, facilitating an understanding of the relative sizes of the training and testing sets Figure 1.

Figure 1.

Train and test percentages.

Various machine learning algorithms were considered, including Decision Tree (DT), Random Forest (RF), Naïve Bayes (NB), Random forest regressor (RFG), XG Boost regressor (XGBoost) and Logistic Regression (LR). These algorithms were selected because of their ability to handle large datasets and their potential for achieving high prediction accuracy.

During training, the algorithms were provided with the input features (i.e., defibrillators data) and the corresponding output labels (i.e., pass or fail status based on the dataset). The algorithms used this data to learn how to accurately predict the classification status of defibrillators based on their performance data. The performance of the trained models was then evaluated using the validation set. Metrics such as accuracy, precision and sensitivity were used to assess the models’ performance. These metrics provided insight into how well the models were able to predict the classification status of ventilators that were not previously processed based on their performance data. Comparing the performance of the machine learning algorithms reveals nuances in their predictive accuracy and effectiveness in addressing defibrillator performance issues. Each algorithm exhibits distinct strengths and considerations, influenced by factors such as model complexity, interpretability, and predictive power.

In evaluating the effectiveness of machine learning models, various metrics are employed to provide a comprehensive assessment. These include:

Accuracy: Measures the proportion of correct predictions among the total predictions made, offering a general sense of the model's overall effectiveness.

Precision: Indicates the proportion of positive identifications that were actually correct, crucial in contexts where false positives have significant consequences.

Sensitivity: Reflects the proportion of actual positives correctly identified, vital in medical scenarios like defibrillator performance prediction where missing a true positive can have serious implications.

3 Results and discussion

The specific parameter settings chosen for each algorithm were based on randomized values that could potentially achieve the desired results. These parameters are crucial in determining the accuracy and performance of the model, and their selection requires careful consideration to obtain the best results. Below are the Confusion Matrix tables (Tables 1, 2, 3 and 4) which depict the results that were achieved when implementing the selected machine learning algorithms. The results achieved across all four implemented algorithms are presented in Table 5, which shows results of the accuracy, precision, and sensitivity.

Table 1.
Performance assessment of decision tree algorithm.

Decision tree algorithm Predicted positive Predicted negative

Actual positive 240 23

Actual negative 15 408

Accuracy 93.3%

Precision 91.1%

Sensitivity 88.7%

Decision tree algorithm	Predicted positive	Predicted negative
Actual positive	240	23
Actual negative	15	408
Accuracy	93.3%
Precision	91.1%
Sensitivity	88.7%

Table 2.

Performance assessment of Random Forest algorithm.

Random forest algorithm	Predicted positive	Predicted negative
Actual positive	231	32
Actual negative	17	406
Accuracy	90.2%
Precision	91.5%
Sensitivity	89.6%

Table 3.

Performance assessment of Naïve Bayes algorithm.

Naïve Bayes	Predicted positive	Predicted negative
Actual positive	225	8
Actual negative	34	389
Accuracy	94.8%
Precision	92.7%
Sensitivity	91.0%

Table 4.

Performance assessment of Linear Regression algorithm.

Linear Regression	Predicted positive	Predicted negative
Actual positive	238	25
Actual negative	29	394
Accuracy	91.6%
Precision	90.4%
Sensitivity	90.3%

Table 5.

Performance comparison across all predictive models.

Algorithm comparison	Decision tree	Random forest	Naïve Bayes	Logistic regression
Accuracy	93.3%	90.2%	94.8%	91.6%
Precision	91.1%	91.5%	92.7%	90.4%
Sensitivity	88.7%	89.6%	91.0%	90.3%

These results suggest that the Naive Bayes algorithm performed the best among the four algorithms in classifying the dataset, achieving the highest accuracy, precision, and sensitivity. Possible explanation for such results may be that the Naive Bayes algorithm is a probabilistic algorithm that assumes independence between the features, making it simple yet effective in classification tasks. The Decision Tree algorithm also performed well, achieving high accuracy, precision, and sensitivity values. The Random Forest algorithm, which uses an ensemble of decision trees, did not perform as well as expected, with a lower accuracy and sensitivity than the other algorithms. The Logistic Regression algorithm also achieved good results, but not as high as those of the Naive Bayes algorithm.

Naive Bayes algorithm was found to be the most effective in classifying the dataset in this study. However, the performance of each algorithm may vary depending on the nature of the dataset and the features used. These findings can be useful in the development of future studies on classification tasks and may aid in the development of more accurate and reliable machine learning algorithms for real-world applications. In terms of accuracy, the naive Bayes algorithm achieved the highest accuracy value of 94.8%, followed closely by the decision tree algorithm with an accuracy of 93.3%. The logistic regression algorithm had an accuracy of 91.6%, while the random forest algorithm had the lowest accuracy of 90.2%. Regarding precision, the Naive Bayes algorithm again achieved the highest precision value of 92.7%. The decision tree and random forest algorithms had very similar precision values of 91.1% and 91.5%, respectively, while the logistic regression algorithm had the lowest precision value of 90.4%. In terms of sensitivity, the Naive Bayes algorithm achieved the highest sensitivity value of 91%, followed by the logistic regression algorithm with a sensitivity of 90.3%. The decision tree algorithm had a sensitivity of 88.7%, and the random forest algorithm had the lowest sensitivity value of 89.6%.

When putting this research into perspective with previous research done on the topic, the developed DT classifier achieves comparable performance with the work of Badnjević et al. from 2019.²⁷ The higher overall performance in comparison to the work presented in²⁷ is due to a significantly larger dataset used that attributes for more variability between the defibrillators. Hence, the work presented here additionally confirms the potential of ML algorithms used for defibrillator performance and confirms the results presented in.²⁷ Hence, a step forward was taken and regressor algorithms were modeled to predict the individual measurement errors of defibrillators. Table 6 shows the performance comparison across regressors.

Table 6.

Performance comparison across regressors.

Algorithm	Random Forest Regressor	XG Boost
MSE	36.54	0.45
R²	0.3	0.99

The Random Forest Regressor is trained on various features relevant to defibrillator functionality. The application of the Random Forest Regressor to the defibrillator performance dataset yielded noteworthy results. The Mean Squared Error (MSE), calculated at 36.54, provides insight into the precision of the algorithm in predicting. errors. Additionally, the R-squared Score (R²) at 0.30 indicates the model's ability to explain variance in the target variable. While the achieved results demonstrate a reasonable level of predictive accuracy, further exploration and fine-tuning of the model are warranted.

The XGBoost Regressor operates by constructing a multitude of decision trees iteratively. Each tree corrects the errors of its predecessor, gradually improving the model's predictive accuracy. The algorithm utilizes gradient boosting techniques, assigning more weight to instances with larger errors to prioritize the rectification of challenging predictions. The XGBoost Regressor, operating through iterative construction of decision trees, showcased exceptional predictive accuracy with an MSE of 0.45 and an impressive R² of 0.99. Focused on key columns related to measured values, errors, and allowed deviations, it proved highly effective in capturing variability within the dataset. The model's performance has profound implications for clinical engineering, enabling early identification of potential issues and contributing to enhanced patient safety.

4 Conclusion

This project serves as an excellent example of how AI, particularly machine learning, can elevate healthcare practices. By using data-driven insights, healthcare professionals have access to an innovative tool that aids in optimizing resources, reducing risks, and improving patient safety. As the integration of AI in medicine continues to evolve, initiatives like this pave the way for inventive approaches to medical device management, opening up a future where predictive analytics substantially contribute to the efficiency and effectiveness of healthcare delivery.

The highest accuracy achieved on this dataset was 94.8% using the Naive Bayes algorithm. This accuracy was used to predict the performance of defibrillators in post-market surveillance. The Random Forest Regressor, with its adaptability and resilience, demonstrated commendable error prediction accuracy. While achieving reasonable results, further refinement is required to unlock its full potential for proactive quality control and maintenance. The XGBoost Regressor emerged as a powerful tool, showcasing exceptional predictive accuracy and the ability to capture a substantial portion of the dataset's variability. Its implications for clinical engineering include early identification of potential issues, contributing to enhanced patient safety and device reliability.

These findings can be significant for future prospective because it offers a solution for performance prediction of defibrillators. By using machine learning algorithms, it is possible to identify potential issues with defibrillators and prevent potential failures that can lead to severe consequences for patients. This study provides evidence that machine learning algorithms can be effective tools for post-market surveillance of medical devices and can be used to improve patient safety.

Footnotes

ORCID iD

Lemana Spahić

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

Badnjević

Cifrek

Magjarević

, et al. Inspection of medical devices. Singapore: Series in biomedical engineering Springer, 2018.

Badnjević

Gurbeta

Bošković

, et al. (eds). Medical devices in legal metrology. In: 2015 4th Mediterranean Conference on Embedded Computing (MECO). IEEE, 2015.

Badnjević

Gurbeta

Bošković

, et al. Measurement in medicine–Past, present, future. Folia Med (Plovdiv) 2015; 50: 43–46.

Badnjević

Pokvić

LG.

Medical devices maintenance. In: Iadanza E (ed.) Clinical engineering handbook. Netherlands: Academic Press, 2020, pp. 520–526.

Badnjević

Pokvić

LG.

Legal metrology framework for medical devices. In: Iadanza E (ed.) Clinical engineering handbook. Netherlands: Academic Press, 2020, pp. 774–779.

Commission

COMMISSION IMPLEMENTING DECISION (EU) 2021/1182. In: Commission E, editor. 2021. p. https://eur-lex.europa.eu/eli/dec\_impl/2021/1182/oj.

ISO. Requirements for medical electrical equipment and medical electrical systems used in the home healthcare environment Geneva, Switzerland. 2015.

ISO. Medical devices-Quality management systems-Requirements for regulatory purposes Standard. 2016.

MAUDE – Manufacturer and User Facility Device Experience [Internet]. 2023 [cited 01.12.2023.]. https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfmaude/search.cfm.

10.

EUDAMED [Internet]. 2023 [cited 01.12.2023.]. https://ec.europa.eu/tools/eudamed.

11.

Badnjević

Pokvić

Deumić

, et al. Post-market surveillance of medical devices: a review. Technol Health Care 2022; 30: 1315–1329.

12.

Pokvić

Deumić

Lutovac

, et al. (eds). Possibility of managing medical device post-market surveillance using artificial intelligence and standardized methodology. In: 2021 10th Mediterranean conference on embedded computing (MECO). IEEE, 2021.

13.

Badnjević

Pokvić

Spahić

Inspection of medical devices. In: Iadanza E (ed.) Clinical engineering handbook. Netherlands: Academic Press, 2020, pp. 491–497.

14.

Badnjević

Vuković

Improving the accuracy and efficiency of medical devices in the legal metrology system of Bosnia and Herzegovina. In: Iadanza E (ed.) Clinical engineering handbook. Netherlands: Academic Press, 2020, pp. 780–786.

15.

Badnjevic

Deumic

Imamovic

, et al. A novel method for conformity assessment testing of defibrillators for post-market surveillance purposes. Technol Health Care 2023; 31: 317–325.

16.

Gurbeta

Dzemic

Bego

, et al. Testing of anesthesia machines and defibrillators in healthcare institutions. J Med Syst 2017; 41: 1–10.

17.

Bošnjaković

Džemić

(eds). Legal metrology: medical devices. In: CMBEBIH 2017: proceedings of the international conference on medical and biological engineering 2017. Springer, 2017.

18.

Deumić

Imamović

Pokvić

, et al. (eds). Performance inspection of patient monitors according to the legal metrology framework: Bosnia and Herzegovina case study. In: International conference on medical and biological engineering. Springer, 2021.

19.

Gurbeta

Džemic

Badnjevic

(eds). Establishing traceability chain of infusion and perfusor pumps using legal metrology procedures in Bosnia and Herzegovina. In: World congress on medical physics and biomedical engineering 2018, Vol. 2, June 3–8, 2018, Prague, Czech Republic. Springer, 2019.

20.

Gurbeta

Sejdinović

Alić

, et al. (eds). Software solution for tracking inspection processes of medical devices from legal metrology system. In: XIV Mediterranean conference on medical and biological engineering and computing 2016: MEDICON 2016, March 31-April 2 2016, Paphos, Cyprus. Springer, 2016.

21.

Gurbeta

Vukovic

Džemic

, et al. (eds). Legal metrology procedures for increasing safety and performance characteristics with cost benefits analysis: case study dialysis machines. In: World congress on medical physics and biomedical engineering 2018, Vol. 2, June 3–8, 2018, Prague, Czech Republic. Springer, 2019.

22.

Vuković

Badnjević

Omanović-Mikličanin

. Cost effectiveness and increasing of accuracy of medical devices in legal metrology system: case study Bosnia and Herzegovina. Inspection of Medical Devices: For Regulatory Purposes 2018; 1: 271–289.

23.

Vuković

Šumar

Dupovac

, et al. Cost effectiveness analysis of medical devices in legal metrology system. In: Badnjevic A, Cifrek M, Magjarević R, et al. (eds). Inspection of medical devices: for regulatory purposes. Cham: Springer Nature Switzerland, 2023, pp.399–415.

24.

Džemić

Bošnjaković

Badnjević

Healthcare quality and ISO/IEC 17020. In: Iadanza E (ed.) Clinical engineering handbook. Netherlands: Academic Press, 2020, pp.731–735.

25.

Taghipour

Banjevic

Jardine

. Reliability analysis of maintenance data for complex medical devices. Qual Reliab Eng Int 2011; 27: 71–84.

26.

Nichols

Linberg

KR.

Apparatus and method to automatic remote software updates of medical device systems. Google Patents, 2002.

27.

Badnjević

Pokvić

Hasičić

, et al. Evidence-based clinical engineering: machine learning algorithms for prediction of defibrillator performance. Biomed Signal Process Control 2019; 54: 101629.

Machine learning for improved medical device management: A focus on defibrillator performance

Abstract

Background

Objective

Methods

Results

Conclusion

Keywords

1 Introduction

2 Materials and methods

Table 1. Performance assessment of decision tree algorithm. Decision tree algorithm Predicted positive Predicted negative Actual positive 240 23 Actual negative 15 408 Accuracy 93.3% Precision 91.1% Sensitivity 88.7%

Footnotes

ORCID iD

Funding

Declaration of conflicting interests

References

Table 1.
Performance assessment of decision tree algorithm.

Decision tree algorithm Predicted positive Predicted negative

Actual positive 240 23

Actual negative 15 408

Accuracy 93.3%

Precision 91.1%

Sensitivity 88.7%