Machine learning algorithms to predict seizure due to acute tramadol poisoning

Abstract

Introduction:

This study was designed to develop and evaluate machine learning algorithms for predicting seizure due to acute tramadol poisoning, identifying high-risk patients and facilitating appropriate clinical decision-making.

Methods:

Several characteristics of acute tramadol poisoning cases were collected in the Emergency Department (ED) (2013–2019). After selecting important variables in random forest method, prediction models were developed using the Support Vector Machine (SVM), Naïve Bayes (NB), Artificial Neural Network (ANN) and K-Nearest Neighbor (K-NN) algorithms. Area Under the Curve (AUC) and other diagnostic criteria were used to assess performance of models.

Results:

In 909 patients, 544 (59.8%) experienced seizures. The important predictors of seizure were sex, pulse rate, arterial blood oxygen pressure, blood bicarbonate level and pH. SVM (AUC = 0.68), NB (AUC = 0.71) and ANN (AUC = 0.70) models outperformed k-NN model (AUC = 0.58). NB model had a higher sensitivity and negative predictive value and k-NN model had higher specificity and positive predictive values than other models.

Conclusion:

A perfect prediction model may help improve clinicians’ decision-making and clinical care at EDs in hospitals and medical settings. SVM, ANN and NB models had no significant differences in the performance and accuracy; however, validated logistic regression (LR) was the superior model for predicting seizure due to acute tramadol poisoning.

Keywords

Machine learning prediction seizure tramadol

Introduction

The mortality and morbidity associated with the complications and side-effects of drug poisoning, especially with narcotics and analgesic drugs such as tramadol, are still a major challenge across the globe.^1,2

Tramadol is a potent analgesic drug with an easy availability that is excessively prescribed and has a euphoric effect. There is an increased prevalence of tramadol poisoning in the Middle East and Iran.^3
–5 Seizure, which is defined as a sudden, uncontrolled electrical disturbance in the brain that causes changes in behavior, sensory perception or motor activity, is an important and common neurological effect of tramadol poisoning reported in 15–30% of the cases as generalized tonic-clonic seizure, occurring as a result of the inhibition of the Gamma Aminobutyric Acid (GABA) receptors and serotonin toxicity.^6,7 Several studies have shown that approximately 2–5% of the general population will experience at least one non-febrile seizure in their lifetime that may result from medication or drug use and that causes death in nearly 2% of the cases.^8,9 Previous investigations have demonstrated that delays in the diagnosis and prognosis of seizure causes greater morbidity and mortality in poisoned patients. Therefore, predicting seizure in acute tramadol poisoning may help clinicians identify high-risk patients and take appropriate and timely action to reduce the complications.^10

–14

Machine learning (ML) is an alternative analytic method for managing complex interactions in big data, discovering nonlinear interactions between a number of predictors, and generating actionable predictions in clinical settings. ML algorithms focus on prediction models that learn from available data directly and automatically and are highly flexible methods requiring penalization to prevent overfitting.^15,16 Recently, many studies have reported the application of Machine Learning algorithms in activities from fraud detection to cancer genomics.^17

–20 Nonetheless, these studies have been confined to small sample sizes and narrow clinical and socio-demographic risk factors. Although one study showed a prediction model for tramadol poisoning using Logistic Regression (LR), no studies have yet examined the use of ML models for predicting this poisoning in large populations of adult patients in Emergency Departments (EDs).²¹ The common ML models that can forecast clinical outcomes include k-Nearest Neighbor (k-NN), Naïve Bayes (NB), Support Vector Machine (SVM) and Artificial Neural Network (ANN).

The K-NN algorithm is easy and simple method which has few hyper-parameters to tune. However, k should be wisely selected and computation analysis could be large when sample size is large. The Naïve Bayes method is much faster than KNN. Unlike K-NN, the Naive Bayes method is a parametric method. SVM could handle outliers better than K-NN in all conditions. In addition, SVM performs better with more variables and less training data. However, ANN model is used in larger number of variables and lesser training data. In order to achieve a good accuracy, the large training data is needed in the neural network method. Moreover, there are too many hyper-parameters in neural network to tune modeling in training data.^22

–27

The primary goal of this study was to develop ML algorithms using routinely available data to accurately predict seizure caused by acute tramadol poisoning. The secondary goals included the comparison of the predictive performance of these models and the selection of superior algorithms to provide the knowledge and tools needed for improving the ability to predict seizure.

The proposed study will thus apply machine learning to develop prediction algorithms that can more accurately identify patients at high risk of seizure with tramadol poisoning using data sources that are readily available in health care systems.

Methods

Study design and setting

This project is a cross-sectional, single-center study of routinely-collected data. The reporting guideline from the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis statement (TRIPOD) was used in the study for the development and validation of a prediction model of seizure. The ethics committee of Shahid Beheshti University of Medical Sciences approved the study (IR.SBMU.PHNS.REC.1398.110).

Study population and data sources

This research was conducted on all the patients with acute tramadol poisoning admitted to Baharloo Hospital (a poisoning referral center) in Tehran, Iran, from September 2015 to November 2019. The patients with tramadol poisoning confirmed by laboratory tests were included in the study (900 patients). The patients with an age less than 15 years, a history of renal, cardiovascular, hepatic, and respiratory disorders, epilepsy, co-ingestion of other drugs, recent seizure history, pregnancy and missing data in their clinical records were excluded from the study. Data were extracted from the patients’ electronic clinical records and the information they had given at the time of their admission to the ED using a pre-made checklist designed by two trained researchers who were blinded to the findings of the study.

Predictors

The extracted data included:

Demographic variables: Age, sex, ingested dose, manner of poisoning, history of addiction, and interval between consumption and seizure

Clinical variables: Systolic and diastolic blood pressure, Glasgow Coma Scale (GCS) score, Pulse Rate (PR), and Respiratory Rate (RR)

Paraclinical variables: Arterial blood oxygen pressure (PO₂), partial pressure of carbon dioxide (PCO₂), blood oxygen saturation (O₂ sat), blood bicarbonate level, platelet count, hemoglobin level (Hb), white blood cell count (WBC), blood sugar, and blood sodium and potassium levels.

The manner of poisoning was presented as a categorical variable with the values being suicide, accidental, overdose and unknown. History of addiction and sex were binary indicators, and the other predictors were continuous variables.

Clinical outcome

The occurrence of seizure due to acute tramadol poisoning during the first 24 hours after admission to ED was assessed as the clinical outcome of this study. Seizure was diagnosed by clinical symptoms and confirmed with Electroencephalogram (EEG) or Computed Tomography (CT) in suspected patients by an expert physician.

Statistical analysis

A statistical analysis was performed using Stata version 16.1 and R version 3.6.2 software. As for the descriptive statistics, the patients were stratified by the occurrence of seizure (with and without seizure) and their characteristics were also compared. The qualitative data were reported as frequencies and percentages and the quantitative data as mean ± standard deviation. The confidence interval (CI) was 95% and P-values <0.05 indicated statistical significance.

Machine learning algorithms

The study population was randomly split into 70% of the samples (641 patients) as training cohort for presenting a seizure risk algorithm and 30% of the samples (268 patients) as validation cohort in which the algorithms were tested for development of predictive model. Each patient could only appear in either the training or validation cohort. Four machine learning models were developed using the demographic, clinical and paraclinical variables, including (1) the k-Nearest Neighbor (k-NN), (2) the Naïve Bayes (NB), (3) the Support Vector Machine (SVM), and (4) the Artificial Neural Network (ANN). The parameters of each algorithm were tuned so as to accurately predict the seizure risk.

Variable importance

Variable importance was determined by the random forest method. This technique examined the impact of each predictor alone and in interaction with the other predictor variables.

The selection of important variables in the random forest method was based on the Mean Decrease Accuracy (MDA) and the Mean Decrease Gini (MDG). MDA was calculated as the normalized difference between the Out of Bag (OOB) accuracy of the original observation to permuted variable randomly, and MDG was calculated by summing the entire decrease in the Gini impurity at each tree node split.^28
–30 The MDA method was used in the random forest algorithm for selecting importance variables.

Performance of the machine learning algorithms

The performance of the four ML models was evaluated in the validation cohort by: (1) The k-fold cross validation (k = 10) method using the Area Under the Curve (AUC) in the Receiver Operating Characteristic (ROC) analysis, (2) the confusion matrix findings, consisting of sensitivity, specificity, positive predictive value, and negative predictive value, (3) the accuracy of the prediction models, and (4) the net benefit through the decision curve analysis.

As AUC is a threshold-independent measure of discrimination, it was chosen to compare the performance of the four ML models using pairwise comparisons and Bonferroni’s method.

Finally, the superior prediction model was chosen based on the performance of these models for application in clinical decision-making.

Results

Patient characteristics

Out of the 909 patients examined, 544 (59.85%) had seizures during the first 24 hours after tramadol consumption. The mean ± SD of age in the seizure and no seizure cases were 23.7 ± 7.1 and 23.8 ± 8.0 years, respectively. Out of the 544 cases with seizure and the 365 cases without seizure, 437 (80.3%) and 246 (67.4%) were male, respectively. Table 1 presents the other characteristics, including the cause of poisoning, addiction history, last dose of ingestion, the time from ingestion to ED admission, vital signs, and laboratory values.

Table 1.

The patients’ characteristics, stratified by seizure status.

Characteristic		n (%) / Mean ± SD
Characteristic		No seizure (n = 365)	Seizure (n = 544)
Age		23.8 ± 8.0	23.7 ± 7.1
Sex
	Female	119 (32.6)	107 (19.7)
	Male	246 (67.4)	437 (80.3)
Cause of poisoning
	Suicide	272 (74.5)	372 (68.4)
	Accidental	9 (2.5)	22 (4.0)
	Overdose	66 (18.1)	130 (23.9)
	Unknown	16 (4.4)	19 (3.5)
Addiction history
	No	215 (58.9)	267 (49.1)
	Yes	150 (41.1)	277 (50.9)
Last dose of ingestion (mg)		152.59 ± 1388.3	1949.7 ± 2051.8
Time from ingestion to admission (hour)		4.9 ± 4.0	5.0 ± 4.2
Vital sign
Systolic blood pressure (mmHg)		121.6 ± 16.6	1222.1 ± 15.6
Diastolic blood pressure (mmHg)		75.1 ± 11.1	74.9 ± 11.4
Respiratory rate (breath/minute)		16.2 ± 3.1	16.4 ± 3.2
Pulse rate (pulse/minute)		93.5 ± 17.9	101.8 ± 21.0
Glasgow Coma Scale score		14.5 ± 1.2	13.9 ± 2.2
Laboratory values
Hemoglobin (g/dl)		14.4 ± 2.3	14.7 ± 2.5
Platelet (103/mm)		239.9 ± 67.9	242.5 ± 80.0
pH		7.3 ± 0.1	6.9 ± 0.3
PO2 (mmHg)		47.3 ± 188	53.1 ± 24.8
PCO2 (mmHg)		43.0 ± 10.1	44.8 ± 10.3
Blood bicarbonate level (mmol/l)		22.7 ± 3.9	21.6 ± 3.9
O2 saturation (%)		0.69 ± 0.1	0.9 ± 0.4
Blood sugar (mg/dl)		107.9 ± 44.7	115.5 ± 44.8
Sodium (meq/l)		141.1 ± 4.2	138.8 ± 3.9
Potassium (meq/l)		3.9 ± 0.4	4.0 ± 0.4

Algorithm variable importance

Figure 1 shows variable importance of predictors in the random forest models. The variable importance is a scaled measure to have a maximum value of MDA that consist of PR, blood bicarbonate level (HCO3), pH, PO2, and sex.

Figure 1.

The variable importance of the predictors in the random forest model based on the mean decrease accuracy measure.

Model performance

The Confusion Matrix and the prediction performance of the machine learning models in the validation cohort is shown in Tables 2 and 3. The accuracy of SVM (0.65, 95% CI: 0.59–0.71), NB (0.69, 95% CI: 0.63–0.74), and neural network (0.68, 95% CI: 0.62–0.73) was significantly higher than the No Information Rate (NIR) value (P < 0.001); however, the accuracy of k-NN (0.55, 95% CI: 0.49–0.61) was not significantly different from the NIR value (P = 0.972). The highest sensitivity (0.83) and NPV (0.71) were reported for the NB model and the highest specificity (0.78) and PPV (0.73) for the k-NN model. The AUC of NB (0.71, 95% CI: 0.65–0.78) and neural network (0.71, 95% CI: 0.65–0.77) was significantly higher than the AUC of k-NN (0.68, 95% CI: 0.52–0.64) that was shown in Figure 2.

Table 2.

The Confusion Matrix of the machine learning models in the validation cohort.

Models	Prediction	Reference
Models	Prediction	No seizure	Seizure
Support Vector Machine (SVM)	No seizure	71	42
Support Vector Machine (SVM)	Seizure	52	108
Naïve Bayes (NB)	No seizure	64	26
Naïve Bayes (NB)	Seizure	59	124
Artificial Neural Network (ANN)	No seizure	80	46
Artificial Neural Network (ANN)	Seizure	43	104
K-Nearest Neighbor (k-NN)	No seizure	84	99
K-Nearest Neighbor (k-NN)	Seizure	24	66

Table 3.

The prediction performance of the machine learning models in the validation cohort.

Model	AUC	Accuracy	P-value*	Sensitivity	Specificity	PPV	NPV
Support Vector Machine (SVM)	0.68 (0.61–0.74)	0.65 (0.59–0.71)	<0.001	0.72 (0.64–0.79)	0.58 (0.49–0.66)	0.68 (0.60–0.74)	0.63 (0.54–0.71)
Naïve Bayes (NB)	0.71 (0.65–0.78)	0.69 (0.63–0.74)	<0.001	0.83 (0.76–0.88)	0.52 (0.43–0.61)	0.68 (0.61–0.74)	0.71 (0.61–0.79)
Artificial Neural Network (NN)	0.71 (0.65–0.77)	0.68 (0.62–0.73)	<0.001	0.70 (0.62–0.77)	0.65 (0.56–0.73)	0.71 (0.63–0.78)	0.65 (0.55–0.72)
K-Nearest Neighbor (k-NN)	0.59 (0.52–0.64)	0.55 (0.49–0.61)	0.972	0.40 (0.33–0.48)	0.78 (0.69–0.85)	0.73 (0.63–0.81)	(0.46 (0.39–0.53)

AUC: Area Under the Curve, PPV: Positive Predictive Value, NPV: Negative Predictive Value.

*P-value was measured to compare the accuracy of the No Information Rate (NIR) with that of each machine learning model.

Figure 2.

The prediction ability of the machine learning models, including Naïve-Bayes, neural-network and SVM, for predicting seizure in the validation cohort. The receiver-operating-characteristic (ROC) curves.

Discussion

Based on the data of 909 patients, the comparison of the four machine learning algorithms showed no significant difference to predict the occurrence of seizure due to acute tramadol poisoning. The Naïve Bayesian algorithm showed a better performance based on the AUC and accuracy, but a similar research article showed that the logistic regression model is superior for the prediction of seizure.²¹ According to the literature review, AUC is an important criterion for evaluating the efficiency of the models.^31,32 Relying on AUC alone, however, may lead to overestimating the advantages of a prediction model, and it is recommended to report other performance criteria as well to present the power performance characteristics of a specific model. Based on the present findings comparing the prediction performance of these ML algorithms, the Naïve Bayesian model had higher sensitivity and NPV, while the k-NN model had a higher specificity and PPV than the other models. The low accuracy and AUC in the k-NN model revealed the superiority of the other ML algorithms over this model for predicting seizure caused by tramadol poisoning.

Several studies have developed prediction models for several types of seizure using important predictive factors and various analysis methods, such as frequency-based methods using EEG signals, the Least Absolute Shrinkage and Selection Operator (LASSO) regression, nonlinear dynamics (chaos), and logistic regression and machine learning models such as k-NN, SVM and the deep learning classifier.^{17

–20,33
–35} Nonetheless, the superiority of the application of ML algorithms over conventional logistic regression models to predict seizure remains a controversial assumption. The review of literature did not show any studies that had managed to forecast seizure caused by tramadol poisoning.

As too many features can decrease the accuracy of a prediction model in the machine learning approach, the use of the Random Forest (RF) method to reduce the variables and consider the correlation of feature is outperformed for selection important variables. Many studies similar to the present study have applied the RF method to develop prediction models for clinical outcomes and drug consumption complications.^36
–38

Comparing these ML models with the LR model (AUC = 0.77, sensitivity = 0.8, specificity = 0.6, Positive Predictive Value = 0.71, Negative Predictive Value = 0.64 and accuracy = 0.75) with the same predictors and outcome demonstrated that the LR model has a superior performance in the prediction of seizure caused by tramadol poisoning on the criteria performance with equal number of important variables.²¹

The main goal of Emergency Department (ED) triage is to accurately identify high-risk patients to enhance decision-making using the available patient information. Previous studies have reported many related and predictive variables for the occurrence of seizure in acute tramadol poisoning for differentiating high and low-risk patients. Nevertheless, this study showed that the use of demographic, clinical and paraclinical data in ML methods may improve the prediction ability due to the times limitations and the priority of patients in EDs. The selected predictive variables of the four ML models included pulse rate, pH, blood bicarbonate level, pressure of oxygen in blood and sex, which is consistent with the results of other studies.^{21,39

–43}

Similar to the present research, a previous systematic review showed no performance benefits for ML over LR models to predict various clinical outcomes.¹⁵ The regression logistic model outperformed the others due to its better interpretation model, simple analysis and representation of the model as a nomogram with equal variables. Additionally, LR models can be used for qualitative and semi-quantitative predictors. Log transformation can be used in LR models to transform the nonlinear relationships between the dependent and independent variables into linear relationships with lower restriction conditions and data types.^44,45 Therefore, improvements in methodology and reporting are necessary for investigations that compare prediction modeling.

The strengths of this study include being the first research to predict acute tramadol-poisoning-induced seizure using the routine demographic, clinical and paraclinical information taken from patients and the utilization of ML models. Also, compared to previous studies conducted to predict the risk of opioid use disorder, our models applied fewer important variables to predict seizure caused by tramadol as an opioid drug. Therefore, these models are practically more efficient than previously trained ML algorithms for predicting complication of poisoning.

The potential limitations of this study include the development of prediction algorithms for patients with tramadol poisoning presenting to a medical setting, which means that the findings may not be generalizable to the general population. Also, data were collected from a single medical center, while a multicenter research could improve the efficiency of these models. Nevertheless, the examined center was a referral poisoning center in Tehran, Iran, to which patients with different demographic profiles are often admitted. Second, not all the important patient information, such as EEG reports, were available in the clinical records and hospital electronic system. Hence, the researchers recommend the development of predictive models based on other important available data as well. Third, our models do not provide estimates of errors to predict seizure individually. This limitation can be resolved in future developments of robust prediction models. The fourth limitation was the selection of four rather common ML algorithms from a much larger pool of ML algorithms. Therefore, future studies are recommending to extend other ML algorithms to predict seizure poisoning and compare them with our proposed models.

Conclusion

This study presented the first prognostic algorithm to predict seizure caused by acute tramadol poisoning applying common ML models and using routine demographic, clinical and paraclinical data, and the important predictive variables were selected with the RF method. The support vector machine, neural network and naïve Bayesian models had no significant difference in the AUC and other performance criteria. The k-NN was not an appropriate model for forecasting seizure in tramadol poisoning due to its low accuracy and AUC. In contrast, the validated logistic regression was superior algorithm for predicting seizure in EDs because of its better interpretation and performance model, simple analysis, representation of the model as a decision tool, fit for qualitative and semi-quantitative predictors and possibility of transformation of linear relationships between dependent and independent variables with less restriction conditions and lower data types and ultimately its ability to guide clinicians toward better decision-making and thus improving the quality of care.

Footnotes

Acknowledgements

We express our gratitude to the team involved in the collection of the data at Baharloo Hospital (a poisoning referral center) in Tehran, Iran.

Author contributions

All authors contributed equally to this work. Also, all authors read and approved the final version of the manuscript and met the criteria of authorship based on the recommendations of the international committee of medical journal editors.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

E Bazmi

References

Liang

Goros

Turner

. Drug overdose: differing risk models for women and men among opioid users with non-cancer pain. Pain Med 2016; 17(12): 2268–2279.

Hylan

Von Korff

Saunders

, et al. Automated prediction of risk for problem opioid use in a primary care setting. J Pain 2015; 16(4): 380–387.

Miotto

Cho

Khalil

, et al. Trends in tramadol: pharmacology, metabolism, and misuse. Anesth Analg 2017; 124(1): 44–51.

Mehrpour

. Addiction and seizure ability of tramadol in high-risk patients. Indian J Anaesth 2013; 57(1): 86–87.

Bigdeli

Corazza

Aslanpour

, et al. Novel psychoactive substances (NPS): a study on Persian language websites. Iran J Public Health 2013; 42(5): 511–515.

Rehni

Singh

Kumar

. Tramadol-induced seizurogenic effect: a possible role of opioid-dependent gamma-aminobutyric acid inhibitory pathway. Basic Clin Pharmacol Toxicol 2008; 103(3): 262–266.

Baldo

. Opioid analgesic drugs and serotonin toxicity (syndrome): mechanisms, animal models, and links to clinical effects. Arch Toxicol 2018; 92(8): 2457–2473.

Sansone

. Tramadol: seizures, serotonin syndrome, and coadministered antidepressants. Psychiatry (Edgmont) 2009; 6(4): 17–21.

Petramfar

. Tramadol induced seizure: report of 106 patients. Iran Red Crescent Med J 2010; 1: 49–51.

10.

Chari

Yadav

Nishijima

, et al. Improving the ability of ED physicians to identify subclinical/electrographic seizures on EEG after a brief training module. Int J Emerg Med 2019; 12(1): 11.

11.

Abbasi

Goldenholz

. Machine learning applications in epilepsy. Epilepsia 2019; 60(10): 2037–2047.

12.

Iasemidis

. Seizure prediction and its applications. Neurosurg Clin N Am 2011; 22(4): 489–506.

13.

Steyerberg

. Clinical prediction model. New York, NY: Springer, 2009.

14.

Taghaddosinejad

Mehrpour

Afshari

, et al. Factors related to seizure in tramadol poisoning and its blood concentration. J Med Toxicol 2011; 7(3): 183–188.

15.

Christodoulou

Collins

, et al. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol 2019; 110: 12–22.

16.

Kuhn

Johnson

. Applied predictive modeling. New York, NY: Springer-Verlag, 2013.

17.

Zhang

Han

Zhao

, et al. Personalized prediction model for seizure-free epilepsy with levetiracetam therapy: a retrospective data analysis using support vector machine. Br J Clin Pharmacol 2018; 84(11): 2615–2624.

18.

Yadollahpour

AMJ

Jalilifar

. Seizure prediction methods: a review of the current predicting techniques. Biomed Pharmacol J 2014; 7(1).

19.

Allotey

Fernandez-Felix

Zamora

, et al. Predicting seizures in pregnant women with epilepsy: development and external validation of a prognostic model. PLoS Med 2019; 16(5): e1002802.

20.

Hussain

. Detecting epileptic seizure with different feature extracting strategies using robust machine learning classification techniques by applying advance parameter optimization approach. Cogn Neurodynamics 2018; 12(3): 271–294.

21.

Bazmi

Behnoush

Hashemi Nazari

, et al. Seizure prediction model in acute tramadol poisoning; a derivation and validation study. Arch Acad Emerg Med 2020; 8(1): e59.

22.

Bishop

. Pattern recognition and machine learning. London: Springer, 2006.

23.

Yang

Jin

. Distance metric learning: a comprehensive survey. Technical report, Michigan State University, Vol. 2(2), 2006, p. 4.

24.

Zou

Han

. Overview of artificial neural networks. Methods Mol Biol 2008; 458: 15–23.

25.

Ben-Hur

Weston

. A user’s guide to support vector machines. Methods Mol Biol 2010; 609: 223–239.

26.

Greiner

Schuurmans

. In: Proceedings of 21st international conference on machine learning, Pasadena, California. AAAI Press, 2004.

27.

Domingos

Pazzani

. On the optimality of the simple Bayesian classifier under zero-one loss. Mach Learn 1997; 29(2–3): 103–130.

28.

Cutler

Edwards

TCJ

Beard

, et al. Random forests for classification in ecology. Ecology 2007; 88(11): 2783–2792.

29.

Calle

Urrea

. Letter to the editor: stability of random forest importance measures. Brief Bioinform 2011; 12(1): 86–89.

30.

Strobl

Statistical issues in machine learning: towards reliable split selection and variable importance measure. PhD Thesis, Ludwig-Maximilians-Universitat Munchen, German, 2008.

31.

Zou

O’Malley

Mauri

. Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models. Circulation 2007; 115: 654–657.

32.

Hand

Till

. A simple generalisation of the area under the ROC curve for multiple class classification problems. Mach Learn 2001; 45: 171–186.

33.

Siuly

Kabir

Wang

, et al. Exploring sampling in the detection of multicategory EEG signals. Comput Math Methods Med 2015; 2015: 576437.

34.

Sharif

Jafari

. Prediction of epileptic seizures from EEG using analysis of ictal rules on Poincaré plane. Comput Methods Programs Biomed 2017; 145: 11–22.

35.

Kang

J-H

Chung

Kim

S-P

. An efficient detection of epileptic seizure by differentiation and spectral analysis of electroencephalograms. Comput Biol Med 2015; 66: 352–356.

36.

Cui

Luo

Tseng

, et al. Combining handcrafted features with latent variables in machine learning for prediction of radiation-induced lung damage. Med Phys 2019; 46(5): 2497–2511.

37.

Darst

Malecki

Engelman

. Using recursive feature elimination in random forest to account for correlated variables in high dimensional data. BMC Genet 2018; 19(Suppl 1): 65.

38.

Rai

. Feature selection and predictive modeling of housing data using random forest. Int J Bus Econ Eng 2017; 11(4): 919–923.

39.

Habibollahi

Garjani

Shams Vahdati

, et al. Severe complications of tramadol overdose in Iran. Epidemiol Health 2019; 41: e2019026.

40.

Dadpour

Alizadeh

Vahabzadeh

, et al. Arterial blood gas analysis of patients with tramadol-induced seizure; a cross sectional study. Arch Acad Emerg Med 2020; 8(1): e14.

41.

Morrow

Dormuth

Paterson

, et al. Tramadol and the risk of seizure: nested case-control study of US patients with employer-sponsored health benefits. BMJ Open 2019; 9(3): e026705.

42.

Nass

Sassen

Elger

, et al. The role of postictal laboratory blood analyses in the diagnosis and prognosis of seizures. Seizure 2017; 47: 51–65.

43.

Moridani

Farhadi

. Heart rate variability as a biomarker for epilepsy seizure prediction. Bratisl Lek Listy 2017; 118(1): 3–8.

44.

Biglarian

Bakhshi

Gohari

, et al., Artificial neural network for prediction of distant metastasis in colorectal cancer. Asian Pac J Cancer Prev 2012; 13(3): 927–930.

45.

Fei

Gao

, et al. Predicting risk for portal vein thrombosis in acute pancreatitis patients: a comparison of radical basis function artificial neural network and logistic regression models. J Crit Care 2017; 39: 115–123.