Development of Machine Learning Models for the Identification of Elevated Ketone Bodies During Hyperglycemia in Patients with Type 1 Diabetes

Abstract

Aims:

Diabetic ketoacidosis (DKA) is a serious life-threatening condition caused by a lack of insulin, which leads to elevated plasma glucose and metabolic acidosis. Early identification of developing DKA is important to start treatment and minimize complications and risk of death. The aim of the present study is to develop and test prediction model(s) that gives an alarm about their risk of developing elevated ketone bodies during hyperglycemia.

Methods:

We analyzed data from 138 type 1 diabetes patients with measurements of ketone bodies and continuous glucose monitoring (CGM) data from over 30,000 days of wear time. We utilized a supervised binary classification machine learning approach to identify elevated levels of ketone bodies (≥0.6 mmol/L). Data material was randomly divided at patient level in 70%/30% (training/test) dataset. Logistic regression (LR) and random forest (RF) classifier were compared.

Results:

Among included patients, 913 ketone samples were eligible for modeling, including 273 event samples with ketone levels ≥0.6 mmol/L. An area under the receiver operating characteristic curve from the RF classifier was 0.836 (confidence interval [CI] 90%, 0.783–0.886) and 0.710 (CI 90%, 0.646–0.77) for the LR classifier.

Conclusions:

The novel approach for identifying elevated ketone levels in patients with type 1 diabetes utilized in this study indicates that CGM could be a valuable resource for the early prediction of patients at risk of developing DKA. Future studies are needed to validate the results.

Introduction

Diabetic ketoacidosis (DKA) is a serious life-threatening problem caused by a lack of insulin, which leads to elevated plasma glucose, metabolic acidosis, and ketones in the serum or the urine.^1

–4 The clinical symptoms of DKA are normally polyuria, polydipsia, polyphagia, weight loss, vomiting, abdominal pain, and fatigue.¹ DKA occurs among both type 1 and type 2 diabetes, however, most often it occurs among children or adolescents with type 1 diabetes.^1,4 Even though, the mortality rate of single DKA is only 5.2%, a recurrent attack is around 23.4%.^2,5 From literature, it is known that too late treatment leads to severe complications for example, edema in the brain.⁶ Thus, there is a need of investigating the potential in early detection of DKA among this group of patients, who potentially could avoid DKA to occur.^1,7
–9

The etiology of DKA is related to insulin requirement and the precipitating factors are omission of insulin; intercurrent illness; drugs that affect carbohydrate metabolism; mechanical problems with continuous subcutaneous insulin infusion devices, and damaged insulin.¹⁰

For many years, continuous glucose monitoring (CGM) has been used by patients with type 1 diabetes to learn about their blood glucose level and thus learn to live with diabetes.¹¹ In fact, several studies demonstrate how the use of CGM has shown better glycemic results and significant improvement in the level of HbA1c.^11

–14 Literature shows that having knowledge and insight into ones' own blood glucose level contribute to a better management of the diabetes.^15
–17 Furthermore, CGM allows professionals to track patients' blood glucose levels over time and guide patients the best ways to avoid DKA.¹²

Prediction models have been used in many health care research areas and practices and have contributed to predict whether an event will happen, and likewise, predict an event before it happens.^18,19 A health care research area where prediction models have gained large profit is within the area of chronic obstructive pulmonary disease.²⁰ Literature shows how it is now possible to predict an exacerbation before the exacerbation occurs, and hence the patient will receive information about the future event if he/she does not act, for example, take antibiotics to avoid the exacerbation.^21
–23 Similar prediction models are available within the area of DKA, for example, mortality prediction models.^24,25

Also, there has been conducted studies within DKA, which investigated different predictors for adverse outcome in DKA.^7,26
–28 For instance, a study published by Li et al. 2021 tested several predictors from the electronic health record for example, HbA1c, BMI, height, white blood cell count, pulse rate, respiratory rate, oxygen, and so on. Li et al. did not, however, investigate the potential in using CGM as a predictor for developing DKA.²⁸

The development of a prediction model based on CGM values, capable of issuing an alert indicating the imminent risk of DKA if insulin is not administered, would be a valuable approach to reduce the onset of DKA. Thus, the aim of the present study is to develop and test a prediction model for the risk of elevated ketone bodies during hyperglycemia.

Materials and Methods

Study data

To investigate whether patterns from CGM could help identify elevated ketone bodies as a risk factor for DKA in patients with diabetes, data from The International Diabetes Closed 1 Loop (iDCL) trial: Clinical Acceptance of the Artificial Pancreas ²⁹ were analyzed. Their study was a multicenter randomized controlled trial of the at-home closed-loop system versus the standard of care.

The study²⁹ included patient with type 1 diabetes aged 14–71. The patients used the Dexcom G6 CGM system and a t:slim X2 with Control-IQ (intervention group) for 6 months. The patients were also equipped with a blood ketone meter and test strips and were instructed to measure ketone levels if glucose readings were >300 mg/dL. The study included 168 patients with type 1 diabetes, HbA1c 7.4 (1.0) % (9.2 mmol/L), with an average age of 33 (16) years, 54% were female, and median duration of diabetes of 17(8–28) years.

For this analysis, inclusion criteria were patients with ketone measurements and associated CGM data in a period for 4 h before the ketone measurements. The CGM periods needed to have ≥70% wear time. The CGM system had a sample rate of 12/h, so inclusion required ≥34 glucose samples in the 4-h window.

This study adheres to the advised standards defined by the “Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis” (TRIPOD).³⁰

Model development

We utilized a supervised binary classification machine learning approach to identify elevated levels of ketone bodies.^31
–33 Ketone levels <0.6 mmol/L are in the normal range and ≥0.6 mmol/L increases the risk of DKA significantly.³⁴ We defined the binary classification problem as an increased/nonincreased risk (ketone bodies ≥0.6 mmol/L; ketone bodies <0.6 mmol/L) during high-glucose reading. Features from a fixed period of 4 h before the ketone measurement were used to predict increased ketone bodies. Data material was then randomly divided at patient level: 70% was used for training the models and 30% was used for testing the final models. This process ensured that cases from a specific patient were exclusively assigned to either the training or test datasets, and the results were not prone to in-sample overfitting, allowing for better transferability to a similar cohort.³⁵ Figure 1 illustrates the method.

FIG. 1.

Illustrates the model development. For each patient, a window with CGM readings was extracted before each ketone measurement. Eligible samples were defined as ketone measurements with a window containing ≥ data completion of CGM samples. Binary sample classes were defined as ketone measurements with a level ≥0.6 mmol/L (class = 1, otherwise class = 0). For each eligible sample, features were extracted from the window of CGM readings before the ketone test measurement. The total eligible samples were divided on the patient level into a 70% training dataset and a 30% test dataset. Selection of features, model hyperparameter estimation and training were performed on the training dataset. The final model(s) performance was assessed using the test dataset. CGM, continuous glucose monitoring.

The analysis and models were implemented using MATLAB R2020b (The MathWorks, Inc., Natick, MA) and Python 3 with packages (sklearn 1.0.2, pandas 1.4.2, numpy 1.21.5, matplotlib 3.7).

Feature extraction and reduction

In total, 14 features were extracted from CGM data in a 4-h window before the ketone samples. An example of a 4-h window from a patient is shown in Figure 2. The features comprised mathematical transformations of the signal to describe the dynamics, range of change, distribution, and extreme values. The approach was data-driven and explorative, as we did not have prior knowledge on which features of the signal would hold discriminative information. Glucose dynamics are influenced by daily cycles, partly induced by hormone secretion and habits such as meals and sleep. Therefore, a feature on the hour of day Let CGM_k (k = −48, −47, …, −1) be the CGM measurements in the 4-h window before each ketone sample was included. Features are calculated as described in Table 1.

FIG. 2.

Illustrates a ketone measurement with adjacent CGM measurement for 4 h prior.

Table 1.

The Equations Show How the Features Are Extracted from the Continuous Glucose Monitoring Window

Feature description	Equation(s)
First value in CGM window	$C G M_{- 1}$
Last value in CGM window	$C G M_{- 48}$
Minimum value in CGM window	${min}_{k} C G M_{k}$
Maximum value in CGM window	${max}_{k} C G M_{k}$
Summation in CGM window	$\sum C G M_{k}$
Mean value from CGM window	$\frac{1}{n} \sum C G M_{k}$
Standard deviation from CGM window	$\sqrt{\frac{1}{n} \sum_{k = 1}^{n} {(C G M_{k} - μ)}^{2}}$
Difference between time point in CGM window	$C G M_{- 3} - C G M_{- 1}$
	$C G M_{- 6} - C G M_{- 1}$
	$C G M_{- 12} - C G M_{- 1}$
Skewness of distribution	${\tilde{μ}}_{3} (C G M_{k})$
Kurtosis of distribution	$\frac{{\tilde{μ}}_{4} (C G M_{k})}{σ^{4} (C G M_{k})}$
Slope of linear regression	$β^{1}$
Hour of day	$h o u r (C G M_{- 1})$

CGM, continuous glucose monitoring.

Choice of machine learning algorithm

Simple linear classifier, logistic regression (LR), and a more complex nonlinear classifier random forest (RF) were compared. LR was used due to the simplicity of the model with a linear combination of features, which gave an easily interpretable model.

Given the binary outcome and the need for a nonlinear approach, prediction model(s) using RF classification were developed. RF is a versatile and robust machine learning method³⁶ that uses a collection of decision trees to make predictions. RF has several benefits, including low variance, high accuracy, relative robustness to overfitting, ability to handle nonlinear relationships, and interpretability using feature importance.

Hyperparameter tuning (estimators, learn rate, leaf size, and max splits) was determined using grid search and threefold cross-validation on the training dataset.³⁷ The final model was then tested on the test dataset. In this study, the model was not recalibrated after training the classifier.

Performance metrics

Several common performance metrics for binary classifiers were calculated.³⁸ To evaluate the general performance of the model, the area under the receiver operating characteristic (ROC) curve (ROC-AUC) was used for comparison.³⁹ Confidence intervals were estimated using bootstrap replicates (n = 10,000).⁴⁰

The performance for specific thresholds from the ROC-AUC to compare in clinical use situations for the patients was also calculated. Every sample in a testing procedure has a real class label and a predicted class label. The real label indicates the class to which the testing sample truly belongs. The predicted label is the output of the classifier at a specific threshold. Let S_k (k = 1, 2, …, n) be the test samples, y(S_k) be the true label of S_k and y_p (S_k) be the class prediction of S_k. Thus, the performance assessing true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) were calculated: $T P = \sum \{S_{k} |y (S_{k}) = 1, Y_{p} (S_{k}) = 1\}$ $T N = \sum \{S_{k} |y (S_{k}) = 0, Y_{p} (S_{k}) = 0\}$

F P = \sum \{S_{k} |y (S_{k}) = 0, Y_{p} (S_{k}) = 1\}

F N = \sum \{S_{k} |y (S_{k}) = 1, Y_{p} (S_{k}) = 0\}

The number of predicted episodes with elevated ketone levels out of the total number of true episodes is important from a clinical perspective. However, patients with diabetes are also prone to device alarm fatigue.⁴¹ Hence, the percentage of true alarm is an important consideration. Hence, the sensitivity (Sen), positive predictive value (PPV) as well as the specificity (Spe), and negative predictive value (NPV) in the reporting of performance metrics were calculated: $S e n = \frac{T P}{T P + F N}$ $S p e = \frac{T N}{T N + F P}$

N P V = \frac{T N}{T N + F N}

P P V = \frac{T P}{T P + F P}

Results

In total, 138 patients were included in the analysis. Thirty patients did not have qualified ketone measurements with a CGM window. Among the included patients, 913 ketone samples were eligible for modeling, including 273 event samples with ketone levels ≥0.6 mmol/L. The patients had over 9 million CGM measurements combined corresponding to over 30,000 days with full-time monitoring.

Performance

Random forest

The ROC-AUC from the training dataset was 0.836 (confidence interval [CI] 90%, 0.809–0.862) and 0.836 (CI 90%, 0.783–0.886) for the test dataset. The ROC curves for the training/test dataset are plotted in Figure 3.

FIG. 3.

ROC curve(s) from the training/test dataset of the RF model. RF, random forest; ROC, receiver operating characteristic.

As concrete examples: based on the ROC-AUC, Table 2, if a sensitivity of 70% (identifying ∼3/4 elevated ketone levels) were chosen that would correspond to a PPV of 57%. This means that ∼3/4 of elevated ketone levels could be predicted, and among episodes predicted as potentially elevated ketone levels, ∼6/10 would be true and ∼4/10 would be false predictions. If a lower sensitivity of 49% (predicting ∼1/2 elevated ketone levels) would be acceptable, that would yield a PPV of 67%. This means that if a lower sensitivity could be accepted, the amount of false elevated ketone level prediction would be substantially reduced.

Table 2.

Performance Measures for the Two Models, One Cutoff with High Sensitivity and One Cutoff with Low Sensitivity

Model(s)	Performance measure
Model(s)	Sensitivity	Specificity	PPV	NPV
Cutoff = 0.25
RF	0.70	0.79	0.57	0.86
LR	0.68	0.65	0.44	0.83
Cutoff = 0.40
RF	0.49	0.90	0.67	0.81
LR	0.48	0.87	0.60	0.81
Cutoff = 0.50
RF	0.39	0.95	0.74	0.79
LR	0.30	0.94	0.67	0.77

LR, logistic regression; NPV, negative predictive value; PPV, positive predictive value; RF, random forest.

Feature importance is illustrated in Figure 4, the five most important features are the following: maximum value, mean value, summation, last value, and hour of day.

FIG. 4.

Feature importance of the included features (MDI). MDI, mean decrease in impurity.

Logistic regression

The ROC-AUC from the training dataset was 0.725 (CI 90%, 0.687–0.762) and 0.710 (CI 90%, 0.646–0.77) for the test dataset. The difference between the ROC-AUC (RF vs. LR) models was significant (P < 0.001).

As concrete examples based on the ROC-AUC, Table 2, if a sensitivity of 68% (identifying ∼3/4 elevated ketone levels) were chosen that would correspond to a PPV of 44%. This means that ∼3/4 of elevated ketone levels could be predicted, and among episodes predicted as potentially elevated ketone levels, ∼4/10 would be true and ∼6/10 would be false predictions. If a lower sensitivity of 48% (predicting ∼1/2 elevated ketone levels) would be acceptable that would yield a PPV of 60%.

Discussion

The aim of this study was to develop and test a prediction model that could alert about their risk of developing DKA in the form of elevated ketone bodies. The results demonstrate that patterns from CGM collected in patients with type 1 diabetes have the potential to be utilized to identify and alarm patients at risk of elevated ketone levels. Elevated ketone levels are a precursor for the development of a dangerous and life-threatening condition, DKA. To our knowledge, this study is the first to investigate the potential of using these types of data for the prediction of acute increased levels of ketone bodies.

The performance of the LR was low, and the ROC-AUC of 0.71 makes it less unfeasible in practice, as more FP would need to be accepted. However, the performance of the RF algorithm was acceptable, with an ROC-AUC of 0.83. The most important features of elevated ketone levels were maximum value, mean value, summation, last value, and hour of day. These features indicated that patterns from the dynamics, distribution, extreme values of the CGM signal and diurnal patterns could be used for the identification of elevated ketone levels.

Implications

The goal of intensive diabetes management is to obtain optimal glycemic control to prevent and reduce diabetes complications such as hypoglycemia, hyperglycemia, and diabetic ketoacidosis.⁴² CGM has been demonstrated to help improve glycemic control while reducing the incidence of hypoglycemia and DKA when compared to regular capillary self-measurement of blood glucose.^43
–45 Hence, the adoption of CGM is increasing rapidly among people with diabetes.⁴⁶ While CGM as an independent system has proven beneficial, it also opens for layering enhancing algorithms on top of the system. Such an enhancing algorithm could be a model and an alarm for identifying elevated ketone levels.

In the daily life of people using CGM, an alarm for the risk of elevated ketone levels, could notify the user on the need for subsequent testing with a blood or urine ketone testing kit and even starting early on preventing measures for developing DKA. It is likely that such measures may help reduce the number of DKA events and decrease insecurity in this group. A group of users who would be interested in the prediction model is parents to children having type 1 diabetes. Moreover, literature shows how these parents constantly worry/fear about their children or adolescents experiencing hypoglycemia or anything else related to the disease.^47,48

Strengths and limitations

Living a life with type 1 diabetes prescribes a life of several restrictions to avoid complications. Therefore, initiatives who help the patients to live a normal life as possible, may have a large impact on quality of life. In this study, we utilized patterns from CGM to identify increased levels of ketone bodies. To our knowledge, this is the first study that aims to do so. Data used in the study were derived from a large sample of the studied population, patients with type 1 diabetes.

Even though our study has a strong design with a large data material and clear separation between training and testing, there are some limitations that are important to mention. Despite the inclusion of a large sample with many episodes, the number of outcome events was rather small. Over 30,000 days with full-time CGM were analyzed, and only 273 events with ketone levels ≥0.6 mmol/L were found. This is also seen from the test sample results, where the ROC-AUC had a broad CI. This means that the expected performance on new data is not yet reliable, even though the results evidently indicate that the data hold valuable information. An attempt to improve the performance could be to conduct the analysis on a larger dataset.

The included population in the study covers adolescents and adult patients with ketone measurements during hyperglycemic ranges, which gives us an indication of performance in that population. However, the performance in subgroups is not covered and could potentially be subject to intervariability. Noteworthy, DKA can also appear during euglycemic ranges and the CGM patterns are potentially significantly different than the patterns observed during hyperglycemia.

Also, the study's participant cohort comprised individuals engaged in a technological investigation, introducing a potential limitation associated with their inherent characteristics. Specifically, participants involved in device-centric studies often exhibit reduced HbA1c levels compared to the broader population, demonstrating increased adherence to diabetes management protocols, including the vigilant execution of pump site changes, insulin administration, and prompt treatment of hyperglycemic episodes. Consequently, the glycemic patterns observed preceding and during ketone development in this study may deviate from those of populations less proactive in diabetes management.

To enhance the comprehensiveness of the findings and their potential applicability, a valuable avenue for future research involves comparing the model's performance with cohorts characterized by previous admissions for DKA or elevated HbA1c levels. Such cohorts represent individuals with a higher likelihood of benefiting from DKA prediction. In addition, it is noteworthy that the model's evaluation focused on pump users, and the outcomes may not be directly extrapolated to injection users relying on long-acting insulin, potentially influencing the glucose-ketone relationship as monitored by CGM systems. This consideration underscores the importance of diversifying the study population to encompass various diabetes management modalities for a more comprehensive understanding of the model's generalizability.

It is not possible to compare the results from our proposed novel approach to existing research since this is the first study to investigate the potential of utilizing CGM patterns to identify elevated ketone levels. The pathophysiology of DKA is well known.¹⁰ It involves the complex combination of insulin deficiency, lack of exogen-administered insulin and increased counterregulatory hormones such as glucagon, growth hormone, catecholamines, and cortisol. This can lead to altered glucose production and consumption and to increased lipolysis and subsequent production of ketone bodies.^49,50

Future research

Future work includes validating the results in patients with different characteristics such as children, high DKA risk groups, individuals with high HbA1c and relying on insulin injections. It would also be interesting to add additional information to the models such as meal intake, insulin administration, insulin-on-board estimation, and demographic information—which all have an important physiological role in carbohydrate and lipid metabolism.⁵¹

However, a notable concern arises with the utilization of insulin data obtained from pump devices. It is imperative to acknowledge the inherent risk of deriving inaccurate assessments of the actual insulin present in the body when relying on pump-collected data. This concern emanates from a prevalent issue in pump users, wherein one of the significant causes of DKA is attributed to infusion set malfunctions. In such cases, the pump may erroneously indicate the presence of insulin on board, when, in reality, the intended insulin delivery might not have reached the patient. This potential source of misinformation underscores the necessity for cautious consideration and validation when incorporating pump-derived insulin data into models.

There are many alternative machine learning algorithms for binary classification, including but not limited to ensemble methods, K-nearest neighbors, support vector machines, and neural networks. We have recently shown on large datasets, including CGM, that neural networks have good performance for predictive modeling.^52,53 Future research could compare the performance of more algorithms simultaneously.

Finally, the clinical impact of the proposed model needs to be evaluated before the implications can be fully understood.

Conclusion

The novel approach for identifying elevated ketone levels during hyperglycemia in patients with type 1 diabetes utilized in this study indicates that CGM could be a valuable resource for the early prediction of patients at risk of developing DKA. Future studies are needed to validate the results further and to explore the potential improvement of adding information from the insulin regime and dosing in the models.

Footnotes

Authors' Contributions

S.L.C. had access to all data analyzed in this study. S.L.C. takes responsibility for the integrity and accuracy of the study data analysis and results. S.L.C. and C.B. were involved in the study design, concept, analysis, and interpretation of the data. S.L.C. drafted the article and performed the statistical analysis. C.B. were involved in critical revision of the article.

Disclaimer

The source of the data is from the DCLP3 trial, but the analyses, content, and conclusions presented herein are solely the responsibility of the authors and have not been reviewed or approved by DCLP3 group.

Author Disclosure Statement

The research was funded by i-SENS, Inc. (Seoul, South Korea) and S.L.C.'s involvement with the company did not influence the design, implementation, or interpretation of the study. S.L.C. have received research funding from i-SENS, Inc. (Seoul, South Korea), which manufactures some of the product types discussed in this article. However, the study was conducted independently, and the authors declare that their involvement with i-SENS, Inc. (Seoul, South Korea) did not influence the findings or conclusions of the study.

Funding Information

The study was funded by i-SENS, Inc. (Seoul, South Korea).

References

Evans

. Diabetic ketoacidosis: Update on management. Clin Med, 2019; 19(5):396–398.

Fazeli Farsani

, Brodovicz

, Soleymanlou

. Correction: Incidence and prevalence of diabetic ketoacidosis (DKA) among adults with type 1 diabetes mellitus (T1D): A systematic literature review. BMJ Open, 2017; 7(8):e016587; doi: 10.1136/bmjopen-2017-016587corr1)

Long

, Lentz

, Koyfman

, et al. Euglycemic diabetic ketoacidosis: Etiologies, evaluation, and management. Am J Emerg Med, 2021; 44:157–160.

Dhatariya

. Defining and characterising diabetic ketoacidosis in adults. Diabetes Res Clin Pract, 2019 Sep;155:107797. doi: 10.1016/j.diabres.2019.107797. Epub 2019 Jul 22. PMID: 31344382.

Brandstaetter

, Bartal

, Sagy

, et al. Recurrent diabetic ketoacidosis. Arch Endocrinol Metab, 2019; 63(5):531–535.

Barski

, Eshkoli

, Brandstaetter

, et al. Euglycemic diabetic ketoacidosis. Eur J Intern Med, 2019; 63:9–14.

Reinhart

. Early detection of diabetic ketoacidosis by breathalyzer in a sailor reporting for duty. Mil Med, 2019; 184(11–12):e951–e952.

Eledrisi

, Elzouki

A-N

. Management of diabetic ketoacidosis in adults: A narrative review. Saudi J Med Med Sci, 2020; 8(3):165.

Usman

, Bakry

, Mustafa

, et al. Correlation of acidosis-adjusted potassium level and cardiovascular outcomes in diabetic ketoacidosis: A systematic review. Diabetes Metab Syndr Obes, 2019; 12:1323–1338.

10.

Nyenwe

, Kitabchi

. The evolution of diabetic ketoacidosis: An update of its etiology, pathogenesis and management. Metabolism, 2016; 65(4):507–521.

11.

DeSalvo

, Miller

, Hermann

, et al. Continuous glucose monitoring and glycemic control among youth with type 1 diabetes: International comparison from the T1D Exchange and DPV Initiative. Pediatr Diabetes, 2018; 19(7):1271–1275.

12.

Azhar

, Gillani

, Mohiuddin

, et al. A systematic review on clinical implication of continuous glucose monitoring in diabetes management. J Pharm Bioallied Sci, 2020; 12(2):102–111.

13.

Poolsup

, Suksomboon

, Kyaw

. Systematic review and meta-analysis of the effectiveness of continuous glucose monitoring (CGM) on glucose control in diabetes. Diabetol Metab Syndr, 2013; 5(1):39.

14.

Laffel

, Kanapka

, Beck

, et al. Effect of continuous glucose monitoring on glycemic control in adolescents and young adults with type 1 diabetes: A randomized clinical trial. JAMA, 2020; 323(23):2388–2396.

15.

Haris

, Kristianti

. The correlation between the knowledge level of diabetes management toward the preprandial glucose levels. Indones J Nurs Pract, 2020; 4(1):21–27.

16.

, Van Wieren Jones

, Kyle Hansen

, et al. The impact of self-monitoring blood glucose adherence on glycemic goal attainment in an indigent population, with pharmacy assistance. P T, 2019; 44(9):554–559.

17.

Telo

, De Souza

, Andrade

, et al. Comparison between adherence assessments and blood glucose monitoring measures to predict glycemic control in adults with type 1 diabetes: A cross-sectional study. Diabetol Metab Syndr, 2016; 8(1):4–9.

18.

Locquet

, Diep

, Beaudart

, et al. A systematic review of prediction models to diagnose COVID-19 in adults admitted to healthcare centers. Arch Public Health, 2021; 79(1):1–15.

19.

de Hond

AAH

, Leeuwenberg

, Hooft

, et al. Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: A scoping review. NPJ Digit Med, 2022; 5(1):2.

20.

Guerra

, Gaveikaite

, Bianchi

, et al. Prediction models for exacerbations in patients with COPD. Eur Respir Rev, 2017; 26(143):160061.

21.

Kronborg

Predicting Exacerbations in Patients with Chronic Obstructive Pulmonary Disease. Aalborg Universitetsforlag. Aalborg Universitet. Det Sundhedsvidenskabelige Fakultet. Ph.D.-serien; 2019.

22.

Kronborg

, Mark

, Cichosz

, et al. Population exacerbation incidence contains predictive information of acute exacerbations in patients with chronic obstructive pulmonary disease in telecare. Int J Med Inform, 2018; 111(December 2017):72–76.

23.

Kronborg

, Hangaard

, Cichosz

, et al. A two-layer probabilistic model to predict COPD exacerbations for patients in telehealth. Comput Biol Med, 2021; 128:104108.

24.

Efstathiou

, Tsiakou

, Tsioulos

, et al. A mortality prediction model in diabetic ketoacidosis. Clin Endocrinol (Oxf), 2002; 57(5):595–601.

25.

Siregar

, Soewondo

, Subekti

, et al. Seventy-two hour mortality prediction model in patients with diabetic ketoacidosis: A retrospective cohort study. J ASEAN Fed Endocr Soc, 2018; 33(2):124–129.

26.

, Broome

, Bena

, et al. Predictors for adverse outcomes in diabetic ketoacidosis in a multihospital health system. Endocr Pract, 2020; 26(3):259–266.

27.

, Lv

, Chen

, et al. Early prediction models for prognosis of diabetic ketoacidosis in the emergency department: A protocol for systematic review and meta-analysis. Medicine, 2021; 100(21):e26113.

28.

, Lee

, Zhou

, et al. Performance assessment of different machine learning approaches in predicting diabetic ketoacidosis in adults with type 1 diabetes using electronic health records data. Pharmacoepidemiol Drug Saf, 2021; 30(5):610–618.

29.

Brown

, Kovatchev

, Raghinaru

, et al. Six-month randomized, multicenter trial of closed-loop control in type 1 diabetes. N Engl J Med, 2019; 381(18):1707–1717.

30.

Moons

KGM

, Altman

, Reitsma

, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): Explanation and elaboration. Ann Intern Med, 2015; 162(1):W1–W73.

31.

Lantz

Machine Learning with R. Packt Publishing: Birmingham; 2013.

32.

Duda

, Hart

, Stork

. Pattern Classification, 2nd ed. John Wiley & Sons, Inc.: New York, NY, USA; 2001.

33.

Brownlee

. Machine Learning Mastery with R. 2016. Available from: https://machinelearningmastery.com/machine-learning-with-r [Last accessed: August 28, 2023 ].

34.

Dhatariya

. Blood ketones: Measurement, interpretation, limitations, and utility in the management of diabetic ketoacidosis. Rev Diabet Stud [Internet], 2016; 13(4):217. Available from: /pmc/articles/PMC5734222/ [Last accessed: August 15, 2023].

35.

Kohavi

. A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection [Internet]. 1995. Available from: https://robotics.stanford.edu/∼ronnyk [Last accessed: August 15, 2023 ].

36.

Breiman

. Random forests. Mach Learn [Internet] 2001; 45(1):5–32. Available from: https://link-springer-com-443.web.bisu.edu.cn/article/10.1023/A:1010933404324 [Last accessed: August 15, 2023 ].

37.

Yang

, Shami

. On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing, 2020; 415:295–316.

38.

Jiao

, Du

. REVIEW Performance measures in evaluating machine learning based bioinformatics predictors for classifications. Available from: https://www.genome.gov/sequencing- [Last accessed: August 15, 2023 ].

39.

Zweig

, Campbell

. Receiver-operating characteristic (ROC) plots: A fundamental evaluation tool in clinical medicine. >Clin Chem [Internet] 1993; 39(4):561–577. Available from: https://academic-oup-com-s.web.bisu.edu.cn/clinchem/article/39/4/561/5646806 [Last accessed: August 15, 2023 ].

40.

DiCiccio

, Efron

. Bootstrap confidence intervals. 1996; 11(3):189–228; doi: 101214/ss/1032280214. Available from: https://projecteuclid.org/journals/statistical-science/volume-11/issue-3/Bootstrap-confidence-intervals/10.1214/ss/1032280214.full [Last accessed: August 15, 2023 ].

41.

Shivers

, Mackowiak

, Anhalt

D.o H

, et al. “Turn it off!”: Diabetes device alarm fatigue considerations for the present and the future. J Diabetes Sci Technol [Internet] 2013; 7(3):789–794. Available from: https://journals-sagepub-com-s.web.bisu.edu.cn/doi/abs/10.1177/193229681300700324 [Last accessed: August 15, 2023 ].

42.

Suh

, Choi

, Oh

, et al. Association between early glycemic management and diabetes complications in type 1 diabetes mellitus: A retrospective cohort study. Prim Care Diabetes [Internet], 2023; 17(1):60–67. Available from: https://pubmed.ncbi.nlm.nih.gov/36609067/

43.

Beck

, Riddlesworth

, Ruedy

, et al. Effect of continuous glucose monitoring on glycemic control in adults with type 1 diabetes using insulin injections: The DIAMOND randomized clinical trial. JAMA [Internet] 2017; 317(4):371–378. Available from: https://jamanetwork.com/journals/jama/fullarticle/2598770 [Last accessed: August 15, 2023 ].

44.

Tauschmann

, Hermann

, Freiberg

, et al. Reduction in diabetic ketoacidosis and severe hypoglycemia in pediatric type 1 diabetes during the first year of continuous glucose monitoring: A multicenter analysis of 3,553 subjects from the DPV registry. Diabetes Care [Internet] 2020; 43(3):e40–e42. Available from: https://diabetesjournals.org/care/article/43/3/e40/35632/Reduction-in-Diabetic-Ketoacidosis-and-Severe [Last accessed: August 15, 2023 ].

45.

Bergenstal

, Kerr

MSD

, Roberts

, et al. Flash CGM is associated with reduced diabetes events and hospitalizations in insulin-treated type 2 diabetes. J Endocr Soc [Internet] 2021;5(4):1–9. Available from: https://academic-oup-com-s.web.bisu.edu.cn/jes/article/5/4/bvab013/6126709 [Last accessed: August 15, 2023 ].

46.

Battelino

, Danne

, Bergenstal

, et al. Clinical targets for continuous glucose monitoring data interpretation: Recommendations from the international consensus on time in range. Diabetes Care [Internet] 2019;42(8):1593–1603. Available from: https://diabetesjournals.org/care/article/42/8/1593/36184/Clinical-Targets-for-Continuous-Glucose-Monitoring [Last accessed: August 15, 2023 ].

47.

Van Name

, Hilliard

, Boyle

, et al. Nighttime is the worst time: PARENTAL fear of hypoglycemia in young children with type 1 diabetes (T1D). Pediatr Diabetes [Internet], 2018; 19(1):114. Available from: /pmc/articles/PMC5650950/ [Last accessed: August 15, 2023].

48.

Marker

, Monzon

, Nelson

, et al. An intervention to reduce hypoglycemia fear in parents of young kids with type 1 diabetes through video-based telemedicine (REDCHiP): Trial design, feasibility, and acceptability. Diabetes Technol Ther [Internet] 2020; 22(1):25–33. Available from: https://pubmed.ncbi.nlm.nih.gov/31517512/ [Last accessed: August 15, 2023 ].

49.

Laffel

. Ketone bodies: a review of physiology, pathophysiology and application of monitoring to diabetes. Diabetes Metab Res Rev, 1999 Nov–Dec;15(6):412–26. doi: 10.1002/(sici)1520-7560(199911/12)15:6<412::aid-dmrr72>3.0.co;2-8. PMID: 10634967.

50.

Umpierrez

, Kitabchi

. Diabetic ketoacidosis: Risk factors and management strategies. Treat Endocrinol [Internet] 2003; 2(2):95–108. Available from: https://link-springer-com-443.web.bisu.edu.cn/article/10.2165/00024677-200302020-00003 [Last accessed: August 15, 2023 ].

51.

Hassell Sweatman

CZW

. Mathematical model of diabetes and lipid metabolism linked to diet, leptin sensitivity, insulin sensitivity and VLDLTG clearance predicts paths to health and type II diabetes. J Theor Biol [Internet] 2020;486. Available from: https://pubmed.ncbi.nlm.nih.gov/31626814/ [Last accessed: August 15, 2023 ].

52.

Cichosz

, Jensen

, Hejlesen

. Short-term prediction of future continuous glucose monitoring readings in type 1 diabetes: Development and validation of a neural network regression model. Int J Med Inform, 2021; 151:104472.

53.

Cichosz

, Kronborg

, Jensen

, et al. Penalty weighted glucose prediction models could lead to better clinically usage. Comput Biol Med, 2021; 138:104865.