Machine-Learning-Based Prediction of a Missed Scheduled Clinical Appointment by Patients With Diabetes

Nomenclature

“Factors,” which denotes patient characteristics (age, sex, etc.) and phenomena that can affect an MA (how and when an appointment was made, etc.), are called “features” when part of our mathematical models.

Model Design

We designed a logistic regression model that predicts a class y ∈ { + 1 , − 1 } from a feature vector x of a patient’s target appointment, with y representing a clinical appointment missed ( y = + 1 ) or kept ( y = − 1 ) and x generated from the patient’s EHR, representing the time from the initial visit to just before the target appointment. We modeled the probability of an appointment with x attributed to y with a logistic regression, ( y | x ; w ) = 1 1 + e x p ( − y w ⋅ x ) , where w is a weight vector. The w ⋅ x denotes the scalar product of two vectors, w and x . We computed the w estimate by using training data { ( x 1 , y 1 ) ,…, ( x n , y n ) ,…, ( x N , y N )}.

In “Feature design” below, observed values of each quantitative variable, for example, blood test results, were linearly transformed (normalized) to make a variance of each variable = 1 and assigned to vector x .

Modeling With L2-Norm Regularization

When the training data size, N, is smaller than the dimension of feature vectors, or a training data sampling is biased, maximum-likelihood estimation often overfits a logistic regression model into the training data, leading the model to classify many new appointments inaccurately. ⁶ We used a machine learning software, Classias, ⁷ which employs an L2-norm regularization method to mitigate overfitting and improve the model’s generalizability. With the regularization, we estimated w as w ^ = a r g m a x w { ∑ n = 1 N log P ( y n | x n ; w ) − λ ‖ w ‖ 2 2 } , where the squared L2-norm of w, ‖ w ‖ 2 2 , is an L2-norm regularizer, acting as a mitigating penalty to provide large absolute weight values only to features occurring frequently in training data. The symbol λ is a hyperparameter for regularization and needs to be tuned. Note that L2-norm regularization reduces to the maximum-likelihood estimation when λ = 0 .

Evaluation Method

The receiver operating characteristic (ROC) curve, area under the curve (AUC), precision, recall, and F-measure were used to measure prediction performance.

We used 10-fold cross-validation to assess how accurately the model performed for unknown data. Cross-validation partitions the whole data into 10 subsets and repeats the performance measure 10 times. Each time, one of the 10 subsets is used as the test data, the other subsets as the training data. The model was trained by using the training data and then was tested by using the test data. The final score was the average performance of all 10 performance measures. For each test, we varied the hyper parameter λ of the regularization, as explained above, from 0.001 to 100, calculated the model’s measures for each λ by using another 10-fold cross-validation on only the training data, and set the best one to λ .

To determine each feature’s contribution to prediction in the model, we examined the absolute values of each feature’s weight learned by using the whole data. Using 10-fold cross-validation on the whole data, we set λ at 0.5 for the highest AUC value. Note that each feature’s contribution was measured by its weight in the regularized model, not necessarily representing its importance as a risk factor. Its characteristic was different from the weight and odds measured by the maximum-likelihood estimation.

Feature Design

Features from a patient’s EHR were designed to fully cover factors related to treatment discontinuation, one set related to the patient’s clinical condition (X1, n = 29 025), the other to previous findings (X2, n = 97).

Table 1 shows examples of X1. To classify clinical conditions chronologically, we generated two features for each, one fulfilled within 90 days before the target appointment and the other without limit. The X1 group has numerous features but most with a zero value, hence a very sparse construction.

OPEN IN VIEWER

Table 1.

Examples of X1.

Category	Feature
Consultation	Past/recent consulted medical departments
Disease	Past/recent diagnosed diseases, recovered from/under treatment
Medicine	Past/recent prescribed medicines
	Number of doses taken per day

Table 2 lists examples of X2, most of whose features are quantitative variables. In designing X2, we referred to previous research such as examinations of discontinued treatments, interviews with specialists and patients, and human behavior analysis. Since previous research had identified sex, age, distance from house to hospital and travel time, and laboratory results as factors related to treatment discontinuation,^2-5 we used them, too. As an example of our design technique, because a diabetes specialist told us that for patients who often forget their medication, the specialist monitored compliance by comparing the intervals between scheduled clinical appointments, length of prescription term, and amount of medication the patient still had, we incorporated this calculation as a check on compliance. Because a diabetic patient told us that the very accumulation of experiences with medical treatment in general stimulated his motivation to continue diabetes treatments, we used the history of clinic visits and number of diagnosed diseases. Because human behavior analysis uses behavior patterns and external factors, we used the day of the week an appointment was scheduled. To classify behavior patterns by medical department, one feature included all departments, the other only the diabetes unit.

OPEN IN VIEWER

Table 2.

Examples of X2.

Category	Feature
Property	Sex, age
	Distance and time-distance from the house to the hospital by public transport
Consultation	Frequency of clinic visits
	Probability of visit on a given day of the week
Appointment	Interval between the date on which a clinical appointment was made and scheduled appointment date
	Day of the week when an appointment was made
	Whether the patient kept an appointment on the day it was made or missed the appointment without notice
	Weather on the appointment day
Medicine	Length of the prescription term of the latest medicine
	Total amount of medication a day
	How many times a day medication is taken
	Maximum size of prescribed tablets
Laboratory result	HbA1c, HDL-C, LDL-C, TG, and T-Cho

Class Design

Scheduled clinical diabetes appointments were classified as to whether they were kept or missed. The missed appointments (MAs) had been made but were missed on the scheduled day. The kept appointments (KAs) were kept on the scheduled day. Appointments—not patients—were analyzed because a patient who had missed an appointment at some point might have kept others, so if we had analyzed patients, we would have been unable to distinguish the status of “kept” or “missed” for that patient.

Appointments the patient changed to another day (and kept) were obviously eliminated from the MAs. Also, if a patient missed an appointment but had an appointment in the diabetes department for a later day—and kept it—the first appointment was removed from the MAs.

A few inaccuracies unavoidably tainted the MAs. For example, if patients changed clinics without notice but continued treatment or severe weather or traffic accidents prevented appointments from being kept, such cases do not involve “treatment discontinuation,” so they should be removed from the MAs, but they are so hard to detect in the EHR that they ended up included.

Data Set

All prediction experiments were performed by using records from the University of Tokyo Hospital that included the history of 16 026 clinical appointments scheduled by 879 patients whose initial clinical visit had been made after January 1, 2004, who had diagnostic codes indicative of diabetes, and whose HbA1c had been tested within 3 months after their initial visit. The records were dated between April 1, 2011 and June 30, 2014. The 879 patients kept 15 104 of the 16 026 appointments scheduled, and the average number of appointments per patient was 17.4 with a standard deviation of 11.0. Hence, 922 appointments were missed by 285 patients, the average number per patient was 3.2, and the standard deviation was 3.9. Of the 285 patients, 207 had fewer MAs than the average (3.2), and their MAs accounted for about 40% of the total.

Prediction Performance

We examined the difference in prediction performance between models by using the X1, X2, and X1+X2 groups. Figure 1 and Tables 3 and 4 show the prediction performance. The X2-group model (AUC = 0.943) predicted the MAs with higher accuracy than did the X1 group (AUC = 0.912), and using both the X1 and X2 groups (AUC = 0.958) yielded even greater accuracy. The precision, recall, and F-measure of the model when using both X1 and X2 groups were, respectively, 0.757, 0.659, and 0.704. For reference, Tables 3 and 4 show, respectively, the AUC value and F-measure of the models learned by maximum-likelihood estimation ( λ = 0 ). The model using both X1 and X2 groups improved both the AUC value and F-measure by L2-norm regularization.

OPEN IN VIEWER

Figure 1.

ROC curve that measures the prediction performance of the logistic regression model with L2-norm regularization by using the X1, X2, and X1+X2 groups. The axes of the ROC curve consist of 1-specificity (TN/(FP+TN)) and sensitivity (TP/(TP+FN)), where TN = true negative, FP = false positive, TP = true positive, and FN = false negative.

OPEN IN VIEWER

Table 3.

AUC Value.

Feature	L2-norm regularization	Maximum-likelihood estimation
X1	0.912	0.902
X2	0.943	0.945
X1+X2	0.958	0.917

OPEN IN VIEWER

Table 4.

Precision, Recall, and F-measure.

	L2-norm regularization			Maximum-likelihood estimation
Features	Precision	Recall	F-measure	F-measure
X1	0.466	0.438	0.449	0.441
X2	0.750	0.577	0.652	0.650
X1+X2	0.757	0.659	0.704	0.587

Figures 2 and 3 show AUC and F-measure scores obtained with 10-fold cross-validation of the whole data when λ was fixed to a value between 0.001 and 100 for training models. The models using the X1 group and both X1 and X2 groups reached the largest AUC and F-measure scores with λ between 0.1 and 1. This result indicates that L2-norm regularization can improve prediction performance when an appropriate λ is selected.

OPEN IN VIEWER

Figure 2.

AUC scores obtained with 10-fold cross-validation of the whole data when λ was fixed to a value between 0.001 and 100 for training models. The AUC is the area under the ROC curve. The models reached the largest AUC score with λ between 0.1 and 1.

OPEN IN VIEWER

Figure 3.

F-measure scores obtained with 10-fold cross-validation of the whole data when λ was fixed to a value between 0.001 and 100 for training models. F-measure is the harmonic mean of precision and recall, where precision = (TP/(TP+FP)) and recall = (TP/(TP+FN)). The models reached the largest F-measure score with λ between 0.1 and 1.

Prediction Model’s Performance and Future

Our model performed well in predicting MAs, with an AUC at 0.958—an AUC value over 0.9 is considered to be an excellent prediction score. ¹¹ This promises to make the model a powerful tool for showing clinicians the suitable timing for intervening to keep patients continuing hospital visits.

Of course, earlier prediction of MAs is required to prevent discontinuation of treatment. We will need to improve our current model by adding new features such as changes of diabetes treatment over time and progression of diabetes. Furthermore, the quality of interaction perceived by the patient may be another factor determining whether the patient returned; we plan to evaluate this in a future model. If we can predict MAs early enough for clinicians to encourage diabetic patients to continue treatment, the number of MAs will decrease significantly, and diabetes care will be also improved.

One of our authors found that “DialBetics,” the telemedicine diabetes self-management system that can automatically give advice on the basis of a diabetic patient’s input and registry data, led patients to improve their dietary habits.^12,13 We will combine DialBetics with our model and investigate whether coordination of clinical appointments with our model more effectively motivates patients to keep clinical appointments.

General Applicability

Our model was evaluated with outpatient EHR from one hospital in Japan. To assess its general applicability, we must evaluate the model with patients at other hospitals in several different countries that have different demographics and regional characteristics. We will also evaluate our model with patients who suffer from other diseases, like asthma and other chronic diseases, that require similarly continual outpatient treatment, and we will investigate the differences in features.

Design of Features in the Model

The features we deemed possibly relevant to MAs (X2 group) rendered a higher prediction accuracy compared with those related to patients’ clinical condition (X1 group). This suggests that features derived from knowledge about and experience with the diabetes treatment of clinicians is more effective for improving accuracy in predicting MAs. Nevertheless, X1 is also valuable since use of both X1 and X2 resulted in a higher prediction accuracy. Therefore, in a future study, we will identify the most contributive features in X1 to construct a simpler model with fewer features.

Machine Learning of Features’ Weight

The highest prediction accuracy was obtained by the model constructed by machine learning without maximum-likelihood estimation but with L2 regularization using only the X1 group or both X1 and X2 groups. Since the number of features in these groups exceeds that of the training data, it carries a high risk of overtraining a part of the training data. We confirmed that L2 regularization reduced over-fitting well. The accuracy of the model that learned with L2 regularization was almost the same as that of the model constructed with only the X2 group using maximum-likelihood estimation because overfitting was effectively suppressed by regularization reducing the number of features to under 1% of the whole training data.

The appointment records we used were possibly uneven because of records obtained from patients who made many appointments. We will consider the relationship between prediction accuracy and the uniformity of appointment records.

Machine learning is generally applied to “bigger” data than we used in this study (16 026). ¹⁴ The prediction accuracy will be improved by using more data because machine-learning generalization increases as the number of training data increases. ¹⁵ SS-MIX2, which is currently being developed to standardize medical data stored in the EHR systems of hospitals, will enable us to use “big data” and improve the prediction accuracy of our model in the near future.^16,17

How Features Contribute to MA Predictions

Tables 5 and 6 show the top five qualitative and quantitative features, respectively. These features were given the largest absolute weights in our trained model and thus contributed strongly to MA prediction. We found that features related to when and how appointments were made, rather than to patients’ clinical condition, influenced the accuracy in predicting MAs. It would seem that irregularly scheduled appointments are more apt to be missed. We examined the largest qualitative/quantitative features in detail.

OPEN IN VIEWER

Table 5.

Qualitative Features That Contribute to the Prediction.

Qualitative features	Weight
Appointment was made on a Sunday	2.22
Appointment was scheduled for a Friday	1.91
Graves’ disease was diagnosed and recovered from	−1.36
Diabetic ketoacidosis with type 2 diabetes diagnosed and under treatment	1.32
Rilmazafone Hydrochloride Hydrate recently prescribed	1.29

OPEN IN VIEWER

Table 6.

Quantitative Features That Contribute to the Prediction.

Quantitative features	Weight
Frequency of visiting the clinic’s diabetes department in the past	−1.20
Number of medical departments consulted in the past	−0.999
Probability of the previous kept appointment was a Friday	0.965
Interval between when a clinical appointment was made and the date for which it was scheduled	−0.912
Probability of the previous appointment was made on a Monday	0.897

The largest qualitative feature was “Appointment made on a Sunday.” In all, 164 (about 1%) of total appointments were made on Sundays; 73 (44.5%) of those appointments were MAs. The percentage of MAs in the appointments made on Sunday was much higher than that of the total appointments (5.75%). Physicians do not usually make appointments on Sundays, so consultations on Sunday are very unlikely except for visits to the emergency room. This means that those 164 Sunday appointments were made for the patients who needed to have a follow-up visit and be examined by the physicians who examined them in the emergency room. It seems that some patients came on a Sunday because they had used up their medications: the prescription for their latest medicine of 39 of the 73 MAs was expired. Another possibility is that appointments made on Sundays, and appointments made only 8 days after a previous one, suggest a sicker patient or more complex condition. The reason the rest of the appointments were missed is not clear. Further investigation is needed to clarify why the appointments made on Sundays are more likely to be missed.

The qualitative feature “Appointment scheduled for a Friday” was second largest: 493 (17.4%) of 2831 appointments scheduled for a Friday were MAs. We examined the previous appointments of the KAs scheduled for a Friday and compared them with the previous appointments of MAs. Generally, the previous appointments of the KAs were likewise scheduled on Fridays, suggesting that KA patients incorporated visits to the hospital for diabetes treatment into their life as a matter of routine. In contrast, the previous appointments of MAs were less likely to have been scheduled on a Friday, suggesting that Friday was not their regular hospital-visit day. This suggests that the same day of the week as the patient’s previous hospital visits should be considered when making an appointment.

Last, we examined appointment intervals, which is the fourth largest quantitative feature (the third largest involved diabetes history). The average interval in the case of KAs was 45 days, while that in the case of MAs was only 8 days. Next appointments in the case of KAs are usually made on the day of patients’ hospital visits. Of course, the interval between appointments is dictated by a patient’s health status; but, in practice, it generally falls between one and two months. As expected, most MAs were made on a day when a patient did not visit the hospital according to our examination. This suggests that appointments should be made on days when patients have a hospital visit.

We will analyze other of the largest features related to clinical condition. For example, the feature, “Rilmazafone Hydrochloride Hydrate recently prescribed,” may indicate that those patients had a sleep disorder, which might suggest a useful hypothesis about why appointments are missed.