Reduction of Number and Duration of Hypoglycemic Events by Glucose Prediction Methods: A Proof-of-Concept In Silico Study

Abstract

Background:

Hypoglycemia prevention is one of the major challenges in diabetes research. Recently, it has been suggested that continuous glucose monitoring (CGM)-based short-term glucose prediction algorithms could be exploited to generate alerts when hypoglycemia is forecasted, allowing the patient to take appropriate countermeasures to avoid/mitigate the event. However, quantifying the potential benefits of prediction in terms of reduction of number/duration of hypoglycemia requires an in silico assessment that is the object of the present article.

Materials and Methods:

Data for 50 virtual subjects were generated by using the University of Virginia/Padova type 1 diabetes simulator (54-h monitoring), made more credible by adding realistic measurement noise and perturbations of meals and insulin injections. CGM was assumed to be well calibrated. Occurrence and duration of hypoglycemic events were compared in three scenarios: (1) hypoglycemia was not recognized and not dealt with; (2) 15 g of carbohydrates was ingested when CGM crossed the hypoglycemia threshold; or (3) 15 g of carbohydrates was ingested when the 30-min ahead-of-time CGM prediction crossed the hypoglycemia threshold. The effectiveness of alerts was investigated also in the case of delayed/absent ingestion of carbohydrates.

Results:

In Scenario 1, each virtual subject spent 17.7% of the time in the hypoglycemic range, with a median of four events of 120 min in the 54-h period monitored. In Scenario 2, the time spent in hypoglycemia was reduced to 4.7% (four events of 40 min). In Scenario 3, the time spent in hypoglycemia was further reduced to 1.2% (one event of 15 min). Absent/delayed patient's responses to alerts slightly increase these percentages, but improvements remain significant.

Conclusions:

This in silico proof-of-concept study demonstrates that using predicted rather than measured CGM allows a significant reduction of the number of hypoglycemic events and the time spent in hypoglycemic range both by 75%, stimulating further research and clinical investigation on the generation of preventive hypoglycemic alerts exploiting glucose prediction methods.

Introduction

O ne of the major issues in diabetes management is to limit hypoglycemic and hyperglycemic events. Hyperglycemia causes long-term invalidating pathologies, whereas hypoglycemia has threatening short-term consequences because, if not quickly detected, it could lead to coma and even to death. Continuous glucose monitoring (CGM) devices can improve diabetes management by generating suitable visual/acoustic alarms in real time when measured glycemia crosses critical thresholds (e.g., 70 mg/dL for hypoglycemia).¹ Preventing rather than simply detecting critical events when they have already occurred would be preferable,^2,3 and, to do so, methods for glucose prediction in the short term (30–45 min) were solicited. In the last decade several prediction methods have been proposed in the literature. Making a comprehensive analysis of them is outside the scope of the present article, and we refer the reader to Sparacino et al.⁴ and Bequette⁵ for reviews, but it is useful to cite, as representative samples of various classes of prediction techniques, autoregressive time-series models,^6

–9 neural networks,^10
–12 kernel-based predictors,¹³ or even combination of multiple models.¹⁴ It has also been suggested that prediction methods may be also useful in compensating the time lag due to both blood-to-interstitium glucose transport and sensor technology.¹⁵

In the literature, the benefit deriving from prediction methods to prevent/mitigate hypoglycemia by soliciting appropriate treatments (e.g., sugar intake and/or pump basal suspension) has been assessed from real data.^14,16

–19 However, as better explained in Hypoglycemia Reduction Through Prediction-Based Preventive Treatment: State of the Art, it would be of interest to compare different scenarios occurring for the same patient and starting from the very same patient conditions, which is not possible in clinical studies, where every action has an effect on glycemia and, unavoidably, excludes the possibility of seeing what would have happened if different decisions were made. Thus, the aim of this article is to use an in silico environment to quantify the potential benefits, in terms of number and duration of hypoglycemia events, coming from the use of predicted, rather than measured, glucose, in 50 synthetic subjects. Virtual patients were created by the University of Virginia/Padova type 1 diabetes simulator,^20,21 which is approved by the Food and Drug Administration as a potential route to clinical testing as a substitute for trial on animals. The synthetic patients were virtually monitored in horizons of 54 h (including two lunch, two dinner, and three breakfast events per patient), in the presence of additive white noise with realistic variance corrupting CGM data and of sources of uncertainty on the quantity of ingested carbohydrates (CHOs) and injected insulin. Three parallel scenarios were considered: (1) the subject was unaware of hypoglycemia, no alert was generated, and no countermeasures were taken when the blood glucose concentration fell below the hypoglycemic threshold (worst case); (2) a hypoglycemia alarm was triggered based on CGM measurements and 15 g of CHO was ingested by the patient; and (3) a hypoglycemic alert was given on the basis of the 30-min ahead-of-time predicted glycemia, obtained via a recently published neural network algorithm,¹² and 15 g of CHO was ingested by the patient, as in Scenario 2. Results demonstrate that under Scenario 3 the number of hypoglycemic events and the time spent in hypoglycemic range are reduced by 75% and 93%, respectively, relative to Scenario 1. Even more remarkable is that comparison of Scenario 2 and Scenario 3 shows that generating alerts on the basis of predicted rather than measured CGM leads to a reduction of both number and duration of hypoglycemic events of 75%. A significant reduction of number and duration of hypoglycemic episodes (25% and 59%, respectively) is observed even in the presence of delayed/absent ingestion of CHO after hypoglycemia alerts.

Hypoglycemia Reduction Through Prediction-Based Preventive Treatment: State of the Art

The aim of this section is to illustrate some results recently obtained in the literature on the possibility of avoiding/mitigating hypoglycemic events by treating them in advance, on the basis of glucose values predicted ahead in time.

In the work of Buckingham et al.,^16,17 nocturnal hypoglycemia was induced in 15 hospitalized subjects by increasing basal insulin infusion. Five different literature glucose prediction techniques (all solely based on past glycemic data measured by CGM) were simultaneously used to forecast hypoglycemia, and hypoglycemia alerts were generated on the basis of a predefined voting scheme. The triggering of a hypoglycemic alarm involved the suspension of basal insulin infusion, until recovering of blood glucose concentration to a safe value, either for a maximum of 90 min. This strategy allowed preventing the majority of the nocturnal induced hypoglycemia crisis.

A retrospective study, dealing with prediction of nocturnal hypoglycemia in children with type 1 diabetes, was conducted by Dassau et al.¹⁴ Following a protocol similar to that of Buckingham et al.,^16,17 low glucose concentration was induced by gradual increases in basal insulin infusion rate. Five different literature glucose prediction algorithms were individually run, and a hypoglycemia alarm was triggered if the cumulative number of alerts generated was above a preset voting threshold. The method was assessed, retrospectively, on the basis of the number of hypoglycemia events correctly forecasted, by evaluating and comparing the performance obtained with various prediction horizons, different voting thresholds, and different hypoglycemic thresholds. Harvey et al.¹⁹ proposed a method to compare hypoglycemia prediction algorithms and to optimally tune their parameters for different applications, as pump suspension or alerting for rescue CHO ingestion. Eren-Oruklu et al.^8,18 proposed prediction algorithms based on autoregressive moving average models exploiting past CGM readings solely and also evaluated three methods for generating hypoglycemic preventive alerts on data collected in type 2 diabetes subjects.⁹

From the above studies it emerges that glucose prediction methods allow prevention, or at least mitigation, of hypoglycemic events. However, all the studies above are done in vivo/on real data. A more comprehensive quantitative assessment would require comparison of different parallel scenarios for the same patient. In clinical studies, once an action on a patient is taken (e.g., administration of sugar at time X), there is no way of coming back to time X and evaluating what would have happened if different actions were adopted (e.g., administration of sugar at time X+Y or no administration at all). The simulation environment allows overcoming this limitation because, for the same subject and the same hypoglycemic event, we can run different simulations (corresponding to parallel alternative and mutually exclusive scenarios) and compare the effect of different actions, starting from the very same initial conditions. To clarify this concept, in Figure 1 we show a conceptual example obtained by creating, through the University of Virginia/Padova type 1 diabetes simulator,²⁰ a subject (#1) experiencing a nocturnal hypoglycemic situation. Given the purely conceptual purpose of this specific example, noise-free data are considered at the moment. At time 02:00, glucose concentration is 120 mg/dL and exhibits a descending trend (continuous black line), which ultimately leads to a hypoglycemia event that could be revealed at time 04:00 (blue alarm bell) when glucose concentration falls below the 70 mg/dL threshold (thin horizontal dashed black line). A short-term glucose prediction can be computed in real time, and in Figure 1 we report the 30-min ahead-of-time glucose prediction profile (continuous magenta line) until time 03:30 (for easy reading), when it falls below the 70 mg/dL threshold and a hypoglycemia alert may be generated (red alarm bell). The simulation environment allows us to consider three scenarios for the hypoglycemia event occurring in this virtual patient. In Scenario 1, no hypoglycemia alert is generated, and the subject is unaware of hypoglycemia; after threshold crossing at time 04:00, the glucose profile takes the shape of the green dotted line. The virtual subject has breakfast at 08:00 (black arrow) and returns within the euglycemic range only at time 08:20, after having spent 260 min (more than 4 h) in the hypoglycemic range, reaching a glycemic concentration as low as 54 mg/dL. In Scenario 2, a bolus of 15 g of CHO (blue arrow) is given at 04:00 (blue alarm bell) with the glucose profile shown by the blue dotted line. It is notable that this countermeasure allows re-entering the euglycemic range at 04:40, thus reducing the time spent below the hypoglycemic threshold to 40 min (horizontal blue line), with a lowest glucose concentration of 66 mg/dL. In Scenario 3, the 15-g CHO bolus for hypoglycemia treatment is administered immediately after the generation of the alert at 03:30 (red alarm bell) with the glucose profile shown by the red dotted line. It is notable that in this case the hypoglycemic event is totally avoided.

FIG. 1.

Virtual subject #1, noise-free data, showing the three scenarios of glucose concentration behavior that may occur in the presence of a nocturnal hypoglycemia (hypo) event (the thin horizontal dashed black line denotes the 70 mg/dL hypo threshold, the continuous black line represents glucose concentration until 04:00, the continuous magenta line identifies the 30-min ahead-of-time glucose prediction [pred] until time 03:30, and the black arrow indicates breakfast time at 08:00): Scenario 1 (green), no hypo alert (green dotted line); Scenario 2 (blue), alarm (blue bell) with 15 g of carbohydrate (CHO) bolus (blue arrow) at 04:00 and subsequent glucose concentration (blue dotted line); and Scenario 3 (red), alarm (red bell) with 15 g of CHO bolus (red arrow) at time 03:30 and subsequent glucose concentration (red dotted line). CGM, continuous glucose monitoring. Color image available online at www.liebertonline.com/dia

This conceptual example shows the potential of generating alerts on the basis of predicted glucose. To the best of our knowledge, a comprehensive simulation study to assess the benefits, in terms of number and duration of occurred hypoglycemia events, of ingesting a small amount of CHO in response to hypoglycemia alerts generated on the basis of measured versus predicted glucose profiles has not been performed. To the best of our knowledge, the only study published so far that adopts simulation strategies in a similar context is that of Hughes et al.,²² where the authors presented a method to detect/predict the risk of hypoglycemia and to perform a gradual attenuation of insulin delivery on the basis of risk factors instead of an instantaneous pump shutoff and assessed it through the University of Virginia/Padova type 1 diabetes simulator.²⁰ Their results indicate that attenuating insulin delivery reduces, or at least delays, the onset of hypoglycemia, especially if rescue CHO is given sufficiently ahead in time. However, differently from the open-loop control framework considered in the present study, the algorithm of Hughes et al.²² was thought to work in a closed-loop artificial pancreas framework, and the amount of insulin released by the pump could be modified at each time step.

Materials and Methods

In silico database

The database consists of 50 type 1 diabetes virtual patients, extracted from the University of Virginia/Padova type 1 diabetes simulator,^20,21 without reference to any particular cohorts of the population. For each subject, one CGM time series has been simulated, with sampling time of 5 min. The choice of this specific sampling time is due to the fact that it coincides with that of two of the most used CGM devices: the Seven^® Plus (DexCom Inc., San Diego, CA) and the Guardian^® Real-time (Medtronic Inc., Northridge CA). Each simulated time series consists of 54 h of monitoring, from 03:00 of Day 1 to 09:00 of Day 3. The monitoring interval was chosen in order to be long enough to observe at least one hypoglycemic event for each subject. Moreover, because breakfast is administered between 06:00 and 08:00, termination of the monitoring interval at 09:00 allows patients to complete the recovery from an eventual nocturnal hypoglycemia. Three meals per day were considered in the simulated scenario. To render the profiles more realistic, CHO intake quantities and meal timings were differentiated from meal to meal and from day to day. Breakfast was randomly located in the time interval 06:00–08:00 and with CHO randomly chosen between 35 and 55 g, lunch was in the interval 12:00–14:00 with 60–90 g of CHO, and finally dinner was in the interval 19:00–21:00 with 70–100 g of CHO. For what concerns insulin, a basal-bolus infusion scheme was adopted, with boluses computed according to the CHO content of the meal. To obtain additional hypoglycemic events in the simulated profiles, overdosage of insulin was considered. In particular, everyday basal insulin was increased twice for 30 min of a random amount sampled from a uniform distribution in 0–3 U/h (this action has also an effect on glucose similar to an increase of insulin sensitivity or to a mild physical activity). Furthermore, for half of the patients randomly chosen, one insulin bolus was augmented once a day of a random percentage sampled from a uniform distribution in 0–30%. For the other half of the patients, the size of one of the meals was simulated to be wrongly estimated, and the amount of CHO effectively ingested was decreased by a percentage randomly chosen in the interval 0–30%. Finally, in order to mimic the random measurement error affecting CGM, a white noise sequence whose samples were extracted from a Gaussian distribution with zero mean, as in Chase et al.,²³ and variance equal to 4, in line with Guerra et al.²⁴ and Facchinetti et al.,²⁵ have been added to each time series. Simulations have been drawn without considering any sensor drift/colored noise because, to the best of our knowledge, no well-established models to describe these sources of sensor inaccuracy are available (see, for example, the discussion of Facchinetti et al.²⁶). However, this simplification does not limit the validity of the results. In fact, sensor drift would affect alerts generated on the basis of prediction, but also, in a very similar way, alerts generated on the basis of CGM (in other words, deterioration of results would occur in both Scenario 2 and Scenario 3, but the relative difference between the two scenarios, which is the key object of the present investigation, would not significantly change). It is also worthwhile recalling that, in a real world environment, enhancement/calibration methods^24,27
–29 could be used to mitigate the effects of sensor drift by exploiting sparse reference self-monitoring of blood glucose samples (see also Limitations of the adopted in silico environment).

To quantify the benefits of prediction-based hypoglycemia alerts, we compared the three scenarios already illustrated in Figure 1. In Scenario 1, hypoglycemia is not recognized and dealt with. This corresponds to a sort of worst-case situation for the diabetes patient, although possible especially during the night.³⁰ In Scenario 2 the alert is triggered on the basis of the measured CGM readings. In Scenario 3 the alarm is generated on the basis of predicted glycemia, obtained through the algorithm summarized in The neural network predictor and described by Zecchin et al.¹² Alert generation obeyed the simple strategy explained in The hypoglycemic alert generation strategy. In both Scenarios 2 and 3, a bolus of 15 g CHO was ingested in the 5 min following the alert. Scenario 2 and Scenario 3 were assessed also in the presence of randomly delayed/absent ingestion of CHO. Results are quantified in terms of number of hypoglycemic events, their duration, and total time in the hypoglycemic range. In addition, we computed also the distribution of glucose concentration and the Low and High Blood Glucose Indexes (LBGI and HBGI, respectively),³¹ two commonly adopted indicators of the risk of hypoglycemia and hyperglycemia, respectively. The higher the value of these indexes, the higher the associated risk. For example, in Figure 1 LBGI results equal to 14.5, 4.5, and 3.6 in Scenario 1 (green dotted line), Scenario 2 (blue dotted line), and Scenario 3 (red dotted line), respectively. The nonparametric Mann–Whitney U test was used to compute P values to assess the significance of differences among the scenarios considered.

Remark 1

In the spirit of keeping the protocol as simple as possible, the action associated with hypoglycemia alert was standardized to the ingestion of 15 g of CHO. According to Briscoe and Davis,³² such a measure is commonly adopted by diabetes patients and has the effect of raising glycemia of about 50 mg/dL in approximately 15 min. In addition, basal insulin infusion was neither suspended nor attenuated (differently from Buckingham et al.^16,17 and Hughes et al.²²), also because this would be expected to have a delayed effect.

The neural network predictor

The glucose predictor is that recently proposed by our group,¹² to which we refer the reader for details. In brief, the predictor is based on a feed-forward neural network and on a time-varying linear extrapolation algorithm,⁶ used in parallel to forecast, respectively, nonlinear and linear dynamics of glycemic time series. Figure 2 shows a block scheme with the structure of the predictor. Inputs of the linear extrapolation predictor are past CGM readings measured by the sensor. Inputs of the neural network block include the past error committed by the linear strategy, together with its trend (computed through first-order differences), the actual CGM readings and the trend of the CGM signal, and four other inputs that depend on meal information, namely, the glucose rate of appearance after a meal and three of its future time derivatives. The four inputs reflecting meal information are obtained using the physiological model of Dalla Man et al.,³³ with population parameters estimated in Dalla Man et al.³⁴ The output of the neural network block is the error currently committed by the linear model in forecasting future glycemia. Finally, the linear model and neural network outputs are combined together to obtain the prediction of future glycemia. The algorithm was trained on profiles of 10 patients (not included in the test set) simulated for 4 days. The prediction horizon used in this study is 30 min.

FIG. 2.

Block scheme of the prediction strategy of Zecchin et al.¹² CGM, continuous glucose monitoring; CHO, carbohydrate.

Remark 2

In Zecchin et al.,¹² the predictor proved to be robust against errors in estimation of meal size (±20%) and timing (±10 min). Therefore, we expect that input errors have a minor impact on the results reported in the present article. However, it is worthwhile stressing that the specific prediction algorithm chosen in this proof-of-concept study is not a central issue. In fact, results very similar to those reported in the present article could have been obtained with other prediction methods available in the literature. We chose our previously described method¹² for obvious practical reasons and because it is one of the few literature approaches allowing us to explicitly take into account meal information.

The hypoglycemia alert generation strategy

Because the focus here is on benefits of considering predicted rather than measured glucose for triggering hypoglycemia alarms, we prefer not to consider sophisticated algorithms to generate alerts. Rather, we consider a very basic procedure that generates an alert when the glucose profile (measured by the CGM sensor in Scenario 2, forecasted by the prediction algorithm in Scenario 3) crosses 70 mg/dL and is lower than this threshold for at least two consecutive sampling times (checking for the presence of two consecutive samples in the hypoglycemic range delays the alarm by 5 min but limits the problem of dealing with false alerts). After 30 min from the first hypoglycemia alert, if the subject is still in the hypoglycemic range, a second alarm is generated, and another 15 g of CHO is ingested by the patient.

Results

Figure 3, top panel, once again shows subject #1 of Figure 1 but this time in the presence of noise (lines meaning is as in Fig. 1). In Scenario 3 (glucose concentration denoted by the dashed red line), the nocturnal hypoglycemia is avoided (the lowest glucose concentration is 72 mg/dL, and the LBGI equals 3.6) thanks to the generation of the alert (followed by CHO ingestion) at 03:25. In Scenario 2 (glucose concentration denoted by the dotted blue line), the alarm is given at 04:05, and the hypoglycemic event can only be mitigated; in fact, the subject spends 60 min in the hypoglycemic range, reaching a lowest value of 63 mg/dL (LBGI of 5.0). Without hypoglycemia alert generation (Scenario 1, glucose concentration denoted by the dotted green line), the virtual subject experiences a threatening nocturnal hypoglycemia, with a minimum value of 53 mg/dL (LBGI of 14.8), which lasts for 255 min (approximately 4 h). The bottom panel of Figure 3 shows another representative subject. In Scenario 3 (glucose concentration denoted by the dashed red line), the prediction-based alert (followed by CHO ingestion) is generated 20 min ahead of time. However, hypoglycemia in this case is not totally avoided, but mitigated: the subject spends 20 min in the hypoglycemic range (lowest glycemia level of 67 mg/dL and LBGI of 5.9). In Scenarios 2 and 1 the time spent in the hypoglycemic range is 50 and 55 min, respectively (respective lowest glycemia levels of 64 mg/dL and 59 mg/dL and respective LBGI values of 7.9 and 9.6), with recovery from the event only after dinner.

FIG. 3.

(Top panel) Same as in Figure 1 but in the presence of noise. A 15-g carbohydrate (CHO) bolus is given at 04:05 and at 03:25 in Scenarios 2 and 3, respectively. (Bottom panel) Another representative test subject. A 15-g CHO bolus is given at 19:40 and at 19:20 in Scenarios 2 and 3, respectively. CGM, continuous glucose monitoring; hypo, hypoglycemia; pred, prediction. Color image available online at www.liebertonline.com/dia

Results computed on the 50 virtual subject dataset considering the entire period of monitoring (54 h) are given in terms of median and 5^th and 95^th percentiles. Table 1 and Figure 4 summarize the number of hypoglycemic events, their duration, and total time in hypoglycemic range. In Scenario 1 (unawareness of hypoglycemia, no alerts), patients experience a median of four (5^th, 95^th percentiles, two, seven) hypoglycemic episodes, with a median duration of 120 (10, 330) min. The total time spent in the hypoglycemic range is 9 h 30 min (4 h 5 min, 20 h 30 min) over the 54-h monitoring period, which corresponds to 17.7% (7.6%, 38.0%) of the total time of monitoring. In Scenario 2, the number of hypoglycemic events is similar to that of Scenario 1. This is expected because in Scenario 2 the alarm is CGM-based; thus it is triggered when the subject is de facto already in hypoglycemia. However, the severity of hypoglycemia events is significantly mitigated (P<0.01), with a median duration of 40 (10, 70) min, for a total time in hypoglycemia of 2 h 35 min (1 h 35 min, 5 h 0 min), corresponding to 4.7% (2.9%, 9.2%) of the total time. In Scenario 3 patients could potentially avoid, or at least mitigate, many hypoglycemic events by ingesting CHO in advance. In fact, the number of hypoglycemic events is one (none, four), 75% lower than Scenarios 2 and 1 (P<0.01). In addition, in Scenario 3 the median duration of hypoglycemia events is 15 (10, 45) min, significantly shorter than in Scenario 2 (−62.5%, P<0.01) and Scenario 1 (−87.5%, P<0.01). Furthermore, in Scenario 3, the percentage of time spent in the hypoglycemic range is 1.2% (0.0%, 2.9%), corresponding to 0 h 35 min (0 h 0 min, 1 h 35 min), with a reduction of 74.5% and 93.2% with respect to Scenario 2 and Scenario 1, respectively. Figure 4 graphically summarizes the results of Table 1. In the top panel the histogram of the count of number of hypoglycemic events per patient, during the monitoring period, clearly shows that in Scenario 3 the majority of patients experience only none to three hypoglycemic events, whereas in Scenario 2 and in Scenario 1 the majority of patients experience two to four hypoglycemic events. In addition, as shown in the box plots in the bottom panels of Figure 4, hypoglycemia duration (left panel) and total time in hypoglycemic range (right panel) considerably decrease moving from Scenario 1 and Scenario 2 to Scenario 3.

FIG. 4.

(Top panel) Histogram of the number of hypoglycemic episodes per subject, observed during the period of monitoring in the three scenarios. (Bottom panels) Boxplots of (left panel) the duration (in min) of hypoglycemic events and (right panel) the total time (in h) spent in the hypoglycemic range in the three scenarios (green, blue, and red for Scenarios 1, 2, and 3, respectively). CGM, continuous glucose monitoring; hypo, hypoglycemia; pred, prediction. Color image available online at www.liebertonline.com/dia

Table 1.

Median Results and 5^th and 95^th Percentiles for Number of Hypoglycemic Events, Average Length of Hypoglycemia Events, and Time Spent in Hypoglycemia During the Total Period of Monitoring (54 H)

	Number of hypoglycemic events				Hypoglycemia duration (min)				Total time in hypoglycemic range (h min and %)
	5^th	50^th	95^th	P value	5^th	50^th	95^th	P value	5^th	50^th	95^th	P value
Scenario 1 (no alert)	2	4	7		10	120	330		4 h 5 min	9 h 30 min	20 h 30 min
									7.6%	17.7%	38.0%
Scenario 2 (CGM-based alert)	2	4	7	0.29	10	40	70	<0.01	1 h 35 min	2 h 35 min	5 h 0 min	<0.01
									2.9%	4.7%	9.2%
Scenario 3 (prediction-based alert)	0	1	4	<0.01	10	15	45	<0.01	0 h 0 min	0 h 35 min	1 h 35 min	<0.01
				<0.01				<0.01	0.0%	1.2%	2.9%	<0.01

P values are computed with the nonparametric Mann–Whitney U test. For Scenario 2, the P value is for comparison with Scenario 1; for Scenario 3, the top P value refers to the comparison with Scenario 1, whereas the bottom P value is for comparison with Scenario 2.

CGM, continuous glucose monitoring.

Table 2 reports, for each scenario, the median and 5^th and 95^th percentiles of the distribution of glucose concentration values and of LBGI and HBGI in the 50 virtual patients. As expected, the 5^th percentile of the distribution of glucose concentration gradually increases in moving from Scenario 1 to Scenario 3. At the same time, the 95^th percentile of glucose concentration distribution does not significantly change between Scenario 1 and Scenario 3, indicating that hypoglycemia treatments do not significantly increase the highest hyperglycemic value. This is confirmed also by the estimated distribution of glycemic values, in the three scenarios, plotted in the top panel of Figure 5. In fact, the percentage of glycemic values in the hypoglycemic range is 19% in Scenario 1 and decreases to 5% in Scenario 2 and to 1% in Scenario 3. The percentage of glycemic values in the hyperglycemic range is 14% in Scenario 1 and 17% in both Scenario 2 and Scenario 3. Also, the analysis of LBGI and of HBGI, summarized in Table 2, confirms that the risk of hypoglycemia is significantly reduced (P<0.01), without any increased risk of hyperglycemia (P>0.5), in moving from Scenario 1 to Scenario 2 and to Scenario 3. This can be deduced also by visual inspection of the box plots of the distribution of LBGI and HBGI values in all the 50 subjects (bottom panels of Fig. 5).

FIG. 5.

(Top panel) Distribution of glycemia in the three scenarios. (Bottom panels) Boxplots of (left) Low Blood Glucose Index (LBGI) and (right) High Blood Glucose Index (HBGI) in the three scenarios (green, blue, and red for Scenarios 1, 2, and 3, respectively). CGM, continuous glucose monitoring; pred, prediction. Color image available online at www.liebertonline.com/dia

Table 2.

Median and 5 ^th and 95^th Percentiles for Results of Glucose Concentration Distributions and Low and High Blood Glucose Indexes Calculated in the Three Scenarios

	Glucose concentration (mg/dL)				LBGI				HBGI
	5^th	50^th	95^th	P value	5^th	50^th	95^th	P value	5^th	50^th	95^th	P value
Scenario 1 (no alert)	57	110	219		4.0	5.9	13.5		1.9	4.6	12.4
Scenario 2 (CGM-based alert)	70	119	231	<0.01	2.6	3.6	4.5	<0.01	1.9	5.1	12.8	0.17
Scenario 3 (prediction-based alert)	74	119	230	<0.01	2.0	2.9	3.4	<0.01	2.2	5.4	12.7	0.19
				<0.01				<0.01				0.87

P values computed with the nonparametric Mann–Whitney U test are also reported. For Scenario 2, the P value is for comparison with Scenario 1; for Scenario 3, the top P value refers to the comparison with Scenario 1, whereas the bottom P value is for comparison with Scenario 2.

CGM, continuous glucose monitoring; HBGI, High Blood Glucose Index; LBGI, Low Blood Glucose Index.

Robustness and Domain of Validity of the Analysis

Delayed/absent patient's response to alerts

In the previous section we simulated the virtual patients responding to alerts in no more than 5 min in both Scenario 2 (CGM-based alerts) and Scenario 3 (prediction-based alerts). However, in real-life conditions, subjects could be unable to promptly ingest CHO or to hear the alarm. For example, in the real case studies documented in Buckingham et al.³⁵ young patients did not respond to 34% of the alerts. In Choudhary et al.³⁶ patients did not respond to hypoglycemia alerts in 4.2% of the cases, and it took them on average 17 min during daytime, and 60 min during nighttime, to take countermeasures in case of hypoglycemia.

To assess the effect of delayed/absent responses to alerts, we did additional simulations introducing delays in CHO ingestion and the possibility of no CHO ingestion at all, in both Scenarios 2 and 3. In particular, every time an alarm is triggered (on the basis of either CGM or prediction), the probability is 0.85 that the subject ingests 15 g of CHO after a delay uniformly distributed in the time interval 0–30 min, versus a probability of 0.15 that no ingestion of CHO at all occurs. In the case of absent response, if the subject is still in hypoglycemia, a new alert is triggered after 30 min. In the case of delayed response, the new alert is generated 30 min after the subject has effectively ingested CHO, if he or she is still in hypoglycemia. Every time an alert is given, the same procedure just described is repeated (i.e., the patient either ignores the alert or ingests CHO with a certain delay).

Figure 6 and Table 3 summarize the results in terms of number of hypoglycemic events per patient, their duration, and total time in the hypoglycemic range during the monitoring period. By comparing results with those of the best-case scenario of Figure 4 and Table 1, we can clearly note a deterioration of the benefits of CGM-based and prediction-based alerts coupled with CHO ingestion. There is still a visible and significant reduction of number of hypoglycemic events and of their duration passing from Scenario 2 to Scenario 3 (P<0.01). In Scenario 2, as expected, the number of hypoglycemic events in Table 3 cannot worsen with respect to Table 1 because in both cases the CGM-based alerts are generated when the subject is already in hypoglycemia. In Scenario 3 the number of hypoglycemic events in Table 3 increases with respect to Table 1, and a median of three (one, five) hypoglycemic events per subject was observed. In particular, moving from Scenario 2 to Scenario 3, the number of hypoglycemic events significantly decreases by 25% (P<0.01). Hypoglycemia duration in Table 3 is longer than in Table 1 and is equal to 50 (10, 149) min in Scenario 2 and to 25 (10, 112) min in Scenario 3. In fact, in Scenario 3 the duration of hypoglycemic events decreases by 50% with respect to Scenario 2 (P<0.01). The total time in the hypoglycemic range is equal to 3 h 35 min (1 h 55 min, 10 h 35 min) in Scenario 2 and to 1 h 30 min (0 h 20 min, 5 h 45 min) in Scenario 3 (significant reduction of 59%, P<0.01). The distribution of glycemic values is reported in Table 4 and in Figure 7, top panel. The shape of the distribution of glucose shows a mild increase in the percentage of values in hypoglycemic range in Scenarios 2 and 3, with respect to Table 2 and to Figure 5, top panel. However, as confirmed by P values, ingestion of CHO on the basis of predicted glycemia, even if delayed or sometimes ignored, still significantly increases the lowest glycemic concentration experienced by patients. In fact, the percentage of glycemic values in the hypoglycemic range is equal to 8% in Scenario 2 and to 4% in Scenario 3. Analysis of LBGI and HBGI values, reported in Table 4 and in Figure 7, bottom panels, confirms that, moving from Scenario 2 to Scenario 3, a significant decrease of the risk of hypoglycemia occurs, without any parallel increase of the hyperglycemia risk.

FIG. 6.

As in Figure 4 but in the presence of delays in answering to alerts. CGM, continuous glucose monitoring; prediction. Color image available online at www.liebertonline.com/dia

FIG. 7.

As in Figure 5 but in the presence of delays in answering to alerts. CGM, continuous glucose monitoring; HBGI, High Blood Glucose Index; LBGI, Low Blood Glucose Index; pred, prediction. Color image available online at www.liebertonline.com/dia

Table 3.

As in Table 1 but in the Presence of Delays in Answering to Alerts

	Number of hypoglycemic events				Hypoglycemia duration (min)				Total time in hypoglycemic range (h min and %)
	5^th	50^th	95^th	P value	5^th	50^th	95^th	P value	5^th	50^th	95^th	P value
Scenario 1 (no alert)	2	4	7		10	120	330		4 h 5 min	9 h 30 min	20 h 30 min
									7.6%	17.7%	38.0%
Scenario 2 (CGM-based alert)	2	4	7	0.57	10	50	149	<0.01	1 h 55 min	3 h 35 min	10 h 35 min	<0.01
									3.5%	6.6%	19.6%
Scenario 3 (prediction-based alert)	1	3	5	<0.01	10	25	112	<0.01	0 h 20 min	1 h 30 min	5 h 45 min	<0.01
				<0.01				<0.01	0.6%	2.7%	10.6%	<0.01

CGM, continuous glucose monitoring.

Table 4.

As in Table 2 but in the Presence of Delays in Answering to Alerts

	Glucose concentration (mg/dL)				LBGI				HBGI
	5^th	50^th	95^th	P value	5^th	50^th	95^th	P value	5^th	50^th	95^th	P value
Scenario 1 (no alert)	57	110	219		4.0	5.9	13.5		1.9	4.6	12.4
Scenario 2 (CGM-based alert)	67	118	229	<0.01	2.9	3.9	6.3	<0.01	2.5	5.2	12.9	0.22
Scenario 3 (prediction-based alert)	71	119	231	<0.01	2.3	3.2	6.4	<0.01	2.1	5.2	12.7	0.16
				<0.01				<0.01				0.88

CGM, continuous glucose monitoring; HBGI, High Blood Glucose Index; LBGI, Low Blood Glucose Index.

In conclusion, delays in responding to hypoglycemia alerts, or absence of response, obviously worsen the results obtained in Scenarios 2 and 3 in Results. However, because delayed/absent ingestion of CHO affects in a similar way Scenarios 2 and 3, the relative difference between these two scenarios remain significant: in fact, passing from Scenario 2 to Scenario 3, the total time in the hypoglycemic range decreases by 59%, the hypoglycemia duration decreases by 50%, and the number of hypoglycemic events decreases by 25%.

Limitations of the adopted in silico environment

In the present study we resorted to the simulation environment because it allows a fair comparison of different scenarios. Indeed, starting from a given initial patient condition, three alternative scenarios could be considered, corresponding to different actions, which in real experiments would have mutually excluded each other. However, even if widely used to preliminarily test new algorithms, given its sufficient realism,^37

–40 the adopted in silico environment of Kovatchev et al.²⁰ has some limitations.

A first limit is that it is not possible yet to take into account intrasubject variability, and there is no model for the various factors that influence glycemia in real life, like, for example, stress and illness. This issue has been partially dealt with by simulating a large number of patients (50 synthetic subjects) for a short period of time (54 h), rather than simulating a few patients for longer periods.

A second restriction is the use of a simple model for sensor noise (i.e., white Gaussian noise with constant variance). Several studies indicated that correlated errors/drift are likely to occur in CGM,⁵ but well-established models, more sophisticated than that used in the present article, are, at present, unavailable. Only a few literature contributions devoted some space to sensor noise characterization: in Chase et al.²³ sensor noise was modeled as a random white Gaussian process with coefficient of variation constant in time and arbitrarily chosen by the user; Breton and Kovatchev⁴¹ proposed an autoregressive model to describe a combination of sensor noise plus blood glucose-to-interstitial glucose kinetics model error, but the domain of validity of this model is limited²⁶; and, finally, our group⁴² showed that the Menarini (Florence, Italy) Glucoday^® CGM sensor data are consistent with a time-varying measurement error variance, but no models of noise heteroscedasticity were investigated. However, although simulating correctly the different components of sensor noise would be obviously important in comparing the ideal (glucose concentration is measured in the blood) versus the practically feasible (glucose concentration is measured with error in the subcutis by a CGM sensor) situation, the conclusions drawn from the key comparison of the present article (i.e., Scenario 2 vs. Scenario 3) would remain similar. Indeed, all the above-cited sources of error would affect both scenarios. As far as the ideal versus practically feasible situation is concerned, it is also worthwhile recalling that, when real CGM data affected by systematic errors are considered, enhancement/calibration methods exploiting sparse SMBG references can be used to improve their accuracy,^{24,25,27
–29} as already mentioned in In silico database.

To conclude, we remark that the simulated analysis performed in the present article cannot capture all the aspects of reality, but any bias equally affects results observed in Scenario 2 (CGM-based alert) as well as in Scenario 3 (prediction-based alert). Thus, on one hand, the absolute results presented in this article could be considered an upper bound of what could be observed in real life. On the other hand, the relative difference between the results obtained in Scenario 2 (CGM-based) and Scenario 3 (prediction-based alert) would probably not change significantly. It is also useful mentioning that research work is in progress to improve the University of Virginia/Padova simulator (e.g., by developing tools to take into account intrapersonal physiological changes⁴³ [this will allow to simulate longer monitoring periods] and by considering more realistic models of CGM sensor errors⁴⁴).

Conclusions

Since the pioneering study by Bremer and Gough,² CGM-based short-term glucose prediction algorithms are believed to allow the patient to take appropriate countermeasures to avoid/mitigate the hypoglycemia event. However, a quantitative comprehensive assessment is still lacking in the literature because the only assessments published so far concern clinical studies in humans where multiple parallel scenarios for the same subject cannot be investigated. By generating data for 50 virtual subjects, in this work we compared occurrence and duration of hypoglycemic events in three scenarios occurring in the same patient (i.e., hypoglycemia unawareness and no countermeasure [Scenario 1], ingestion of 15 g of CHO as glucose concentration measured by CGM sensor crosses the hypoglycemic threshold [Scenario 2], and ingestion of 15 g of CHO as the glucose concentration predicted 30 min ahead of time crosses the hypoglycemic threshold [Scenario 3]). Results show that, by generating hypoglycemia alerts based on prediction, hypoglycemia occurrence could be mitigated and almost totally avoided (a median of one hypoglycemic event in 54 h of monitoring), and time spent in hypoglycemia could be reduced to 1.2% of the period of monitoring, corresponding to 35 min in 54 h. A specific prediction algorithm (i.e., the recently presented neural network-based algorithm of Zecchin et al.¹²), was chosen, but the assessment could be repeated by forecasting glycemia through any of the several prediction strategies proposed in the literature. Needless to say, algorithms particularly accurate in predicting future hypoglycemia would generate highly effective prediction-based alerts.

Because the focus of this article was in assessing how much the number and duration of hypoglycemic events can be potentially reduced thanks to the use of prediction methods, we have not considered the problem of how to generate alerts and have limited ourselves to using a simple threshold comparison strategy. In fact, generating alerts from CGM profiles is a critical issue because of data noise. In the literature, a few results are available, and the efficacy of existing alerting systems, especially in terms of robustness against false alerts, is controversial.^45,46 In addition, because the uncertainty in the CGM data due to noise is amplified when predictions are estimated, the problem of avoiding false alerts is exacerbated when predicted profiles are considered (for example, in Scenario 3 we had, on average, one false alert every 39 h). This suggests that, although it is true that important improvements in the signal-to-noise ratio can be achieved by denoising techniques like those proposed previously,^{24,25,27
–29} efforts of investigators should concentrate on the development of new “smart” alert generation strategies exploiting the potential benefits of glucose prediction algorithms.

Footnotes

Author Disclosure Statement

No competing financial interests exist.

References

Bode

, Gross

, Rikalo

, Schwartz

, Wahl

, Page

, Gross

, Mastrototaro

. Alarms based on real-time sensor glucose values alert patients to hypo- and hyperglycemia: the Guardian continuous monitoring system. Diabetes Technol Ther, 2004; 6:105–113.

Bremer

, Gough

. Is blood glucose predictable from previous values? A solicitation for data. Diabetes, 1999; 48:445–451.

Buckingham

. Hypoglycemia detection, and better yet, prevention, in pediatric patients. Diabetes Technol Ther, 2005; 7:792–796.

Sparacino

, Facchinetti

, Cobelli

. “Smart” continuous glucose monitoring sensors: on-line signal processing issues. Sensors, 2010; 10:6751–6772.

Bequette

. Continuous glucose monitoring: real-time algorithms for calibration, filtering, and alarms. J Diabetes Sci Technol, 2010; 4:404.

Sparacino

, Zanderigo

, Corazza

, Maran

, Facchinetti

, Cobelli

. Glucose concentration can be predicted ahead in time from continuous glucose monitoring sensor time-series. IEEE Trans Biomed Eng, 2007; 54:931–937.

Gani

, Gribok

, Rajaraman

, Ward

, Reifman

. Predicting subcutaneous glucose concentration in humans: data-driven glucose modeling. IEEE Trans Biomed Eng, 2009; 56:246–254.

Eren-Oruklu

, Cinar

, Quinn

, Smith

. Estimation of future glucose concentrations with subject-specific recursive linear models. Diabetes Technol Ther, 2009; 11:243–253.

Eren-Oruklu

, Cinar

, Rollins

, Quinn

. Adaptive system identification for estimating future glucose concentrations and hypoglycemia alarms. Automatica, 2012; 48:1892–1897.

10.

Pérez-Gandía

, Facchinetti

, Sparacino

, Cobelli

, Gómez

, Rigla

, de Leiva

, Hernando

. Artificial neural network algorithm for online glucose prediction from continuous glucose monitoring. Diabetes Technol Ther, 2010; 12:81–88.

11.

Pappada

, Cameron

, Rosman

, Bourey

, Papadimos

, Olorunto

, Borst

. Neural network-based real-time prediction of glucose in patients with insulin-dependent diabetes. Diabetes Technol Ther, 2011; 13:135–141.

12.

Zecchin

, Facchinetti

, Sparacino

, De Nicolao

, Cobelli

. Neural network incorporating meal information improves accuracy of short-time prediction of glucose concentration. IEEE Trans Biomed Eng, 2012; 59:1550–1560.

13.

Naumova

, Pereverzyev

, Sivananthan

. A meta-learning approach to the regularized learning—case study: blood glucose prediction. Neural Networks, 2012; 33:181–193.

14.

Dassau

, Cameron

, Lee

, Bequette

, Zisser

, Jovanovic

, Chase

, Wilson

, Buckingham

, Doyle

III . Real-time hypoglycemia prediction suite using continuous glucose monitoring: a safety net for the artificial pancreas. Diabetes Care, 2010; 33:1249–1254.

15.

Keenan

, Mastrototaro

, Voskanyan

, Steil

. Delays in minimally invasive continuous glucose monitoring devices: a review of current technology. J Diabetes Sci Technol, 2009; 3:1207–1214.

16.

Buckingham

, Cobry

, Clinton

, Gage

, Caswell

, Kunselman

, Cameron

, Chase

. Preventing hypoglycemia using predictive alarm algorithms and insulin pump suspension. Diabetes Technol Ther, 2009; 11:93–97.

17.

Buckingham

, Chase

, Dassau

, Cobry

, Clinton

, Gage

, Caswell

, Wilkinson

, Cameron

, Lee

, Bequette

, Doyle

III . Prevention of nocturnal hypoglycemia using predictive alarm algorithms and insulin pump suspension. Diabetes Care, 2010; 33:1013–1017.

18.

Eren-Oruklu

, Cinar

, Quinn

. Hypoglycemia prediction with subject-specific recursive time-series models. J Diabetes Sci Technol, 2010; 4:25–33.

19.

Harvey

, Dassau

, Zisser

, Bevier

, Seborg

, Jovanovič

, Doyle

III . Clinically relevant hypoglycemia prediction metrics for event mitigation. Diabetes Technol Ther, 2012; 14:719–727.

20.

Kovatchev

, Breton

, Cobelli

, Dalla Man

. Method, system, computer simulation environment for testing of monitoring, control strategies in diabetesPatent WO/2008/1577881.2008.

21.

Kovatchev

, Breton

, Dalla Man

, Cobelli

. Biosimulation modeling for diabetes. in silico preclinical trials: a proof of concept in closed-loop control of type 1 diabetes. J Diabetes Sci Technol, 2009; 3:1374–1381.

22.

Hughes

, Patek

, Breton

, Kovatchev

. Hypoglycemia prevention via pump attenuation and red-yellow-green “traffic” lights using continuous glucose monitoring and insulin pump data. J Diabetes Sci Technol, 2010; 4:1146–1155.

23.

Chase

, Hann

, Jackson

, Lin

, Lotz

, Wong

, Shaw

. Integral based filtering of continuous glucose sensor measurements for glycaemic control in critical care. Comp Methods Prog Biomed, 2006; 82:238–247.

24.

Guerra

, Facchinetti

, Sparacino

, De Nicolao

, Cobelli

. Enhancing the accuracy of subcutaneous glucose sensors: a real-time deconvolution-based approach. IEEE Trans Biomed Eng, 2012; 59:1658–1669.

25.

Facchinetti

, Sparacino

, Cobelli

. Enhanced accuracy of continuous glucose monitoring by online extended Kalman filtering. Diabetes Technol Ther, 2010; 12:353–356.

26.

Facchinetti

, Sparacino

, Cobelli

. Modeling the error of continuous glucose monitoring sensor data: critical aspects discussed through simulation studies. J Diabetes Sci Technol, 2010; 4:4–14.

27.

Knobbe

, Buckingham

. The extended Kalman filter for continuous glucose monitoring. Diabetes Technol Ther, 2005; 7:15–27.

28.

Facchinetti

, Sparacino

, Cobelli

. Online denoising method to handle intraindividual variability of signal-to-noise ratio in continuous glucose monitoring. IEEE Trans Biomed Eng, 2011; 58:2664–2671.

29.

Barcelo-Rico

, Bondia

, Diez

, Rossetti

. A multiple local models approach to accuracy improvement in continuous glucose monitoring. Diabetes Technol Ther, 2012; 14:74–82.

30.

Cryer

. Mechanisms of hypoglycemia-associated autonomic failure and its component syndromes in diabetes. Diabetes, 2005; 54:3592–3601.

31.

Kovatchev

, Cox

, Gonder-Frederick

, Young-Hyman

, Schlundt

, Clarke

. Assessment of risk for severe hypoglycemia among adults with IDDM: validation of the low blood glucose index. Diabetes Care, 1998; 21:1870–1875.

32.

Briscoe

, Davis

. Hypoglycemia in type 1 and type 2 diabetes: physiology, pathophysiology, and management. Clin Diabetes, 2006; 24:115–121.

33.

Dalla Man

, Camilleri

, Cobelli

. A system model of oral glucose absorption: validation on gold standard data. IEEE Trans Biomed Eng, 2006; 53:2472–2478.

34.

Dalla Man

, Rizza

, Cobelli

. Meal simulation model of the glucose-insulin system. IEEE Trans Biomed Eng, 2007; 54:1740–1749.

35.

Buckingham

, Block

, Burdick

, Kalajian

, Kollman

, Choy

, Wilson

, Chase

. Response to nocturnal alarms using a real-time glucose sensor. Diabetes Technol Ther, 2005; 7:440–447.

36.

Choudhary

, Shin

, Wang

, Evans

, Hammond

, Kerr

, Shaw

JAM

, Pickup

, Amiel

. Insulin pump therapy with automated insulin suspension in response to hypoglycemia. Diabetes Care, 2011; 34:2023–2025.

37.

Magni

, Forgione

, Toffanin

, Dalla Man

, Kovatchev

, De Nicolao

, Cobelli

. Run-to-run tuning of model predictive control for type 1 diabetes subjects: in silico trial. J Diabetes Sci Technol, 2009; 3:1091–1098.

38.

Kovatchev

, Cobelli

, Renard

, Anderson

, Breton

, Patek

, Clarke

, Bruttomesso

, Maran

, Costa

, Avogaro

, Dalla Man

, Facchinetti

, Magni

, De Nicolao

, Place

, Farrett

. Multinational study of subcutaneous model-predictive closed-loop control in type 1 diabetes mellitus: summary of the results. J Diabetes Sci Technol, 2010; 4:1374–1381.

39.

Van Heusden

, Dassau

, Zisser

, Seborg

, Doyle

III . Control-relevant models for glucose control using a priori patient characteristics. IEEE Trans Biomed Eng, 2012; 59:1839–1849.

40.

Patek

, Magni

, Dassau

, Karvetski

, Toffanin

, De Nicolao

, Del Favero

, Breton

, Dalla Man

, Renard

, Zisser

, Doyle

III , Cobelli

, Kovatchev

. Modular closed-loop control of diabetes. IEEE Trans Biomed Eng, 2012; 59:2986–2999.

41.

Breton

, Kovatchev

. Analysis, modeling, and simulation of the accuracy of continuous glucose sensors. J Diabetes Sci Technol, 2008; 2:853–862.

42.

Facchinetti

, Sparacino

, Cobelli

. Online denoising method to handle intraindividual variability of signal-to-noise ratio in continuous glucose monitoring. IEEE Trans Biomed Eng, 2011; 58:2664–2671.

43.

Visentin

, Dalla Man

, Cobelli

. Cloning a day of T1DM individual subjects from the FDA-accepted simulator by a Bayesian approach. Presented at the 12th Annual Diabetes Technology Meeting, Bethesda, MD, November 8–10, 2012.

44.

Facchinetti

, Del Favero

, Castle

, Ward

, Sparacino

, Cobelli

. Model of sensor error from multiple simultaneous CGMs in the same subject. Presented at the 12th Annual Diabetes Technology Meeting, Bethesda, MD, November 8–10, 2012.

45.

Klonoff

. The need for separate performance goals for glucose sensors in the hypoglycemic, normoglycemic, and hyperglycemic ranges. Diabetes Care, 2004; 27:834–836.

46.

Ward

. The role of new technology in the early detection of hypoglycemia. Diabetes Technol Ther, 2004; 6:115–117.