Abstract
Schematic of the stages towards the development of a fully automated dual-hormone closed-loop system. Reprinted with permission of Aaron Kowalski, Juvenile Diabetes Research Foundation.
As closed-loop studies move from inpatient to outpatient, the methods for assessing the efficacy and safety of closed-loop systems will become a challenge. Ultimately, for pivotal studies to bring a product to market, key outcomes will include glycated hemoglobin (HbA1c) levels and development of acute complications such as severe hypoglycemia and diabetic ketoacidosis. However, for these outcomes to be feasible, studies likely will need to be at least 6 months and require a large sample size, particularly to be able to assess adverse events because the rates will be low. Prior to the conduct of these pivotal studies, there will be the need for many shorter-duration studies with smaller sample sizes, for which these outcomes will not be meaningful. For such studies, measurement of glucose levels will be an important outcome.
Inpatient closed-loop studies rely on blood glucose measurements to assess the function of the closed-loop system. Current-generation blood glucose meters are sufficiently accurate to serve this role in the outpatient setting. However, feasibility of using blood glucose meter measurements as a primary outcome in most outpatient study designs will be limited. Requiring patients to do six to eight self-monitoring of blood glucose (SMBG) measurements per day at specified time intervals for long periods of time will be too burdensome for most patients, and compliance with frequent middle-of-the-night measurements is likely to be low. In addition, SMBG measurements may tend to oversample high and low values because patients will tend to measure the blood glucose level when concerned that the glucose level may be low or high. Intermittent seven-point testing (before and after meals and at bedtime) was used in the Diabetes Control and Complications Trial. 1 However, a more recent study found it difficult to achieve a high compliance rate, at least in children. 2 Even with seven glucose measurements a day, much will be missed, particularly in assessing closed-loop control overnight where at best there could be a single blood glucose measurement.
By recording glucose values continually, continuous glucose monitoring (CGM) solves the problem of infrequent, potentially biased glucose measurements. The value of CGM as a primary outcome measure has been demonstrated in two randomized trials evaluating CGM as an intervention in individuals with type 1 diabetes with HbA1c in the excellent range. 3,4 The Food and Drug Administration in a guidance on artificial pancreas device systems recognized the important role CGM can play in assessing outcomes in closed-loop studies. 5 Unfortunately, although using current-generation CGM to assess outcome eliminates the problems described above for SMBG measurements, it creates other problems when it is used as an outcome measure.
The first issue to address is CGM accuracy. Current generation CGM devices are not as accurate as SMBG measurements. 6 Nevertheless, when CGM glucose measurements were compared with reference blood glucose measurements in data from several inpatient studies, glycemia metrics computed from CGM approximated glycemia metrics computed from reference blood glucose measurements with the exception that CGM tended to slightly underestimate the extremes of hypoglycemia and hyperglycemia as well as glucose variability. 7 In designing a clinical trial using CGM as an outcome, the inaccuracy will not affect the type 1 error rate but will affect statistical power, although this can be accounted for in the study design by increasing sample size if the degree of inaccuracy can be estimated.
In this issue of the journal, Hovorka et al. 8 evaluate the effect of using CGM in closed-loop studies to simultaneously inform the control algorithm and assess performance. The authors evaluate two approaches to modifying CGM glucose values to reduce bias when using CGM for measuring glycemic outcomes: a stochastic approach and retrospective recalibration. Their stochastic approach assumed 15% error to estimate the probability that the true glucose level is above or below a certain threshold. The retrospective recalibration approach utilized blood glucose measurements at the beginning and end of an overnight CGM glucose tracing to adjust the glucose values between these two time points. The authors found that unadjusted CGM glucose values tended to overestimate the overnight percentage of values in the target range and to underestimate the percentage below target, more so during closed-loop control than during conventional pump therapy. Bias for time in target and below target appeared less when adjusting the glucose values using the stochastic approach, but on the downside, the stochastic approach tended to overestimate glucose variability. Retrospective recalibration, surprisingly, did not improve accuracy.
Under what circumstances could stochastic adjustment reduce bias when using CGM to measure outcome? First, it is important to understand that the potential bias occurs when a categorical outcome variable (e.g., in range versus out of range) is created from a continuous measure such as glucose. Thus with the model for stochastic adjustment used by Hovorka et al. 8 (CGM glucose value=blood glucose+fixed error), mean glucose would not be expected to be biased when using CGM as an outcome measure assuming that the sensor is not biased (i.e., a sensor glucose value is just as likely to be too high as too low). However, a more realistic model might be CGM glucose value=slope×blood glucose+intercept+ error, where slope is <1. In this case, CGM values will tend to be higher than low true blood glucose values, and vice versa. Second, bias would not be expected to be present when CGM is used as an outcome measure in a clinical trial if the null hypothesis of no difference between groups is true. 7 This latter point means that the bias described by the authors would not increase the probability of a type 1 error (concluding that a treatment group difference exists when in fact there is not a true difference) and would not affect statistical power. However, the bias could produce an overestimate of the true treatment effect assuming that a true treatment group difference exists.
Assuming that time in range is being used as the outcome metric and a true treatment effect exists, why does stochastic adjustment reduce bias in the overestimate of the treatment effect? As explained by the authors, the closed-loop system strives to achieve glucose levels in the target range. In doing so, more CGM glucose values will end up in this range than are present in truth. It is likely that the controller induces a correlation between the CGM measurement error and the true blood glucose that does not exist during open loop and alters the balance of false positives and false negatives. Using a second sensor that is not part of the closed-loop system to measure outcome would be expected to reduce at least some of the bias depending on the degree of correlation of the second sensor with the sensor driving the controller. However, such a second sensor may not be practical for outpatient studies of more than short duration.
It is surprising that retrospective recalibration did not improve the accuracy of the sensor glucose measurements. In an unpublished paper prepared for the Food and Drug Administration in 2010 entitled “Retroactive Adjustment of Continuous Glucose Monitoring (CGM) Data Enabling the Use of CGM for the Assessment of Primary Study Endpoints,” Boris Kovatchev and Marc Breton described a retrospective recalibration procedure similar to that of Hovorka et al. 8 that when applied to a CGM dataset substantially improved the concordance between CGM and reference glucose values. Without a clear explanation for why these results are not consistent with those of Kovatchev and Breton, it is possible that retrospective recalibration may help in some circumstances and not in others. Further evaluation of the effect of retrospective recalibration using other datasets will be important. In addition, evaluating the combination of the stochastic approach and retrospective recalibration would be of interest.
Use of CGM as an outcome measure will be a necessity in many closed-loop studies. The authors have described an innovative approach to account for sensor inaccuracy in estimating the true treatment effect in such studies. However, there are some limitations, largely noted by the authors, that are important to recognize. The analysis approach required certain assumptions such as a uniform sensor error, whereas almost certainly the error varies considerably based on patient and sensor factors. The analysis was of data from a single study, and it is possible that in other scenarios with different degrees of sensor inaccuracy and different patient cohorts, the results could be different. Simulations of glucose data that cover a spectrum of situations to compare the stochastic and unadjusted approach would be valuable. As the analysis was performed on data collected overnight, it will be important to assess both the stochastic and retrospective recalibration approaches in other datasets and on data collected during the daytime, where there will be much greater fluctuations in glucose levels than overnight. Thus, although the results of the analyses are important, they should be viewed as an important first step and not the final word.