Accuracy of CGM During Closed-Loop Home Studies

CGM Performance Criteria for AP Systems—Considerations: Decoupling Component Performance and Clinical Utility

Performance criteria or specifications are critical for all engineered systems. In diabetes, blood glucose meters must meet ISO 15197:2013 standard requirements. ³ There also exists a Clinical and Laboratory Standards Institute (CLSI) guideline for evaluation of CGM systems, ⁴ although these guidelines are not a mandatory component of the regulatory approval process in the United States for CGM devices. Why the lack of uptake on the CLSI guidelines? It may be to the distinctly different clinical utilization of the information provided by a capillary blood glucose meter. Whereas self-monitoring of blood glucose (SMBG) relies on the precision and accuracy of the meter reading to accurately make diabetes decisions, the CGM provides an important temporal component that SMBG does not. The commonly used analogy here of car safety is appropriate. The radar gun is analogous to the blood glucose meter. Precision and accuracy are critical in measure automobile speeds on a highway. The tracking, trending, and alarms provided by CGM devices are incorporated into diabetes management schemes to allow more proactive management of the diabetes, and this additional information has been shown to be valuable in improving diabetes outcomes (reduction of A1c, improved time in target, reduction in hypoglycemia, etc.) despite being “less accurate” than capillary blood glucose meters. ^5,6

The struggle to develop widely accepted CGM evaluation criteria highlights a tension between the mechanical performance characteristics of diabetes devices and their clinical utility. The performance characteristics of SMBG devices are easy to define and well accepted because the link between the data and the clinical application is direct. There is a direct cause/effect between the information derived and the clinical decision at hand (i.e., the blood glucose level is 50 mg/dL, have a snack quickly). Precision and accuracy are critical in this paradigm. However, the link is different with CGM devices—the precision and accuracy become less important because of the additional temporal data. That is, blood sugar is in a normal range and heading downward to a low range—treat with carbohydrates proactively. The numerical accuracy and precision are balanced by the additional information provided by the directionality and rate of change of the glucose readings. Groups have described novel means to mathematically quantify this benefit. ^7,8

Furthermore, the development of high-risk products such as AP systems are tightly regulated in both the United States and the European Union. Medical device developers must follow a formal risk management process and demonstrate that risk of harm to a patient or operator have been considered and addressed, following the internationally recognized medical device risk management standard ISO 14971. ⁹ This specifically addresses:

• Have risks been designed out where possible?

AP Systems: Mechanical Performance Criteria Versus Clinical Utility: How Far Under the Hood?

The performance characteristics of CGM devices within AP systems are further complicated by the fact that the CGM is one part of a multicomponent system. The performance characteristics within the system will be different in an AP than in open-loop control, where the human is interpreting the data and making diabetes decisions. In 2011, the JDRF and the Helmsley Charitable Trust held a series of meetings with thought leaders that focused on the characteristics of continuous monitors that would allow for safe automated dosing of insulin. It was clear that although sensor accuracy is very important, in the context of an AP system, there are many more specifications that are also worthy of significant consideration.

In a subsequent request for applications (RFA), potential applicants were asked to address how a sensor suitable for an AP system would perform/operate within the following categories: Calibration, Accuracy, Failure Detection, Sensor Drift, Hypoglycemia Performance, Form Factors, Redundancy, Advanced Algorithms, Reduced Warm Up, Duration of Wear, GPS, Connectivity, Ability to Direct/Own Data—Cloud Based, and Risk Management Approaches in Case the Proposed Technology Does Not Meet Specifications. This list highlights that the performance criteria for CGM devices when framed in the context of a closed-loop AP system are broader and include several characteristics that are critical in minimizing the risk for system driven overdosing of insulin. A common theme that arose in the process of developing this RFA was that tolerance for inaccuracy could be balanced by error-detection systems and redundancy in the system.

The challenge in developing performance criteria for CGM devices is the different approaches that different systems use to mitigate potentially anomalous sensor readings. The clinical performance of the system is confounded further by other potential sources of variability including infusion set failure, insulin degradation, blood glucose meter performance, etc. The performance of closed-loop AP systems will ultimately be judged by their safety and efficacy in well-designed clinical trials.

If a glucose meter is analogous to a radar gun, and a CGM to a speedometer and windshield, then a closed-loop AP may be analogous to a Google car (or maybe a modern car with advanced safety systems). The goal in utilizing each of these technologies is to improve on the road safety. Advanced automobiles utilize a variety of mechanisms to improve safety and minimize risk of accidents. Similarly, AP systems will utilize a variety of mechanisms to improve insulin dosing to minimize hypo- and hyperglycemia exposure. The question that Thabit et al. ² raise and is worthy of discussion and debate is the following: how far under the hood do we go? Each component of the system will need to meet specifications based on the system's overall performance. Fortunately, Thabit et al. ² and others have demonstrated that a variety of different AP approaches have been successful in safely automating various degrees of insulin delivery.

Summary

The evolution of diabetes technologies is pivoting from component-based approaches to systems. With this transition the performance of each component will be judged in the context of the performance of the overall system. The overall system will not be judged by metrics such as sensor accuracy and precision; rather, they will be evaluated on their clinical performance defined by their safety and efficacy. The CGM is a critical component of closed-loop AP systems. Thabit et al. ² provide the first careful look at the performance of a CGM device in home closed-loop studies. The performance of the sensor—particularly its accuracy in the euglycemic and hyperglycemic range—combined with the safety and efficacy of the overall system suggest that there are sensor characteristics that may be defined to aid in further AP system development. Ultimately, critical CGM specifications will need to be developed in the broader context of the AP system. It will be the sum of the system that defines the achievement of the ultimate metric for success—improved diabetes outcomes.