Evaluation of the Utility of a Glycemic Pattern Identification System

Abstract

Background:

With the increasing prevalence of systems allowing automated, real-time transmission of blood glucose data there is a need for pattern recognition techniques that can inform of deleterious patterns in glycemic control when people test. We evaluated the utility of pattern identification with a novel pattern identification system named Vigilant™ and compared it to standard pattern identification methods in diabetes.

Method:

To characterize the importance of an identified pattern we evaluated the relative risk of future hypoglycemic and hyperglycemic events in diurnal periods following identification of a pattern in a data set of 536 patients with diabetes. We evaluated events 2 days, 7 days, 30 days, and 61-90 days from pattern identification, across diabetes types and cohorts of glycemic control, and also compared the system to 6 pattern identification methods consisting of deleterious event counts and percentages over 5-, 14-, and 30-day windows.

Results:

Episodes of hypoglycemia, hyperglycemia, severe hypoglycemia, and severe hyperglycemia were 120%, 46%, 123%, and 76% more likely after pattern identification, respectively, compared to periods when no pattern was identified. The system was also significantly more predictive of deleterious events than other pattern identification methods evaluated, and was persistently predictive up to 3 months after pattern identification.

Conclusions:

The system identified patterns that are significantly predictive of deleterious glycemic events, and more so relative to many pattern identification methods used in diabetes management today. Further study will inform how improved pattern identification can lead to improved glycemic control.

Keywords

analysis blood diabetes glucose identification pattern

For insulin-dependent diabetes patients and their care providers, the ability to quickly and efficiently identify deleterious patterns in blood glucose (BG) data can have a significant impact on improving diabetes management outcomes and quality of life.^1,2 Standard tools for aiding in identifying patterns typically involve (1) paper log books or (2) software on a desktop computer, BG meter, or other handheld device. In the case of software, data logs may be accompanied by weekly averages or point charts, or more complex analytical tools in the form of modal days, percentage of readings in and out of a target range, or averages by diurnal time period.^2,3

These methods have several shortcomings, including (1) the limited characterization of the utility of the information provided and (2) the significant time and mental workload required to extract distinct patterns or other actionable information.^3-5 With the increasing prevalence of BG meters with advanced processing and display functionality, and technologies that transmit BG data to smartphone or cloud based systems in real time, there is an opportunity to develop an intelligent system that can overcome the above shortcomings.

There is support for the management of diurnal BG patterns with behavioral or treatment related interventions.^2,3,6,7 We are evaluating a novel system named Vigilant™ (investigational use software device from InSpark Technologies, Inc, Charlottesville, VA) that is designed to message users about diurnal patterns in upcoming daily time periods when the patient is testing, and to optimize the frequency of pattern messaging to within a desirable range. Patterns of hyperglycemia and hypoglycemia are identified if the product of the following exceeds a predetermined threshold: (1) the percentage of BG readings in a diurnal time period exceeding a clinical threshold and (2) the probability that the identified diurnal BG pattern is deviant relative to the rest of the patient’s BG levels. This method, hereinafter referred to as “method 1,” has been described in detail in a prior publication.⁸

Methods

We are reporting separately on the ability of the system to deliver a desirable frequency of messaging.⁹ Based on a review of prior message-based interventions,^10-12 we determined a target frequency range that is frequent enough to promote active self-management of patterns but not so frequent as to be obtrusive. The system achieved 77% of patients meeting an average of 0.5-2.5 messages per day in silico with optimized default settings across a population of 436 patients (146 T1, 278 T2, 12 N/A, HbA1c = 8.3% (SD 1.4%), daily test frequency mean 3.3 (SD 1.2)).

A desirable frequency of messages notwithstanding, messages will only have an impact if the pattern information has utility. In addition, the importance of identified patterns should be well characterized so that clinicians and patients can appropriately understand and prioritize the information that is presented.

Evaluating the Utility of Pattern Information

Prior studies of diabetes pattern analysis tools have measured self-reported insights or survey results as a measure of utility.^2,13 While user preference is important to ensure a management tool is adopted, self-reported insights are not a measure of anticipated outcome, or of the value of the information provided in terms of health benefits. In addition, preference surveys are prone to a number of possible biases such as sample selection bias, social desirability bias, and construct validity.¹⁴

We propose that the utility of an identified pattern should be associated with the degree to which that pattern predicts patient outcomes, preferably patient outcomes that can be mitigated by diabetes management decisions. HbA1c is the gold standard of diabetes control because it is correlated with a risk of long-term complications.¹⁵ If it was not, it would be misleading as a benchmark of control. Similarly, any identified BG patterns should be predictive of future deleterious events or complications. Otherwise the utility of the presentation of that pattern is unclear.

Our approach is similar to the method outlined by Kovatchev et al in the evaluation of a new measure of glycemic variability called the average daily risk range, or ADRR.¹⁶ In this study the ADRR was evaluated for its ability to predict future glycemic events that were out of target range compared to other measures of glycemic variability. This is an effective way of determining the capacity for a pattern, or an index, of determining potential glycemic outcomes that can be mitigated by management decisions. It follows that the greater the capacity of a pattern to predict deleterious events, the greater the propensity for management action addressing these patterns to ameliorate glycemic control.

All data analyses were performed on a data set of 536 patients with 422 188 BG readings over an average of 230 days (201 T1, 323 T2, 12 N/A; daily test frequency mean (SD) = 3.75 (1.52); mean BG (SD) = 181 (93) mg/dL) using MatLab® from MathWorks and Microsoft® Excel. In this data set we ran method 1 and determined the number of hyperglycemic (BG ≥ 180 mg/dL), severe hyperglycemic (BG ≥ 400 mg/dL), hypoglycemic (BG ≤ 70 mg/dL), and severe hypoglycemic (BG ≤ 40 mg/dL) episodes that did or did not follow a pattern message for hyperglycemia or hypoglycemia.

To convey the predictive ability of pattern messages, we will be reporting results in terms of the relative risk of deleterious glucose events, which enumerates the increased probability of a deleterious event occurring in diurnal time periods following a pattern message. A relative risk of 1 would signify that the pattern message is not predictive of future deleterious events at all, whereas a relative risk of 2 would signify that a deleterious event is twice as likely to occur in diurnal time periods following the pattern message.

To characterize the longitudinal risk of deleterious events following pattern messages, we evaluated relative risk across 4 different time windows after a pattern message; 2 days, 7 days, 30 days, and 61-90 days. To identify whether the relative risk is a broad effect or if it is localized to a subset of patients, we also evaluated patient cohorts of diabetes type (type 1 and type 2), glycemic control and BG test frequency.

Comparison with Other Pattern Identification Methods

We benchmarked the utility of pattern messages against other pattern identification methods in diabetes. Comparative pattern identification methods included (1) a predetermined count of hypoglycemic or hyperglycemic events occurring in a diurnal time period over 5, 14, or 30 days, or (2) a predetermined percentage of the readings that were hypoglycemic or hyperglycemia in a diurnal time period over 5, 14, or 30 days. These comparative methods were chosen because identification of BG patterns via inspection of logbooks or software tools may take the form of a count or percentage methodology in clinical practice.^2,17,18 Percentage of readings in and out of target range is a standard graphical tool in diabetes management software,^2,3 and there is currently an automated pattern identification system that uses a count criteria for identifying patterns.¹⁴

In any comparative evaluation of the utility of identified patterns, there must be consideration for the impact of pattern identification frequency. Some pattern identification criteria may have a high predictive ability but occur so infrequently as to have limited overall utility. Therefore we opted to limit the comparison of pattern identification criteria to within a frequency of messaging nearest to the predetermined messaging frequency of method 1. This will produce a better like-for-like comparison of the predictive ability of method 1 versus other pattern identification methods because the impact of message frequency is limited as a confounding factor.

Thus methods 2, 3, and 4 represent a pattern identification threshold consisting of meeting or exceeding a count of 5, 10, or 19 hyperglycemic episodes, and 2, 3, or 5 hypoglycemic episodes, in the previous 5, 14, and 30 days, respectively. Methods 5, 6, and 7 represent a pattern identification threshold consisting of meeting or exceeding 100%, 79%, or 77% hyperglycemic episodes, and 21%, 21%, or 19% hypoglycemic episodes, in the previous 5, 14, and 30 days, respectively. These specific thresholds were chosen because they represent the integer count or integer percentage that results in the closest possible messaging frequency to method 1, but that are still less than the messaging frequency of method 1.

It is possible that evaluating a single messaging frequency across these pattern identification models could produce results that are an outlier relative to other messaging frequencies. Area under the curve for the receiver operating characteristic (AUROC) is a common method for evaluating clinical algorithms and medical diagnostics¹⁹ which could address this issue. However AUROC ignores the goodness-of-fit of the model, and it summarizes the test performance over regions of the AUROC space in which one would not normally operate.^20-22 In the case of method 1, an AUROC would integrate the value of messaging scenarios that are not user friendly, including when messages are so frequent they are obtrusive. However, a partial AUROC can be performed in a specificity range where messaging frequency is not excessive. Therefore we examined partial AUROC in a range of specificities resulting in up to 2 times the messaging frequency of the evaluated method 1 settings.

Results

For our standalone evaluation of method 1, in Table 1 and Figure 1 we summarize results for a 7-day predictive window. As shown in Figure 1A, method 1 was predictive of significantly more hyperglycemia (46% more after a hyperglycemia pattern, P < .001) and hypoglycemia (120% more after a hypoglycemia pattern, P < .001) in the next 7 days when compared to periods when no pattern was identified. As shown in Figure 1B, method 1 pattern messages were also predictive of significantly more severe hyperglycemia (76% more after a hyperglycemia pattern, P < .001) and severe hypoglycemia (124% more after a hypoglycemia pattern, P < .001).

Table 1.

Event Rates and Proportion of Deleterious Glucose Events After Method 1-Identified Diurnal Patterns Versus When No Pattern Has Been Identified Over a 7-Day Prediction Window.

	Number of events in data set (n, %)	Proportion of events after pattern identification (%)	Proportion of events when no pattern is identified (%)
Hyperglycemia	181 405 (43.0)	61.3	42.0
Hypoglycemia	33 235 (7.9)	16.2	7.3
Severe hyperglycemia	11 423 (2.7)	5.1	2.9
Severe hypoglycemia	2765 (0.7)	1.2	0.6

Figure 1.

(A) Increased likelihood of hyperglycemic and hypoglycemic events in the same diurnal time period following a method 1 pattern message versus no pattern message, over a subsequent 7-day window. (B) Increased likelihood of severe hyperglycemic and severe hypoglycemic events in the same diurnal time period following a method 1 pattern message versus no message, over a subsequent 7-day window. Black bars indicate proportion of events following a method 1 message; gray bars indicate proportion of events when no message is given. *Indicates a statistically significant difference (P < .001) using the Wilcoxon signed rank test.

Table 2 demonstrates that patterns identified by method 1 have immediate and long-term utility for predicting deleterious glycemic events across all cohorts of diabetes type and control. Statistical significance was reached across all cohort comparisons for the relative risk of hyperglycemic and hypoglycemic events, and most other comparisons as well. A paucity of evaluable data could be a contributing factor to lack of statistical significance in certain comparisons. To wit, the majority of table cells with a lack of significance are evaluating severe glycemic events (≥400 mg/dL or ≤40 mg/dL), which are generally infrequent in the data set (2.7% and 0.7% of readings overall, respectively).

Table 2.

The Relative Risk of Glycemic Events After Method 1-Identified Diurnal Patterns Across Diabetes Types and Glycemic Control Cohorts.

		Hyperglycemia patterns		Hypoglycemia patterns
	Evaluable patients	BG ≥ 180 mg/dL	BG ≥ 400 mg/dL	BG ≤ 70 mg/dL	BG ≤ 40 mg/dL
2 days after a message
All data	536	1.46*	1.70*	2.23*	2.45*
Type 1	201^†	1.31*	1.52*	1.60*	2.01*
Type 2	323^†	1.55*	1.95*	2.99*	2.93*
BG group 1	126	2.51*	2.67	2.03*	1.72*
BG group 2	164	1.41*	1.51*	1.99*	2.49*
BG group 3	133	1.32*	1.80*	2.41*	2.41*
BG group 4	113	1.12*	1.29*	2.85*	4.76*
61-90 days after a message
All data	536	1.36*	1.67*	2.14*	2.05*
Type 1	201^†	1.26*	1.49*	1.70*	1.66
Type 2	323^†	1.41*	1.80*	2.44*	2.32*
BG group 1	126	1.99*	2.41	1.88*	1.78*
BG group 2	164	1.31*	1.47	1.97*	1.88*
BG group 3	133	1.22*	1.75*	2.16*	1.61
BG group 4	113	1.10*	1.28*	2.95*	4.29*

Cohorts for glycemic control are BG group 1: mean BG < 154 mg/dL, BG group 2: 154 ≤ mean BG < 183 mg/dL, BG group 3: 183 ≤ mean BG < 212 mg/dL, and BG group 4: mean BG ≥ 212 mg/dL. *Indicates that the relative risk is statistically significant (P < .05) using the Wilcoxon signed rank test. ^†Indicates that 12 patients were of unknown type and therefore excluded from the diabetes type subgroup analysis.

Perhaps what is most noteworthy about Table 2 is the long-term persistency of relative risk after a message. A method 1 pattern message of hypoglycemia remained 114% more predictive of hypoglycemia in diurnal time periods 3 months after the message is given, and a similar persistency was observed for severe hypoglycemia (105%), hyperglycemia (36%), and severe hyperglycemia (67%). These findings suggest that, in the absence of management intervention, there are persistent idiosyncratic patterns of deleterious glycemia in diurnal periods in this population. The presence of these strong patterns highlights the importance of identifying and managing diurnal patterns irrespective of pattern identification methodology.

Table 3 demonstrates the effect of test frequency on relative risk. Pattern identification is predictive of future hypoglycemia and hyperglycemia across all test frequency cohorts. It is notable that relative risk declines at higher test frequencies. The reduced relative risk can perhaps be explained by the increased prevalence of deleterious events at higher test frequencies, indicating there is more background hyperglycemia and hypoglycemia in these groups.

Table 3.

The Relative Risk of Glycemic Events After Method 1-Identified Diurnal Patterns Across Test Frequency Groups Over a 7-day Prediction Window.

	Evaluable patients	% hyper BG readings	% hypo BG readings	Relative risk after hyperglycemia patterns		Relative risk after hypoglycemia patterns
	Evaluable patients	% hyper BG readings	% hypo BG readings	BG ≥ 180 mg/dL	BG ≥ 400 mg/dL	BG ≤ 70 mg/dL	BG ≤ 40 mg/dL
TF group 1	186	42	5.4	1.50*	1.71*	2.94*	3.01
TF group 2	189	43	7.4	1.50*	1.85*	2.21*	2.55*
TF group 3	88	45	10.5	1.35*	1.47*	1.85*	1.97*
TF group 4	73	42	12.3	1.39*	2.04*	1.64*	1.48

Average daily test frequency groups are TF group 1 (≥ 2 and < 3 tests per day), TF group 2 (≥ 3 and < 4 tests per day), TF group 3 (≥ 4 and < 5 tests per day), and TF group 4 (≥ 5 tests per day). *Statistically significant (P < .05) using the Wilcoxon signed rank test.

As shown in Table 4, across the majority of time frames and messaging types there is a statistically significant higher relative risk of deleterious events after method 1 messages compared to the comparative pattern identification methods. In only 1 case was there a higher relative risk for the comparative methods. For the prediction of severe hyperglycemia in a 2 day window with method 6, the relative risk in this case was 1.75 versus 1.70 for method 1, but this difference was not statistically significant in favor of method 6.

Table 4.

The Statistical Significance of the Difference in Relative Risk of Deleterious Events for Method 1 Pattern Identification Versus 6 Comparative Methods.

		Comparative pattern identification methods
Type of event	Time range	2	3	4	5	6	7
Hyperglycemia	2 days	†	*	*	—	*	*
	7 days	*	*	*	†	†	†
	30 days	*	*	*	*	—	—
	61-90 days	*	*	*	*	*	*
Severe hyperglycemia	2 days	*	*	*	*	—	*
	7 days	*	*	*	*	*	*
	30 days	*	*	*	*	*	*
	61-90 days	*	*	*	*	*	*
Hypoglycemia	2 days	*	†	*	—	—	—
	7 days	*	*	*	—	—	—
	30 days	*	*	*	*	*	*
	61-90 days	*	*	*	*	*	*
Severe hypoglycemia	2 days	*	*	*	*	*	*
	7 days	†	*	*	†	†	†
	30 days	*	*	*	*	*	*
	61-90 days	†	—	†	—	—	—

Relative risk was compared for hyperglycemic, severe hyperglycemic, hypoglycemic and severe hypoglycemic events across windows of 2 days, 7 days, 30 days, and 61-90 days. —Indicates no statistically significant difference. *Indicates there a statistically significant higher relative risk of deleterious events after a method 1 message (P < .05) using the Wilcoxon signed rank test corrected by the Holm–Bonferroni method. †Indicates there is a statistically significant higher relative risk of deleterious events prior to P value correction via the Holm–Bonferroni method.

Figure 2A shows the relative risk of deleterious events after a method 1 message compared to the mean relative risk for the 6 other methods tested. Method 1 outperformed the 6 other methods in the prediction of hyperglycemic episodes (0.07 to 0.11 higher relative risk) and hypoglycemic episodes (0.29 to 0.39 higher relative risk). For the prediction of severe episodes, the differences were more pronounced. Figure 2B demonstrates method 1 outperforming the other methods in the prediction of severe hyperglycemia (0.12 to 0.39 higher relative risk) and severe hypoglycemia (0.42 to 0.73 higher relative risk).

Figure 2.

(A) The relative risk of subsequent deleterious events for method 1 when compared to the mean of 6 methods of identifying patterns for hyperglycemic and hypoglycemic events across windows of 2 days, 7 days, 30 days, and 61-90 days. (B) The relative risk of subsequent deleterious events for the method 1 algorithm when compared to the mean of 6 comparative methods of identifying patterns for severe hyperglycemic and severe hypoglycemic events across windows of 2 days, 7 days, 30 days, and 61-90 days. Black bars indicate the relative risk for the method 1 algorithm; gray bars indicate the relative risk for the mean of the 6 comparative methods of identifying patterns.

By plotting receiver operating characteristic curves and analyzing messaging frequencies we found that a specificity of 0.8 or less for pattern identification thresholds resulted in a doubling or more of hyperglycemia pattern identification and a specificity of 0.7 or less resulted in a doubling or more of hypoglycemia pattern identification. Therefore if we calculate partial AUROC above these specificity cutoffs we effectively evaluate the pattern identification methods in a useful range of messaging frequencies. Using a 7-day predictive event horizon, the partial AUROC for method 1 was significantly greater than all comparative methods (P < .01 in all cases), verifying that method 1 maintains predictive strength across a range of useful messaging frequencies. Partial receiver operator characteristic curves are illustrated in Figure 3 and areas under the curves can be found in Table 5.

Figure 3.

Partial receiver operating characteristic curves for method 1 and the 6 comparative methods for messaging frequencies up to 2 times method 1’s default messaging frequency. (A) Hyperglycemia, (B) hypoglycemia, (C) severe hyperglycemia, and (D) severe hypoglycemia.

Table 5.

Partial AUROC Calculations for Method 1 and the 6 Comparative Methods.

	Hyperglycemia	Hypoglycemia	Severe hyperglycemia	Severe hypoglycemia
Method 1	.0328	.0732	.0384	.0918
Method 2	.0309	.0600	.0317	.0744
Method 3	.0284	.0627	.0301	.0810
Method 4	.0286	.0647	.0300	.0862
Method 5	.0309	.0600	.0317	.0744
Method 6	.0322	.0639	.0337	.0827
Method 7	.0323	.0647	.0338	.0867

AUROC was evaluated for specificities of between .8 to 1.0 for hyperglycemia and severe hyperglycemia and specificities of between .7 to 1.0 for hypoglycemia and severe hypoglycemia. Method 1 partial AUROC was significantly greater than all other methods evaluated for each curve (P < .01). Statistical significance was evaluated using the method developed by Hanley and McNeil.²³

Discussion

There are several limitations to this study, and considerations that should be noted in the context of diabetes management in general.

For one, the comparative pattern identification methods do not comprise the full breadth of pattern identification methods in diabetes. For practical and rational reasons, we have chosen pattern identification methods in common use today, but for which there are many other possible permutations of the window of data, counts, or percentages of deleterious events. While we controlled count and percentage thresholds to align with the pattern identification frequency of method 1, using windows of data other than 5, 14, and 30 days may achieve different results, for example.

Despite attempts to minimize the effect of message frequency, in certain cases the limited degrees of freedom in pattern identification methods resulted in significant differences in messaging frequency. Method 2 for hypoglycemia (at least 2 episodes of hypoglycemia in the last 5 days) identified a pattern less than half the time than method 1 (3.9% of time periods versus 7.9% for method 1), even though it was the closest proxy to pattern identification frequency for a count of hypoglycemia in 5 days. It is noteworthy that method 1 patterns remained more predictive of hypoglycemic events despite having twice the messaging frequency of method 2. Also, the hyperglycemic threshold for method 5 (100% of diurnal readings being hyperglycemic over the last 5 days) reached a maximum before we could obtain a lower messaging frequency than method 1. However at this threshold method 5 pattern identification frequency was nearly identical to method 1 (7.8% of time periods versus 7.5% for method 1), and the AUROC analysis evaluated performance of method 5 versus method 1 across a range of useful messaging frequencies, and thus we opted to proceed with the evaluation using method 5.

This study is limited to pattern recognition paradigms utilizing episodic BG data and not continuous glucose monitoring (CGM) data. We acknowledge that CGM has the potential to provide a richer data stream, which may allow for more predictive pattern recognition algorithms. However, as of the time of writing, most insulin-using patients do not use CGM, and those that do often use it intermittently. Therefore there remains a significant opportunity to support patients and clinicians with episodic BG pattern recognition.

The analysis of pre- and postprandial glucose levels is well supported as a tenet of diabetes management,^6,17,24 and method 1 does not evaluate prandial data. However, we do not view an understanding of pre- and postprandial glucose dynamics as an alternate or competing pattern identification practice, but rather one that is complementary. For example, if one has a pattern of hyperglycemia identified in the evening, one may want to check pre- and postprandial glucose levels in the evening to see if the pattern is manifesting as a result of a meal-/bolus-related management behavior.

Also, while evaluation of pre- and postprandial glucose data is an accepted and important clinical practice, electronic capture of data for analysis requires the patient to flag the glucose result, whereas the evaluated pattern identification techniques do not require flagging. In addition, postprandial glucose can be difficult to define due to varied diurnal eating habits, snacking times and meal sizes. Thus we must accept that automated assessment of pre- and postprandial data may present an incomplete picture due to inaccurate or inadequate information, and higher level patterns such as the ones identified by method 1 can help fill in the gaps.

The ultimate test of utility would be to determine improved clinical outcomes from the use of method 1 in the hands of patients. We have not evaluated user preference and human factors in this analysis, which would contribute to our understanding of adherence, patient comprehension and what management actions may be useful to address patterns. However, we believe we have established that diurnal patterns identified by method 1 are important insofar as they are significantly predictive of deleterious BG events, and more so relative to many standard pattern identification methods used in diabetes management today.

Conclusions

In this study it was found that the diurnal patterns identified by method 1 are predictive of future hyperglycemia, severe hyperglycemia, hypoglycemia, and severe hypoglycemia in the same diurnal time period across the evaluated population and within subgroups of diabetes type, glycemic control, and test frequency. Method 1 is also predictive of persistent diurnal patterns over 2 to 3 months, and significantly more predictive of deleterious glycemia than the other pattern identification techniques evaluated across a range of useful messaging frequencies.

Footnotes

Abbreviations

AUROC, area under the curve of the receiver operator characteristic; BG, blood glucose; CGM, continuous glucose monitoring.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Erik A. Otto is an employee and equity holder of InSpark Technologies. Vinay Tannan is a consultant to InSpark Technologies and receives compensation for services rendered.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by InSpark Technologies, Inc.

References

Bergenstal

Callahan

Johnson

. Management principles that most influence glycemic control: a follow-up study of former DCCT participants. Diabetes. 1996;45(suppl 2):124A.

Parkin

Davidson

. Value of self-monitoring blood glucose pattern analysis in improving diabetes outcomes. J Diabetes Sci Technol. 2009;3(3):500-508.

Rodbard

. Optimizing display, analysis, interpretation and utility of self monitoring of blood glucose data for management of patients with diabetes. J Diabetes Sci Technol. 2007;1(1):62-71.

Klonoff

. Improved outcomes from diabetes monitoring: the benefits of better adherence, therapy adjustments, patient education, and telemedicine support. J Diabetes Sci Technol. 2012;6(3):486-490.

Rao

Hou

Golnik

Flaherty

. Evolution of data management tools for managing self-monitoring of blood glucose results: a survey of iPhone applications. J Diabetes Sci Technol. 2010;4(4):949-957.

Monnier

Lapinski

Colette

. Contributions of fasting and postprandial plasma glucose increments to the overall diurnal hyperglycemia of type 2 diabetic patients: variations with increasing levels of HbA(1c). Diabetes Care. 2003;26(3):881-885.

Weintrob

Schechter

Benzaquen

. Glycemic patterns detected by continuous subcutaneous glucose sensing in children and adolescents with type 1 diabetes mellitus treated by multiple daily injections vs continuous subcutaneous insulin. Arch Pediatr Adolesc Med. 2004;158(7):677-684.

Kovatchev

Price

Otto

. Systems, methods and computer program codes for recognition of patterns of hyperglycemia and hypoglycemia, increased glucose variability, and ineffective self-monitoring in diabetes. US patent application number US2007943226A. November 20, 2007.

Otto

Tannan

. Optimizing pattern messaging frequency to improve interventional outcomes. Poster presented at: Diabetes Technology Meeting; October 31, 2013; San Francisco, CA.

10.

Krishna

Boren

Balas

. Healthcare via cell phones: a systematic review. Telemed J E Health. 2009;15(3):231-240.

11.

Cole-Lewis

Kershaw

. Text messaging as a tool for behavior change in disease prevention and management. Epidemiol Rev. 2010;32(1):56-69.

12.

Fjeldsoe

Marshall

Miller

. Behavior change interventions delivered by mobile telephone short-message service. Am J Prev Med. 2009;36(2):165-173.

13.

Grady

Campbell

MacLoed

Srinivasan

. Evaluation of a blood glucose monitoring system with automatic high-and low-pattern recognition software in insulin-using patients: pattern detection and patient-reported insights. J Diabetes Sci Technol. 2013;7(4):970-978.

14.

Shaughnessy

Zechmeister

Jeanne

. Research Methods in Psychology. 9th ed. New York, NY: McGraw-Hill; 2011:161-175.

15.

Diabetes Control and Complications Trial Research Group. The effect of intensive treatment of diabetes and the development and progression of long-term complications in insulin-dependent diabetes mellitus. N Engl J Med. 1993;329(14):977-986.

16.

Kovatchev

Otto

Cox

Gonder-Frederick

Clarke

. Evaluation of a new measure of blood glucose variability in diabetes. Diabetes Care. 2006;29(11):2433-2438.

17.

International Diabetes Center. Insulin Basics. Minneapolis, MN: IDC; 2008.

18.

Pearson

Bergenstal

. Fine-tuning control: pattern management versus supplementation: view 1: pattern management: an essential component of effective insulin management. Diabetes Spectrum. 2001;14(2):75-78.

19.

Zweig

Campbell

. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem. 1993;39(4):561-577.

20.

Cook

. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115:928-935.

21.

Obuchowski

. Fundamentals of clinical research for radiologists: ROC analysis. AJR. 2005;184(2).

22.

Lobo

Jimenez-Valverde

Real

. AUC: a misleading measure of the performance of predictive distribution models. Global Ecol Biogeogr. 2007;17(2):145-151.

23.

Hanley

McNeil

. A method of comparing areas under the receiver operating characteristic curve derived from the same cases. Radiology. 1983;148(3):839-843.

24.

IDF Clinical Guidelines Taskforce. 2011 Guideline for Management of Post Meal Glucose in Diabetes. 2011.