The Rectangle Target Plot

Abstract

Background:

The measurement accuracy of systems for self-monitoring of blood glucose (SMBG) is usually analyzed by a method comparison in which the analysis results are displayed using difference plots or similar graphs. However, such plots become difficult to comprehend as the number of data points displayed increases. This article introduces a new approach, the rectangle target plot (RTP), which aims to provide a simplified and comprehensible visualization of accuracy data.

Methods:

The RTP is based on ISO 15197 accuracy evaluations of SMBG systems. Two-sided tolerance intervals for normally distributed data are calculated for absolute and relative differences at glucose concentrations <100 mg/dL and ≥100 mg/dL. These tolerance intervals provide an estimator of where a 90% proportion of results is found with a confidence level of 95%.

Results:

Plotting these tolerance intervals generates a rectangle whose center indicates the systematic measurement difference of the investigated system relative to the comparison method. The size of the rectangle depends on the measurement variability.

Conclusions:

The RTP provides a means of displaying measurement accuracy data in a simple and comprehensible manner. The visualization is simplified by reducing the displayed information from typically 200 data points to just 1 rectangle. Furthermore, this allows data for several systems or several lots from 1 system to be displayed clearly and concisely in a single graph.

Keywords

ISO 15197 rectangle target plot self-monitoring of blood glucose system accuracy

The measurement accuracy of systems for self-monitoring of blood glucose (SMBG) is generally characterized using parameters such as bias or percentage of values within International Organization for Standardization (ISO) 15197 accuracy limits.^1,2 Results are presented in tables or graphs which require detailed knowledge and experience to interpret the displayed data and which sometimes miss relevant information such as bias or variability. There is a need for a new, more comprehensible approach to the visualization of measurement accuracy data that facilitates the easy interpretation of data, also for the lay person.

Measurement accuracy is usually analyzed by determining the bias between an SMBG system and a comparison method of a higher metrological order (as described in the ISO 15197 standard) when measuring the same sample and analyte. The Clinical and Laboratory Standards Institute (CLSI) guideline EP09-A3 describes the experimental procedures for a method comparison and bias estimation using patient samples.³ The ISO 15197 standard specifies testing procedures and accuracy criteria for SMBG systems and is commonly accepted as the international standard for evaluation of SMBG systems before market launch.^1,2

The system accuracy criteria defined in the 2003 version of the ISO 15197 standard¹ require that at least 95% of individual SMBG measurements are within ±15 mg/dL of the comparison measurement results at blood glucose (BG) concentrations <75 mg/dL and within ±20% at BG concentrations ≥75 mg/dL. The revised ISO 15197:2013 standard² includes more stringent accuracy criteria requiring at least 95% of individual meter results within ±15 mg/dL for BG concentrations <100 mg/dL and within ±15% for BG concentrations ≥100 mg/dL. Compliance with the accuracy criteria has to be demonstrated in the SMBG system’s labeling. For this purpose, tables are included showing the system accuracy results, for example, number and percentage of SMBG measurement results within the specified limits. ISO 15197 also describes a plot for the visualization of accuracy data showing the difference between each individual measurement result of the SMBG system and the respective measurement result of the comparison method.

Other approaches to the assessment and presentation of measurement accuracy can be applied as well, for example, bias analysis according to Bland and Altman,⁴ regression analysis according to Passing and Bablok,⁵ and calculation of the mean absolute relative difference (MARD).⁶

However, regardless of the approach taken to system accuracy evaluation, the presentation of the numerical data in tables is often not concise and difficult to interpret. In addition, commonly used graphical representations like difference plots become increasingly difficult to comprehend as the number of data points displayed increases.

This article introduces the rectangle target plot (RTP) as a new approach to the visualization of the measurement accuracy of an SMBG system. The RTP is based on an ISO 15197 accuracy evaluation and is designed with the aim of providing a simplified and comprehensible visualization of accuracy data obtained for a given system.

Methods

The RTP approach presented here was developed based on data sets obtained in accuracy evaluation studies following ISO 15197.^1,2 Each individual rectangle represents the data assessment of a method comparison following ISO 15197 and shows a single combination of meter type and test strip lot of a given system.

Data for the Calculation of the Rectangle Target Plot

Following the ISO 15197 evaluation protocol, 200 measured values obtained from at least 100 capillary samples from at least 100 different subjects were analyzed for each combination of meter type and test strip lot of a given system. Following ISO 15197:2013, the samples were distributed into different BG concentration categories (Table 1).

Table 1.

Distribution of the Capillary Blood Samples Into Different Glucose Concentration Categories According to ISO 15197:2013.²

Concentration category	Percentage of samples	Glucose concentration (mg/dL)
1	5	≤50
2	15	>50-80
3	20	>80-120
4	30	>120-200
5	15	>200-300
6	10	>300-400
7	5	>400

The samples were assigned to the categories according to the mean BG result of the respective comparison method. For the calculation of the RTP, the data were divided into BG concentrations <100 mg/dL and BG concentrations ≥100 mg/dL. The cut-off value of 100 mg/dL was chosen because of the ISO 15197:2013 system accuracy criterion A (clause 6.3.3), which considers absolute differences below 100 mg/dL and relative differences ≥100 mg/dL.² Since the concentration category 3 includes samples with BG concentrations from >80-120 mg/dL, the numbers of data points for BG concentrations <100 mg/dL and ≥100 mg/dL depend on the specific distribution within concentration category 3. For example, if category 3 includes only samples <100 mg/dL for a given system, 80 data points <100 mg/dL and 120 data points ≥100 mg/dL are obtained (duplicate measurements on each sample). If category 3 includes only samples ≥100 mg/dL, 40 data points <100 mg/dL and 160 data points ≥100 mg/dL are obtained. This variation of sample sizes is taken into consideration by the RTP approach.

Calculation of the Rectangle Target Plot

The RTP is based on a 2-sided tolerance interval for normally distributed data.⁷ The tolerance interval is defined by lower and upper tolerance limits. The tolerance limits are calculated separately for BG concentrations <100 mg/dL and BG concentrations ≥100 mg/dL.

The tolerance interval is a statistical tool that covers a specified proportion of data points (eg, p = 90%) with a stated confidence level (usually γ = 95%). In other words, if an unlimited number of tests were performed, in 95% of tests a proportion of at least 90% of results would be found within the tolerance interval. Thus, based on study data, the RTP can be used to predict the general performance of the SMBG system. The following formulas⁷ are used for the calculation:

YL = \bar{Y} - k 2 s

YU = \bar{Y} + k 2 s

where YL is the lower tolerance limit, YU is the upper tolerance limit, and $\bar{Y}$ and s are the mean and standard deviation of the differences between the SMBG measurements and the comparison measurements, respectively. Absolute and relative differences are calculated for BG concentrations <100 mg/dL and ≥100 mg/dL, respectively. The 2-sided k-factor k2 is dependent on the specific p and γ.⁷

The approximate value for k2 as a function of p and γ for a 2-sided interval⁷ is calculated using the following formula:

k 2 = \sqrt{\frac{v (1 + \frac{1}{N}) z_{(1 - p) / 2}^{2}}{χ_{1 - γ, v}^{2}}} .

where $χ_{1 - γ, ν}^{2}$ is the critical value of the chi-square distribution with degrees of freedom ν (ν = N – 1) that is exceeded with probability γ, and $z_{(1 - p) / 2}^{2}$ is the critical value of the normal distribution associated with the cumulative probability (1 – p) / 2.⁷

The k-factor depends on the numbers of data points (N) for BG concentrations <100 mg/dL and BG concentrations ≥100 mg/dL, respectively. Thus, the k-factor for a given system depends on the distribution of samples in glucose concentration category 3 as described above. As N will usually be different for BG concentrations <100 mg/dL and ≥100 mg/dL, 2 different k-factors have to be calculated, one for absolute differences (with N for BG concentrations <100 mg/dL) and one for relative differences (with N for BG concentrations ≥100 mg/dL).

Plotting the Data and the Tolerance Interval

After calculation of the tolerance interval using formulas (1) to (3), the data are displayed in an RTP. The RTP shows the tolerance intervals of a given system (combination of meter type and test strip lot). Each rectangle in the RTP is limited by the lower (YL) and upper (YU) tolerance limits for BG concentrations <100 mg/dL (x-axis: x YU and x YL) and for BG concentrations ≥100 mg/dL (y-axis: y YU and y YL). Thus, a rectangle is described by the following 4 corner points: (x YU | y YU), (x YU | y YL), (x YL | y YL), (x YL | y YU).

Template for the Rectangle Target Plot

A template for the visualization of an SMBG system’s measurement accuracy in an RTP is shown in Figure 1. Absolute differences between the SMBG system’s measurement and the comparison measurement at BG

Figure 1.

Template for the visualization of an SMBG system’s measurement accuracy in a rectangle target plot (RTP).

concentrations <100 mg/dL are plotted on the x-axis, relative differences at BG concentrations ≥100 mg/dL are plotted on the y-axis. Since the RTP was designed using data sets obtained from ISO 15197 system accuracy evaluations, the light gray square in the template indicates the system accuracy limits as stipulated by ISO 15197:2013 (±15 mg/dL for BG concentrations <100 mg/dL and ±15% for BG concentrations ≥100 mg/dL). The dark gray square indicates more stringent limits (±10 mg/dL for BG concentrations <100 mg/dL and ±10% for BG concentrations ≥100 mg/dL).

Results

Visualization of system accuracy data using the RTP is described and shown below for 2 example systems with 1 test strip lot each. In addition, examples are provided for RTPs showing system accuracy for multiple test strip lots of a system.

Examples of the General RTP Principle

Data for 2 example systems were obtained in accuracy evaluation studies following ISO 15197. In these studies, system 1 showed 100% and system 2 showed 80% of measurements within the accuracy limits of ISO 15197:2013 (±15 mg/dL for BG concentrations <100 mg/dL and ±15% for BG concentrations ≥100 mg/dL).

In each case, the calculation of the RTP was based on 200 measurement data points (Table 2). The RTP for system 1 included 52 data points with BG concentrations <100 mg/dL (21.6 to 99.7 mg/dL) and 148 data points with BG concentrations ≥100 mg/dL (102.3 to 488.4 mg/dL). The RTP for system 2 included 62 data points with BG concentrations <100 mg/dL (27.9 to 97.4 mg/dL) and 138 data points with BG concentrations ≥100 mg/dL (102.0 to 527.7 mg/dL).

Table 2.

Calculation of the Tolerance Intervals for the rectangle target plot (RTP) for System 1 and System 2.

	System 1		System 2
BG concentration	<100 mg/dL	≥100 mg/dL	<100 mg/dL	≥100 mg/dL
N	52	148	62	138
k2	1.988	1.827	1.951	1.824
$\bar{Y}$	0.4 mg/dL	–1.2%	8.3 mg/dL	–3.6%
s	2.6 mg/dL	2.8%	9.8 mg/dL	9.5%
Min, max	–4.9 mg/dL, 6.2 mg/dL	–9.1%, 6.5%	–16.1 mg/dL, 31.4 mg/dL	–24.9%, 23.5%
YU	5.5 mg/dL	3.9%	27.5 mg/dL	13.7%
YL	–4.8 mg/dL	–6.3%	–10.9 mg/dL	–21.0%

For both systems, the total number of included data points ( N ), the 2-sided k-factor ( k2 ), the mean difference ( $\bar{Y}$ ) with standard deviation ( s ) between the SMBG system’s measurements and the comparison measurements, the minimum and maximum individual differences, and the lower and upper tolerance limits ( YL and YU ) are shown for BG concentrations <100 mg/dL and ≥100 mg/dL.

For both systems, absolute and relative tolerance intervals were calculated for BG concentrations <100 mg/dL and ≥100 mg/dL using formulas (1) to (3). Table 2 shows the calculated values.

For each system, the tolerance intervals and mean differences for BG concentrations <100 mg/dL and ≥100 mg/dL were visualized in RTPs using the template shown in Figure 1.

Figure 2 shows the RTPs for each system including both mean and individual measurement differences.

Figure 2.

Visualization of the measurement accuracy of system 1 and system 2 in rectangle target plots (RTPs). The colored rectangles show the tolerance intervals. Individual absolute and relative measurement differences are indicated as crosses for BG concentrations <100 mg/dL and ≥100 mg/dL. The mean absolute and relative differences are indicated as black dots. Individual measurement differences in the respective BG ranges are plotted horizontally and vertically about the mean difference in each case.

Figure 3 shows the same RTPs with mean measurement differences but without the individual measurement differences. To compare several products or several lots from 1 product, it is recommended not to show the individual measurement differences.

Figure 3.

Visualization of the measurement accuracy of system 1 and system 2 in rectangle target plots (RTPs). For both systems, the tolerance intervals are indicated as rectangles and the absolute and relative mean differences are indicated as black dots.

The examples in Figure 2 and Figure 3 illustrate how an SMBG system’s accuracy can be interpreted using the RTP by looking at the location and the size of the rectangle.

The center of the rectangle is defined by the mean absolute difference (for BG concentrations <100 mg/dL; x-coordinate of the center) and mean relative difference (for BG concentrations ≥100 mg/dL; y-coordinate of the center) between SMBG measurement results and comparison method results. A rectangle whose center is at the origin of the graph indicates no systematic measurement difference (bias). For system 1, the center is closer to the origin than for system 2; thus system 1 shows a lower bias than system 2.

The size of the rectangle is defined by the standard deviation of these differences and by the number of results <100 mg/dL and ≥100 mg/dL. For an identical number of results, a smaller size of the rectangle corresponds to smaller measurement variability.

As the RTP is calculated based on parameters for the systematic measurement error (mean difference) and random measurement errors (standard deviation of differences), it has the quality of a total error analysis.

The system 1 RTP example shows a minimal positive bias at BG concentrations <100 mg/dL and a small negative bias at BG concentrations ≥100 mg/dL. System 2 shows a considerable positive bias at BG concentrations <100 mg/dL and a negative bias at BG concentrations ≥100 mg/dL.

The light and dark gray squares in the RTPs serve as a reference point to assess the system accuracy relative to predefined criteria. For system 1, the tolerance intervals are within the system accuracy limits as stipulated by ISO 15197:2013 (±15 mg/dL for BG concentrations <100 mg/dL and ±15% for BG concentrations ≥100 mg/dL) and are also within the more stringent limits of ±10 mg/dL for BG concentrations <100 mg/dL and ±10% for BG concentrations ≥100 mg/dL. The RTP example of system 2 indicates that the tolerance intervals exceed the ISO 15197:2013 accuracy criteria.

Examples for Comparison of Multiple Test Strip Lots

Additional RTP examples are shown in Figure 4, which compare tolerance intervals for multiple test strip lots of the same SMBG system.

Figure 4.

Rectangle target plots (RTPs) for multiple test strip lots of the same SMBG system.

For an even faster understanding of the plots, the single rectangles can be colored to demonstrate the performance. In the left-hand plot, the light green rectangle represents the tolerance intervals for a test strip lot that meets the ±15 mg/dL/±15% system accuracy limits as stipulated by ISO 15197:2013, while the dark green rectangles represent tolerance intervals for test strip lots that meet more stringent criteria of ±10 mg/dL/±10% for system accuracy.

In the right-hand plot, the orange rectangles show tolerance intervals for test strip lots that meet the system accuracy criteria of ISO 15197:2003, but not of ISO 15197:2013. The red rectangles show tolerance intervals for test strip lots that do neither meet the system accuracy criteria of ISO 15197:2013 nor of ISO 15197:2003.

Different sets of colors or line types can be used, depending on the specific intended use (eg, highlighting compliance with certain criteria, accessibility).

Discussion

The RTP represents a new approach to the visualization of the measurement accuracy of an SMBG system. It can be applied to data obtained in an ISO 15197 accuracy evaluation to provide further information.

System accuracy evaluation in accordance with ISO 15197 is important to demonstrate that SMBG systems comply with established accuracy requirements. In accordance with ISO 15197, accuracy is visualized in a difference plot that shows the deviation between each individual measurement and the respective comparison result. The ISO 15197 difference plot^1,2 and the bias plot according to Bland and Altman,⁴ which is also often used for the presentation of system accuracy, allow for the visualization of a system’s bias and precision over the entire measurement range.

Since accuracy is an important aspect of an SMBG system’s analytical performance which is extensively discussed by experts from science, medicine, and industry and is seriously considered by health care professionals and patients, an intuitive and comprehensible presentation of accuracy data is important. However, standard approaches like difference plots and newer approaches like radar plots^8,9 are difficult to comprehend because of the high number of data points displayed, for example when data from multiple test strip lots are shown. The visualization of data obtained in an ISO 15197 system accuracy evaluation in an RTP is a new approach that provides information about an SMBG system’s measurement accuracy in a manner that is easy to understand, also for the lay person.

The RTP shows whether the tolerance interval that covers 90% of measurements with a confidence of 95% complies with established accuracy limits, for example, the ISO 15197:2013 accuracy limits. According to ISO 15197:2013 accuracy criteria, 95% of measurements must be within the limits; however, the tolerance interval only covers 90% of data points with a 95% probability. Although this proportion of data points could be adjusted to other values as well, changing the proportion of the tolerance interval showed that the 90% level leads to a very good consistency with the 95% accuracy criteria of the ISO 15197:2013. The proportion of 90% was also chosen because in a large data set used for calculation of RTPs the rectangles often exceeded the light gray area of the RTP (highlighting ISO 15197:2013 acceptance criterion A) only unilaterally, so that it was highly likely that only 5% of data were found outside the rectangle and, simultaneously, outside the light gray area. Thus, the RTP is mostly consistent with a total error analysis that has the requirement of 95% of values being within the tolerance. Because the RTP approach is a predictive tool, it is not meant as substitute for system accuracy analysis following ISO 15197 (2003 or 2013 version).

It must be also considered that the basic assumption for the RTP is that the data are normally distributed; this is required for the calculation of k2. Deviation from normal distribution may be determined, for example using the Shapiro–Wilk test. In cases of pronounced deviations from normal distribution, for example caused by outliers, robust estimators¹⁰ like median and Sn (scale estimator) may be used instead of mean and standard deviation for calculation of the tolerance intervals.

The RTP also provides information about the direction of a potential measurement deviation, that is, positive or negative bias, separately for lower (<100 mg/dL) and higher (≥100 mg/dL) BG concentrations. Since the accuracy of SMBG systems is probably not constant over the complete measurement range, such information may allow patients and health care professionals a better interpretation of measurement results of a given system as well as better categorization of different systems. However, the RTP only considers analytical measurement accuracy.

Currently, the percentage of measurements fulfilling the ISO 15197:2013 acceptance criteria for analytical system accuracy (clause 6.3) and the MARD are often used for comparative purposes.^2,11-13 In both cases the performance of a system is reduced to just 1 number. However, the direction of measurement deviation, that is, positive or negative, is not considered. Applying the RTP for comparative purposes, for example by plotting rectangles of different test strip lots or SMBG systems in the same graph, not only provides information about the measurement differences of an SMBG system but also shows whether the direction of measurement differences is comparable or not. Plotting the tolerance intervals separately for low and high glucose concentrations in 1 rectangle allows a differentiated evaluation of a system. As shown in the system 2 RTP example (Figure 3), results for low glucose concentrations are clearly overestimated, while results for higher glucose concentrations are underestimated.

Although the RTP approach was established based on data points from ISO 15197 system accuracy evaluations, it is not intended to replace the analyses recommended in ISO 15197, and the RTP approach is not limited to data sets obtained in this kind of accuracy evaluation.

The RTP is a visualization approach that aims to simplify displayed accuracy information by reducing the displayed information from typically 200 data points to just 1 rectangle. Furthermore, this allows data for several systems or several lots from 1 system to be displayed clearly and concisely in a single graph.

Footnotes

Abbreviations

BG, blood glucose; CLSI, Clinical and Laboratory Standards Institute; ISO, International Organization for Standardization; MARD, mean absolute relative difference; RTP, rectangle target plot; SMBG, self-monitoring of blood glucose

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: PS is an employee of Roche Diabetes Care GmbH. GF is general manager of the Institut für Diabetes-Technologie Forschungs- und Entwicklungsgesellschaft mbH an der Universität Ulm, Ulm, Germany (IDT), which carries out studies on the evaluation of BG meters and medical devices for diabetes therapy on behalf of various companies. GF/IDT have received speakers’ honoraria or consulting fees from Abbott, Bayer, Berlin-Chemie, Becton-Dickinson, Dexcom, LifeScan, Menarini Diagnostics, Novo Nordisk, Roche Diagnostics, Sanofi, and Ypsomed. CS, SP, and CH are employees of IDT. PM is an employee of Roche Diagnostics GmbH.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by Roche Diagnostics GmbH, Mannheim, Germany.

References

International Organization for Standardization. In vitro diagnostic test systems—requirements for blood-glucose monitoring systems for self-testing in managing diabetes mellitus. EN ISO 15197:2003 E.

Clinical and Laboratory Standards Institute. EP09-A3. Measurement procedure comparison and bias estimation using patient samples; approved guideline—third edition. 2013.

Bland

Altman

. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307-310.

Passing

Bablok

. A new biometrical procedure for testing the equality of measurements from two different analytical methods. application of linear regression procedures for method comparison studies in clinical chemistry, part I. J Clin Chem Clin Biochem. 1983;21(11):709-720.

Wilmoth

. The relationships between common measures of glucose meter performance. J Diabetes Sci Technol. 2012;6(5):1087-1093.

NIST/SEMATECH E-handbook of statistical methods, 7.2.6.3. Tolerance intervals for a normal distribution. October 30, 2013. Available at: http://www.itl.nist.gov/div898/handbook/prc/section2/prc263.htm. Accessed June 8, 2015.

Pardo

Pflug

Dunne

Simmons

. Using radar plots to evaluate five blood glucose monitoring systems for accuracy and precision in 2014 diabetes technology meeting abstracts. J Diabetes Sci Technol. 2015;9(2):430.

Pardo

Pflug

Dunne

Simmons

. Evaluating the accuracy and precision of six blood glucose monitoring systems using radar plots in 2014 diabetes technology meeting abstracts. J Diabetes Sci Technol. 2015;9(2):431.

10.

Rousseeuw

Croux

. Alternatives to the median absolute deviation. J Am Stat Assoc. 1993;88(424):1273-1283.

11.

Klaff

Brazg

Hughes

. Accuracy evaluation of contour next compared with five blood glucose monitoring systems across a wide range of blood glucose concentrations occurring in a clinical research setting. Diabetes Technol Ther. 2015;17(1):8-15.

12.

DeSalvo

Shanmugham

Wilson

Buckingham

. Accuracy evaluation of blood glucose monitoring systems in children on overnight closed-loop control. J Diabetes Sci Technol. 2014;8(5):969-973.

13.

Halldorsdottir

Warchal-Windham

Wallace

Pardo

Parkes

Simmons

. Accuracy evaluation of five blood glucose monitoring systems: the North American Comparator Trial. J Diabetes Sci Technol. 2013;7(5):1294-1304.