Abstract
Background:
Anecdotal blood glucose assessments conducted by health care professionals (HCPs) in the field have highlighted differences in results when methodology used is not according to best practices for measuring blood glucose. This study assessed the impact on accuracy of blood glucose measurements when methodology deviates from the recommended study design and recommended reference instrument.
Methods:
Adults with type 1 or type 2 diabetes provided capillary and venous blood samples for accuracy assessments using OneTouch® Verio® (Verio) and OneTouch® Ultra 2® (Ultra) blood glucose meters (BGM) and two different reference instruments.
Results:
Increases in mean bias were observed when comparing capillary to venous samples tested on the BGMs and the recommended reference instrument. Mean bias was even greater when a hospital blood glucose analyzer was used to measure venous plasma glucose. Increases in mean bias observed for Ultra BGM when testing venous blood on the meter compared to the recommended reference instrument was likely due to the interfering effects of low oxygen levels in the venous blood sample. Conversely, Verio meters, which are insensitive to low oxygen levels, showed little difference from baseline when testing venous blood on the meter compared to results from the same venous sample measured on a reference instrument.
Conclusions:
Deviations from the best practice study design of comparing capillary blood glucose results tested on the blood glucose meter with the manufacturer’s stated reference instrument will affect accuracy of blood glucose measurements.
It has long been recognized that accuracy of blood glucose monitors (BGM) can be influenced by multiple factors which can lead to the introduction of variation and even inaccurate results (Ginsberg; 1 Mahoney and Ellison; 2 Demircik et al 3 ). Checklists have been developed to distinguish device error from protocol specific bias and random patient interferences; papers have reviewed the assessment of analytical accuracy studies and the assessment of the influence of interferents (Thorpe; 4 Erbach et al 5 ). Guidance has been developed for comparison of different sample types (Swaminathan et al 6 ); observations have been conducted on accuracy highlighting the importance of sample processing time and reference instrument selection (Schrot et al; 7 Twomey 8 ); and information has been published regarding the ineffectiveness of sodium fluoride as an inhibitor of glycolysis (Gambino 9 ). Best practice suggests that the reference instrument the manufacturer states the product has been calibrated against should be used when testing a BGM for accuracy. However, this is not stipulated in ISO15197 standards (ISO15197:2003; 10 ISO15197:2015 11 ). But, comparing capillary whole blood (fingerstick) to the same capillary sample measured on the reference instrument is recommended by ISO15197:2015 for these standards. However, it is recognized that this method can often be difficult or impractical to implement in the clinical setting.
There is currently an increased focus on the accuracy assessment of BGM with the publication of FDA guidance for assessment of self-monitoring blood glucose devices, 12 where the expectation of performance is greater than ISO 15197: 2015 (Freckmann et al 13 ). In addition, the In-Vitro Diagnostic Regulation (IVDR) 14 mandates that manufacturers implement a review of clinical data as part of postmarket surveillance activities.
As part of addressing the IVDR requirements, notified bodies will conduct their own accuracy assessments of product performance where no requirements on test methodology are provided. This could lead to independent testing institutions developing their own test methods different from those used by the manufacturer resulting in increased variation in blood glucose accuracy measurement. This could also be true of assessment of the device under review as per country specific requirements for product registration.
This study explores the impact of different study designs, some nonstandard, on accuracy performance of OneTouch Ultra and Verio BGM systems.
Methods
This open, nonrandomized, single visit clinical evaluation was conducted at clinical sites at the Highland Diabetes Institute (Inverness, UK), the Royal Infirmary of Edinburgh (Edinburgh, UK), and Heartlands Hospital (Birmingham, UK). Testing was performed between August 10 and August 31, 2016.
The flow of activities during a single clinical site visit was as follows:
Study staff provided potential subjects with a description of the study, including requirements for participation, specific study activities and procedures, and informed consent was obtained.
Subjects completed an in-clinic orientation session which included one practice fingerstick with a microtainer lancet (Becton Dickinson, Oxford, UK), according to the lancet labeling and had an opportunity to familiarize themselves with the BGM system and owner’s guides. No practice glucose tests with the BGMs were performed.
Venous blood collection was performed by health care professionals (HCPs), either trained research nurses or phlebotomists. Subjects were not required to be fasted. Samples were collected in tubes coated with lithium heparin and were tested on the study BGM. A Siemens 238 blood gas analyzer was used to measure the oxygen level of the sample. The venous blood sample was centrifuged immediately, and within one minute after centrifugation the plasma fraction was tested in duplicate on the YSI 2300 reference instrument.
As soon as possible, but no longer than 5 minutes after venous collection, the subject (a) lanced a fingertip using a microtainer lancet, (b) obtained a drop of blood large enough for 4 meter tests; and (c) performed a self-test using two Verio and two Ultra meters.
Study staff observed and evaluated the subject’s self-testing technique and recorded the evaluations, including any relevant comments.
Study Staff then took the subjects’ finger and applied blood to glucose test strips in two Verio and two Ultra meters and collected blood from the same finger puncture for hematocrit, oxygen and reference plasma glucose testing. Capillary samples for testing on the YSI 2300 and Siemens Blood Gas Analyzer were collected in tubes coated with lithium heparin.
YSI 2300 reference analyzers at all 3 sites were situated in the same room as subject testing to ensure centrifugation and testing could occur immediately after sample collection to reduce any effect of glycolysis. Hospital analyzers used were Siemens ADVIA 2400 at one site and Abbott Architect C16000 at 2 sites. Each analyzer used hexokinase enzyme methodology for the glucose assay and duplicate tests were performed for each sample. All venous samples collected for testing in hospital laboratories were per hospital procedure and the processing of study samples was included in other hospital sample testing to ensure no influence of the study protocol on the test process. Calibration, maintenance, and use of the hospital analyzers were per individual hospital protocol and traceability and performance requirements were maintained throughout study conduct. Trueness and precision of the YSI 2300 glucose analyzer was verified during the test process by daily quality control measurements which followed LifeScan internal standard operating procedures.
Blood glucose test strips lots were obtained from LifeScan Scotland Ltd. Strip lot numbers for Verio were 40230246/4005113 and Ultra were 4030703/4030724.
Accuracy comparisons were examined in this study as presented in Table 1: Group 1 used capillary whole blood (fingerstick) samples on meters versus capillary plasma (taken from the same fingerstick) on reference instruments. Sampling was performed by subjects and HCPs. Group 2 used capillary whole blood (fingerstick) samples on meters versus venous plasma on YSI and hospital analyzer reference instruments. Sampling was performed by subjects and HCPs. Group 3 used venous whole blood on meters versus venous plasma (taken from same venous sample) on both YSI and hospital reference instruments.
Study Groups for Blood Glucose Accuracy Evaluations.
Hospital, Abbott Architect C16000 and Siemens ADVIA 2400 (see Methods); Ultra, OneTouch Ultra blood glucose strip; Verio, OneTouch Verio blood glucose strip; YSI, Yellow Springs Instruments, YSI 2300.
Statistical Analyses
Continuous demographic variables were described by mean and range (min to max) and the count (percentage) provided for each categorical variable. Accuracy was determined by assessing bias from reference against appropriate specification limits. Minitab 17.0 and SPSS V21.0 were used for all analyses.
Results
Subjects
Baseline characteristics of all 120 subjects are shown in Table 2. Approximately half were male and half had type 1 diabetes. A broad age range of subjects (20 to 78 years) were recruited with a mean age of 52 years. Likewise, time from diagnosis varied from 1 to 60 years with a mean of 20 years. Of subjects, 75% reported performing self-monitoring of blood glucose (SMBG) two or more times per day, and 78% of all subjects used insulin therapy. Half of subjects had an education level of college or greater.
Subject Demographics.
A total of 109 subjects produced valid capillary samples and 120 subjects had valid venous samples. Of the 11 capillary samples that were excluded from the study, 7 had oxygen levels outside of the manufacturer’s specified range of 7-12 kPa for Ultra, where oxygen levels were just outside the range; 3 had sampling errors; and 1 sample was not tested because YSI results were outside of the manufacturer’s specified drift criteria.
Group 1: Capillary Result on Meter Versus Capillary Result on YSI—Subject and HCP Testing
The bias plots for Ultra and Verio products versus YSI reference instrument when the subject performs the fingerstick testing are shown in Figures 1A and B, respectively. Accuracy versus ISO 15197:2015 (Verio) and ISO 15197:2003 (Ultra) are shown for both subject and HCP testing in Table 3. The results demonstrate that subject and HCP testing of capillary blood using Verio and Ultra meters compared to values using the YSI reference instrument meet the appropriate ISO 15197 accuracy acceptance criteria. In all cases, more than 95% of the results were within the appropriate ISO accuracy criteria. Verio was evaluated by ISO15197:2015 specifications; Ultra was evaluated by ISO15197:2003 specifications as per product clearance for each device. This comparison is considered the standard test method (best practice) and is the baseline for comparison to the other study designs in this study.
Difference plot of capillary result on meter versus capillary result on YSI for lay user performance. (A) Ultra product. Bias was calculated as an absolute bias for all bias results <75 mg/dL and percentage bias for all bias results ⩾75 mg/dL. Of results, 97.7% were within ± 15 mg/dL or 20% of reference. Average bias was 1.1. (B) Verio product. Bias was calculated as an absolute bias for all bias results <100 mg/dL and percentage bias for all bias results ⩾100 mg/dL. Of results, 95.9% were within ± 15 mg/dL or 15% of reference. Average bias was 5.5.
Accuracy of Blood Glucose Testing: Capillary Versus Capillary; Capillary Versus Venous; and Venous Versus Venous Comparisons.
Hospital, Abbott Architect C16000 and Siemens ADVIA 2400; YSI, Yellow Springs Instrument 2300.
Cut point for analysis is 100 mg/dL.
Cut point for analysis is 75 mg/dL.
Group 2A/B: Capillary Result on Meter Versus Venous Result on YSI—Subject and HCP Testing
The capillary samples used in Groups 1A and 1B were compared to the venous blood samples taken from each subject in Groups 2A and 2B. Accuracy results per the appropriate ISO 15197 guidance criteria for Verio and Ultra meters are shown in Table 3. The results demonstrate a reduced ability to meet the accuracy criteria when capillary blood tested on the meter is compared to testing the venous sample on the YSI device. For Verio meters, <95% of the samples were within the accuracy specifications when compared to venous samples. This was true whether the capillary samples were collected by the subject or HCP. Results for Ultra meters were close to 95%.
Group 2C/D: Capillary Result on Meter Versus Venous Result on Hospital Reference—Subject and HCP Testing
This comparison measures the capillary samples tested on the meters versus venous blood glucose as measured using the hospital analyzer. The accuracy results against the appropriate ISO 15197 guidance are shown in Table 3. The results demonstrate that the subject and the HCP testing of capillary blood comparing the venous sample tested on the YSI does not meet the appropriate ISO 15197 acceptance criteria for Ultra or Verio. BGM accuracy is reduced when comparing capillary samples tested versus venous samples tested on a reference instrument against which the product was not calibrated, that is, the hospital reference instrument.
Group 3: Venous Result on Meter Versus Venous Result on YSI and Hospital Reference Analyser
For group 3A, venous blood tested on meters was compared to venous blood tested on the YSI. Both Verio and Ultra met the acceptance criteria as per the appropriate ISO 15197 guidance (Table 3). Verio had 97.5% (234/240) of results within specification and Ultra had 97.1% (233/240) results within specification. The results obtained for Group 3B, where the reference testing occurred using the hospital reference analyzer did not meet the acceptance criteria as per the appropriate ISO 15197 guidance (Table 3). Verio had 94.2% (224/240) and Ultra had 93.3% (224/240) of the meter results within the guidance when meter results were compared to a hospital reference instrument.
Comparison of All Study Groups
An interval plot as shown as Figure 2 demonstrates the mean change in bias for all groups and both BGM product types. The change in mean bias from baseline for subject testing on Verio versus venous blood tested on YSI was 2.5 and the difference for HCP testing was 2.6. The change in mean bias from baseline for subject testing on Ultra versus venous blood tested on YSI was 2.6 and the difference for HCP testing was 2.5. The change in mean bias from baseline for subject testing for Verio and Ultra meters versus venous blood tested on Hospital Reference was 3.5 and 3.3, respectively.
Mean bias for all comparison groups and meter types. Each point represents the mean bias ± 95% confidence interval of two lots calculated as an absolute mean bias for all bias results <100 mg/dL plus mean percentage bias for all bias results ⩾100 mg/dL. Open symbols represent HCP facilitated testing and closed symbols represent self-testing by the study subjects. Self, subject performs test on the meter; HCP, health care professional performs the test on the meter; Ultra, meter uses OneTouch Ultra test strips; Verio, meter uses OneTouch Verio test strips. Cap-Cap, capillary whole blood as sample source for meter and reference instrument; Cap-Ven, capillary whole blood as sample source for meter and venous blood as sample source for reference instrument; Ven-Ven, venous blood as sample source for meter and reference instrument; YSI, Yellow Springs Instruments 2300; Hosp, Abbott Architect C16000 and Siemens ADVIA 2400. Group nomenclature follows description in Tables 1 and 3.
When comparing venous blood tested on the meter with venous blood tested on the YSI reference instrument, the change in mean bias was 0.1 for Verio and 3.4 for Ultra. The change increases when comparing the venous blood tested on the meter to venous blood tested on the hospital analyzer. The mean bias for this comparison was 1.2 for Verio and 4.8 for Ultra.
When comparing all study groups for Ultra the bias increases the further one moves from the best practice study design of testing capillary blood on the meter and comparing the same capillary sample on the appropriate reference (YSI). The mean bias response increases even further when venous blood is tested on the blood glucose meter.
For Verio, a similar increase in bias is observed when the method deviates from the best practice study design of testing capillary blood on the meter compared to the same capillary sample on the appropriate reference instrument. Comparing capillary samples on the BGM with venous samples on a hospital analyzer produced the greatest bias.
The bias is not affected to the same extent when testing a venous sample on either reference instrument compared to a venous sample tested on the blood glucose meter.
Comparing Effects of Oxygen Interference on Ultra
Because of improved product technology, Verio has no sensitivity to the oxygen level of the blood sample and therefore does not require a stated oxygen test range. This is not true for Ultra where the stated operating range is 7-12 kPa. For this reason, Ultra is not intended for use with venous blood samples. For capillary blood samples tested on Ultra, changes in bias across the oxygen range were observed. A mean change in bias of −2.1 was observed across the operating range of 7-12 kPa oxygen, the operating range of Ultra. However venous blood samples with oxygen <7 kPa showed an increase in mean bias of 3.9, a statistically significant difference from the mean bias in the 7-12 kPa oxygen range (P < .05). This is not unexpected because most unmanipulated venous sample oxygen levels are outside of the claimed operating range of the Ultra system.
Comparing Reference Instrument Performance
There was a change in bias when using the two reference instruments (YSI and hospital) across the glucose range when comparing the glucose results of the same venous blood samples. For samples <250 mg/dL, the mean bias difference was slightly positive whereas a negative bias occurred for samples >250 mg/dL (Figure 3).
Bias of difference between YSI and hospital analyser results for venous blood samples. Bias was calculated as an absolute bias for all bias results <100 mg/dL and percentage bias for all bias results ⩾100 mg/dL. The hospital analyzers used were Siemens ADVIA 2400 at one site and Abbott Architect C16000 at 2 sites.
Using a regression calculation to estimate bias at 100 mg/dL glucose showed that the difference in response between the analyzers (ie, bias) was 3.7 mg/dL. This bias became increasingly more negative at 400 mg/dL glucose where the mean bias was −5.3%. At more extreme glucose levels of 600 mg/dL, this negative bias was increased further to −11.3%. This highlights that the performance of the reference instrument changes across the blood glucose range of samples tested.
Safety and Tolerability
There were no adverse and no serious adverse events during the conduct of the study.
Discussion
When Verio and Ultra products are tested utilizing the manufacturers’ stated recommended capillary test design (ie, capillary blood tested on the BGM compared to the same capillary sample tested on the YSI reference instrument), the results meet the appropriate ISO 15197 accuracy acceptance criteria for glucose measurement. In addition, for both types of strips, accuracy results are independent of whether an HCP or lay user is conducting the test. This demonstrates that there is little impact of user testing on meter results when conducted using the appropriate testing design.
When venous blood is used as the BGM test sample, the mean bias increases for both product types compared to using the recommended capillary sample. This is especially true for Ultra strips which are more sensitive to blood oxygen concentration and accuracy using venous blood may not meet the appropriate ISO 15197 acceptance criteria. This demonstrates that when testing a BGM system, care should be taken to identify the intended sample type for the device. In addition, accuracy is reduced if capillary or venous samples are tested on the BGM against a reference instrument with which the product is not calibrated. Once again, this highlights the risk of using methodology that has not been recommended by the manufacturer to assess accuracy of blood glucose meters.
Careful consideration should be taken to understand the differences in accurately measuring glucose levels in capillary and venous blood samples since blood oxygen levels can have an interfering effect on BGM results. Results from this study confirm that testing samples on BGM where the oxygen level is outside the stated operating range for oxygen sensitive products such as Ultra increases the blood glucose result on the meter, impacting the bias compared to the reference instrument. Because anecdotal evidence from the field indicates that assessments of this type do occur, it is important to highlight the impact of sample type on BGM accuracy.
When examining Verio and Ultra strip results obtained using reference instruments (eg, hospital lab analyzers) different than those against which the meter was calibrated (eg, YSI), results were more positively biased. This is likely due to the difference in enzyme methodology since literature (Twomey 8 ) suggests that the hexokinase method is more positively biased for plasma glucose than glucose oxidase methodology. We also demonstrated that results obtained when comparing YSI to the other hospital analyzers depend on the glucose level. For samples ⩽250 mg/dL, the mean bias difference of the YSI compared to the two hospital analyzers was slightly positive whereas for samples >250 mg/dL bias becomes increasingly negative. This change in bias could be attributed to the capabilities of the instrument in accurately detecting glucose levels as they increase in concentration. Therefore, one cannot assume that different reference instruments will give the same results, especially as glucose levels change. Careful consideration should be given to which reference instrument is used and its capability to provide accurate results across the entire claimed glucose range of the BGM under evaluation. Ideally, blood glucose meters should only be compared with the reference instrument against which it has been calibrated. It is understood, however, that this may be difficult in a hospital setting. However, a thorough understanding of reference instrument performance is critical to understanding any potential impact on BGM bias. It is also critical to understand performance of the reference analyzer relating to imprecision, calibration, and traceability (eg, to NIST standards) at the time of the evaluation.
Conclusion
Altering specific variables from best practice clinical accuracy study design such as comparing capillary blood tested on the blood glucose meter to venous blood tested on the reference analyzer or comparing venous blood tested on the blood glucose meter compared to venous blood tested on the reference analyzer can produce significantly different results depending on the blood glucose monitoring system under evaluation. In addition, the difference in results for oxygen sensitive blood glucose monitoring systems is more pronounced when testing venous blood. The choice of laboratory reference instrument should be carefully considered in the clinical accuracy study design.
Footnotes
Acknowledgements
The authors would like to thank Barry Irvine for leadership of the study setup and conduct, and employees at Heartlands Hospital Birmingham (Clara Nicholls, Chhaya Sankhalpara, Misaba Altaf, Cheryl Davies), Edinburgh Royal Infirmary (Constance Jones, Jane Irwin, Alison Sudworth), and Highland Diabetes Institute (Shannon Proctor, Emma Coates, Jill Jones, Fiona Barrett) for patient interaction and study conduct.
Abbreviations
AHA, antihyperglycemic agents; BGM, blood glucose monitor; HCP, health care professional; IVDR, In-Vitro Diagnostic Regulation; SMBG, self-monitoring of blood glucose; Ultra, OneTouch Ultra 2; Verio, OneTouch Verio; YSI, Yellow Springs Instrument.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: KM is an employee of LifeScan Scotland, Ltd. LBK is an employee of LifeScan, Inc. HC is a contractor of LifeScan Scotland, Ltd.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by LifeScan, Inc.