Application of the sigma metrics to evaluate the analytical performance of cystatin C and design a quality control strategy

Abstract

Background

Sigma metrics are commonly used to evaluate laboratory management. In this study, we aimed to evaluate the analytical performance of cystatin C using sigma metrics and to develop an individualized quality control scheme for cystatin C concentrations.

Methods

Bias was calculated based on the samples used for the external quality assessment. The coefficient of variation was calculated using six months of internal quality control measurements at two levels, and desirable specification derived from biological variation was used as the quality goal. The sigma value for cystatin C was calculated using the above data. The internal quality control scheme and improvement measures were formulated according to the Westgard sigma standards for batch size and quality goal index.

Results

The sigma values for cystatin C, for quality control levels 1 and 2, were 3.04 and 4.95, respectively. The 1_3s/2_2s/R_4s/4_1s/6_x multirules (n = 6, R = 1), with a batch size of 45 patient samples, were selected as the internal quality control schemes for cystatin C. With different concentrations of cystatin C, the power function graph showed a probability for error detection of 94% and 100% and a probability for false rejection of 4% and 2%, respectively. According to the quality goal index of cystatin C, its precision needs to be improved.

Conclusions

With a ‘desirable’ biological variation of 6.50%, the Westgard rule 1_3s/2_2s/R_4s/4_1s/6_x (n = 6, R = 1, batch size of 45) with high efficacy for determining the detection error is recommended for individualized quality control schemes of cystatin C.

Keywords

Cystatin C sigma metrics biological variation quality goal index

Introduction

Cystatin C is an endogenous marker for the glomerular filtration rate, and it is widely used during the early diagnosis of compromised renal function and the evaluation and monitoring of renal function recovery.^1–3 Therefore, the accuracy of cystatin C measurements can directly affect the diagnosis and evaluation of clinically related diseases. However, at present, no standard reference measurement procedures have been established for cystatin C.⁴ The European Institute for Reference Materials and Measurements released an international reference substance for cystatin C (ERM-DA471) in 2010, but its labelled value of 5.48 ± 0.15 mg/L is significantly higher than the level used for medical decision-making (the reference range for cystatin C is 0.61–1.01 mg/L).⁵ Hence, the application of this reference substance for cystatin C measurements has been limited.

He et al.⁶ conducted a survey in China from 2014 to 2017 examining the internal quality control (IQC) standards for cystatin C measurements. These authors showed that the results generated by different laboratories varied greatly due to the lack of standardized quality control (QC) materials and detection systems. Eckfeldt et al.⁷ investigated the recommendations made by the College of American Pathologists for the detection of cystatin C. Using the commutability quality assessment substances CYS-WC1 and CYS-WC2, at concentrations of 0.96 mg/L and 2.37 mg/L, respectively, the maximum differences in measurements made by 123 laboratories for these different reagent groups were 29% and 36%, respectively. In view of the above findings, the objective evaluation of the accuracy and analytical performance of cystatin C detection methods is very important.

The six-sigma theory, as an important method of laboratory quality management, was first applied to the medical field by Nevalainen et al.⁸ Sigma metrics have also been commonly applied to the fields of clinical haematology and clinical chemistry to assess the QC process for different analytes. Sigma metrics have played significant roles in the evaluation of analytical performance, the formulation of IQC schemes and the assessment of continuous quality improvements.^9–12 Therefore, this study aimed to apply sigma metrics to the objective evaluation of the analytical performance of cystatin C measurements and to formulate an individualized QC scheme for cystatin C.

Materials and methods

Instruments and reagents

Cystatin C was measured by latex-enhanced immunoturbidimetry, using an AU5800 automatic biochemical analyser and corresponding reagents (Beckman Co., Kraemer Boulevard Brea, USA). The calibrator (lots: AUZ7167; traceability: ERM DA471/IFCC) was set to the original matched reagent. IQC materials were obtained from Qualab (lots 1:379C4E4 and 2:379EBF4; Shanghai, China). External quality assessment (EQA) samples (lots: 202015 and 202011) were provided by the National Center for Clinical Laboratories (NCCL, Beijing, China).

Assessment of measurement performance

Calculation of sigma metrics

Sigma metrics were calculated using the following formula: sigma = [TEa (%) − |bias (%)|]/CV (%). Allowable total error (TEa) was derived from the quality specification of the ‘desirable’ biological variation that was established by the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM).¹³ CV was the coefficient of variation for the IQC data for a consecutive six months (January to June 2020). According to the two levels of IQC, the CVs of cystatin C (CV1 and CV 2) were calculated using the following formula

C V = \sqrt{\frac{C V_{Jan}^{2} + C V_{Feb}^{2} + C V_{Mar}^{2} + C V_{Apr}^{2} + C V_{May}^{2} C V_{Jun}^{2}}{6}}

Bias was evaluated based on the data of EQA. According to the two levels of IQC, two EQA samples with similar levels were selected. Bias 1 was matched with CV 1 and bias 2 was matched with CV 2. The target value of the bias was calculated with the average value of all participating laboratories (n = 60) using the same detection system (the detection platform of the AU5800 automatic biochemical analyser). The EQA samples were measured five times repeatedly, and five measured values were obtained. Then, the percentage difference between the measured value and the target value was calculated, and the average value of the absolute value of the percentage difference was taken as the bias (Bias 1 and Bias 2). Biases were calculated using the following formula

\begin{array}{l} B ias = \frac{Valu e_{measured} - c}{C V_{Jan}^{2}} \times 100 % \\ B i a s_{average} = \frac{\begin{array}{l} | B i a s_{F i r s t} | + | B i a s_{S e c o n d} | + | B i a s_{T h i r d} | \\ + | B i a s_{F o u r t h} | + | B i a s_{F i f t h} | \end{array}}{5} \end{array}

The purpose of matching the IQC concentration of cystatin C with the EQA sample concentration is to ensure the accuracy of the sigma metrics (Tables 1 and 2).

Table 1.

CV determined by IQC for cystatin C (January to June 2020).

Parameter	Levels of IQC (mg/L)	CV values for different months (%)
Parameter	Levels of IQC (mg/L)	January (n = 31)	February(n = 29)	March (n = 31)	April (n = 30)	May (n = 31)	June (n = 30)	Average (n = 182)
CV 1	0.51	1.94	1.68	1.76	2.06	1.87	1.73	1.84
CV 2	1.65	1.01	1.06	1.20	0.92	1.03	1.34	1.10

CV: coefficient of variation; IQC: internal quality control.

Table 2.

Bias of EQA for cystatin C (2020).

Parameter	Lot of EQA	Target value (mg/L, n = 60)	Measured value (repeat five times, mg/L)	Bias_average (%)
Bias 1	202015	0.88	0.86	0.88	0.89	0.88	0.89	0.91
Bias 2	202011	1.71	1.72	1.76	1.73	1.71	1.70	1.05

Parameter

Lot of EQA

Target value (mg/L, n = 60)

Measured value (repeat five times, mg/L)

Bias_average

(%)

First

Second

Third

Fourth

Fifth

Bias 1

202015

0.88

0.86

0.88

0.89

0.88

0.89

0.91

Bias 2

202011

1.71

1.72

1.76

1.73

1.71

1.70

1.05

EQA: external quality assessment.

Standardized sigma performance verification diagram for cystatin C

A standardized sigma performance verification diagram was constructed (Figure 1), according to the instructions available on www.clinet.com.cn. The name of the analyte, TEa, bias and CV values were entered in order. In Figure 1, the abscissa depicts CV/TEa(%), and the ordinate depicts bias/TEa(%), which was divided into six regions that represent the different analytical performances of the analyte. From the bottom left to the top right, Figure 1 depicts sigma values greater than 6 (world-class), between 5 and 6 (excellent), between 4 and 5 (good), between 3 and 4 (marginal), between 2 and 3 (poor) and less than 2 (unacceptable).¹⁴

Figure 1.

Standard sigma performance verification chart for cystatin C.

IQC scheme and improvement measures

An individualized IQC scheme for cystatin C was formulated by applying Figure 2 (Westgard sigma rules with batch size), which was obtained from the website http://www.clinet.com.cn/sigmapv/#sgm4. In addition, the power function graph, provided by Q-expert Quality Management Software (Qualab Biotech Co., Shanghai, China), was used to evaluate the performance characteristics of the QC method.¹⁵ Moreover, the quality goal index (QGI) for cystatin C was applied, which is calculated for sigma values less than 6 to determine the causes of poor performance and the corresponding improvement measures. QGIs were calculated using the formula: QGI=bias(%)/1.5×CV(%). A QGI value of less than 0.8 indicates that precision is the primary factor affecting analytical performance and it must be improved prior to the implementation of any other improvement measures. A QGI value greater than 1.2 indicates that accuracy is the primary factor affecting analytical performance and must be improved prior to the implementation of any other improvement measures. A QGI value between 0.8 and 1.2 indicates that both precision and accuracy are affecting analytical performance and both must be improved.¹⁶

Figure 2.

Westgard sigma rules with batch size.

Results

Distribution of sigma metrics for different concentrations of cystatin C

Using the desirable specification derived from the biological variation as the quality goal for our laboratory, we found differences in the sigma values calculated for different concentrations of cystatin C. At the level of QC 1, the sigma value of cystatin C was 3.04, with an analytical performance of ‘marginal’. However, at the level of QC 2, the sigma value of cystatin C was 4.95, with an analytical performance of ‘good’ (Figure 1 and Table 3).

Table 3.

Distribution of sigma values for two concentrations of cystatin C.

Analyte	TEa (%)	QC 1			QC 2
Analyte	TEa (%)	CV 1 (%)	Bias 1 (%)	Sigma	CV 2 (%)	Bias 2 (%)	Sigma
Cystatin C	6.50	1.84	0.91	3.04	1.10	1.05	4.95

CV: coefficient of variation; TEa: allowable total error; QC: quality control.

IQC strategy and improvement measures for cystatin C

Individualized IQC schemes for cystatin C were designed based on its analytical performance and the Westgard sigma rules with the batch size. With a sigma value of 4.95 for cystatin C at the QC 2 level, the 1_3s/2_2s/R_4s/4_1s multirules (n = 4, R = 1), with a batch size of 200 patient samples, was recommended. However, with a sigma value of 3.04 for cystatin C at the QC 1 level, the 1_3s/2_2s/R_4s/4_1s/6_x multirules (n = 6, R = 1), with a batch size of 45 patient samples, were used for the IQC scheme of cystatin C. Besides, we also used the power function graph to evaluate the performance of the IQC scheme for cystatin C. Our study showed a probability for error detection (Ped) of 94%, 100% and a probability for false rejection (Pfr) of 4%, 2% based on the levels of QC1, QC2 respectively. In addition, the results also indicated that QGI values of cystatin C for the QC 1 and 2 levels were 0.33 and 0.64, respectively, indicating that the precision was the primary factor affecting analytical performance. Therefore, precision requires improvement prior to any other improvement measures (Figures 2 and 3).

Figure 3.

Power function graph for different levels of cystatin C. (a) quality control 1, using 1_3s/2_2s/R_4s/4_1s/6_x multi-rules; (b) quality control 2, using 1_3s/2_2s/R_4s/4_1s/6_x multi-rules.

Discussion

The sigma symbol represents the ‘standard deviation’ in mathematical statistics, and its value usually reflects the testing quality level.¹⁷ Six sigma represents a ‘world-class’ quality level, indicating that for every one million products (or services), only 3.4 defects can be detected, which is the expected quality goal.¹⁷ The calculation of sigma depends on three variables: TEa, precision and accuracy. TEa represents the quality goals selected by the laboratory, and previous studies have shown that there are significant differences in sigma levels for the quality goals used for the same analytes in the same laboratories.^18,19 Thus, based on the model developed at the Milan Conference,²⁰ the desirable specification derived from biological variation was selected as the quality goal for cystatin C, and the sigma values were calculated based on the concentrations of two IQC materials. Our study found differences in the analytical performance for cystatin C at different concentrations. When cystatin C was at QC 2 levels, analytical performance reached a sigma value close to 5. However, analytical performance decreased slightly at the QC 1 level, with a sigma value between 3 and 4. Wang et al.²¹ applied sigma metrics to the evaluation of analytical performance for urinary albumin and revealed significant performance differences when the analyte levels differed. Furthermore, Đido et al.²² applied sigma metrics to the evaluation of the analytical performance for serum enzymes and showed significant differences in the sigma values for analytes at different concentrations.

Our research findings are consistent with the above studies. Hence, our study confirmed that when sigma metrics are used to evaluate the analytical performance for an analyte, the sigma metrics should be adjusted according to the concentration of the analyte in question, especially analyte levels associated with medical decision-making, to ensure the objective and reasonable evaluation of analytical performance. Our study investigated the analytical performance of cystatin C at different concentrations by applying a standardized sigma performance verification chart. The chart is simple and clear, which can help laboratories easily determine their analytical performance for cystatin C.

Once a quality goal has been determined by a laboratory, the sigma value should reflect the changes in precision and accuracy.^23,24 Precision is usually expressed as the long-term cumulative CV in each laboratory, which is a feasible assessment for conventional laboratories to perform using data over a six-month period.^23,24 For assessments of accuracy, reference measurement procedures or reference materials are preferred.²⁵ Because a reference measurement procedure has not yet been established for cystatin C and the concentration of its reference substance is much higher than the upper limit of its reference interval, EQA samples were used as candidate materials for the evaluation of cystatin C measurement accuracy. Based on the concentrations of two QC levels of cystatin C, we selected two EQA samples that had similar concentrations to the IQC material. This approach ensured that the sigma metrics applied to cystatin C at different concentrations were objective and reasonable.

Westgard et al.^26–29 have long been engaged in the field of laboratory quality management and have updated many quality control strategies. However, laboratories do not apply laboratory quality control measures uniformly and the implementation of laboratory quality control measures can often be associated with difficulty and confusion.^30,31 When a laboratory does not evaluate the analytical performance of each analyte using the sigma metric approach, the true capacity of the laboratory cannot be accurately reflected, and they may implement inappropriate IQC schemes.

The IQC schemes used in our laboratory applied the 1_3s/2_2s/R_4s/4_1s/10_X multirules for all analytes, but detection efficiency was not being evaluated effectively. Therefore, whether the multirules met the requirements for testing quality was impossible to determine. Based on the sigma metrics for cystatin C and the Westgard sigma rules with batch size, this study established an individualized IQC scheme for cystatin C, which applied the 1_3s/2_2s/R_4s/4_1s/6_x multirules and a batch size of 45 patient samples.

Westgard et al.²⁷ also showed that using different batch sizes for different sigma values facilitates the development of appropriate QC strategies for laboratories and reduces patient risk. The performance characteristics of the QC method included Ped and Pfr. The QC method generally meets the requirements for testing quality when Ped is greater than 90% and Pfr is less than 5%.^28,29 Our study showed that when cystatin C was applied to the above QC scheme, the Ped and Pfr values of QC 1 and 2 levels all met the requirements for testing quality. In this study, the IQC scheme for cystatin C was formulated by applying sigma metrics, which significantly increased Ped, reduced Pfr, reduced costs and reduced time waste associated with the processing of false out-of-control signals, which will reduce the time needed for patients to obtain reports.

In addition, because the sigma value for cystatin C at the QC 1 level was lower than 6, based on the QGI calculation, precision must be improved. Verma et al.³² evaluated analytical performance for 16 clinical analytes using sigma metrics and developed quality improvement measures based on QGI calculations. Therefore, to improve the analytical performance for cystatin C, we plan to establish the following quality improvement programs: (1) strengthen the maintenance and repair procedures for the automatic biochemical analyser; (2) improve the training of team members to ensure that QC products, calibrators and instruments meet specifications; (3) select reagents from different manufacturers for comparative testing, objectively evaluate their analytical performance and select kits whose quality meets the requirements and (4) dynamically evaluate the sigma values for each analyte to ensure the continuous improvement of laboratory testing capabilities.

However, the limitations of our study merit discussion. First, TEa (quality goal) is an important parameter that affects sigma metrics. Therefore, setting the TEa value in each laboratory is very important. If the selected TEa value is too large (minimum specification derived from the biological variation for cystatin C was 9.70%) or too small (optimum specification derived from the biological variation for cystatin C was 3.20%),¹³ the sigma metrics will overestimate or underestimate the analytical performance. Unfortunately, no accepted standards exist to guide laboratories when choosing quality goals. Therefore, our study selected the desirable specification derived from biological variation for the evaluation of analytical performance for cystatin C. Second, EQA samples provided by the NCCL were selected to perform bias assessments in this study. However, the target value of the EQA samples was not a ‘true’ value because it was not derived using a reference measurement procedure but instead used the average value from all of the participating laboratories. Third, uncertainty is involved in the entire measurement process.³³ However, we ignored the impact of uncertainty for cystatin C on the research results. Therefore, we need to pay more attention to the measurement uncertainty of analytes to ensure that the results are more scientific and reliable in the future.

In conclusion, a TEa of 6.50% (derived from the ‘desirable’ biological variation) and the Westgard rule 1_3s/2_2s/R_4s/4_1s/6_x (n = 6, R = 1, batch size of 45) with a high efficacy for determining the detection error is recommended for individualized IQC schemes of cystatin C.

Footnotes

Acknowledgements

We would like to thank Jiahao Zhu, application engineer of Qualab Biotech Co., who gave important technical assistance.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the study, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by Jiangsu Commission of Health (M2020079), Key Project of Scientific Research Development Fund of Kangda College of Nanjing Medical University (KD2019KYJJZD010), and the Health Scientific Research Project in Lianyungang (201817).

Ethical approval

Not applicable.

Guarantor

FMY.

Contributorship

QL and FMY conceived and designed the experiments. QL, WJZ, GRB and CXZ performed the experiments. WL, QL and WJZ analysed the data. QL, WJZ and FMY wrote the paper.

ORCID iDs

Qian Liu

Fumeng Yang

References

Deng

Wang

Hou

, et al. The influence of glycemic status on the performance of cystatin C for acute kidney injury detection in the critically ill. Ren Fail 2019; 41: 139–149.

Lopes

Costa-Rama

Beirão

, et al. Disposable electrochemical immunosensor for analysis of cystatin C, a CKD biomarker. Talanta 2019; 201: 211–216.

Zonoozi

Ramsay

Papacosta

, et al. Chronic kidney disease, cardiovascular risk markers and total mortality in older men: cystatin C versus creatinine. J Epidemiol Community Health 2019; 73: 645–651.

Delanaye

Cavalier

Cristol

, et al. Calibration and precision of serum creatinine and plasma cystatin C measurement: impact on the estimation of glomerular filtration rate. J Nephrol 2014; 27: 467–475.

Erlandsen

Randers

Reference intervals for plasma cystatin C and plasma creatinine in adults using methods traceable to international calibrators and reference methods. J Clin Lab Anal 2018; 32: e22433.

Wang

Zhao

, et al. The observation and analysis of internal quality control of cystatin C in China from 2014 to 2017. Clin Lab 2018; 64: 1709–1718.

Eckfeldt

Karger

Miller

, et al. Performance in measurement of serum cystatin C by laboratories participating in the College of American Pathologists 2014 CYS Survey. Arch Pathol Lab Med 2015; 139: 888–893.

Nevalainen

Berte

Kraft

, et al. Evaluating laboratory performance on quality indicators with the six sigma scale. Arch Pathol Lab Med 2000; 124: 516–519.

Zeng

Qin

, et al. Practical application of the sigma-metric run size nomogram for multistage bracketed statistical quality control analysis of eight enzymes. Clin Chim Acta 2019; 492: 57–61.

10.

Huang

Zhang

Zhao

, et al. Performance evaluation of HbA1c measurement systems with sigma metric for 1066 laboratories in China. Clin Chim Acta 2018; 487: 281–286.

11.

Hollestelle

Ruinemans-Koerts

Idema

, et al. Determination of sigma score based on biological variation for haemostasis assays: fit-for-purpose for daily practice? Clin Chem Lab Med 2019; 57: 1235–1241.

12.

Wang

Wen

Wang

, et al. Application of biological variation and six sigma models to evaluate analytical quality of six HbA1c analyzers and design quality control strategy. Artif Cells Nanomed Biotechnol 2019; 47: 3598–3602.

13.

EFLM. List of matrices with biological variation datasets, https://biologicalvariation.eu/search?q=Cystatin+C (2020, accessed 2 July 2020).

14.

Westgard

Bayat

Westgard

JO.

Analytical Sigma metrics: a review of Six Sigma implementation tools for medical laboratories. Biochem Med (Zagreb)) 2018; 28: 020502.

15.

Kilpatrick

ES.

Quality control failures exceeding the weekly limit (QC FEWL): a simple tool to improve assay error detection. Ann Clin Biochem 2019; 56: 668–673.

16.

Zhang

Zhao

Wang

, et al. The application of Six Sigma techniques in the evaluation of enzyme measurement procedures in China. Clin Lab 2015; 61: 461–465.

17.

El Sharkawy

Westgard

Awad

, et al. Comparison between Sigma metrics in four accredited Egyptian medical laboratories in some biochemical tests: an initiative towards sigma calculation harmonization. Biochem Med (Zagreb) 2018; 28: 020711.

18.

Varela

Pacheco

Comprehensive evaluation of the internal and external quality control to redefine analytical quality goals. Biochem Med (Zagreb) 2018; 28: 020710.

19.

Liu

Yang

, et al. Application of Six Sigma for evaluating the analytical quality of tumor marker assays. J Clin Lab Anal 2019; 33: e22682.

20.

Panteghini

Sandberg

Total error vs. measurement uncertainty: the match continues. Clin Chem Lab Med 2016; 54: 195–196.

21.

Wang

Shan

Evaluating the analytical quality control of urinary albumin measurements using sigma metrics. Clin Biochem 2019; 73: 109–111.

22.

Đido

Ćorić

Mujić

, et al. Determination of Six Sigma metric in control of enzymes determination in human serum. Clin Lab 2019; 65: 1309–1312.

23.

Guo

Zhang

Gao

, et al. Sigma metrics for assessing the analytical quality of clinical chemistry assays: a comparison of two approaches: electronic supplementary material available online for this article. Biochem Med (Zagreb) 2018; 28: 020708.

24.

Yang

Wang

Liu

, et al. The application of Six Sigma to perform quality analyses of plasma proteins. Ann Clin Biochem 2020; 57: 121–127.

25.

Carter

Ahmed

Berry

, et al. External quality assessment of 24,25-dihydroxyvitamin D(3) (24,25(OH)(2)D(3)) assays. J Steroid Biochem Mol Biol 2019; 187: 130–133.

26.

Westgard

SA.

Establishing evidence-based statistical quality control practices. Am J Clin Pathol 2019; 151: 364–370.

27.

Westgard

JO.

A total quality-control plan with right-sized statistical quality-control. Clin Lab Med 2017; 37: 137–150.

28.

Westgard

SA.

Quality control review: implementing a scientifically based quality control system. Ann Clin Biochem 2016; 53: 32–50.

29.

Westgard

SA.

Six Sigma quality management system and design of risk-based statistical quality control. Clin Lab Med 2017; 37: 85–96.

30.

Westgard

Bayat

Westgard

JO.

Mistaken assumptions drive new Six Sigma model off the road. Biochem Med (Zagreb) 2019; 29: 010903.

31.

Westgard

Bayat

Westgard

JO.

Special issue on Six Sigma metrics – experiences and recommendations. Biochem Med (Zagreb) 2018; 28: 020301.

32.

Verma

Dahiya

Ghalaut

, et al. Assessment of quality control system by sigma metrics and quality goal index ratio: a roadmap towards preparation for NABL. WJM 2018; 8: 44–50.

33.

Oosterhuis

Bayat

Armbruster

, et al. The use of error and uncertainty methods in the medical laboratory. Clin Chem Lab Med. 2018; 56: 209–219.