Assessing agreement between two measurement systems: An alternative to the limits of agreement approach

Abstract

The comparison of two measurement systems is important in medical and other contexts. A common goal is to decide if a new measurement system agrees suitably with an existing one, and hence whether the two can be used interchangeably. Various methods for assessing interchangeability are available, the most popular being the limits of agreement approach due to Bland and Altman. In this article, we review the challenges of this technique and propose a model-based framework for comparing measurement systems that overcomes those challenges. The proposal is based on a simple metric, the probability of agreement, and a corresponding plot which can be used to summarize the agreement between two measurement systems. We also make recommendations for a study design that facilitates accurate and precise estimation of the probability of agreement.

Keywords

Bias homoscedasticity interchangeability measurement error probability of agreement repeatability

Get full access to this article

View all access options for this article.

References

Dunn

. Statistical evaluation of measurement errors: Design and analysis of reliability studies, 2nd ed. London: Arnold, 2004.

Altman

Bland

. Measurement in medicine: The analysis of method comparison studies. Statistician 1983; 32: 307–317.

Bland

Altman

. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; i: 307–310.

Carstensen

. Comparing clinical measurement methods: A practical guide, Chichester: Wiley, 2010.

U.S. Food and Drug Administration. Premarket notification. http://www.fda.gov/medicaldevices/deviceregulationandguidance/howtomarketyourdevice/premarketsubmissions/premarketnotification510k/default.htm#se (accessed 13 March 2015).

Mayo Clinic. Communique. Assay validation: What is it, why do we need it, and how do we do it? http://www.mayomedicallaboratories.com/articles/communique/2010/09.html (accessed 13 March 2015).

Clinical and Laboratory Standards Institute. Measurement procedure comparison and bias estimation using patient samples; approved guidelines. 3rd ed. Technical Report, EP09-A3, 2013.

Statistical guidelines for the annals of clinical biochemistry. http://acb.sagepub.com/site/includefiles/Statistical%20Guidelines.pdf (accessed 1 July 2015).

Clinical chemistry author instructions. http://www.clinchem.org/site/info_ar/info_authors.xhtml#standards (accessed 1 July 2015).

10.

Barnett

. Simultaneous pairwise linear structural relationships. Biometrics 1969; 25: 129–142.

11.

Ludbrook

. Confidence in Altman-Bland plots: A critical review of the method of differences. Clin Exp Pharmacol Physiol 2010; 37: 143–149.

12.

Choudhary

Nagaraja

Measuring agreement in method comparison studies - A review. In: Balakrishnan

et al. (ed). Advances in ranking and selection, multiple comparisons, and reliability: Methodology and applications, Boston: Birkhauser, 2005, pp. 215–244.

13.

Barnhart

Haber

. An overview on assessing agreement with continuous measurements. J Biopharm Stat 2007; 17: 529–569.

14.

Lin

. Overview of agreement statistics for medical devices. J Biopharm Stat 2008; 18: 126–144.

15.

Ryan

Woodall

. The most-cited statistical papers. J Appl Stat 2005; 32: 461–474.

16.

Bland

Altman

. Measuring agreement in method comparison studies. Stat Meth Med Res 1999; 8: 135–160.

17.

Bland

Altman

. Agreement between methods of measurement with multiple observations per individual. J Biopharmaceut Stat 2007; 17: 571–582.

18.

Voelkel JG and Siskowski BE. A study of the Bland-Altman plot and its associated methodology. Technical Report, Center for Quality and Applied Statistics, Rochester Institute of Technology, 2005.

19.

Bland

Altman

. Comparing methods of measurement: Why plotting difference against standard method is misleading. Lancet 1995; 346: 1085–1087.

20.

Bland

Altman

. Applying the right statistics: Analyses of measurement studies. Ultrasound Obstet Gynecol 2003; 22: 85–93.

21.

Mantha

Roizen

Fleisher

et al.

Comparing methods of clinical measurement: Reporting standards for Bland and Altman analysis. Anesth Analg 2000; 90: 593–602.

22.

Dewitte

Fierens

Stöckl

et al.

Application of the Bland-Altman plot for interpretation of method-comparison studies: A critical investigation of its practice. Clin Chem 2002; 48: 799–801.

23.

Lin

Hedayat

Sinha

et al.

Statistical methods in assessing agreement: Models, issues, and tools. J Am Stat Assoc 2002; 97: 257–270.

24.

Oldford RW. On modified QQ plots. Personal communication, 2014.

25.

O’Brien Petrie

Littler

et al.

The British Hypertension Society protocol for the evaluation of measuring devices. J Hypertension 1993; 11: S43–S62.

26.

Stevens NT. Assessment and comparison of continuous measurement systems. PhD Thesis, University of Waterloo, Canada, 2014.

27.

Lin

. Total deviation index for measuring individual agreement with applications in laboratory performance and bioequivalence. Stat Med 2000; 19: 255–270.

28.

Maple 18. Maplesoft. Waterloo, ON: Waterloo Maple Inc. www.maplesoft.com.

29.

Matlab 8.2.0. Natick, MA: The MathWorks Inc. www.mathworks.com.