Decision-Analytic Modeling to Evaluate Benefits and Harms of Medical Tests: Uses and Limitations

Abstract

The clinical utility of medical tests is measured by whether the information they provide affects patient-relevant outcomes. To a large extent, effects of medical tests are indirect in nature. In principle, a test result affects patient outcomes mainly by influencing treatment choices. This indirectness in the link between testing and its downstream effects poses practical challenges to comparing alternate test-and-treat strategies in clinical trials. Keeping in mind the broader audience of researchers who perform comparative effectiveness reviews and technology assessments, the authors summarize the rationale for and pitfalls of decision modeling in the comparative evaluation of medical tests by virtue of specific examples. Modeling facilitates the interpretation of test performance measures by connecting the link between testing and patient outcomes, accounting for uncertainties and explicating assumptions, and allowing the systematic study of tradeoffs and uncertainty. The authors discuss challenges encountered when modeling test-and-treat strategies, including but not limited to scarcity of data on important parameters, transferring estimates of test performance across studies, choosing modeling outcomes, and obtaining summary estimates for test performance data.

Keywords

Bayesian meta-analysis comparative effectiveness multiparameter evidence synthesis systematic reviews meta-analysis evidence synthesis decision analysis.

Get full access to this article

View all access options for this article.

References

Bossuyt P. , McCaffery K. Patient outcome after testing. Unpublished manuscript. 2008 .

Bossuyt PM , Lijmer JG , Mol BW Randomised comparisons of medical tests: sometimes invalid, not always efficient . Lancet. 2000;356:1844-7.

Lord SJ , Irwig L. , Simes RJ

When is measuring sensitivity and specificity sufficient to evaluate a diagnostic test, and when do we need randomized trials?

Ann Intern Med . 2006;144:850-5.

Lijmer JG , Leeflang M. , Bossuyt PMM. Proposals for a phased evaluation of medical tests. Med Decis Making. 2009; 29:E13-E21.

Lord SJ , Irwig L. , Bossuyt PMM. Using the principles of randomized controlled trial design to guide test evaluation . Med Decis Making. 2009; 29:E1-E12.

Siebert U. When should decision analytic modeling be used in the economic evaluation of health care? Eur J Health Econ. 2003; 4:143-50.

Tatsioni A. , Zarin DA , Aronson N. , et al. Challenges in systematic reviews of diagnostic technologies . Ann Intern Med. 2005; 142:1048-55.

Hunink M. , Glasziou P. , Siegel J. , Pliskin J. , Elstein A. , Weinstein M. Decision Making in Health and Medicine: Integrating Evidence and Values . Cambridge, UK: Cambridge University Press; 2001.

Sox H. , Blatt M. , Higgins M. , Marton K. Medical Decision Making. Philadelphia, PA: American College of Physicians; 2007.

10.

Mushlin AI Challenges and opportunities in economic evaluations of diagnostic tests and procedures. Acad Radiol. 1999; 6(suppl 1):S128-31.

11.

Davies MJ , Tringham JR , Troughton J. , Khunti KK Prevention of type 2 diabetes mellitus: a review of the evidence and its application in a UK setting. Diabet Med. 2004;21:403-14.

12.

Waugh N. , Scotland G. , McNamee P. , et al. Screening for type 2 diabetes: literature review and economic modelling. Health Technol Assess. 2007;11:iii-xi, 1.

13.

Gillies CL , Lambert PC , Abrams KR , et al. Different strategies for screening and prevention of type 2 diabetes in adults: cost effectiveness analysis. BMJ. 2008;336(7654):1180-5.

14.

Pauker SG , Kassirer JP Decision analysis. N Engl J Med. 1987 ; 316:250-8.

15.

Cooper NJ , Sutton AJ , Abrams KR , Turner D. , Wailoo A. Comprehensive decision analytical modelling in economic evaluation: a Bayesian approach. Health Econ. 2004;13:203-26.

16.

Sutton AJ , Cooper NJ , Goodacre S. , Stevenson M. Integration of meta-analysis and economic decision modeling for evaluating diagnostic tests. Med Decis Making. 2008;28(5):650-7.

17.

Habbema JD , van Oortmarssen GJ , Lubbe JT , van der Maas PJ The MISCAN simulation program for the evaluation of screening for disease . Comput Methods Programs Biomed. 1985;20:79-93.

18.

Barza M. , Pauker SG The decision to biopsy, treat, or wait in suspected herpes encephalitis . Ann Intern Med. 1980;92:641-9.

19.

Braun P. The clinical management of suspected herpes virus encephalitis: a decision-analytic view. Am J Med. 1980;69:895-902.

20.

Lau J. , Ioannidis JP , Balk EM , et al. Evaluation of Technologies for Identifying Acute Cardiac Ischemia in Emergency Departments. Evidence Report/Technology Assessment No26. Rockville, MD: Agency for Healthcare Research and Quality; 2001.

21.

Loeve F. , Boer R. , van Oortmarssen GJ , van BM , Habbema JD The MISCAN-COLON simulation model for the evaluation of colorectal cancer screening. Comput Biomed Res. 1999;32:13-33.

22.

Zauber AG , Lansdorp-Vogelaar I. , Knudsen AB , Wilschut J. , van BM , Kuntz KM Evaluating test strategies for colorectal cancer screening: a decision analysis for the U.S. Preventive Services Task Force. Ann Intern Med . 2008;149:659-69.

23.

Screening for colorectal cancer: U.S. Preventive Services Task Force recommendation statement. Ann Intern Med. 2008;149: 627-37.

24.

Duke Center for Clinical Health Policy Research and Evidence Practice Center. Positron Emission Tomography, Single Photon Emission Computed Tomography, Functional Magnetic Resonance Imaging, and Magnetic Resonance Spectroscopy for the Diagnosis and Management of Alzheimers Dementia. Evidence Report/Technology Assessment No 26. Rockville, MD: Agency for Healthcare Research and Quality ; 2001.

25.

Decision analytic modelling in the economic evaluation of health technologies. A consensus statement. Consensus Conference on Guidelines on Economic Modelling in Health Technology Assessment. Pharmacoeconomics . 2000;17:443-4.

26.

Richardson WS , Detsky AS ; Evidence Based Medicine Working Group.

Users’ guides to the medical literature. VII. How to use a clinical decision analysis. B. What are the results and will they help me in caring for my patients?

JAMA. 1995;273:1610-3.

27.

Richardson WS , Detsky AS ; Evidence Based Medicine Working Group.

Users’ guides to the medical literature. VII. How to use a clinical decision analysis. A. Are the results of the study valid?

JAMA. 1995;273:1292-5.

28.

Philips Z. , Ginnelly L. , Sculpher M. , et al. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technol Assess. 2004;8:iii-xi, 1.

29.

Philips Z. , Bojke L. , Sculpher M. , Claxton K. , Golder S. Good practice guidelines for decision-analytic modelling in health technology assessment: a review and consolidation of quality assessment. Pharmacoeconomics. 2006;24:355-71.

30.

Sculpher M. , Fenwick E. , Claxton K. Assessing quality in decision analytic cost-effectiveness models: a suggested framework and example of application. Pharmacoeconomics. 2000;17:461-77.

31.

Weinstein MC , O’Brien B. , Hornberger J. , et al. Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices- Modeling Studies . Value Health . 2003;6:9-17.

32.

Meyer KB , Pauker SG Screening for HIV: can we afford the false positive rate? N Engl J Med. 1987;317:238-41.

33.

Trikalinos TA , Ip S. , Raman G. , et al. Home Diagnosis of Obstructive Sleep Apnea-Hypopnea Syndrome. Evidence Report/ Technology Assessment. Rockville, MD: Agency for Healthcare Research and Quality ; 2007.

34.

Trikalinos TA , Lau J. Obstructive Sleep Apnea-Hypopnea Syndrome: Modeling Different Diagnostic Strategies. Evidence Report/Technology Assessment . Rockville, MD: Agency for Healthcare Research and Quality; 2007.

35.

Bossuyt PM , Reitsma JB , Bruns DE , et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med. 2003;138:W1-12.

36.

Bossuyt PM , Reitsma JB , Bruns DE , et al. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD Initiative. Ann Intern Med. 2003;138:40-4.

37.

Lijmer JG , Mol BW , Heisterkamp S. , et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA. 1999; 282:1061-6.

38.

Rutjes AW , Reitsma JB , Di NM , Smidt N. , van Rijn JC , Bossuyt PM Evidence of bias and variation in diagnostic accuracy studies. CMAJ. 2006;174:469-76.

39.

Whiting P. , Rutjes AW , Reitsma JB , Glas AS , Bossuyt PM , Kleijnen J. Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med. 2004;140:189-202.

40.

Hawkins DM , Garrett JA , Stephenson B. Some issues in resolution of diagnostic tests using an imperfect gold standard . Stat Med. 2001;20:1987-2001.

41.

Joseph L. , Gyorkos TW Inferences for likelihood ratios in the absence of a ‘‘gold standard .’’ Med Decis Making. 1996;16:412-7.

42.

Walter SD , Irwig L. , Glasziou PP Meta-analysis of diagnostic tests with imperfect reference standards . J Clin Epidemiol. 1999; 52:943-51.

43.

Dendukuri N. , Joseph L. Bayesian approaches to modeling the conditional dependence between multiple diagnostic tests. Biometrics. 2001;57:158-67.

44.

Irwig L. , Bossuyt P. , Glasziou P. , Gatsonis C. , Lijmer J. Designing studies to ensure that estimates of test accuracy are transferable . BMJ. 2002;324:669-71.

45.

Loong TW Understanding sensitivity and specificity with the right side of the brain . BMJ. 2003;327:716-9.

46.

Goehring C. , Perrier A. , Morabia A. Spectrum bias: a quantitative and graphical analysis of the variability of medical diagnostic test performance. Stat Med. 2004;23:125-35.

47.

Mulherin SA , Miller WC Spectrum bias or spectrum effect? Subgroup variation in diagnostic test evaluation . Ann Intern Med . 2002;137:598-602.

48.

Leeflang MMG , Bossuyt PM , Irwig L. Diagnostic accuracy may vary with prevalence: implications for evidence-based diagnosis. J Clin Epidemiol. 2009;62(1):5-12

49.

Lejeune C. , Al ZK , Woronoff-Lemsi MC , et al. Use of a decision analysis model to assess the medicoeconomic implications of FDG PET imaging in diagnosing a solitary pulmonary nodule . Eur J Health Econ. 2005;6:203-14.

50.

Cantor T. , Yang Z. , Caraiani N. , Ilamathi E. Lack of comparability of intact parathyroid hormone measurements among commercial assays for end-stage renal disease patients: implication for treatment decisions . Clin Chem. 2006;52:1771-6.

51.

Foley RN , Li S. , Liu J. , Gilbertson DT , Chen SC , Collins AJ The fall and rise of parathyroidectomy in U.S. hemodialysis patients, 1992 to 2002. J Am Soc Nephrol. 2005;16:210-8.

52.

Dukic V. , Gatsonis C. Meta-analysis of diagnostic test accuracy assessment studies with varying number of thresholds. Biometrics. 2003;59:936-46.

53.

Littenberg B. , Moses LE Estimating diagnostic accuracy from multiple conflicting reports: a new meta-analytic method. Med Decis Making. 1993;13:313-21.

54.

Reitsma JB , Glas AS , Rutjes AW , Scholten RJ , Bossuyt PM , Zwinderman AH Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol. 2005;58:982-90.

55.

Rutter CM , Gatsonis CA Regression methods for meta-analysis of diagnostic test data. Acad Radiol. 1995;2(suppl 1): S48-S56.

56.

Harbord RM , Deeks JJ , Egger M. , Whiting P. , Sterne JA A unification of models for meta-analysis of diagnostic accuracy studies . Biostatistics. 2007;8:239-51.

57.

Karnon J. , Goyder E. , Tappenden P. , et al. A review and critique of modelling in prioritising and designing screening programmes. Health Technol Assess. 2007;11:iii-xi, 1.

58.

National Cancer Institute, Cancer Intervention and Surveillance Modeling Network. Available from: URL: http://cisnetcancergov/2008

59.

Loeve F. , Brown ML , Boer R. , van BM , van Oortmarssen GJ , Habbema JD Endoscopic colorectal cancer screening: a cost-saving analysis. J Natl Cancer Inst. 2000;92:557-63.