An investigation of new toxicity test method performance in validation studies: 3. sensitivity and specificity are not independent of prevalence or distribution of toxicity
Free accessResearch articleFirst published online June, 2002
An investigation of new toxicity test method performance in validation studies: 3. sensitivity and specificity are not independent of prevalence or distribution of toxicity
Often, the only measures of toxicity test performance provided in validation studies are the contingent probability statistics (CPS) sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). Sensitivity and specificity are generally used in preference to NPV and PPV since NPV and PPV are assumed to vary with changes in prevalence while sensitivity and specificity are assumed to be independent of changes in prevalence. The purpose of the studies reported here was to test whether or not sensitivity and specificity are actually independent of changes in prevalence. Results derived from these studies indicate that sensitivity and specificity vary significantly depending on the prevalence of toxic substances in the set of chemicals being tested. This means sensitivity and specificity should not always be considered constant indicators of toxicity test performance.
Spielmann H , et al. The International EU/COLIPA in vitro phototoxicity validation study: results of phase II (blind trial): Part 1: the 3T3 NRU phototoxicity test . Toxicol In Vitro1998; 12: 305–327 .
2.
Spielmann H , Liebsch M , Doering B , Moldenhauer F. First results of the EU/COLIPA validation trial ‘in vitro phototoxicity testing’ . In Vitro Toxicol1996; 9: 325–338 .
3.
Haseman JK , et al. Predicting rodent carcinogenicity from four in vitro genetic toxicity assays: an evaluation of 114 chemicals studied by the National Toxicology Program . J Am Stat Assoc1990; 85: 964–971 .
4.
Aardema MJ , Isfort RJ , Thompson ED , LeBoeuf RA . The low pH Syrian hamster embryo (SHE) cell transformation assay: a revitalized role in carcinogen prediction . Mutat Res1996; 356: 5–9 .
5.
Fentem JH , et al. The ECVAM International Validation Study on in vitro tests for skin corrosivity: 2. Results and evaluation by the management team . Toxicol In Vitro1998; 12: 483–524 .
6.
Olson H , et al. Concordance of the toxicity of pharmaceuticals in humans and in animals . Regul Toxicol Pharmacol2000; 32: 56–67 .
7.
Cooper JA , Saracci R , Cole P.Describing the validity of carcinogen screening tests . Br J Cancer1979; 39: 87–89 .
8.
Purchase IF.Procedures for screening chemicals for carcinogenicity . Br J Ind Med1980; 37: 1–10 .
9.
Purchase IF.Validation of tests for carcinogenicity . IARC Sci Publ1980; 27: 343–349 .
10.
Balls M , et al. Report and recommendations of the CAAT/ ERGATT Workshop on the validation of toxicity test procedures . Alternatives Lab Anim1990; 18: 313–336 .
11.
Feinstein AR.On the sensitivity, specificity, and discrimination of diagnostic tests . Clin Pharmacol Ther1975; 17: 104–116 .
12.
Brenner H , Gefeller O.Variation of sensitivity, specificity, likelihood ratios and predictive values with disease prevalence . Stat Med1997; 16: 981–991 .
13.
Anonymous . Evaluation of the validation status of toxicological methods: general guidelines for submissions to ICCVAM. Research Triangle Park, National Toxicology Prograame Inter Agency Center for the Evaluation of Alternative Toxicological Methods (NICEATM), NC : 1999: 1–13.
14.
Anonymous . Final report of the OECD workshop on harmonisation of validation and acceptance criteria for alternative toxicological test methods. Paris: Organisation for Economic Cooperation and Development (OECD) , ENV/MC/CHEM/TG (96)9, 1996: 1–51.
15.
Anonymous . Validation and regulatory acceptance of toxicological test methods. A report of the ad hoc Interagency Co-ordinating Committee on the Validation of Alternative Methods. Research Triangle Park, NC: National Institutes of Health , NIH Publication Number 97-3981, 1997: 1–105.
16.
Bruner LH , Carr GJ , Harbell JW , Curren RD . An investigation of new toxicity test method performance in validation studies: 1. toxicity test methods which have predictive capacity no greater than chance . Hum Exp Toxicol2002; 21: 327–334 .
17.
Bruner LH , Carr GJ , Harbell JW , Curren RD . An investigation of new toxicity test method performance in validation studies: 2. comparative assessment of three measures of toxicity test performance . Hum Exp Toxicol2002; 21: 305–315 .
18.
Vose D.Monte Carlo simulations: a guide to Monte Carlo simulation modelling. New York: John Wiley & Sons , 1996.
19.
Southee JA , et al. The performance of the tissue equivalent assay using the Skin2TM ZK1200 model in COLIPA International Validation Study on Alternatives to the Draize Eye Irritation Test . Toxicol In Vitro1999; 13: 355–373 .
20.
Brantom PG , et al. A summary report of the COLIPA International validation study on alternatives to the Draize rabbit eye irritation test . Toxicol In Vitro1997; 11: 141–179 .
21.
Bruner LH , Carr GJ , Chamberlain M , Curren RD . Validation of alternative methods for toxicity testing . Toxicol In Vitro1996; 10: 479–501 .
22.
Draize JH , Woodard G , Calvery JO . Methods for the study of irritation and toxicity of substances applied topically to the skin and mucous membranes . J Pharmacol Exp Pharmacol1944; 82: 377–390 .
23.
Forthofer RN, Lee J.Introduction to biostatistics. A guide to design, analysis, and discovery. San Diego: Academic Press , 1995.
24.
Guggenmoos-Holzmann I , van Houwelingen H.The (in)validity of sensitivity and specificity . Stat Med2000; 19: 1783–1792 .
25.
Choi BC.Causal modeling to estimate sensitivity and specificity of a test when prevalence changes . Epidemiology1997; 8: 80–86 .
26.
Tennant R , et al. Prediction of chemical carcinogenicity in rodents from in vitro genetic toxicity assays . Science2002; 236: 933–941 .
27.
Zeiger E.An interesting state of affairs in genetic toxicology . Environ Mol Mutagen2000; 35: 82–85 .
