Are the Naranjo Criteria Reliable and Valid for Determination of Adverse Drug Reactions in the Intensive Care Unit?

Abstract

Spanish

French

BACKGROUND

The Naranjo criteria are frequently used for determination of causality for suspected adverse drug reactions (ADRs); however, the psychometric properties have not been studied in the critically ill.

OBJECTIVE

To evaluate the reliability and validity of the Naranjo criteria for ADR determination in the intensive care unit (ICU).

METHODS

All patients admitted to a surgical ICU during a 3-month period were enrolled. Four raters independently reviewed 142 suspected ADRs using the Naranjo criteria (review 1). Raters evaluated the 142 suspected ADRs 3–4 weeks later, again using the Naranjo criteria (review 2). Inter-rater reliability was tested using the kappa statistic. The weighted kappa statistic was calculated between reviews 1 and 2 for the intra-rater reliability of each rater. Cronbach alpha was computed to assess the inter-item consistency correlation. The Naranjo criteria were compared with expert opinion for criterion validity for each rater and reported as a Spearman rank (r_s) coefficient.

RESULTS

The kappa statistic ranged from 0.14 to 0.33, reflecting poor inter-rater agreement. The weighted kappa within raters was 0.5402–0.9371. The Cronbach alpha ranged from 0.443 to 0.660, which is considered moderate to good. The r_s coefficient range was 0.385–0.545; all r_s coefficients were statistically significant (p < 0.05).

CONCLUSIONS

Inter-rater reliability is marginal; however, within-rater evaluation appears to be consistent. The inter-item correlation is expected to be higher since all questions pertain to ADRs. Overall, the Naranjo criteria need modification for use in the ICU to improve reliability, validity, and clinical usefulness.

Keywords

adverse drug reactions Naranjo probability scale critical care unit

Get full access to this article

View all access options for this article.

References

1. Kane-Gill

, Dasta

, Schneider

, Cook

. Monitoring abnormal laboratory values as antecedents to drug induced injury. J Trauma 2005 (in press).

2. Karch

, Smith

, Kerzner

, Mazzullo

, Weintraub

, Lasagna

. Adverse drug reactions—a matter of opinion. Clin Pharmacol Ther 1976; 19: 489–92.

3. Koch-Weser

, Sellers

, Zacest

. The ambiguity of adverse drug reactions. Eur J Clin Pharmacol 1977; 11: 75–8.

4. Karch

, Lasagna

. Toward the operational identification of adverse drug reactions. Clin Pharmacol Ther 1977; 21: 247–54.

5. Kramer

, Leventhal

, Huchinson

, Feinstein

. An algorithm for the operational assessment of adverse drug reactions. JAMA 1979; 242: 623–32.

6. Jones

. Definition of events associated with drugs: regulatory perspectives. J Rheumatol 1988; 15: 14–9.

7. Naranjo

, Busto

, Sellers

, Sandor

, Ruiz

, Roberts

, A method for estimating the probability of adverse drug reactions. Clin Pharmacol Ther 1981; 30: 239–45.

8. Kelly

. The quality of published adverse drug event reports. Ann Pharmacother 2003; 37: 1774–8. Epub Nov 5, 2003. DOI 10.1345/aph.1D202

9. Kane-Gill

, Devlin

, Bentley

, Herout

, Barletta

, Cohen

. A national survey of adverse event reporting in the intensive care unit (abstract). Crit Care Med 2004; 32: A25.

10.

10. Suggested definitions and relationships among medication misadventures, medication errors, adverse drug events, and adverse drug reactions. Am J Health Syst Pharm 1998; 55: 165–6.

11.

11. Pathak

, Meinhold

, Fisher

. Research design: sampling techniques. Am J Hosp Pharm 1980; 37: 998–1005.

12.

12. Peter

J P

. Reliability: a review of psychometric basics and recent marketing practices. J Mark Res 1979; 16: 6–17.

13.

13. Streiner

, Norman

. Reliability and validity. In: Streiner

, Norman

, eds. Health measurement scales—a practical guide to their development and use. 2nd ed. Oxford: Oxford University Press, 1998: 5–14, 62, 104–61.

14.

14. Stewart

. Psychometric considerations in functional status instruments. In: WONCA Classification Committee. Functional status measurement in primary care. New York: Springer-Verlag, 1990: 1–26.

15.

15. Stokes

, Davis

, Knoch

. Categorical data analysis using SAS system. Cary, NC: Prentice Hall, 1999: 163–200.