Caution regarding the choice of standard deviations to guide sample size calculations in clinical trials

Abstract

Background

The method used to determine choice of standard deviation (SD) is inadequately reported in clinical trials. Underestimations of the population SD may result in underpowered clinical trials.

Purpose

This study demonstrates how using the wrong method to determine population SD can lead to inaccurate sample sizes and underpowered studies, and offers recommendations to maximize the likelihood of achieving adequate statistical power.

Methods

We review the practice of reporting sample size and its effect on the power of trials published in major journals. Simulated clinical trials were used to compare the effects of different methods of determining SD on power and sample size calculations.

Results

Prior to 1996, sample size calculations were reported in just 1%–42% of clinical trials. This proportion increased from 38% to 54% after the initial Consolidated Standards of Reporting Trials (CONSORT) was published in 1996, and from 64% to 95% after the revised CONSORT was published in 2001. Nevertheless, underpowered clinical trials are still common. Our simulated data showed that all minimal and 25th-percentile SDs fell below 44 (the population SD), regardless of sample size (from 5 to 50). For sample sizes 5 and 50, the minimum sample SDs underestimated the population SD by 90.7% and 29.3%, respectively. If only one sample was available, there was less than 50% chance that the actual power equaled or exceeded the planned power of 80% for detecting a median effect size (Cohen’s d = 0.5) when using the sample SD to calculate the sample size. The proportions of studies with actual power of at least 80% were about 95%, 90%, 85%, and 80% when we used the larger SD, 80% upper confidence limit (UCL) of SD, 70% UCL of SD, and 60% UCL of SD to calculate the sample size, respectively. When more than one sample was available, the weighted average SD resulted in about 50% of trials being underpowered; the proportion of trials with power of 80% increased from 90% to 100% when the 75th percentile and the maximum SD from 10 samples were used. Greater sample size is needed to achieve a higher proportion of studies having actual power of 80%.

Limitations

This study only addressed sample size calculation for continuous outcome variables.

Conclusions

We recommend using the 60% UCL of SD, maximum SD, 80th-percentile SD, and 75th-percentile SD to calculate sample size when 1 or 2 samples, 3 samples, 4–5 samples, and more than 5 samples of data are available, respectively. Using the sample SD or average SD to calculate sample size should be avoided.

Get full access to this article

View all access options for this article.

References

Cohen

. Statistical Power Analysis for the Behavioral Sciences (2nd edn). Academic Press, New York, 1988.

Cohen

. To be or not to be: Control and balancing of Type I and Type II errors. Eval Program Plann 1982; 5: 247–53.

Maggard

O’Connell

Liu

Etzioni

. Sample size calculations in surgery: Are they done correctly? Surgery 2003; 134: 275–79.

Vickers

. Underpowering in randomized trials reporting a sample size calculation. J Clin Epidemiol 2003; 56: 717–20.

Weaver

Leonardi-Bee

Bath-Hextall

Bath

. Sample size calculations in acute stroke trials: A systematic review of their reporting, characteristics, and relationship with outcome. Stroke 2004; 35: 1216–24.

Gotzsche

. Methodology and overt and hidden bias in reports of 196 double-blind trials of nonsteroidal anti-inflammatory drugs in rheumatoid arthritis. Control Clin Trials 1989; 10: 31–56.

Moher

Dulberg

Wells

. Statistical power, sample size, and their reporting in randomized controlled trials. JAMA 1994; 272: 122–24.

DerSimonian

Charette

Mcsteller

. Reporting on methods in clinical trials. N Engl J Med 1982; 306: 1332–37.

Pocock

Hughes

Lee

. Statistical problems in the reporting of clinical trials. A survey of three medical journals. N Engl J Med 1987; 317: 426–32.

10.

Altman

Dore

. Randomisation and baseline comparisons in clinical trials. Lancet 1990; 335: 149–53.

11.

Thornley

Adams

. Content and quality of 2000 controlled trials in schizophrenia over 50 years. BMJ 1998; 317: 1181–84.

12.

Yuen

Pope

. Learning from past mistakes: Assessing trial quality, power and eligibility in non-renal systemic lupus erythematosus randomized controlled trials. Rheumatology 2008; 47: 1367–72.

13.

Adetugbo

Williams

. How well are randomized controlled trials reported in the dermatology literature? Arch Dermatol 2000; 136: 381–85.

14.

Dimick

Diener-West

Lipsett

. Negative results of randomized clinical trials published in the surgical literature. Arch Surg 2001; 136: 796–800.

15.

Ruiz-Canela

De Irala-Estevez

Martínez-González

Gómez-Gracia

Fernández-Crehuet

. Methodological quality and reporting of ethical requirements in clinical trials. J Med Ethics 2001; 27: 172–76.

16.

Greenfield

Rosenberg

O’Reilly

. The quality of randomized controlled trials in major anesthesiology journals. Anesth Analg 2005; 100: 1759–64.

17.

Keen

Pile

Hill

. The prevalence of underpowered randomized clinical trials in rheumatology. J Rheumatol 2005; 32: 2083–88.

18.

Charles

Giraudeau

Dechartres

Baron

Ravaud

. Reporting of sample size calculation in randomised controlled trials: Review. BMJ 2009; 338: 1–6.

19.

Begg

Cho

Eastwood

. Improving the quality of reporting of randomized controlled trials: The CONSORT statement. JAMA 1996; 274: 637–39.

20.

Altman

Schulz

Moher

. The revised CONSORT statement for reporting randomized trials: Explanation and elaboration. Ann Intern Med 2001; 134: 663–94.

21.

Moher

Schulz

Altman

. The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomized trials. Lancet 2001; 357: 1191–4.

22.

Bezeau

Graves

. Statistical power and effect sizes of clinical neuropsychology research. J Clin Exp Neuropsychol 2001; 23: 399–406.

23.

Cashen

Geiger

. Statistical power and the testing of null hypotheses: A review of contemporary management research and recommendations for future studies. Organ Res Methods 2004; 7: 151–67.

24.

Maxwell

. The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychol Methods 2004; 9: 147–63.

25.

Kieser

Wassmer

. On the use of the upper confidence limit for the variance from a pilot sample for sample size determination. Biometrical J 1996; 8: 941–49.

26.

Browne

. Using the sample range as a basis for calculating sample size in power calculations. Am Stat 2001; 55: 293–98.

27.

Hyndman

Fan

. Sample quantiles in statistical packages. Am Stat 1996; 40: 361–65.

28.

Goodman

Berlin

. The use of predicted confidence intervals when planning experiments and the misuse of power when interpreting results. Ann Intern Med 1994; 121: 200–06.

29.

Kelley

Rausch

. Sample size planning for the standardized mean difference: Accuracy in parameter estimation via narrow confidence interval. Psychol Methods 2006; 11: 363–85.