A Monte-Carlo Estimation of Effect Size Distortion Due to Significance Testing

Abstract

A Monte-Carlo study was done with true effect sizes in deviation units ranging from 0 to 2 and a variety of sample sizes. The purpose was to assess the amount of bias created by considering only effect sizes that passed a statistical cut-off criterion of α = .05. The deviation values obtained at the .05 level jointly determined by the set effect sizes and sample sizes are presented. This table is useful when summarizing sets of studies to judge whether published results reflect an accurate appraisal of an underlying effect or a distorted estimate expected because significant studies are published and nonsignificant results are not. The table shows that the magnitudes of error are substantial with small sample sizes and inherently small effect sizes. Thus, reviews based on published literature could be misleading and especially so if true effect sizes were close to zero. A researcher should be particularly cautious of small sample sizes showing large effect sizes when larger samples indicate diminishing smaller effects.

Get full access to this article

View all access options for this article.

References

Begg

C. B.

(1994) Publication bias. In Cooper

Hedges

L. V.

(Eds.), The handbook of research synthesis. New York: Russell Sage Foundation.

Cohen

(1992) A power primer. Psychological Bulletin, 112, 155–159.

Greenwald

A. G.

(1975) Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1–20.

June Issue. (2002) Journal of Experimental Psychology: Human Perception and Performance, 28, 499–747.

Kalat

J. W.

(1996) Introduction to psychology. (4th ed.) New York: Brooks/Cole.

Reichardt

C. S.

Gollob

H. F.

(1997) When confidence intervals should be used instead of statistical tests, and vice versa. In Harlow

L. L.

Mulaik

S. A.

Steiger

J. H.

(Eds.), What if there were no significance tests? Mahwah, NJ: Erlbaum. Pp. 259–284.

Rosenthal

(1979) The “file-drawer problem” and tolerance for null results. Psychological Bulletin, 86, 638–641.

Sharpe

(1997) Of apples and oranges, file drawers and garbage: why validity issues in meta-analysis will not go away. Clinical Psychology Review, 17, 881–901.

Smith

M. L.

Glass

G. V.

(1977) Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, 752–760.

10.

Sterling

T. D.

(1959) Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association, 54, 30–34.