Meta-Analysis Calculations Based on Independent and Nonindependent Cases

Abstract

Meta-analysis calculations, including both validity generalization and reliability generalization, are based on the assumption that the correlations or standardized mean differences are statistically independent. This assumption is frequently violated because studies often report more than one correlation or effect size based on the same sample. The purpose of this study was to determine the effect of including nonindependent correlations in Hunter and Schmidt’s meta-analysis method on the estimated population standard deviation. First, the method for estimating the population standard deviation was examined and four new estimation methods were suggested and evaluated. The evaluation indicated that the Hunter and Schmidt method will underestimate the true population standard deviation, and the new methods were developed to correct for this. Second, a pseudo–Monte Carlo study consisting of 36 different simulated cases was conducted to illustrate the behavior of both the Hunter and Schmidt estimators and the new methods.

Get full access to this article

View all access options for this article.

References

Bangert-Drowns, R. L. (1986). Review of developments in meta-analytic method. Psychological Bulletin, 99, 388-399.

Cornwell, J. M. , & Ladd, R. T. (1993). Power and the accuracy of the Schmidt and Hunter meta-analytic procedures. Educational and Psychological Measurement, 53, 877-895.

Glass, G. V. (1976). Primary, secondary, and meta-analysis research. Educational Researcher, 5, 3-8.

Hedges, L. V. (1989). An unbiased correction for sampling error in validity generalization studies. Journal of Applied Psychology, 74, 469-477.

Hedges, L. V. , & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.

Hunter, J. E. , & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.

Hunter, J. E. , Schmidt, F. L. , & Jackson, G. B. (1982). Meta-analysis: Cumulating research findings across studies. Beverly Hills, CA: Sage.

Johnson, N. M. , & Kotz, S. (1970). Distributions in statistics: Continuous univariate distributions 2. New York: John Wiley.

Koslowsky, M. , & Sagie, A. (1994). Components of artifactual variance in meta-analytic research. Personnel Psychology, 47, 561-574.

10.

Marascuilo, L. A. , Busk, P. L. , & Serlin, R. C. (1988). Large sample multivariate procedures for comparing and combining effect sizes within a single study. Journal of Experimental Education, 57, 69-85.

11.

Martinussen, M. (1996). Psychological measures as predictors of pilot performance: A meta-analysis. International Journal of Aviation Psychology, 6,1-20.

12.

Martinussen, M., & Fjukstad, B. (1995). Metados: A computer program performing meta-analysis calculation. Unpublished manual, University of Tromsø, Norway.

13.

Rosenthal, R. (1984). Meta-analytic procedures for social research. Beverly Hills, CA: Sage.

14.

Rosenthal, R. , & Rubin, D. B. (1986). Meta-analytic procedures for combining studies with multiple effect sizes. Psychological Bulletin, 99, 400-406.

15.

Roth, L. , & Sackett, P. R. (1991). Development and Monte-Carlo evaluation of meta-analytic estimators for correlated data. Psychological Bulletin, 110, 318-327.

16.

Schmidt, F. L. , & Hunter, J. E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529-540.

17.

Tracz, S. M. , Elmore, P. B. , & Pohlmann, J. T. (1992). Correlational meta-analysis: Independent and nonindependent cases. Educational and Psychological Measurement, 52, 879-888.

18.

Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6-20.