Construct Validity Invariance and Discrepancies in Meta-Analytic Effect Sizes Based on Different Measures

Abstract

Critical to meta-analysis is the presumption that effect sizes based on different measures are directly comparable. Recent theoretical work has shown that an invariance condition—universe score, or construct, validity invariance—must hold for either observed score or reliability-corrected effect sizes based on different measures to be directly comparable. To date, however, no research has been done investigating how crucial violations of this invariance condition are for either effect size discrepancy across different measures or the outcomes of a meta-analysis. This article reports the results of a simulation study of the possible effects that violations of construct validity invariance have on (a) the differences in population construct level standardized mean difference effect sizes based on different measurement procedures; (b) the variability in population construct level correlation effect sizes based on different measurement procedures; and (c) the results of a meta-analysis. Results suggest that considerable variability in effect sizes can exist across measurement procedures that fail to meet universe score validity invariance and that this variability has the potential to negatively affect meta-analytic results.

Keywords

effect sizes meta-analysis effect size comparability effect size invariance construct validity and meta-analysis construct validity and effect sizes

Get full access to this article

View all access options for this article.

References

Achenbach, T. (1991). Integrative guide to the 1991 CBCL/4-18, YSR, and TRF profiles. Burlington: University of Vermont, Department of Psychology.

Achenbach, T. , McConaughy, S. , & Howell, C. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101, 213-232.

Brennan, R. (2001). Generalizability theory. New York: Springer-Verlag.

Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159.

Cohen, J. , Cohen, P. , West, S. , & Aiken, L. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.

Cooper, H. , & Hedges, L. (1994). Research synthesis as a scientific enterprise . In H. Cooper & L. Hedges (Eds.), Handbook of research synthesis (pp. 3-14). New York: Russell Sage.

Cronbach, L. , Gleser, G. , Nanda, H. , & Rajaratnam, N. (1972). The dependability of behavioral measurement procedures: Theory of generalizability for scores and profiles . New York: John Wiley.

Hedges, L. , & Olkin, I. (1985). Statistical methods for meta-analysis. New York: Academic Press.

Hunt, M. (1997). How science takes stock. New York: Russell Sage.

10.

Kane, M. (1982). A sampling model for validity. Applied Psychological Measurement, 6, 125-160.

11.

Kazdin, A. (1987). Children's depression scale: Validation with psychiatric inpatients. Journal of Child Psychology and Psychiatry, 28, 29-41.

12.

Lam, R.W. , & Kennedy, S.H. (2005). Using meta-analysis to evaluate evidence: Practical tips and traps. Canadian Journal of Psychiatry, 50, 167-174.

13.

Lipsey, M. , & Wilson, D. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181-1209.

14.

Lipsey, M. , & Wilson, D. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.

15.

Lord, F. , & Novick, M. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.

16.

Maier, W. , & Moller, H. (2005). Meta-analyses: Highest level of empirical evidence? European Archives of Psychiatry & Clinical Neuroscience , 255, 369-370.

17.

Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-104). New York: Macmillan .

18.

Nugent, W. (2006). The comparability of the standardized mean difference effect size across different measures of the same construct: Measurement considerations . Educational and Psychological Measurement, 66, 612-623.

19.

Nugent, W. (in press-a). Meta-analysis as a tool for research synthesis: Cause for concern. Journal of Social Service Research.

20.

Nugent, W. (in press-b). The (non)comparability of the correlation effect size across different measurement procedures: A challenge to meta-analysis as a tool for identifying ``evidence based practices.'' Journal of Evidence Based Social Work.

21.

Nunnally, J. , & Bernstein, I. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.

22.

Petitti, D. (2000). Meta-analysis, decision analysis, and cost-effectiveness analysis: Methods for quantitative synthesis in medicine (2nd ed.). New York: Oxford University Press.

23.

Reynolds, W. (1989). Reynolds Child Depression Scale. Lutz, FL: Psychological Assessment Resources.

24.

Rosenthal, H. (1994). Parametric measures of effect size. In H. Cooper & L. Hedges (Eds.), Handbook of research synthesis (pp. 231-244). New York: Russell Sage.

25.

Rotundo, N. , & Hensley, V. (1985). The children's depression scale: A study of its validity. Journal of Child Psychology and Psychiatry, 26, 917-927.

26.

Solow, D. (2002). How to read and do proofs. New York: John Wiley.

27.

Weine, A. , Phillips, J. , & Achenbach, T. (1995). Behavioral and emotional problems among Chinese and American children: Parent and teacher reports for ages 6-13. Journal of Abnormal Child Psychology, 23, 619-639.

28.

Wilson, D. , & Lipsey, M. (2001). The role of method in treatment effectiveness research: Evidence from meta-analysis. Psychological Methods , 6, 413-429.