Significance Tests are Not Enough

Abstract

Chow (1991) distinguishes between `practical impact' and `conceptual rigor' research, and he concludes that effect-size estimation is useful only in practical impact research. I argue that significance tests do not answer substantive questions about the data and are useful only as a check that the results are unlikely to have occurred by chance. Chow's decision to regard the similarity between data and prediction as being a dichotomous judgment made on the basis of significance testing is therefore unwise. I conclude that effect sizes are the single best index of the relationship between theoretical predictions and the obtained data. The role of replications and meta-analysis in advancing theory is also discussed.

Get full access to this article

View all access options for this article.

References

Bakan, D. (1966). The effect of significance in psychological research. Psychological Bulletin, 66, 423-437.

Chow, S.L. (1988). Significance test or effect size? Psychological Bulletin, 103, 105-110.

Chow, S.L. (1989). Significance tests and deduction: Reply to Folger (1989). Psychological Bulletin, 106, 161-165.

Chow. S.L. (1991). Conceptual rigor versus practical impact. Theory and Psychology, 1(3), 337-360.

Cohen, J. (1977). Statistical power analysis for the behavioral sciences (Rev. ed.). New York: Academic Press.

Cooper, H.M. , & Rosenthal, R. (1980). Statistical versus traditional procedures for summarizing research findings. Psychological Bulletin, 87, 442-449.

Grant, D.A. (1962). Testing the null hypothesis and the strategy and tactics of investigating theoretical models. Psychological Review, 69, 54-61.

Harris, M.J. (1991). Controversy and cumulation: Meta-analysis and research on interpersonal expectancy effects. Personality and Social Psychology Bulletin, 17, 316-322.

Hedges, L.V. , & Olkin, I. (1985). Statistical methods for meta-analysis. New York: Academic Press.

10.

Lykken, D. (1968). Statistical significance in psychological research. Psychological Bulletin, 70, 151-159.

11.

Meehl, P.E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806-834.

12.

Rosenthal, R. (1984). Meta-analytic procedures for social research. Beverly Hills, CA: Sage.

13.

Rosenthal, R. (1990). Replication in behavioral research. In J.W. Neuliep (Ed.), Handbook of replication research in the behavioral and social sciences, [Special Issue]. Journal of Social Behavior and Personality, 5, 1-30.

14.

Rosenthal, R. , & Rosnow, R.L. (1985). Contrast analysis: Focused comparisons in the analysis of variance. Cambridge: Cambridge University Press.