Statistical Significance Tests,Effect Size Reporting and the Vain Pursuit of Pseudo-Objectivity

Abstract

Two themes are argued in this comment on the use of statistical significance tests. First, effect sizes are an important aspect of results that should be reported. However, 10 empirical studies (some of several different journals) of articles in various disciplines demonstrate that effect sizes are still not usually being reported, notwithstanding the admonitions of the 1994 American Psychological Association (APA) Publication Manual. Second, using statistical significance tests does not (and cannot) make scientists (or their science) objective.

Keywords

effect size replication statistical significance

Get full access to this article

View all access options for this article.

References

American Psychological Association . (1994) Publication manual of the American Psychological Association (4th ed.). Washington, DC: Author.

Biskin, B.H. (1998). Comment on significance testing. Measurement and Evaluation in Counseling and Development, 31, 58–62.

Cohen, J. (1994). The earth is round (p, .05). American Psychologist, 49, 997–1003.

Cortina, J.M. , & Dunlap, W.P. (1997). Logic and purpose of significance testing. Psychological Methods, 2, 161–172.

Frick, R.W. (1996). The appropriate use of null hypothesis testing. Psychological Methods, 1, 379–390.

Frick, R.W. (1999). Defending the statistical status quo. Theory & Psychology, 9, 183–189.

Hays, W.L. (1981). Statistics (3rd ed.). New York: Holt, Rinehart & Winston.

Heldref Foundation . (1997). Guidelines for contributors. Journal of Experimental Education, 65, 287–288.

Huberty, C.J. , & Morris, J.D. (1988). A single contrast test procedure. Educational and Psychological Measurement, 48, 567–578.

10.

Keselman, H.J. , Huberty, C.J. , Lix, L.M. , Olejnik, S. , Cribbie, R. , Donahue, B. , Kowalchuk, R.K. , Lowman, L.L. , Petoskey, M.D. , Keselman, J.C. , & Levin, J.R. (1998). Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA and ANCOVA analyses. Review of Educational Research, 68, 350–386.

11.

Kirk, R. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746–759.

12.

Murphy, K.R. (1997). Editorial. Journal of Applied Psychology, 82, 3–5.

13.

Nilsson, J. , & Vacha-Haase, T. (1998, August). A review of statistical significance reporting in the Journal of Counseling Psychology. Paper presented at the annual meeting of the American Psychological Association, San Francisco, CA.

14.

Piel, G. (1978). Research for action. Educational Researcher, 7(2), 8–12.

15.

Robinson, D. , & Levin, J. (1997). Reflections on statistical and substantive significance, with a slice of replication. Educational Researcher, 26(5), 21–26.

16.

Rosnow, R.L. , & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276–1284.

17.

Thompson, B. (1993). The use of statistical significance tests in research: Bootstrap and other alternatives. Journal of Experimental Education, 61, 361–377.

18.

Thompson, B. (1994). Guidelines for authors. Educational and Psychological Measurement, 54, 837–847.

19.

Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25(2), 26–30.

20.

Thompson, B. (1997). Editorial policies regarding statistical significance tests: Further comments. Educational Researcher, 26(5), 29–32.

21.

Thompson, B. (1998). In praise of brilliance, where that praise really belongs. American Psychologist, 53, 799–800.

22.

Thompson, B. (1999). If statistical significance tests are broken/misused, what practices should supplement or replace them? Theory & Psychology, 9, 165–181.

23.

Thompson, B. (in press). Journal editorial policies regarding statistical significance tests: Heat is to fire as p is to importance. Educational Psychology Review.

24.

Vacha-Haase, T. , & Nilsson, J.E. (1998). Statistical significance reporting: Current trends and usages within MECD. Measurement and Evaluation in Counseling and Development, 31, 46–57.

25.

Vacha-Haase, T. , & Thompson, B. (1998). Further comments on statistical significance tests. Measurement and Evaluation in Counseling and Development, 31, 63–67.