The recent fourth edition of the American Psychological Association Publication Manual emphasized that p values are not acceptable indices of effect and `encouraged' effect-size reporting. However, empirical studies of reporting practices of diverse journals unequivocally indicate that this new encouragement has to date been ineffective. Here two additional multi-year studies of APA journals are reported. Additionally, all 50 APA editorials that have been published since 1990 were reviewed to determine how many editors with approval have articulated policies more forceful than the APA Publication Manual's vague and seemingly self-canceling encouragement. It is suggested that changes in editorial policies will be required before improved reporting will become routine.
Abelson, R.P. (1997). A retrospective on the significance test ban of 1999 (If there were no significance tests, they would be invented). In L.L. Harlow, S.A. Mulaik, & J.H. Steiger (Eds.), What if there were no significance tests? (pp. 117-141). Mahwah, NJ: Erlbaum.
2.
American Psychological Association . (1994). Publication manual of the American Psychological Association (4th ed.). Washington, DC: Author.
3.
Anderson, D.R. , Burnham, K.P., & Thompson, W.L. (1999). Null hypothesis testing in ecological studies: Problems, prevalence, and an alternative. Manuscript submitted for publication.
4.
Carver, R. (1993). The case against statistical significance testing, revisited. Journal of Experimental Education, 61, 287-292.
5.
Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003.
6.
Falk, R. (1998). Replication-a step in the right direction (Commentary on Sohn). Theory & Psychology, 8, 313-321.
7.
Falk, R. , & Greenbaum, C.W. (1995). Significance tests die hard: The amazing persistence of a probabilistic misconception. Theory & Psychology, 5, 75-98.
8.
Frick, R.W. (1996). The appropriate use of null hypothesis testing. Psychological Methods, 1, 379-390.
9.
Glantz, S.A. (1980). Biostatistics: How to detect, correct and prevent errors in the medical literature. Circulation, 61, 1-7.
10.
Heldref Foundation . (1997). Guidelines for contributors. Journal of Experimental Education, 65, 95-96.
11.
Hill, C.E. (1993). Editorial. Journal of Counseling Psychology, 40, 252-256.
12.
Hunt, E. (1990). The traditional editorial statement. Journal of Experimental Psychology: General, 119, 3-4.
13.
Kendall, P.C. (1997). Editorial. Journal of Consulting and Clinical Psychology, 15, 3-5.
14.
Keselman, H.J. , Huberty, C.J, Lix, L.M., Olejnik, S., Cribbie, R., Donahue, B., Kowalchuk, R.K., Lowman, L.L., Petoskey, M.D., Keselman, J.C., & Levin, J.R. (1998). Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA and ANCOVA analyses. Review of Educational Research, 68, 350-386.
15.
Kirk, R. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.
16.
Lance, T. , & Vacha-Haase, T. (1998, August). The Counseling Psychologist: Trends and usages of statistical significance testing. Paper presented at the annual meeting of the American Psychological Association, San Francisco.
17.
Levant, R.F. (1992). Editorial. Journal of Family Psychology, 6, 3-9.
McLean, J.E. , & Ernest, J.M. (1998). The role of statistical significance testing in educational research. Research in the Schools, 5(2), 15-22.
20.
Murphy, K.R. (1997). Editorial. Journal of Applied Psychology, 82, 3-5.
21.
Neeley, J.H. (1995). Editorial. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21, 261.
22.
Nelson, N. , Rosenthal, R., & Rosnow, R.L. (1986). Interpretation of significance levels and effect sizes by psychological researchers. American Psychologist, 41, 1299-1301.
23.
Oakes, M. (1986). Statistical inference: A commentary for the social and behavioral sciences. New York: Wiley.
24.
Rayner, K. (1990). Editorial. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16, 3-4.
25.
Robinson, D. , & Levin, J. (1997). Reflections on statistical and substantive significance, with a slice of replication. Educational Researcher, 26(5), 21-26.
26.
Rosenthal, R. , & Gaito, J. (1963). The interpretation of level of significance by psychological researchers. Journal of Psychology, 55, 33-38.
27.
Rosnow, R.L. , & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284.
28.
Rozeboom, W.W. (1997). Good science is abductive, not hypothetico-deductive. In L.L. Harlow, S.A. Mulaik, & J.H. Steiger (Eds.), What if there were no significance tests? (pp. 335-392). Mahwah, NJ: Erlbaum.
29.
Schmidt, F. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for the training of researchers. Psychological Methods, 1, 115-129.
30.
Sedlmeier, P. , & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309-316.
31.
Shaver, J. (1985). Chance and nonsense. Phi Delta Kappan, 67(1), 57-60.
32.
Snyder, P.A. , & Lawson, S. (1993). Evaluating results using corrected and un-corrected effect size estimates. Journal of Experimental Education, 61, 334-349.
33.
Snyder, P.A. , & Thompson, B. (1998). Use of tests of statistical significance and other analytic choices in a school psychology journal: Review of practices and suggested alternatives. School Psychology Quarterly, 13, 335-348.
34.
Sohn, D. (1998). Statistical significance and replicability: Why the former does not presage the latter. Theory & Psychology, 8, 291-311.
35.
Thompson, B. (1993). The use of statistical significance tests in research: Bootstrap and other alternatives. Journal of Experimental Education, 61, 361-377.
36.
Thompson, B. (1994). Guidelines for authors. Educational and Psychological Measurement, 54, 837-847.
37.
Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25(2), 26-30.
38.
Thompson, B. (1999a). If statistical significance tests are broken/misused, what practices should supplement or replace them? Theory & Psychology, 9(2), 167-183.
39.
Thompson, B. (1999b). Improving research clarity and usefulness with effect size indices as supplements to statistical significance tests. Exceptional Children, 65(3), 329-337.
40.
Thompson, B. (1999c). Journal editorial policies regarding statistical significance tests: Heat is to fire as p is to importance. Educational Psychology Review, 11, 157-169.
41.
Thompson, B. (1999d). Statistical significance tests, effect size reporting, and the vain pursuit of pseudo-objectivity. Theory & Psychology, 9(2), 191-196.
42.
Thompson, B. , & Snyder, P.A. (1997). Statistical significance testing practices in theJournal of Experimental Education. Journal of Experimental Education, 66, 75-83.
43.
Thompson, B. , & Snyder, P.A. (1998). Statistical significance and reliability analyses in recent JCD research articles. Journal of Counseling and Development, 76, 436-441.
44.
Vacha-Haase, T. , & Ness, C. (1999). Statistical significance testing as it relates to practice: Use withinProfessional Psychology. Professional Psychology: Research and Practice, 30, 104-105.
45.
Vacha-Haase, T. , & Nilsson, J.E. (1998). Statistical significance reporting: Current trends and usages withinMECD. Measurement and Evaluation in Counseling and Development, 31, 46-57.
46.
Wilkinson, L. , & The APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604 [reprint available through the APA Home Page: http://www.apa.org/journals/amp/amp548594.html].
47.
Zuckerman, M. , Hodgins, H.S., Zuckerman, A., & Rosenthal, R. (1993). Contemporary issues in the analysis of data: A survey of 551 psychologists. Psychological Science, 4, 49-53.
