Given some consensus that statistical significance tests are broken, misused or at least have somewhat limited utility, the focus of discussion within the field ought to move beyond additional bashing of statistical significance tests, and toward more constructive suggestions for improved practice. Five suggestions for improved practice are recommended; these involve (a) required reporting of effect sizes, (b) reporting of effect sizes in an interpretable manner, (c) explicating the values that bear upon results, (d) providing evidence of result replicability, and (e) reporting confidence intervals. Though the five recommendations can be followed even if statistical significance tests are reported, social science will proceed most rapidly when research becomes the search for replicable effects noteworthy in magnitude in the context of both the inquiry and personal or social values.
American Psychological Association . (1994). Publication manual of the American Psychological Association (4th ed.). Washington, DC: Author.
2.
Carver, R. (1978). The case against statistical significance testing. Harvard Educational Review, 48, 378–399.
3.
Carver, R. (1993). The case against statistical significance testing, revisited. Journal of Experimental Education, 61, 287–292.
4.
Chow, S.L. (1988). Significance test or effect size?Psychological Bulletin, 103, 105–110.
5.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
6.
Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304–1312.
7.
Cohen, J. (1994). The earth is round (p, .05). American Psychologist, 49, 997–1003.
8.
Cortina, J.M. , & Dunlap, W.P. (1997). Logic and purpose of significance testing. Psychological Methods, 2, 161–172.
9.
Cronbach, L.J. (1975). Beyond the two disciplines of psychology. American Psychologist, 30, 116–127.
10.
Dar, R. (1987). Another look at Meehl, Lakatos, and the scientific practices of psychologists. American Psychologist, 42, 145–151.
11.
Dar, R. , Serlin, R.C., & Omer, H. (1994). Misuse of statistical tests in three decades of psychotherapy research. Journal of Consulting and Clinical Psychology, 62, 75–82.
12.
Diaconis, P. , & Efron, B. (1983). Computer-intensive methods in statistics. Scientific American, 248(5), 116–130.
13.
Falk, R. , & Greenbaum, C.W. (1995). Significance tests die hard: The amazing persistence of a probabilistic misconception. Theory & Psychology, 5, 75–98.
14.
Frick, R.W. (1996). The appropriate use of null hypothesis testing. Psychological Methods, 1, 379–390.
15.
Friedman, H. (1968). Magnitude of experimental effect and a table for its rapid estimation. Psychological Bulletin, 70, 245–251.
16.
Gage, N.L. (1978). The scientific basis of the art of teaching. New York: Teachers College Press.
17.
Glass, G.V. (1979). Policy for the unpredictable (uncertainty research and policy). Educational Researcher, 8(9), 12–14.
18.
Greenwald, A.G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1–20.
19.
Greenwald, A.G. , Gonzalez, R., Harris, R.J., & Guthrie, D. (1996). Effect size and p-values: What should be reported and what should be replicated?Psychophysiology, 33, 175–183.
20.
Hagen, R.L. (1997). In praise of the null hypothesis statistical test. American Psychologist, 52, 15–24.
21.
Harlow, L.L. , Mulaik, S.A., & Steiger, J.H. (Eds.). (1997). What if there were no significance tests?Mahwah, NJ: Erlbaum.
22.
Harris, M.J. (1991). Significance tests are not enough: The role of effect size estimation in theory corroboration. Theory & Psychology, 1, 375–382.
Heldref Foundation . (1997). Guidelines for contributors. Journal of Experimental Education, 65, 287–288.
25.
Howson, C. , & Urbach, P. (1994). Probability, uncertainty and the practice of statistics. In G. Wright & P. Ayton (Eds.), Subjective probability (pp. 39–51). Chichester: Wiley.
26.
Huberty, C.J. , & Morris, J.D. (1988). A single contrast test procedure. Educational and Psychological Measurement, 48, 567–578.
27.
Hudson, W.D. (1969). The is/ought question. London: Macmillan.
28.
Humphreys, L.G. , & Fleishman, A. (1974). Pseudo-orthogonal and other analysis of variance designs involving individual-differences variables. Journal of Educational Psychology, 66, 464–472.
29.
Hunter, J.E. (1997). Needed: A ban on the significance test. Psychological Science, 8, 3–7.
30.
Kirk, R. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746–759.
31.
Knapp, T.R. (1978). Canonical correlation analysis: A general parametric significance testing system. Psychological Bulletin, 85, 410–416.
32.
Kraemer, H.C. (1992). Reporting the size of effects in research studies to facilitate assessment of practical or clinical significance. Psychoendocrinology, 17, 527–536.
33.
Kupfersmid, J. (1988). Improving what is published: A model in search of an editor. American Psychologist, 43, 635–642.
Lunneborg, C.E. (1987). Bootstrap applications for the behavioral sciences. Seattle: University of Washington Press.
36.
Meehl, P.E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806–834.
37.
Meehl, P.E. (1997). The problem is epistemology, not statistics: Replace significance tests by confidence intervals and quantify accuracy of risky numerical predictions. In L.L. Harlow, S.A. Mulaik, & J.H. Steiger (Eds.), What if there were no significance tests? (pp. 391–423). Mahwah, NJ: Erlbaum.
38.
Morrison, D.E. , & Henkel, R.E. (Eds.). (1970). The significance test controversy. Chicago, IL: Aldine.
39.
Murphy, K.R. (1997). Editorial. Journal of Applied Psychology, 82, 3–5.
40.
Nelson, N. , Rosenthal, R., & Rosnow, R.L. (1986). Interpretation of significance levels and effect sizes by psychological researchers. American Psychologist, 41, 1299–1301.
41.
Nunnally, J. (1960). The place of statistics in psychology. Educational and Psychological Measurement, 20, 641–650.
42.
Oakes, M. (1986). Statistical inference: A commentary for the social and behavioral sciences. New York: Wiley.
43.
Olejnik, S.F. (1984). Planning educational research: Determining the necessary sample size. Journal of Experimental Education, 53, 40–48.
44.
Ottenbacher, K. (1984). Measures of relationship strength in occupational therapy research. The Occupational Therapy Journal of Research, 4, 271–285.
45.
Parker, S. (1990). A note on the growth of the use of statistical tests inPerception & Psychophysics. Bulletin of the Psychonomic Society, 28, 565–566.
46.
Pedhazur, E.J. , & Schmelkin, L.P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, NJ: Erlbaum.
47.
Pocock, S.J. , Hughes, M.D., & Lee, R.J. (1987). Statistical problems in the reporting of clinical trials. The New England Journal of Medicine, 317, 426–432.
48.
Robinson, D. , & Levin, J. (1997). Reflections on statistical and substantive significance, with a slice of replication. Educational Researcher, 26(5), 21–26.
49.
Rosenthal, R. (1979). The ‘file drawer problem’ and tolerance for null results. Psychological Bulletin, 86, 638–641.
50.
Rosenthal, R. (1991). Effect sizes: Pearson’s correlation, its display via the BESD, and alternative indices. American Psychologist, 46, 1086–1087.
51.
Rosenthal, R. , & Gaito, J. (1963). The interpretation of levels of significance by psychological researchers. Journal of Psychology, 55, 33–38.
52.
Rosnow, R.L. , & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276–1284.
53.
Rozeboom, W.W. (1960). The fallacy of the null hypothesis significance test. Psychological Bulletin, 57, 416–428.
54.
Saunders, S.M. , Howard, K.I., & Newman, F.L. (1988). Evaluating the clinical significance of treatment effects: Norms and normality. Behavioral Assessment, 10, 207–218.
55.
Sawyer, A.G. , & Peter, J.P. (1983). The significance of statistical significance tests in marketing research. Journal of Marketing Research, 20, 122–123.
56.
Schmidt, F.L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for the training of researchers. Psychological Methods, 1, 115–129.
57.
Schmidt, F.L. , & Hunter, J.E. (1997). Eight common but false objections to the discontinuation of significance testing in the analysis of research data. In L.L. Harlow, S.A. Mulaik, & J.H. Steiger (Eds.), What if there were no significance tests? (pp. 37–64). Mahwah, NJ: Erlbaum.
58.
Serlin, R.C. (1993). Confidence intervals and the scientific method: A case for Holm on the range. Journal of Experimental Education, 61, 350–360.
59.
Shaver, J. (1985). Chance and nonsense. Phi Delta Kappan, 67, 57–60.
60.
Shea, C. (1996). Psychologists debate accuracy of ‘significance test’. Chronicle of Higher Education, 42, A12-A12, A16-A16.
61.
Shrout, P.E. (1997). Should significance tests be banned? Introduction to a special section exploring the pros and cons. Psychological Science, 8, 1–2.
62.
Snyder, P.A. , & Lawson, S. (1993). Evaluating results using corrected and uncorrected effect size estimates. Journal of Experimental Education, 61, 334–349.
63.
Snyder, P.A. , & Thompson, B. (1998). Use of tests of statistical significance and other analytic choices in a school psychology journal: Review of practices and suggested alternatives. School Psychology Quarterly, 13, 335–348.
64.
Strike, K.A. (1979). An epistemology of practical research. Educational Researcher, 8(1), 10–16.
65.
Thompson, B. (1992a). DISCSTRA: A computer program that computes bootstrap resampling estimates of descriptive discriminant analysis function and structure coefficients and group centroids. Educational and Psychological Measurement, 52, 905–911.
66.
Thompson, B. (1992b). Two and one-half decades of leadership in measurement and evaluation. Journal of Counseling and Development, 70, 434–438.
67.
Thompson, B. (1993). The use of statistical significance tests in research: Bootstrap and other alternatives. Journal of Experimental Education, 61, 361–377.
68.
Thompson, B. (1994a). The concept of statistical significance testing (An ERIC/AE Clearinghouse Digest #EDO-TM-94-1). Measurement Update, 4, 5–6. (ERIC Document Reproduction Service No. ED 366 654)
69.
Thompson, B. (1994b). Guidelines for authors. Educational and Psychological Measurement, 54, 837–847.
70.
Thompson, B. (1994c). The pivotal role of replication in psychological research: Empirically evaluating the replicability of sample results. Journal of Personality, 62, 157–176.
71.
Thompson, B. (1995). Exploring the replicability of a study’s results: Bootstrap statistics for the multivariate case. Educational and Psychological Measurement, 55, 84–94.
72.
Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25(2), 26–30.
73.
Thompson, B. (1997). Editorial policies regarding statistical significance tests: Further comments. Educational Researcher, 26(5), 29–32.
74.
Thompson, B. (1998a). Review ofWhat if there were no significance tests? Educational and Psychological Measurement, 58.
75.
Thompson, B. (1998b, January). Why ‘encouraging’ effect size reporting isn’t working: The etiology of researcher resistance to changing practices. Paper presented at the annual meeting of the Southwest Educational Research Association, Houston, TX. (ERIC Document Reproduction Service No. ED 416 214)
76.
Thompson, B. (in press). Canonical correlation analysis. In L. Grimm & P. Yarnold (Eds.), Reading and understanding multivariate statistics, Vol. 2. Washington, DC: American Psychological Association.
77.
Thompson, B. , & Snyder, P.A. (1997). Statistical significance testing practices in theJournal of Experimental Education. Journal of Experimental Education, 66, 75–83.
78.
Thompson, B. , & Snyder, P.A. (1998). Statistical significance and reliability analyses in recent JCD research articles. Journal of Counseling and Development, 76, 436–441.
79.
Vacha-Haase, T. , & Nilsson, J.E. (1998). Statistical significance reporting: Current trends and usages within MECD. Measurement and Evaluation in Counseling and Development, 31, 46–57.
80.
Young, M.A. (1993). Supplementing tests of statistical significance: Variation accounted for. Journal of Speech and Hearing Research, 36, 644–656.
81.
Zuckerman, M. , Hodgins, H.S., Zuckerman, A., & Rosenthal, R. (1993). Contemporary issues in the analysis of data: A survey of 551 psychologists. Psychological Science, 4, 49–53.
82.
Zwick, R. (1997, March). Would the abolition of significance testing lead to better science? Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
