Purpose: The purpose of this study was to review research published by Educational Administration Quarterly (EAQ) during the past 10 years to determine if confidence intervals and effect sizes were being reported as recommended by the American Psychological Association (APA) Publication Manual .
Research Design:The author examined 49 volumes of the journal, which included 245 articles from the most recent 10-year time frame. As the debate on statistical reform is focused on reporting confidence intervals and properly interpreting p values and null hypothesis statistical testing (NHST), the author examined all empirical studies from the 10-year time frame to determine if confidence intervals were included in the results and, if so, interpreted correctly. Because effect size measures are, alongside confidence intervals, at the heart of statistical reform goals in psychology, the author also examined effect-size practices in EAQ. Descriptive statistics (frequencies and percentages) were utilized to report the findings of the analysis.
Findings: The results of the descriptive analysis indicated that effect sizes were being reported in the majority of quantitative studies with limited interpretation. In contrast, no quantitative study examined from the 10-year time frame reported confidence intervals, despite the recommendations of the APA Task Force on Statistical Inference and guidelines outlined in the most recent edition of the APA Publication Manual. Recommendations for statistical reform in reporting quantitative results in EAQ are presented, and future direction is discussed.
American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.
2.
Anderson, D.R., Burnham, K.P., & Thompson, W.L. (2000). Null hypothesis testing: Problems, prevalence and an alternative. Journal of WildlifeManagement, 64, 912—923.
3.
Cohen, J. (1990). Things I have learned so far. American Psychologist, 45, 1304-1312.
4.
Cohen, J. (1994). The earth is round (p @#x3C; . 05). American Psychologist, 49, 997-1003.
5.
Cumming, G., & Finch, S. (2001). A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions . Educational and Psychological Measurement, 61, 530-572.
6.
Cumming, G., & Finch, S. (2005). Inference by eye: Confidence intervals and how to read pictures of data. American Psychologist, 60, 170-180.
7.
Fidler, F., Thomason, N., Cumming, G., Finch, S., & Leeman, J. (2005). Still much to learn about confidence intervals: Reply to Rouder and Morey. Psychological Science, 16, 494-495.
8.
Fidler, F., Thomason, N., Cumming, G., Finch, S., & Leeman, J. (2004). Editors can lead researchers to confidence intervals, but can't make them think: Statistical reform lessons from medicine. Psychological Science, 15, 119—126.
9.
Finch, S., Cumming, G., & Thomason, N. (2001). Reporting of statistical inference in the Journal of Applied Psychology: Little evidence of reform. Educational and Psychological Measurement, 61, 181-210.
10.
Finch, S., Thomason, N., & Cumming, G. (2002). Past and future American Psychological Association guidelines for statistical practice. Theory and Psychology , 12, 825-853.
11.
Harlow, L.L., Mulaik, S.A., & Steiger, J.H. (1997). What if there were no significance tests?Mahwah, NJ: Lawrence Erlbaum.
12.
Hunter, J.E., & Schmidt, F.L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.
13.
Kirk, R.E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.
14.
Morrison, D.E., & Henkel, R.E. (1970). Significance tests reconsidered. In D. E. Morrison & R. E. Henkel (Eds.), The significance test controversy (pp. 182—199). Chicago: Aldine.
15.
Nickerson, R. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods , 5, 241-301.
16.
Oakes, M. (1986). Statistical inference: A commentary for the social and behavioural sciences. Chichester, UK: Wiley.
17.
Schmidt, F.L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist , 47, 1173-1181.
18.
Schmidt, F.L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115-129.
19.
Schmidt, F., & Hunter, J. (1997). Eight common but false objections to the discontinuation of significance testing in analysis of research data. In L. Harlow, S. Mulaik, & J. Steiger (Eds.), What if there were no significance tests? (pp. 37-63). Mahwah, NJ: Erlbaum.
20.
Snyder, P., & Lawson, S. (1993). Evaluating results using corrected and uncorrected effect size estimates. Journal of Experimental Education, 61, 334-349.
21.
Thompson, B. (1994). Guidelines for authors. Educational and Psychological Measurement, 54, 837 -847.
22.
Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25, 26-30.
23.
Thompson, B. (1999). Why “encouraging” effect size reporting is not working: The etiology of researcher resistance to changing practices . Journal of Psychology, 133, 133-140.
24.
Thompson, B. (2002). What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31, 24-31.
25.
Wilkinson, L., Task Force on Statistical Inference, & American Psychological Association. (1999). Statistical methods in psychology journals: Guidelines and explanations . American Psychologist, 54, 594-604.
