The purpose of this study is to determine how well contemporary management re-search fares on the issue of statistical power with regard to studies specifically predicting null relationships between phenomena of interest. This power assessment differs from traditional power studies because it focuses solely on studies that offered and tested null hypotheses. A sample of studies containing hypothesized null relationships was taken from five mainstream management journals over the 1990 to 1999 time period. Results of the power assessment suggest that management researchers’ abilities to affirm null hypotheses are low. On average, the power assessment revealed that for those studies that found nonsignificance of results and consequently affirmed their null hypotheses, the actual Type II error rate was nearly 15 times greater than what is advocated in the literature when failing to reject a false null hypothesis. Recommendations for researchers proposing and testing formal null hypotheses are also discussed.
Get full access to this article
View all access options for this article.
References
1.
Atkinson, D. R., Furlong, M. J., & Wampold, B. E. (1982). Statistical significance, reviewer evaluations, and scientific process: Is there a (statistically) significant relationship? Journal of Counseling Psychology, 29, 189-194 .
2.
Baroudi, J. J., & Orlikowski, W. J. (1989). The problem of statistical power in MIS research . MIS Quarterly, 13, 87-106 .
3.
Brewer, J. K. (1972). On the power of statistical tests in the American Educational Research Journal . American Educational Research Journal, 9, 391-401 .
4.
Camilleri, S. F. (1962). Theory, probability, and induction in social research . American Sociological Review, 27, 170-178 .
5.
Chase, L. J., & Chase, R. B. (1976). A statistical power analysis of applied psychological re-search . Journal of Applied Psychology, 61, 234-237 .
6.
Chase, L. J., & Tucker, R. K. (1975). A power-analytic examination of contemporary communication research . Speech Monographs, 42, 29-41 .
7.
Chow, S. L. (1996). Statistical significance: Rationale, validity, and utility. Thousand Oaks, CA: Sage .
8.
Cohen, J. (1962). The statistical power of abnormal-social psychological research . Journal of Abnormal and Social Psychology, 65, 145-153 .
9.
Cohen, J. (1977). Statistical power analysis for the behavioral sciences (Rev. ed.). New York: Academic Press .
10.
Cohen, J. (1990). Things I have learned (so far) . American Psychologist, 45, 1304-1312 .
11.
Cohen, J. (1992). A power primer . Psychological Bulletin, 112, 155-159 .
12.
Cortina, J. M., & Dunlap, W. P. (1997). On the logic and purpose of significance testing . Psychological Methods, 2, 161-172 .
13.
Cortina, J. M., & Folger, R. G. (1998). When is it acceptable to accept a null hypothesis: No way, Jose? Organizational Research Methods, 1, 334-350 .
14.
Cowles, M. P., & Davis, C. (1982). On the origins of the .05 level of statistical significance . American Psychologist, 37, 553-558 .
15.
Ferguson, T. D., & Ketchen, D. J., Jr. (1999). Organizational configurations and performance: The role of statistical power in extant research . Strategic Management Journal, 20, 385-395 .
16.
Frick, R. W. (1995). Accepting the null hypothesis . Memory & Cognition, 23, 132-138 .
17.
Gomez-Mejia, L. R., & Balkin, D. B. (1992). Determinants of faculty pay: An agency theory perspective . Academy of Management Journal, 35, 921-955 .
18.
Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis . Psychological Bulletin, 82, 1-20 .
19.
Greenwald, A. G. (1993). Consequences of prejudice against the null hypothesis. In G. Keren & C. Lewis (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 419-448). Hillsdale, NJ: Lawrence Erlbaum .
20.
Hambrick, D. C., Cho, T. S., & Chen, M. J. (1996). The influence of top management team heterogeneity on firms’ competitive moves . Administrative Science Quarterly, 41, 659-684 .
21.
Howard, G. S., Maxwell, S. E., & Fleming, K. J. (2000). The proof of the pudding: An illustration of the relative strengths of null hypothesis, meta-analysis, and Bayesian analysis . Psychological Methods, 5, 315-332 .
22.
Hubbard, R., Vetter, D. E., & Little, E. L. (1998). Replication in strategic management: Scientific testing for validity, generalizability, and usefulness . Strategic Management Journal, 19, 243-254 .
23.
Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage .
24.
Katzer, J., & Sodt, J. (1973). An analysis of the use of statistical testing in communication re-search . Journal of Communication, 23, 251-265 .
25.
Lane, P. J., Cannella, A. A., & Lubatkin, M. H. (1998). Agency problems as antecedents to unrelated mergers and diversification: Amihud and Lev reconsidered . Strategic Management Journal, 19, 555-578 .
26.
Lawless, M. W., & Anderson, P. C. (1996). Generational technological change: Effects of innovation and local rivalry on performance . Academy of Management Journal, 39, 1185-1217 .
27.
Mazen, A. M., Graf, L. A., Kellogg, C. E., & Hemmasi, M. (1987). Statistical power in contemporary management research . Academy of Management Journal, 30, 369-380 .
28.
Mazen, A. M., Hemmasi, M., & Lewis, M. F. (1987). Assessment of statistical power in contemporary strategy research . Strategic Management Journal, 8, 403-410 .
29.
Meehl, P. E. (1991). Why summaries of research on psychological theories are often uninterpretable. In R. E. Snow & D. E. Wilet (Eds.), Improving inquiry in social science (pp. 13-59). Hillsdale, NJ: Lawrence Erlbaum .
30.
Mone, M. A., Mueller, G. C., & Mauland, W. (1996). The perceptions and usage of statistical power in applied psychology and management research . Personnel Psychology, 49, 103-120 .
31.
Myers, B. L., & Melcher, A. J. (1969). On the choice of risk levels in managerial decisionmaking . Management Science, 16, B31-B39 .
32.
Nickerson, R. S. (2000). Null hypothesis significance testing: Areview of an old and continuing controversy . Psychological Methods, 5, 241-301 .
33.
Popper, K. (1959). The logic of scientific discovery. New York: Basic Books .
34.
Rossi, J. S. (1990). Statistical power of psychological research: What have we gained in 20 years? Journal of Consulting and Clinical Psychology, 58, 646-656 .
35.
Rudner, R. (1953). The scientist qua scientist makes value judgments . Philosophy of Science, 20, 1-6 .
36.
Sauley, K. S., & Bedeian, A. G. (1989). .05: A case of the tail wagging the distribution . Journal of Management, 15, 335-344 .
37.
Sawyer, A. G., & Ball, D. (1981). Statistical power and effect size in marketing research . Journal of Marketing Research, 18, 275-290 .
38.
Schmidt, F. L., & Hunter, J. E. (1997). Eight common but false objections to the discontinuation of significance testing in the analysis of research data. In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 37-64). Hillsdale, NJ: Lawrence Erlbaum .
39.
Schwenk, C. R., & Dalton, D. R. (1991). The changing shape of strategic management research. In P. Shrivastava, A. Huff, & J. Dutton (Eds.), Advances in strategic management (Vol. 7, pp. 277-300). Greenwich, CT: JAI .
40.
Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309-316 .
41.
Sharplin, A. D., & Mabry, R. H. (1985). The relative importance of journals used in management research: An alternative ranking . Human Relations, 38, 139-149 .
42.
Wiersema, M. F. (1992). Strategic consequences of executive succession within diversified firms . Journal of Management Studies, 29, 73-94 .
