In management research, theory testing confronts a paradox described by Meehl in which designing studies with greater methodological rigor puts theories at less risk of falsification. This paradox exists because most management theories make predictions that are merely directional, such as stating that two variables will be positively or negatively related. As methodological rigor increases, the probability that an estimated effect will differ from zero likewise increases, and the likelihood of finding support for a directional prediction boils down to a coin toss. This paradox can be resolved by developing theories with greater precision, such that their propositions predict something more meaningful than deviations from zero. This article evaluates the precision of theories in management research, offers guidelines for making theories more precise, and discusses ways to overcome barriers to the pursuit of theoretical precision.
Bakan, D. ( 1966). The test of significance in psychological research. Psychological Bulletin, 66, 423-437.
11.
Bass, B.M., Cascio, W.F., & O’Connor, E.J. (1974 ). Magnitude estimations of expressions of frequency and amount . Journal of Applied Psychology, 59, 313-320.
12.
Binder, A. ( 1963). Further considerations on testing the null hypothesis and the strategy and tactics of investigating theoretical models. Psychological Review, 70, 107-115.
13.
Blanton, H., & Jaccard, J. ( 2006). Arbitrary metrics in psychology. American Psychologist, 61, 27-41.
14.
Bollen, K.A. ( 1989). Structural equations with latent variables. New York, NY: John Wiley.
15.
Borenstein, M., Hedges, L.V., Higgins, J.P.T., & Rothstein, H.R. (2009). Introduction to meta-analysis . New York, NY: John Wiley.
16.
Carlson, K.D., & Ji, F.X. ( 2009). Citing, reporting, interpreting, and building on metaanalytic findings. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, New Orleans, LA.
17.
Carver, R.P. ( 1978). The case against statistical significance testing. Harvard Education Review, 48, 378-399.
18.
Carver, R.P. ( 1993). The case against statistical significance testing, revisited . Journal of Experimental Education, 61, 287-292.
19.
Champoux, J.E. ( 1992). A multivariate analysis of curvilinear relationships among job scope, work context satisfactions, and affective outcomes. Human Relations, 45, 87-111.
20.
Cohen, J. ( 1988). Statistical power analysis for the behavioral sciences (2nd ed.). New York, NY: Academic Press.
21.
Cohen, J. ( 1994). The earth is round (p < .05). American Psychologist, 49, 997-1003.
22.
Cortina, J.M., & Dunlap, W.P. ( 1997). On the logic and purpose of significance testing. Psychological Methods, 2, 161-172.
23.
Cudeck, R., & Browne, M.W. ( 1983). Cross-validation of covariance structures. Multivariate Behavioral Research, 18, 147-167.
24.
Dar, R. ( 1987). Another look at Meehl, Lakatos, and the scientific practices of psychologists. American Psychologist, 42, 145-151.
25.
Davis, J.P., Eisenhardt, K.M., & Bingham, C.B. ( 2007). Developing theory through simulation methods. Academy of Management Review, 32, 480-499.
26.
Diener, E., Ng, W., & Tov, W. ( 2009). Balance in life and declining marginal utility of diverse resources. Applied Research in Quality of Life, 3, 277-291.
27.
Dubin, R. ( 1978). Theory building. New York, NY: Free Press.
28.
Eden, D. ( 2004). Reflections on the AMJ Associate Editor Role. Academy of Management Journal, 47, 167-173.
29.
Edwards, J.R. ( 2008). To prosper organizational psychology should . . . overcome methodological barriers to progress. Journal of Organizational Behavior, 29, 469-491.
30.
Edwards, J.R., & Shipp, A.J. ( 2007). The relationship between person-environment fit and outcomes: An integrative theoretical framework. In C. Ostroff & T. A. Judge (Eds.), Perspectives on organizational fit (pp. 209-258). San Francisco, CA: Jossey-Bass.
31.
Eisenhardt, K.M. ( 1989). Building theories from case study research. Academy of Management Review, 14, 532-550.
32.
Falk, R., & Greenbaum, C.W. (1995). Significance tests die hard: The amazing persistence of a probabilistic misconception. Theory & Psychology, 5, 75-98.
33.
Ferris, G.R., Bowen, M.G., Treadway, D.C., Hochwarter, W.A., Hall, A.T., & & Perrewé, P.L. (2006). The assumed linearity of organizational phenomena: Implications for occupational stress and well-being . In P. L. Perrewé & D. C. Ganster (Eds.), Research in occupational stress and well-being (Vol. 5, pp. 205-232). Oxford, UK: Elsevier Science Ltd.
34.
Fowler, R.L. ( 1985). Testing for substantive significance in applied research by specifying nonzero effect null hypotheses. Journal of Applied Psychology, 70, 215-218.
35.
Freese, L. ( 1980). Formal theorizing. Annual Review of Sociology , 6, 187-212.
36.
Frick, R.W. ( 1996). The appropriate use of null hypothesis testing. Psychological Methods, 1, 379-390.
37.
Gigerenzer, G. ( 1998). Surrogates for theories. Theory and Psychology , 8, 195-204.
38.
Gigerenzer, G. ( 2004). Mindless statistics. The Journal of Socio-Economics , 33, 587-606.
39.
Gillett, R. ( 1994). Post hoc power analysis. Journal of Applied Psychology, 79, 783-785.
40.
Guion, R.M. ( 1998). Some virtues of dissatisfaction in the science and practice of personnel selection. Human Resource Management Review, 8, 351-365.
41.
Hall, J.A., & Rosenthal, R. ( 1991). Testing for moderator variables in meta-analysis: Issues and methods. Communication Monographs, 58, 437-448.
42.
Hambrick, D.C. ( 2007). The field of management‘s devotion to theory: Too much of a good thing? Academy of Management Journal, 50, 1346-1352.
43.
Harrison, J.R., Lin, Z., Carroll, G.R., & Carley, K.M. ( 2007). Simulation modeling in organizational and management research. Academy of Management Review, 32, 1229-1245.
44.
Helfat, C.E. ( 2007). Stylized facts, empirical research and theory development in management. Strategic Organization, 5, 185-192.
45.
Hodges, J., & Lehmann, E. ( 1954). Testing the approximate validity of statistical hypotheses . Journal of the Royal Statistical Society (B), 16, 261-268.
46.
Hubbard, R., & Armstrong, J.S. (1997). Publication bias against null results. Psychological Reports, 80, 337-338.
47.
Hubbard, R., Vetter, D.E., & Little, E.L. ( 1998). Replication in strategic management: Scientific testing for validity, generalizability, and usefulness. Strategic Management Journal, 19, 243-254.
48.
Hulin, C. L. & Ilgen, D. R. (Eds.). (2000). Computational modeling of behavior in organizations. Washington, DC: American Psychological Association.
Jaccard, J., Turrisi, R., & Wan, C.K. ( 1990). Interaction effects in multiple regression. Newbury Park, CA: Sage.
51.
Johns, G. ( 1991). Substantive and methodological constraints on behavior and attitudes in organizational research. Organizational Behavior and Human Decision Process, 49, 80-104.
52.
Kacmar, K.M., & Whitfield, J.M. (2000). An additional rating method for journal articles in the field of management. Organizational Research Methods, 3, 392-406.
53.
Kazdin, A.E. ( 2006). Arbitrary metrics: Implications for identifying evidence-based treatments. American Psychologist, 61, 42-49.
54.
Kingstrom, P.O., & Bass, A.R. ( 1981). A critical analysis of studies comparing behaviorally anchored rating scales (bars) and other rating formats. Personnel Psychology, 34, 263-289.
55.
Kirk, R.E. ( 1996). Practical significance: A concept whose time has come . Educational and Psychological Measurement, 56, 746-759.
56.
Krueger, J. ( 2001). Null hypothesis significance testing: On the survival of a flawed method. American Psychologist, 56, 16-26.
57.
Lakatos, I. ( 1978). Falsification and the methodology of scientific research programs. In J. Worrall & G. Currie (Eds.), The methodology of scientific research programs: Imre Lakatos philosophical papers (Vol. 1). Cambridge, England: Cambridge University Press.
MacCallum, R., Roznowski, M., Mar, C.M., & Reith, J.V. ( 1994). Alternative strategies for cross-validation of covariance structure models. Multivariate Behavioral Research, 29, 1-32.
65.
Mahoney, M.J. ( 1985). Open exchange and epistemic progress. American Psychologist, 40, 29-39.
66.
McKinley, W. ( 2010). Organizational theory development: Displacement of ends?Organization Studies, 31, 47-68.
67.
McNemar, Q. ( 1962). Psychological statistics (3rd ed.). New York, NY: John Wiley .
68.
Meehl, P.E. ( 1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103-115.
69.
Meehl, P.E. ( 1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806-834.
70.
Meehl, P.E. ( 1990a). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1, 108-141.
71.
Meehl, P.E. ( 1990b). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195-244.
72.
Meehl, P.E. ( 1997). The problem is epistemology, not statistics: Replace significance tests by confidence intervals and quantify accuracy of risky numerical predictions. In L. L. Harlow , S. A. Mulaik & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 391-423). Hillsdale, NJ: Lawrence Erlbaum.
73.
Mosier, C.I. ( 1951). Symposium: The need and means of cross-validation. Educational and Psychological Measurement, 11, 5-11.
74.
Mulaik, S.A., Raju, N.S., & Harshman, R.A. (1997). There is a time and place for significance testing. In L. L. Harlow , S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 65-116). Hillsdale, NJ: Erlbaum.
75.
Murphy, K.R. ( 1990). If the null hypothesis is impossible, why test it?American Psychologist, 45, 403-404.
76.
Murphy, K.R., Myors, B., & Wolach, A. ( 2009). Statistical power analysis: A simple and general model for traditional and modern hypothesis tests (3rd ed.). Mahwah, NJ: Lawrence Erlbaum .
77.
Muse, L.A., Harris, S.G., & Field, H.S. ( 2003). Has the inverted-U theory of stress and job performance had a fair test? Human Performance, 16, 349-364.
78.
Neuliep, J.W., & Crandall, R. ( 1991). Editorial bias against replication research. In J. W. Neuliep (Ed.), Replication research in the social sciences (pp. 85-90). Newbury Park, CA : Sage.
79.
Nickerson, R.S. ( 2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241-301.
80.
Northcraft, G.B., Preston, J.N., Neale, M.A., Kim, P.H., & Thomas-Hunt, M.C. (1998). Non-linear preference functions and negotiated outcomes. Organizational Behavior and Human Decision Processes, 73, 54-75.
81.
Pedhazur, E.J. ( 1997). Multiple regression in behavioral research (3rd ed.). New York, NY: Holt.
82.
Pentland, B.T. ( 1999). Building process theory with narrative: From description to explanation. Academy of Management Review, 24, 711-724.
83.
Pfeffer, I. ( 1982). Organizations and organization theory. Marshfield, MA: Pitman.
84.
Pfeffer, J. ( 1993). Barriers to the advance of organizational science: Paradigm development as a dependent variable. Academy of Management Review, 18, 599-620.
85.
Pfeffer, J. ( 2007). A modest proposal: How we might change the process and product of managerial research. Academy of Management Journal , 50, 1334-1345.
Poole, M.S., & Van de Ven, A.H. (1989). Using paradox to build management and organization theories. Academy of Management Review, 14, 562-578.
88.
Popper, K.R. ( 1959). The logic of scientific discovery. New York, NY: Basic Books.
89.
Rice, R.W., McFarlin, D.B., Hunt, R.G., & Near, J.P. ( 1985). Organizational work and the perceived quality of life: Toward a conceptual model. Academy of Management Review, 10, 296-310.
90.
Rodgers, J.L. ( 2010). The epistemology of mathematical and statistical modeling: A quiet methodological revolution. American Psychologist, 65, 1-12.
91.
Rosenthal, R. ( 1979). The ‘‘file drawer problem’’ and tolerance for null results. Psychological Bulletin, 86, 638-641.
92.
Rotton, J., Foos, P.W., Vanmeek, L., & Levitt, M. ( 1995). Publication practices and the file drawer problem: A survey of published authors. Journal of Social Behavior and Personality, 10, 1-13.
93.
Rozeboom, W.W. ( 1960). The fallacy of the null-hypothesis significance test . Psychological Bulletin, 57, 416-428.
94.
Runkel, P.J., & McGrath, J.E. ( 1972). Research on human behavior: A systematic guide to method . New York, NY: Holt.
95.
Schmidt, F.L. ( 1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist , 47, 1173-1181.
Schriesheim, C.A., & Castro, S.L. ( 1996). Referent effects in the magnitude estimation scaling of frequency expressions for response anchor sets: An empirical investigation . Educational and Psychological Measurement, 56, 557-569.
98.
Sechrest, L., McKnight, P., & McKnight, K. ( 1996). Calibration of measures in psychotherapy outcome studies . American Psychologist, 51, 1065-1071.
99.
Serlin, R.C. ( 1993). Confidence intervals and the scientific method: A case for Holm on the range. Journal of Experimental Education, 61, 350-360.
100.
Serlin, R.C., & Lapsley, D.K. ( 1985). Rationality in psychological research: The good-enough principle. American Psychologist, 40, 73-83.
101.
Smith, P.C., & Kendall, L. ( 1963). Retranslation of expectation: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 47, 149-155.
102.
Snee, R.D. ( 1977). Validation of regression models: Methods and examples . Technometrics, 19, 415-428.
103.
Spencer, B. ( 1995). Correlations, sample size, and practical significance: A comparison of selected psychological and medical investigations. Journal of Psychology, 129, 469-475.
104.
Sterling, T.D., Rosenbaum, W.L., & Weinkam, J.J. ( 1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 49, 108-112.
Sudman, S., Bradburn, N.M., & Schwarz, N. ( 1996). Thinking about answers: The application of cognitive processes to survey methodology. San Francisco, CA: Jossey-Bass.
107.
Sutton, R.I., & Staw, B.M. ( 1995). What theory is not. Administrative Science Quarterly, 40, 371-384.
108.
Tryon, W.W. ( 2001). Evaluating statistical difference, equivalence, and indeterminancy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests. Psychological Methods, 6, 371-386.
109.
Tukey, J.W. ( 1969). Analyzing data: Sanctification or detective work? American Psychologist, 24, 83-91.
110.
Tukey, J.W. ( 1991). The philosophy of multiple comparisons. Statistical Science, 6, 100-116.
111.
Tversky, A., & Kahneman, D. ( 1992). Advances in prospect theory: Cumulative representations of uncertainty. Journal of Risk and Uncertainty, 5, 297-323.
112.
Vandenberg, R.J., & Grelle, D.M. ( 2009). Alternative model specifications in structural equation modeling: Facts, fictions, and truth. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and methodological myths and urban legends: Received doctrine, verity, and fable in the organizational and social sciences (pp. 165-191). New York, NY: Routledge.
113.
Wainer, H. ( 1999). One cheer for null hypothesis significance testing. Psychological Methods, 4, 212-213.
114.
Weick, K.E. ( 1989). Theory construction as disciplined imagination. Academy of Management Review, 14, 516-531.
115.
Whetten, D.A. ( 1989). What constitutes a theoretical contribution? Academy of Management Review, 14, 490-495.
116.
Xie, J.L., & Johns, G. ( 1995). Job scope and stress: Can job scope be too high? Academy of Management Journal, 38, 1288-1309.
