The argument that empirical studies should be appraised in terms of multiple context-dependent criteria is reviewed. A distinction is made between `practical impact' appraisal and `conceptual rigor' assessment. The idea of replacing a test of statistical significance with multiple criteria is assessed and found inappropriate when we are concerned with conceptual rigor. A test of statistical significance can be profitably used to ascertain whether or not a theory is tenable vis-a-vis a set of experimental data. It is an indispensable step in the multi-step theory-corroboration procedure which is hypothetico-deductive in nature. The binary nature of a significance test is compatible with the evolving nature of scientific knowledge, which should be based on internally and externally valid research results. The use of multiple context-dependent criteria of impressiveness, on the other hand, is antithetical to the requirements of internal validity and of external validity. A case is made that criticisms of the use of significance tests and of the hypothetico-deductive approach do not detract from the contribution to conceptual rigor due to the use of a test of significance and of the hypothetico-deductive nature of theory corroboration experimentation.
Get full access to this article
View all access options for this article.
References
1.
Anderson, J.W. , Story, L., Sieling, B., Chen, W.J., Petro, M.S., & Story, J. (1984). Hypocholesterolemic effects of oat-bran or bean intake for hyper-cholesterolemic men. American Journal of Clinical Nutrition, 40, 1146-1155.
2.
Boring, E.G. (1954). The nature and history of experimental control. American Journal of Psychology, 67, 573-589.
3.
Boring, E.G. (1969). Perspective: Artifact and control. In R. Rosenthal & R.L. Rosnow (Eds.), Artifact in behavioral research (pp. 1-11). New York: Academic Press.
4.
Campbell, D.T. , & Stanley, J.L. (1966). Experimental and quasi-experimental designs for research. Chicago: Rand McNally.
Chow, S.L. (1987b). Science, ecological validity and experimentation. Journalfor the Theory of Social Behavior, 17, 181-194.
7.
Chow, S.L. (1987c). Meta-analysis of pragmatic and theoretical research: A critique. Journal of Psychology, 121, 259-271.
8.
Chow, S.L. (1988). Significance test or effect size?Psychological Bulletin, 103, 105-110.
9.
Chow, S.L. (1989). Significance tests and deduction: Reply to Folger (1989). Psychological Bulletin, 106, 161-165.
10.
Cohen, M.R. , & Nagel, E. (1934). An introduction to logic and scientific method. London: Routledge & Kegan Paul.
11.
Cook, T.D. , & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally.
12.
Cooper, H.M. , & Rosenthal, R. (1980). Statistical versus traditional procedures for summarizing research findings. Psychological Bulletin, 87, 442-449.
13.
Garner, W.R. , Hake, H.W., & Eriksen, C.W. (1956). Operationalism and the concept of perception. Psychological Review, 63, 149-159.
14.
Glass, G.V. , & Kliegl, R.M. (1983). An apology for research integration in the study of psychotherapy. Journal of Consulting and Clinical Psychology, 51, 28-41.
15.
Glass, G.V. , McGaw, B., & Smith, M.L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
16.
Glymour, C. (1980). Hypothetico-deductivism is hopeless. Philosophy of Science, 47, 322-325.
17.
Green, D.M. , & Swet, J.A. (1966). Signal detection theory & psychophysics. New York: Wiley.
18.
Harris, M.J. , & Rosenthal, R. (1985). Mediation of interpersonal expectancy effects: 31 meta-analyses. Psychological Bulletin, 97, 363-386.
19.
Jackson, G.B. (1980). Methods for integrating reviews. Review of Education Research, 50, 438-460.
20.
Light, R.J. , & Smith, P.V. (1971). Accumulating evidence: Procedures for resolving contradictions among different research studies. Harvard Educational Review, 41, 429-471.
21.
McNicol, D. (1972). A primer of signal detection theory. London: George Allen.
22.
Meehl, P.E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806-834.
23.
Meehl, P.E. (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1, 108-141.
24.
Mill, J.S. (1973). A system of logic: Ratiocinative and inductive. Toronto: University of Toronto Press.
25.
Morrison, D.E. , & Henkel, R.E. (1970). The significance test controversy: A reader. Chicago: Aldine.
26.
Nunnally, J. (1960). The place of statistics in psychology. Educational and Psychological Measurement, 20, 641-650.
27.
Popper, K. (1968). Conjectures and refutations: The growth of scientific knowledge. New York: Harper & Row. (Original work published 1962.)
28.
Rosenthal, R. (1984). Meta-analytic procedures for social research. Beverly Hills, CA: Sage.
29.
Rosenthal, R. , & Rubin, D.B. (1978). Interpersonal expectancy effects: The first 345 studies. Behavioral and Brain Sciences, 3, 377-386.
30.
Rosnow, R.L. , & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284.
31.
Rozeboom, W.W. (1970). The art of metascience, or, what should a psychological theory be? In J.R. Royce (Ed.), Toward unification in psychology (pp. 54-163). Toronto: University of Toronto Press.
32.
Rozeboom, W.W. (1972). Scientific inference: The myth and the reality. In S.R. Brown & D.J. Brenner (Eds.), Science, psychology, and communication: Essays honoring William Stephenson (pp. 95-118). New York: Columbia University Press.
33.
Rozeboom, W.W. (1982). Let's dump hypothetico-deductivism for the right reasons. Philosophy of Science, 49, 637-647.
34.
Rozeboom, W.W. (1990). Hypothetico-deductivism is a fraud. American Psychologist, 45, 555-556.
35.
Schneider, W. , & Shiffrin, R.M. (1977). Controlled and automatic human information processing: I. Detection, search, and attention. Psychological Review, 84, 1-66.
36.
Swain, J.F. , Rouse, I.L., Curley, C.B., & Sacks, F.M. (1990). Comparison of the effects of oat bran and low-fiber wheat on serum lipoprotein levels and blood pressure. New England Journal of Medicine, 322, 147-152.
37.
Tukey, J.W. (1960). Conclusions vs. decisions. Technometrics, 2, 1-11.
38.
Wampold, B.E. , Davis, B., & Good, R.H. III (1990). Methodological contributions to clinical research: Hypothesis validity of clinical research. Journal of Consulting & Clinical Psychology, 58, 360-367.
