This article introduces and urges consideration of the incomplete block design for policy-capturing research. The authors compare incomplete block designs with full factorial and fractional factorial designs along several dimensions, including survey length, analysis limitations, and impact on study participants. An initial empirical investigation of an incomplete block design and a full factorial design is performed in a policy-capturing study of 183 job seekers. As predicted, the two designs generated equivalent results on the main effects and most interactions; however, the designs yielded divergent results on some interactions. Study participants who received the incomplete block design were more positive about the survey length and experienced less stress and exhaustion than those receiving the full design. The authors discuss how methodologists might examine and compare policy-capturing designs in future research.
Get full access to this article
View all access options for this article.
References
1.
American Psychological Association. (1992). Ethical principles of psychologists and code of conduct. American Psychologist, 47, 1597-1611.
2.
American Sociological Association. (1997). Code of ethics. Washington, DC: Author.
3.
Baker, D. D., Ravichandran, R., & Randall, D. M. (1989). Exploring contrasting formulations of expectancy theory. Decision Sciences, 20, 1-13.
4.
Birnbaum, M. H. (1999). How to show that 9 > 221: Collect judgments in a between-subjects design. Psychological Methods, 43, 243-249.
5.
Cable, D. M., & Judge, T. A. (1994). Pay preferences and job search decisions: A person-organization fit perspective. Personnel Psychology, 47, 317-348.
6.
Cochran, W. G., & Cox, G. M. (1957). Experimental designs (2nd ed.). New York: Wiley.
7.
Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlation analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
8.
Dey, A. (1985). Orthogonal fractional factorial designs. New York: John Wiley & Sons.
9.
Federer, W. T. (1955). Experimental design. New York: Macmillan.
10.
Fombrun, C., & Shanley, M. (1990). What’s in a name? Reputation building and corporate strategy. Academy of Management Journal, 33, 233-258.
11.
Gomez-Mejia, L. R., & Balkin, D. B. (1992). Determinants of faculty pay: An agency theory perspective. Academy of Management Journal, 35, 921-955.
12.
Graves, L. M., & Karren, R. J. (1992). Interviewer decision processes and effectiveness: An experimental policy capturing investigation. Personnel Psychology, 45, 313-340.
13.
Hardy, M. A. (1993). Regression with dummy variables. Newbury Park, CA: Sage.
14.
Hobson, C. J., & Gibson, F. W. (1983). Policy capturing as an approach to understanding and improving performance appraisal: A review of the literature. Academy of Management Review, 8, 640-649.
15.
Hoffman, P. J. (1960). The paramorphic representation of clinical judgement. Psychological Bulletin, 57, 116-131.
16.
John, P.W.M. (1984). The growth of experimental design in engineering. In H. A. David & H. T. David (Eds.), Statistics: An appraisal (pp. 129-151). Ames: Iowa State University Press.
17.
Judge, T. A., & Bretz, R. D., Jr. (1992). Effects of work values on job choice decisions. Journal of Applied Psychology, 77, 261-271.
18.
Klaas, B. S., & Dell’Omo, G. G. (1991). The determinants of disciplinary decisions: The case of employee drug use. Personnel Psychology, 44, 813-834.
19.
Klaas, B. S., & Wheeler, H. N. (1990). Managerial decision making about employee discipline: A policy capturing approach. Personnel Psychology, 43, 117-134.
20.
Martocchio, J. J., & Judge, T. A. (1995). When we don’t see eye to eye: Discrepancies between supervisors and subordinates in absence disciplinary decisions. Journal of Management, 21, 251-278.
21.
Monahan, C. J., & Muchinsky, P. M. (1985). Intrasubject predictions of vocational preference: Convergent validation via the decision theoretic paradigm. Journal of Vocational Behavior, 27, 1-18.
22.
Naylor, J. C., & Wherry, R. J. (1965). The use of simulated stimuli and the “JAN” technique to capture and cluster the policies of raters. Educational and Psychological Measurement, 25, 969-986.
23.
Neter, J., Wasserman, W., & Kutner, M. H. (1990). Applied linear statistical models (3rd ed.). Homewood, IL: Irwin.
24.
Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231-259.
25.
Pablo, A. L. (1994). Determinants of acquisition integration level: A decision-making perspective. Academy of Management Journal, 37, 803-836.
26.
Roehling, M. V. (1993). “Extracting” policy from judicial opinions: The dangers of policy capturing in a field setting. Personnel Psychology, 46, 477-502.
27.
Rynes, S., & Lawler, J. (1983). A policy-capturing investigation of expectancies in decisions to pursue job alternatives. Journal of Applied Psychology, 68, 620-631.
28.
Rynes, S. L., McNatt, B. D., & Bretz, R. D. (1999). Academic research inside organizations: Inputs, processes, and outcomes. Personnel Psychology, 52, 869-898.
29.
Rynes, S. L., & Miller, H. E. (1983). Recruiter and job influences on candidates for employment. Journal of Applied Psychology, 68, 147-154.
30.
Schwab, D. P., Rynes, S. L., & Aldag, R. (1987). Theories and research on job search and choice. In G. R. Ferris & K. M. Rowland (Eds.), Research in personnel and human resources management (Vol. 5, pp. 129-166). Greenwich, CT: JAI Press.
31.
Slovic, P., & Lichtenstein, S. (1971). Comparison of Bayesian and regression approaches to the study of information processing in judgment. Organizational Behavior and Human Performance, 6, 649-744.
32.
Stahl, M. J., & Harrell, A. M. (1983). Using decision modeling to measure second level valences in expectancy theory. Organizational Behavior and Human Performance, 32, 23-34.
33.
Stahl, M. J., & Zimmerer, T. W. (1984). Modeling strategic acquisition policies: A simulation of executives’ acquisition decisions. Academy of Management Journal, 27, 369-383.
34.
Stumpf, S. A., & London, M. (1981). Capturing rater policies in evaluating candidates for promotion. Academy of Management Journal, 24, 752-766.
35.
Wherry, R. J., Naylor, J. C., Wherry, R. J., Jr., & Fallis, R. F. (1965). Generating multiple samples of multivariate data with arbitrary population parameters. Psychometrika, 30, 303-313.
36.
Winer, B. J., Brown, D. R., & Michels, K. M. (1991). Statistical principles in experimental design (3rd ed.). New York: McGraw-Hill.
37.
Yates, F. (1936). Incomplete randomized blocks. Annals of Eugenics, 7, 121-140.
38.
Zedeck, S., & Kafry, D. (1977). Capturing rater policies for processing evaluation data. Organizational Behavior and Human Performance, 18, 269-294.
