Interest in on-demand noncognitive assessment has flourished due to advances in computer technology and studies demonstrating noteworthy predictive validities for organizational outcomes. Computerized adaptive testing (CAT) based on the Zinnes-Griggs (ZG) ideal point item response theory (IRT) model may hold promise for organizational settings, because a large pool of items can be created from a modest number of stimuli, and the items have been shown to be resistant to some types of rater bias. However, sample sizes needed for marginal maximum likelihood (MML) estimation of statement parameters are quite large and could thus limit usefulness in practice. This article addresses that concern and its ramifications for CAT. Specifically, we conducted empirical and simulation studies to examine whether subject matter expert (SME) ratings of statement extremity (location) can be substituted for MML estimates to streamline test development and launch. Results showed that error in SME-based location estimates had little detrimental effect on score accuracy or validity, regardless of whether measures were constructed adaptively or nonadaptively. Implications for research involving small samples and CAT in field settings are discussed.
Arthur, W., Jr., Bell, S.T., Villado, A.J., & Doverspike, D. (2006). The use of person-organization fit in employment decision making: An assessment of its criterion-related validity. Journal of Applied Psychology, 91, 786-801.
2.
Birnbaum, A. ( 1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 397-472). Reading, MA : Addison-Wesley.
3.
Bock, R.D., & Aitkin, M. ( 1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443-459.
4.
Borman, W.C., Buck, D.E., Hanson, M.A., Motowidlo, S.J., Stark, S., & Drasgow, F. ( 2001). An examination of the comparative, reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales. Journal of Applied Psychology, 86, 965-973.
5.
Borman, W.C. & Motowidlo, S.J. ( 1997). Task performance and contextual performance: The meaning for personnel selection research. Human Performance, 10, 99-109.
6.
Campbell, J.P. ( 1990). Modeling the performance prediction problem in industrial and organizational psychology. In M. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 1, 2nd ed., pp. 687-731). Palo Alto, CA: Consulting Psychologists Press.
7.
Campbell, J.P., Dunnette, M.D., Arvey, R.D., & Hellervik, L.V. (1973). The development and evaluation of behaviorally based rating scales. Journal of Applied Psychology , 57, 15-22.
8.
Chernyshenko, O.S., Stark, S., Chan, K.Y., Drasgow, F., & Williams, B. ( 2001). Fitting item response theory models to personality inventories: Issues and insights. Multivariate Behavioral Research, 36, 523-562.
9.
Chernyshenko, O.S., Stark, S., Drasgow, F., & Roberts, B.W. ( 2007). Constructing personality scales under the assumptions of an ideal point response process: Toward increasing the flexibility of personality measures. Psychological Assessment, 19, 88-106.
10.
Chernyshenko, O.S., Stark, S., & Williams, A. ( 2009). Latent trait theory approach to measuring person-organization fit: Conceptual rationale and empirical evaluation. International Journal of Testing, 9, 358-380.
11.
Chuah, S.C., Drasgow, F., & Luecht, R. ( 2006). How big is big enough? Sample size requirements for CAST item parameter estimation. Applied Measurement in Education , 19, 241-255.
12.
Drasgow, F., Levine, M.V., Tsien, S., Williams, B.A., & Mead, A.D. ( 1995). Fitting polytomous item response theory models to multiple-choice tests. Applied Psychological Measurement, 19, 143-165.
13.
Drasgow, F., & Olson-Buchanan, J.B. (Eds.). (1999). Innovations in computerized assessment. Mahwah, NJ: Lawrence Erlbaum.
14.
Earley, P.C., & Ang, S. ( 2003). Cultural intelligence: Individual interactions across cultures. Stanford, CA: Stanford University Press.
15.
Griffin, M.A., Neal, A., & Neale, M. ( 2000). The contribution of task performance and contextual performance to effectiveness: Investigating the role of situational constraints. Applied Psychology: An International Review, 49, 517-533.
16.
Hambleton, R.K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston, MA: Kluwer-Nijhoff Publishing.
17.
Hanisch, K.A., & Hulin, C.L. ( 1991). General attitudes and organizational withdrawal: An evaluation of a causal model. Journal of Vocational Behavior, 39, 110-128.
18.
Harter, J.K., Schmidt, F.L., & Hayes, T.L. ( 2002). Business unit-level relationship between employee satisfaction, employee engagement, and business outcomes: A meta analysis. Journal of Applied Psychology, 87, 268-279.
19.
Hedge, J.W., Borman, W.C., Bruskiewicz, K.T., & Bourne, M.J. ( 2004). The development of an integrated performance category system for supervisory jobs in the U.S. Navy. Military Psychology , 16, 231-243.
20.
Hogan, R. ( 2005). In defense of personality measurement: New wine for old whiners. Human Performance, 18, 331-341.
21.
Houston, J.S., Borman, W.C., Farmer, W.L., & Bearden, R.M. ( 2005). Development of the enlisted computer adaptive personality scales (ENCAPS), renamed Navy Computer Adaptive Personality Scales (NCAPS) . Technical Report No. 502. Navy Personnel Research, Studies, and Technology (NPRST). Navy Personnel Command .
22.
Landy, F.J., Farr, J.L., Saal, F.E., & Freytag, W.R. ( 1976). Behaviorally anchored scales for rating the performance of police officers. Journal of Applied Psychology, 61, 750-758.
23.
Law, K.S., Wong, C.S., & Song, L.J. ( 2004). The construct and criterion validity of emotional Intelligence and its potential utility for management studies. Journal of Applied Psychology, 89, 483-496.
24.
Legree, P.J., Psotka, J., Tremble, T.R., & Bourne, D. ( 2005). Using consensus based measurement to assess emotional intelligence. In R. Schulze & R. Roberts (Eds.), International handbook of emotional intelligence (pp. 99-123). Berlin: Hogrefe & Huber.
25.
Lievens, F., & Sackett, P.R. ( 2006). Video-based versus written situational judgment tests: A comparison in terms of predictive validity. Journal of Applied Psychology, 9, 1181-1188.
26.
McDaniel, M.A., & Nguyen, N.T. ( 2001). Situational judgment tests: A review of practice and constructs assessed. International Journal of Selection and Assessment, 9, 103-113.
27.
Mills, C. N., Potenza, M. T., Fremer, J. J., & Ward, W. C. (Eds.). (2002). Computer-based testing: Building the foundation for future assessments. Mahwah, NJ: Lawrence Erlbaum.
28.
Organ, D.W. ( 1988). Organizational citizenship behavior: The good soldier syndrome. Lexington, MA: Lexington Books.
29.
Roberts, B. ( 2001). Study behavior questionnaire. Unpublished manuscript . Department of Psychology, University of Illinois, Urbana-Champaign.
30.
Roberts, B.W., Chernyshenko, O., Stark, S., & Goldberg, L. ( 2005). The structure of conscientiousness: An empirical investigation based on seven major personality questionnaires. Personnel Psychology, 58, 103-139.
31.
Sackett, P.R., Berry, C.M., Wiemann, S.A., & Laczo, R.M. ( 2006). Citizenship and counterproductive behavior: Clarifying the relationships between the two domains. Human Performance , 19, 441-464.
32.
Sackett, P.R., Schmitt, N., Ellingson, J.E., & Kabin, M.B. ( 2001). Highstakes testing in employment, credentialing, and higher education: Prospects in a post-affirmative action world. American Psychologist, 56, 302-318.
33.
Sireci, S. ( 1995). Using subject-matter experts to assess content representation: An MDS analysis. Applied Psychological Measurement, 19, 241-265.
34.
Smith, P.C., & Kendall, L.M. ( 1963). Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 47, 149-155.
35.
Stark, S. ( 2006a). MODFIT 2.0: An Excel VBA program for examining model-data fit with dichotomous and polytomous IRT models. Unpublished manuscript. University of South Florid.
36.
Stark, S. ( 2006b). ZG-EAP: A computer program for EAP scoring with the Zinnes-Griggs model. Unpublished manuscript. University of South Florida.
37.
Stark, S., Chernyshenko, O.S., & Drasgow, F. ( 2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise preference model. Applied Psychological Measurement , 29, 184-201.
38.
Stark, S., & Drasgow, F. ( 1998). Application of an IRT ideal point model to computer adaptive assessment of job performance. Paper presented at the 13th annual conference for the Society of Industrial and Organizational Psychology. Dallas, TX.
39.
Stark, S., & Drasgow, F. ( 2002). An EM approach to parameter estimation for the Zinnes and Griggs paired comparison IRT model. Applied Psychological Measurement, 26, 208-227.
40.
Thissen, D., & Wainer, H. (Eds.). (2001). Test scoring. Mahwah, NJ: Lawrence Erlbaum.
41.
Thurstone, L.L. ( 1927). Psychophysical analysis. American Journal of Psychology, 38, 368-389.
42.
Vickers, R.R., Jr., Conway, T.L., & Hervig, L.K. ( 1990). Demonstration of replicable dimensions of health behaviors . Preventive Medicine, 19, 377-401.
43.
Viswesvaran, C., Deller, J., & Ones, D.S. ( 2007). Personality measures in personnel selection: Some new contributions. International Journal of Selection and Assessment , 15, 354-358.
44.
Wainer, H. (Ed.). (2000). Computerized-adaptive testing: A primer (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.
45.
Wanous, J.P., Reichers, A.E., & Hudy, M.J. ( 1997). Overall job satisfaction: How good are single-item measures ? Journal of Applied Psychology, 82, 247-252.
