The cross-mode equivalence of paper-and-pencil (P&P) and computer-based clerical tests was examined. 411 undergraduate students at two large Midwestern universities were administered 10 timed clerical tests. Although Mead & Drasgow's (1993) meta-analysis found that speededness moderated cross-mode equivalence, the current study found no differences across modes between P&P tests and computer-based tests (CBTs). When speeded CBTs follow the same administration and response procedures as the P&P format, differences across modes can be minimized. Equivalence is discussed at three levels: parallel, r-equivalent, and congeneric. The statistical and practical implications of test equivalence are presented. Although structural equation modeling determined the tests to be congeneric, results showed only small distributional differences cross-modally. No differential validity across formats was found.
Get full access to this article
View all access options for this article.
References
1.
Allen, M. J. , & Yen, W. M. (1979). Introduction to measurement theory. Monterey CA: Brooks/Cole.
2.
Anderson, J. C. , & Gerbing, D. W. (1984). The effect of sampling error on convergence, improper solutions, and goodness-of-fit indices for maximum likelihood confirmatory factor analysis. Psychometrika, 49, 155-173.
3.
Ben-Porath, Y. S. , & Butcher, J. N. (1986). Computers in personality assessment: A brief past, an ebullient present, and an expanding future. Computers in Human Behavior, 2, 167-182.
4.
Bentler, P. M. , & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588-606.
5.
Burke, M. J. (1992). Computerized psychological testing. In N. Schmitt & W. C. Borman (Eds.), Personnel selection (pp. 203-239). San Francisco: Jossey-Bass.
Divgi, D. R. (1989). Effect of the medium of administration on ASVAB item response curves (CRM Research Memorandum 86-24). Arlington VA: Center for Naval Analyses.
8.
Eaves, R. C. , & Smith, E. (1986). The effect of the media and amount of microcomputer experience on examination scores. Journal of Experimental Education, 55, 23-26.
9.
Greaud, V. A. , & Green, B. F. (1986). Equivalence of conventional and computer presentation of speed tests. Applied Psychological Measurement, 10, 23-24.
10.
Hattrup, K. , Schmitt, N., & Landis, R. S. (1992). Equivalence of constructs measured by jobspecific and commercially available aptitude tests. Journal of Applied Psychology, 77, 298-308.
11.
Howell, D. C. (1992). Statistical methods for psychology. Belmont CA: Duxbury Press.
12.
James, L. R. , Demaree, R. G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal ofApplied Psychology, 69, 85-98.
13.
Joreskog, K. G. , & Sorbom, D. (1994). LISREL VIII: A guide to the program and applications (3rd ed.). Chicago: SPSS, Inc.
14.
Kenny, D. A. (1979). Correlation and causality. New York: Wiley.
15.
Lee, J. A. , Moreno, K. E., & Sympson, J. B. (1986). The effects of mode of test administration on test performance. Educational and Psychological Measurement, 46,467-474.
16.
Linn, P. C. , & Werts, C. E. (1979). Covariance structures and their analysis. In R. Traub (Ed.), New directions for testing and measurement: Methodological developments (pp. 53-73). San Francisco: Jossey-Bass
17.
Lord, F. M. , & Novick, M. R. (1968). Statistical theories of mental test scores. Reading MA: Addison-Wesley.
18.
Marsh, H. W. , Balla, J. R., & McDonald, R. P. (1988). Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103, 391-410.
19.
Mazzeo, J., & Harvey, A. L. (1988). The equivalence of scores from automated and conventional educational and psychological tests: A review of the literature (College Board Report No 91-5). Princeton NJ: Educational Testing Service.
20.
McHenry, J. J. , Hough, L. M., Toquam, J. L., Hanson, M., & Ashworth, S. (1990). Project A validity results: The relationship between predictor and criterion domains. Personnel Psychology, 43, 335-354.
21.
Mead, A. D. , & Drasgow, F. (1993). Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. Psychological Bulletin, 114, 449-458.
22.
Muthen,O. (1989). Factor structure in groups selected on observed scores. British Journal of Mathematical and Statistical Psychology, 42, 81-90.
23.
Schmit, M. J. , & Ryan, A. M. (1993). Test-taking disposition: A missing link?Journal of Applied Psychology, 77, 629-637.
24.
Schmitt, N. , Gilliland, S. W., Landis, R. S., & Devine, D. (1993). Computer-based testing applied to selection of secretarial applicants. Personnel Psychology, 46, 149-165.
25.
Schmitt, N. , & Stults, D. M. (1986). Methodology review: Analysis of multitrait-multimethod matrices. Applied Psychological Measurement, 10, 1-22.
Silver, E. M. , & Bennett, C. (1987). Modification of the Minnesota Clerical Test to predict performance on video display terminals. Journal of Applied Psychology, 72, 153-155.
29.
Skinner, H.A. , & Pakula, A. (1986). Challenge of computers in psychological assessment. Professional Psychology, 17, 44-50.
30.
Spray, J. A. , Ackerman, T. A., Reckase, M. D., & Carlson, J. E. (1989). Effect of medium of item presentation on examinee performance and item characteristics. Journal of Educational Measurement, 26, 261-271.
31.
Tabachnik, B. G. , & Fidell, L. S. (1989). Using multivariate statistics, 2nd Ed.New York: Harper Collins, Inc.
32.
Tucker, L. R. , & Lewis, C. (1973). The reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1-10.
33.
T'urban, D. B. , Sanders, P. A., Francis, D. J., & Osburn, H. G. (1989). Construct equivalence as an approach to replacing validated cognitive ability selection tests. Journal of Applied Psychology, 74(1), 62-71.
34.
United States Department of Labor . (1970). General Aptitude Test Battery: Section III. Development. Washington DC: U.S. Government Printing Office.
35.
van de Vijver, F. J. R. , & Harsveld, M. (1994). The incomplete equivalence of the paper-and-pencil and computer versions of the General Aptitude Test Battery. Journal of Applied Psychology, 79, 852-859.
