Multidimensional item response theory (MIRT) com puterized adaptive testing, building on recent work by Segall (1996), is applied in a licensing/certification context. An example of a medical licensure test is used to demonstrate situations in which complex, integrated content must be balanced at the total test level for va lidity reasons, but items assigned to reportable subscore categories may be used under a MIRT adaptive paradigm to improve the reliability of the subscores. A heuristic optimization framework is outlined that gen eralizes to both univariate and multivariate statistical objective functions, with additional systems of con straints included to manage the content balancing or other test specifications on adaptively constructed test forms. Simulation results suggested that a multivariate treatment of the problem, although complicating some what the objective function used and the estimation of traits, nonetheless produces advantages from a psycho metric perspective.
Aquinis, H. (1994). A QuickBasic program for generating correlated multivariate random normal scores. Educational and Psychological Measurement, 54, 687-690.
Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F. Lord & M. Novick, Statistical theories of mental test scores (pp. 397-479). Reading MA: Addison-Wesley.
4.
Bloxom, B., & Vale, C.D. (1987, June). Multidimensional adaptive testing: A procedure for sequential estimation of the posterior centroid and dispersion of theta . Paper presented at the annual meeting of the Psychometric Society, Montreal.
5.
Bock, R.D. (1985). Contributions of empirical Bayes and marginal maximum likelihood methods to the measurement of individual differences. In E. E. Roskam (Ed.), Measurement and personality assessment (pp. 75-99). North Holland, The Netherlands: Elsevier.
6.
Bock, R.D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika , 46, 29-51.
7.
Bock, R.D., Gibbons, R., & Muraki, E. (1988). Full-information factor analysis. Applied Psychological Measurement, 12, 261-280.
8.
Bock, R.D., & Mislevy, R.J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431-444.
9.
Doody-Bogan, E., & Yen, W.M. (1983, April). Detecting multidimensionality and examining its effects on vertical equating with the three-parameter logistic model. Paper presented at the annual meeting of the American Educational Research Association, Montreal.
10.
Federation of State Medical Boards & National Board of Medical Examiners. (1996a). Step 1: General instructions, content description, and sample items. Philadelphia PA : National Board of Medical Examiners.
11.
Federation of State Medical Boards & National Board of Medical Examiners. (1996b). Step 2: General instructions, content description, and sample items. Philadelphia PA : National Board of Medical Examiners.
12.
Fraser, C. (1986). NOHARM: An IBM PC computer program for fitting both unidimensional and multidimensional normal ogive models of latent trait theory. ArmidaleNew South Wales, Australia: The University of New England.
13.
Glas, C.A.W. (1992). A Rasch model with a multivariate distribution of ability. In M. Wilson (Ed.), Objective measurement: Theory into practice (pp. 236-258). Norwood NJ: Ablex.
14.
Green, B.F. (1988, October). Notes on the item information function in the multidimensional compensatory IRT model (Research Report 88-10. Baltimore MD: Psychometric Laboratory, The Johns Hopkins University.
15.
Greenburg, A.G., Case, S.M., Golden, G.S., & Melnick, D.E. (1996, June). Core content of Step 2 of the USMLE using surgery as an example. Paper presented at the 7th Annual International Ottawa Conference, Maastricht, The Netherlands. (Proceedings in press).
16.
Kendall, M.G., & Stuart, A. (1967). The advanced theory of statistics (Vol. 2). New York: Hafner .
17.
Kingsbury, G.G., & Weiss, D.J. (1983). A comparison of IRT-based adaptive mastery testing and a sequential mastery testing procedure. In D. J. Weiss (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 257-283). New York: Academic Press.
18.
Lehmann, E.L. (1983). Theories of point estimation. New York: Wiley.
Lord, F.M. (1980). Applications of item response theory to practical testing problems. Hillsdale NJ: Erlbaum.
21.
Luecht, R.M. (in press). A generalized heuristic for test construction using item response theory. Applied Psychological Measurement.
22.
Luecht, R.M., & Hirsch, T.M. (1992). Item selection using an average growth approximation of target information functions. Applied Psychological Measurement , 16, 41-51.
23.
Luecht, R.M., & Miller, T.R. (1992). Unidimensional calibrations and interpretations of composite traits for multidimensional tests. Applied Psychological Measurement, 16, 279-294.
24.
McDonald, R.P. (1982). Linear versus nonlinear models in item response theory. Applied Psychological Measurement, 6, 379-396.
25.
Miller, T., Reckase, R., Spray, J., Luecht, R., & Davey, T. (in press). Multidimensional item response theory. Iowa City IA: ACT Publications.
Owen, R.J. (1975). A Bayesian sequential procedure for quantal response in the context of adaptive mental testing. Journal of the American Statistical Association, 70,351-356.
29.
Reckase, M.R. (1985). The difficulty of test items that measure more than one dimension. Applied Psychological Measurement, 9, 401-412.
30.
Reckase, M.R., & McKinley, R.L. (1983, April). The definition of difficulty and discrimination for multidimensional item response theory models . Paper presented at the annual meeting of the American Educational Research Association, Montreal.
Stocking, M.L. (1993). Controlling item exposure rates in a realistic adaptive testing paradigm (ETS Research Report 93-2. Princeton NJ: Educational Testing Service.
33.
Stocking, M.L., & Lewis, C. (1995). A new method of controlling item exposure in computerized adaptive testing (ETS Research Report 95-25. Princeton NJ: Educational Testing Service.
34.
Stocking, M.L., & Swanson, L. (1993). A method for severely constrained item selection in adaptive testing. Applied Psychological Measurement, 17, 277-292.
35.
Swanson, D.B., Case, S.M., Kelley, P.R., Lawley, J.L., Nungester, R.J., Powell, R.D., & Volle, R.L. (1991). Phase-in of the NBME Comprehensive Part I Examination . Ideas for Medical Education, 66, 443-444.
36.
Swanson, L., & Stocking, M.L. (1993). A model and heuristic for solving large item selection problems. Applied Psychological Measurement, 17, 151-166.
37.
Sympson, J.B., & Hetter, R.D. (1985). Controlling item exposure rates in computerized adaptive testing. Proceedings of the 27th Annual Meeting of the Military Testing Association (pp. 973-977). San Diego CA: Navy Personnel Research and Development Center.
38.
Verhelst, N.D., & Glas, C.A.W. (1995). The one parameter logistic model. In G. H. Fischer & I. W. Molenaar (Eds.), Rasch models: Foundations, recent developments and applications (pp. 215-237). New York: Springer-Verlag.
39.
Wainer, H., Dorans, N.J., Flaugher, R., Green, B.F., Mislevy, R., Steinberg, L., & Thissen, D. (1990). Computerized adaptive testing: A primer. Hillsdale NJ: Erlbaum.
40.
Wang, M.M. (1986, April). Fitting a unidimensional model to multidimensional item response data. Paper presented at the Office of Naval Research Contractors' Meeting, Knoxville TN.
41.
Weiss, D.J., & Kingsbury, G.G. (1984). Application of computerized adaptive testing to educational problems. Journal of Educational Measurement, 21, 361-375.
42.
Wilson, D., Wood, R.L., & Gibbons, R. (1984). TESTFACT: Test scoring and item factor analysis [Computer program]Chicago: Scientific Software, Inc.
43.
Wright, B.D., & Stone, M.H. (1979). Best test design. Chicago: MESA Press.
