Using Multidimensional CAT to Administer a Short,Yet Precise,Screening Test

Abstract

Multidimensional computerized adaptive testing (MCAT) provides a mechanism by which the simultaneous goals of accurate prediction and minimal testing time for a screening test could both be met. This article demonstrates the use of MCAT to administer a screening test for the Computerized Adaptive Testing–Armed Services Vocational Aptitude Battery (CAT-ASVAB) under a variety of manipulated conditions. CAT-ASVAB is a test battery administered via unidimensional CAT (UCAT) that is used to qualify applicants for entry into the U.S. military and assign them to jobs. The primary research question being evaluated is whether the use of MCAT to administer a screening test can lead to significant reductions in testing time from the full-length selection test, without significant losses in score precision. Different stopping rules, item selection methods, content constraints, time constraints, and population distributions for the MCAT administration are evaluated through simulation, and compared with results from a regular full-length UCAT administration.

Keywords

CAT domain scores MCAT MIRT multidimensional item response theory multidimensional information overall scores subscores

Introduction

In a selection setting, significant savings could be realized if potential applicants could be administered a screening test beforehand that would yield an accurate prediction/estimation of their true score with a minimal amount of testing time. The selection test could then be administered only to examinees whose predicted/estimated performance from the screening test suggests they would meet the qualification requirements on the full-length test. Multidimensional computerized adaptive testing (MCAT) provides a mechanism by which the simultaneous goals of accurate prediction and minimal testing time for the screening test could both be met. This is similar to previous research for paper-and-pencil (PP) tests that showed using multidimensional item response theory (MIRT) can improve the precision of ability estimates by borrowing information from each content domain (Yao, 2010; Yao & Boughton, 2007), while also considering the response time for each of the content domains in a CAT mode.

This article demonstrates the use of MCAT to administer a screening test for the Computerized Adaptive Testing–Armed Services Vocational Aptitude Battery (CAT-ASVAB) under a variety of manipulated conditions. CAT-ASVAB is a test battery administered via unidimensional CAT (UCAT) that is used to qualify applicants for entry into the U.S. military and assign them to jobs. The primary research question being evaluated is whether the use of MCAT to administer a screening test can lead to significant reductions in testing time from the full-length selection test, without significant losses in score precision.

To answer the primary research question, a simulation study is conducted using ASVAB item parameters to administer a 20-item screening test under a variety of manipulated conditions: (a) fixed-length test versus variable-length test; (b) content constraints versus no content constraints; (c) three conditions regarding response time management: no response time constraint, Methods 1 and 2 with response time constraints; (d) four levels of correlations between domains for the population distribution; and (e) three sets of CAT pools. The manipulated conditions were chosen to help evaluate how best to implement a CAT-ASVAB screening test under MCAT administration. The results for all varying conditions on the screening test are compared over 20 replications by examining average bias in ability estimates, average absolute bias, correlations between estimated abilities and their true values, average response time used, and average test length. Results from the screening test are also compared with results from a full-length unidimensional CAT-ASVAB administration using the same criteria.

Multidimensional Item Response Theory (MIRT) Models

Following the notation of the MIRT model in Yao and Schwarz (2006), for a dichotomously scored item $j,$ the probability of a correct response to item $j$ for an examinee with the ability $\vec{θ} = (θ_{1}, \dots, θ_{D})$ for the multidimensional three-parameter logistic (M-3PL; Reckase, 1997) model is as follows:

P_{j 1} = P_{j 1} (\vec{θ}) = P (x_{j} = 1 | \vec{θ}, {\vec{β}}_{j}) = β_{3 j} + \frac{1 - β_{3 j}}{1 + e^{(- {\vec{β}}_{2 j} ⊙ {\vec{θ}}^{T} + β_{1 j})}},

where $x_{j} = 0$ or $1$ is the response of the examinee to item $j .$ ${\vec{β}}_{2 j} = (β_{2 j 1}, \dots, β_{2 jD})$ is a vector of dimension $D$ for the item discrimination parameters. $β_{1 j} / ‖ {\vec{β}}_{2 j} ‖$ is the difficulty parameter, $β_{3 j}$ is the guessing parameter, and ${\vec{β}}_{2 j} ⊙ {\vec{θ}}_{i}^{T} = \sum_{l = 1}^{D} β_{2 jl} θ_{il} .$ The norm is defined as $‖ {\vec{β}}_{2 j} ‖ = \sqrt{\sum_{l = 1}^{D} β_{2 j l}^{2}} .$ The parameters for the $j th$ item are ${\vec{β}}_{j} = ({\vec{β}}_{2 j}, β_{1 j}, β_{3 j}) .$ Please note that only the M-3PL model is introduced here, as the items in the simulation study are all multiple-choice items.

The item and test information function (indicated by $I_{j} (\vec{θ})$ for item $j$ and $I_{J} (\vec{θ})$ for a total of $J$ items, respectively) and the standard error of estimates (SEE) were derived in Yao (2012). The composite score $θ_{\vec{α}} = \sum_{l = 1}^{D} θ_{l} w_{l}$ has a standard error of measurement $S E E (θ_{\vec{α}}) = V {(θ_{\vec{α}})}^{1 / 2},$ where $V (θ_{\vec{α}}) = \vec{w} V (\vec{θ}) {\vec{w}}^{T},$ $\vec{w} = (w_{1}, …, w_{D}) .$ $V (\vec{θ})$ can be approximated by $I_{J} {(\vec{θ})}^{- 1} .$ The SEE for the Bayesian version is similarly derived as the non-Bayesian version, with the information matrix added by $Σ^{- 1},$ where $\vec{θ} ~ N (μ, Σ) .$

In this study, Maximum a posteriori (MAP) is used by finding the mode that maximizes the posterior distribution. Yao (in press) has found that MAP yields better precision than maximum likelihood (MLE). Strong priors are applied, which may yield better precision.

Applications

Data

The purpose of the CAT-ASVAB screening test is to predict/estimate a score on the Armed Forces Qualification Test (AFQT). AFQT scores are computed from four ASVAB subtests, Arithmetic Reasoning (AR), Word Knowledge (WK), Paragraph Comprehension (PC), and Mathematical Knowledge (MK), and are used for selection into the military.

Simulation studies using three sets of item pools are used to evaluate the MCAT approach to estimating AFQT scores. The item parameters used to generate responses come from real item pools for the AR, WK, PC, and MK tests. The first item pool contains a large number of item parameters sampled proportionally from operational CAT pools, and represents an idealized case for a screening test, where there are many items available to select from in the MCAT administration. The pool contains 2,845 items in total, with 807, 839, 446, and 753 items for AR, WK, PC, and MK, respectively.

For the second item pool, the item parameters come from four operational PP forms; they are combined and put together to form the item pool. There are 420 available items in total, with 120, 140, 60, and 100 items for AR, WK, PC, and MK, respectively.

The third item pool contains a small number of item parameters sampled proportionally from operational CAT pools, and represents a more likely scenario for a screening test, where there are relatively few items available to select from in the MCAT administration. The pool contains 253 items in total, with 69, 74, 43, and 67 items for AR, WK, PC, and MK, respectively.

The summary statistics for the third item pool (small number of CAT items) are very similar to the first item pool (large number of CAT items). The second item pool (moderate number of PP items) has lower information than the other item pools (CAT items). The summary statistics for the item pools are displayed in Table 1.

Table 1.

Item Statistics for the Three Item Pools.

	M			SD
	Discrimination	Difficulty	Guessing	Discrimination	Difficulty	Guessing
Content	$β_{j 1} / 1.7$	$β_{j 2} / β_{j 1}$	$β_{j 3}$	$β_{j 1} / 1.7$	$β_{j 2} / β_{j 1}$	$β_{j 3}$	Number of items	Average response time
Item Pool 1 (CAT)
AR	1.43	−0.03	0.179	0.40	1.15	0.069	807	79
WK	1.57	−0.11	0.213	0.39	1.24	0.074	839	14
PC	1.29	−0.31	0.189	0.31	1.02	0.059	446	69
MK	1.51	0.33	0.162	0.31	1.02	0.073	753	42
Item Pool 2 (PP)
AR	1.14	0.31	0.192	0.34	0.95	0.092	120	79
WK	1.07	−0.15	0.223	0.39	1.43	0.113	140	14
PC	0.92	−0.56	0.186	0.29	0.96	0.076	60	69
MK	1.11	0.19	0.210	0.44	0.77	0.100	100	42
Item Pool 3 (CAT)
AR	1.32	0.01	0.174	0.28	1.11	0.077	69	79
WK	1.53	−0.15	0.226	0.36	1.23	0.081	74	14
PC	1.26	−0.37	0.199	0.25	1.06	0.050	43	69
MK	1.40	0.45	0.158	0.46	0.97	0.075	67	42

Note. The response time is in unit second. AR = Arithmetic Reasoning; WK = Word Knowledge; PC = Paragraph Comprehension; MK = Mathematical Knowledge; PP = Paper and Pencil; CAT = computerized adaptive testing.

The AFQT composite score for the item pools derived from operational CAT pools (Pools 1 and 3) is obtained based on the four content scores, expressed as $θ_{AFQT} = 1.34 [w_{AR} θ_{AR} + w_{MK} θ_{MK} + w_{PC} θ_{PC} + w_{MK} θ_{WK}] - 0.33,$ where $w_{AR} = 0.25,$ $w_{WK} = 0.31,$ $w_{PC} = 0.22,$ and $w_{MK} = 0.22 .$ The slope is $a = 1.34,$ and the intercept is $b = - 0.33$ . For the pool created from operational PP forms (Pool 2), different weights were derived and the relation is expressed as $θ_{AFQT} = 1.18 [w_{AR} θ_{AR} + w_{MK} θ_{MK} + w_{PC} θ_{PC} + w_{MK} θ_{WK}] - 0.28,$ where $w_{AR} = 0.23,$ $w_{WK} = 0.34,$ $w_{PC} = 0.20,$ and $w_{MK} = 0.23 .$ Theslope is $a = 1.18,$ and the intercept is $b = - 0.28 .$ More details are presented in Segall (2004).

The average item response times for the four content areas AR, WK, PC, and MK are 79, 14, 69, and 42, in unit seconds, respectively; they are derived by computing the means of the response times for all the items in each content area for all 4,839,372 examinees taking the CAT-ASVAB from 1998 to 2012. Using separate response times for each item is possible, however, using the average response time for each content area makes the computation simple and should not affect the results of this study.¹

MCAT Simulation Design

Item selection procedures

For the AFQT composite score, the pre-determined weight $\vec{w} = (w_{AR}, w_{WK}, w_{PC}, w_{WK})$ is described earlier. Please note that in practice, the overall score may not be a linear combination of the domain scores as in this study; conditional SEE (conditioning on the $\vec{θ}$ ) may be used in a similar manner (Yao, 2012). The criteria is to minimize the error variance for the AFQT score (van der Linden, 1999), while also minimizing the total test time. As the response times across the four content areas have large variation, the selection procedure will favor items in the order of WK, MK, PC, and AR, if no content constraints are specified.

For each item $j,$ assign a variable ${Time}_{j}$ based on its content. Let $W_{time}$ be a weight imposed on the time. For the MCAT item selection procedure, the initial abilities are set to ${\vec{θ}}_{0} = (0, 0, \dots, 0) .$ The first item selected is an item $j$ such that $\vec{w} {(I_{j} ({\vec{θ}}_{0}))}^{- 1} {(\vec{w})}^{T} + W_{time} \times Tim e_{j} = min {\vec{w} {(I_{m} ({\vec{θ}}_{0}))}^{- 1} {(\vec{w})}^{T} + W_{time} \times Tim e_{m}, m = 1, \dots, J} .$ For $j = 2, …, J,$ suppose $j - 1$ items have been selected. To select the next $j th$ item, suppose the updated ability is ${\vec{θ}}^{j - 1} .$ The following steps are used in selecting items.

For each item m, compute its information and add to the current posterior information, $I_{j - 1}^{m} = I_{j - 1} + I_{m} + Σ^{- 1} .$

Select item $m = j$ such that,

\vec{w} {[I_{j - 1}^{m} ({\vec{θ}}^{j - 1})]}^{- 1} {(\vec{w})}^{T} + W_{time} \times {Time}_{m}

has a minimum value (Method 1, labeled “D1” in the results), or

\frac{{SEE}_{j - 1} ({\vec{θ}}^{j - 1}) - \vec{w} {[I_{j - 1}^{m} ({\vec{θ}}^{j - 1})]}^{- 1} {(\vec{w})}^{T}}{{Time}_{m}^{W_{time}}}

has a maximum value (Method 2, labeled “D2” in the results).

Update ability ${\vec{θ}}^{j}$ and information $I_{j} ({\vec{θ}}^{j})$ based on the selected $j$ items.

Compute the current error variance $SE E_{j} ({\vec{θ}}^{j}) = \vec{w} {[I_{j} ({\vec{θ}}^{j})]}^{- 1} {(\vec{w})}^{T} .$

Compute the current subscale estimates ${\vec{θ}}^{j}$ and the updated AFQT score.

Item exposure rate is not considered in the item selection.²

Population distributions

Three samples of 3,000 examinees (labeled P1-P3 in the results) are simulated from a multivariate normal distribution with a mean of (0,0,0,0) where correlations between the domains are $. 2,$ $. 5,$ and $. 8,$ respectively. The correlations for the prior in the MAP ability estimates are set to be .2, .3, and .6 for the three populations, respectively. To mimic the real situation, a fourth sample of size 3,000 (labeled P4 in the results) is simulated from a multivariate normal distribution with a mean of (0,0,0,0) where correlations between the domains are those observed in CAT-ASVAB data. The prior used in these MAP estimates is the same as the true values; in practice, the correlations can be obtained from row data. Given the four “true” domain abilities AR, WK, PC, and MK, the “true” overall ability AFQT score is obtained depending on the item pool used. For each of the 3,000 examinees, 20 different seeds are used, resulting in 20 replications.

Response time constraints

There are two variations for $W_{time}$ when using item selection Method 1 to select items: They are 0 (labeled W1 in the results) and 0.0005 (labeled W2 in the results). There are two variations for $W_{time}$ when using item selection Method 2 to select items; they are 0.5 (labeled W1 in the results) and 1 (labeled W2 in the results). The choices for $W_{time}$ were based on the values of the two parts in Equation 2 (SEE and the response time) and the top and bottom of Equation 3 (difference of SEEs and response time). Larger weight on response time results in smaller test time but smaller reliability, and smaller weight results in larger test time but larger reliability; experimenting with the weight is suggested. For a screening test, it is desirable to have a test response time less than 15 min and in the meantime, it is desirable to have a good reliability. Please note that when $W_{time} = 0,$ Methods 1 and 2 are the same.

Content constraints

Constraints are imposed here that require the minimum number of items administered to be 0, 1, 2, or 3 for each of the four content areas under MCAT administration. The conditions are labeled M0, M1, M2, or M3, respectively, in the results.

Fixed versus variable-length tests

A CAT administration process is a cyclical procedure that utilizes a stopping rule (Reckase, 2009; Wainer, 2000). The stopping rule can specify a fixed number of test items be administered (a fixed-length test), or a desired level of precision be reached (a variable-length test). Previous studies on MCAT item selection procedures have been conducted under the fixed-length condition (Huang, Chen, & Wang, 2012; Li & Schafer, 2005; Luecht, 1996; Mulder & van der Linden, 2009; Veldkamp & van der Linden, 2002; W.-C. Wang & Chen, 2004; Yao, 2012, 2014a). However, MCAT selection procedures with varying test lengths using different stopping rules deserve more attention (C. Wang, Chang, & Boughton, 2013; Yao, 2013). In Yao (2013), variable-length stopping rules were used for five MCAT item selection procedures, but the precision requirement and the selection criteria were imposed on the subscores, not the composite scores. This study focuses on the composite score in both the selection criteria and the precision requirement. To achieve the same level of precision, test length may need to vary across examinees, depending on the examinees’ ability, the quality of the item pool, and the CAT selection procedures.

For each variation, two procedures are applied; one is a fixed-length test of 20 items (labeled F in the results), and another is a variable-length test (labeled V in the results) that is stopped when a maximum test length of 20 is reached or when the precision or SEE for the AFQT score reaches a level of 0.25. The choice of the precision level depends on the quality of the item pool (i.e., information); a precision level of 0.25 is appropriate for the item pools in this study, and it is comparable with the precision level for a test of fixed length 20.³ The simulation conditions are summarized in Table 2.

Table 2.

Simulation Conditions Under MCAT Administration.

Item pool	Method	Weight $W_{time}$	Population	Minimum required number of items	Number of replications	Test length
Pool 1	FD1	w1 = 0	$P_{1} (r = . 2)$	0	20	20
Pool 2	VD1	w2 = .0005	$P_{2} (r = . 5)$	1
Pool 3	FD2	w1 = .5	$P_{3} (r = . 8)$	2
	VD2	w2 = 1	$P_{4} (r = observedcorrelation)$	3

Note. MCAT = multidimensional computerized adaptive testing; F = fixed length; V = variable length; D1 = Method 1; D2 = Method 2.

Evaluation Criteria

A number of criteria are used to compare results across the different conditions. They include (a) the bias and absolute value of the bias, averaged over replications and examinees (labeled Average Bias [BIAS] and Average Absolute Value of the Bias [ABSBIAS], respectively); (b) the correlation between estimated and true ability values, averaged over replications; (c) the reliability of the ability estimates; (d) false positive and negative rates; (e) response time, averaged over examinees and replications; and (f) test length, averaged over examinees and replications. The reliability is the average of the reliabilities over n replications and the reliability is the square of the correlations between the estimates and the true values. False positive and negative rates at two cut score points are examined: AFQT scores of 31 and 50.⁴ False negative is defined as the percentage where the true value is classified as pass but the estimate is classified as fail; false positive is defined as the percentage where the true value is classified as fail but the estimate is classified as pass. AFQT cut scores of 31 and 50 correspond to $θ_{AFQT} = - 0.5$ and $θ_{AFQT} = 0.06$ for the CAT and PP item pools.

Results

Comparison of MCAT screening test and UCAT full-length test

First, results are compared between the MCAT screening test and the UCAT full-length test for population P4 using Pool 3; the results are presented in Table 3. The bottom line in Table 3 presents the results for a UCAT simulation under the current operational CAT-ASVAB setting and is considered as the baseline from which to compare performance of the MCAT administrations. As expected, the MCAT results all show much shorter average testing times than the UCAT administration (7-17 min vs. 45 min). The reduced number of items administered (20 or fewer items for the MCAT case vs. 55 items for the UCAT case) is a contributing factor to that finding. As might be expected, reliability is highest for the UCAT case, again corresponding to the longer test length. However, reliability of some MCAT conditions approaches that of the UCAT case, for fixed-length test conditions. On all evaluation criteria, the MCAT results show some loss in precision when compared with the UCAT case; however, the loss in precision does not appear substantial, especially for the fixed-length conditions. For a screening test, a smaller false negative rate (the percentage where the true value is classified as pass but the estimate is classified as fail) is desirable. There are some 20-item MCAT conditions that have a smaller false negative rate than the full-length UCAT test, which means that fewer qualified examinees will be screened out by the screening test. However, a non-qualified examinee (AFQT $< 31$ ) may be classified as qualified if taking a full-length CAT-ASVAB test (false positive rate at 31 = 4.795). However, if a MCAT screening test is taken beforehand, then most of those non-qualified examinees will be screened out (e.g., false positive rate at 31 = 5.98 for condition FP4D2M1W2). Because screened out examinees will not be allowed to take the full-length test, significant savings could be achieved, although there may be still some non-qualified examinees that pass the screening test and go on to take the official full-length test.

Table 3.

BIAS, ABSBIAS, Correlations, Reliability, False Negative and False Positive Rates at Cut Score AFQT = 31 and 50, Average Response Time (in Minutes), and Average Test Length for Population P4 for Pool 3 for MCAT With Test Length 20 and a Full-Length UCAT.

Method and condition	BIAS	ABSBIAS	Correlation	Reliability	At AFQT = 31		At AFQT = 50		Time	Length
					False negative	False positive	False negative	False positive
MCAT Test length of 20
FP4D1M0W1	.091	.270	.9546	.9113	2.985	6.138	3.323	4.723	16.44	20
FP4D1M1W1	.088	.269	.9550	.9120	3.015	6.018	3.370	4.683	16.52	20
FP4D1M0W2	.075	.367	.9142	.8358	4.840	8.156	5.368	5.818	6.96	20
FP4D1M1W2	.071	.357	.9191	.8447	4.440	7.625	5.833	5.113	7.70	20
FP4D2M0W1	.091	.270	.9546	.9113	2.985	6.138	3.323	4.723	16.44	20
FP4D2M1W1	.088	.269	.9550	.9120	3.015	6.018	3.370	4.683	16.52	20
FP4D2M0W2	.086	.283	.9497	.9019	3.436	6.230	3.588	4.643	11.95	20
FP4D2M1W2	.084	.282	.9501	.9027	3.411	5.980	3.765	4.545	12.18	20
VP4D1M0W1	.102	.325	.9335	.8714	2.148	3.473	4.308	6.085	10.74	13
VP4D1M1W1	.098	.325	.9335	.8714	3.341	8.021	4.438	5.726	10.90	13
VP4D1M0W2	.074	.367	.9141	.8356	3.400	7.776	5.411	5.816	6.96	20
VP4D1M1W2	.073	.360	.9180	.8427	4.796	8.128	5.711	5.173	7.70	20
VP4D2M0W1	.102	.325	.9335	.8714	4.451	7.796	4.308	6.085	10.74	13
VP4D2M1W1	.098	.325	.9335	.8714	3.341	8.021	4.438	5.726	10.90	13
VP4D2M0W2	.087	.317	.9363	.8767	3.400	7.776	4.468	5.683	9.06	15
VP4D2M1W2	.089	.318	.9360	.8760	3.518	7.540	4.575	5.415	9.33	15
UCAT with 10 PC items and 15 items each for AR, WK, and MK for a total of 55 items
FP4D1M0W1	.048	.240	.96420	.92968	3.201	4.795	3.44	3.553	45.25	55

Note. BIAS = Average Bias; ABSBIAS = Average Absolute Value of the Bias; AFQT = Armed Forces Qualification Test; MCAT = multidimensional computerized adaptive testing; UCAT = unidimensional computerized adaptive testing; F = fixed length; V = variable length; D1 = Method 1; D2 = Method 2; M0 = minimum item number is 0; M1 = minimum item number is 1; W1 = $W_{time}$ is 0 for Method 1; W2 = $W_{time}$ is 0.0005 for Method 1; W1 = $W_{time}$ is 0.5 for Method 2; W2 = $W_{time}$ is 1 for Method 2.

Comparison of MCAT conditions

To further evaluate how MCAT might be implemented operationally, results are compared between item selection Method 1 (D1) and item selection Method 2 (D2), between content constraints (M1, M2, M3) and no content constraints (M0), between weights for the time constraints (W0 vs. W1), between variable-length (V) tests and fixed-length (F) tests, across the three item pools, and across the four sets of samples (P1-P4). For populations P1 to P3 under MCAT administration, content constraints M0 and M1 were imposed. For population P4 under MCAT administration, content constraints M0, M1, M2, and M3 were imposed. In general, it is observed that for item selection Method 1, if there are no time constraints and no content constraints (condition D1M0W1), the procedure selects items from all four content areas, where WK has the most items selected, followed by AR, MK, and PC. The number of items selected are similar in the condition where there are no time constraints and there are content constraints (D1M1W1). For Method 1, when there are time constraints, the procedure selects most of the items from WK, with some from MK. AR and PC items are selected for only some examinees but the numbers are very small. Compared with Method 1, Method 2 selects more items from AR, PC, and MK, and the response time is higher. Tables and figures are used to demonstrate the results. Both summary statistics and individual-level information are examined.

Table 4 summarizes the BIAS, ABSBIAS, correlation, reliability, percentage of misclassification, average response time, and average test length for population P2 (r = .5) for the three pools and for both the fixed-length and variable-length tests. For Method 2, two weights for the times $W_{time} = 0.5$ and $W_{time} = 1$ were applied. The results for Pools 1 and 3 are very similar, even though Pool 1 is much larger than Pool 3, likely because their item statistics are very similar. Compared with Pools 1 and 3, Pool 2 has a longer average test length for the variable-length test; this is expected, as the information for Pool 2 is smaller than the other two pools. Condition D1M0W2 has the smallest response time and also the smallest correlation/reliability and the highest percentage of misclassification. Compared with condition D1M0W2, condition D1M1W2, which requires a minimum of one item for each content area has a slightly higher response time and correlation/reliability, and smaller percentage of misclassification. Condition VD2M0W2 has the next smallest response time.

Table 4.

BIAS, ABSBIAS, Correlations, Reliability, Percentage of Misclassification, Average Response Time (in Minutes), and Average Test Length for Population P2 for the Three Pools With Test Length = 20.

Method and condition	BIAS	ABSBIAS	Correlation	Reliability	At AFQT = 31		At AFQT = 50		Time	Length
					False negative	False positive	False negative	False positive
Pool 1
FP2D1M0W1	.115	.256	.9559	.9138	0.23	4.07	0.70	2.00	16.41	20
FP2D1M1W1	.115	.256	.9562	.9143	0.30	4.00	0.77	1.77	16.42	20
FP2D1M0W2	.091	.367	.9125	.8326	1.87	5.90	4.77	3.20	7.63	20
FP2D1M1W2	.082	.365	.9152	.8375	1.60	6.10	4.80	2.53	7.74	20
FP2D2M0W1	.118	.258	.9553	.9126	0.23	4.30	0.70	2.17	14.75	20
FP2D2M1W1	.116	.257	.9556	.9132	0.27	3.83	0.80	1.70	14.84	20
FP2D2M0W2	.124	.272	.9513	.9050	0.37	4.57	0.73	2.50	13.07	20
FP2D2M1W2	.123	.270	.9522	.9067	0.27	4.13	0.87	2.20	13.18	20
VP2D1M0W1	.135	.378	.9055	.8200	0.07	8.23	2.63	2.20	8.27	10
VP2D1M1W1	.130	.374	.9064	.8216	0.20	7.37	2.73	2.13	8.35	10
VP2D1M0W2	.094	.370	.9107	.8293	1.33	6.80	4.20	3.70	7.39	19
VP2D1M1W2	.085	.370	.9132	.8340	1.33	5.97	4.70	2.60	7.65	20
VP2D2M0W1	.125	.370	.9072	.8230	0.17	7.67	2.63	2.73	7.72	11
VP2D2M1W1	.124	.371	.9090	.8262	0.30	7.13	2.87	2.40	7.80	11
VP2D2M0W2	.113	.360	.9128	.8331	0.50	7.07	2.93	3.20	7.50	12
VP2D2M1W2	.116	.362	.9137	.8348	0.17	6.90	2.70	2.67	7.64	12
Pool 2
FP2D1M0W1	.096	.278	.9342	.8727	2.921	8.238	4.258	5.861	15.48	20
FP2D1M1W1	.095	.278	.9339	.8723	2.960	8.481	4.260	5.793	15.50	20
FP2D1M0W2	.091	.355	.8908	.7935	4.255	10.406	6.773	6.578	7.26	20
FP2D1M1W2	.082	.350	.8958	.8025	3.993	10.130	6.808	6.233	7.55	20
FP2D2M0W1	.098	.280	.9327	.8700	2.850	8.465	4.190	5.955	13.52	20
FP2D2M1W1	.095	.281	.9332	.8708	2.950	8.556	4.345	5.871	13.63	20
FP2D2M0W2	.096	.290	.9283	.8618	3.036	8.813	4.531	6.196	11.57	20
FP2D2M1W2	.095	.290	.9286	.8624	3.036	8.761	4.728	5.866	11.65	20
VP2D1M0W1	.098	.297	.9240	.8538	2.925	9.005	4.755	6.555	13.31	17
VP2D1M1W1	.094	.297	.9249	.8554	2.878	8.856	4.763	6.443	13.33	17
VP2D1M0W2	.091	.355	.8908	.7935	4.255	10.406	6.773	6.578	7.26	20
VP2D1M1W2	.082	.350	.8958	.8025	3.993	10.130	6.808	6.233	7.55	20
VP2D2M0W1	.097	.296	.9248	.8553	2.903	9.083	4.633	6.753	11.90	18
VP2D2M1W1	.095	.297	.9247	.8550	2.931	8.913	4.565	6.456	12.02	18
VP2D2M0W2	.095	.299	.9243	.8543	2.950	8.888	4.725	6.490	10.76	19
VP2D2M1W2	.093	.300	.9228	.8516	3.081	8.951	4.755	6.500	10.91	19
Pool 3
FP2D1M0W1	.099	.281	.9473	.8974	3.096	6.551	3.790	4.903	16.49	20
FP2D1M1W1	.099	.280	.9472	.8971	2.941	6.591	3.813	4.810	16.52	20
FP2D1M0W2	.073	.381	.9044	.8179	4.475	8.793	6.576	5.416	7.46	20
FP2D1M1W2	.070	.378	.9060	.8208	4.171	9.015	6.583	5.253	7.51	20
FP2D2M0W1	.099	.284	.9468	.8964	3.028	6.405	3.948	4.818	14.69	20
FP2D2M1W1	.099	.281	.9472	.8973	3.103	6.443	3.835	4.763	14.74	20
FP2D2M0W2	.097	.292	.9428	.8889	3.268	6.613	4.130	4.750	12.73	20
FP2D2M1W2	.094	.290	.9434	.8899	3.238	6.765	4.095	4.791	12.80	20
VP2D1M0W1	.103	.341	.9212	.84864	3.346	8.851	5.085	5.975	10.65	13
VP2D1M1W1	.103	.339	.9231	.85222	3.155	9.001	5.126	5.978	10.63	13
VP2D1M0W2	.073	.381	.9044	.81793	4.475	8.793	6.576	5.416	7.46	20
VP2D1M1W2	.070	.378	.9060	.82084	4.171	9.015	6.583	5.253	7.51	20
VP2D2M0W1	.100	.335	.9237	.85335	3.255	8.723	4.923	5.720	9.88	14
VP2D2M1W1	.096	.336	.9232	.85247	3.345	8.760	5.161	5.691	9.91	13
VP2D2M0W2	.095	.334	.9253	.85627	3.378	8.393	5.151	5.585	9.48	15
VP2D2M1W2	.090	.333	.9254	.85634	3.225	8.563	5.173	5.498	9.55	15

Note. BIAS = Average Bias; ABSBIAS = Average Absolute Value of the Bias; AFQT = Armed Forces Qualification Test; F = fixed length; V = variable length; D1 = Method 1; D2 = Method 2; M0 = minimum item number is 0; M1 = minimum item number is 1; W1 = $W_{time}$ is 0 for Method 1; W2 = $W_{time}$ is 0.0005 for Method 1; W1 = $W_{time}$ is 0.5 for Method 2; W2 = $W_{time}$ is 1 for Method 2.

To examine the effect of population correlations, Figures 1 and 2 show the correlations, response time, and test length for the fixed-length and variable-length tests for Pool 3 for populations P1 to P3 for content constraints M0 and M1, respectively. As the population correlation increases, response time decreases and the correlation between the estimates and the true values for the four content areas and AFQT increases, especially for AR and MK. As strong priors are applied in the MAP estimate, the subscore estimates for AR and WK are improved as the correlation between the content areas increases. Method 1 with time constraints of $W_{time} = 0.0005$ and no minimum item requirement (condition D1M0W2) yields the smallest response time; on average, the response times are 7.35, 7.39, and 6.99 min for the three populations P1 to P3, respectively, for Pool 1.

Figure 1.

Correlations and response time for a fixed-length test for the three populations P1-P3, two item selection methods Dl and D2, and time constraints and content constraints for Pool 3 with a 20-item test and 20 replications.

Figure 2.

Correlations, response time, and test length for a variable-length test for the three populations P1-P3, item selection methods Dl and D2, time constraints and content constraints for Pool 3 with a 20-item test and 20 replications.

Compared with no content constraints (i.e., no minimum required item number, indicated by M0), a slightly larger time is observed when content constraints are imposed (indicated by M1). If time is not considered (indicated by conditions D1M0W1 and D1M1W1) in selecting items, the precision is the best and the test length is the shortest, however, the response time is the largest and, on average, it can be as high as 16.5 min for the fixed-length test and 8.27 min for the variable-length test. Figure 3 shows the response time and test length against their true ability values for all 3,000 examinees for population P4 using Pool 3 for MCAT administration with a 20-item test. Differences between Methods 1 and 2, between fixed-length and variable-length tests, and between time constraints and no time constraints can be visually observed.

Figure 3.

Response time and test length against their true ability values for the 3,000 examinees for P4, applied to item Pool 3, with a 20-item test using MCAT.

To examine the differences between fixed-length and variable-length tests and how the stopping rules perform, SEEs are examined for some conditions. Figure 4 displays the SEE against the true ability values for all 3,000 examinees for population P2 with Pool 1 with a 20-item test for both Methods 1 and 2. It is observed that when there are no time constraints and content constraints (conditions FP2D1M0W1, VP2D1M0W1), the SEEs are smaller than 0.25 for most of the examinees. The fixed-length test has a smaller SEE than the variable-length test (e.g., FP2D1M0W1 vs. VP2D1M0W1), as the test length for the variable length is at most 20 items (the length for the fixed-length test). When there are time constraints, the SEEs are higher than those without time constraints (e.g., FP2D1M0W2 vs. FP2D1M0W1). Method 2 has smaller SEEs than Method 1 (e.g., FP2D2M0W2 vs. FP2D1M0W2 and VP2D2M0W2 vs. VP2D1M0W2).

Figure 4.

SEEs against their true ability for the 3,000 examinees for Methods 1 and 2 for population P2 with Pool 1 with a 20-item test.

A SAS reg procedure was conducted for the estimated AFQT and the estimated four content domain area scores and the true relations were recovered for all the conditions with the default error. Table 5 lists the number of selected items for each of the four content areas for the fixed-length and variable-length tests for population P2 for content constraints M0 and M1; it is clear that many more items from WK were selected for Method 1 than Method 2.

Table 5.

Number of Selected Items by Content Area for a 20-Item Test With Population P2 and Pool 1 Averaged Over Replications.

Method and conditions	AR	WK	PC	MK
FP2D1M0W1	50	67	36	39
FP2D1M1W1	51	67	35	39
FP2D1M0W2	8	168	9	9
FP2D1M1W2	4	168	6	9
FP2D2M0W1	38	93	29	38
FP2D2M1W1	38	91	28	36
FP2D2M0W2	29	116	26	35
FP2D2M1W2	27	116	25	35
VP2D1M0W1	20	30	20	15
VP2D1M1W1	21	29	20	15
VP2D1M0W2	8	169	9	9
VP2D1M1W2	4	166	6	9
VP2D2M0W1	17	45	19	15
VP2D2M1W1	14	44	17	15
VP2D2M0W2	13	68	16	13
VP2D2M1W2	15	67	16	14

Note. AR = Arithmetic Reasoning; WK = Word Knowledge; PC = Paragraph Comprehension; MK = Mathematical Knowledge; F = fixed length; V = variable length; D1 = Method 1; D2 = Method 2; M0 = minimum item number is 0; M1 = minimum item number is 1; W1 = $W_{time}$ is 0 for Method 1; W2 = $W_{time}$ is 0.0005 for Method 1; W1 = $W_{time}$ is 0.5 for Method 2; W2 = $W_{time}$ is 1 for Method 2.

Discussion

The simulation results suggest that using MCAT to administer a 20-item screening test can shorten testing time substantially over the unidimensional full-length CAT-ASVAB, while still yielding a precise estimate of actual overall ability. The approach demonstrated here could be useful in any selection setting where a short, yet precise screening test is desired to reduce the rate of unqualified examinees taking the full-length selection test.

Two methods in selecting items were studied. Compared with Method 2, Method 1 selects more WK items (smallest response time) and fewer items from the other three content areas (longer response times), has a longer test length, and has lower precision; the precision differences between the two methods are smaller when the correlations between subscores are higher than 0.5—such correlations can be met in practice. For MCAT ability estimates, strong priors are used, and as the correlation between the abilities for the population distribution increases, the correlation between the estimates and the true values for the four content areas and the overall increases.

Variable-length tests may be considered; certain examinees can reach the precision level with much fewer items and much lower response times. However, fixed-length tests may be easier for users to understand and/or justify.

Three different item pools were examined. For pools with more high-quality items (i.e., more informative), the required precision level can be reached with lower response times if a variable-length test is used. As this is a screening test, no item exposure control is considered. Therefore, a larger item pool does not necessarily yield better results than a smaller item pool, even if the item statistics for these pools are similar (i.e., Pool 1 vs. Pool 3).

A key element to a shorter testing time in the MCAT screening test is the modeling of response time in the item selection method. For example, for item selection Method 1, average testing time is much higher when there is no response time constraint (i.e., the weight assigned to response time in the model is set to 0) than when a response time constraint is used (16-17 min vs. 7-8 min, respectively, for a fixed-length test). When response time is accounted for in the item selection model, the result is a much shorter testing time, but the tradeoff is an imbalance in content, as the test with the shortest item response times (WK) dominates the selected items. The addition of content constraints has a minimal effect on reducing the number of WK items administered, mainly because only a small number of items per content area are constrained to be administered. If content constraints are included that result in a more balanced distribution of content, then the likely tradeoff would be an increase in testing time and possible reduction in reliability.

If high reliability is the primary desired outcome, increasing the test length may not necessarily improve the precision level when response time constraints are used, because many items are selected from WK due to its shorter response time, and even more WK items would be expected to be administered, which may not increase the precision level. For condition D1M0W2, additional simulations were conducted for a test length of 30 and a precision level of 0.2, and the correlation/reliability for AFQT score was no better than that for a test length of 20 and a precision level of 0.25. Therefore, a screening test with item selection Method 1, variable-length test, response time constraints, and a precision level of 0.25 could be implemented operationally with no anticipated loss in score precision when compared with a 30-item test length.

A secondary research question regarding how the MCAT administration compares with a UCAT administration of the screening test was answered through additional simulation, namely, would it be worth it to implement MCAT instead of UCAT for an operational CAT-ASVAB screening test? Readers can find the research design and the results in the online version of the article at the appendix.

Compared with UCAT, the precision increased when MCAT was applied. To facilitate the comparison of UCAT and MCAT, a 55-item MCAT was also simulated. MCAT outperformed UCAT at both the 55- and 20-item test lengths with regard to reliability and test time, but there was a greater gain in performance with the shorter test length. However, there are some UCAT 20-item screening test conditions that look like they could be acceptable in practice, in terms of their relative reliability and testing times. A good UCAT screening test condition can be realized but is best chosen based on the success of the MCAT screening test. A requirement for the minimum number of items in each of the content areas increases the reliability for the MCAT administration, but also increases the average response time. If minimizing testing time is the primary interest in a screening test, the minimum required item number should be less than or equal to 3; under these conditions in MCAT, the rest of the items are selected from WK for most of the examinees, especially for Method 1. More items from the test with the shortest response times (relative to the other three content areas) will decrease both the reliability and the average response time.

In conclusion, MCAT certainly looks to be a promising approach to a screening test; MCAT can incorporate content constraints and response time simultaneously for better prediction and short testing time. In practice, the balance between the precision and the response time (the weight for the two methods) needs to be adjusted based on their values in the application data. The tradeoffs between reliability, testing time, content representation, and the labor in implementing MCAT methods are all factors that need to be considered in choosing an approach for operational implementation of a screening test.

Footnotes

Acknowledgements

We thank the reviewers and the editor for their valuable input on the earlier version of this manuscript.

Authors’ Note

The views expressed are those of the authors and not necessarily those of the Department of Defense or the U.S. government.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Huang

H.-Y.

Chen

P.-H.

Wang

W.-C.

(2012). Computerized adaptive testing using a class of high-order item response theory models. Applied Psychological Measurement, 36, 689-706.

Y.-H.

Schafer

W. D.

(2005). Trait parameter recovery using multidimensional computerized adaptive testing in reading and mathematics. Applied Psychological Measurement, 29, 3-25.

Luecht

R. M.

(1996). Multidimensional computerized adaptive testing in a certification or licensure context. Applied Psychological Measurement, 20, 389-404.

Mulder

van der Linden

W. J.

(2009). Multidimensional adaptive testing with optimal design criterion for item selection. Psychometrika, 74, 273-296.

Reckase

M. D.

(1997). The past and future of multidimensional item response theory. Applied Psychological Measurement, 21, 25-36.

Reckase

M. D.

(2009). Multidimensional item response theory. New York, NY: Springer.

Segall

D. O.

(2004). Development and evaluation of the 1997 ASVAB score scale (Technical Report No. 2004-002). Seaside, CA: Defense Manpower Data Center.

van der Linden

W. J.

(1999). Multidimensional adaptive testing with a minimum error-variance criterion. Journal of Educational and Behavioral Statistics, 24, 398-412.

Veldkamp

B. P.

van der Linden

W. J.

(2002). Multidimensional adaptive testing with constraints on test content. Psychometrika, 67, 575-588.

10.

Wainer

(Ed.). (2000). Computerized adaptive testing: A primer (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

11.

Wang

Chang

H.-H.

Boughton

(2013). Deriving stopping rules for multidimensional computerized adaptive testing. Applied Psychological Measurement, 37, 99-122.

12.

Wang

W.-C.

Chen

P.-H.

(2004). Implementation and measurement efficiency of multidimensional computerized adaptive testing. Applied Psychological Measurement, 28, 295-316.

13.

Yao

(2010). Reporting valid and reliability overall score and domain scores. Journal of Educational Measurement, 47, 339-360.

14.

Yao

(2012). Multidimensional CAT item selection methods for domain scores and composite scores: Theory and applications. Psychometrika, 77, 495-523.

15.

Yao

(2013). Comparing the performance of five multidimensional CAT selection procedures with different stopping rules. Applied Psychological Measurement, 37, 3-23.

16.

Yao

(2014a). Multidimensional CAT item selection procedures with item exposure control and content constraints. Journal of Educational Measurement, 51, 18-38.

17.

Yao

(in press). Multidimensional Item Response Theory for Score Reporting. In Chang

Cheng

H.-H.

(Eds.), Advances in modern, international testing: Transition from summative to formative assessment. Charlotte, NC: Information Age Publishing.

18.

Yao

Boughton

K. A.

(2007). A Multidimensional item response modeling approach for improving subscale proficiency estimation and classification. Applied Psychological Measurement, 31, 83-105.

19.

Yao

Schwarz

R. D.

(2006). A multidimensional partial credit model with associated item and test statistics: An application to mixed-format tests. Applied Psychological Measurement, 30, 469-492.