Adaptive Testing With a Hierarchical Item Response Theory Model

Multidimensional Item Selection and Interim Scoring

The multidimensional item selection method used in this study selects the items that maximize the determinant of the Fisher information matrix in Equation 2:

I ( θ ) = ∑ k = 1 S − 1 I k ( θ ) + I j ( θ ) .

There are S– 1 items administered, and one of the remaining items, j, is selected from the item pool. I _k (θ) and I _j (θ) follow the same equation shown in Equation 3, where P jt ( θ ) is the probability of answering item j in or above score category t.

I j ( θ ) = ∑ t = 1 M { P jt ( θ ) × [ 1 − P jt ( θ ) ] − P jt + 1 ( θ ) × [ 1 − P jt + 1 ( θ ) ] } 2 P jt ( θ ) − P jt + 1 ( θ ) [ a j 1 2 a j 1 a j 2 a j 1 a j 2 a j 2 2 … a j 1 a jD … a j 2 a jD … … a jD a j 1 a jD a j 2 … … … a jD 2 ] ,

where M is the number of score categories and D is the number of dimensions included in calculation. For the multidimensional method, D is equal to the number of H-IRT model dimensions, which includes all theta dimensions.

To achieve content balance, the multidimensional method selects the same number of items for each loading pattern by cycling through different groups of item pools. The cycling order of groups was predetermined by an arbitrary dimension sequence established by the content-structure index for both simulation and real item pool studies of this article. The content-structure index for the real item pool of this article is provided by the content experts. Using ARMM as an example, this method first selects one item measuring c factor (as shown in Figure 1) from all items that measure c factor, using the information function in Equation 2. Then the thetas of all dimensions are updated after administering this item. The same procedure is repeated for each of the 14 groups of items, resulting in 15 administered items in the first round. This process is repeated until the predetermined test length is reached.

The multidimensional interim scoring method in this study uses the Bayesian method to estimate the thetas of all dimensions (Segall, 1996; Seo, 2011; van der Linden, 1999; Veldkamp & van der Linden, 2002). When tests are short and the number of dimensions is high, Bayesian estimation methods that are implemented with a reasonable prior distribution provide estimations that converge faster and are more accurate than the maximum likelihood estimation (Segall, 1996; van der Linden, 1999; Veldkamp & van der Linden, 2002). The Bayesian estimation finds the theta vector that maximizes the posterior density function, f ( θ | u ) . The posterior density function follows the Bayes rule and is the combination of the prior distribution function, f ( θ ) , the marginal density of the response data, f ( u ) , and the likelihood function, L ( u | θ ) (Segall, 1996):

f ( θ | u ) = L ( u | θ ) f ( θ ) f ( u ) .

The prior distribution, f ( θ ) , is usually chosen to be a multivariate normal distribution with mean vector µ and covariance matrix Φ. In this study, the prior distribution has µ as a vector, with all elements equal to 0 and Φ as an identity matrix. Thus, the dimensions of the prior distribution are independent. This distribution is also the prior distribution used for ARMM scoring. As thetas of all dimensions are estimated, the likelihood function includes the responses of all items administered. The Newton–Raphson algorithm is used to find the set of thetas that maximize Equation 4. The first derivative vector (also called the Jacobian matrix), ∂ ∂ θ lnf ( θ | u ) , and the second derivative matrix (also called the Hessian matrix), J ( θ ) , of f ( θ | u ) are shown in Equations 5 and 6:

∂ ∂ θ lnf ( θ | u ) = [ ∂ ∂ θ 1 lnf ( θ | u ) ∂ ∂ θ 2 lnf ( θ | u ) … ∂ ∂ θ d lnf ( θ | u ) ] ;

J ( θ ) = [ ∂ 2 ∂ θ 1 2 lnf ( θ | u ) ∂ 2 ∂ θ 1 ∂ θ 2 lnf ( θ | u ) ∂ 2 ∂ θ 2 2 lnf ( θ | u ) … ∂ 2 ∂ θ 1 ∂ θ d lnf ( θ | u ) … ∂ 2 ∂ θ 2 ∂ θ d lnf ( θ | u ) … ∂ 2 ∂ θ D 2 lnf ( θ | u ) ]

The Newton–Raphson algorithm, as in Equation 7, is applied until the difference between the previous and current theta solutions, δ ( j ) , is smaller than a chosen tolerance level:

δ ( j ) = [ J ( θ ( j ) ) ] − 1 × ∂ ∂ θ lnf ( θ ( j ) | u ) .

As the prior distribution function is applied to the likelihood function for Bayesian estimation, the information matrix can also include prior distribution to gain more information in a Bayesian framework. The posterior information matrix, W ( θ ) , which is the expectation of the Hessian matrix including the prior distribution (Segall, 1996), is shown in Equation 8:

W ( θ ) = I ( θ ) + Φ − 1 .

Simplified Item Selection and Interim Scoring

Using a general MIRT model, Mulder and van der Linden (2009) compared the multidimensional item selection method with a simplified method that considered only some of the thetas, which they referred to as intentional thetas. The results indicated the multidimensional method leads to more accurate estimations. As the general MIRT model does not differentiate dimensions, Mulder and van der Linden (2009) randomly chose intentional thetas in their simulation studies. However, for the H-IRT model, choosing intentional thetas is usually model based; for example, a general- or a sub-level theta can be the intentional theta. Randomly choosing intentional thetas for an H-IRT model might not be practical. C. Wang (2014) also compared the multidimensional item selection method with the simplified method, which considers only one theta dimension at a time and found that the multidimensional method leads to more precise results. For the simplified method, only the intentional theta was updated during interim scoring. However, the IRT model used in C. Wang’s (2014) study is a higher order IRT model, and the higher order dimension does not directly govern the item responses. Weiss and Gibbons (2007) proposed a simplified item selection method when using the bifactor model. In this method, item selection is based on the theta of one dimension at a time. They first focused on the general-factor theta, selecting items that can provide maximum Fisher information for this theta; only the general-factor theta was estimated during interim scoring. After the precision criterion of the general-level theta estimation was reached, the previously selected items were used to calculate the initial estimation of theta for the first second-level dimension. This procedure was repeated for all other second-level thetas. After 615 items were administered, the results of their simulation study showed that this simplified item selection and interim scoring method can provide accurate estimations for both general- and second-level thetas, although it is not realistic for students to take a 615-item test. Seo (2011) compared this simplified method with the multidimensional method, using Fisher information during the application of a bifactor model to CAT when the number of items administered was fixed for both item selection methods. Comparison results indicated similar accuracy of the general-level theta estimation between the two methods. However, items selected by the multidimensional method provided a more accurate estimations of the second-level thetas. Seo and Weiss (2015) compared another simplified item selection method with the multidimensional method for bifactor-model CATs under a fixed-length condition. This simplified method considered only the general factor and selected items that maximized the information of the general factor only. Results indicated this simplified method improved proficiency estimations on the general factor, but not on the second-level factors, when compared with the multidimensional method. It is not clear whether similar results would occur with the more complex H-IRT model, such as an unbalanced three-level H-IRT. Moreover, those studies did not compare methods on their computation efficiency. Therefore, this study compares the multidimensional method with two simplified item selection and interim scoring methods—unidimensional and local dimensional methods—on the accuracy and efficiency of final scoring.

Unidimensional item selection and interim scoring

The unidimensional item selection and interim scoring method in this study is very similar to the one used by Weiss and Gibbons (2007). Like the multidimensional method, this method selects items by cycling through loading patterns according to the predetermined order. Only the lowest level dimension theta with nonzero loading (e.g., a1 dimension or c dimension, as shown in Figure 1) is used in item selection and is updated in interim scoring because one item loads only on one lowest dimension and all dimensions are independent. The lowest level dimension is chosen in case different items load on different dimension levels. As only one theta dimension is considered, the IRT model in Equation 1 is treated as a unidimensional model, and the information function in Equation 3 is not a matrix but a value, as shown in Equation 9:

I j ( θ d ) = ∑ t = 1 M { P jt ( θ d ) × [ 1 − P jt ( θ d ) ] − P jt + 1 ( θ d ) × [ 1 − P jt + 1 ( θ d ) ] } 2 P jt ( θ d ) − P jt + 1 ( θ d ) a jd 2 ,

where dimension d is the lowest level dimension with nonzero loading for the current loading pattern. The item that maximizes Equation 9 in the item pool with the current loading pattern is selected.

Only θ d is estimated after administering this item because only θ d is used for selection of items with the current loading pattern. The same Bayesian method is used for scoring, but first and second derivatives of the posterior density function are single values rather than matrices. Thus, Equation 7 is changed to a single-value calculation in Equation 10:

δ d ( j ) = ∂ ∂ θ d lnf ( θ d ( j ) | u ) ∂ 2 ∂ θ d 2 lnf ( θ d ( j ) | u ) ,

where dimension d is the lowest level dimension with nonzero loading for the current loading pattern and δ d ( j ) is the difference between the previous and current theta estimation. For the posterior density function, f ( θ d | u ) , only the responses of the items administered with the current loading pattern are included in the likelihood function, and the standard normal distribution, N(0, 1), is used as the prior distribution. This unidimensional item selection and interim scoring procedure involves only one dimension, thereby reducing the computational burden.

Local multidimensional item selection and interim scoring

The second simplified method has characteristics of both the unidimensional and multidimensional approaches, called local multidimensional item selection and interim scoring in this study. This method has not been studied in other research. As in the multidimensional and unidimensional methods, the first step is cycling through loading patterns according to the predetermined order. Instead of using only the lowest level dimension theta with nonzero loading, though, this method uses thetas of all the dimensions with nonzero loadings, including a first-level dimension, a second-level dimension, and perhaps a third-level dimension. For example, the thetas of the gen and c dimensions shown in Figure 1 are used for selecting items measuring c factor; the thetas of the gen, a , and a 1 dimensions shown in Figure 1 are used for selecting items measuring a 1 factor. As two or three dimensions are considered during item selection, the IRT model in Equation 1 is still a multidimensional model but with fewer dimensions, and the information function in Equation 3 is still a matrix but with fewer dimensions, as shown in Equation 11:

I j ( θ ) = ∑ t = 1 M { P jt ( θ ) × [ 1 − P jt ( θ ) ] − P jt + 1 ( θ ) × [ 1 − P jt + 1 ( θ ) ] } 2 P jt ( θ ) − P jt + 1 ( θ ) [ a j d 1 2 a j d 1 a j d 2 a j d 1 a j d 3 a j d 1 a j d 2 a j d 2 2 a j d 2 a j d 3 a j d 1 a j d 3 a j d 2 a j d 3 a j d 3 2 ] ,

where dimension d₁ is the general-level dimension, dimension d₂ is the second-level dimension, and dimension d₃ is the third-level dimension, if there is a nonzero third-level dimension for the current loading pattern. The item, in the item pool with the current loading pattern, that maximizes the determinant of the Fisher information matrix of Equation 2 is selected. The I _k (θ) as well as I _j (θ) in Equation 2 follows Equation 11, where S– 1 is the number of items administered with the current loading pattern.

Only the thetas of three dimensions in Equation 11 are estimated after administering this item because only the thetas of these three dimensions are used for selection of items with the current loading pattern. The same Bayesian method is used for scoring, but the Jacobian and Hessian matrices have fewer dimensions, as shown in Equations 12 and 13:

∂ ∂ θ lnf ( θ | u ) = [ ∂ ∂ θ d 1 lnf ( θ | u ) ∂ ∂ θ d 2 lnf ( θ | u ) ∂ ∂ θ d 3 lnf ( θ | u ) ] ;

J ( θ ) = [ ∂ 2 ∂ θ d 1 2 lnf ( θ | u ) ∂ 2 ∂ θ d 1 ∂ θ d 2 lnf ( θ | u ) ∂ 2 ∂ θ d 1 ∂ θ d 3 lnf ( θ | u ) ∂ 2 ∂ θ d 2 2 lnf ( θ | u ) ∂ 2 ∂ θ d 2 ∂ θ d 3 lnf ( θ | u ) ∂ 2 ∂ θ d 3 2 lnf ( θ | u ) ] ,

where these three dimensions are the same dimensions in Equation 11. For the posterior density function, f ( θ | u ) , only the responses of the items administered with the current loading pattern are included in the likelihood function, and θ is a vector with thetas of two or three dimensions. The prior distribution is a multivariate normal distribution with two or three dimensions, and all dimensions are independent. This local multidimensional item selection and interim scoring procedure involves only two or three dimensions, reducing the computational burden.

The intent of this study is to compare three item selection and interim scoring methods on the accuracy and efficiency of theta estimations using both a simulated item pool and a real ARMM item pool. The three item selection and interim scoring methods used are multidimensional, local multidimensional, and unidimensional. These three methods differ only on item selection and interim scoring. When the desired test length is reached, the final scoring procedures for these three methods are the same. In final scoring, all dimensions were estimated based on all item responses for all three methods.

Simulation H-IRT Models

This simulation study used two unbalanced H-IRT models. The first model was the simplest: five dimensions comprising one general-level dimension, two second-level dimensions, and two three-level dimensions. Items were grouped into three groups based on their loading patterns. In the first group of items, nonzero discrimination parameters were loaded on the general-level factor and on one of the two second-level factors. In the second and third groups of items, a nonzero discrimination parameter was loaded on the general-level factor, on the other second-level factor, and on one of the third-level factors. The second H-IRT model was more complex, composed of 13 dimensions: one general-level dimension, six second-level dimensions, and six third-level dimensions. Items were organized into nine groups based on their loading patterns. The first three groups of items had nonzero discrimination parameters loaded on the general-level factor and on one of the two second-level factors. The rest of the six groups of items had a nonzero discrimination parameter loaded on the general-level factor, on one of the second-level factors, and on one of the third-level factors.

Item Pools

Two sets of item pools were simulated, one for each H-IRT model, with each pool comprising 270 items. The ratio of item pool size to the longest test length in this study (270:90) was very similar to the ARMM’s ratio. The design of the item pool represents the best scenario in real testing: an equal number of items in the item pool for each loading pattern, meaning each loading pattern had the same number of items to select from. Therefore, in the item pool for the simple model, 90 items were generated for each of three loading patterns. In the complex-model item pool, 30 items were generated for each of nine loading patterns. For all items, the nonzero discrimination parameters for each dimension were drawn randomly and independently from the log-normal distribution (M = 0, SD = 0.4). The item difficulty parameter was drawn randomly and independently from a standard normal distribution (M = 0, SD = 1.0). Thus, all items had only two score categories.

Generating Data

After an item was selected, Monte Carlo simulation generated a response using the H-IRT model from Equation 1. Two groups of 10,000 theta vectors were randomly generated from the multivariate normal distribution, N(µ, Σ), where µ is a vector with all elements equal to 0 and Σ is an identity matrix. These 10,000 theta vectors were the true theta vectors used for simulation. One group of theta vectors was for the simple H-IRT model with five dimensions; the other group of theta vectors was for the complex H-IRT model with 13 dimensions. Fixed test length was used as the termination criterion because it makes the desired content balance easier to achieve (van der Linden, 1999; van der Linden & Reese, 1998). No replication was conducted in this study because the overall performance of methods across theta dimensions was considered (Seo & Weiss, 2015).

Simulation Conditions

For each item selection and interim scoring method, simulation conditions varied by test length and H-IRT model. Ten conditions were studied using five test lengths (i.e., 18, 36, 54, 72, and 90 items) and two H-IRT models. Using test length as one of the simulation conditions provides evidence about whether the number of items affects the accuracy of the three item selection and interim scoring methods. All these test lengths also ensured that equal numbers of items could be administered to each loading pattern for two H-IRT models. Therefore, for the simple H-IRT model, one third of the test items were administered for each of the three loading patterns. For the complex H-IRT model, one ninth of the test items were administered for each of the nine loading patterns. These two H-IRT models included a model with a small number of theta dimensions and a model with a large number of theta dimensions. The difference in number of theta dimensions may affect the efficiency of the three item selection and interim scoring methods. Table 1 summarizes the simulation design.

OPEN IN VIEWER

Table 1.

Simulation Condition Information.

Condition	H-IRT model	Items on test (n)	Unique loading patterns (n)	Items administered per loading pattern (n)
1	Simple	18	3	6
2	Simple	36	3	12
3	Simple	54	3	18
4	Simple	72	3	24
5	Simple	90	3	30
6	Complex	18	9	2
7	Complex	36	9	4
8	Complex	54	9	6
9	Complex	72	9	8
10	Complex	90	9	10

Note. H-IRT = hierarchical item response theory.

For each of the 10 conditions within each of the three item selection and interim scoring methods, the initial value of each element of the theta vector was set to 0. Next, the appropriate item selection method was used to select one item for one loading pattern. After a new item was administered for each loading pattern, the corresponding interim scoring method was applied. The same procedure was repeated until the test-length criterion was reached. After the simulated administration was completed, the final theta of all dimensions was estimated, based on all item responses and using the Bayesian method introduced above.

Evaluation Criteria

Two criteria were used to compare the accuracy of final scoring of different item selection and interim scoring methods under the 10 different conditions: the Pearson correlation between the true and estimated thetas of each dimension and the root mean square error (RMSE) of the estimation of each dimension. The Pearson correlation, r( θ d , θ ^ d ), of every dimension was calculated across 10,000 theta nodes of each method and condition. The equation for calculating the RMSE _d of one dimension, d, is shown in Equation 14:

RMS E d = ∑ k = 1 K ( θ kd − θ ^ kd ) 2 K ,

where K is the number of true theta vectors (10,000 in this study). The RMSEs were calculated for each method and condition, and the criteria of different dimensions within each loading level were averaged arithmetically for that loading level of theta dimensions. The correlation average was achieved through Fisher’s z transformation. Thus for every condition, two criteria were reported for four loading levels of theta dimensions: the general level (e.g., gen theta), the second level with the third level loaded (e.g., a theta), the second level without the third level loaded (e.g., c theta), and the third level (e.g., a1 theta).

We used computation time to compare the efficiency of different item selection and interim scoring methods under the 10 different conditions. The simulation study program was written in Microsoft Visual Studio 2010, Version 10.0.40219.1, using Fortran and executed on a computer with an Intel Xeon 2.4 GHz Processor and 12 gigabytes of RAM. The computation times of one student finishing one test and 10,000 students finishing tests were reported for each method and condition.

H-IRT Model and Item Pool

The real item pool study used the ARMM’s three-level unbalanced H-IRT model with 18 dimensions. The ARMM’s adaptive test item pool is used in this study. This item pool included 12 items for each loading pattern, except for items measuring b2 factor, as shown in Figure 1. Of the 174 total items in the pool, only six items measured b2 factor, and there were six score categories for all items. There were 174 first-level item discrimination parameters (M = 1.61, SD = 0.88), 174 second-level item discrimination parameters (M = 1.61, SD = 0.89), 102 third-level item discrimination parameters (M = 1.28, SD = 0.71), and 870 item difficulty parameters (M = 0.03, SD = 2.71).

Generating Data and Result Evaluation

After an item was selected, Monte Carlo simulation generated a response using the H-IRT model from Equation 1. There were 10,000 theta vectors with 18 dimensions randomly generated from the multivariate normal distribution, N(µ, Σ), where µ is a vector with all elements equal to 0 and Σ is an identity matrix. These 10,000 theta vectors are the true theta vectors. The ARMM test length was fixed at 45 items; thus, three items were administered for every loading pattern. No replication was conducted in this study.

For each of the three item selection and interim scoring methods, the initial values of each element of the theta vector were set to 0. Next, the appropriate item selection method was used to select one item for one loading pattern. After a new item was administered for each loading pattern, the corresponding interim scoring method was applied. The same procedure was repeated until the test-length criterion was reached. After the simulated administration was completed, the final theta of all dimensions was estimated, based on all item responses and using the Bayesian method introduced above.

The results of the real item pool study were evaluated using the same evaluation criteria from the simulation study. The accuracy and efficiency of three different item selection and interim scoring methods were compared when the ARMM item pool was used.

Results of Simulation Study

Figure 2 presents the correlations between the true and estimated thetas of four loading levels for each method and condition. For each model and item selection and interim scoring method, the correlations between true and estimated thetas of all loading levels increased as test length increased. The increase rate slowed, however, with longer tests. Comparing methods, the local multidimensional method had high correlations on the general and second loading levels. The unidimensional method had high correlations on the second level without the third level loaded and on the third loading level. The multidimensional method had relatively high correlations across four loading levels. The correlation differences among the methods were very large for the shortest test but became smaller as test length increased. For the 90-item test, the methods had similar, high correlations for all loading levels. When comparing the correlations of the same test length but different H-IRT models, the correlation of the simple model was higher.

OPEN IN VIEWER

Figure 2.

Correlations by H-IRT model, theta-loading level, and test length.

Figure 3 presents the RMSEs of four loading levels for each method and condition. For each model and item selection and interim scoring method, RMSEs of all loading levels decreased as test length increased. However, the decrease rate slowed as test length increased. Comparing methods, the local multidimensional methods had low RMSEs on the general and second loading levels. The unidimensional method had low RMSEs on the second level without the third level loaded and on the third loading level. The multidimensional method had relatively low RMSEs across four loading levels. There were differences among the methods for the shortest test, but the differences became smaller as test length increased. For the longest test, the methods had similar, low RMSEs on all loading levels. When comparing RMSEs of the same test length but different H-IRT models, the RMSEs of the simple model were lower.

OPEN IN VIEWER

Figure 3.

RMSEs by H-IRT model, theta-loading level, and test length.

Table 2 presents the computation times in seconds of a single test and 10,000 tests, respectively, for all three methods under 10 simulation conditions. For a single test, in general, the computation time for each model and item selection and interim scoring method increased as test length increased. The three methods required similar computation time for the simple model, but the multidimensional method required longer computation time for the complex model. For 10,000 tests, the computation time for each model and item selection and interim scoring method increased as test length increased. Comparing methods, the multidimensional method required the longest computation time and the unidimensional method required the shortest computation time. For the simple model, the multidimensional method required more than 6 times the computation time of the unidimensional method. For the complex model, the computation time of the multidimensional method was 20 times longer than that of the unidimensional method. When comparing computation times for the same test length but different H-IRT models, the computation time of the simple model was shorter.

OPEN IN VIEWER

Table 2.

Computation Time, in Seconds, of Three Methods for 10 Conditions.

		Time, in seconds
		1 test			10,000 tests
H-IRT model	Items on test (n)	M	LM	U	M	LM	U
Simple	18	0.00	0.00	0.00	42.79	28.72	18.03
	36	0.02	0.00	0.00	92.99	43.57	23.42
	54	0.02	0.00	0.02	142.96	60.54	28.05
	72	0.03	0.00	0.00	199.60	76.64	35.40
	90	0.03	0.02	0.02	259.06	95.27	42.00
Complex	18	0.05	0.02	0.02	487.27	75.71	72.57
	36	0.11	0.00	0.00	1,000.11	87.16	79.87
	54	0.16	0.00	0.02	1,543.21	101.90	89.68
	72	0.20	0.00	0.02	2,111.66	114.02	94.66
	90	0.28	0.02	0.02	2,707.24	125.80	103.73

Note. H-IRT = hierarchical item response theory; M = multidimensional item selection and interim scoring method; LM = local multidimensional item selection and interim scoring method; U = unidimensional item selection and interim scoring method.

Results of Real Item Pool Study

Table 3 presents the correlations and RMSEs of four loading levels, as well as the computation times in seconds of a single test and 10,000 tests for the real item pool study. For the accuracy comparison criteria, the results of the real item pool study showed a slightly different pattern than did the results of the simulation study. The unidimensional method provided more accurate estimations on the general and second loading levels. The multidimensional method provided more accurate estimations on the second level without the third level loaded and on the third loading level. The local multidimensional method provided relatively accurate estimations across four loading levels. In general, the differences in accuracy comparison criteria among methods were very small. However, these accuracy results were not as accurate as the simulation study values, especially for the third loading level. We believe this is because the lower level item discrimination parameters of the real item pool were small. For the efficiency comparison criterion, the results of the real item pool study showed the same pattern as the simulation study results in that the computation time for multidimensional method was much longer than for the unidimensional or local multidimensional method. For 10,000 tests, the multidimensional method required about 15 times the computation time of the unidimensional or local multidimensional method.

OPEN IN VIEWER

Table 3.

Theta Estimation Accuracy and Computation Time, in Seconds, of Three Methods Used in Real Item Pool Study.

	Correlation				RMSE				Time, in seconds
Method	G	SwT	S	T	G	SwT	S	T	1 test	10,000 tests
M	.84	.51	.65	.34	0.86	0.90	0.91	0.97	0.25	2,656.85
LM	.85	.50	.63	.32	0.85	0.90	0.91	0.98	0.02	179.73
U	.86	.56	.63	.25	0.85	0.88	0.91	0.99	0.02	164.49

Note. RMSE = root mean square error; G = general level; SwT = second level with third level loaded; S = second level without third level loaded; T = third level; M = multidimensional item selection and interim scoring method; LM = local multidimensional item selection and interim scoring method; U = unidimensional item selection and interim scoring method.

Suggestions for Future Study

Several questions require further investigation. First, additional multidimensional information calculation methods can be considered, such as the KL method; this study used only the Fisher method. Chang (2015), however, stated that Fisher information is based on local information and has limitations. Second, the distribution used to generate theta vectors was also used as the prior distribution for Bayesian theta estimation. The same distribution of the prior and true thetas can lead to spuriously accurate theta estimations. The true theta distribution is unknown, and the prior distribution used may be different from the true theta distribution in real testing situations. Thus, future studies could investigate the effect on theta estimation accuracy of different prior distribution. Third, results of the real item pool study have low accuracy on the third-level theta estimation because item discrimination parameters of this level are low. However, item discrimination parameters of different levels of the simulated item pool are very similar and lead to more accurate theta estimations on the third-level, which may be one reason the accuracy results of the two studies have different patterns. Thus, future studies could differentiate the item discrimination parameter distributions of different loading levels for simulated item pools. Fourth, future studies could compare different item exposure rate control methods for CATs based on H-IRT models. Fifth, the interim theta in this study was updated after each item was answered. For the item selection and interim scoring methods in which the general theta is updated after each item is administered, the general theta for item selection of the nth loading pattern will be more accurate than for item selection of the first loading pattern. More research is needed to determine whether these differences in general theta accuracy lead to the selection of different items. Finally, the item selection order of different dimensions in this study was determined by an arbitrary dimension sequence. Future research could investigate and compare different rules of determining the item selection dimension order, such as random selection of dimensions.