A Stochastic Method for Balancing Item Exposure Rates in Computerized Classification Tests

Abstract

Computerized classification tests (CCTs) classify examinees into categories such as pass/fail, master/nonmaster, and so on. This article proposes the use of stochastic methods from sequential analysis to address item overexposure, a practical concern in operational CCTs. Item overexposure is traditionally dealt with in CCTs by the Sympson-Hetter (SH) method, but this method is unable to restrict the exposure of the most informative items to the desired level. The authors’ new method of stochastic item exposure balance (SIEB) works in conjunction with the SH method and is shown to greatly reduce the number of overexposed items in a pool and improve overall exposure balance while maintaining classification accuracy comparable with using the SH method alone. The method is demonstrated using a simulation study.

Keywords

computerized testing classification test security

Computerized classification tests (CCTs) are measurement instruments that classify examinees into categories such as pass/fail, master/nonmaster, and so on, often for the purpose of professional licensure or certification. In many cases, the sequential probability ratio test (SPRT) of Wald (1947) is used in CCTs to terminate the test when a classification decision can be made with a high degree of certainty. Clearly, it is undesirable for testing to go on indefinitely and the entire item pool to be exposed to a single examinee, so a maximum number of items is specified for almost all operational CCTs. In addition, a minimum number of items is often specified as well, and the test may be terminated according to the SPRT when the number of items is between the minimum and the maximum. (Some authors refer to SPRTs constrained by a maximum number of items as “truncated SPRTs,” to differentiate them from pure SPRTs with no such constraint. However, because many operational SPRTs do in fact impose a maximum number of items, the authors simply use the term SPRT.)

Finkelman (2008) proposed the method of stochastically curtailed SPRTs (SC-SPRTs), an innovation designed to make CCTs more efficient, that is, decrease the number of items administered to examinees while still classifying them accurately. He motivates the topic by demonstrating that under some conditions, the SPRT will continue to administer items although the current interim classification decision would not change before the maximum number of items is reached. Thus, the remaining items that are administered are wasted if there is no chance that the responses to them will change the classification decision. The method of the SC-SPRT computes the probability that the current interim classification decision will not change before the maximum number of items is reached. If this probability is very high, the SC-SPRT terminates the test, even if the traditional SPRT would have kept administering items. It is shown that the SC-SPRT is very effective in reducing the average test length (ATL) while maintaining classification accuracy equal or nearly equal to that of the traditional SPRT.

This article proposes the use of these stochastic methods to address another practical concern in operational CCTs, that of item overexposure. Item overexposure is traditionally dealt with in CCTs by the Sympson-Hetter (SH) method (Sympson & Hetter, 1985), but this method is unable to ensure that the exposure rate of all items will be kept at or below the desired maximum exposure rate (Barrada, Abad, & Veldkamp, 2009). In other words, when using the SH method, the most popular (i.e., informative) items will usually be administered to a larger proportion of examinees than was intended by the test developers.

To remedy this situation, the authors propose a method that uses the mathematical concepts of the SC-SPRT to work in conjunction with the SH algorithm to help balance item exposure rates in CCTs that specify a maximum and minimum number of items. As Thompson (2011) stated, a minimum serves a public relations function in that it prevents examinees from failing after responding to only a few items, which would likely result in complaints to the testing organization. From a psychometric standpoint, however, it is often possible to classify examinees accurately before they reach the minimum, such as in the case of examinees with either very high or very low ability levels. For example, for a CCT with a minimum test length of 50 items, a high-ability examinee may be able to be classified correctly as passing by the 40th item, but 10 more items must still be administered to meet the minimum requirement. In this situation, it would be unnecessary to draw these last 10 items from a portion of the item pool consisting of often exposed items; it is reasonable that these 10 positions may be filled with items that have been underexposed.

In the proposed method, before the minimum number of items is reached, an interim classification decision is made and the probability of this decision being retained for the remainder of the test is computed. If this probability is very high, the item selection algorithm will be routed to a subset of the item pool consisting of the most underexposed items, which are often the least informative. Responses to these items continue to contribute to the classification of the examinee; however, the classification is not likely to change, given the high probability of retention. In this manner, classification accuracy will remain high, and rarely administered items will have a much larger chance of being used, thereby preventing the use of more popular items. For examinees with interim classification decisions that are more uncertain, that is, interim decisions with a higher chance of changing, the test will proceed normally and terminate according to the traditional SPRT.

This article is structured as follows: The section titled ‘Method’ briefly summarizes the concepts of Finkelman’s (2008) SC-SPRT and presents the proposed method, which applies these ideas to the problem of improving item exposure balance in CCTs. ‘Simulations’ describes two simulation studies designed to examine the performance of the method in contrasting CCT settings, and ‘Results’ presents the results of the simulation. ‘Discussion’ concludes the article with discussion and avenues for further research.

Method

Traditional CCTs and SC-SPRTs

This article deals with CCTs that perform dichotomous classifications based on item response theory (IRT), specifically, the three-parameter logistic model (Birnbaum, 1968). For a detailed treatment of CCTs and the SPRT, see Wald (1947) and Spray and Reckase (1994, 1996). Practical considerations when designing CCTs include the aforementioned issue of item exposure control and item selection methods. A traditional method of item selection, and the one used in this article, is choosing the item that maximizes the Fisher information (FI) at the cut point θ₀, which was shown by Spray and Reckase (1994) to compare favorably with other methods. This method of item selection is nonadaptive, as it does not depend on an examinee’s interim ability estimate. For cases in which a CCT uses an item selection method that is adaptive, Finkelman (2008) presented a slight modification of his method of stochastic curtailment. Thus, the general method of stochastic curtailment, as well as the methodology proposed below, would be applicable to CCTs using either method of item selection.

As mentioned above, Finkelman (2008) proposed shortening CCTs via the SC-SPRT. To review the method, the authors introduce the following notation. Items administered in a CCT are indexed by the letter j, and the minimum and maximum number of items allowed are denoted by J_min and J_max, respectively. The authors differentiate the interim classification decision after the administration of j′ items, denoted as $D_{j'}^{*}$ , from the final classification decision, D_T. $P (D_{j'}^{*} = D_{T})$ is computed using statistical concepts from sequential analysis, which is the probability that the current interim decision will not change before the test ends, that is, the final decision will be the same as the current interim decision. If $P (D_{j'}^{*} = D_{T})$ is greater than some probability threshold γ, the SC-SPRT terminates the test, even if the traditional SPRT would have continued administering items. For the sake of brevity, the reader seeking the mathematical details of computing $P (D_{j'}^{*} = D_{T})$ is referred to Finkelman (2008).

Stochastically Balancing Item Exposure Rates

As discussed previously, Finkelman (2008) proposed the use of the probability $P (D_{j'}^{*} = D_{T})$ to shorten test lengths for some examinees, thereby decreasing the ATL for a given sample of examinees. This article proposes the use of $P (D_{j'}^{*} = D_{T})$ for a different purpose, that of balancing item exposure rates in CCTs with a maximum and minimum number of items. Under this new method, items in the CCT are selected either nonadaptively by maximizing the FI at θ₀ or adaptively by maximizing the FI at the interim theta estimate, and the SH method is applied. Then, $P (D_{j'}^{*} = D_{T})$ is computed for each $j' < J_{min}$ . If at some point, $P (D_{j'}^{*} = D_{T}) \geq γ$ , then the remaining $(J_{min} - j')$ items will be selected randomly from a subset of the item pool containing items with low exposure rates. The total set of items in the pool and the set of lesser exposed items are denoted as M and M_low, respectively. Then, the examinee is classified after the administration of J_min items, and the test is terminated. Hereafter, this new method of stochastic item exposure balance is referred to as SIEB. From an intuitive viewpoint, SIEB is reasonable because examinees producing response patterns that yield large values of $P (D_{j'}^{*} = D_{T})$ relatively early in the test most likely possess very high or very small values of θ and thus can be classified with a high degree of certainty before J_min items are administered. As soon as an interim classification is made that has a high probability of being retained for the remainder of the test, the item selection may be routed to M_low. For examinees with interim classifications that are less certain, that is, with $P (D_{j'}^{*} = D_{T}) < γ$ after J_min items have been administered, the test will proceed normally and terminate according to the traditional SPRT, and the item selection procedure will not be altered.

The exact set of items selected to be contained in M_low may be specified at the discretion of the psychometrician. For CCTs using the SH method for item exposure control, a reasonable way to construct the subset would be to include all items with a SH exposure parameter of 1, because it is these items that are exposed less often. Another simple way to construct M_low would be to include the x least informative (and therefore, most seldom exposed) items in the pool, for x = 100, 150, 200 . . . depending on the size of the pool and the number of items that tend to be over-/underexposed.

Simulations

Two simulation studies were performed to examine the ability of the new SIEB method to improve item underexposure while maintaining classification accuracy comparable with that of the traditional SPRT. The two studies use CCTs with differing specifications so that the proposed methodology may be demonstrated in two contrasting situations. As is described in the following, eight different test conditions were simulated in each study, and each condition was run for 50 replications. The item pools generated for each study were kept constant throughout all conditions and replications. Also, for each replication of each condition n = 1,000 simulated examinees were generated from the Normal(0,1) distribution.

Study 1

For the first simulation study, an item pool consisting of 750 items was generated with IRT a, b, and c parameters drawn from the Normal(0.7,0.2), Normal(−0.75,2.0), and Normal(0.25,0.03), respectively. These values were chosen to be similar to those used in previous CCT studies (Thompson, 2009). SH exposure control parameters were set for the items in the pool via initial simulations run according to the method outlined by Sympson and Hetter (1985), and the target maximum exposure rate was set at 0.2, a commonly chosen rate (Leung, Chang, & Hau, 2002). The exposure control parameters obtained at the 10th iteration of the SH algorithm were used. For four of the eight conditions, content domain constraints were imposed; in these cases, items in the pool were randomly assigned to one of three equally balanced content domains.

For all conditions, the following CCT parameters were set: $J_{min} = 50$ , $J_{max} = 75$ , $α = β = . 05$ (resulting in log A = −2.94 and log B = 2.94), $δ = . 15$ , and $θ_{0} = . 5$ . (For a detailed explanation of these parameters, the reader is referred to Spray & Reckase, 1994, 1996.) For all conditions, items were selected via maximizing the FI at $θ_{0}$ . Then, four different item exposure controls were tested: SH with SIEB( $γ = . 80$ ), SH with SIEB( $γ = . 90$ ), SH with SIEB( $γ = . 99$ ), and SH only, which serves as a baseline comparison for the SIEB conditions. $M_{low}$ contained the 300 least informative items of the pool for the SIEB conditions.

In summary, the conditions that were varied for Study 1 include two levels of content domain constraints (present and not present) and four-item different exposure control methods, resulting in 2×4 = 8 conditions. For the conditions incorporating content domain constraints, content balance was achieved using the spiraling method proposed by Kingsbury and Zara (1989), with the three contents being assigned equal importance.

Study 2

The design of the second study was similar to that of the first, but several CCT settings were varied for contrast. For this study, an item bank of 500 items was generated with IRT parameters drawn from the same distributions as in Study 1. The following CCT settings were implemented: $J_{min} = 30$ , $J_{max} = 50$ , $α = β = . 05$ , $δ = . 25$ , and $θ_{0} = . 75$ . For the conditions imposing content domain constraints on the CCT, items were randomly assigned to two equally balanced content domains. As in Study1, items were chosen by maximizing FI at $θ_{0}$ , and content control was achieved using the spiraling method. Also, the same four exposure control methods were tested, but in the second study $M_{low}$ consisted of the 150 least informative items of the pool.

Results

The results of the simulation study were evaluated according to five different criteria: (a) the percentage of correct classifications (PCC); (b) ATL; (c) a chi-square statistic used by Wen, Chang, and Hau (2000) to summarize the item exposure rate balance (or imbalance) of the items in a pool for variable-length computer adaptive test or CCT (lower values signify better exposure balance); (d) the proportion of low-exposed items (LEXP), that is, items seen by less than 2% of examinees; and (e) the proportion of highly exposed items (HEXP), that is, those items exposed to greater than 20% of the examinees, the maximum rate specified when running the SH simulations.

In general, the results of the simulation studies show that SIEB is capable of significantly improving item exposure balance, including decreasing the proportion of overexposed items in the pools without sacrificing classification accuracy. Table 1 displays the results of Study 1, averaged over the 50 replications for each condition (standard deviations are in parentheses). For conditions both with and without content domain constraints, the SH with SIEB( $γ = . 99$ ) exposure control yields a vast improvement in item exposure balance and proportion of overexposed item compared with SH only, as indicated by the χ² and HEXP statistics, while maintaining virtually identical classification accuracy. For SIEB( $γ = . 90$ ) and SIEB( $γ = . 80$ ), we see continual improvement in χ² and HEXP. However, for SIEB( $γ = . 80$ ), a slight decrease in classification accuracy is apparent, whereas the improvements in χ² and HEXP are less marked.

Table 1.

Results for Study 1

Content	Exposure control	PCC	ATL	χ²	LEXP	HEXP
No	SH only	89.7 (0.93)	69.5 (0.28)	82.2 (0.42)	0.52 (<0.01)	0.25 (0.01)
	SH with SIEB ( $γ = . 99$ )	89.5 (1.00)	59.7 (0.36)	36.0 (0.95)	0.12 (<0.01)	0.04 (<0.01)
	SH with SIEB ( $γ = . 90$ )	89.6 (1.00)	54.8 (0.32)	24.7 (0.48)	0.13 (<0.01)	0.02 (<0.01)
	SH with SIEB ( $γ = . 80$ )	88.7 (1.05)	52.5 (0.19)	25.8 (0.65)	0.13 (<0.01)	0.01 (<0.01)
	SH only	89.6 (1.01)	69.5 (0.31)	84.5 (0.45)	0.51 (<0.01)	0.21 (<0.01)
Yes	SH with SIEB ( $γ = . 99$ )	89.6 (1.07)	59.7 (0.42)	36.4 (0.88)	0.11 (<0.01)	0.04 (<0.01)
	SH with SIEB ( $γ = . 90$ )	89.5 (0.93)	54.9 (0.33)	24.3 (0.48)	0.12 (<0.01)	0.02 (<0.01)
	SH with SIEB ( $γ = . 80$ )	89.0 (1.09)	52.5 (0.21)	24.7 (0.63)	0.12 (<0.01)	<0.01 (<0.01)

Note: PCC = percentage of correct classifications; ATL = average test length; LEXP = low-exposed items; HEXP = highly exposed items; SH = Sympson-Hetter; SIEB = stochastic item exposure balance.

Also, in Table 1, there is very little difference between the conditions with content constraints and those without. With the studied pool, the content constraints do not affect the classification accuracy. A difference may be seen, perhaps, if the bank was made smaller and/or a larger set of constraints was used.

The results for Study 2 are displayed in Table 2. The patterns described for the results of Study 1 all hold for Study 2: SIEB with γ values close to 1 yield significant improvement in χ² and HEXP while maintaining similar classification accuracy as SH only, whereas SIEB( $γ = . 80$ ) starts to show a dip in accuracy with marginal improvements in χ² and HEXP compared with SIEB( $γ = . 90$ ) and SIEB( $γ = . 90$ ).

Table 2.

Results for Study 2

Content	Exposure control	PCC	ATL	χ²	LEXP	HEXP
No	SH only	89.0 (0.82)	43.4 (0.28)	57.7 (0.41)	0.55 (<0.01)	0.22 (0.02)
	SH with SIEB ( $γ = . 99$ )	89.0 (0.95)	39.7 (0.31)	31.9 (0.85)	0.25 (<0.01)	0.04 (0.01)
	SH with SIEB ( $γ = . 90$ )	88.7 (1.11)	35.7 (0.32)	21.8 (0.40)	0.25 (<0.01)	0.02 (<0.01)
	SH with SIEB ( $γ = . 80$ )	88.1 (0.99)	33.5 (0.26)	20.9 (0.44)	0.25 (<0.01)	0.02 (<0.01)
Yes	SH only	88.8 (0.78)	43.5 (0.24)	56.5 (0.35)	0.53 (0.01)	0.22 (0.02)
	SH with SIEB ( $γ = . 99$ )	88.7 (0.93)	39.9 (0.31)	31.4 (0.80)	0.23 (0.01)	0.04 (0.01)
	SH with SIEB ( $γ = . 90$ )	88.8 (0.88)	35.9 (0.29)	21.1 (0.39)	0.24 (0.01)	0.02 (<0.01)
	SH with SIEB ( $γ = . 80$ )	88.5 (1.00)	33.6 (0.24)	19.9 (0.40)	0.24 (0.01)	0.01 (<0.01)

Note: PCC = percentage of correct classifications; ATL = average test length; LEXP = low-exposed items; HEXP = highly exposed items; SH = Sympson-Hetter; SIEB = stochastic item exposure balance.

In addition to the SIEB yielding lower χ² and HEXP values across testing conditions in both studies, the ATLs decrease as lower values of γ are used in the SIEB. This is due to the fact that under the SIEB, the tests for examinees with $P (D_{j'}^{*} = D_{T}) \geq γ$ for $j' < J_{min}$ are terminated at $J_{min}$ , while many of these examinees would have continued testing under the traditional SPRT. While the aim of SIEB is to improve item exposure balance, especially item overexposure, the lower ATL is an additional benefit that comes at little cost, because classification accuracy is not adversely affected in the conditions studied.

A question that arises naturally is how many examinees are routed to $M_{low}$ during the course of test administration. Table 3 displays the average proportion across replications of examinees routed to $M_{low}$ for all simulation conditions. As expected, this proportion increases as the threshold γ decreases, and the proportions for a given γ are higher in Study 1 than in Study 2. In practice, the proportion of examinees routed to $M_{low}$ would depend on a number of factors besides γ, including the size of the item pool, the psychometric quality of the items in the pool (specifically, the shape of the pool information function at $θ_{0}$ ), and the conservativeness of the SPRT as dictated by the quantities α, β, and δ. Also, a practicing psychometrician may be concerned that while SIEB decreases the overall HEXP, this quantity may be inflated for examinees routed to $M_{low}$ . The right most column in Table 3 displays $HEXP_M_{low}$ , the HEXP calculated based only on examinees routed to $M_{low}$ . It can be seen that these values are essentially identical to the overall values reported in Tables 1 and 2, suggesting that there is no inflation of item exposure among examinees routed to $M_{low}$ .

Table 3.

Proportion of Examinees Routed to $M_{low}$ and HEXP Based on These Examinees

	Content	Exposure control	Routed to $M_{low}$	HEXP_ $M_{low}$
Study 1	No	SH with SIEB ( $γ = . 99$ )	0.61 (0.02)	0.04 (<0.01)
		SH with SIEB ( $γ = . 90$ )	0.81 (0.01)	0.02 (<0.01)
		SH with SIEB ( $γ = . 80$ )	0.90 (0.01)	0.01 (<0.01)
	Yes	SH with SIEB ( $γ = . 99$ )	0.61 (0.02)	0.03 (<0.01)
		SH with SIEB ( $γ = . 90$ )	0.80 (0.01)	0.02 (<0.01)
		SH with SIEB ( $γ = . 80$ )	0.90 (0.01)	0.01 (<0.01)
Study 2	No	SH with SIEB ( $γ = . 99$ )	0.51 (0.02)	0.04 (<0.01)
		SH with SIEB ( $γ = . 90$ )	0.71 (0.01)	0.02 (<0.01)
		SH with SIEB ( $γ = . 80$ )	0.82 (0.01)	0.02 (<0.01)
	Yes	SH with SIEB ( $γ = . 99$ )	0.50 (0.02)	0.04 (<0.01)
		SH with SIEB ( $γ = . 90$ )	0.71 (0.01)	0.02 (<0.01)
		SH with SIEB ( $γ = . 80$ )	0.82 (0.01)	0.01 (<0.01)

Note: HEXP = highly exposed items; SH = Sympson-Hetter; SIEB = stochastic item exposure balance.

Discussion

The SH method is a tried and true means of limiting item overexposure in CCTs. One of the standard criticisms of this method is that many items may remain overexposed, exceeding the desired maximum item exposure rate. While this may be a security concern for test developers and stakeholders, it would be undesirable for any method aiming to improve item overexposure to sacrifice a significant degree of classification accuracy, which is usually a much higher priority. This article proposes SIEB, a stochastic method of improving item exposure balance that works in conjunction with the SH method for CCTs, which specify a minimum number of items that must be administered to all candidates. This new combined method exposure control has been shown to aid in balancing item exposure rates while maintaining classification accuracy virtually equal to that of the SH method used alone. Specifically, SIEB greatly reduces the number of items that display an exposure rate greater than the maximum desired exposure rate specified when using the SH method. The strength of the proposed method is due to the calculation of the probability $P (D_{j'}^{*} = D_{T}) \geq γ$ , the calculation of which gives assurance that the current interim classification decision will be retained until the end of the test.

The idea of improving item exposure balance by administering low-discriminating and/or low-information items at a point in an exam when doing so is psychometrically “inexpensive,” that is, unlikely to negatively affect examinee ability measurement, is not a new one. For example, the technique of a-stratification (Chang & Ying, 1999) administers low-discriminating (and therefore underexposed) items at the beginning of a computerized adaptive test when an examinee’s theta value has not yet been estimated precisely. In contrast, SIEB administers rarely exposed items in a CCT after the examinee has been classified with certainty, but the two methods are similar in spirit and aim. a-Stratification has not yet been studied in a CCT setting; the randomization approach for exposure control used by Lin (2011) appears to be analogous. Further research may attempt to combine SIEB with such methods.

There are myriad ways to alter the design of CCT, including varying the minimum and maximum test length, false negative and false positive rates α and β, length of indifference region, item pool size, item parameter distribution, and content domain constraints. No single study can coherently examine all these factors; this article presented two contrasting simulation studies in which the item pool size and content domain constraints were varied, as well as the SIEB parameter γ. Researchers wishing to use the proposed methodology should experiment with different γ values for use with their particular CCTs. While the authors do not wish to overgeneralize, their results suggest that even very conservative values of γ (close to 1) may yield significant improvements in item exposure balance while maintaining classification accuracy.

An issue that would arise if this method were to be used operationally is how to select $M_{low}$ . Some suggestions were provided in a previous section, but care must be taken in the real-world setting. Clearly, if this subset is relatively small and γ is too low, the items in $M_{low}$ may actually end up being overexposed due to a large number of examinees being routed to a small set of items. Future research should focus on how to optimally choose this subset under a variety of testing conditions.

Footnotes

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Barrada

J. A.

Abad

F. J.

Veldkamp

B. P.

(2009). Comparison of methods for controlling maximum exposure rates in computerized adaptive testing. Psicothema, 21, 313-320.

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 397-472). Reading, MA: Addison-Wesley.

Chang

H. H.

Ying

(1999). a-Stratified multistage computer adaptive testing. Applied Psychological Measurement, 23, 211-222.

Finkelman

(2008). On using stochastic curtailment to shorten the SPRT in sequential mastery testing. Journal of Educational and Behavioral Statistics, 33, 442-463.

Kingsbury

G. G.

Zara

A. R.

(1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2, 359-375.

Leung

C. K.

Chang

H. H.

Hau

K. T.

(2002). Item selection in computerized adaptive testing: Improving the a-stratified design with the Sympson-Hetter algorithm. Applied Psychological Measurement, 26, 376-392.

Lin

C. J.

(2011). Item selection criteria with practical constraints for computerized adaptive testing. Educational and Psychological Measurement, 71, 20-36.

Spray

J. A.

Reckase

M. D.

(1994, April). The selection of test items for decision making with a computerized adaptive test. Paper presented at the Annual Meeting of the National Council for Measurement in Education, New Orleans, LA.

Spray

J. A.

Reckase

M. D.

(1996). Comparison of SPRT and sequential Bayes procedures for classifying examinees into two categories using a computerized adaptive test. Journal of Educational and Behavioral Statistics, 21, 405-414.

10.

Sympson

J. B.

Hetter

R. D.

(1985). Controlling item exposure rates in computerized adaptive testing. In Proceedings of the 27th annual meeting of the Military Testing Association (pp. 937-977). San Diego, CA: Navy Personnel Research and Development Center.

11.

Thompson

N. A.

(2009). Item selection in computerized classification testing. Educational and Psychological Measurement, 69, 778-793.

12.

Thompson

N. A.

(2011). Termination criteria for computerized classification testing. Practical Assessment, Research, & Evaluation, 16. Available at http://pareonline.net/pdf/v16n4.pdf

13.

Wald

(1947). Sequential analysis. New York, NY: Wiley.

14.

Wen

J. B.

Chang

H. H.

Hau

K. T.

(2000, April). Adaptation of a-stratified method in variable length computerized adaptive testing. Paper presented at the American Educational Research Association Annual Meeting, Seattle, WA.