An Empirical Evaluation of the Slip Correction in the Four Parameter Logistic Models With Computerized Adaptive Testing

Abstract

In a selected response test, aberrant responses such as careless errors and lucky guesses might cause error in ability estimation because these responses do not actually reflect the knowledge that examinees possess. In a computerized adaptive test (CAT), these aberrant responses could further cause serious estimation error due to dynamic item administration. To enhance the robust performance of CAT against aberrant responses, Barton and Lord proposed the four-parameter logistic (4PL) item response theory (IRT) model. However, most studies relevant to the 4PL IRT model were conducted based on simulation experiments. This study attempts to investigate the performance of the 4PL IRT model as a slip-correction mechanism with an empirical experiment. The results showed that the 4PL IRT model could not only reduce the problematic underestimation of the examinees’ ability introduced by careless mistakes in practical situations but also improve measurement efficiency.

Keywords

item response theory (IRT)computerized adaptive testing (CAT)upper-asymptote parameter four-parameter logistic (4PL) IRT model

Introduction

In a selected response test, examinees’ responses could be classified into the following three types: (a) responses reflecting examinees’ true ability, (b) correct responses made through lucky guesses, and (c) false responses derived from anxiety, carelessness, or distraction. The latter two types of aberrant responses might cause error in ability estimation because they would not reflect the actual knowledge that examinees possessed. According to the classical test theory (CTT), the aberrant responses might have little effect on ability estimation in a traditional paper-and-pencil (P&P) or computer-based test (CBT) for each item if items are equally weighted. Yet, these aberrant responses may actually lead to estimation error in computerized adaptive tests (CAT).

CAT was designed to tailor item administration in accordance with each examinee’s ability, whereas item response theory (IRT) was used to estimate an examinee’s ability. Each subsequent item would then be selected to be administered based on the test taker’s estimated ability. Hence, an examinee’s aberrant response could cause an estimation error, and accordingly, an inappropriate item selection might occur. To lessen the excessive punishment for careless errors (or “slip”), Barton and Lord (1981) proposed the four-parameter logistic (4PL) IRT model. Later, Rulison and Loken’s (2009) study showed that the 4PL IRT model could effectively reduce underestimation caused by careless error. However, most relevant studies on the 4PL IRT model were conducted based on simulation experiments; few empirical experiments were included.

Therefore, this study aims to investigate the performance of the 4PL IRT model as a slip-correction mechanism in an empirical experiment. The measurement precision and efficiency of the 4PL and 3PL IRT models were discussed and compared with each other within two different administration conditions. The two conditions were a standard condition and a poor-start condition, and, in the latter, the first two items were always scored incorrect.

4PL IRT Model

Generally speaking, there are three kinds of IRT models: the 1PL model, 2PL model, and 3PL model. In the 1PL and 2PL models, the probability of a pass ranges between 0 and 1 as $θ$ goes from $- \infty$ to $\infty$ . On a selected response test, however, in practice, the probability of making the correct choice does not even approach 0 for low-ability students. Birnbaum (1968) introduced a lower asymptote to model the situation in which that examinees would make a random guess or answer on the basis of their knowledge would be reflected. The resulting 3PL model is

P_{3 PL} (θ) = c + (1 - c) \frac{1}{1 + \exp [- 1.702 a (θ - b)]},

where the lower asymptote c (guessing parameter) represents the probability that an examinee with extremely low ability would get the item with difficulty b correct. Parameter a is called the discrimination parameter and allows an item to discriminate among the examinees (Harvey & Hammer, 1999).

In the 1PL and 2PL models, the probability should approach 0 when a low-ability student answers difficult items correctly, and it should approach 1 when a high-ability student answers an easy item. Strictly speaking, this assumption might not always be true because an examinee knowing nothing could still select the correct answer by chance. Moreover, the chance would be even stronger if the examinee possessed partial knowledge (Bar-Hillel, Budescu, & Attali, 2005; Burton, 2002; Gardner-Medwin & Gahan, 2003; Yen, Ho, Chen, Chou, & Chen, 2010). However, high-ability students might on occasion miss items that they should have answered correctly when they are anxious, careless, unfamiliar with computer techniques, distracted by poor testing conditions, or even when they misread the question (Hockemeyer, 2002; Rulison & Loken, 2009). Under such conditions, the 3PL model might severely penalize a high-ability student who makes a careless error on an easy item (Barton & Lord, 1981; Rulison & Loken, 2009). More specifically, the lower asymptote in the 3PL IRT model can accommodate the situation when a low-ability student makes a correct guess on a difficult item, but the upper asymptote of 1 in the 3PL model assigns a probability of 0 when a high-ability student fails on an easy item (Loken & Rulison, 2010).

To correct the potential estimation error, Barton and Lord (1981) introduced an upper-asymptote parameter, expressed by the lowercase d, into the 3PL model:

P_{4 PL} (θ) = c + (d - c) \frac{1}{1 + \exp [- 1.702 a (θ - b)]} .

Whereas $P_{3 PL} (θ)$ ranges from the lower asymptote, c, to 1, $P_{4 PL} (θ)$ ranges from c to the upper-asymptote parameter, d, for item-specific “carelessness.” Figure 1 illustrates two typical 4PL IRT ICCs with b = 0, a = 1, c = 0.1, d = 0.8 and b = 0, a = 1, c = 0.05, d = 0.9, respectively.

Figure 1.

Two typical ICCs for the 4PL IRT model

To evaluate whether changing the upper asymptote is likely to improve the measurement precision of standardized tests, Barton and Lord (1981) compared the 3PL model with the 4PL model under two upper-asymptote values, d = 0.99 and d = 0.98. Reestimating test scores from four data sets: the Scholastic Aptitude Test (SAT) Verbal, SAT Math, Graduate Record Examinations (GRE) Verbal, and Advanced Placement (AP) Calculus AB led to the conclusion that the changes in ability estimation were too small to be of practical significance (Barton & Lord, 1981). However, it should be emphasized that the study mentioned was carried out based on the fixed response data from administered tests in which all examinees received predetermined items from the entire ability range. Hence, the item following the aberrant response was not dynamically selected from the item bank according to examinees’ accumulating information.

Rulison and Loken (2009) conducted two CAT simulation experiments to reevaluate the effect of the upper asymptote on ability estimation in a dynamic CAT environment. Figure 2 shows the trajectories of $θ$ estimation based on 3PL (solid line) and 4PL (dashed line) IRT models for a high-ability student (with true $θ$ = 2) who failed the first two items. It was obvious that the initial drop of estimated ability was followed by a very slow ascent in $\hat{θ}$ under the 3PL model, and this estimation failed to reach the examinee’s true ability. In 4PL-based CAT, the early errors resulted in an identical drop, but the $\hat{θ}$ ascended faster because the upper asymptote discounted the early mistakes. According to Rulison and Loken’s study, the 4PL model (with the upper asymptote d = 0.98) could reduce estimation error for high-ability students with a poor start. In other words, the 4PL IRT model provides an opportunity for examinees to remedy their careless mistakes in a CAT.

Figure 2.

Ability estimation in a poor-start 3PL and 4PL IRT model CAT

To further investigate the general implementation of the 4PL model, Loken and Rulison (2010) conducted their parameter estimation of the 4PL model, assessed its model fit, and evaluated its performance on an empirical, nonstandard IRT test (a self-report measure of delinquency). Instead of using a testwide upper asymptote, an item-specific upper asymptote was estimated in their simulation, and the results from the simulation and the empirical experiment indicated that the 4PL model could provide a refined model fit over the 2PL and 3PL models.

Green (2011) argued that high-ability students made careless errors in only a few cases and that the early errors noted by Rulison and Loken might occur at any time during the test. Hence, the effect was seen on the conventionally administered tests and the CAT, except that the CAT lessens the effect in comparison to a conventionally administered test. However, Green agreed that the asymmetry between lower and upper asymptotes in the 3PL model was the reason why the forced correct responses did not benefit low-ability students to a similar extent as careless errors penalized high-ability students. The empirical experiment that follows was thus conducted to reevaluate the performance of the 4PL model for students of all ability levels.

Method

In this study, an empirical experiment was conducted to compare the measurement precision and efficiency of the 3PL and 4PL IRT models under ordinary administration conditions, and to evaluate the performance of these two models in regard to diminishing estimation error caused by missing early items in a test. The four CAT systems, participant demography, item bank characteristics, and procedure are described in the following.

CAT Systems

Four versions of fixed-length web-based CAT systems were developed for this study: P3CAT, P4CAT, N3CAT, and N4CAT. Within P3CAT and P4CAT, the participants of this study were unaware that the first two items would be judged as wrong regardless of their responses, and that was to create a poor-start administration condition. The remaining test was administered according to the ordinary CAT procedure. The third and fourth tests (N3CAT and N4CAT) were administered according to ordinary CAT procedure, but N3CAT was based on 3PL IRT and N4CAT was based on 4PL IRT. The four CATs are categorized as follows, and the corresponding administration flowcharts are shown in Figure 3:

P3CAT: A CAT based on the 3PL model under a poor-start administration condition

P4CAT: A CAT based on the 4PL model under a poor-start administration condition

N3CAT: A normally administered 3PL-based CAT

N4CAT: A normally administered 4PL-based CAT

Figure 3.

The flowchart of four versions of CAT

Participants

A total of 212 senior high school students (115 boys and 97 girls) from six classes of three different senior high schools participated in this experiment. All participants were aged from 17 to 18, with learning experience via the Internet and computers for more than 3 years, and with learning experience in English for more than 6 years. None of them possessed experience with CAT. As they logged in to the web-based CAT system, the participants would be randomly assigned to one of the four CATs. The participants’ distribution to the four CATs is shown in Table 1.

Table 1.

Gender Distribution of Participants in Four CAT Conditions

	CAT type
Gender	P3CAT	P4CAT	N3CAT	N4CAT	Total
Male	30	31	27	27	115
Female	27	24	25	21	97
Total	57	55	52	48	212

Note: CAT = computerized adaptive test; P3CAT = a CAT based on the 3PL model under a poor-start administration condition; P4CAT = a CAT based on the 4PL model under a poor-start administration condition; N3CAT = a normally administered 3PL-based CAT; N4CAT = a normally administered 4PL-based CAT.

Item Bank

The item bank used in this experiment consisted of 84 items, each with four response options. This item bank was suggested in the study of Ho and Yen (2005), with each item drawn from the English Ability Test for college entrance in Taiwan, administered by the College Entrance Examinations Center (CEEC). The properties of the three parameters in the item bank are shown in Table 2. The test information curve is shown in Figure 4, and it shows that the item bank provided higher information amounts for middle- and high-ability examinees.

Table 2.

Distributional Characteristics of 3PL Item Parameters

Parameter	M	SD	Minimum	Maximum
b	−0.09	0.77	−1.93	1.67
a	1.10	0.38	0.12	1.88
c	0.20	0.10	0.00	0.58

Note: Number of items = 84. SD = standard deviation.

Figure 4.

Test information curve of the CEEC item bank

Experimental Procedure

The participants were requested to take a computerized English test in a computer lab, and they were informed that the result would be an important reference for their English ability. A demonstration of how to login and operate the web-based CAT system was then achieved by the experimenter. This was followed by an opportunity to ask questions. Participants were then required to complete the test within 40 min.

All CATs began with an item of medium difficulty to obtain each examinee’s initial ability, and terminated when a preset maximum test length (30 items) was reached. The test length was set to 30 because that should be sufficient to properly estimate the examinee’s ability (McBride, Wetzel, & Hetter, 1997). Expected a posteriori Bayesian estimation (EAP; Bock & Mislevy, 1982) with $p (θ) ~ N (0, 2)$ was used to estimate the participants’ ability in the four CATs, and items were selected using a simple maximum-information criterion. According to the study of Rulison and Loken (2009), using a less informative prior would make $\hat{θ}$ ascend rapidly and further improve the accuracy of the estimation. The value of the upper asymptote for the two 4PL CATs was set to 0.98. Fisher’s information for P3CAT and N3CAT was given as

I_{j} (θ) = \frac{{(1.7 a_{j})}^{2} (1 - c_{j})}{(c_{j} + e^{1.7 a_{j} (θ - b_{j})}) {(1 + e^{- 1.7 a_{j} (θ - b_{j})})}^{2}},

and Fisher’s information function for the P4CAT and N4CAT was given as

I_{j} (θ) = \frac{{(1.7 a_{j})}^{2} {(0.98 - c_{j})}^{2}}{(c_{j} + 0.98 e^{1.7 a_{j} (θ - b_{j})}) (1 - c_{j} + 0.02 e^{1.7 a_{j} (θ - b_{j})}) {(1 + e^{- 1.7 a_{j} (θ - b_{j})})}^{2}},

where

$I_{j} (θ)$ was the item information for item j,

$a_{j}$ was the item discrimination parameter for item j,

$b_{j}$ was the difficulty parameter for item j,

$c_{j}$ was the lower asymptote parameter for item j, and

$d$ was the upper-asymptote parameter with value of 0.98.

Results

To demonstrate the theta convergence of 3PL- and 4PL-based CAT under a practical situation, from each CAT system, a student of average ability was selected. The results of the estimation trace plots for the four students are shown in Figure 5. The four students were considered to possess identical English competence, as they all got a score of 80 in the English midterm. As the convergence curves show, the P3CAT suffered a more serious initial drop compared with that of the P4CAT, and the P4CAT ascended more rapidly than the P3CAT. However, the trajectory oscillation of N4CAT’s theta estimation ascended more steadily and faster than that of N3CAT. In addition, the final theta estimated by the 4PL-based CAT (P4CAT and N4CAT) was higher than that estimated by the 3PL-based CAT (P3CAT and N3CAT).

Figure 5.

The theta convergence of four CATs in practical situations

The Precision of 3PL- and 4PL-Based CAT

The English midterm test scores of participants were used as indicators to investigate whether participants assigned to different CATs were of the same level of English ability. Table 3 displays the descriptive statistics of participants’ English midterm scores in P3CAT, P4CAT, N3CAT, and N4CAT conditions. No significant difference was found in the English midterm scores among these four groups, F(3, 208) = 0.11, p = .9534.

Table 3.

Descriptive Statistics of Participants’ Midterm Scores in the Four CAT Conditions

	CAT type
	P3CAT (n = 57)		P4CAT (n = 55)		N3CAT (n = 52)		N4CAT (n = 48)
Variable	M	SD	M	SD	M	SD	M	SD
Midterm English score	76.04	12.06	75.96	11.14	76.80	9.96	76.96	11.17

The keynote of this article was the performance of the two IRT models under different testing conditions. The means and standard deviations of estimated theta, along with the corresponding standard errors (SEs), are presented in Tables 4 and 5. The results were subjected to two-way ANOVAs to evaluate how the IRT models (3PL and 4PL) and testing conditions (poor-start and ordinary condition) might affect the theta estimation. As Table 6 shows, no significant interaction was detected between IRT models and testing conditions for estimated theta, F(1, 208) = 0.68, p > .05. However, a significant main effect of the IRT model, F(1, 208) = 12.21, p < .001, and test type, F(1, 208) = 6.62, p < .05, was found. The results revealed that the theta estimated by P4CAT was significantly higher than that estimated by P3CAT in both testing conditions, and the theta estimated in the ordinary condition was higher than that in the poor-start condition.

Table 4.

Means (SE) of the Estimated Theta Based on the 3PL and 4PL Models in Different Test Conditions

	Test condition
IRT model	Poor start	Ordinary start
3PL	−0.34 (0.64)	0.01 (0.43)
4PL	−0.07 (0.63)	0.15 (0.66)

Note: SE = standard error; IRT = item response theory; 3PL = three-parameter logistic; 4PL = four-parameter logistic.

Table 5.

Means (SE) of the SE Based on the 3PL and 4PL Model in Different Test Conditions

	Test condition
IRT model	Poor start	Ordinary start
3PL	0.23 (0.04)	0.22 (0.02)
4PL	0.22 (0.04)	0.21 (0.02)

Note: SE = standard error; IRT = item response theory; 3PL = three-parameter logistic; 4PL = four-parameter logistic.

Table 6.

Two-Way ANOVA Results of Theta (N = 212)

Source	SS	df	MS	F
IRT model (3PL, 4PL)	4.32	1	4.32	12.21***
Testing condition (Poor, Ordinary)	2.34	1	2.34	6.62*
IRT model × Test condition	0.24	1	0.24	0.68
Error	73.58	208	0.35

Note: IRT = item response theory; 3PL = three-parameter logistic; 4PL = four-parameter logistic.

p < .05. ***p < .001.

However, Table 7 shows no significant interaction between IRT models and testing conditions in term of SE, F(1, 208) = 0.41, p > .05. The only significant effect found was between different IRT models, F(1, 208) = 8.26, p < .01, which indicated that the SE of the 4PL model was smaller than the 3PL model in both test conditions.

Table 7.

Two-Way ANOVA Results of SE (N = 212)

Source	SS	df	F
IRT model (3PL, 4PL)	0.01	1	8.26**
Testing condition (poor, ordinary)	0.00	1	1.08
IRT model × testing condition	0.00	1	0.41
Error	0.22	208

Note: SE = standard error; IRT = item response theory; 3PL = three-parameter logistic; 4PL = four-parameter logistic.

p < .01.

The Efficiency of 3PL- and 4PL-Based CAT

The efficiency of 3PL- and 4PL-based CAT was evaluated by comparing the number of items required for these CATs to achieve a specified estimation precision. A few stopping criteria for these CATs were set, including an SE reaching 0.45, 0.40, 0.35, and 0.30. Based on these criteria, the numbers of the required items within P3CAT, P4CAT, N3CAT, and N4CAT were compared. Figure 6 depicts the required items for the four CATs as the corresponding SE reaches the different stopping criteria. According to the plots for test length, the efficiency of the 4PL-based CATs (P4CAT and N4CAT) was higher than that of the 3PL-based CATs (P3CAT and N3CAT) under both testing conditions.

Figure 6.

Mean of test length for four CAT conditions

To verify the efficiency statistics of 3PL- and 4PL-based CATs under poor-start and ordinary testing conditions in practical situations, two sets of unpaired t tests were conducted. The results indicated that the efficiency improvement introduced by the 4PL IRT model under the poor-start condition was not statistically significant (as shown in Table 8). However, Table 9 reveals that the 4PL model could improve the CAT’s efficiency significantly under ordinary conditions.

Table 8.

Efficiency Statistics of P3CAT (n = 57) and P4CAT (n = 55)

Criterion	CAT type	Mean test length	Difference of means	t
SE≤0.45	P3CAT	6.49	0.71	1.45
	P4CAT	5.78
SE≤0.40	P3CAT	8.09	0.80	1.36
	P4CAT	7.29
SE≤0.35	P3CAT	10.56	0.89	1.26
	P4CAT	9.67
SE≤0.30	P3CAT	15.26	1.01	1.05
	P4CAT	14.25

Note: CAT = computerized adaptive test; SE = standard error.

Table 9.

Efficiency Statistics of N3CAT (n = 52) and N4CAT (n = 48)

Criterion	CAT type	Mean test length	Difference of means	t
SE≤0.45	N3CAT	6.17	1.34	3.99***
	N4CAT	4.83
SE≤0.40	N3CAT	7.65	1.70	4.33***
	N4CAT	5.94
SE≤0.35	N3CAT	10.50	2.56	4.98***
	N4CAT	7.94
SE≤0.30	N3CAT	14.92	2.59	2.70***
	N4CAT	12.33

Note: CAT = computerized adaptive test; SE = standard error.

***

p < .001.

According to this empirical experiment, the following results were obtained: (a) The theta estimated by P4CAT was significantly higher than that estimated by the P3CAT in both tests, (b) the theta estimated under ordinary conditions was also significantly higher than that in poor-start conditions, and (c) under ordinary conditions, test efficiency could be improved significantly with implementation of the 4PL IRT model. The slightly smaller SE of 4PL seems inconsistent with the hypothesis that the addition of the extra parameter would increase the SE value. However, as the 4PL model required fewer items than did the 3PL model to reach a certain SE, as shown in Table 8, it was reasonable that the 4PL estimates had slightly smaller SE than did their 3PL counterparts when the stopping criterion was set to the same fixed numbers of items. Another explanation for the smaller SE of the 4PL model could be that the 4PL model makes the estimation more robust against violations of the know-correct assumptions of IRT (Ackerman, 1989). As a result, the forced errors inconsistent with the examinee’s ability degrade the theta estimate less with the 4PL model than that with the 3PL model.

Conclusion

The objective of assessment is to measure examinees’ true ability accurately. In real testing situations, the confounding effect of lucky guesses and careless errors should be taken into consideration, especially for selected response tests. If these careless errors occur at the beginning of the test, the ability could be underestimated seriously under the 3PL IRT model. Simulation experiments made previously showed that the 4PL IRT model could diminish estimation error due to missing items early on, and that the measurement efficiency and precision could be improved with the model (Rulison & Loken, 2009). Green (2011) further suggested that the choice of the 4PL model meant losing precision for all scores to lessen the impact of one or two blunders on scores for the highly proficient.

In this study, the theta estimated by the 4PL model was higher than that estimated by the 3PL model regardless of the initial errors. Also, the difference in the average theta of 4PL and 3PL in poor-start conditions (0.27, P4CAT-P3CAT) was more obvious than that in ordinary conditions (0.14, N4CAT-N3CAT). The upper asymptote in this study means more tolerance for careless errors from the perspective of the examinees. A similar conclusion has also been made in Green (2011). Meanwhile, it seems that the upper asymptote lessens not only the excessive punishment for early errors but also the punishment for errors at any point of the test. According to this study, the precision of the 4PL IRT model is better than that of the 3PL model in both administration conditions. Finally, under ordinary administration conditions, the efficiency of CAT could be improved with the 4PL model.

In sum, the 4PL IRT model could not only reduce the problems of ability underestimation introduced by initial careless mistakes in practical situations but also improve measurement efficiency. To investigate the evaluation of the 4PL model comprehensively, further studies are required to (a) evaluate the performance of the 4PL model with an item bank in which all four parameters are item specific and calibrated based on empirical response patterns (Loken & Rulison, 2010); (b) investigate the effect of the 4PL model on theta estimation with forced errors occurring in the beginning, middle, and end of CAT; and (c) have examinees take 3PL and 4PL model CATs to compare the differences between these two models.

Footnotes

Acknowledgements

The authors would like to thank the National Science Council of Taiwan for financial support.

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported in part by the Science Education Division of Taiwan National Science Council under Grant No. NSC 99-2511-S-003-010-MY2.

References

Ackerman

T. A.

(1989). Unidimensional IRT calibration of compensatory and noncompensatory multidimensional items. Applied Psychological Measurement, 13, 113-127.

Bar-Hillel

Budescu

Attali

(2005). Scoring and keying multiple choice tests: A case study in irrationality. Mind & Society, 4, 3-12.

Barton

M. A.

Lord

F. M.

(1981). An upper asymptote for the three-parameter logistic item-response model. Princeton, NJ: Educational Testing Services.

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 397-479). London, England: Addison-Wesley.

Bock

R. D.

Mislevy

R. J.

(1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431-444.

Burton

R. F.

(2002). Misinformation, partial knowledge and guessing in true/false tests. Medical Education, 36, 805-811.

Gardner-Medwin

A. R.

Gahan

(2003). Formative and summative confidence-based assessment. Paper presented at the Proceedings of the 7th International Computer-Aided Assessment Conference, Loughborough, England.

Green

B. F.

(2011). A comment on early student blunders on computer-based adaptive tests. Applied Psychological Measurement, 35, 165-174.

Harvey

R. J.

Hammer

A. L.

(1999). Item response theory. The Counseling Psychologist, 27, 353-384.

10.

R.-G.

Yen

Y.-C.

(2005). Design and evaluation of an XML-based platform-independent computerized adaptive testing system. IEEE Transactions on Education, 48, 230-237.

11.

Hockemeyer

(2002). A comparison of non-deterministic procedures for the adaptive assessment of knowledge. Psychologische Beiträge, 44, 495-503.

12.

Loken

Rulison

(2010). Estimation of a four-parameter item response theory model. British Journal of Mathematical and Statistical Psychology, 63, 509-525.

13.

McBride

J. R.

Wetzel

C. D.

Hetter

R. D.

(1997). Preliminary psychometric research for CAT-ASVAB: Selecting an adaptive testing strategy. In Sands

W. A.

Waters

B. K.

McBride

J. R.

(Eds.), Computerized adaptive testing: From inquiry to operation (pp. 83-95). Washington, DC: American Psychological Association.

14.

Rulison

Loken

(2009). I’ve fallen and I can’t get up: Can high-ability students recover from early mistakes in CAT? Applied Psychological Measurement, 33, 83-101.

15.

Yen

Y.-C.

R.-G.

Chen

L. J.

Chou

K. Y.

Chen

Y. L.

(2010). Development and evaluation of a confidence-weighting computerized adaptive testing. Educational Technology & Society, 13, 163-176.