Detection of Item Preknowledge Using Likelihood Ratio Test and Score Test

Abstract

An increasing concern of producers of educational assessments is fraudulent behavior during the assessment (van der Linden, 2009). Benefiting from item preknowledge (e.g., Eckerly, 2017; McLeod, Lewis, & Thissen, 2003) is one type of fraudulent behavior. This article suggests two new test statistics for detecting individuals who may have benefited from item preknowledge; the statistics can be used for both nonadaptive and adaptive assessments that may include either or both of dichotomous and polytomous items. Each new statistic has an asymptotic standard normal n distribution. It is demonstrated in detailed simulation studies that the Type I error rates of the new statistics are close to the nominal level and the values of power of the new statistics are larger than those of an existing statistic for addressing the same problem.

Keywords

cheating fraud test security

Fraudulent behavior during educational assessments is an increasingly common problem. Naturally, there is a growing interest in methods for detection of fraudulent behavior, which is evident from the publication of three recent edited volumes on the topic (Cizek & Wollack, 2017; Kingston & Clark, 2014; Wollack & Fremer, 2013) in addition to numerous journal articles (e.g., Belov, 2013, 2016; McClintock, 2015; McLeod et al., 2003; Romero, Riascos, & Jara, 2015; Segall, 2002; Shu, Henson, & Luecht, 2013; van der Linden, 2009; van der Linden & Jeon, 2012; van der Linden & Lewis, 2015) and conference presentations (e.g., Eckerly, Babcock, & Wollack, 2015) on the topic.

While there are several types of fraudulent behavior (Kingston & Clark, 2014), the interest in this article will be on item preknowledge in which a “source” shares assessment questions and/or their answers (where the source could be a teacher, a test preparation company, a website, or individual examinees) and then several beneficiaries memorize the assessment questions and/or answers. For example, Educational Testing Service discovered in 2002 that many students in several countries were benefiting from websites showing live items used in the Graduate Record Examination (GRE); the phenomenon was so widespread that average scores on GRE verbal increased by 100 points (out of a possible 800 points) in one country and 50 points in another (Kyle, 2002). The shared/memorized items are usually referred to as “compromised” items. Item preknowledge results in inflated scores for some examinees and should be identified (Belov, 2013, p. 142). The focus of this article will be on detecting examinees who may have benefited from item preknowledge, both for nonadaptive assessments and for computerized adaptive tests (CATs) that are more susceptible to item preknowledge due to more frequent testing.

This article considers only the case when the investigator knows which items are compromised; Belov (2013), Eckerly (2017), and Shu, Henson, and Luecht (2013) considered this case. Typically, such a case arises when the assessment administrators become aware after an assessment about some items possibly being compromised. Such a case may also arise when the assessment administrators have applied a method to detect compromised items (e.g., that suggested by Veerkamp & Glas, 2000) to find that some items may have been compromised. The real data examples later in this article demonstrate that such a situation can arise in practice.

This article suggests two new statistics for detection of examinees who may have benefited from item preknowledge. Both statistics are based on item response theory (IRT) and quantify the difference between the ability estimates computed from the compromised items and noncompromised items and are based on two classical statistical tests, namely, the likelihood ratio test (LRT) and the score test (e.g., Cox & Hinkley, 1974; Rao, 1973); the asymptotic null distributions of these statistics are known. The new statistics are computed for two real data sets from a nonadaptive assessment. The Type I error rates and power of the new statistics are examined and are compared to those of the statistic of Belov (2013) for both nonadaptive and adaptive assessments.

As in Belov (2013), this article involves the assumption that the item parameters are known. While Belov seems to have considered only dichotomous items,¹ the two statistics suggested in this article apply to an assessment consisting of a mix of dichotomous and polytomous items; thus, the approaches suggested in this article are the first that can be used with an assessment that includes some polytomous items.

Background

Let us consider an assessment that consists of a mix of dichotomous and polytomous items whose parameters are assumed known (and are equal to the estimates computed from a previous calibration using IRT). Let y_i denote the score of an examinee (with true ability of θ) on item i and let $P_{i} (y_{i} | θ)$ denote the probability of the score y_i on item i for ability θ. For example, if the three-parameter logistic model (3PLM) is used for item i, then:

P_{i} (y_{i} = 1 | θ) = c_{i} + (1 - c_{i}) \frac{\exp [a_{i} (θ - b_{i})]}{1 + \exp [a_{i} (θ - b_{i}]}, and P_{i} (y_{i} = 0 | θ) = 1 - P_{i} (1 | θ),

where a_i, b_i, and c_i, respectively, are the slope, difficulty, and guessing parameters of item i (e.g., Birnbaum, 1968), whereas if the generalized partial credit model is used for item i that has the score categories 0, 1,…, m_i, then:

P_{i} (y_{i} = j | θ) = \frac{\exp [\sum_{h = 0}^{j} a_{i} (θ - b_{i h})]}{\sum_{c = 0}^{m_{i}} \exp [\sum_{h = 0}^{c} a_{i} (θ - b_{i h})]}, j = 0, 1, …, m_{i},

where a_i is the slope parameter and b_ih’s are the location parameters of the item (e.g., Muraki, 1992). Let us assume that a known subset S of items in the item pool is suspected to have been compromised. Let s denote the subset of the compromised items that was administered to the examinee. Depending on the extent of item preknowledge and the design of the assessment, an examinee could receive between 0 and all of the compromised items (i.e., s could be between a null set and S). For example, it is possible that 50 items from a CAT pool are compromised (so that S includes 50 items) and an examinee knows about all of them but receives only 10 of them (so that s includes 10 items) when he or she is tested. Let y _s denote the collection of scores of the examinee on the items in s, that is, $y_{s} = {y_{i}, i \in s}$ . Let $\bar{s}$ denote the set of noncompromised items that were administered to the examinee. Let $y_{\bar{s}}$ denote the collection of scores of the examinee on the items in $\bar{s}$ . Let N denote the total number of items administered to the examinee.

The Research Problem

Given the estimated parameters of the items, the scores of several examinees on several items, and a known set S of items that is suspected to have been compromised, the goal here is to detect examinees who may have benefited from item preknowledge on either a nonadaptive test or a CAT that may include both dichotomous and polytomous items. Further, it is assumed that if an examinee benefited from item preknowledge, then he or she has preknowledge of all items in S and (unfairly) obtains a score larger than what is expected from the IRT model and the examinee’s true ability on any item on S. This research problem is essentially the same as Case 1 considered in Belov (2013).

Review of a Statistic Based on the Kullback–Leibler Divergence

Belov (2013) suggested using the approximate Kullback–Leibler divergence (Kullback & Leibler, 1951) between the posterior distribution of the ability given the item scores on s and the posterior distribution of the ability given the item scores on $\bar{s}$ as the statistic (denoted as h henceforth) for detecting item preknowledge. If the examinee performed significantly better on s than on $\bar{s}$ , then the posterior distribution based on s is located far to the right of that based on $\bar{s}$ and h is large. Belov (2013) suggested two empirical approaches (referred to as algorithm 1 and algorithm 2) to find appropriate critical values of h for his Case 1 (that is considered here). Only algorithm 2, which led to larger power than algorithm 1 in Belov, is considered here. The algorithm involves the assumption that the examinees belong to several assessment centers (that could be determined by the testing location, college, region, etc., of the examinees). Then 20% assessment centers with the largest values of h on average are flagged, a critical value (denoted w henceforth) of h determined from the remaining (and not flagged) 80% assessment centers, and an examinee is flagged for possibly benefiting from item preknowledge only if he or she is in one of the 20% flagged assessment centers and the value of his or her h statistic is larger than w.

The above algorithm has at least three practical limitations. First, if examinees in more than 20% assessment centers benefited from item preknowledge, then the above strategy would fail to flag the examinees in them. Belov (2013) did not discover this problem because he simulated 20% or fewer aberrant assessment centers (an aberrant examinee is one who had item preknowledge and an aberrant assessment center is one that includes one or more aberrant examinees). Second, if the actual number of assessment centers whose examinees benefited from item preknowledge is smaller than 20%, then the flagging of 20% assessment centers and computing w from the remaining 80% assessment centers could lead to the loss of quite a bit of information (especially if there are only a few assessment centers) as well as causing some actually nonaberrant assessment centers appear aberrant. Third, for assessments that involve continuous administration and every center is not involved in every administration, it is not clear which assessment centers should be included in the computations.

It is clear from the above limitations that there is a scope of further research on this area. Especially, it can be beneficial if the investigator could use a test statistic with a known null distribution to detect item preknowledge—that would obviate the need to determine w empirically and would overcome the above-mentioned limitations of the algorithm of Belov (2013).

Review of Other Methods of Detecting Item Compromise

McLeod, Lewis, and Thissen (2003) suggested a Bayesian method to detect item preknowledge in CATs; they did not assume that the set of aberrant items is known—so their method is expected to lead to smaller power than the statistics considered in this article (some limited simulations confirmed this). Researchers such as Segall (2002), Shu et al. (2013), and Eckerly, Babcock, and Wollack (2015) suggested IRT models for detection of item preknowledge for nonadaptive assessments. van der Linden and van Krimpen-Stoop (2003) suggested a Bayesian method that uses item scores and response times to detect item preknowledge for nonadaptive assessments. Qian, Staniewska, Reckase, and Woo (2016) used a similar approach in which the residuals from the joint application of the Rasch or the two-parameter logistic model to item scores and the response-time model of van der Linden (2006) to item response times were used to flag examinees for possible item preknowledge in both adaptive and nonadaptive assessments. Researchers such as Veerkamp and Glas (2000) discussed how one can detect items that have been compromised. These methods are not considered henceforth in this article. Note that all of these methods apply to tests with only dichotomous items.

Two New Statistics Based on the LRT and the Score Test

The rationale behind the new statistics

The rationale of the h statistic is that the posterior distributions of a nonaberrant examinee (one who did not benefit from item preknowledge) based on s and $\bar{s}$ are close to each other, but, for an aberrant examinee (one who benefited from item preknowledge), the posterior distribution based on s is located far to the right of that based on $\bar{s}$ .

If, for an examinee, the posterior distributions based on s and $\bar{s}$ are close to each other, then her ability estimates based on s and $\bar{s}$ are close to each other as well; on the other hand, if the posterior distribution based on s is located far to the right of that based on $\bar{s}$ , then her ability estimate based on s is expected to be substantially larger than that based on $\bar{s}$ . Thus, instead of testing for the closeness of the posterior distributions (that is the basis of h) based on s and $\bar{s}$ , it is possible to test if the abilities based on s and $\bar{s}$ are significantly different. Two widely used statistical tests² that can be used to test for the equality of the model parameters underlying two subsets of data (or two samples) are the LRT and the score test (e.g., Cox & Hinkley, 1974; Lehmann & Romano, 2005; Rao, 1973). For example, in educational measurement, Finkelman, Weiss, and Kim-Kang (2010) used the LRT to examine whether the difference between the ability estimates of an examinee from two different assessments is significant in the context of measurement of change, Guo and Drasgow (2010) used the LRT to test if the scores on an unproctored version of a test was larger than that on a proctored version, Klauer and Rettig (1990) used both of these tests to examine whether the difference between the ability estimates from multiple subtests administered to an examinee is significant in the context of person-fit assessment, Romero, Riascos, and Jara (2015) used the LRT to detect whether an examinee was involved in answer copying, and Glas and Dagohoy (2007) used the score test to examine whether the performance of an examinee varied over several subtests (involving polytomous items).

Both the LRT and the score test statistic have a known asymptotic null distribution, the χ² distribution with one degree of freedom (e.g., Finkelman, Weiss, & Kim-Kang, 2010; Klauer & Rettig, 1990). Further, hypothesis tests based on likelihood ratios are asymptotically the most powerful in general³ (e.g., Cox & Hinkley, 1974, pp. 312, 320; Drasgow, Levine, & McLaughlin, 1987) because of the Neyman–Pearson lemma (e.g., Cox & Hinkley, 1974; Lehmann & Romano, 2005; Romero et al., 2015), and the score test is asymptotically equivalent to the LRT (e.g., Cox & Hinkley, 1974; Lehmann & Romano, 2005)—therefore, these tests may perform better than the h statistic.

A Statistic Based on the LRT

For an examinee, let us define the maximum likelihood estimate (MLE) or the weighted maximum likelihood estimate (WLE; Warm, 1989) of the examinee ability from the scores on s as ${\hat{θ}}_{s}$ , that from the scores on $\bar{s}$ as ${\hat{θ}}_{\bar{s}}$ , and that from the scores on all the items as ${\hat{θ}}_{0}$ .

The LRT statistic (e.g., Cox & Hinkley, 1974; Lehmann & Romano, 2005; Rao, 1973) for testing the null hypothesis of equality of the examinee ability over s and $\bar{s}$ is given by

L = 2 [L ({\hat{θ}}_{s}; y_{i}, i \in s) + L ({\hat{θ}}_{\bar{s}}; y_{i}, i \in \bar{s}) - L ({\hat{θ}}_{0}; y_{i}, i = 1, 2, …, N)],

where

L ({\hat{θ}}_{s}; y_{i}, i \in s) = log likelihood of the scores on s at {\hat{θ}}_{s},

L ({\hat{θ}}_{\bar{s}}; y_{i}, i \in \bar{s}) = log likelihood of the scores on \bar{s} at {\hat{θ}}_{\bar{s}},

and $L ({\hat{θ}}_{0}; y_{i}, i = 1, 2, …, N)$ = log-likelihood of the scores on all the items at ${\hat{θ}}_{0}$

Because, for example,

L ({\hat{θ}}_{s}; y_{i}, i \in s) = \sum_{i \in s} \log P_{i} (y_{i} | {\hat{θ}}_{s}),

the LRT statistic given in Equation 1 can be expressed as

L = 2 {\sum_{i \in s} \log P_{i} (y_{i} | {\hat{θ}}_{s}) + \sum_{i \in \bar{s}} \log P_{i} (y_{i} | {\hat{θ}}_{\bar{s}}) - \sum_{i = 1}^{N} \log P_{i} (y_{i} | {\hat{θ}}_{0})} \cdot

The above result holds for nonadaptive assessments due to local independence given the examinee ability and for CATs due to local independence given the examinee ability and item strings (e.g., Mislevy & Chang, 2000). Finkelman et al. (2010) showed in the context of measurement of change that a statistic similar to the above statistic follows an asymptotic $χ_{1}^{2}$ null distribution for both CAT and nonadaptive assessments. The value of L is large for an examinee who benefited from item preknowledge.

The statistic L is appropriate for two-sided alternative hypotheses. However, the alternative hypothesis in our case is one sided—it is desired here to detect those who performed better on the compromised items and not to detect those who performed better on the noncompromised items. For one-sided alternatives, researchers such as Cox (2006, p. 104), Cox and Hinkley (1974, p. 315), and Biehler, Holling, and Doebler (2015) suggested the use of the signed likelihood ratio statistic, which, in our context, is given by

L_{s} = {\begin{array}{l} \sqrt{L} if {\hat{θ}}_{s} \geq {\hat{θ}}_{\bar{s}}, \\ - \sqrt{L} if {\hat{θ}}_{s} < {\hat{θ}}_{\bar{s}} \cdot \end{array}

Thus, the absolute value of L_s is equal to the square root of L, and L_s is positive if the examinee’s estimated ability based on s is larger than or equal to that based on $\bar{s}$ and negative otherwise. A large value of L_s (that means a large value of L accompanied with ${\hat{θ}}_{s} > {\hat{θ}}_{\bar{s}}$ ) leads to the rejection of the null hypothesis of no item preknowledge and the flagging of the corresponding examinee for possibly benefiting from item preknowledge.

The statistic L_s has an asymptotic standard normal distribution (e.g., Cox, 2006, p. 104) under the null hypothesis of no item preknowledge. It is well known that the square of a standard normal random variable has a $χ_{1}^{2}$ distribution (e.g., Casella & Berger, 2002, p. 219)—so it is natural that L_s, which is the square root of L (that has a $χ_{1}^{2}$ null distribution) and has a positive or negative sign equally often under the null hypothesis,⁴ has a standard normal null distribution.

A Statistic Based on the Score Test

Rao’s (1973) score test statistic or Lagrange multiplier test statistic for testing the null hypothesis of equality of the ability parameter over s and $\bar{s}$ is provided by

R = \frac{{(\nabla ({\hat{θ}}_{0}; y_{i}, i \in s))}^{2}}{I_{s} ({\hat{θ}}_{0})} + \frac{{(\nabla ({\hat{θ}}_{0}; y_{i}, i \in \bar{s}))}^{2}}{I_{\bar{s}} ({\hat{θ}}_{0})},

where, for example, $\nabla ({\hat{θ}}_{0}; y_{i}, i \in s)$ is the first derivative of the log likelihood of y_i, i ∈ s at $θ = {\hat{θ}}_{0}$ , and $I_{s} ({\hat{θ}}_{0})$ is the estimated information on s at ${\hat{θ}}_{0}$ . Expressions of $\nabla ({\hat{θ}}_{0}; y_{i}, i \in s)$ and $I_{s} ({\hat{θ}}_{0})$ for common IRT models can be found in, for example, Baker and Kim (2004, pp. 64–71). The R statistic is asymptotically equivalent to L and has an asymptotic $χ_{1}^{2}$ null distribution (e.g., Cox & Hinkley, 1974; Rao, 1973). The value of R is large for an examinee who benefited from item preknowledge.

Because the alternative hypothesis in our case is one sided as discussed earlier, it is more appropriate to use the signed score statistic (Cox & Hinkley, 1974, Cox, 2006, p. 104), which, in our context, is given by:

R_{s} = {\begin{array}{l} \sqrt{R} if {\hat{θ}}_{s} \geq {\hat{θ}}_{\bar{s}}, \\ - \sqrt{R} if {\hat{θ}}_{s} < {\hat{θ}}_{\bar{s}} \cdot \end{array}

A large value of R_s (that means a large value of R accompanied with ${\hat{θ}}_{s} > {\hat{θ}}_{\bar{s}}$ ) leads to the rejection of the null hypothesis and the flagging of the corresponding examinee for possibly benefiting from item preknowledge.

Like L_s, the statistic R_s has an asymptotic standard normal distribution under the null hypothesis of no item preknowledge (e.g., Cox, 2006, p. 104).

The Advantage of L_s and R_s Over h

Both the L_s and R_s statistics have a known asymptotic null distribution, the standard normal distribution—that is a clear advantage over the h statistic when the number of compromised and noncompromised items are large. Further, the LRT and the score tests, which are the basis of L_s and R_s, are quite popular in statistical applications (e.g., Cos & Hinkley, 1974) and in educational measurement (Finkelman, Weiss, & Kim-Kang, 2010; Guo & Drasgow, 2010; Klauer & Rettig, 1990; Romero et al., 2015)—so these statistics would be intuitively appealing to measurement practitioners. Further, h is appropriate for two-sided alternative hypotheses and would be large for those who performed considerably better on $\bar{s}$ than on s and would flag such examinees. For example, consider an assessment with 20 items, all with $a_{i} = 1, b_{i} = 0$ , and c_i = 0. Suppose further that both s and $\bar{s}$ include 10 items. Let us further consider two examinees both of whom answered 10 items correctly. Let us assume that one of them is aberrant and answered 8 items on s correctly, while the other is nonaberrant and answered 2 items on s correctly. The value of h is 9.52 for both of these examinees and would flag both of them. The statistics L_s and R_s do not suffer from this problem—they would be positive only when an examinee performs better on s than on $\bar{s}$ . Finally, both L_s and R_s can be applied to both dichotomous and polytomous items, whereas Belov (2013) seems to have considered only dichotomous items.

A Real Data Application

Let us consider item response data from two forms of a nonadaptive licensure assessment. The data sets were analyzed in several chapters (such as Eckerly, 2017) of Cizek and Wollack (2017) and in Sinharay (2016). Both forms include 170 operational items that are dichotomously scored; 87 items are common between the two forms. Item scores were available for 1,636 examinees for Form 1 and 1,644 examinees for Form 2. The licensure organization (who provided the data) identified 63 and 61 items as compromised on Forms 1 and 2, respectively (though several of these items were common between the forms, so that the combined number of compromised items was 64). The organization also flagged 46 and 48 individuals on Forms 1 and 2, respectively, as possible cheaters from a variety of statistical analysis and an investigative process that brought in other information. The examinees for each form were distributed in several hundred assessment centers.

The values of h, L_s, and R_s were computed from the two data sets. The Rasch model is operationally used in the assessment—the operational difficulty parameter estimates were used in the calculations. The WLEs of abilities were used to compute the L_s and R_s statistics primarily because the WLEs were always finite (that agrees with similar findings in Magis & Verhelst, 2014), whereas MLEs are infinite for all correct or all incorrect response patterns. The WLEs were computed using the Newton–Raphson algorithm.

The proportions of examinees flagged by the three statistics at levels of .001, .01, and .05 are provided in Table 1. The first three lines provide the proportions flagged among all examinees. The last three lines provide the proportions flagged only among the (46 or 48) examinees who were flagged by the licensure organization; thus, for example, the proportion .15 for L_s at level .001 for Form 1 in the fifth row of numbers denotes that among the 46 examinees flagged by the licensure organization, 7 examinees were flagged (note that 7/46 ≈ 0.15) at level .001 by L_s. The proportions flagged among those who were not flagged by the licensure organization are very close to those shown in the first three rows (that is expected as the organization flagged only about 2.4% of the examinees).

Table 1.

The Proportion of Examinees Flagged for the Real Data.

Group	Statistic	Form 1			Form 2
Group	Statistic	.001	.01	.05	.001	.01	.05
All	h	.01	.02	.03	.02	.03	.05
All	L_s	.01	.03	.09	.04	.09	.17
All	R_s	.01	.03	.08	.04	.09	.17
Flagged	h	.13	.24	.26	.10	.21	.27
Flagged	L_s	.15	.26	.37	.23	.31	.38
Flagged	R_s	.15	.24	.35	.22	.30	.37

Table 1 shows that L_s and R_s flag more examinees than h in almost all cases. The smaller flagging rates of h, especially in the last three rows of the table, can most likely be attributed to the fact that the examinees who had large values of L_s or R_s were distributed among many more than 20% assessment centers and the examinees flagged by the licensure organization were also distributed among several assessment centers rather than being concentrated in a few of them; h is expected to perform relatively well when the examinees with item preknowledge are included in 20% or fewer assessment centers. Table 1 also shows that the proportion flagged for all the statistics is much larger among the examinees flagged by the licensure organization (bottom three rows of the table) compared to among all examinees (top three rows of the table)—this result provides some evidence that the suggested statistics are somewhat successful. Note that item compromise was not the only reason of flagging by the licensure organization—so a proportion smaller than 1.0 in the bottom three rows is not a severe limitation of the suggested statistics.

The first five rows of Table 2 provide, for five examinees flagged by the licensure organization on Form 2, the proportion correct scores (the number correct divided by the total number of items) on the compromised items (P_s) and noncompromised items ( $P_{\bar{s}}$ ), the ability estimates ${\hat{θ}}_{s}$ and ${\hat{θ}}_{\bar{s}}$ , and the values of L_s, R_s, and h. The last row of the table provides the same numbers for one examinee who was not flagged by the licensure organization. The values of the statistics that are statistically significant at level .001 are accompanied by three asterisks, those significant at level .01 are accompanied by two asterisks, and those not significant at level .05 are accompanied by no asterisks.

Table 2.

Some Details for Six Examinees for the Real Data.

P_s	$P_{\bar{s}}$	${\hat{θ}}_{s}$	${\hat{θ}}_{\bar{s}}$	L_s	R_s	h
87	0.60	2.45	0.34	5.16***	4.99***	20.48***
84	0.69	2.16	0.81	3.30***	3.23***	8.64**
90	0.82	2.81	1.62	2.43**	2.37**	4.84
62	0.77	0.86	1.30	−1.16	−1.16	1.17
62	0.82	0.86	1.62	−1.95	−1.95	3.33
57	0.87	0.62	2.09	−3.70	−3.65	12.42***

The performance on the compromised items was much better than that on the other items (with respect to both proportion corrects and ability estimates) for the first three examinees; among them, the first was flagged by all the three statistics at level .001, the second was flagged by L_s and R_s at level .001 but flagged by h at level .01 (and not at level .001), and the third was flagged by L_s and R_s at level .01 but not flagged by h at level .05.⁵ Thus, Table 2 shows that h may not flag examinees who are flagged by L_s and R_s. The examinees corresponding to the fourth and fifth row of the table performed worse on compromised items than on noncompromised items (and must have been flagged by the licensure organization on other grounds) and were not flagged by any of the three statistics. The examinee corresponding to the sixth row was not flagged by L_s and R_s but was flagged by h; however, this examinee performed much worse on the compromised items than on the other items—so there does not exist enough evidence to flag the examinee for item preknowledge—the use of h leads to a false alarm for the examinee due to the inappropriateness of h to test one-sided hypotheses.

A Simulation Study Involving a Nonadaptive Assessment

A simulation study was performed to compare the Type I error rates and power of the h statistic to those of L_s and R_s for nonadaptive assessments.

Design of the Simulation

The simulation study involved a 170-item assessment calibrated using the Rasch model like the above-mentioned operational assessment. The operational item parameter estimates of Form 1 of the assessment were used as the true item parameters to generate data in the simulations and also in the computation of the statistics for detecting item preknowledge. The true abilities of all the examinees were simulated from the $N (0, 1)$ distribution.

A total of 1,000 assessment centers were used, with 100 examinees in each of them. The percentage of aberrant test centers was 10 or 20. Note that any value larger than 20 would clearly lead to a smaller power of the h statistic (as described earlier); however, no such value was used. Thus, the simulation cases considered here are the best-case scenarios for the h statistic. The percentage of aberrant examinees in the aberrant assessment centers was 10 or 20; the percentage of aberrant examinees in the nonaberrant assessment centers was 0. The size of S was assumed to be 10, 20, or 30. It was assumed that each aberrant examinee had preknowledge of the whole of S.⁶

For any simulation condition (where an example of a simulation condition is “10 aberrant items, 10% aberrant assessment centers, and 10% aberrant examinees”), the set S was chosen as a random subset of 10, 20, or 30 items from all items of Form 1. Then, 10% or 20% centers were randomly chosen from the 1,000 centers as aberrant. Then, 10% or 20% aberrant examinees were randomly chosen from the 100 examinees in each aberrant center. Then, the following steps were performed for each examinee:

Simulate the true ability parameter (θ) from the $N (0, 1)$ distribution;

If the examinee is not an aberrant examinee, then simulate the item scores from the Rasch model using the above-mentioned true item parameters and θ simulated in the above step;

If the examinee is an aberrant examinee, then simulate the item scores on the items that are not in S from the Rasch model using the above-mentioned true item parameters and θ simulated above, but simulate the item scores on the items that are in S as draws of a Bernoulli random variable with a success probability of .9 (rather than being simulated from the Rasch model);

Compute three WLEs of the ability of the examinee, one each from the simulated item scores on s = S, the simulated item scores on $\bar{s}$ , and the simulated item scores on the whole test, using the Newton–Raphson algorithm; and

Compute each test statistic for the examinee. Use the WLEs computed in the above step to compute L_s and R_s.

For L_s and R_s, the Type I error rates were computed at levels of .001, .01, and .05 as the proportion of the values of the corresponding statistic among the nonaberrant examinees that were larger than the corresponding critical values (3.09, 2.33, and 1.64, respectively) from the standard normal distribution. The Type I error rate of the h statistic was computed as the proportion of the nonaberrant examinees that were flagged by algorithm 2 of Belov (2013). The standard error (SE) corresponding to the Type I error is about .0003 when level .01 and about .0007 when level .05.⁷

For L_s and R_s, the power was computed as the proportion of the values of the corresponding statistic among the aberrant examinees that were larger than the corresponding critical values from the standard normal distribution. The power of the h statistic was computed as the proportion of the aberrant examinees that were flagged by algorithm 2 of Belov (2013). The SE corresponding to the power is smaller than .015.⁸

The value of L was negative and close to zero for a very few examinees in the simulations. While L should be positive in theory, negative values may occur for small number of items or sampling fluctuations; the values of the other statistics indicate that the corresponding examinees performed very similarly on compromised and noncompromised items. For such examinees, L_s was set equal to 0.

Results on Type I Error Rates

The Type I error rates hardly varied over the simulation conditions (governed by the percentage of aberrant assessment centers and examinees) when the size of S is fixed—so they were pooled together for each size of S. Table 3 displays the pooled Type I error rates of the three statistics for levels of .001, .01, and .05 for each size of S. While the Type I error rates of h are considerably smaller than the nominal level in all conditions, those of L_s and R_s are close to the nominal level. The Type I error rates of L_s are occasionally slightly larger than the nominal level, for example, for 10 compromised items, but are satisfactory in all cases according to Cochran’s (1952) criterion for robustness (p. 394) that deems Type I error rates smaller than .06, .015, and .0015 to be satisfactory at levels .05, .01, and .001, respectively; also, the Type I error rates of L_s and R_s become closer to the nominal level as the size of S increases, which provide evidence that their null distribution converges to the standard normal distribution as the number of items in S increases.

Table 3.

Type I Error Rates for Nonadaptive Assessments.

Size of S	Statistic	Level .001	Level .01	Level .05
10	h	.000	.001	.008
	R_s	.000	.007	.047
	L_s	.001	.012	.055
20	h	.000	.002	.008
	R_s	.001	.008	.049
	L_s	.001	.011	.053
30	h	.000	.001	.009
	R_s	.001	.009	.049
	L_s	.001	.011	.051

Results on Power

Table 4 presents the values of power for different simulation conditions for levels of .001, .01, and .05. The first column (with title “Size of S”) denotes the number of items in S and the second column (with title “Statistic”) denotes the statistic. Columns 3 through 5, with title “10%, 10%” with “.001, .01, .05” below it, show the power for 10% aberrant assessment centers and 10% aberrant examinees at levels of .001, .01, and .05, respectively; columns 6 through 8, with title “10%, 20%” with “.001, .01, .05” below it, show the power for 10% aberrant assessment centers and 20% aberrant examinees for the three levels; columns 9 through 11, with title “20%, 10%” with “.001, .01, .05” below it, show the power for 20% aberrant assessment centers and 10% aberrant examinees for the three levels; and columns 12–14, with title “20%, 20%” with “.001, .01, .05” below it, show the power for 20% aberrant assessment centers and 20% aberrant examinees for the three levels.

Table 4.

Power for Nonadaptive Assessments.

Size of S	Statistic	10%, 10%			10%, 20%			20%, 10%			20%, 20%
Size of S	Statistic	.001	.01	.05	.001	.01	.05	.001	.01	.05	.001	.01	.05
10	h	.11	.31	.50	.12	.36	.61	.10	.27	.42	.10	.30	.48
	R_s	.09	.38	.70	.10	.41	.71	.10	.39	.70	.10	.40	.70
	L_s	.21	.48	.72	.22	.51	.75	.19	.49	.73	.21	.49	.72
20	h	.40	.66	.82	.42	.69	.86	.29	.56	.69	.29	.59	.76
	R_s	.38	.70	.88	.41	.72	.91	.39	.71	.89	.39	.71	.90
	L_s	.48	.74	.89	.50	.77	.91	.50	.76	.90	.50	.76	.91
30	h	.61	.82	.93	.60	.81	.92	.41	.69	.79	.40	.72	.84
	R_s	.61	.86	.95	.60	.84	.95	.61	.84	.95	.59	.84	.95
	L_s	.68	.89	.96	.66	.86	.95	.69	.87	.96	.68	.87	.95

Note. Columns 3–5 show the power for 10% aberrant assessment centers and 10% aberrant examinees at levels of .001, .01, and .05, respectively; columns 6–8 show the power for 10% aberrant assessment centers and 20% aberrant examinees for the three levels; columns 9–11 show the power for 20% aberrant assessment centers and 10% aberrant examinees for the three levels; columns 12–14 show the power for 20% aberrant assessment centers and 20% aberrant examinees for the three levels.

The table shows that h is the least powerful among the three statistics and L_s is the most powerful. The power of L_s is often substantially larger than that of h (e.g., .73 vs. .42 for 10 compromised items, 20% aberrant centers, and 10% aberrant examinees). The power of R_s is quite close to that of L_s in all cases. The power of each statistic increases as S becomes larger.

The Type I error rate and power of the $l_{z}^{*}$ statistic (Snijders, 2001), which is a popular person-fit statistic, were also computed in the above simulations. Person-fit statistics have been used to detect item preknowledge (see, e.g., a discussion on this issue in Eckerly, 2017). But the power of $l_{z}^{*}$ was much smaller compared to the statistics considered here. For example, for 20 compromised items, 20% aberrant centers, and 20% aberrant examinees, the power of $l_{z}^{*}$ was .17 at the level of .05 (whereas that of all the three statistics in Table 4 is 0.76 or larger). Therefore, $l_{z}^{*}$ is not considered any further in this article.

A Simulation Study Involving a CAT

A detailed simulation study, somewhat similar to that in Belov (2013), was performed to compare the Type I error rates and power of the h statistic to those of the new statistics for CATs.

Design of the Simulation

The simulation study involved a 50-item CAT. The 3PLM was used as the IRT model. As in Belov (2013), an item pool of 500 items was used in all simulations. The true item parameters of the items of the pool were simulated. The true a_i or slope parameters of the items in the pool were simulated from an $N (mean = 0.75, variance = 0.06)$ distribution (and then truncated above 0), the true b_i or difficulty parameters were simulated from an $N (mean = 0.49, variance = 1.27)$ distribution, and the true guessing parameters were simulated from an $N (mean = 0.17, variance = 0.01)$ distribution. These means and variances match those of the item pool in Belov (2013, p. 157). The item parameters were assumed known and equal to the simulated true values during the computation of the statistics for detecting item preknowledge. The true abilities of the nonaberrant and aberrant examinees were simulated from the $N (0, 1)$ distribution. Each item of the CAT was selected as the one that has maximum information at the posterior mean (computed using a uniform prior and over a grid of 41 points between −5 and 5) estimated from the scores on the previous items; the posterior mean was assumed to be 0 at the beginning of the CAT. As in Belov, no exposure control method was used. A total of 1,000 assessment centers was used with 100 examinees in each of them. The size of S (or, the number of compromised items) was 4, 12, 20, or 30. The percentage of aberrant assessment centers was 10 or 20; note that any value larger than 20 would clearly lead to a smaller power of the h statistic (as described earlier); however, no such value was used. Thus, the simulation cases considered here are the best case scenarios for the h statistic. The percentage of aberrant examinees in the aberrant assessment centers was 10 or 20; the percentage of aberrant examinees in the nonaberrant assessment centers was 0.

Computation

The steps in the simulation were to:

Simulate a CAT without item preknowledge for the 1,000 centers with 100 examinees each; all the scores are generated from the 3PLM,

Compute for each item the exposure rate, which is the percentage of examinees who received the item,

Pick the 4, 12, 20, or 30 items from the item pool that have the largest exposure rate, and let these items constitute the set S,

Randomly select 10% or 20% aberrant assessment centers and randomly select 10% or 20% aberrant examinees in each aberrant assessment center,

Simulate CAT only for the aberrant assessment centers and aberrant examinees under the assumption that the set S determined in Step 3 is compromised; to do this, for each aberrant examinee:

Simulate a CAT as above (using the 3PLM), except that when the examinee receives any item in S, simulate the score on the item as a draw of a Bernoulli random variable with a success probability of .9 (rather than simulating the score from the 3PLM) and

Let the scores of these aberrant examinees replace the scores of the same number of nonaberrant examinees generated in the first step (to ensure that each assessment center ends up having the same number of examinees).

Compute three WLEs of the ability of the examinee, one each from s, $\bar{s}$ , and all items, and

Compute each test statistic for each examinee. Use the WLEs computed in the above step to compute L_s and R_s.

The average size of s over the examinees was 3.3, 8.8, 12.2, and 14.3, respectively, when S included 4, 12, 20, and 30 items, respectively. Note that the above design was used to replicate the simulations of Belov (2013). In limited simulations under a few other designs of assigning compromised items to aberrant students, the Type I error rates of the statistics were very similar and power of the statistics were different, but the relative performance of the statistics was very similar.

A Fortran 90 computer program written by the authors was used for all the computations including the computation of the ability estimates. The WLEs were computed using the Newton–Raphson algorithm.

For L_s and R_s, the Type I error rate was computed as the proportion of the values of the corresponding statistic among the nonaberrant examinees that were larger than the corresponding critical values from the standard normal distribution. The Type I error rate of the h statistic was computed as the proportion of the nonaberrant examinees that were flagged by algorithm 2 of Belov (2013).

For L_s and R_s, the power was computed as the proportion of the value of the corresponding statistic among the aberrant examinees that were larger than the critical values from the standard normal distribution. The power of the h statistic was computed as the proportion of the aberrant examinees that were flagged by algorithm 2 of Belov (2013).

Results on Type I Error Rates and Power

The Type I error rates hardly varied over the simulation conditions (governed by the percentage of aberrant assessment centers and examinees) when the number of aberrant items is fixed—so they were pooled together. Table 5 displays the pooled Type I error rates of the three statistics for levels of .001, .01, and .05 for 4, 12, and 20 aberrant items—the rates for 30 aberrant items were very close to those for 20 aberrant items and are not shown. While the Type I error rates of h are considerably smaller than the nominal level in all conditions, those of L_s and R_s are very close to the nominal level. The Type I error rates of R_s are slightly smaller than those of L_s.

Table 5.

Type I Error Rates for CAT.

Size of S	Statistic	Level .001	Level .01	Level .05
4	h	.000	.002	.008
	R_s	.001	.005	.035
	L_s	.001	.006	.041
12	h	.000	.002	.009
	R_s	.001	.008	.046
	L_s	.001	.01	.049
20	h	.000	.002	.008
	R_s	.001	.009	.048
	L_s	.001	.010	.051

Note. CAT = computerized adaptive test.

Table 6 displays the Type I error rates at three levels of significance when the size of s is 1–4 in the case of 4 aberrant items. The percentage of examinees for each size of s is also provided in the second column of the table. The table shows that L_s and R_s become more conservative as the size of s decreases. For example, at level .05, the Type I error rate of R_s was .050 when the size of s is 4 but only .001 when the size of s is 1; one source of this conservativeness is that the average SE of the WLE is 1.37, 0.99, 0.97, and 0.91 when the size of s is 1, 2, 3, and 4, respectively. While these numbers indicate that L_s and R_s will suffer from small power for very small sizes of s, they also indicate that L_s and R_s will not incorrectly flag too many nonaberrant examinees in these cases.

Table 6.

Type I Error Rates of L_s and R_s for CATs With 4 Aberrant Items.

Size of s	%	R_s			L_s
Size of s	%	α = .001	α = .01	α = .05	α = .001	α = .01	α = .05
1	9	.000	.000	.001	.000	.000	.000
2	12	.000	.001	.008	.000	.000	.002
3	15	.002	.005	.049	.001	.002	.049
4	64	.001	.009	.050	.002	.013	.059

Note. Columns 3–5 provide the Type I error rates of R_s and columns 6–8 provide the Type I error rates of L_s, at significance levels (α) of .001, .01, and .05. CAT = computerized adaptive test.

Table 7, which looks very similar to Table 4, presents the power values for different simulation conditions for levels of .001, .01, and .05. The table shows that the power of L_s is the largest followed by that of R_s and then that of the h statistic. The power of L_s is often substantially larger than that of h (e.g., .76 vs. .55 for 12 compromised items, 20% aberrant centers, and 10% aberrant examinees at level .05). The power of each statistic increases as the number of compromised items increases because it is easier to detect item preknowledge when the number of compromised items is larger.

Table 7.

Power for CAT.

Size of S	Statistic	10%, 10%			10%, 20%			20%, 10%			20%, 20%
Size of S	Statistic	.001	.01	.05	.001	.01	.05	.001	.01	.05	.001	.01	.05
4	h	.04	.11	.19	.08	.21	.36	.05	.11	.18	.07	.20	.32
	R_s	.02	.13	.32	.07	.21	.40	.04	.18	.41	.12	.25	.45
	L_s	.04	.17	.36	.07	.24	.43	.05	.21	.42	.11	.27	.48
12	h	.33	.51	.69	.23	.41	.58	.24	.42	.55	.22	.43	.58
	L_s	.28	.53	.75	.21	.41	.61	.31	.54	.74	.29	.52	.71
	R_s	.39	.60	.77	.27	.48	.64	.39	.59	.76	.36	.57	.73
20	h	.38	.55	.70	.39	.59	.76	.24	.43	.58	.25	.47	.63
	L_s	.42	.62	.78	.44	.66	.82	.41	.61	.76	.40	.60	.78
	R_s	.48	.66	.81	.51	.71	.84	.46	.65	.78	.46	.65	.80
30	h	.27	.48	.67	.32	.51	.71	.19	.36	.52	.25	.53	.70
	L_s	.50	.70	.84	.49	.71	.86	.43	.66	.82	.57	.77	.89
	R_s	.54	.74	.85	.55	.74	.87	.51	.72	.83	.64	.81	.90

The power of h in Table 7 is often smaller than that in Belov (2013). That is primarily because the true abilities of the aberrant examinees were simulated in Belov from a uniform distribution between −3 and 0 but from a standard normal distribution here.⁹

Discussion on the Type I Error Rates and Power

The above simulations show that the h statistic has the smallest Type I error rates among the three statistics considered here, but it also has the smallest power, often by a large margin. Combined with the practical limitations of the h statistic mentioned earlier, they provide strong evidence that the L_s and R_s statistics should be preferred over the h statistic.

Further, note that the above simulations, with 20% or fewer aberrant assessment centers, represented the best case scenarios for the h statistic. When the number of aberrant items is large, the number of aberrant assessment centers is likely to be large as well; in that case, the choice of not flagging any examinee in 80% assessment centers in algorithm 2 of Belov (2013) would lead to the low power of h. To examine this issue further, some limited simulations were performed with 12 aberrant items, 20% aberrant assessment centers with 10% aberrant examinees each and 10% more aberrant assessment centers with 5% aberrant examinees each; the power of the h statistic was much smaller than that of the L_s and R_s because the h statistic failed to detect any of the aberrant examinees in the assessment centers with 5% aberrant examinees each.

Conclusions

Standard 6.6 of the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, & National Council for Measurement in Education, 2014, p. 137) includes the recommendation that when the assessment results have important consequences, score integrity should be supported through active efforts to prevent, detect, and correct scores obtained by fraudulent or deceptive means. As a consequence, there is a growing interest in detection of test fraud. This article suggests two new statistics for detecting individuals who may have benefited from item preknowledge on nonadaptive and adaptive tests that may include a mix of dichotomous and polytomous items. The suggested statistics are first, according to the authors’ knowledge, that apply to assessments that include some polytomous items. In simulation studies, for both nonadaptive and adaptive assessments, the Type I error rate of the suggested statistics were found close to the nominal level and the values of power were found considerably larger than that of a statistic based on the Kullback–Leibler divergence (Belov, 2013) on average.

Statistical indices for detecting item preknowledge are useful for providing confirming evidence of inappropriate behavior when evidence from other sources also exist, but the evidence provided by statistical indices is insufficient by itself. For example, Hanson, Harris, and Brennan (1994) commented that no statistical method on its own can provide conclusive proof that copying occurred (p. 25); the comment is true about item preknowledge as well. Researchers such as Tendeiro and Meijer (2014, p. 257) recommended complementing statistical indices of detecting irregularities with other sources of information such as seating charts, video surveillance, or follow-up interviews.

There are several limitations of this article and, consequently, several related topics can be further investigated. First, while our simulation study was detailed, it is possible to perform more simulations, possibly with polytomous items, different true item parameters, different test lengths, different IRT models, and different types of preknowledge. Second, the behavior of the suggested statistics can be further examined for a small number of compromised or noncompromised items. The asymptotic null distribution of these statistics is expected to hold only when the numbers of compromised items and noncompromised items administered to an examinee are large; if one of these numbers is small, a simulation-based null distribution should be used, or one should not use any of these statistics. Third, the item parameters were treated as known in this article. This assumption is reasonable especially in the context of CATs where item parameters are assumed known and the ability parameters are estimated. However, it is possible to estimate item parameters and consider the effect of the estimation on the properties of the indices considered here. Fourth, this article considers the case when the investigator precisely knows which items are compromised; in reality, there may be uncertainty with the identity of these items; in the presence of uncertainty, the suggested statistics can still be used, but they would lead to smaller power than that found in the simulations here. More research is possible on this issue. Fifth, the Type I error rate and power of the statistics suggested here may be compared to the Bayesian index suggested by McLeod et al. (2003) in future research. The latter does not involve the assumption that the set of aberrant items is known—so it is expected to lead to smaller power (some limited simulations confirmed this) than the statistics in this article. Sixth, the article by Belov (2016) was published at a later stage of this article—a comparison of L_s and R_s with the best performing statistics in Belov is a topic of future research. Finally, it would be interesting to combine the statistics here with the methods of van der Linden and van Krimpen-Stoop (2003) and Qian et al. (2016) that use response times to detect item preknowledge.

Footnotes

Acknowledgments

The author would like to thank the editor Daniel McCaffrey and the three anonymous reviewers for several helpful comments that led to a significant improvement of the article. The author would also like to thank James A. Wollack for sharing a data set that was used in this article.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

American Educational Research Association, American Psychological Association, & National Council for Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

Baker

F. B.

Kim

H. S.

(2004). Item response theory: Parameter estimation techniques (2nd ed.). New York, NY: Marcel Dekker.

Belov

D. I.

(2013). Detection of test collusion via Kullback-Leibler divergence. Journal of Educational Measurement, 50, 141–163.

Belov

D. I.

(2016). Comparing the performance of eight item preknowledge detection statistics. Applied Psychological Measurement, 40, 83–97.

Biehler

Holling

Doebler

(2015). Saddlepoint approximations of the distribution of the person parameter in the two parameter logistic model. Psychometrika, 80, 665–688.

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 397–479). Reading, MA: Addison-Wesley.

Casella

Berger

R. L.

(2002). Statistical inference (2nd Ed.). Pacific Grove, CA: Duxbury.

Cizek

G. J.

Wollack

J. A.

(2017). Handbook of detecting cheating on tests. Washington, DC: Routledge.

Cochran

W. G.

(1952). The χ² test of goodness of fit. Annals of Mathematical Statistics, 23, 315–345.

10.

Cox

D. R.

(2006). Principles of statistical inference. New York, NY: Cambridge University Press.

11.

Cox

D. R.

Hinkley

D. V.

(1974). Theoretical statistics. London, England: Chapman and Hall.

12.

Drasgow

Levine

M. V.

McLaughlin

M. E.

(1987). Detecting inappropriate test scores with optimal and practical appropriateness indices. Applied Psychological Measurement, 11, 59–79.

13.

Eckerly

C. A.

(2017). Detecting item preknowledge and item compromise: Understanding the status quo. In Cizek

G. J.

Wollack

J. A.

(Eds.), Handbook of detecting cheating on tests. Washington, DC: Routledge.

14.

Eckerly

C. A.

Babcock

B. G.

Wollack

J. A.

(2015, 4). Preknowledge detection using a scale-purified deterministic gated IRT model. Paper presented at the annual meeting of the National Council of Measurement in Education, Chicago, IL.

15.

Finkelman

Weiss

D. J.

Kim-Kang

(2010). Item selection and hypothesis testing for the adaptive measurement of change. Applied Psychological Measurement, 34, 238–254.

16.

Glas

C. A. W.

Dagohoy

A. V. T.

(2007). A person fit test for IRT models for polytomous items. Psychometrika, 72, 159–180.

17.

Guo

Drasgow

(2010). Identifying cheating on unproctored internet tests: The z-test and the likelihood ratio test. International Journal of Selection and Assessment, 18, 351–364.

18.

Hanson

B. A.

Harris

D. J.

Brennan

R. L.

(1994). A comparison of several statistical methods for examining allegations of copying (ACT research report series no. 87-15). Iowa City, IA: American College Testing.

19.

Kingston

Clark

(2014). Test fraud: Statistical detection and methodology. New York, NY: Routledge.

20.

Klauer

K. C.

Rettig

(1990). An approximately standardized person test for assessing consistency with a latent trait model. British Journal of Mathematical and Statistical Psychology, 43, 193–206.

21.

Kullback

Leibler

R. A.

(1951). On information and sufficiency. The Annals of Mathematical Statistics, 22, 79–86.

22.

Kyle

(2002). Cheating scandal rocks GRE, ETS. Retrieved May 12, 2016, from http://thedartmouth.com/2002/08/09/cheating-scandal-rocks-gre-ets/

23.

Lehmann

E. L.

Romano

J. P.

(2005). Testing statistical hypotheses (3rd ed.). New York, NY: Springer-Verlag.

24.

Magis

Verhelst

(2014, 11). On the finiteness and uniqueness of the weighted likelihood estimator of ability in polytomous IRT models. Research Center for Examination and Certification Workshop on IRT and Educational Measurement, University of Twente, the Netherlands.

25.

McClintock

J. C.

(2015). Erasure analyses: Reducing the number of false positives. Applied Measurement in Education, 28, 14–32.

26.

McLeod

L. D.

Lewis

Thissen

(2003). A Bayesian method for the detection of item preknowledge in computerized adaptive testing. Applied Psychological Measurement, 27, 121–137.

27.

Mislevy

R. J.

Chang

H. H.

(2000). Does adaptive testing violate local independence? Psychometrika, 65, 149–156.

28.

Muraki

(1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159–176.

29.

Qian

Staniewska

Reckase

Woo

(2016). Using response time to detect item preknowledge in computer-based licensure examinations. Educational Measurement: Issues and Practice, 35, 38–47.

30.

Rao

C. R.

(1973). Linear statistical inference and its applications (2nd Ed.). New York, NY: John Wiley.

31.

Romero

Riascos

Jara

(2015). On the optimality of answer-copying indices: Theory and practice. Journal of Educational and Behavioral Statistics, 40, 435–453.

32.

Segall

D. O.

(2002). An item response model for characterizing test compromise. Journal of Educational and Behavioral Statistics, 27, 163–179.

33.

Shu

Henson

Luecht

(2013). Using deterministic, gated item response theory model to detect test cheating due to item compromise. Psychometrika, 78, 481–497.

34.

Sinharay

(2016). Asymptotic corrections of standardized extended caution indices. Applied Psychological Measurement, 40, 418–433.

35.

Snijders

(2001). Asymptotic distribution of person-fit statistics with estimated person parameter. Psychometrika, 66, 331–342.

36.

Tendeiro

J. N.

Meijer

R. R.

(2014). Detection of invalid test scores: The usefulness of simple nonparametric statistics. Journal of Educational Measurement, 51, 239–259.

37.

van der Linden

W. J.

(2006). A lognormal model for response times on test forms. Journal of Educational and Behavioral Statistics, 31, 181–204.

38.

van der Linden

W. J.

(2009). A bivariate lognormal response-time model for the detection of collusion between test takers. Journal of Educational and Behavioral Statistics, 34, 378–394.

39.

van der Linden

W. J.

Jeon

(2012). Modeling answer changes on test items. Journal of Educational and Behavioral Statistics, 37, 180–199.

40.

van der Linden

W. J.

Lewis

(2015). Bayesian checks on cheating on tests. Psychometrika, 80, 689–706.

41.

van der Linden

W. J.

van Krimpen-Stoop

E. M. L. A.

(2003). Using response times to detect aberrant responses in computerized adaptive testing. Psychometrika, 68, 251–265.

42.

Veerkamp

W. J. J.

Glas

C. A. W.

(2000). Detection of known items in adaptive testing with a statistical quality control method. Journal of Educational and Behavioral Statistics, 25, 373–389.

43.

Warm

T. A.

(1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54, 427–450.

44.

Wollack

J. A.

Fremer

J. J.

(2013). Handbook of test security. New York, NY: Routledge.

Detection of Item Preknowledge Using Likelihood Ratio Test and Score Test

Abstract

Keywords

Background

The Research Problem

Review of a Statistic Based on the Kullback–Leibler Divergence

Review of Other Methods of Detecting Item Compromise

Two New Statistics Based on the LRT and the Score Test

The rationale behind the new statistics

A Statistic Based on the LRT

A Statistic Based on the Score Test

The Advantage of Ls and Rs Over h

A Real Data Application

A Simulation Study Involving a Nonadaptive Assessment

Design of the Simulation

Results on Type I Error Rates

Results on Power

A Simulation Study Involving a CAT

Design of the Simulation

Computation

Results on Type I Error Rates and Power

Discussion on the Type I Error Rates and Power

Conclusions

Footnotes

Acknowledgments

Declaration of Conflicting Interests

Funding

Notes

References

The Advantage of L_s and R_s Over h