Bayesian Change-Point Analysis Approach to Detecting Aberrant Test-Taking Behavior Using Response Times

Abstract

Change-point analysis (CPA) is a method for detecting abrupt changes in parameter(s) underlying a sequence of random variables. It has been applied to detect examinees’ aberrant test-taking behavior by identifying abrupt test performance change. Previous studies utilized maximum likelihood estimations of ability parameters, focusing on detecting one change point for each examinee. This article proposes a Bayesian CPA procedure using response times (RTs) to detect abrupt changes in examinee speed, which may be related to aberrant responding behaviors. The lognormal RT model is used to derive a procedure for detecting aberrant RT patterns. The method takes the numbers and locations of the change points as parameters in the model to detect multiple change points or multiple aberrant behaviors. Given the change points, the corresponding speed of each segment in the test can be estimated, which enables more accurate inferences about aberrant behaviors. Simulation study results indicate that the proposed procedure can effectively detect simulated aberrant behaviors and estimate change points accurately. The method is applied to data from a high-stakes computerized adaptive test, where its applicability is demonstrated.

Keywords

aberrant behavior Bayesian change-point analysis response times computerized testing

When test results are used for high-stakes decisions, such as tracking students’ growth trajectory, career planning, and other professional decisions, test security is an important validity issue. With the rapid development of computer technology, many standardized tests are computer-based testing (CBT) due to its operational advantages. Most importantly, CBT enhances test security given data from multiple sources, such as item responses and response times (RTs) can be collected and utilized for aberrant responding behavior detection.

Item responses in a test provide information for the estimation of latent ability underlying the responses, while RTs provide information about the working speed of the respondent. In the lognormal RT model proposed by van der Linden (2006), it is assumed that the working speed of an examinee is constant throughout the test. However, in many cases, the working speed may vary during a test. One scenario is related to different response strategies that an examinee might use (Fox & Marianti, 2016). Another scenario is when examinees show aberrant test-taking behaviors due to, for example, the warm-up effect, test speededness, a decline in motivation, or preknowledge of some items (Bradlow et al., 1998; Cheng & Shao, 2022; Marianti et al., 2014). Aberrant responding behaviors not only lead to errors in item parameter estimation but also to inaccurate estimations of ability, which ultimately lead to invalid inferences about students’ abilities (Douglas et al., 1998).

The working speed of an examinee is auxiliary information for a better understanding of examinees’ response behavior. In other words, RTs may provide important information for the detection of aberrant behavior as unexpected patterns in RTs may indicate certain types of aberrant responding behaviors. Specifically, if examinees have speededness or preknowledge of some items, they are likely to respond more quickly on these items than on others (van der Linden & van Krimpen-Stoop, 2003). On the other hand, examinees affected by the warm-up effect are likely to respond more slowly at the beginning of a test (Shao, 2017). In addition, RTs could be used to improve test design, item selection in computerized adaptive tests, item calibration, and cognitive diagnosis (Zhan et al., 2018).

There are two common approaches to dealing with aberrant responding behaviors. One approach is to model the aberrant responses using mixture models. For example, Bolt et al. (2002) used a two-class mixture Rasch model to classify examinees into speeded and nonspeeded classes. Other researchers (e.g., Meyer, 2010; Schnipke & Scrams, 1997; Wang & Xu, 2015; Wise & DeMars, 2006) proposed mixture models to differentiate solution behaviors from rapid guessing behaviors. The other approach is to detect aberrant behaviors. Previous research has mainly developed person-fit statistics (PFSs) and change-point analysis (CPA) methods. PFSs provide a numerical measure of how well a person’s response pattern fits an item response theory (IRT) model, which allows the detection of aberrant response patterns (Embretson & Reise, 2000; Meijer, 2002; Meijer & Sijtsma, 2001). However, PFSs only show the model-data fit issue, and they do not provide more detailed information to distinguish between the normal and aberrant responses. Moreover, the calculation of PFSs requires examinees’ ability parameters that are known, and for examinees with aberrant behaviors, their ability estimations can also be seriously biased (Shao et al., 2016; Sinharay, 2016).

CPA is a method from the field of statistical quality control (e.g. Allalouf, 2007; Montgomery, 2013; von Davier, 2012) for detecting any change in the parameter(s) underlying a sequence of random variables. It has been used in many studies to detect aberrant responding behaviors by identifying an abrupt change in an examinee’s performance (Cheng & Shao, 2022; Choe et al., 2018; Shao et al., 2016; Sinharay, 2016; Zhang, 2014; Zhang & Li, 2016). A unique advantage of CPA is that it provides an estimate of the exact location (change point) at which an aberrant behavior occurs. This information is very useful. First, it helps test designers to develop tests in a more rational way. Second, the change points can tell us which responses are aberrant, so that the data excluding the aberrant item responses can be used to ensure the accuracy of parameter estimation.

Some areas remained untapped in previous studies on CPA. First, these studies considered and detected one change point only. However, it is possible for examinees to engage in both a warm-up effect and speededness, associated with two change points. Sinharay (2016) found that in some simulation conditions with more than one true change point, only one true change point was able to be detected, whereas the others could not be detected. Second, the maximum likelihood estimation (MLE) method of the examinee’s ability is often used in CPA. However, with fewer items, the estimated ability parameter based on the MLE method is less accurate and will thus influence the change point detection results. To increase the power of the test, Sinharay (2016) restricted the change point to the middle range of the items. Shao et al. (2016) also found that as the starting point of speeded responses gets closer to the end of the test, both the classification accuracy rate and power drop. Third, the detection results may be affected by some item characteristics. Shao et al. (2016) found that when items on a test are ordered from easy to difficult, the increasing difficulty parameters of items may be confounded with test speededness.

This article proposes a Bayesian CPA method using RTs to detect aberrant test-taking behaviors when there is more than one change point. The following sections first introduce current CPA methods for detecting aberrant behaviors. Next, the proposed Bayesian change-point detection procedure is presented, and simulation studies are conducted to investigate the effectiveness of the proposed approach. Real data analysis is then used to demonstrate its applicability. Finally, the limitations of the current study are addressed and future lines of research are suggested.

CPA for Aberrant Responding Behavior Detection

Various aberrant responding behaviors are often associated with an abrupt change in an examinee’s performance in taking a test. Zhang (2014) proposed a real-time sequential item monitoring procedure based on CPA to detect compromised items in computerized adaptive testing (CAT). Having compromised items in an item bank may cause substantial bias and root mean square error in ability estimates, jeopardizing the validity and fairness of the test (Liu et al., 2019; Zhang et al., 2012). Further, Zhang and Li (2016) proposed a sequential test method based on an IRT model. In this method, the MLE of the ability parameter is used. If there is a large difference between the expected and observed probability of a correct response, the item is likely to have been leaked. Choe et al. (2018) also proposed a change-point detection method in combination with RT data for detecting compromised items. Shao et al. (2016) proposed a likelihood ratio test statistic based on CPA to detect test speededness. This procedure can both identify speeded examinees and locate the point at which an examinee started to speed. The ability estimation was improved by removing suspected speeded responses rather than using the entire sets of responses consisting of those from examinees suspected of test speededness. Sinharay (2016) suggested three new PFSs based on tests for a change point to detect an abrupt change in an examinee’s ability related to aberrant behaviors in CAT. Among the three new PFSs, the power of the one based on the Wald test (Rao, 1973) was the largest. More recently, Cheng and Shao (2022) proposed a CPA procedure based on RTs to detect test speededness by detecting an increase in the working speed.

Unlike previous studies, we introduce a Bayesian approach to CPA (e.g., Chen & Gupta, 2012; Inclán, 1993; Smith, 1975) for two reasons. First, it does not rely on the asymptotic assumptions about test statistics in frequentist methods. This assumption can be problematic in situations, where the parametric models are restricted to a finite, possibly small interval of time (a small number of items). Second, a Bayesian approach allows the quantification of uncertainty in both the number and the location of the change points, which makes it possible to detect multiple change points and multiple aberrant behaviors simultaneously (Chopin, 2007; Ruggieri & Antonellis, 2016).

Further, we propose to analyze item RTs in the Bayesian CPA method. In general, RTs are continuous rather than binary, which can offer more information on aberrances (van der Linden & Guo, 2008). Van der Linden and van Krimpen-Stoop (2003) proposed using a lognormal RT model to detect preknowledge and speededness in CAT. They found that the methods using response data had nearly no power, but that Bayesian approach with RTs almost doubled detection rates relative to the classical methods based on item responses. Qian et al. (2016) applied the residual of the lognormal RT model to detect possible item preknowledge in computer-based licensure exams. Choe et al. (2018) found that, compared to the analysis of responses alone, preknowledge detection based on RTs can provide evident improvements in detection power with fewer false positives. Cheng and Shao (2022) also found that the power of CPA procedure to detect test speededness using RTs is improved compared with using dichotomous item response data.

In this article, the proposed Bayesian CPA method detects aberrant test-taking behaviors that are associated with abrupt changes in the examinee’s speed. The number and location of the change points are model parameters that are inferred based on their posterior distributions. The speed of the examinee in each segment of the test partitioned by the change point is also estimated. Using these detection results, the types of responding behaviors can be inferred more comprehensively with greater accuracy.

Method

The Model

Suppose the lognormal RT model (van der Linden, 2006) fits the regular RTs from examinees. Consider examinees $p (p = 1, \dots, N)$ and items $i (i = 1, \dots, n)$ in a test. The observed RT is $t_{i p}$ , and the model is a lognormal density for the distribution of the RT $t_{i p}$ as presented in the following:

f (t_{i p}; τ_{p}, α_{i}, β_{i}) = \frac{α_{i}}{t_{i p} \sqrt{2 π}} exp {- \frac{1}{2} {[α_{i} (ln t_{i p} - (β_{i} - τ_{p}))]}^{2}},

where parameter $τ_{p} \in (- \infty, + \infty)$ represents the speed at which examinee p operates during the test. $β_{i} \in (- \infty, + \infty)$ is a parameter for the time intensity of item i, related to the amount of time required from the examinees. $α_{i} \in (0, + \infty)$ is the discrimination parameter of item i; a larger value of $α_{i}$ means less dispersion for the log RT distribution on item i for the examinees. The values of item parameters $α_{i}$ and $β_{i}$ can be estimated during item calibration. Accordingly, the values of $α_{i}$ and $β_{i}$ are assumed to have been estimated with known values (e.g., Marianti et al., 2014; van der Linden & Guo, 2008; van der Linden & van Krimpen-Stoop, 2003).

In Equation 1, the log RT follows a normal distribution:

ln t_{i p} \sim N (β_{i} - τ_{p}, α_{i}^{- 2}) .

The RT variables are independent between persons as well as between items given a person’s latent speed parameter under the local independence assumption (van der Linden, 2006). From Equation 2, we have

Y_{i p} = β_{i} - ln t_{i p} \sim N (τ_{p}, α_{i}^{- 2}) .

That is, for each examinee p, $Y_{i p}$ is normally distributed with unknown mean $τ_{p}$ and known variance $α_{i}^{- 2}$ . Because we test each examinee one by one, let ${Y_{i}}$ represents the observations of an examinee on items $i = 1, \dots, n$ . The RTs by the examinee on the items in the test will be checked to detect the existence of more than one abrupt change in speed:

τ_{i} = η_{0}, \dots 1 \leq i \leq k_{1},

= η_{1}, \dots k_{1} < i \leq k_{2},

⋮

= η_{q}, \dots k_{q} < i \leq n,

where $τ_{i}$ denotes the speed on item i of the examinee. Note that $τ_{1} = τ_{2} = \dots = τ_{n}$ if there is no change of speed. q is the unknown number of change points in the n observations. $1 \leq k_{1} < \dots < k_{q} < n$ are the unknown locations of the change points, respectively, which denote the indices of the observations before the speed changes. Specifically, k_j indicates that there is an abrupt change of speed at item $k_{j} + 1$ . $k_{1}, \dots, k_{q}$ partition the data set into $q + 1$ segments, and $η_{j} (j = 0, \dots, q)$ are the examinee’s speeds in the $q + 1$ segments, respectively. Note that the first item can be a change point (in the case where the examinee’s speed changes from the second item), but the last item cannot be a change point.

In theory, there are at most n − 1 change points for n observations. However, the change points to be detected here are not scattered in each item but appear as the last item of one stage, which means that the next stage starts with the next item. This is mainly caused by the warm-up effect, speededness, fatigue, decline in motivation, and so on—hence, the number of change points is small. In this study, at most two or three change points in dozens of observations are assumed. To illustrate how the procedure works, let the maximum number of change points for each examinee be m. Thus, $q = 0, 1, \dots, m$ . $q = 0$ means that there is no abrupt change in speed (no change point), and other values correspond to the number of change points.

Using the convention that $k_{0} = 0$ and $k_{q + 1} = n$ , let

Y = (Y_{1}, \dots, Y_{n}) ’, η = (η_{0}, \dots, η_{q}) ’, k = (k_{1}, \dots, k_{q}) ’,

Given local independence, the joint density for Y can be written as

f (Y | η, k, q) = \prod_{j = 0}^{q} \prod_{i = k_{j} + 1}^{k_{j + 1}} \frac{1}{\sqrt{2 {πα}_{i}^{- 2}}} exp (- \frac{{(Y_{i} - η_{j})}^{2}}{2 α_{i}^{- 2}}) .

Assume that r is the probability of observing a change point for an examinee in a test. This implies that the number of change points q can be assumed to follow a binomial (m, r) distribution for each examinee. That is,

f (q | r) = (\begin{matrix} m \\ q \end{matrix}) r^{q} {(1 - r)}^{m - q}, q = 0, \dots, m .

By selecting appropriate values of r, the number of change points of the examinees can meet certain prior information. Using Equation 7, the posterior probability of q can be computed to determine which value is better supported by the data.

Assume that $q + 1$ values of $η_{j}$ are conditionally independently drawn from the normal density and are independent of q and k:

f (η_{j} | μ, σ) = \frac{1}{\sqrt{2 {πσ}^{2}}} exp (- \frac{{(η_{j} - μ)}^{2}}{2 σ^{2}}), j = 0, \dots, q .

The three quantities r, $μ$ , and $σ$ are the hyperparameters that are not modeled. Given that there are n items and that the last item cannot be the change point, there are $(\begin{matrix} n - 1 \\ q \end{matrix})$ ways to place q change points, and the probabilities of all possible locations for the change points $k_{1} < \dots < k_{q}$ are equal:

f (k | q) = \frac{1}{(\begin{matrix} n - 1 \\ q \end{matrix})} .

Given the likelihood and the priors, the joint probability density function of $Y, η, and k$ is presented as follows:

f (Y, η, k | q) = f (Y | η, k, q) \times f (η | μ, σ^{2}) \times f (k | q) .

Integrating over each $η_{j}$ , given that $η_{j}^{'} s$ are independent, we get

f (Y, k | q) = \int \dots \int f (Y, η, k | q) d η = f (k | q) \times \prod_{j = 0}^{q} \int f (Y_{k_{j} + 1,} \dots, Y_{k_{j + 1}} | η_{j}, k, q) f (η_{j} | μ, σ^{2}) d η_{j},

= f (k | q) \times \prod_{j = 0}^{q} f (Y_{k_{j} + 1}, \dots, Y_{k_{j + 1}} | k, q),

= f (k | q) \times \prod_{j = 0}^{q} \frac{1}{2 π^{(k_{j + 1} - k_{j} + 1) / 2} {| \sum |}^{1 / 2}} exp (- {(Y - μ)}^{′} \sum^{- 1} (Y - μ) / 2),

where

\sum = [\begin{matrix} α_{k_{j} + 1}^{- 2} + σ^{2} & σ^{2} & \dots & σ^{2} \\ σ^{2} & α_{k_{j} + 2}^{- 2} + σ^{2} & \dots & ⋮ \\ ⋮ & ⋮ & ⋱ & σ^{2} \\ σ^{2} & \dots & σ^{2} & α_{k_{j + 1}}^{- 2} + σ^{2} \end{matrix}],

(Y - μ) = [\begin{matrix} Y_{k_{j} + 1} - μ \\ ⋮ \\ Y_{k_{j + 1}} - μ \end{matrix}] .

The posterior probability function $f (k | Y, q)$ , which provides the information about the location of the change points, is

f (k | Y, q) = \frac{f (Y, k | q)}{f (Y | q)},

where

f (Y | q) = \sum_{k_{1}} \dots \sum_{k_{q}} f (Y, k | q) .

The sums on the right side cover all the possible values of k: $k_{1} = 1, 2, \dots, n - q$ ; $k_{2} = k_{1} + 1, \dots, n - q + 1$ ; $\dots k_{q} = k_{q - 1} + 1, \dots, n - 1$ . The joint mode of k is the estimation of the location of the change points.

For assessing the evidence on the data about the number of change points q, we use the posterior distribution of q:

f (q | Y) \propto f (Y | q) f (q) .

The value of q is empirically determined by comparing the posterior probability of each value of q:

f (q = 0 | Y) \propto f (Y | q = 0) f (q = 0),

f (q = 1 | Y) \propto f (Y | q = 1) f (q = 1) = \sum_{k = 1}^{n - 1} f (Y, k | q = 1) f (q = 1)

⋮

f (q = m | Y) \propto f (Y | q = m) f (q = m),

= \sum_{k_{1} = 1}^{n - m} \sum_{k_{2} = k_{1} + 1}^{n - m + 1} \dots \sum_{k_{m} = k_{m - 1} + 1}^{n - 1} f (Y, k_{1}, \dots, k_{m} | q = m) f (q = m) .

The evidence that supports the number of change points detected is the one with the largest posterior probability, that is, the Bayesian estimate of q obtained by the posterior mode:

\overset{\land}{q} = max_{q = {0, \dots, m}} f (q | Y) .

Given the posterior estimate of q, we obtain the posterior estimate of k from Equation 12. For example, if the estimated number of change points is three, then we have:

f (k_{1}, k_{2}, k_{3} | Y, q = 3) = \frac{f (Y, k_{1}, k_{2}, k_{3} | q = 3)}{f (Y | q = 3)}, \overset{\land}{(k_{1}}, \overset{\land}{k_{2}}, \overset{\land}{k_{3}}) = mod e (k_{1}, k_{2}, k_{3}),

1 \leq k_{1} \leq \dots \leq n - 3, k_{1} + 1 \leq k_{2} \leq \dots \leq n - 2, k_{2} + 1 \leq k_{3} \leq \dots \leq n - 1.

Specification of the Hyperparameters

The hyperparameters need to be specified for r, $μ$ , and $σ$ . Those values should reflect what we assume about the process. r is the only parameter in the binomial distribution, and it represents the probability of observing a change point for an examinee during a test. As we have no prior knowledge of which examinees will have change points, we assume that the probability of normal answers for each examinee is the largest. Thus, r will be a small number (to maximize the probability of having zero change points for each examinee), and the larger the number of change points, the smaller the probability. For example, when m = 2, q follows a binomial (2, r) distribution, and the values of r should be chosen from (0,1/3), which satisfies p(q = 0) > p(q = 1) > p(q = 2). When m = 3, q follows a binomial (3, r) distribution, the values of r should be chosen from (0,0.25), which satisfies p(q = 0) > p(q = 1) > p(q = 2) > p(q = 3). Figure 1 shows the probability of different values of q as r changes for binomial (2, r) and binomial (3, r), respectively.

Figure 1.

Probability of different values of q as r changes for binomial (2, r) and binomial (3, r).

From one point of view, the prior of q is equivalent to adding a penalty term to the model. As the number of change points increases, the model becomes more complex, and the likelihood function usually increases. We can reduce the influence of this by adding certain prior knowledge (a penalty term) in order to have stronger evidence when choosing a more complicated model (lager number of change points). At this point, the posterior mode estimation of q is obtained by the product of the likelihood function and the penalty term.

The choice of $μ$ should be related to the location of $η_{j}$ , j = 0,…, q for each examinee. It can be equal to the overall location of examinees’ speed, which is usually set to zero (e.g., Marianti et al., 2014; Shao, 2017; van der Linden, 2006; van der Linden & Guo, 2008; van der Linden & van Krimpen-Stoop, 2003). Note that $μ$ is irrelevant to the change of speed; what matters for finding the change points is the relative speed at adjacent segments of items. $σ$ is the standard deviation (SD) of the prior distribution of $η_{j}$ , which presents the range of possible values for the speed. The greater the value of $σ$ , the wider the range of possible values for the speed. A relatively large value (1 or 2) can be chosen to make detection more efficient (under the premise of a reasonable false alarm rate).

Inference About $η$

The speed of each examinee at each stage of the test can also be inferred. Given the data and the modal value of the points of change $\hat{k}$ , the posterior distribution of $η$ can be written as

f (η | Y, \overset{\land}{k}, q) \propto f (Y | η, \overset{\land}{k}, q) f (η |, μ, σ)

\propto \prod_{j = 0}^{q} \frac{1}{\sqrt{2 π \frac{1}{2 A}}} exp (- (η_{j} - \frac{B}{A})^{2} / (2 \frac{1}{2 A})),

with

A = \sum_{i = {\overset{\land}{k}}_{j} + 1}^{{\overset{\land}{k}}_{j + 1}} \frac{1}{2 α_{i}^{- 2}} + \frac{1}{2 σ^{2}} and B = \sum_{i = {\overset{\land}{k}}_{j} + 1}^{{\overset{\land}{k}}_{j + 1}} \frac{Y_{i}}{2 α_{i}^{- 2}} + \frac{μ}{2 σ^{2}}

In this case, the posterior mean and SD of $η_{j}$ , j = 0,…, q can be obtained explicitly:

E (η_{j} | Y, \overset{\land}{k}, q) = \frac{B}{A} = \frac{\sum_{i = {\overset{\land}{k}}_{j} + 1}^{i = {\overset{\land}{k}}_{j + 1}} \frac{Y_{i}}{2 α_{i}^{- 2}} + \frac{μ}{2 σ^{2}}}{\sum_{i = {\overset{\land}{k}}_{j} + 1}^{i = {\overset{\land}{k}}_{j + 1}} \frac{1}{2 α_{i}^{- 2}} + \frac{1}{2 σ^{2}}},

var (η_{j} | Y, \overset{\land}{k}, q) = \frac{1}{2 A} = \frac{1}{2 (\sum_{i = {\overset{\land}{k}}_{j} + 1}^{i = {\overset{\land}{k}}_{j + 1}} \frac{1}{2 α_{i}^{- 2}} + \frac{1}{2 σ^{2}})} .

If we want to calculate the marginal posterior of $η$ , it is necessary to integrate out k, that is:

f (η | Y, q) = \sum_{k} f (η, k | Y, q) = \sum_{k_{1}} \dots \sum_{k_{q}} f (η | Y, k, q) f (k | Y, q) .

At this point, $η_{j}$ follows a mixture distribution, where the mass function is the posterior distribution of k. Then, the posterior mean and SD of $η_{j}$ can be obtained as follows:

E (η_{j} | Y, q) = \sum_{k_{1}} \dots \sum_{k_{q}} \frac{B ’}{A ’} f (k | Y, q),

var (η_{j} | Y, q) = \sum_{k_{1}} \dots \sum_{k_{q}} \frac{1}{2 A ’} {(f (k | Y, q))}^{2},

with

A ’ = \sum_{i = k_{j} + 1}^{k_{j + 1}} \frac{1}{2 α_{i}^{- 2}} + \frac{1}{2 σ^{2}} and B ’ = \sum_{i = k_{j} + 1}^{k_{j + 1}} \frac{Y_{i}}{2 α_{i}^{- 2}} + \frac{μ}{2 σ^{2}}

The moments are easy to evaluate but require many calculations, which can become unfeasible for lager values of q (Inclán, 1993). In this case, it would be necessary to consider the modal approximation given in Equation 16. In fact, the results given in Equations 16 and 18 are very close to each other, and this is further illustrated in the simulation study. Using the location of the change points and the estimation of the speed at different segments partitioned by the change points, we can infer aberrant behaviors flagged by the method more accurately.

Simulation Study

Simulation Design

The performance of the proposed method based on RTs is investigated for the following aberrant test-taking behaviors in a computerized test. These aberrant behaviors represent specific situations with abrupt changes in speed. It is worthy of note that this study focuses on the RT-based CPA method for the detection of changes in the responding behavior of examinees. The aberrant responding behavior simulated does not reflect in the true ability but manifested in the working speed, which result in aberrant RT patterns.

Case 1: Speededness

When there is not sufficient time to work carefully on all the items, examinees may rapidly respond to remaining items before time is up. In this type of aberrant pattern, the RTs to a set of consecutive items at the end of the test are noticeably shorter compared to items in the early parts of a test.

Case 2: Warm-up effect and speededness

As examinees begin to take a test, they may have trouble settling in or warming up to the normal test-taking process. As there is no time stress, the examinees may spend more time on fully reviewing the early items. This can be found in conjunction with speededness behavior when too much time is spent on early items, leading to the presence of two aberrant behaviors.

We simulate $q = 1$ in the case that an examinee has only the aberrant behavior in Case 1, and $q = 2$ when an examinee has warm-up and speededness behaviors as in Case 2. Figure 2 graphically illustrates these two types of aberrant response behaviors.

Figure 2.
Two cases of aberrant test-taking behaviors. Note. In all scenarios, the number line represents the order of items and the arrow indicates the location of the change point; that is, there is an abrupt change in the examinee’s speed at the next item. The items below the bracket indicate where the aberrant behavior occurs. The label above the bracket gives the name of the aberrant behavior.

One way to generate the RT data affected by behavior in Case 1 is to subtract a constant from $τ_{p}$ in the lognormal RT model, as proposed by van der Linden and van Krimpen-Stoop (2003):

$ln T_{i p} \sim N (β_{i} - τ_{p} + L, α_{i}^{- 2}), L < 0,$

where L is a speed shift caused by the aberrant behavior and represents the degree of an abrupt change in an examinee’s speed. RTs with the warm-up effect can be generated analogously from

$ln T_{i p} \sim N (β_{i} - τ_{p} + L, α_{i}^{- 2}), L > 0,.$

when $L = 0$ and

$ln T_{i p} \sim N (β_{i} - τ_{p}, α_{i}^{- 2}),$

which represents the normal situation.

The simulation study is based on a nonadaptive computerized test of three different test lengths: 20 items (short test), 50 items (moderate test), and 80 items (long test). The discrimination parameters are generated from $U (1.5, 2.5)$ , and examinees’ speed parameters are generated from $N (0, 0.5)$ . In van der Linden and Guo’s (2008) paper, the estimates of the time intensity parameters $β_{i}$ were in the range 2.68–6.70; we therefore generated the intensity parameters from $N (5, 0.5)$ .

In the simulation, we use $ω$ (0 ≤ $ω$ ≤ 1) to represent the location of a change point and set $ω$ to follow a beta distribution. It is known that if there is a change point in a set of data, the closer its location to the center of the data, the higher the detection efficiency will be (Julious, 2001). Therefore, in order to explore the performance of the proposed method when the change point is at a more extreme position in the test, in the case of $q = 1$ , we set two medians of $ω$ : 0.7 and 0.8. Each case has two different speed shifts: $L = - 1$ and $L = - 2$ . In total, therefore, there are 12 conditions: 3 test lengths × 2 locations of change point × 2 different shifts.

In the case of $q = 2$ , $ω_{1}$ represents the position where the warm-up effect finishes, $ω_{2}$ represents the position where the speededness begins, and L ₁ and L ₂ represent the corresponding speed shift caused by the warm-up effect and the speededness. For the medians of $ω_{1}$ and $ω_{2}$ , we set two combinations: (0.3, 0.7) and (0.2, 0.8). The variances of $ω_{1}$ and $ω_{2}$ are set to 0.001. For speededness, there are also two different speed shifts: $L_{2} = - 1$ and $L_{2} = - 2$ . In practice, the degree of speed change affected by the warm-up may not be very large, and so we keep $L_{1} = 1$ . Hence, there are again 12 conditions.

The total sample size is set at $N = 500$ in each case. In each condition of the simulation, 30% of the examinees (30% × 500 = 150) have aberrant behaviors, and the remaining 350 examinees respond normally. The performance of the proposed method is evaluated in four aspects: hit rate, false detection rate, false alarm rate, and lag. For the detection of the change points, the hit rate is the proportion of examinees who have aberrant behaviors with $q = 1$ or $q = 2$ and who are successfully detected. The false detection rate is the proportion of examinees with q change points who are incorrectly detected as having another number of change points. The false alarm rate is the proportion of examinees with no change point who are incorrectly detected as having change points. We also examine the accuracy of our procedure in locating the change points. The difference between the estimated change point and the true change point is defined as lag (see Figure 3). We use the absolute value of lag and then calculate its mean and SD. In the case where there are two change points, we calculate their mean absolute lag.

Figure 3.
Difference between the estimated change point and the true change point.

Parameter Selection

First, let m = 3 for each examinee; that is, the possible number of change points for each examinee is 0, 1, 2, or 3. Therefore, as discussed before, the hyperparameter r should be chosen from (0, 0.25). In order to obtain the appropriate value of σ, we calculate the hit rate and the false alarm rate for our approach in each condition for 100 different values of σ: 0.05, 1,…, 4.95, 5, with μ = 0 and r is chosen from (0, 0.25) for each examinee. The sample size is 500. We also compute the corresponding results when m = 2 and r is from (0, 1/3). Figure 4 summarizes the results for the false alarm rate about σ when all 500 examinees respond normally for m = 3 and m = 2. Figures 5 and 6 summarize the results when all 500 examinees have aberrant behaviors, with the hit rate about σ in each condition for m = 3 and m = 2, respectively.

Figure 4.
False alarm rate about σ when m = 3 (left) and m = 2 (right).

Figure 5.
Hit rate about σ when m = 3.

Figure 6.
Hit rate about σ when m = 2.

The results show that, in all conditions, as the value of σ increases from 0.05 to 5, the false alarm rate and hit rate first increase and then level off after about σ = 2 for three test lengths. This trend is expected; a smaller σ indicates a narrower range for the change of speed, and hence, it is more likely that no change point will be detected. By contrast, a larger value of σ indicates a larger range for the change of speed, which makes it easier to detect the change points. However, after about σ = 2, there is no obvious influence on the detection results. Note that the results for m = 2 and m = 3 are very similar, which indicates that the results are not sensitive to m in some way. (Because the space is limited, the hit rates of case 1 are shown here, and the regularity of the results of case 2 is similar to those). Based on the above analysis, m = 3, r ∈ (0, 0.25), σ = 2, and μ = 0 are used for the simulation study to evaluate the performance of the method.

Results of the Simulation

Tables 1 and 2 summarize the detection results for examinees who have the simulated aberrant behaviors with one and two change points. The hit rate, false detection rate, false alarm rate, and mean and SD of the absolute lag, calculated on the basis of 50 replications, are shown in each table. As shown in Table 1 for q = 1, in almost all conditions, the hit rate is high and the mean and SD of the absolute lag are small. The hit rate increases as the test length increases, because the longer the test, the more items will be affected by speededness. For a test of 20 items, the hit rate drops slightly as ω increases (the starting point of the speeded responses gets closer to the end of the test), but this is not found for tests of 50 or 80 items. This drop is expected because fewer items are affected by speededness, especially in a short test, and it is consistent with the findings of Shao et al. (2016). Note that the power range reported in Shao et al. (2016) was 0.49–0.88 for a test of 50 items, where the medians of ω were 0.5, 0.6, and 0.7. The Bayesian CPA method is therefore more efficient. An important reason for this is the use of RTs and the greater information that can be obtained from the data. In addition, the hit rate is influenced by the degree of speed shift, mainly for the test length is 20 items, which shows that the bigger shift, the higher the hit rate.

Table 1.
Detection for q = 1

Shift (L) n Median Hit Rate False Detection Rate False Alarm Rate Lag

ω 0 2 3 Mean Mean SD

−1 20 .7 .854 .127 .019 0 .049 .817 1.558

50 .7 .983 .002 .015 0 .029 .685 1.324

80 .7 .985 0 .015 0 .024 .663 1.175

20 .8 .700 .286 .014 0 .051 .878 1.967

50 .8 .967 .017 .016 0 .031 .745 1.581

80 .8 .985 0 .015 0 .024 .647 1.241

−2 20 .7 .970 0 .030 0 .049 .076 0.285

50 .7 .983 0 .017 0 .029 .058 0.250

80 .7 .986 0 .014 0 .024 .061 0.250

20 .8 .972 .002 .026 0 .051 .074 0.307

50 .8 .981 0 .019 0 .031 .064 0.251

80 .8 .985 0 .015 0 .024 .062 0.253

When comparing the results of the lag, we find that conditions with a bigger speed shift ( $L = - 2$ ) of the speeded examinees yield a much smaller mean and SD of the absolute lag, indicating that the shift is a key factor that influences the absolute lag. For conditions where the speed shift is $L = - 1$ , the mean and SD of the absolute lag increase slightly as $ω$ increases, and they decrease slightly as the test length increases. However, they are not greatly affected by $ω$ or the test length when the shift is $L = - 2$ . Note that Shao et al. (2016) found that it was harder to locate the exact location of the change point accurately for a longer test (50 or 80 items).

Table 2 shows the detection results for aberrant behaviors when $q = 2$ . The hit rate is evidently higher when the test length is 50 or 80 items, because in those cases, sufficient number of items are affected by the warm-up effect and speededness. When the test length is 20 items, the hit rate is less than that for $q = 1$ , and it drops as the change point gets closer to the beginning or the end of the test. The hit rate also shows the bigger shift, the higher the hit rate, especially when the test length is 20 items. On the whole, conditions with a bigger degree of speed shift yield a much smaller mean and SD of absolute lag.

Table 2.
Detection for $q = 2$

Shift ( $L_{1}, L_{2}$ ) n Median Hit Rate False Detection Rate False Alarm Rate Lag

$ω_{1}$ $ω_{2}$ 0 1 3 Mean Mean SD

(1, −1) 20 .3 .7 .553 0 .447 0 .051 .676 1.110

50 .3 .7 .970 0 .025 .005 .029 .734 1.291

80 .3 .7 .994 0 0 .006 .023 .680 1.261

20 .2 .8 .487 0 .513 0 .054 .667 1.185

50 .2 .8 .941 0 .053 .006 .029 .707 1.384

80 .2 .8 .991 0 .004 .005 .023 .694 1.328

(1, −2) 20 .3 .7 .760 0 .234 .006 .054 .395 0.809

50 .3 .7 .986 0 .009 .005 .029 .400 0.904

80 .3 .7 .994 0 0 .006 .023 .365 0.839

20 .2 .8 .700 0 .295 .005 .054 .405 0.901

50 .2 .8 .976 0 .017 .007 .029 .408 1.034

80 .2 .8 .993 0 .002 .005 .023 .377 0.897

The false alarm rate is around 0.05 when the test length is 20 items, 0.03 for 50 items, and 0.02 for 80 items. Therefore, the false alarm rate can be well controlled within a reasonable range for tests of different lengths. For the false detection rate, the corresponding three columns in Tables 1 and 2 show the results of misidentifying the true number of change points as the other values, which can be seen that there is relatively small possibility of “magnifying” the number of change points.

Inference About the Types of Aberrant Behaviors

In addition to the detection of the change points, the value of $η_{j}$ (j = 0,…, q) of each examinee can be estimated. Table 3 presents the posterior means and SDs of $η_{j}$ for two examinees in the simulation study when the test length is 50 items. It includes both the results from the mixture $E (η_{j} | Y, q)$ and the results from the modal approximation $E (η_{j} | Y, \overset{\land}{k}, q)$ . By combing the results of their numbers of change points q and the locations of the change points $\overset{\land}{k}$ , the numbers and types of aberrant behaviors can be inferred.

Table 3.
Posterior Expected Value and Standard Deviation of $η_{j}$

q $\hat{k}$ j $E (η_{j} | Y, \overset{\land}{k}, q)$ $s d (η_{j} | Y, \overset{\land}{k}, q)$ $E (η_{j} | Y, q)$ $s d (η_{j} | Y, q)$

1 41 0 0.623 .076 0.616 .077

1 1.564 .168 1.510 .162

2 11, 42 0 −0.783 .139 −0.768 .138

1 0.153 .089 0.152 .090

2 1.258 .171 1.224 .169

Table 3 shows that the estimations of $η_{j}$ are very close to each other. The first examinee was detected as having one change point. The location of the change point was the 41st item, and the speed was abnormally fast in the second segment relative to the first segment. Thus, it is quite possible that the examinee was speeded in the test. The second examinee was detected as having two change points, the first at the 11th item and the second at the 42nd item; the speed was abnormally slow at the beginning of the test and abnormally fast at the end. Hence, we have evidence that this examinee may have engaged in both the warm-up effect and speededness.

Robustness of the Proposed Method to Assumption Violations

Violation of Lognormal RT Models

As the Bayesian CPA method is established for the lognormal RT model, some RT data that are not from the lognormal model are simulated. The distribution of RT data is typically positively skewed (Fazio, 1990). Assuming that the transformed RTs are normal, we simulated the χ² distribution of RTs with degrees of freedom ${(β_{i} - τ_{p})}^{2}$ and took the square root to generate noise in the RT data to see how and to what extent this disturbance might affect the method. Taking the test length of 50 items as an example, for Cases 1 and 2, we set two ratios of noise: 0.1 and 0.2. Specifically, 0.1 × 50 = 5 or 0.2 × 50 = 10 RT data from χ² distribution were both set in the examinees with aberrant behaviors and without aberrant behaviors. The detection results are shown in Table 4 (the results of q = 1 are shown here, and the regularity of the results of q = 2 is similar to those).

Table 4.
Detection When There Is Noise in the Lognormal Response Time Model

q Noise Ratio Median $ω$ Hit Rate False Alarm Rate Lag

Mean Mean SD

1 0 .7 .983 .029 .685 1.324

0 .8 .967 .031 .745 1.581

.1 .7 .972 .055 .788 1.407

.1 .8 .953 .056 .836 1.768

.2 .7 .950 .084 .880 1.651

.2 .8 .938 .089 .953 2.135

Note. The proportion of examinees with aberrant behaviors is assumed to be 30%. When $q = 1$ , the speed shift is $L = - 1$ .

Table 4 indicates that as the noise ratio increases, the hit rate of the method decreases, the false alarm rate increases, and the mean and SD of the lag also increase. These results suggest that the square root performs worse than the log and the reciprocal in terms of reducing positive skewness (Stocké, 2004; van der Linden et al., 1999), and so this RT model deviates further from the assumption. However, the performance of this method is slightly worse in the presence of such disturbances. It can therefore be concluded that the method is not very sensitive to the assumption of the lognormal RT model.

Violation of the Prior With the Binomial Distribution

Assuming that each examinee may have 0, 1, 2, or 3 numbers of change points, which do not follow binomial (3, r) distributions, the proposed detection model (with binomial distribution prior) was used to identify the change points. In addition, we consider another pattern of the speed change, the gradual speed change, to examine the performance of the method.

The simulation settings are specified as follows.
Two test scenarios: In Case 1, the largest proportion of examinees comprises those who have aberrant test-taking behaviors. The truncated Poisson distribution is used to generate the number of change points for each examinee, which is less than or equal to 3. The parameter $λ$ is set as 1.5, meaning that the zero change point does not have the highest probability. In Case 2, the largest proportion of examinees comprises those who answer normally. The number of change points of each examinee is generated by the truncated zero-inflated Poisson model, the probability function of which is 0 with probability 0.7, and Poisson (1.5) with probability 0.3. In this case, the zero change point has the highest probability.

Two speed change patterns: When $q = 1$ , the speed is abruptly changed by the speededness, and $L = - 1$ , the median of $ω$ is 0.8; when $q = 2$ , $L_{1} = 1$ , and $L_{2} = - 1$ , the medians of $ω_{1}$ and $ω_{2}$ are 0.2 and 0.8, respectively. For the warm-up, two patterns of speed change are considered—one is the abrupt change and the other is a gradual change:
$ln T_{i p} \sim N (β_{i} - τ_{p} + L_{1} (1 - (i - 1) / (n ω (p))), α_{i}^{- 2}), i = 1, \dots, n$

where $n ω (p)$ represents the change point for examinee p. In this pattern, the speed of examinee p starts from $τ_{p} - L_{1}$ , gradually increases to $τ_{p}$ , and is then sustained at $τ_{p}$ after the change point; when $q = 3$ , $L_{1} = 1$ , $L_{2} = - 1$ , and $L_{3} = - 2$ , the medians of $ω_{1}$ , $ω_{2}$ , and $ω_{3}$ are 0.2, 0.5, and 0.8, and there are still two patterns of the speed change for the warm-up.

Four test lengths: 20, 40, 60, and 80 are considered in each condition to get a more detailed picture of how detection results change with test length.

The false alarm rate, the hit rate of aberrant behaviors (examinees who have 1, 2, or 3 change points), and the overall correct classification rate for the number of change points are calculated to evaluate the performance of the method. The results are shown in Table 5.

Table 5.
Detection When the Underlying Prior Distribution is Different From the Binomial (3, r)

Prior Speed Change n Correct Classification Rate Hit Rate False Alarm Rate

1 2 3

Case 1 Abrupt 20 .650 .724 .411 .018 .052

40 .853 .936 .864 .427 .040

60 .957 .982 .975 .829 .028

80 .983 .986 .991 .966 .023

Gradual 20 .526 .692 .184 .012 .050

40 .733 .940 .524 .194 .036

60 .848 .981 .741 .485 .025

80 .913 .987 .869 .694 .026

Case 2 Abrupt 20 .839 .692 .399 .018 .049

40 .930 .936 .856 .422 .035

60 .968 .977 .973 .817 .026

80 .979 .983 .994 .964 .022

Gradual 20 .828 .693 .183 .019 .051

40 .897 .943 .520 .215 .035

60 .932 .980 .728 .464 .028

80 .957 .986 .861 .704 .024

Note. “Abrupt” indicates that the speed change of the warm-up effect is abrupt; “gradual” indicates that the speed change of the warm-up effect is gradual.

First, the false alarm rates under various conditions are basically consistent with those shown in Tables 1 and 2. The longer the test length, the lower the false alarm rate. When the test length is 40, the false alarm rate is around 0.04. Second, in Cases 1 and 2, when the speed change is abrupt, the hit rates for one and two change points are roughly the same as those shown in Tables 1 and 2 and are largely unaffected by the underlying prior distribution. When the warm-up is subject to a gradual change in speed, the hit rates for detecting the change points are decreased compared to the abrupt change (see the hit rates for two and three change points). In this situation, the speed of the examinee is constantly changing across a series of items, and the degree of change at the change point is small, so the detection efficiency decreases significantly. In addition, when there are three change points, the hit rate is relatively lower than that of other numbers of change points, and this increases with increased test length. Regarding the correct classification rate of examinees’ test-taking behaviors (according to the number of change points), those in Case 2 are higher than in Case 1. As the number of examinees with zero change points is the largest in Case 2, which is consistent with our assumption (the probability of having zero change points is the largest for each examinee), the overall classification rate is higher.

Summary of the Simulation Results

In summary, the Bayesian CPA method is effective in detecting aberrant test-taking behaviors when there is more than one change point in the test performance of an examinee, and the false alarm rate can be controlled in a reasonable range. The location of change points for each examinee can also be estimated accurately. When this is combined with the estimation of the speed in each segment partitioned by the change points, it is possible to determine the number and types of aberrant behaviors flagged by the method. The results indicate that the test length, the location of change points, and the degree of speed changes influence the efficiency of the detection results. The longer the test length, the greater the degree of speed changes at change points, and the higher the efficiency of the method. In addition, the proposed method is robust to the violation of assumptions of the lognormal model and that the underlying distribution of the change point is not binomial.

Real Data Analysis

To illustrate the application of the proposed method in real data, we used data from a high-stakes standardized CAT program that was analyzed by Meng et al. (2015). In high-stakes tests, when the examinee may feel nervous or under time pressure, the warm-up effect and speededness behaviors are likely to occur. We compared the Bayesian CPA method with RTs and the Wald test statistic based on item responses (Sinharay, 2016) in the real data analysis.

The data set consists of 2,061 examinees and the test length is 37 items; that is, each examinee responds to 37 items delivered adaptively from a pool of 620 items. We used the lognormal RT model to fit the data and estimate the parameters in the model. According to Meng et al. (2015), the fit of the lognormal RT model for this empirical data set is satisfactory. The estimated intensity parameter has a mean of 4.45 and variance of 0.23, and the discrimination parameter has a mean of 1.66 and variance of 0.21.

First, we used the Bayesian CPA method to calculate the false alarm rate through a simulation with real data conditions (n = 37 and N = 2,061), assuming a maximum of three change points, using hyperparameters $μ = 0$ , $σ = 2$ , and r ∈ (0.01∼0.25) for each examinee. The results show that the maximum false alarm rate is less than 10% (see left panel of Figure 7). There are two marked points where the false alarm rate is closest to 1% at Data 1 and 5% at Data 2, and the corresponding values of r are 0.05 and 0.17. The relationship between the false alarm rate and the hit rate is also shown (in the right panel of Figure 7), with conditions q = 1 (ω = 0.8), q = 2 ( $ω_{1}$ = 0.2 and $ω_{2}$ = 0.8), and q = 3 ( $ω_{1}$ = 0.2, $ω_{2}$ = 0.5, and $ω_{3}$ = 0.8), and the speed shift of each change is 1 in each condition. According to these results, we used r = 0.17 to detect each examinee’s test-taking behavior and make the false alarm rate as close to 0.05 as possible.

Figure 7.
False alarm rate about r (left) and its relationship with the hit rate (right).

For each examinee, the posterior probability of the number of change points of the RT pattern was computed. We found that 1,389 examinees had no change point, 616 examinees had one change point, 53 examinees had two change points, and three examinees had three change points. The examinees who had no change point did not show any abrupt change in speed in their test performance, and hence, these examinees could be deemed as normal responding examinees. The examinees who had change points showed abrupt changes in speed in their test performance, indicating suspicious response behavior. We estimated the locations of the change points for each examinee and the speed in each segment partitioned by the change points. These results allowed us to infer the types of the suspected behaviors, as discussed in the following.

Among examinees who had one change point, those whose estimated speed in the second segment was faster may have had speededness or the warm-up effect. Specifically, if the speed was relatively normal (moderate) in the first segment but abnormally fast in the second segment, the behavior could be related to speededness. If the speed in the first segment was abnormally slow but relatively normal in the second segment, and the change point was in the first few items, the warm-up effect is possible. For examinees whose speed in the second segment was slow (perhaps because of their response strategy), it is possible that later items were more difficult, and therefore, they spent more time on those items.

For examinees with two change points, if the locations of the change points were at the beginning and the end of the test, respectively, and the speed was abnormally slow at the beginning, moderate in the middle, and abnormally fast at the end, then it is possible that speededness and the warm-up effect occurred simultaneously. In addition, if the two change points were relatively close to each other, with the speed abnormally high in the middle but relatively normal on either side, then rapid guessing on the middle items is likely. Similarly, for examinees with three change points, one can make inferences about their test-taking behaviors on the basis of the change points and the speed. It should be noted that the presence and the types of aberrant behaviors need to be determined based on these empirical results holistically.

Figure 8 shows the RT patterns and the detection results of five examinees who had change points. The item number is shown along the x-axis and the RTs (in seconds) along the y-axis. The dashed vertical line represents the location of the change point for each examinee. We can see that there is a clear abrupt increase or drop-off in RT after the detected change point. The change point for the first examinee is at Item 27, with an estimated speed of −0.30 before the change point and 1.33 after the change point. The speed is very fast in the last segment, it is likely that speededness occurred. The change point for the second examinee is at Item 3, with an estimated speed of −0.74 before the change point and 0.23 after the change point, a warm-up effect is likely to be present for this examinee. The change point for the third examinee is at Item 27, with an estimated speed of −0.03 before the change point and −0.94 after the change point, indicating that the examinee most likely changed their response strategy before the end of the test. The change points for the fourth examinee are at Items 7 and 24, with an estimated speed of −0.86 before the first change point, 0.05 between the two change points, and 1.03 after the second change point. The examinee most likely had both a warm-up effect and speededness behaviors. The change points for the fifth examinee are at Items 23 and 35, with estimated speeds of −0.55, 2.70, and −0.72 in each test segment. The speed in the middle segment is abnormally fast, and it is therefore likely that the examinee adopted rapid guessing behavior for those items.

Figure 8.
Response times of five detected examinees.

The Wald test statistics using response data were also applied to analyze the test-taking behavior of this group of examinees. Following Sinharay (2016), the unidimensional Rasch model was used for item calibration. A Bayesian ability estimate was used initially, the MLE of ability was used for the Wald test, and a 5% significance level was chosen (which was nearest to the false alarm rate of the Bayesian CPA method above). For the test with 37 items, 37 × 0.15 = 6 items located at the beginning and the end of the test were removed, giving a corresponding critical value of 8.85. A total of 382 examinees were found to have change points, which means that the correct response probability of these examinees changed significantly at a certain item. Among these examinees, 154 had one change of speed, 20 changed speed twice, one changed speed three times, and 207 examinees did not change speed.

These results can be combined with those obtained using the Bayesian CPA method to make further inferences about the aberrant behavior. For example, if the correct response probability of an examinee decreases after a certain item, and the speed after that is faster than before, it can be further shown that the examinee has speededness. In contrast, if the speed reduces, it is very likely that the decline in the probability of a correct response was caused by item difficulty, not by aberrant behaviors. For the 207 examinees who had no change of speed, there is insufficient evidence to determine the presence of aberrant behavior, despite the changes in their correct response probabilities. In addition, the speed of 497 examinees had one or more changes, but their correct response probabilities were not changed.

The results of the two methods show that when false alarm rates are similar, the Bayesian CPA method using RTs is efficient at detecting change points in the performance of examinees and to provide information about their speed change at each stage. By combining the detection results of the two methods, we are able to better understand the behavior change process of each examinee in the test. Further comparison and generalization of these results are undoubtedly worthy of attention in the future studies.

Summary and Discussion

Various aberrant behaviors are often observed in real test data. It is reasonable to expect that in many cases, there will be more than one change point or more than one aberrant behavior in an examinee’s test performance. RTs in CBT reveal information about the working speed of the examinee. Further, RTs provide information useful for detecting changes in response behaviors. This article proposes a Bayesian CPA approach using RTs to identify aberrant test-taking behaviors on the basis of abrupt changes in the speed of examinees.

The proposed method has the following characteristics and advantages: (1) It does not rely on the asymptotic assumptions about test statistics that can be problematic in situations, where parametric models are restricted to a small number of items; (2) the number and location of change points are taken as parameters in the model, and the prior information is combined to infer these parameters from the posterior distribution, such that an examinee can have multiple change points or multiple aberrant behaviors; (3) RTs are used to provide more information about changes in an examinee’s test performance and to improve the efficiency of detection of aberrant behavior; and (4) the speed of examinees at each stage can be determined, which helps to make more accurate inferences about the types of behaviors flagged by the method. Overall, the detection results provide a more comprehensive understanding of examinee behaviors.

In this study, the number of change points of each examinee is assumed to follow a binomial distribution. The hyperparameter r is associated with the probability of the number of change points; it is selected from a certain range and contributes to making the inference more efficient. This can be seen as a reasonable penalty imposed on the selection of change points, which reduces the effect by which the model itself tends to choose more change points. The hyperparameter $σ$ presents the range of possible values for the speed of each examinee. A relatively large value is used in this study, and the reasonableness of this value is also justified via numerical simulation. Another option may be to take $σ$ as a smaller value to constrain the inference of q; however, this raises other questions of how best to select the prior distribution of q and whether noninformative priors are also appropriate.

Several limitations of this study should be noted. First, the assumption of the proposed method for detecting aberrant behavior is the abrupt changes in speed. This may not be the only underlying factor in the occurrence of aberrant behavior for examinees. Accordingly, an important distinction between the detected change points is whether they are caused by aberrant behavior, item characteristics, or different strategies that an examinee might legitimately use. Our results indicate that it is possible to combine an examinee’s change points with the speed of each segment to distinguish different causes. In addition, the assumption of the procedure is that aberrant behavior will manifest in different responding speeds irrespective of performance. These results can be combined with information from the CPA results based on the response data. Testing the differences in response accuracy partitioned by the change points might be another perspective to validate the results from the proposed method. It is worthy of note that the evidence provided by statistical results should not be the sole source to flag possible aberrant behaviors, and it is highly recommended that such evidence is combined with evidence from other sources for cross-validation (Cheng & Shao, 2022; Marianti et al., 2014; Sinharay, 2016; van der Linden & van Krimpen-Stoop, 2003).

Second, the method is more powerful for longer tests and larger speed shift. If the test length is short (e.g., 20 items), the efficiency of the method is largely determined by the degree of the speed change. Further, if the speed shift is small, or the speed change is gradual, the performance of the method could be significantly worse, including a significant decrease in hit rate and change point estimation accuracy. Moreover, the greater the number of change points for the examinee, the stronger the dependence on these two conditions. Thus, how small speed shifts can be detected is worthy of further exploration.

Third, the method is appropriate when an investigator wants to detect aberrant behavior occurring on a set of consecutive items. However, for the detection of behavior on items scattered throughout a test, the method proposed here may not be appropriate. Fourth, the proposed method is established only for the lognormal RT model, but there may be data sets for which the lognormal model does not fit the data.

Future research may tap the following areas. First, the Bayesian CPA method could be applied to detect item preknowledge. When the investigator knows which items are compromised; the items answered by each examinee can be reordered, with the compromised items on one segment and the rest on another segment (Sinharay, 2017a, 2017b). The Bayesian CPA method can detect item preknowledge by examining whether the speed of an examinee is significantly faster on the compromised items than on the noncompromised items. Second, different prior distributions can be explored. Based on this study, it is critical to ensure the reasonable selection of parameters, which should not only take into account reality but also theoretical beliefs, to make the detection as efficient as possible. Third, different RT models can be applied in future work, such as the Box–Cox normal RT model (Klein Entink et al., 2009), which allows for more flexibility in the transformation of the RT data to obtain a normal distribution. Fourth, the proposed RT method can be combined with IRT models to utilize responses and RTs simultaneously through joint modeling. For example, in the hierarchical model of van der Linden (2007), it models the examinee ability $θ$ and working speed $τ$ following a bivariate normal distribution at the second level. In this case, one can integrate the information of the ability parameters in detecting the change of speed parameters.

Shift (L)	n	Median	Hit Rate	False Detection Rate	False Alarm Rate
−1	20	.7	.854	.127	.019	.049	.817	1.558
	50	.7	.983	.002	.015	.029	.685	1.324
	80	.7	.985	0	.015	.024	.663	1.175
	20	.8	.700	.286	.014	.051	.878	1.967
	50	.8	.967	.017	.016	.031	.745	1.581
	80	.8	.985	0	.015	.024	.647	1.241
−2	20	.7	.970	0	.030	.049	.076	0.285
	50	.7	.983	0	.017	.029	.058	0.250
	80	.7	.986	0	.014	.024	.061	0.250
	20	.8	.972	.002	.026	.051	.074	0.307
	50	.8	.981	0	.019	.031	.064	0.251
	80	.8	.985	0	.015	.024	.062	0.253

Shift ( $L_{1}, L_{2}$ )	n	Median	Hit Rate	False Detection Rate	Lag
(1, −1)	20	.3	.7	.553	.447	0	.051	.676	1.110
	50	.3	.7	.970	.025	.005	.029	.734	1.291
	80	.3	.7	.994	0	.006	.023	.680	1.261
	20	.2	.8	.487	.513	0	.054	.667	1.185
	50	.2	.8	.941	.053	.006	.029	.707	1.384
	80	.2	.8	.991	.004	.005	.023	.694	1.328
(1, −2)	20	.3	.7	.760	.234	.006	.054	.395	0.809
	50	.3	.7	.986	.009	.005	.029	.400	0.904
	80	.3	.7	.994	0	.006	.023	.365	0.839
	20	.2	.8	.700	.295	.005	.054	.405	0.901
	50	.2	.8	.976	.017	.007	.029	.408	1.034
	80	.2	.8	.993	.002	.005	.023	.377	0.897

q	$\hat{k}$	j	$E (η_{j} \| Y, \overset{\land}{k}, q)$	$s d (η_{j} \| Y, \overset{\land}{k}, q)$	$E (η_{j} \| Y, q)$	$s d (η_{j} \| Y, q)$
1	41	0	0.623	.076	0.616	.077
		1	1.564	.168	1.510	.162
2	11, 42	0	−0.783	.139	−0.768	.138
		1	0.153	.089	0.152	.090
		2	1.258	.171	1.224	.169

q	Noise Ratio	Median $ω$	Hit Rate	False Alarm Rate	Lag
1	0	.7	.983	.029	.685	1.324
	0	.8	.967	.031	.745	1.581
	.1	.7	.972	.055	.788	1.407
	.1	.8	.953	.056	.836	1.768
	.2	.7	.950	.084	.880	1.651
	.2	.8	.938	.089	.953	2.135

Prior	Speed Change	n	Correct Classification Rate	Hit Rate	False Alarm Rate
Case 1	Abrupt	20	.650	.724	.411	.018	.052
		40	.853	.936	.864	.427	.040
		60	.957	.982	.975	.829	.028
		80	.983	.986	.991	.966	.023
	Gradual	20	.526	.692	.184	.012	.050
		40	.733	.940	.524	.194	.036
		60	.848	.981	.741	.485	.025
		80	.913	.987	.869	.694	.026
Case 2	Abrupt	20	.839	.692	.399	.018	.049
		40	.930	.936	.856	.422	.035
		60	.968	.977	.973	.817	.026
		80	.979	.983	.994	.964	.022
	Gradual	20	.828	.693	.183	.019	.051
		40	.897	.943	.520	.215	.035
		60	.932	.980	.728	.464	.028
		80	.957	.986	.861	.704	.024

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research and/or authorship of this article: This work was supported by National Natural Science Foundation of China (11871141).

ORCID iD

Hongyue Zhu

References

Allalouf

(2007). Quality control procedures in the scoring, equating, and reporting of test scores. Educational Measurement: Issues and Practice, 26(1), 36–46.

Bolt

D. M.

Cohen

A. S.

Wollack

J. A.

(2002). Item parameter estimation under conditions of test speededness: Application of a mixture Rasch model with ordinal constraints. Journal of Educational Measurement, 39(4), 331–348.

Bradlow

E. T.

Weiss

R. E.

Cho

(1998). Bayesian identification of outliers in computerized adaptive tests. Journal of the American Statistical Association, 93(443), 910–919.

Chen

Gupta

A. K

. (2012). Parametric statistical change point analysis: With applications to genetics. Medicine and Finance. Springer.

Cheng

Shao

(2022). Application of change point analysis of response time data to detect test speededness. Educational and Psychological Measurement, 82(5), 1031–1062.

Choe

E. M.

Zhang

Chang

H. H.

(2018). Sequential detection of compromised items using response times in computerized adaptive testing. Psychometrika, 83(3), 650–673.

Chopin

(2007). Dynamic detection of change points in long time series. Annals of the Institute of Statistical Mathematics, 59(2), 349–366.

Douglas

Kim

H. R.

Habing

Gao

(1998). Investigating local dependence with conditional covariance functions. Journal of Educational and Behavioral Statistics, 23, 129–151.

Embretson

S. E.

Reise

(2000). Item response theory for psychologists. Lawrence Erlbaum.

10.

Fazio

R. H.

(1990). A practical guide to the use of response latency in social psychological research. In Hendrick

Clark

M. S.

(Eds.), Research methods in personality and social psychology (pp. 74–97). Sage.

11.

Fox

J. P.

Marianti

(2016). Joint modeling of ability and differential speed using responses and response times. Multivariate Behavioral Research, 51(4), 540–553.

12.

Inclán

(1993). Detection of multiple changes of variance using posterior odds. Journal of Business and Economics Statistics, 11(3), 289–300.

13.

Julious

S. A.

(2001). Inference and estimation in a changepoint regression problem. Journal of the Royal Statistical Society: Series D (The Statistician), 50(1), 51–61.

14.

Klein Entink

R. H.

van der Linden

W. J.

Fox

J.-P.

(2009). A Box-Cox normal model for response times. British Journal of Mathematical and Statistical Psychology, 62, 621–640.

15.

Liu

Han

K. T.

(2019). Compromised item detection for computerized adaptive testing. Frontiers in Psychology, 10, 829.

16.

Marianti

Fox

Avetisyan

Veldkamp

B. P.

Tijmstra

(2014). Testing for aberrant behavior in response time modeling. Journal of Educational and Behavioral Statistics, 39(6), 426–451.

17.

Meijer

R. R.

(2002). Outlier detection in high-stakes-stakes certification testing. Journal of Educational Measurement, 39(3), 219–233.

18.

Meijer

R. R.

Sijtsma

(2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25(2), 107–135.

19.

Meng

X. B.

Tao

Chang

H. H.

(2015). A conditional joint modeling approach for locally dependent item responses and response times. Journal of Educational Measurement, 52(1), 1–27.

20.

Meyer

J. P.

(2010). A mixture Rasch model with item response time components. Applied Psychological Measurement, 34(7), 521–538.

21.

Montgomery

D. C.

(2013). Introduction to statistical quality control. Wiley.

22.

Qian

Staniewska

Reckase

Woo

(2016). Using response time to detect item Preknowledge in computer-based licensure examinations. Educational Measurement: Issues and Practice, 35(1), 38–47.

23.

Rao

C. R.

(1973). Linear statistical inference and its applications (Vol. 2, pp. 263–270). Wiley.

24.

Ruggieri

Antonellis

(2016). An exact approach to Bayesian sequential change point detection. Computational Statistics & Data Analysis, 97, 71–86.

25.

Schnipke

D. L.

Scrams

D. J.

(1997). Modeling item response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34(3), 213–232.

26.

Shao

. (2017). Aberrant response detection using change-point analysis. University of Notre Dame.

27.

Shao

Cheng

(2016). Detection of test speededness using change-point analysis. Psychometrika, 81(4), 1118–1141.

28.

Sinharay

(2016). Person fit analysis in computerized adaptive testing using tests for a change point. Journal of Educational and Behavioral Statistics, 41(5), 521–549.

29.

Sinharay

(2017a). Detection of item Preknowledge using likelihood ratio test and score test. Journal of Educational and Behavioral Statistics, 42(1), 46–68.

30.

Sinharay

(2017b). Some remarks on applications of tests for detecting a change point to psychometric problems. Psychometrika, 82(4), 1149–1161.

31.

Smith

A. F. M.

(1975). A Bayesian approach to inference about a change-point in a sequence of random variables. Biometrika, 62(2), 407–416.

32.

Stocké

(2004). Measuring information accessibility and predicting response-effects: The validity of response-certainties and response latencies. Metodoloski Zvezki, 1, 33–55.

33.

van der Linden

W. J.

(2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181–204.

34.

van der Linden

W. J.

(2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287–308.

35.

van der Linden

W. J.

Guo

(2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika, 73(3), 365–384.

36.

van der Linden

W. J.

Scrams

D. J.

Schnipke

D. L.

(1999). Using response-time constraints to control for differential speededness in computerized adaptive testing. Applied Psychological Measurement, 23(3), 195–210.

37.

van der Linden

W. J.

van Krimpen-Stoop

E. M.

(2003). Using response times to detect aberrant responses in computerized adaptive testing. Psychometrika, 68(2), 251–265.

38.

von Davier

A. A

. (2012). The use of quality control and data mining techniques for monitoring scaled scores: An overview. ETS Research Report Series, 2012(2), i–18.

39.

Wang

(2015). A mixture hierarchical model for response times and response accuracy. British Journal of Mathematical and Statistical Psychology, 68(3), 456–477.

40.

Wise

S. L.

DeMars

C. E.

(2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43(1), 19–38.

41.

Zhan

Jiao

Liao

(2018). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71(2), 262–286.

42.

Zhang

(2014). A sequential procedure for detecting compromised items in the item pool of a CAT system. Applied Psychological Measurement, 38(2), 87–104.

43.

Zhang

Chang

(2012). Comparing single-pool and multiple-pool designs regarding test security in computerized testing. Behavior Research Methods, 44, 742–752.

44.

Zhang

(2016). Monitoring items in real time to enhance CAT security. Journal of Educational Measurement, 53, 131–151.

Shift (L)	n	Median	Hit Rate	False Detection Rate			False Alarm Rate	Lag
Shift (L)	n	ω	Hit Rate	0	2	3	Mean	Mean	SD
−1	20	.7	.854	.127	.019	0	.049	.817	1.558
	50	.7	.983	.002	.015	0	.029	.685	1.324
	80	.7	.985	0	.015	0	.024	.663	1.175
	20	.8	.700	.286	.014	0	.051	.878	1.967
	50	.8	.967	.017	.016	0	.031	.745	1.581
	80	.8	.985	0	.015	0	.024	.647	1.241
−2	20	.7	.970	0	.030	0	.049	.076	0.285
	50	.7	.983	0	.017	0	.029	.058	0.250
	80	.7	.986	0	.014	0	.024	.061	0.250
	20	.8	.972	.002	.026	0	.051	.074	0.307
	50	.8	.981	0	.019	0	.031	.064	0.251
	80	.8	.985	0	.015	0	.024	.062	0.253

Shift ( $L_{1}, L_{2}$ )	n	Median		Hit Rate			False Alarm Rate	Lag
Shift ( $L_{1}, L_{2}$ )	n	$ω_{1}$	$ω_{2}$	Hit Rate	1	3	Mean	Mean	SD
(1, −1)	20	.3	.7	.553	.447	0	.051	.676	1.110
	50	.3	.7	.970	.025	.005	.029	.734	1.291
	80	.3	.7	.994	0	.006	.023	.680	1.261
	20	.2	.8	.487	.513	0	.054	.667	1.185
	50	.2	.8	.941	.053	.006	.029	.707	1.384
	80	.2	.8	.991	.004	.005	.023	.694	1.328
(1, −2)	20	.3	.7	.760	.234	.006	.054	.395	0.809
	50	.3	.7	.986	.009	.005	.029	.400	0.904
	80	.3	.7	.994	0	.006	.023	.365	0.839
	20	.2	.8	.700	.295	.005	.054	.405	0.901
	50	.2	.8	.976	.017	.007	.029	.408	1.034
	80	.2	.8	.993	.002	.005	.023	.377	0.897

Bayesian Change-Point Analysis Approach to Detecting Aberrant Test-Taking Behavior Using Response Times

Abstract

Keywords

CPA for Aberrant Responding Behavior Detection

Method

The Model

Specification of the Hyperparameters

Inference About η

Simulation Study

Simulation Design

Case 1: Speededness

Case 2: Warm-up effect and speededness

Parameter Selection

Results of the Simulation

Inference About the Types of Aberrant Behaviors

Robustness of the Proposed Method to Assumption Violations

Violation of Lognormal RT Models

Violation of the Prior With the Binomial Distribution

Summary of the Simulation Results

Real Data Analysis

Summary and Discussion

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iD

References

Inference About $η$