A Sequential Process Model for Cognitive Diagnostic Assessment With Repeated Attempts

Abstract

When diagnostic assessments are administered to examinees, the mastery status of each examinee on a set of specified cognitive skills or attributes can be directly evaluated using cognitive diagnosis models (CDMs). Under certain circumstances, allowing the examinees to have at least one opportunity to correctly answer the questions and assessments, with repeated attempts on the items, provides many potential benefits. A sequential process model can be extended to model repeated attempts in diagnostic assessments. Two formulations of the sequential generalized deterministic-input noisy-“and”-gate (G-DINA) model were developed in this study. The first extension uses the latent transition analysis (LTA) approach to model changes in the attributes over attempts, and the second extension constructs a higher order structure of latent continuous variables and latent attributes to account for the dependences of the attributes over attempts. Accurate model parameter estimation and correct classifications of attributes were observed in a series of simulations using Bayesian estimation. The effectiveness of the developed sequential G-DINA model was demonstrated by fitting real data from a longitudinal mathematical test to the developed model and the longitudinal G-DINA model using the LTA approach. Finally, this article closes by discussing several important issues associated with the developed models and providing suggestions for future directions.

Keywords

cognitive diagnosis models the G-DINA model sequential process model latent transition analysis higher order model

Over the past few decades, interest in the development of cognitive diagnostic assessments to provide diagnostic information about students’ strengths and weaknesses according to a set of cognitive skills or attributes has been increasing; this approach provides the advantages of facilitating learning and instruction in educational environments. In contrast to the conventional approaches to scaling and ordering students along a proficiency continuum (i.e., item response theory [IRT] models), cognitive diagnosis models (CDMs) hold great promise for extracting relevant diagnostic information, and they serve as an important means of constructing diagnostic assessments for identifying the mastery status of individuals on a set of multiple fine-grained skills (Leighton & Gierl, 2007; Rupp, Templin, & Henson, 2010). Diagnostic assessments based on certain standards can be used for different purposes. In state-mandated testing programs, for example, school performance can be evaluated using large-scale diagnostic assessments to satisfy state and federal accountability requirements; in practical instructional settings, diagnostic assessments can be used to aid instructors in understanding and addressing the specific needs of students and to guide teachers in designing effective instructional interventions for those in need (Huff & Goodman, 2007).

In response to the demand for more fine-grained information and personalized feedback on students’ strengths and weaknesses, tests that are cognitively diagnostic in nature can be administered to students multiple times in a sequential process by allowing examinees multiple attempts on test items until they submit a correct response to those items. Such an approach, that is, giving examinees repeated attempts at problem solving, was defined terminologically as answer-until-correct test administration (Pressey, 1926), and the literature has documented that the sequential assessment approach has a considerable number of potential benefits that are associated with student learning and development; these benefits include, for example, a reduction in test anxiety because of multiple attempts being allowed and the subsequent feedback when the examinees complete each attempt (Attali & Powers, 2010; DiBattista & Gosse, 2006; Epstein, Epstein, & Brosvic, 2001; Kluger & DeNisi, 1996).

The large body of research on the effectiveness of administering an answer-until-correct test has been supported by experimental evidence with the use of partial feedback and other materials to supplement students’ knowledge content levels (for an intensive discussion, see Ye, Fellouris, Culpepper, & Douglas, 2016). In an introductory psychology course, for example, Epstein et al. (2001) found that students who received immediate response feedback using an answer-until-correct testing procedure on multiple-choice examinations performed better than those using the traditional examination form. Another example was demonstrated with the use of constructed-response items on an answer-until-correct examination by Culpepper (2014). Examinees were allowed to provide a maximum number of five opportunities to submit a correct response to test items in a calculus-based probability theory course, and in this study, the sequential process of cognitive operations over repeated attempts was observed and the variation of content knowledge supplement over attempts was significantly large such that the benefits of retrieval-based activity were acknowledged.

At least two principal psychometric methods for analyzing sequential data collected from repeated trials deserve further discussion. The first method counts the number of attempts that an examinee requires to correctly answer an item, treats the required attempt number that an examinee has as a reciprocal function of an observed score, and then employs polytomous IRT models to analyze the observed scores (e.g., Attali, 2011; Attali & Powers, 2010; Muñiz & Menéndez, 2011). Under this scoring rubric framework, let T denote the maximum number of attempts allowed; when an examinee correctly responds to an item on attempt k ( $k = 1, \dots, T$ ), he or she will be assigned a score of $T - k$ . Although a polytomous IRT model, for example, the graded response model (GRM; Samejima, 1969), can be used to fit the ordered categorical response data and attempt-related item parameters and IRT-based latent trait scores are attainable, the polytomous scoring approach cannot consider individual differences in latent ability over repeated attempts and is not an ideal method for analyzing sequential data (Culpepper, 2014).

When repeated attempts on test items are allowed, a succession of cognitive operations over repeated trials should be considered and can be decomposed into different sequences that are reflective of different success rates on each attempt. Sequential IRT models have been developed for analyzing data that are collected from multiple attempts on items in cognitive tests, survival trials in experiments, or practices in psychomotor research (Akkermans, 2000; Albert & Chib, 2001; Bechger & Akkermans, 2001; Spray, 1990; Tutz, 1990). These proposed sequential IRT models can account for the sequential process of cognitive or psychomotor operation when examinees are allowed to respond to specific tasks via several attempts; however, certain methodological limitations should be noted. For cognitive assessments, it is unlikely to assume that the success rates on each attempt are independent because examinees could receive feedback or instructional intervention subsequently to supplement their content knowledge; consequently, their ability levels are expected to change over attempts. Recently, Culpepper (2014) proposed a series of sequential IRT models for repeated attempts on test items in which the probability of success on an attempt was conditional on that of success on the previous attempt by adding attempt-specific threshold parameters or person-specific growth parameters to model the dependences of multiple attempts. A testing program that allows examinees opportunities to correctly answer questions provides diagnostic information; however, neither Culpepper’s models nor the previous sequential IRT models can provide sufficient diagnostic information, in contrast to CDMs, because the proficiency levels of examinees are scored on a proficiency continuum using conventional IRT models, which offer limited diagnostic information.

Allowing examinees multiple opportunities to answer test items correctly implies that the changes in latent traits or attributes would occur over attempts. In sequential IRT models for repeated attempts, Culpepper (2014) incorporated a linear growth model for assessing the changes in examinees’ latent continuous traits. Note that examinees’ latent traits are not necessarily assumed to increase over attempts in the latent growth model because the growth rate may be negative as they practice different trials. In the sequential CDMs, latent attributes that examinees possess on each attempt are expected to change across multiple measurements, as in sequential IRT models. For the assessment of attribute changes over time, latent transition analysis (LTA) for CDMs (Kaya & Leite, 2017; Li, Cohen, & Bottge, 2016) and multilevel higher order CDMs (Huang, 2017; Huang & Hung, 2014) have been developed for longitudinal surveys in the literature. However, the two approaches do not consider the sequential process of cognitive operations and do not serve as ideal models for analyzing answer-until-correct test data.

In this study, sequential CDMs were developed by the authors to account for the attribute mastery statuses of examinees on multiple successive diagnostic assessments. Variations between the different measurement occasions (i.e., multiple attempts) in the mastery statuses of the individuals with respect to a set of defined attributes can be modeled using the LTA approach or a higher order CDM approach (Huang, 2017; Huang & Hung, 2014; Kaya & Leite, 2017; Li et al., 2016). This article is organized as follows. The most general CDM of the generalized deterministic-input noisy-“and”-gate (G-DINA) model (de la Torre, 2011) is adopted and extended in this study, which is introduced in Online Appendix A in the online supplemental material. The next section describes the sequential CDM framework and discusses the development and extension of the model by originally combining the sequential process model and the G-DINA model, followed by a description of two approaches to modeling changes in the latent attributes. A series of simulations are conducted to assess the parameter recovery and classification accuracy of the examinees, using Bayesian estimation to calibrate the model parameters. An empirical analysis is presented to demonstrate the applications and implications of the proposed models. The final section draws conclusions about the new models and provides suggestions for future research.

Sequential CDM Framework

When allowing examinees multiple attempts on a diagnostic assessment, given a maximum number of attempts T, the number of attempts that examinee i requires to submit a correct response to item j can be recorded and it is denoted as $Y_{ij}$ ( $Y_{ij} = 1, \dots, T$ ). Because the sequential process is modeled by producing the conditional probabilities of a correct response on an attempt given failure on the previous attempt, let $X_{ij} = (X_{ij 1}, \dots, X_{ijT})$ be a vector representing the sequence of Bernoulli trials on multiple attempts whose entry is equal to 1 if examinee i correctly answers item j on attempt m ( $m = 1, \dots, T$ ) and equal to 0 otherwise. The conditional probability of success for attempt m on item j for examinee i can be computed as follows:

P (X_{ijm} = 1 | X_{ij (m - 1)} = 0; α_{ljm (i)}^{*}, δ_{j}) = \frac{P (X_{ijm} = 1; α_{ljm (i)}^{*}, δ_{j})}{P (X_{ij (m - 1)} = 0; α_{ljm (i)}^{*}, δ_{j})} \equiv P (α_{ljm (i)}^{*}),

(1)

and the conditional probability of an incorrect response for attempt m is

P (X_{ijm} = 0 | X_{ij (m - 1)} = 0; α_{ljm (i)}^{*}, δ_{j}) = \frac{P (X_{ijm} = 0; α_{ljm (i)}^{*}, δ_{j})}{P (X_{ij (m - 1)} = 0; α_{ljm (i)}^{*}, δ_{j})} \equiv 1 - P (α_{ljm (i)}^{*}),

(2)

where $α_{ljm (i)}^{*}$ is the reduced attribute profile l that examinee i has for item j on attempt m ( $m = 1, \dots, T$ ), $δ_{j}$ is the G-DINA model parameter vector for item j and is assumed to be invariant across attempts, and $P (α_{ljm (i)}^{*})$ is the probability of success on item j for examinee i with attribute pattern l on attempt m, which can be directly applied from Equation A1 (given in Online Appendix A) in the G-DINA model framework. Note that when examinees make the first attempt, the conditional probability of success is simplified as

P (X_{ij 1} = 1 | X_{ij 0} = 0; α_{lj 1 (i)}^{*}, δ_{j}) = P (X_{ij 1} = 1; α_{lj 1 (i)}^{*}, δ_{j}) .

(3)

Therefore, the probability of making $m_{ij}$ attempt(s) before recording a correct answer for examinee i on item j and the associated probability mass function can be formulated as follows, respectively:

P (Y_{ij} = m_{ij} | α_{lj 1 (i)}^{*}, \dots, α_{ljT (i)}^{*}, δ_{j}) = Π_{w = 1}^{m_{ij} - 1} [1 - P (α_{ljw (i)}^{*})] \times P (α_{ljm (i)}^{*}),

(4)

if $1 \leq m_{ij} \leq T$ and

P (Y_{ij} = m_{ij} | α_{lj 1 (i)}^{*}, \dots, α_{ljT (i)}^{*}, δ_{j}) = Π_{w = 1}^{T} [1 - P (α_{ljw (i)}^{*})],

(5)

if $m_{ij} > T$ . When a sample of I examinees’ attempts to J items are collected, the likelihood function can be presented as follows:

L (Y | α, δ) = Π_{i = 1}^{I} Π_{j = 1}^{J} P (Y_{ij} = m_{ij} | α_{lj 1 (i)}^{*}, \dots, α_{ljT (i)}^{*}, δ_{j}) .

(6)

Because examinees are allowed multiple attempts on test items, the probability of a correct response would change as the number of attempts increases. In such cases, examinees can practice multiple times on test items and may have the opportunity to make use of additional materials through partial feedback in educational interventions to supplement their content knowledge or to refine the response strategies that are related to problem solving. Therefore, attribute mastery statuses of examinees are treated as dynamic over attempts, which is similar to the approach adopted in sequential IRT models for answer-until-correct tests (Culpepper, 2014). In addition to the changes in attribute mastery statuses, items may be easier in later attempts than in early attempts, and item parameter drift may occur. It is thus possible to add an additional item parameter to capture the variation of the item parameter over time for each attempt except for the first attempt. Item parameter invariance over time or attempts is a common assumption in longitudinal surveys (e.g., Huang, 2015, 2017) and item parameter drift may be caused by several factors (Park, Lee, & Xing, 2016). Following reasonable assumptions and to avoid reaching beyond the focus of this study, it can be assumed that the item parameters are invariant over time in this study.

Figure B1 of Online Appendix B illustrates an example in which the maximum number of attempts is three for examinee i in response to item j with a reduced attribute vector $α_{lj}^{*}$ . The reduced attribute vector $α_{lj}^{*}$ that an examinee has could change from one attempt to the next, and transitions in the latent binary variables among the attempts could be caused by factors that arise when students have the opportunity to use notes or other resources to focus on problem-solving strategies by providing partial feedback or instructional intervention. Any changes in the attributes over attempts can be modeled by two different approaches that have been applied in longitudinal diagnostic assessment analyses under the CDM framework; these two approaches are adopted in this study to account for variations in the changes in the latent attributes.

LTA Approach

Stage sequential changes in multiple attributes can be captured by the transition probabilities between two adjacent measurement occasions through LTA (Kaya & Leite, 2017; Li et al., 2016). For notational convenience, the subscript i that indicates a specific person is omitted in the following formulation. Let the entry in $X_{j} = (X_{j 1}, \dots, X_{jT})$ denote the response to item j on attempt m and $X_{jm} = 1$ indicate the correct response, whereas $X_{jm} = 0$ otherwise. The probability function of a given response pattern of $X = (X_{1}, \dots, X_{J})$ is expressed as follows:

P (X) = \sum_{α_{ljm} = 0}^{1} Π_{k = 1}^{K_{j}^{*}} P (α_{lj 1 k}) Π_{m = 2}^{T} Π_{k = 1}^{K_{j}^{*}} P (α_{ljmk} | α_{lj (m - 1) k}) Π_{m = 1}^{T} Π_{j = 1}^{J} P (X_{jm} = 1 | α_{ljm}^{*}),

(7)

where $P (α_{lj 1 k})$ is the probability of membership that masters (i.e., $α_{lj 1 k} = 1$ ) or does not master (i.e., $α_{lj 1 k} = 0$ ) attribute k for latent class l and item j on the first attempt, which can be assumed to follow an independent Bernoulli distribution for each attribute or multivariate Bernoulli distribution for all attributes using the tetrachoric correlation approach (as used in the following simulation study); $P (α_{ljmk} | α_{lj (m - 1) k})$ is the probability of a transition in the mastery/nonmastery status on attempt m conditional on membership on attempt $m - 1$ and is assumed to follow a conditional Bernoulli distribution; and $P (X_{jm} = 1 | α_{ljm}^{*})$ is the probability of a correct response to item j on attempt m given an attribute profile of $α_{ljm}^{*}$ , which can be defined by the G-DINA model, as indicated in Equation A1.

Higher Order Modeling Approach

Because the attributes are seldom independent and their correlations should be considered, a higher order latent variable can be assumed to govern the mastery status of each attribute, and higher order CDMs have been developed to describe the dependences of the multiple attributes (de la Torre & Douglas, 2004). Multilevel extension of the higher order CDMs provides an alternative to assessing the changes in the latent attributes and to simultaneously estimating the continuous latent trait, which can serve as an overall evaluation of the examinees’ performances (Huang, 2017; Huang & Hung, 2014). Let $θ_{im}$ be the continuous latent trait and $α_{imk}$ be the mastery status of the kth attribute for examinee i on attempt m, where $α_{imk} = 1$ indicates that the attribute is present and $α_{imk} = 0$ otherwise. The probability of mastering attribute k on attempt m for examinee i can be expressed as follows:

P (α_{imk} = 1 | θ_{im}) = \frac{\exp [λ_{1 k} (θ_{im} - λ_{0 k})]}{1 + \exp [λ_{1 k} (θ_{im} - λ_{0 k})]},

(8)

where λ_1k is the discrimination parameter and λ_0k is the location parameter for attribute k. The function described by Equation 8 is equivalent to the response function of the two-parameter logistic model (2PLM), but the outcome variables in Equation 8 are latent rather than observed, as in the 2PLM. As with the G-DINA model parameters described above, the λ_1k and λ_0k parameters are assumed to be time-homogeneous and are estimated to be equal for each attempt. Combining Equations A1 and 8 leads to a higher order G-DINA model when translating the full attribute vector $α_{im} = (α_{im 1}, \dots α_{imK})$ to the reduced attribute vector $α_{ljm (i)}^{*} = (α_{ljm 1 (i)}, \dots α_{{ljmK}_{j}^{*} (i)})$ for each item.

In the higher order CDMs, such as the higher order G-DINA model, both $θ$ and $α$ should be estimated. When the assumption of local independence is satisfied conditional on $θ$ , the probability of latent class l for examinee i on attempt m is

P_{l} (α_{im} | θ_{im}) = Π_{k = 1}^{K} P (α_{imk} | θ_{im}) .

(9)

Therefore, the likelihood function of $θ$ for examinee i on attempt m can be written as

L (θ_{im} | α_{im}) = \sum_{l = 1}^{2^{k}} P_{l} (α_{im} | θ_{im}) P_{l} (α_{im}),

(10)

where $P_{l} (α_{im})$ is the posterior probability of latent class l for examinee i on attempt m. With respect to the likelihood function, the Bayesian expected a posteriori (EAP) estimate or the maximum a posteriori estimate can be used to estimate the $θ$ parameters over attempts.

To model the latent growth for the $θ$ variables, the higher order G-DINA model can be extended to have a multilevel structure. In this study, a linear growth model is developed to account for variations in the latent traits over a single person’s attempts. The model is expressed as follows:

θ_{im} = η_{0 i} + (m - 1) η_{1 i} + ε_{im},

(11)

where $η_{0 i}$ and $η_{1 i}$ are the initial status and growth rate of examinee i, respectively, and are assumed to follow a bivariate normal distribution; $ε_{im}$ is the regression residual for examinee i on attempt m and is assumed to be normally distributed with a mean of 0 and a variance of $σ_{ε (m)}^{2}$ . $η_{0 i}$ , $η_{1 i}$ , and $ε_{im}$ are assumed to be independent of the other ε and η values. Specifically, when an examinee is administered a test on four occasions, under the linear growth assumption, his or her latent trait level is equal to $η_{0} + ε_{0}$ , $η_{0} + η_{1} + ε_{1}$ , $η_{0} + 2 \times η_{1} + ε_{2}$ , and $η_{0} + 3 \times η_{1} + ε_{3}$ for the initial status, second-measurement, third-measurement, and fourth-measurement occasions, respectively. For model identification, the mean of $η_{0}$ is constrained to equal 0, and one of the discrimination parameters is set to an arbitrary value (e.g., 1).

Note that although the higher-order-structured parameters (i.e., the $λ$ parameters) are set to be equal across times or attempts, as in the estimation of the G-DINA model parameters, the measurement invariance assumptions regarding the scalar and metric parameters may be violated because attribute parameter drift can occur. Under such circumstances, a subscript m should appear on the $λ$ parameters, and certain constraints should be considered to build a common scale across different measurements. Measurement invariance over time in the multilevel higher order G-DINA model has been addressed in the literature, and readers can refer to the study by Huang (2017).

The research hypotheses and expectations formulated in this study are as follows: (a) the model and person parameters can be recovered satisfactorily in the sequential G-DINA model using the LTA and higher order modeling approaches with the use of Bayesian estimation; (b) a large sample size can enable more precise estimations of the model parameters; (c) the LTA approach can provide higher accuracy classifications than the higher order modeling approach because the transition probabilities between two attempts are directly modeled rather than through a higher order structure; and (d) the use of the sequential G-DINA model to fit the empirical longitudinal data would be more efficient than the use of the non-sequential G-DINA model because fewer items are administered to examinees if they correctly answer the items on early attempts.

Method

Simulation Design

Two simulation studies were conducted to assess the quality of parameter estimation in the sequential G-DINA model with respect to two different approaches of modeling the changes in examinees’ mastery statuses over attempts. The first simulation used the LTA approach, and the simulation design was adapted from an empirical analysis performed by Li et al. (2016). Because the LTA approach was extended in the sequential G-DINA model, the simulation design and generated values were set to be similar to those reported in a previous study rather than simulating the conditions arbitrarily. The test length was fixed to 23 items that measured four attributes, and the construction of the real-data Q-matrix mapping each item to the four attributes is presented in Table C1 of Online Appendix C. The maximum number of attempts was set to four, and the transition probabilities for the four measurement occasions are listed in Table 1 (also see Li et al., 2016).

Table 1.

Transition Probabilities for Attempt Changes.

Attributes	Attempts 1 to 2		Attempts 2 to 3		Attempts 3 to 4
Attribute 1	0.55¹	0.45²	0.66¹	0.34²	0.05¹	0.95²
Attribute 1	0.03³	0.97⁴	0.06³	0.94⁴	0.01³	0.99⁴
Attribute 2	0.71¹	0.29²	0.82¹	0.18²	0.12¹	0.88²
Attribute 2	0.03³	0.97⁴	0.05³	0.95⁴	0.03³	0.97⁴
Attribute 3	0.93¹	0.07²	0.90¹	0.10²	0.65¹	0.35²
Attribute 3	0.87³	0.13⁴	0.54³	0.46⁴	0.51³	0.49⁴
Attribute 4	0.83¹	0.17²	0.74¹	0.26²	0.07¹	0.93²
Attribute 4	0.54³	0.46⁴	0.12³	0.88⁴	0.12³	0.88⁴

Note. The superscript indicates four transition conditions, where the values of 1, 2, 3, and 4 denote the following conditions: remaining in nonmastery status, transitioning from nonmastery to mastery status, transitioning from mastery to nonmastery status, and remaining in mastery status, respectively.

It is likely that certain misconceptions arise when examinees receive more instructional materials on the following measurements (National Research Council, 1997). Therefore, the probability of transitioning from mastery to nonmastery status may be greater than that of remaining in mastery status. In addition, difficult cognitive attributes, for example, problem-solving skill, could result in a higher transition probability from mastery to nonmastery status (Li et al., 2016). The generated transition probability parameters (see Table 1) indicate that a few conditions involved a rather larger probability for the transition from mastery to nonmastery status. For example, the transition probability for the abovementioned situation was sufficiently large from Attempt 1 to 2 but decreased dramatically from Attempt 2 to 3. Acknowledging the possible and occasional backwardness in attitudes, although such a situation is rare in answer-until-correct test administrations, any increasing monotonicity for attribute mastery profile in this study is not assumed.

To generate the mastery proportions of the four attributes similarly to the study of Li et al. (2016), the tetrachoric correlations among the attributes on the first attempt were fixed to .5, and the mastery thresholds for each attribute were set to 0.08, –0.03, 1.28, and 0.58; consequently, the proportions of the examinees who mastered each of the four respective attributes on the first attempt were 0.47, 0.51, 0.10, and 0.28. After generating the attribute mastery status on the first attempt for each examinee, the attribute mastery status of the following attempts could be determined according to the transition probabilities listed in Table 1. Because the saturated G-DINA model was used to generate the item responses and the stabilities of the parameter estimates could be influenced by the number of observations (e.g., de la Torre, 2011), samples of 500 and 1,000 examinees were employed to assess the effects of the sample size on the parameter estimation in the developed sequential G-DINA model.

The parameters in the G-DINA model were generated as follows. When one attribute was measured (i.e., when $K_{j}^{*} = 1$ ), $δ_{jo}$ and $δ_{j 1}$ were set to .05 and .75, respectively. The authors set $δ_{jo} = . 05$ , $δ_{j 1} = δ_{j 2} = . 30$ , and $δ_{j 12} = . 15$ for items that measured two attributes (i.e., when $K_{j}^{*} = 2$ ). The authors set $δ_{jo} = . 05$ , $δ_{j 1} = δ_{j 2} = δ_{j 3} = . 15$ , $δ_{j 12} = δ_{j 13} = δ_{j 23} = . 05$ , and $δ_{j 123} = . 15$ for the items that measured three attributes (i.e., when $K_{j}^{*} = 3$ ). The authors set $δ_{jo} = . 05$ , $δ_{j 1} = δ_{j 2} = δ_{j 3} = δ_{j 4} = . 05$ , $δ_{j 12} = δ_{j 13} = δ_{j 14} = δ_{j 23} = δ_{j 24} = δ_{j 34} = . 05$ , $δ_{j 123} = δ_{j 124} = δ_{j 134} = δ_{j 234} = . 05$ , and $δ_{j 1234} = . 05$ for the items that measured four attributes (i.e., when $K_{j}^{*} = 4$ ). Consequently, the probability of success on item j was equal to .05 for the examinees who had mastered none of the required attributes and .80 for the examinees who had mastered all of the required attributes. Designs similar to that employed in this study have been reported (e.g., Chen, de la Torre, & Zhang, 2013; Huang, 2017; Kunina-Habenicht, Rupp, & Wilhelm, 2012).

In the second simulation study, a higher order model was used to generate examinees’ mastery statuses of the four attributes; the other settings were set the same as in the first simulation study. The responses of a large sample of 3,000 examinees were generated using the LTA approach and were calibrated using the higher order modeling approach to obtain stable parameter estimates. When the G-DINA model parameter estimates were set to the generated values, the parameter estimates of the higher order model (i.e., the discrimination, location, mean, variance, and covariance parameters) were used as the generated values for the higher order sequential G-DINA model. Thus, the results of the second simulation study could be compared with those of the first simulation study.

Therefore, the generated values for the four respective attributes were set to 1, 0.741, 0.710, and 1.927 for the discrimination parameters and 0.182, 0.274, 3.709, and 1.602 for the location parameters. The regression residual was randomly generated from a normal distribution with a mean of 0 and variances of 0.507, 0.033, 0.183, and 0.036 for the four attempts, and the $η_{0 i}$ and $η_{1 i}$ parameters were randomly sampled from a bivariate normal distribution that had a mean vector of $(0, 0.827)$ and a variance–covariance matrix of $[\begin{matrix} 4.038 & - 1.357 \\ - 1.357 & 0.462 \end{matrix}]$ .

For both simulation studies, each condition was replicated 30 times. The authors attempted to increase the number of replications to 100 for several conditions and found that the resulting estimates were very close to those obtained from 30 replications, which implied that the sampling variation was very small and the differences among them could be neglected. Therefore, 30 replications were deemed sufficient to provide reliable results, as in Bayesian CDMs involving fewer replications than those used in this study (e.g., de la Torre & Douglas, 2004, 2008; Huang & Wang, 2014).

Analysis

The computer program WinBUGS 1.4 (Spiegelhalter, Thomas, & Best, 2003), which implements Bayesian estimation with Markov Chain Monte Carlo (MCMC) methods, was used to calibrate the parameters in the developed sequential CDMs. The Metropolis-Within-Gibbs algorithm was used to construct a Markov chain to generate samples for parameter estimation in this study. For the two types of sequential G-DINA models, in addition to the G-DINA model parameters (i.e., the $δ$ parameters) and the attribute mastery classifications, the attribute threshold, attribute correlation, and transition probability parameters should be estimated in the LTA approach, and the discrimination, location, random-slope mean, residual variance, covariance-variance parameters should be estimated in the higher order modeling approach.

Prior distributions should be specified for each model parameter before simulating the Markov chain to obtain the target density of the parameters. The settings of the prior distributions described below were consistent with or similar to those reported in previous studies involving Bayesian estimation in CDMs (e.g., Huang, 2017; Huang & Hung, 2014; Huang & Wang, 2014; Li et al., 2016). A beta distribution with both hyperparameters equal to 1 was used for the G-DINA model probability and transition probability parameters; a normal prior with a mean of 0 and a variance of 4 was used for the threshold, location, and random-slope mean parameters; a uniform distribution between −1 and 1 was used for the attribute correlations; a lognormal prior with a mean of 0 and a variance of 1 was used for the discrimination parameters; a gamma prior with both hyperparameters equal to 0.1 was used for the inverse residual variances; and finally, a Wishart distribution with a diagonal matrix whose nonzero entries were all equal to 0.1 and with two degrees of freedom was used as the prior for the inverse variance–covariance matrix.

The multivariate potential scale reduction factor (Brooks & Gelman, 1998) was used as the criterion for verifying the convergence of the parameter estimates using three parallel chains for five randomly selected simulated data points in each condition. Consequently, the authors performed 15,000 iterations to calculate the posterior mean for the parameter estimates over the iterations after the first 5,000 iterations were used as burn-in. Parameter estimates were then computed by averaging the samples from the remaining 10,000 iterations with a thinning interval of five. In addition, label switching is one major challenge associated with mixture models when Bayesian estimation is implemented for parameter calibration. In the simulation and empirical analyses, no multiple nodes in the marginal posterior distributions were observed for the specified parameters whether within a single MCMC chain or when using different initial values, implying that label switching did not occur. The quality of the model parameter estimation was evaluated by computing the bias and root mean square error (RMSE) for each estimator, and the recovery of the examinees’ attribute mastery statuses was assessed by computing the correct classification rate (CCR) for each individual attribute.

Results

The mean and standard deviation of the bias and RMSE are provided for different types of parameters instead of for individual parameters, owing to space constraints. Tables 2 and 3 summarize the results of the parameter recovery for the sequential G-DINA model using the LTA and higher order modeling approaches, respectively. The two approaches estimated a common set of G-DINA model parameters but used different schemes to account for the dependences of the attributes and to model the stage sequential change. The results show that the two approaches yielded satisfactory parameter recovery for the G-DINA model parameters, and the difference between the two approaches was very small. For the LTA approach, attribute correlations, mastery thresholds, and transition probabilities were estimated acceptably, and the large sample size provided more precise parameter estimation. For the higher order modeling approach, relatively larger bias and RMSE values were observed for the discrimination, location, residual variance, and covariance–variance parameters. This result was obtained because the quality of the latent continuous trait parameter estimation in the higher order CDMs depended strongly on the number of measured attributes (Hsu & Wang, 2015) and the latent growth trajectory would be influenced by the imprecise latent continuous trait estimation (Huang, 2017; Huang & Hung, 2014). Nonetheless, as shown in Table 2, from sample sizes of 500 to 1,000, satisfactory parameter recovery could be obtained when the large sample size was used.

Table 2.

Parameter Recovery for the LTA-Approach Sequential G-DINA Model.

Sample size		500		1,000
Parameter		Bias	RMSE	Bias	RMSE
G-DINA	M	0.004	0.026	0.001	0.019
G-DINA	SD	0.007	0.015	0.004	0.010
Threshold	M	−0.077	0.177	−0.128	0.159
Threshold	SD	0.192	0.147	0.194	0.180
Correlation	M	−0.018	0.100	−0.019	0.074
Correlation	SD	0.020	0.029	0.018	0.016
Transition probability	M	−0.012	0.058	−0.004	0.041
Transition probability	SD	0.032	0.047	0.017	0.039

Note. LTA = latent transition analysis; G-DINA = generalized deterministic-input noisy-“and”-gate; RMSE = root mean square error.

Table 3.

Parameter Recovery for the Higher Order Approach Sequential G-DINA Model.

Sample size		500		1,000
Parameter		Bias	RMSE	Bias	RMSE
G-DINA	M	0.006	0.032	0.001	0.020
G-DINA	SD	0.010	0.018	0.005	0.013
Discrimination	M	0.055	0.392	0.099	0.230
Discrimination	SD	0.107	0.382	0.164	0.205
Location	M	0.218	0.361	0.085	0.232
Location	SD	0.264	0.238	0.124	0.177
Residual variance	M	0.224	0.350	0.126	0.191
Residual variance	SD	0.059	0.084	0.032	0.082
Covariance–Variance	M	0.105	0.588	0.019	0.291
Covariance–Variance	SD	0.516	0.402	0.137	0.183

Note. G-DINA = generalized deterministic-input noisy-“and”-gate; RMSE = root mean square error.

Figure 1 shows the mean CCR for the two approaches across replications for each individual attribute in the four attempts. Although the LTA approach had better attribute recoveries than the higher order modeling approach in most conditions, the mastery status of the attributes appeared to be classified accurately for both approaches because of the higher CCR values. Because Attributes 3 and 4 were measured by a moderate number of items, 8 and 9 items, respectively, in contrast to Attributes 1 and 2, which were measured by a large number of items, 19 and 15 items, respectively, it was not surprising that Attributes 3 and 4 had lower CCR values than those of Attributes 1 and 2. In summary, with a sufficiently large sample, the parameters in the sequential G-DINA model using the two different approaches to account for the dependences of the attributes over attempts were recovered accurately using Bayesian estimation.

Empirical Demonstration

The data used for application demonstration were obtained from a longitudinal basic-ability-assessment survey in which a mathematical test was administered to 3,527 Taiwanese junior high school students at three points in time (Chang, 2007). Four attributes, namely, analysis, numbers, geometry, and algebra, were identified by content experts for a subset of the mathematical test; therefore, seven multiple-choice items were analyzed. The application of the sequential G-DINA model to the multiple-choice items was justified because the mathematical test was treated as a low-stakes examination and because multiple-choice tests have been widely used in the answer-until-correct scoring procedure (e.g., Ben-Simon, Budescu, & Nevo, 1997; Epstein et al., 2001).

The Q-matrix for the real data is presented in Table C2 of Online Appendix C. Because the sequential G-DINA model was formulated for multiple attempts, multiple measures in this example were treated as multiple attempts, and the longitudinal data were converted to a set of sequential responses by removing the responses of the examinees who answered an item correctly on the early attempt but failed on the later attempt. Specifically, for a given item, only the attempt patterns of (1, 1, 1), (0, 1, 1), (0, 0, 1), and (0, 0, 0) were included in the analysis. Note that removing the item responses that were not consistent with the sequential CDM assumption in this example did not imply that the attribute profile was assumed to increase monotonically because the attribute mastery statuses of examinees were allowed to change over attempts.

Table 4 presents the frequency and proportion of the sequential response patterns that satisfy the assumption of the sequential G-DINA model for each item. For each item, more than 60% of the examinees showed consistent response patterns, as specified by the sequential CDMs, and the responses of the 597 examinees who had consistent response patterns for all of the items were included for the fit of the developed model. Note that because a relatively small number of examinees and items was used, the data were originally collected in a longitudinal survey, and a possible misspecification of the Q-matrix could occur; thus, the results should be interpreted with caution.

Table 4.

Frequencies and Percentages for the Specified Attempt Patterns in the Longitudinal Mathematical Test.

Item	Attempt pattern	Frequency
1	(1, 1, 1)	805
	(0, 1, 1)	1100
	(0, 0, 1)	478
	(0, 0, 0)	443
	Percentage	80
2	(1, 1, 1)	934
	(0, 1, 1)	809
	(0, 0, 1)	446
	(0, 0, 0)	551
	Percentage	78
3	(1, 1, 1)	1264
	(0, 1, 1)	577
	(0, 0, 1)	281
	(0, 0, 0)	338
	Percentage	70
4	(1, 1, 1)	1204
	(0, 1, 1)	731
	(0, 0, 1)	432
	(0, 0, 0)	310
	Percentage	76
5	(1, 1, 1)	647
	(0, 1, 1)	493
	(0, 0, 1)	549
	(0, 0, 0)	509
	Percentage	62
6	(1, 1, 1)	2196
	(0, 1, 1)	420
	(0, 0, 1)	211
	(0, 0, 0)	106
	Percentage	83
7	(1, 1, 1)	1473
	(0, 1, 1)	473
	(0, 0, 1)	228
	(0, 0, 0)	321
	Percentage	71

Note. The total number of examinees was 3,257.

The LTA-approach sequential G-DINA model was used to fit the data because it provided more precise parameter estimation and more accurate attribute classification in the simulation study. The same dataset was fit to the LTA-approach G-DINA model developed for assessing the changes in the attributes (Li et al., 2016). A comparison between the two models provides evidence to support the efficacy of the LTA-approach sequential G-DINA model when the attribute classifications of the two models are comparable. For the fit of the LTA-approach G-DINA model, the examinees’ responses to seven items across three measurement times were included for data analysis, whereas each item was scored polytomously with four ordered response categories, where scores of 1, 2, 3, and 4 were recorded for the attempt patterns of (1, 1, 1), (0, 1, 1), (0, 0, 1), and (0, 0, 0), respectively, when the LTA-approach sequential G-DINA model was used.

Table 5 presents the estimated successful probability of the seven items for each of the reduced attribute profiles in the LTA-approach sequential G-DINA model. Note that different items can require different sets of attributes even though they have the same reduced attribute profile. It can be observed that the majority of items functioned in the manner that the disjunctive (or compensatory) CDMs assumed; therefore, highly constrained CDMs, such as the DINA model, were not the best choice for the fitting model. Result patterns similar to those of the LTA-approach sequential G-DINA model can be observed in the LTA-approach G-DINA model, although the results are not discussed in detail, and the correlation of the estimated probabilities between the two models was as high as .88. Because item exposure control should be considered in a large-scale assessment, further scrutiny of test contents in terms of different probabilities for different attribute patterns was impractical in this empirical analysis demonstration.

Table 5.

Probabilities of Success on the Seven Items for Attribute Mastery Patterns.

	Attribute pattern
	0	1
	00	10	01	11
Item	000	100	010	001	110	101	011	111
1	.097	.420	.489	.990
2	.090	.573	.312	.968
3	.334	.615	.381	.725
4	.206	.574	.349	.253	.798	.674	.682	.975
5	.081	.240	.474	.869
6	.132	.621	.344	.872	.964	.998	.991	.999
7	.051	.313	.086	.775	.799	.993	.952	.999

Finally, the attribute classification consistency between the two models was examined, and a higher consistency rate indicated that the LTA-approach sequential G-DINA model was comparable with the LTA-approach G-DINA model and was thus more efficient because the test item would be terminated thereafter if an examinee could correctly answer it on an attempt. The results indicate that the mean consistency rates between the two models over the four attributes were .88, .90, and .91 for the three respective attempts, which suggests that the LTA-approach sequential G-DINA model can serve as an efficient and alternative method when allowing examines to answer test items over repeated attempts.

Summary and Discussion

Allowing examinees multiple attempts on test items until they correctly answer questions has received increasing attention and offers many advantages. Sequential IRT models have been introduced for such occasions when latent continuous traits are intended to be measured (Culpepper, 2014). Cognitive diagnostic assessments can be administered to examinees multiple times and allow the examinees multiple opportunities to submit a correct response. Therefore, a new class of sequential CDMs for identifying examinees’ mastery statuses of the attributes over attempts was developed in this study, and two formulations in the sequential G-DINA model were introduced to model the dependences of the attributes on repeated attempts.

The results of two simulation studies demonstrated that the LTA-approach sequential G-DINA model performed better than the higher order approach sequential G-DINA model in both model parameter estimation and attribute classification. A larger sample size increased the accurate item and person parameter estimation, as documented in previous studies (e.g., de la Torre, Hong, & Deng, 2010; Huang & Wang, 2014). Because the simulation experiment should be as reasonable as practical CDM assessments, the generated values were adopted from a real data analysis study (Li et al., 2016), and several factors were not manipulated in this study. For example, different attribute structures could influence the model parameter estimations, and examinees using multiple strategies could interfere with the sequential process; these issues are left for future study.

A longitudinal mathematical test was implemented in the application of the sequential G-DINA to real data analysis because there is little empirical research regarding answer-until-correct diagnostic assessments to date. The responses to each item that were correct on an early attempt but incorrect on a later attempt were removed from the data analysis because the sequential CDMs assume that once examinees succeed on a question in a trial, the following attempts on that question are treated as successful. The LTA approach to modeling the stage sequential change for the attributes was used in the empirical analysis. Comparing the results obtained by fitting the LTA-approach sequential G-DINA model with those obtained by fitting the LTA-approach G-DINA model provided supporting evidence regarding whether the developed model produced comparable estimation efficiency and served as an alternative when repeated attempts on test items were allowed in cognitive diagnostic assessments. The results of the empirical analysis indicated that the two models produced similar estimated probabilities for different attribute patterns and had high consistency in attribute classifications; therefore, the use of the developed model to fit the repeated-attempt data was justified. Because of item exposure control in the longitudinal ability assessment, as in other international large-scale assessments, the Q-matrix was determined by internal experts and was assumed to be valid in this study. However, the assumption should be examined, and validating the Q-matrix is strongly recommended in practical testing situations by using several empirically based methods for Q-matrix validation (e.g., de la Torre & Chiu, 2016).

It should be noted that the data used for demonstration in the empirical example pertained to a low-stakes assessment; therefore, a relatively high proportion of examinees who recorded a correct response to items on a previous attempt but incorrect responses on the following attempts was observed. Whether high-stakes assessments could increase the number of the response patterns that the sequential CDMs assume deserves further investigation. Nonetheless, test developers were encouraged to devote themselves to designing multiple-choice or constructed-response items with diagnostic function and to provide retrieval-based practice for students with the use of partial feedback that can improve learning as students proceed in a sequence of attempts (Grimaldi & Karpicke, 2012; Karpicke & Blunt, 2011; Karpicke & Grimaldi, 2012). In addition, with ongoing advances in computer technology, a test administration and partial feedback system can be constructed in an e-learning environment, and the effectiveness of diagnostic assessments can be improved when using e-learning tools.

Recently, Ma and de la Torre (2016) proposed a sequential CDM for polytomous responses, which could provide more information than the traditional CDMs. In contrast to the models developed in this study, which involved repeated attempts, Ma and de la Torre’s models focused on a sequence of ordered categorical responses within an item and assumed that an item consisted of a finite number of sequential steps and that each of the ordered categories represented a specific step that can be specified by a different set of attributes. Although the two models were formulated from different methodological perspectives, extending the proposed sequential G-DINA model to fit graded-response data that involve different sequential steps would be an interesting topic for future study.

Supplemental Material

Online_appendix – Supplemental material for A Sequential Process Model for Cognitive Diagnostic Assessment With Repeated Attempts

Supplemental material, Online_appendix for A Sequential Process Model for Cognitive Diagnostic Assessment With Repeated Attempts by Su-Pin Hung and Hung-Yu Huang in Applied Psychological Measurement

Footnotes

Acknowledgements

The authors thank the editor and two anonymous reviewers for their constructive comments on earlier drafts of this article. Finally, the authors would like to express their deepest appreciation to Professor Wen-Chung Wang for his persistent encouragement and substantial feedback on their research career over the past years.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The first author was supported by the Ministry of Science and Technology, Taiwan (Grant NO. 106-2410-H-006-057-MY2). The second author was supported by the Ministry of Science and Technology, Taiwan (Grant NO. 106-2628-H-845-001-MY2).

Supplemental Material

Supplemental material for this article is available online.

References

Akkermans

(2000). Modeling sequentially scored item responses. British Journal of Mathematical and Statistical Psychology, 53, 83-98.

Albert

Chib

(2001). Sequential ordinal modeling with applications to survival data. Biometrics, 57, 829-836.

Attali

(2011). Immediate feedback and opportunity to revise answers: Application of a graded response IRT model. Applied Psychological Measurement, 35, 472-479.

Attali

Powers

(2010). Immediate feedback and opportunity to revise answers to open-ended questions. Educational and Psychological Measurement, 70, 22-35.

Bechger

T. M.

Akkermans

(2001). A note on the equivalence of the graded response model and the sequential model. Psychometrika, 66, 461-463.

Ben-Simon

Budescu

D. V.

Nevo

(1997). A comparative study of measures of partial knowledge in multiple-choice questions. Applied Psychological Measurement, 21, 65-88.

Brooks

S. P.

Gelman

(1998). General methods for monitoring convergence of iterative simulations. Journal of Computational and Graphical Statistics, 7, 434-455.

Chang

L.-Y.

(2007). Taiwan education panel survey: Users’ guide and base year (2007) student questionnaire for junior high school. Taipei, Taiwan: Center for Survey Research, Academia Sinica.

Chen

de la Torre

Zhang

(2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50, 123-140.

10.

Culpepper

S. A.

(2014). If at first you don’t succeed, try, try again: Applications of sequential IRT models to cognitive assessments. Applied Psychological Measurement, 38, 632-644.

11.

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76, 179-199.

12.

de la Torre

Chiu

C.-Y.

(2016). A general method of empirical Q-matrix validation. Psychometrika, 81, 253-273.

13.

de la Torre

Douglas

J. A.

(2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333-353.

14.

de la Torre

Douglas

(2008). Model evaluation and multiple strategies in cognitive diagnosis: An analysis of fraction subtraction data. Psychometrika, 73, 595-624.

15.

de la Torre

Hong

Deng

(2010). Factors affecting the item parameter estimation and classification accuracy of the DINA model. Journal of Educational Measurement, 47, 227-249.

16.

DiBattista

Gosse

(2006). Test anxiety and the immediate feedback assessment technique. The Journal of Experimental Education, 74, 311-328.

17.

Epstein

M. L.

Epstein

B. B.

Brosvic

G. M.

(2001). Immediate feedback during academic testing. Psychological Reports, 88, 889-894.

18.

Grimaldi

Karpicke

(2012). When and why do retrieval attempts enhance subsequent encoding? Memory & Cognition, 40, 505-513.

19.

Hsu

C.-L.

Wang

W.-C.

(2015). Variable-length computerized adaptive testing using the higher order DINA model. Journal of Educational Measurement, 52, 125-143.

20.

Huang

H.-Y.

(2015). A multilevel higher-order item response theory model for measuring latent growth in longitudinal data. Applied Psychological Measurement, 39, 362-372.

21.

Huang

H.-Y.

(2017). Multilevel cognitive diagnosis models for assessing changes in latent attributes. Journal of Educational Measurement, 54, 440-480.

22.

Huang

H.-Y.

Hung

S.-P.

(2014, July). Measuring latent growth in the framework of cognitive diagnosis models. Paper presented at the 28th International Congress of Applied Psychology, Paris, French.

23.

Huang

H.-Y.

Wang

W.-C.

(2014). The random-effect DINA model. Journal of Educational Measurement, 51, 75-97.

24.

Huff

Goodman

D. P.

(2007). The demand for cognitive diagnostic assessment. In Leighton

J. P.

Gierl

M. J.

(Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 19-60). Cambridge, MA: Cambridge University Press.

25.

Karpicke

Blunt

(2011). Retrieval practice produces more learning than elaborative studying with concept mapping. Science, 331, 772-775.

26.

Karpicke

Grimaldi

(2012). Retrieval-based learning: A perspective for enhancing meaningful learning. Educational Psychology Review, 24, 401-418.

27.

Kaya

Leite

L. L.

(2017). Assessing change in latent skills across time with longitudinal cognitive diagnosis modeling: An evaluation of model performance. Educational and Psychological Measurement, 77, 369-388.

28.

Kluger

A. N.

DeNisi

(1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119, 254-284.

29.

Kunina-Habenicht

Rupp

A. A.

Wilhelm

(2012). The impact of model misspecification on parameter estimation and item-fit assessment in log-linear diagnostic classification models. Journal of Educational Measurement, 49, 59-81.

30.

Leighton

J. P.

Gierl

M. J.

(Eds.). (2007). Cognitive diagnostic assessment for education: Theory and applications. Cambridge, MA: Cambridge University Press.

31.

Cohen

Bottge

(2016). A latent transition analysis model for assessing change in cognitive skills. Educational and Psychological Measurement, 76, 181-204.

32.

de la Torre

(2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69, 253-275.

33.

Muñiz

Menéndez

(2011). The answer-until-correct item format revisited. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 7, 103-110.

34.

National Research Council. (1997). Science teaching reconsidered: A handbook. Washington, DC: National Academy Press.

35.

Park

Y. S.

Lee

Y.-S.

Xing

(2016). Investigating the impact of item parameter drift for item response theory models with mixture distributions. Frontiers in Psychology, 7, 255.

36.

Pressey

(1926). A simple apparatus which gives tests and scores—And teaches. School and Society, 23, 373-376.

37.

Rupp

A. A.

Templin

J. L.

Henson

R. A.

(2010). Diagnostic measurement: Theory, methods, and applications. New York, NY: Guilford Press.

38.

Samejima

(1969). Estimation of latent ability using a response pattern of graded scores (Psychometrika Monograph Supplement, No. 17). Richmond, VA: Psychometric Society.

39.

Spiegelhalter

D. J.

Thomas

Best

(2003). WinBUGS version 1.4 [Computer Program.]. Cambridge, UK: MRC Biostatistics Unit, Institute of Public Health.

40.

Spray

(1990). One-parameter item response theory models for psychomotor tests involving repeated, independent attempts. Research Quarterly for Exercise and Sport, 61, 162-168.

41.

Tutz

(1990). Sequential item response models with an ordered response. British Journal of Mathematical and Statistical Psychology, 43, 39-55.

42.

Fellouris

Culpepper

S. A.

Douglas

(2016). Sequential detection of learning in cognitive diagnosis. British Journal of Mathematical and Statistical Psychology, 69, 139-158.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.34 MB