A Bayesian Random Weights Linear Logistic Test Model for Within-Test Practice Effects

Abstract

The present paper introduces a random weights linear logistic test model for the measurement of individual differences in operation-specific practice effects within a single administration of a test. The proposed model is an extension of the linear logistic test model of learning developed by Spada (1977) in which the practice effects are considered random effects varying across examinees. A Bayesian framework was used for model estimation and evaluation. A simulation study was conducted to examine the behavior of the model in combination with the Bayesian procedures. The results demonstrated the good performance of the estimation and evaluation methods. Additionally, an empirical study was conducted to illustrate the applicability of the model to real data. The model was applied to a sample of responses from a logical ability test providing evidence of individual differences in operation-specific practice effects.

Keywords

linear logistic test model random effects ability to learn learning models Markov chain Monte Carlo

Introduction

Research in item response theory has led to considerable advances in measuring individual differences in learning between different administrations of a test (e.g., Andersen, 1985; Bock, 1976; Embretson, 1991). However, the measurement of individual differences in learning within a test has been surprisingly understudied (Lozano & Revuelta, 2023). Within-test learning effects are likely to occur when the items share a set of solution principles that must be repeatedly applied throughout the test, so that respondents can improve their performance during the test as a result of practice. These effects may be facilitated by explicit feedback. For instance, after the respondent answers an item (or a set of items), the examiner (or the test in computerized form) may provide information about the correctness of the response(s) or about the cognitive operations that should have been applied to solve the item(s). However, note that feedback could also occur without the need for explicit information. For example, in a multiple-choice test, the fact that the respondent finds (or does not find) a response alternative that matches the solution he or she has arrived at provides information that such solution is probably correct (or certainly incorrect).

Most of the existing models aimed at measuring within-test learning effects treat learning as a fixed effect constant across examinees (e.g., Spada, 1977; Verguts & De Boeck, 2000; Verhelst & Glas, 1993). Of particular interest for the present paper is the model developed by Spada (1977; see also Fischer & Formann, 1982; Lozano & Revuelta, 2021a; Spada & McGaw, 1985) on the basis of the linear logistic test model (LLTM; Fischer, 1973; Scheiblechner, 1972). This model was conceived to account for within-test learning effects specifically associated with the cognitive operations required to solve the test items. The model assumes that the operation-specific learning effects are the same for all examinees, which may be a too restrictive assumption in some instances, particularly when the items involve relatively complex operations with which examinees do not have previous experience. In such circumstances, examinees may show quite different learning curves throughout the test, which would make the model produce incorrect estimates of person and item parameters. In the present paper, a generalization of this model is proposed in which the practice effects are treated as random effects varying across examinees. This provides the model with greater flexibility, allowing it to account for a wider variety of learning patterns as well as to yield a more precise analysis of person and item properties. The proposed model is therefore a multidimensional model able to measure individual differences in operation-specific learning within a single administration of a test.

A Random Weights Linear Logistic Test Model for Learning

This section describes the model specification and identification.

Model Specification

According to the proposed model, the logit of a correct response for person i (i = 1, 2, . . ., n) on item j (j = 1, 2, . . ., J) is given by:

logit [x_{i j} = 1] = θ_{i} - \sum_{m = 1}^{M} w_{j m} (α_{m} - δ_{i m} \sum_{k = 1}^{j - 1} w_{k m}),

(1)

where

θ_i is the ability of person i;

w_jm is the weight of item j on operation m (m = 1, 2, . . ., M);

α_m is the initial difficulty of operation m;

w_km is the weight of previous item k (k = 1, 2, . . ., j ‒ 1) on operation m; and

δ_im is the effect of previous practice on the difficulty of operation m for person i.

The model is based on a J × M structure matrix W that contains the weights w_jm. These weights represent the frequency with which operation m is required to solve item j. In addition, the model includes V, a J × M matrix that contains the weights v_jm, which represent the frequency with which operation m has been required in previous items times the frequency with which it is required in item j:

v_{j m} = w_{j m} \sum_{k = 1}^{j - 1} w_{k m} .

(2)

In other words, v_jm quantifies the number of times an operation-specific practice effect (δ_im) affects the response to item j. Consequently, Equation (1) could be rewritten as follows:

logit [x_{i j} = 1] = θ_{i} - \sum_{m = 1}^{M} w_{j m} α_{m} - \sum_{m = 1}^{M} v_{j m} δ_{i m} .

(3)

Table 1 shows an example of matrices W and V (for weights representing successively smaller practice effects, see Spada & McGaw, 1985).

Table 1.

Hypothetical Example of Matrices W and V.

	W					V
Item	1	2	3	4	5	1	2	3	4	5
1	1	1	0	0	0	0	0	0	0	0
2	1	0	1	1	0	1	0	0	0	0
3	1	1	1	1	0	2	1	1	1	0
4	0	2	1	1	1	0	4	2	2	0
5	1	1	2	1	1	3	4	6	3	1

The ability parameter θ_i represents the ability of person i to perform the operations involved in the test without practice. Since most psychometric tests are not meant to produce practice effects, θ_i may be conceived as the ability the test is intended to measure. The difficulty parameter α_m represents the initial difficulty of operation m; that is, the difficulty of operation m without practice. The practice parameter δ_im represents the effect of practicing operation m for person i. A positive sign for δ_im indicates a decrease in the difficulty of operation m for person i as a function of practice, which may be interpreted as learning. On the other hand, a negative sign for δ_im indicates an increase in the difficulty of operation m for person i as a function of practice, which may be interpreted as fatigue or loss of interest and/or attention. A value of zero for δ_im entails the absence of practice effect, and therefore that the difficulty of operation m for person i remains equal to α_m throughout the test. Note that the operation difficulty parameters (α_m) are fixed effects constant across examinees, whereas the practice parameters (δ_im) are random effects allowed to vary over individuals. The model is therefore a random weighs linear logistic test model (RWLLTM; Rijmen & De Boeck, 2002). According to this, we will refer to the model as random weighs operation-specific learning model (RWOSLM).

Model Identification

For the RWOSLM to be identified, the matrices W and V⁺ = (V, 1) (i.e., V supplemented with a column vector of ones), as well as the concatenation of both matrices, must have full column rank. That is, rank(W) = M, rank(V⁺) = M + 1, and rank(W, V⁺) = 2M + 1. Additionally, the number of effects should not exceed the number of items 2M + 1 ≤ J. Note however that in Bayesian estimation, the scale of the latent variable (θ_i) is fixed by the corresponding prior distribution and this restriction is relaxed to 2M ≤ J. Finally, for the model to be identified, the covariance matrix of the distribution of the random effects must be symmetric positive definite. This condition is violated when the random effects are linearly dependent or when the variance of a random effect equals zero (Rijmen & De Boeck, 2002).

Relation to Other Models

This section examines the relationship of the RWOSLM to other item response models.

Random Weights Linear Logistic Test Model

As mentioned in the previous section, the proposed model can be considered a RWLLTM (Rijmen & De Boeck, 2002). According to the RWLLTM, the logit of a correct response is given by:

logit [x_{i j} = 1] = θ_{i} - \sum_{m = 1}^{M_{1}} w_{j m} α_{m} - \sum_{m = M_{1} + 1}^{M} w_{j m} α_{i m},

(4)

where M₁ is the number of fixed effects (α_m) and M₂ = M ‒ M₁ is the number of random effects (α_im). In this regard, the RWOSLM is a RWLLTM in which the fixed effects (α_m) represent the operations' difficulties, the random effects (δ_im) represent the operations' practice effects, and the weights of the random effects (v_jm = w_jmΣw_km) count the incidence of previous practice on the response.

Operation-Specific Learning Model

A special case of the RWOSLM takes place when δ_im is constant over i ( $σ_{δ_{i m}}^{2} = 0$ for all m); that is, when the practice effects are the same for all examinees. In that case, the RWOSLM equals the linear logistic test model of learning developed by Spada (1977; see also Fischer & Formann, 1982; Lozano & Revuelta, 2021a; Spada & McGaw, 1985):

logit [x_{i j} = 1] = θ_{i} - \sum_{m = 1}^{M} w_{j m} (α_{m} - δ_{m} \sum_{k = 1}^{j - 1} w_{k m}),

(5)

where both α_m and δ_m are fixed effects and only the intercept θ_i is random over persons. We will refer to this model as operation-specific learning model (OSLM; Lozano & Revuelta, 2021a).

Linear Logistic Test Model

When δ_im is zero for all i and m, the RWOSLM equals the LLTM (Fischer, 1973; Scheiblechner, 1972):

logit [x_{i j} = 1] = θ_{i} - \sum_{m = 1}^{M} w_{j m} α_{m} .

(6)

The LLTM assumes the absence of practice effects during the test. Consequently, item difficulty, as measured by the Rasch model (β_j), is assumed to be exclusively a function of the marginal difficulties of the operations involved in solving the item:

β_{j} = \sum_{m = 1}^{M} w_{j m} α_{m} .

(7)

As can be appreciated, both the OSLM and the LLTM are Rasch models (Rasch, 1960) with linear constraints on the item parameters, whereas the RWOSLM and the RWLLTM would be constrained multidimensional Rasch models.

Bayesian Framework

This section describes a Bayesian framework for model estimation and evaluation.

Model Estimation

Marcov chain Monte Carlo (MCMC) simulation was used to derive an empirical approximation to the posterior distribution of the parameters (Brooks et al., 2011). Specifically, the No-U-Turn (NUTS) algorithm (Hoffman & Gelman, 2014) was used to iteratively draw samples from the posterior distribution. The mean of the posterior draws was used to summarize the estimation, whereas the variance and the quantiles of the posterior distribution were used as a measure of the precision of the estimation. The un-normalized posterior distribution is given by the product of the prior distribution of the parameters and the likelihood of the data given the parameters:

p (θ, α, δ | x, W) \propto p (θ, α, δ) p (x | θ, α, δ, W),

(8)

where x is the n × J matrix of item responses, α = (α₁, . . ., α_m, . . ., α_M) is the vector of fixed-effects parameters, and δ = (δ₁₁, . . ., δ_im, . . ., δ_nM) and θ = (θ₁, . . ., θ_i, . . ., θ_n) are the vectors of random-effects parameters. Assuming conditional independence of the responses given the model parameters as well as independence among subjects, the conditional likelihood of x can be written as:

p (x | θ, α, δ, W) = \prod_{i = 1}^{n} \prod_{j = 1}^{J} p (x_{i j} | θ_{i}, α, δ, W) .

(9)

Based on the independence assumption between the model parameters and on an exchangeability assumption, the joint prior distribution can be factored into the products of common prior distributions:

p (θ, α, δ) = \prod_{i = 1}^{n} p (θ_{i}) \prod_{m = 1}^{M} p (α_{m}) \prod_{i = 1}^{n} \prod_{m = 1}^{M} p (δ_{i m}) .

(10)

The following prior distributions were specified for each of the model parameters:

α_{m} ˜ normal (μ_{α}, σ_{α}),

(11)

θ_{i} ˜ normal (μ_{θ}, σ_{θ}), and

(12)

δ_{i m} ˜ mnormal (μ_{δ}, Σ_{δ}),

(13)

where

μ_{δ} = (μ_{δ_{1}}, . . ., μ_{δ_{m}}, . . ., μ_{δ_{M}})

is the vector of means of practice parameters, and Σ_δ is the variance-covariance matrix of practice parameters, with

μ_{δ_{m}} ˜ normal (μ_{μ_{δ}}, σ_{μ_{δ}}),

(14)

σ_{δ_{m}} ˜ Cauchy (x_{0_{σ_{δ}}}, γ_{σ_{δ}}), and

(15)

L ˜ LKJ (η),

(16)

where

L = R_{δ} R_{δ}^{'}

is the Cholesky factor of the correlation matrix of practice parameters.

Model Evaluation

Model evaluation was conducted using posterior predictive model checking (PPMC; Gelman et al., 1996). PPMC is based on test statistics that capture relevant features of the data. In PPMC, the realized value of the statistic based on the observed data, T(x), is compared to the value of the statistic based on the posterior predictive distribution, T(x^rep):

p (x^{rep} | x) = \int_{θ} \int_{α} \int_{δ} p (x^{rep} | θ, α, δ, W) p (θ, α, δ | x, W) d θ d α d δ,

(17)

where x^rep is the data reproduced by the model. The results are summarized by means of the posterior predictive p value (PPP value; Gelman et al., 1996; Meng, 1994), the tail-area of the posterior predictive distribution of the statistic corresponding to the realized value of the statistic:

PPP = p [T (x^{rep}) \geq T (x) | x] .

(18)

In this study, PPMC was conducted based on the odds ratio statistic (OR; Sinharay, 2005). The OR is a measure of association between pairs of items that has proven useful for detecting learning effects within a test (Lozano & Revuelta, 2021b, 2023). Measures of inter-item associations at the item and test level are obtained by summing the OR values over the pairs of items. PPP values close to .5 indicate that the realized value of the statistic is in the middle of the posterior predictive distribution, evidencing adequate model-data fit; whereas extreme PPP values close to zero (one) indicate that the observed data exhibit more (less) local dependence than expected based on the model.

Model Comparison and Selection

Model comparison and selection was based on two information criteria: the widely applicable information criterion (WAIC; Watanabe, 2013) and the leave-one-out information criterion (LOOIC; Vehtari et al., 2017). WAIC and LOOIC estimate the out-of-sample predictive performance adjusting the log predictive density of the observed data by penalizing for model complexity. Such penalty compensates for the over-fitting exhibited by more complex models by virtue of their higher flexibility. WAIC and LOOIC are used to compare competing models in order to select the one that fits the data best. Lower values indicate better predictive accuracy. WAIC and LOOIC have shown good performance in detecting learning effects within a test (Lozano & Revuelta, 2021b, 2023).

Simulation Study

A simulation study was conducted to examine the performance of the model in combination with Bayesian estimation and evaluation methods. Specifically, the study examined the performance of the NUTS algorithm in parameter recovery and the performance of PPMC and information criteria in model evaluation and selection.

Method

A 4 × 5 factorial design was used for the simulation study. The design results from the combination of four generating models (RWOSLM, OSLM, RWLLTM,¹ and LLTM) and five fitted models (the same four models plus the Rasch model). The combination of generating models and fitted models allowed us to study different conditions of misspecification. The sample size (N = 500), test length (J = 25), cognitive operations (M = 5), weight matrix (Table 2), and true values of the fixed-effects parameters (α_m = 2 and δ_m = 1) were kept constant across conditions. The true values of the random-effects parameters (θ_i, α_im, and δ_im) were randomly generated from a standard normal distribution. The matrix W was balanced so that each operation and combination of operations was measured the same number of times.

Table 2.

Matrix W Used in the Simulation Study.

Item	Operation					Item	Operation
Item	1	2	3	4	5	Item	1	2	3	4	5
1	1	0	0	0	0	14	0	0	1	0	1
2	0	1	0	0	0	15	0	0	0	1	1
3	0	0	1	0	0	16	1	1	1	0	0
4	0	0	0	1	0	17	1	1	0	1	0
5	0	0	0	0	1	18	1	1	0	0	1
6	1	1	0	0	0	19	1	0	1	1	0
7	1	0	1	0	0	20	1	0	1	0	1
8	1	0	0	1	0	21	1	0	0	1	1
9	1	0	0	0	1	22	0	1	1	1	0
10	0	1	1	0	0	23	0	1	1	0	1
11	0	1	0	1	0	24	0	1	0	1	1
12	0	1	0	0	1	25	0	0	1	1	1
13	0	0	1	1	0

The simulation was conducted with R version 3.6.1 (R Development Core Team, 2019) and the RStan R package version 2.19.2 (Stan Development Team, 2019). One hundred data sets of dichotomous responses were simulated from each generating model. The models were estimated from each simulated data set using four Markov chains of 2,000 iterations each. The draws from the first half of the iterations were discarded as burn-in. The potential scale reduction statistic ( $\hat{R}$ ; Gelman & Rubin, 1992) was used to evaluate the convergence of the chains. Values of $\hat{R}$ less than 1.2 are indicative of approximate convergence. No convergence issues were detected.

Based on previous research (Lozano & Revuelta, 2021b, 2023), a normal(0, 100) was specified as prior distribution for the fixed-effects parameters (α_m, β_j, and δ_m), whereas a normal(0, 1) was specified for the parameter θ_i. Additionally, a normal(0, 1) and a Cauchy(0, 5) were specified for the hyper-parameters $μ_{δ_{m}}$ and $σ_{δ_{m}}$ , respectively, whereas a LKJ(2) was specified for the Cholesky factor of the correlation matrix of practice parameters to regularize it to a unit correlation matrix.

PPMC based on the OR statistic was used to assess the fit of the models to the data. The hypothesis that the model fits the data was rejected when the PPP value was less than .05 or greater than .95. The performance of the OR statistic was assessed by the average PPP value over the 100 simulated samples and the empirical proportion of rejections (EPR), that is, the proportion of simulated samples in which the fitted model is rejected. When the fitted model coincides with the model used to generate the data, the EPR is an estimate of the false-positive error rate of the test, whereas when the fitted model and the generating model do not coincide, the EPR is an estimate of the sensitivity of the test.

Additionally, WAIC and LOOIC were used for model comparison and selection. These measures were obtained using the loo R package version 2.3.1 (Vehtari et al., 2020). For each condition of the study, the performance of WAIC and LOOIC was assessed by their average value over the 100 simulated samples and the empirical proportion of selections (EPS), that is, the proportion of simulated samples in which the fitted model is selected.

The accuracy of the parameter estimates was assessed with the root mean square error (RMSE). Lower values of RMSE indicate higher estimation accuracy. The RMSE of the parameter δ_im is defined as follows:

R M S E ({\hat{δ}}_{i m}) = \frac{1}{L} \sum_{l = 1}^{L} \sqrt{\frac{1}{N} \sum_{i = 1}^{N} \frac{1}{M} {\sum_{m = 1}^{M} ({\hat{δ}}_{\lim} - δ_{\lim})}^{2}},

(19)

where l (l = 1, 2, . . ., L) denotes a simulated sample, L is the number of simulated samples (L = 100),

{\hat{δ}}_{\lim}

is the expected a posteriori (EAP) estimate of the mth parameter δ for person i in sample l, and δ_lim is the true value of the parameter for the same person and sample. For a description of the RMSE of the parameters α_m and θ_i, see Lozano and Revuelta (2021b, 2023).

Results

Table 3 shows the mean PPP value and the EPR of the OR statistic for each combination of generating and fitted model. As expected, when the true or a more general model was fitted to the data, the OR statistic yielded mean values of PPP close to .5 and values of EPR close to zero, indicating high specificity. On the other hand, when the fitted model was not the true or a more general model, the OR statistic yielded mean values of PPP close to zero or one and values of EPR close to one, indicating high sensitivity.

Table 3.

Average Posterior Predictive p-value ( $\bar{PPP}$ ) and Empirical Proportion of Rejections (EPR) of the Odds Ratio for each Combination of Generating and Fitted Model.

	Fitted model
Generating model	RWOSLM		OSLM		RWLLTM		LLTM		Rasch
Generating model	$\bar{PPP}$	EPR	$\bar{PPP}$	EPR	$\bar{PPP}$	EPR	$\bar{PPP}$	EPR	$\bar{PPP}$	EPR
RWOSLM	.565	.00	.000	1.00	.001	1.00	.000	1.00	.000	1.00
OSLM	.541	.04	.533	.04	1.000	1.00	1.000	1.00	.547	.07
RWLLTM	.019	.95	.008	.99	.531	.00	.008	.99	.009	.98
LLTM	.594	.04	.418	.12	.583	.14	.486	.13	.305	.16

Table 4 shows the mean values of WAIC and LOOIC and the EPS for each combination of generating and fitted model. In general terms, both indices tended to select the true model, showing relatively good performance in model comparison and selection. Interestingly, the EPS was particularly high for the true model when the data were generated with the RWOSLM. However, when the data were generated with the OSLM, both indices tended to select the RWOSLM, which indicates a tendency to overestimate the presence of individual differences in practice effects.

Table 4.

Average WAIC and LOOIC (Mean) and Empirical Proportion of Selections (EPS) for each Combination of Generating and Fitted Model.

		Fitted model
Generating model		RWOSLM		OSLM		RWLLTM		LLTM		Rasch
Generating model	Statistic	Mean	EPS	Mean	EPS	Mean	EPS	Mean	EPS	Mean	EPS
RWOSLM	WAIC	7198.5	1.00	13221.0	.00	12199.0	.00	14248.0	.00	13008.1	.00
RWOSLM	LOOIC	7713.6	1.00	13223.3	.00	12286.1	.00	14250.2	.00	13010.3	.00
OSLM	WAIC	7785.8	.71	7799.6	.17	12042.2	.00	12022.6	.00	7801.2	.12
OSLM	LOOIC	7789.7	.65	7801.5	.20	12047.7	.00	12025.5	.00	7802.7	.15
RWLLTM	WAIC	17293.6	.00	17975.0	.00	16706.8	1.00	17969.5	.00	17991.8	.00
RWLLTM	LOOIC	17435.4	.00	17977.9	.00	16781.5	1.00	17972.3	.00	17994.8	.00
LLTM	WAIC	4191.1	.22	4193.3	.14	4191.4	.18	4187.6	.44	4209.4	.02
LLTM	LOOIC	4185.2	.38	4193.8	.16	4197.0	.10	4188.1	.35	4209.6	.01

Table 5 shows the RMSE of the parameter estimates for each combination of estimated parameter, generating model, and fitted model. As expected, for each generating model, the RMSE tended to be minimized when the true model was fitted to the data. In general terms, the results revealed inaccuracy in the estimates when a model other than the true model was fitted to the data, particularly when the data were generated with practice effects not considered in the fitted model. Additionally, when the RWOSLM was the generating and fitted model, the recovery of θ_i and α_m was comparable to that observed with other consolidated models such as the LLTM when they were the generating and fitted model. Similarly, when using the RWOSLM as generating and fitted model, the recovery of δ_im was better than that of α_im when the RWLLTM was the generating and fitted model [

RMSE ({\hat{α}}_{i m}) = 0.699

]².

Table 5.

Root Mean Square Error (RMSE) of Parameter Estimates for each Combination of Estimated Parameter, Generating Model, and Fitted Model.

Estimated parameter	Generating model	Fitted model
Estimated parameter	Generating model	RWOSLM	OSLM	RWLLTM	LLTM	Rasch
θ_i	RWOSLM	0.766	1.185	1.093	1.377	1.186
	OSLM	0.597	0.593	1.509	1.513	0.595
	RWLLTM	0.727	0.807	0.689	0.806	0.808
	LLTM	0.770	0.765	0.760	0.766	0.766
α_m	RWOSLM	0.103	0.631		1.792
	OSLM	0.132	0.111		3.348
	LLTM	0.136	0.112		0.088
δ_im	RWOSLM	0.566

Conclusions

The simulation study demonstrated the good performance of PPMC for model evaluation. Specifically, the OR statistic showed high specificity and sensitivity identifying the presence or absence of individual differences in learning affecting the data. Additionally, the information criteria demonstrated good performance in model comparison and selection. Nevertheless, a certain tendency to overestimate the presence of individual differences in practice effects suggests the need to complement the use of information criteria with PPMC. The results indicated that the MCMC algorithm provided accurate estimates for the model parameters.

Empirical Study

An empirical study was conducted to illustrate the applicability of the model to real data. The model was applied to a deductive reasoning test based on several logical operations that previous research has demonstrated to be prone to within-test practice effects (Lozano & Revuelta, 2021a, 2023).

Method

The data consist of responses from 501 examinees to 50 items of the DA5 logical ability test (SHL, 1995). Each item includes a sequence from two to four figures. Symbols are presented along with the figures representing the logical operations that must be applied to transform the figures. The test is based on 10 logical operations: (1) rotate the figure from top to bottom; (2) rotate the figure from left to right; (3) erase the previous figure; (4) erase the next figure; (5) interchange the figure with the previous one; (6) ignore the previous operator; (7) ignore the next operator; (8) reverse the order of the figures; (9) reorder the figures such that 1234 → 3412; and (10) reorder the figures such that 1234 → 2143. The respondent must choose between five response alternatives the one that represents the result of applying the logical operations to the figures. Figure 1 depicts a hypothetical example of DA5 item. The item involves operations 1, 6, 3, 5, and 10, in that order, and the correct response is D. Table 6 shows the hypothesized structure matrix W for the items. The matrices W, V⁺, and the concatenation of both matrices satisfy the full column rank condition for model identification. Additionally, the number of effects does not exceed the number of items. All the models used in the simulation study were fitted to the data. The same estimation method and model evaluation procedures tested in the simulation study were used with the empirical data. The $\hat{R}$ statistic indicated no convergence issues.

Figure 1.

Hypothetical example of DA5 item.

Table 6.

Structure Matrix for the Empirical Study.

	Operation											Operation
Item	1	2	3	4	5	6	7	8	9	10	Item	1	2	3	4	5	6	7	8	9	10
1	1	0	0	0	0	0	0	0	0	0	26	1	0	0	0	1	0	1	1	0	0
2	0	1	0	0	0	0	0	0	0	0	27	1	1	0	0	1	0	0	1	0	0
3	2	0	0	0	0	0	0	0	0	0	28	0	0	0	1	1	1	0	1	0	0
4	1	0	0	0	1	0	0	0	0	0	29	1	0	0	1	2	0	0	0	0	0
5	0	1	0	0	1	0	0	0	0	0	30	0	0	1	1	1	0	1	0	0	0
6	1	0	0	0	0	1	0	0	0	0	31	0	1	0	0	3	0	0	0	0	0
7	1	0	0	0	0	0	1	0	0	0	32	0	2	0	0	1	0	0	1	0	0
8	0	0	0	1	1	0	0	0	0	0	33	0	1	0	1	1	1	0	0	0	0
9	1	0	1	0	0	0	0	0	0	0	34	0	1	1	0	1	1	0	1	0	0
10	0	1	0	1	0	0	0	0	0	0	35	3	0	0	0	1	0	0	0	0	1
11	1	0	0	0	1	0	0	1	0	0	36	1	1	0	0	1	1	0	0	1	0
12	1	1	0	0	1	0	0	0	0	0	37	2	1	0	0	1	0	0	1	0	0
13	1	1	0	0	0	1	0	0	0	0	38	1	1	0	0	2	0	0	1	0	0
14	1	0	0	1	1	0	0	0	0	0	39	2	1	0	0	1	0	0	1	0	0
15	0	0	1	1	0	1	0	0	0	0	40	0	1	1	0	1	0	1	0	1	0
16	0	2	0	0	1	0	0	0	0	0	41	1	1	0	0	2	0	0	0	0	1
17	1	0	0	0	2	0	0	0	0	0	42	0	0	2	1	0	1	0	0	0	1
18	1	3	0	0	0	0	0	0	0	0	43	0	0	1	1	1	0	1	0	1	0
19	1	0	1	1	0	0	1	0	0	0	44	0	0	1	1	2	0	0	0	0	1
20	1	0	0	0	2	0	0	1	0	0	45	0	1	0	0	3	0	0	1	0	0
21	0	0	1	1	2	0	0	0	0	0	46	0	2	0	0	2	0	0	0	1	0
22	0	0	1	1	0	1	1	0	0	0	47	1	1	0	0	2	0	0	0	1	0
23	3	0	0	0	0	0	0	1	0	0	48	0	1	0	0	1	2	0	0	1	0
24	1	1	0	0	2	0	0	0	0	0	49	2	2	0	0	0	0	0	0	0	1
25	1	1	0	0	2	0	0	0	0	0	50	1	1	0	0	2	0	0	0	1	0

Results

Table 7 shows the goodness-of-fit estimates for the fitted models. According to the PPMC, the RWOSLM minimized the distance between the realized and the simulated value of the OR statistic, showing a PPP value closer to .5. The results therefore suggested the presence of inter-item local dependencies in the data that were better accounted for by the RWOSLM. Additionally, the information criteria, WAIC and LOOIC, indicated that the RWOSLM was the model that showed the best balance between fit and parsimony. An inspection of the covariance matrix of the RWOSLM revealed close-to-zero variances for the practice effects of Operations 4 and 5, as well as a high covariance between Operations 9 and 10. Based on these results, a constrained version of the RWOSLM was fitted to the data in which the practice effects of Operations 4 and 5 were treated as fixed effects and Operations 9 and 10 were merged into one single operation (from now on referred to as Operation 9–10). The constrained RWOSLM did not show a substantial decrease in fit, so the following analyses were based on this model.

Table 7.

Goodness-of-fit Estimates of the Fitted Models.

	OR
	Observed	Simulated(Sd)	PPP	WAIC	LOOIC
RWOSLM	1118.51	1081.88(50.47)	.215	18346.4	18394.6
RWOSLM (const.)	1118.51	1082.92(48.53)	.231	18426.1	18464.7
OSLM	1118.51	985.40(52.10)	.005	20016.3	20017.4
RWLLTM	1118.51	863.40(40.70)	.000	22484.0	22502.5
LLTM	1118.51	806.29(34.05)	.000	23092.9	23093.1
Rasch	1118.51	1041.11(47.50)	.041	18502.1	18502.7

Table 8 shows the EAP estimates, posterior standard deviations, and posterior probability intervals of the RWOSLM parameters. Figure 2 shows the difficulty of the operations as a function of practice for all examinees. Although Operations 1, 2, 3, 6, 7, and 8 also showed variability in practice effects, Operation 9–10 showed substantially greater variance. This operation involves a rearrangement of the figures based on an arbitrary order, requiring the storage and manipulation of intermediate results to a higher extent than the other operations. Besides Operation 4, for which all examinees showed a learning effect (

{\hat{δ}}_{4} = 0.099

), the frequencies of examinees with positive values of

{\hat{δ}}_{i m}

revealed the presence of learning effects in Operations 8 (34; 7%) and 9–10 (14; 3%). These frequencies, respectively, represent the number of examinees that showed learning throughout the DA5 items in the mentioned operations, which in Figure 2 corresponds to negative slopes depicting a progressive reduction in difficulty. The rest of the examinees, on the other hand, showed practice effects detrimental to performance, as fatigue or loss of motivation, which in Figure 2 corresponds to positive slopes depicting a progressive increase in difficulty.

Table 8.

Expected a Posteriori Estimates (EAP), Posterior Standard Deviations (SD), and Posterior Probability Intervals (2.5%–97.5%) of the RWOSLM Parameters.

	EAP	SD	2.5%	97.5%		EAP	SD	2.5%	97.5%		EAP	SD	2.5%	97.5%
α₁	‒1.569	0.062	‒1.682	‒1.446	$μ_{δ_{1}}$	‒0.032	0.002	‒0.036	‒0.029	$σ_{δ_{1}}$	9.547e-04	1.221e-04	7.423e-04	1.215e-03
α₂	‒1.938	0.067	‒2.068	‒1.813	$μ_{δ_{2}}$	‒0.058	0.003	‒0.064	‒0.052	$σ_{δ_{2}}$	2.141e-03	3.331e-04	1.557e-03	2.883e-03
α₃	‒1.160	0.101	‒1.355	‒0.959	$μ_{δ_{3}}$	‒0.091	0.011	‒0.113	‒0.068	$σ_{δ_{3}}$	1.114e-02	3.176e-03	5.076e-03	1.794e-02
α₄	0.625	0.078	0.468	0.775	$μ_{δ_{4}}$	0.099	0.012	0.074	0.123	$σ_{δ_{4}}$	-	-	-	-
α₅	‒0.607	0.054	‒0.716	‒0.499	$μ_{δ_{5}}$	‒0.019	0.002	‒0.023	‒0.014	$σ_{δ_{5}}$	-	-	-	-
α₆	‒1.002	0.105	‒1.202	‒0.803	$μ_{δ_{6}}$	‒0.079	0.011	‒0.102	‒0.058	$σ_{δ_{6}}$	1.049e-02	3.425e-03	3.829e-03	1.724e-02
α₇	‒1.870	0.137	‒2.135	‒1.603	$μ_{δ_{7}}$	‒0.216	0.020	‒0.254	‒0.174	$σ_{δ_{7}}$	1.512e-02	7.271e-03	3.695e-03	3.167e-02
α₈	‒0.126	0.093	‒0.303	0.055	$μ_{δ_{8}}$	‒0.015	0.010	‒0.033	0.006	$σ_{δ_{8}}$	1.719e-03	1.169e-03	1.064e-04	4.418e-03
α_9–10	‒1.240	0.151	‒1.531	‒0.946	$μ_{δ_{9 - 10}}$	‒0.454	0.034	‒0.521	‒0.383	$σ_{δ_{9 - 10}}$	2.629e-01	5.364e-02	1.701e-01	3.818e-01

Figure 2.

Difficulty of the DA5 logical operations as a function of practice for all examinees.

Table 9 shows the EAP estimates, posterior standard deviations, and posterior probability intervals of the practice parameters of the RWOSLM for a particular examinee. The posterior distribution of the parameters indicated that for this examinee the probabilities of learning effects, p(δ_im > 0), were .801 and .831 for Operations 8 and 9–10, respectively. Figure 3 shows the evolution of the operations' difficulties throughout the test for this examinee as a function of practice. The figure illustrates the decrease in difficulty during the test in Operations 4, 8, and 9–10. For instance, the initial difficulty of Operation 4 was 0.625, and each time the examinee practiced Operation 4, its difficulty decreased by 0.099. Consequently, after 11 previous presentations, the difficulty of Operation 4 at item 42 for this examinee was

{\hat{α}}_{i 42 4} = 0.625 - 11 (0.099) = - 0.464

. Since item 42 involves also Operations 3 (twice), 6, and 9–10 (after eight, eight, and four previous presentations, respectively), its difficulty for this examinee according to the RWOSLM was:

\begin{array}{l} {\hat{β}}_{i 42} = w_{42 3} ({\hat{α}}_{3} - {\hat{δ}}_{i 3} \sum_{k = 1}^{41} w_{k 3}) + w_{42 4} ({\hat{α}}_{4} - {\hat{δ}}_{4} \sum_{k = 1}^{41} w_{k 4}) + w_{42 6} ({\hat{α}}_{6} - {\hat{δ}}_{i 6} \sum_{k = 1}^{41} w_{k 6}) + w_{42 9 - 10} ({\hat{α}}_{9 - 10} - {\hat{δ}}_{i 9 - 10} \sum_{k = 1}^{41} w_{k 9-10}) \\ = 2 (- 1.160) - 2 (8) (- 0.146) + 0.625 - 11 (0.099) + (- 1.002) - 8 (- 0.252) + (- 1.240) - 4 (0.152) \\ = - 1.282 . \end{array}

Table 9.

Expected a Posteriori Estimates (EAP), Posterior Standard Deviations (SD), and Posterior Probability Intervals (2.5%–97.5%) of the Practice Parameters of the RWOSLM for a Particular Examinee.

	EAP	SD	2.5%	97.5%
δ_i1	‒0.055	0.015	‒0.084	‒0.025
δ_i2	‒0.098	0.026	‒0.152	‒0.050
δ_i3	‒0.146	0.066	‒0.287	‒0.017
δ₄	0.099	0.012	0.074	0.123
δ₅	‒0.019	0.002	‒0.023	‒0.014
δ_i6	‒0.252	0.075	‒0.399	‒0.109
δ_i7	‒0.487	0.110	‒0.717	‒0.270
δ_i8	0.042	0.049	‒0.040	0.151
δ_i9–10	0.152	0.156	‒0.152	0.454

Figure 3.

Difficulty of the cognitive operations as a function of practice for a particular examinee.

Conclusions

This study illustrates the applicability of the RWOSLM for the measurement of individual differences in operation-specific practice effects. The model was fitted to data from the DA5 logical ability test. The results suggest that the practice effects in the DA5 logical operations are a source of individual variability. In this regard, the RWOSLM overcame the OSLM in terms of fit, demonstrating higher flexibility to account for diverse patterns of practice effects. Interestingly, the operations that showed greater variability in practice effects were those that impose a higher load on working memory.

Discussion

The present paper introduces a random weights linear logistic test model for the measurement of individual differences in operation-specific practice effects within a test. A simulation study was conducted to examine the performance of the Bayesian estimation and evaluation procedures. PPMC along with the OR statistic demonstrated good performance in model evaluation, whereas the information criteria performed well in model comparison and selection. Additionally, the MCMC algorithm demonstrated accuracy in recovering the model parameters. The proposed model in combination with the Bayesian procedures thus proved useful in identifying individual differences in learning. Additionally, an empirical study was conducted to illustrate the applicability of the model. The RWOSLM proved its superiority over the OSLM to account for the variability of the practice effects associated with the operations involved in the DA5.

The requirement of relatively complex operations with which respondents do not have previous experience makes more likely the manifestation of individual differences in learning. In such circumstances, the RWOSLM may provide superior performance compared to one-dimensional operation-specific learning models. Despite the applicability of the model, there are certain considerations that must be taken into account. In the present paper, the structure matrix used with the RWOSLM represented the frequency with which the cognitive operations were required to solve the items. However, the model allows for other uses of the matrix, such us differentiating the order in which the operations must be applied to solve the items. In this regard, different columns of the structure matrix may reflect different orders of application of the same set of operations. Additionally, the present paper is focused on the measurement of non-contingent learning. However, other variants of the RWLLTM can be specified to measure individual differences in within-test contingent learning effects (e.g., Lozano & Revuelta, 2023). Future studies may examine the specification and identification conditions of these models along with their performance. Finally, as other operation-specific learning models, the proposed model is based on the assumption that item difficulty can be explained in terms of a well-defined set of operations. Such an assumption will constitute an important limitation when the hypothesized operations do not truly reflect the way in which examinees actually solve the items. In this regard, other item properties may constitute a source of item difficulty or even a source of practice effects throughout the test (e.g., Lozano & Revuelta, 2020). For a detailed discussion of this issue, see Lozano and Revuelta (2021b). Another important limitation of the proposed model is the assumption of linearity. In this regard, the difficulty of the cognitive operations is assumed to be a linear function of previous practice during the test. Despite its parsimony, the linearity of the model may constitute a too restrictive assumption when dealing with more complex patterns of practice effects, as for instance a learning curve with successively smaller reductions in operation difficulty (e.g., Spada, 1977; Spada & McGaw, 1985). Further research will be aimed to overcome these limitations.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Ministerio de Ciencia, Innovación y Universidades (PID2021-124885NB-I00). The computations were run with the support of the Scientific Computing Centre at Universidad Autónoma de Madrid (CCC-UAM).

ORCID iDs

José H. Lozano

Javier Revuelta

Notes

References

Andersen

E. B.

(1985). Estimating latent correlations between repeated testings. Psychometrika, 50(1), 3–16. https://doi.org/10.1007/bf02294143

Bock

R. D.

(1976). Basic issues in the measurement of change. In de Gruijter

D. N. M.

van der Kamp

L. J. T.

(Eds.), Advances in psychological and educational measurement (pp. 75‒96). Wiley.

Brooks

Gelman

Jones

G. L.

Meng

X.-L.

(2011). Handbook of Markov chain Monte Carlo. Chapman and Hall/CRC.

Embretson

S. E.

(1991). A multidimensional latent trait model for measuring learning and change. Psychometrika, 56(3), 495‒515. https://doi.org/10.1007/bf02294487

Fischer

G. H.

(1973). The linear logistic test model as an instrument in educational research. Acta Psychologica, 37(6), 359‒374. https://doi.org/10.1016/0001-6918(73)90003-6

Fischer

G. H.

Formann

A. K.

(1982). Some applications of logistic latent trait models with linear constraints on the parameters. Applied Psychological Measurement, 6(4), 397‒416. https://doi.org/10.1177/014662168200600403

Gelman

Meng

X.-L.

Stern

(1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 6(4), 733‒807.

Gelman

Rubin

D. B.

(1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 457‒472. https://doi.org/10.1214/ss/1177011136

Hoffman

M. D.

Gelman

(2014). The No-U-turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15, 1593‒1623.

10.

Lozano

J. H.

Revuelta

(2020). Investigating operation-specific learning effects in the Raven’s Advanced Progressive Matrices: A linear logistic test modeling approach. Intelligence, 82, 101468. https://doi.org/10.1016/j.intell.2020.101468

11.

Lozano

J. H.

Revuelta

(2021a). Bayesian estimation and testing of a linear logistic test model for learning during the test. Applied Measurement in Education, 34(3), 223‒235. https://doi.org/10.1080/08957347.2021.1933982

12.

Lozano

J. H.

Revuelta

(2021b). A Bayesian generalized explanatory item response model to account for learning during the test. Psychometrika, 86(4), 994‒1015. https://doi.org/10.1007/s11336-021-09786-x

13.

Lozano

J. H.

Revuelta

(2023). A Bayesian general model to account for individual differences in operation-specific learning within a test. Educational and Psychological Measurement, 83(4), 782‒807. https://doi.org/10.1177/00131644221109796

14.

Meng

X.-L.

(1994). Posterior predictive p-values. The Annals of Statistics, 22(3), 1142‒1160.

15.

R Development Core Team (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

16.

Rasch

(1960). Probabilistic models for some intelligence and attainment tests. Danish Institute for Educational Research.

17.

Rijmen

De Boeck

(2002). The random weights linear logistic test model. Applied Psychological Measurement, 26(3), 271‒285. https://doi.org/10.1177/0146621602026003003

18.

Scheiblechner

(1972). Das lernen und lösen komplexer denkaufgaben [The learning and solving of complex cognitive tasks]. Zeitschrift für Experimentelle und Angewandte Psychologie, 19, 476‒506.

19.

SHL (1996). DA5: Diagramas codificados [DA5: Coded diagrams]. SHL, Psicólogos Organizacionales.

20.

Sinharay

(2005). Assessing fit of unidimensional item response theory models using a Bayesian approach. Journal of Educational Measurement, 42(4), 375‒394. https://doi.org/10.1111/j.1745-3984.2005.00021.x

21.

Spada

(1977). Logistic models of learning and thought. In Spada

Kempf

W. F.

(Eds.), Structural models of thinking and learning (pp. 227‒262). Huber.

22.

Spada

McGaw

(1985). The assessment of learning effects with linear logistic test models. In Embretson

(Ed.), Test design: New directions in psychology and psychometrics (pp. 169‒194). Academic Press.

23.

Stan Development Team . (2019). RStan: the R interface to Stan. R package. version 2.19.2. http://mc-stan.org

24.

Vehtari

Gabry

Magnusson

Yao

Bürkner

Paananen

Gelman

(2020). loo: Efficient leave-one-out cross- validation and WAIC for Bayesian models. R package version 2.3.1. Available online at: https://mc-stan.org/loo/

25.

Vehtari

Gelman

Gabry

(2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5), 1413–1432. https://doi.org/10.1007/s11222-016-9696-4

26.

Verguts

De Boeck

(2000). A Rasch model for detecting learning while solving an intelligence test. Applied Psychological Measurement, 24(2), 151–162. https://doi.org/10.1177/01466210022031589

27.

Verhelst

N. D.

Glas

C. A. W.

(1993). A dynamic generalization of the Rasch model. Psychometrika, 58(3), 395–415. https://doi.org/10.1007/bf02294648

28.

Watanabe

(2013). A widely applicable Bayesian information criterion. Journal of Machine Learning Research, 14, 867–897.