Accounting for standard errors of measurement when modeling change

Abstract

Modeling within-person change over time and between-person differences in change over time is a primary goal in prevention science. When modeling change in an observed score over time with multilevel or structural equation modeling approaches, each observed score counts toward the estimation of model parameters equally. However, observed scores can differ in terms of their precision—both within and across participants. We propose an approach to weight observed scores by their level of precision, which is estimated as the inverse of their standard error of measurement in the context of item response modeling. Thus, scores with lower standard errors of measurement have greater weight, and scores with higher standard errors of measurement are down weighted. We discuss the weighting approaches and illustrate how to apply this approach with commonly available software. We then compare this approach to modeling change without weighting based on standard errors of measurement.

Keywords

Growth modeling reliability prevention

Prevention and intervention studies have the goal of changing the behavior of individuals. The main analytic tool to assess individual and group-level change over time is the growth model (Grimm et al., 2017; Laird & Ware, 1982; McArdle & Epstein, 1987; Meredith & Tisak, 1984, 1990; Singer & Willett, 2003). When modeling change with growth models, or any other statistical model, every observed score contributes equally to the estimation of model parameters. If observed scores are equally reliable, then it makes sense that they should all contribute equally. A single reliability for a measurement instrument (i.e., test, scale, or survey) for a given sample is consistent with the notion of reliability from classical test theory (CTT); however, modern measurement theory dictates otherwise. That is, notions of reliability from item response theory (IRT) suggest that the reliability of scores can differ and that there is not simply a single level of reliability.

In this article, we propose an approach to account for the differential reliability of the scores when examining change over time with growth models. We continue with an overview of growth modeling from the mixed-effects modeling perspective, a discussion of reliability in the context of item response modeling, and propose and approach to account for differential reliability when estimating growth models in the mixed-effects modeling framework. We then illustrate this estimation approach with longitudinal mathematics data collected from the Applied Problems test from the Woodcock–Johnson Psycho-educational Battery–III. We conclude with a discussion of the extensions as well as the limitations of this approach.

Mixed-Effects Models

Mixed-effects models are referred to by various names (often depending on the discipline) such as hierarchical linear models and multilevel models. These models are commonly used to analyze data with a nested (clustered) structure. Data with a nested structure have lower level units nested within one or more upper level units. This nested structure can occur in cross-sectional data as well as in longitudinal data. In the prevention sciences, a common example of a cross-sectional nested data is when children are nested within schools. In these situations, children in the same school are likely to be more similar to each other than students from other schools. If the intervention is at the level of the school (e.g., schools are randomized to the intervention and all children in a school are in the same arm of the intervention) or if the intervention is at the level of the student (e.g., students within different schools are randomized to the intervention), it is important to use a mixed-effects model to account for the dependency in the data due to the shared school environment. Longitudinal data are inherently nested because the repeated measures data from one individual are likely to be more similar to one another than observations from different individuals. Intervention studies often collect repeated measures data to assess individual change over time, and whether the intervention is implemented at the between-person level (e.g., different individuals are randomized to the intervention) or the intervention varies within person and over time (e.g., A-B-A Design; all individuals experience the levels of the intervention across time) a mixed-effects model can account for the dependency in the scores due to the repeated measurement protocol.

When applied to longitudinal data, mixed-effects models allow researchers to capture sample-level changes and individual-level changes using the fixed- and random-effects parameters (Searle et al., 1992). Fixed-effect parameters capture aspects of sample-level (mean) change, whereas the random-effects parameters capture aspects of individual differences in the change process. Mixed-effects models can be broken down into two types: linear and nonlinear models (Laird & Ware, 1982; Lindstrom & Bates, 1990). The distinction between the two types of models involves how the fixed and random effects enter the model. More details on each type of model are discussed in the next sections, starting with linear mixed-effects models as nonlinear mixed-effects models are built off the same foundation.

Linear mixed-effects models

Linear mixed-effects models are one of the most commonly used models to analyze longitudinal prevention data. In linear mixed-effects models, both the fixed and random effects enter the model in a linear (additive) fashion. While these models are linear models, they can be used to model nonlinear change patterns. That is, nonlinear change can be modeled using transformations in the timing variable. For example, squaring timing variable (e.g., ${time}^{2}$ ) is often used to model curvature in the change trajectory.

Linear mixed-effects models use fixed and random effects to model the mean and individual change trajectories. Fixed-effects parameters are used to describe the change pattern for the entire sample (sample-level parameters). For example, a fixed-effect parameter for the timing variable describes the expected rate of change in the outcome for a one-unit change in the timing variable for the sample. Random-effects parameters are used to describe the magnitude of differences between individuals. That is, random coefficients allow each person to deviate from a given fixed effect, and the variances and covariances of these random coefficients are random-effects parameters. For example, allowing for the effect of the timing variable to be random (i.e., a random slope) indicates that each individual is allowed to have a different rate of change in the outcome for a one-unit change in the timing variable. The random slope would have an estimated fixed-effect parameter describing the average rate of change for the sample and an estimated random-effect parameter describing the magnitude of differences in the rate of change across individuals.

Linear mixed-effects models take on the general form

y_{i} = X_{i} β + Z_{i} d_{i} + e_{i}

where y _i is a column vector of length T_i containing the observed responses for person i at T_i measurement occasions, X _i is a design matrix defining the fixed effects with fixed-effects parameters in column vector $β$ , Z _i is a design matrix defining the random effects with (level-2) random effects in column vector d _i , and e _i is a column vector of length T_i of residuals for person i at T_i measurement occasions (level-1 random effects). Each column in X _i represents a different fixed effect and each column in Z _i represents a different random effect. X _i and Z _i can have the same structure or different structures (and dimensions) depending on the nature of the specified model.

The level-2 and level-1 random effects are often assumed to follow multivariate normal distributions, such that $d_{i} \sim MVN (0, D)$ and $e_{i} \sim N (0, R_{i})$ . The matrix D is the covariance matrix of the level-2 (i.e., person-level) random effects, and the matrix R _i is the covariance matrix of the level-1 (i.e., within-person) residuals across the $T_{i}$ time points. R _i is typically assumed to be diagonal and have constant variance ( $σ_{e}^{2}$ ), meaning the residuals are assumed to be independent and identically distributed across persons and time points. Readers interested in other forms the level-1 residuals are referred to Grimm and Widaman (2010). The person-level random effects d _i and the level-1 residuals e _i are assumed to be uncorrelated.

As an example of a linear mixed-effects model, a linear growth model with ${day}_{t i}$ as the predictor variable with a random intercept and a random slope can be specified as

y_{t i} = [β_{0} + β_{1} \cdot {day}_{t i}] + [d_{0 i} + d_{1 i} \cdot {day}_{t i}] + e_{t i}

where $y_{t i}$ is the outcome variable for the ith individual ( $i = 1, \dots, n$ ; n indicates the total number of people in the sample) at this individual’s tth time point ( $t = 1, \dots, T_{i}$ ; T_i indicates the number of measurement occasions for individual i), ${day}_{t i}$ is the chosen timing metric and is the number of days since the beginning of the intervention for individual i at the tth measurement occasion, $β_{0}$ and $β_{1}$ are the fixed-effects parameters, $d_{0 i}$ and $d_{1 i}$ are the level-2 random coefficients, and $e_{t i}$ is the level-1 random term. The fixed-effects parameters are used to describe the average trajectory for the sample (also the average of the trajectories in linear mixed-effects models). That is, the average trajectory can be written as $μ_{t} = β_{0} + β_{1} \cdot {day}_{t}$ , where $μ_{t}$ is the predicted mean of $y_{t i}$ at ${day}_{t}$ . Thus, $β_{0}$ is the expected mean of $y_{t i}$ at the beginning of the intervention (i.e., ${day}_{t} = 0$ ) and $β_{1}$ is the expected mean rate of change in $y_{t i}$ for a one-unit change in ${day}_{t}$ . The random coefficients, $d_{0 i}$ and $d_{1 i}$ , are individual-level deviations from their associated fixed-effect for person i and are used to allow each person to have a different intercept (predicted value when ${day}_{t i} = 0$ ) and a different rate of change. That is, the intercept for person i is $β_{0} + d_{0 i}$ , and the rate of change for person i is $β_{1} + d_{1 i}$ . The level-2 random-effects parameters include the variance of the intercept, $σ_{0}^{2}$ , which indicates the magnitude of between-person differences in the intercept (individual differences at the beginning of the intervention), the variance of the slope for ${day}_{t i}$ , $σ_{1}^{2}$ , indicates the magnitude of between-person differences in the rate of change, and the covariance between the intercept and slope, $σ_{10}$ , which represents the strength of the linear association between the random intercept and slope (e.g., $[\begin{matrix} d_{0 i} \\ d_{1 i} \end{matrix}] \sim [[\begin{matrix} 0 \\ 0 \end{matrix}], [\begin{matrix} σ_{0}^{2} & sym . \\ σ_{10}^{} & σ_{1}^{2} \end{matrix}]]$ ). Lastly, the level-1 random-effect parameter is the variance of $e_{t i}$ and indicates the magnitude variability in $y_{t i}$ that is not accounted for by the model (i.e., $e_{t i} \sim N (0, σ_{e}^{2})$ ). We note that equation (1) was written for a vector of scores for each individual (only uses subscript i), whereas Equation (2) was written at the observation level (uses subscript t and i).

Nonlinear Mixed-Effects Models

Nonlinear mixed-effects models are an extension of linear mixed-effects models and are often used to analyze repeated measures data (Davidian & Giltinan, 1995). Nonlinear mixed-effects models are best suited for modeling data whose individual trajectories follow complex nonlinear patterns. These models are more flexible than linear mixed-effects models and can capture the nonlinearity in the data through inherently nonlinear functions (e.g., Gompertz function). Just like linear mixed-effects models, nonlinear mixed-effects models use fixed- and random-effects parameters to model the mean trajectory for the sample and individual-level trajectories. The difference between the two is how the parameters enter the model. In nonlinear mixed-effects models, at least one parameter enters the model in a nonlinear (multiplicative) fashion. Parameters that enter the model in a nonlinear fashion are contained within the mathematical function of time (e.g., parameter in an exponent, parameter raised to an exponent; Burchinal & Appelbaum, 1991).

Grimm et al. (2017) discuss two types of nonlinear mixed-effects models. The first type of nonlinear mixed-effects models are models that are nonlinear with respect to the fixed effects. These models have at least one fixed-effect parameter that enters the model in a nonlinear fashion. In this model, the fixed-effects parameter that enters the model nonlinearly does not have an associated random effect. These models have also been referred to as conditionally linear models (Blozis & Cudeck, 1999) because the random coefficients are additive. An example of this kind of model would be an exponential growth model where the rate parameter is a fixed-effect parameter without an associated random effect (e.g., all participants have the same rate parameter). The second type of nonlinear mixed-effects models are models with at least one random effect that enters the model in a nonlinear fashion. These models are considered to be nonlinear with respect to the random effects and have been referred to as fully nonlinear models. An example of this kind of model would be an exponential growth model where the rate parameter has an associated random effect (e.g., participants are allowed to differ in the rate).

A general expression for a nonlinear mixed-effects model is

y_{t i} = f ({day}_{t i}, b_{i}) + e_{t i}

b_{i} = X_{i} β + Z_{i} d_{i}

where $y_{t i}$ is the outcome variable for person i ( $i = 1, \dots, n)$ at time t ( $t = 1, \dots, T_{i})$ , which is equal to some linear or nonlinear function, $f ()$ , of the timing variable, ${day}_{t i}$ , and individual-specific parameters, b _i . The individual-specific parameters, b _i , in equation (3) are broken down in Equation (4) to highlight how they are composed of fixed effects, $β$ , and random effects, d _i . The fixed effects, $β$ , model aspects of the trajectory that describe the sample. The vector $β$ has length $p \times 1$ , meaning there are p fixed effects in the model. The design matrix X _i has dimension $T_{i} \times p$ , meaning the number of rows of this matrix equals the number of time points for person i (T_i ) and the number of columns equals the number of fixed effects (p). The random effects, d _i , are individual-specific deviations from the fixed effects. The vector d _i has length $k \times 1$ , meaning there are k random effects in the model. Lastly, the design matrix Z _i has dimension $T_{i} \times k$ , meaning the number of rows of this matrix equals the number of time points for person i (T_i ) and the number of columns equals the number of random effects (k).

The fixed-effect parameters, $β$ , are estimated and used to define the trajectory for the average person, and the level-2 random effects are assumed to follow a multivariate normal distribution with mean zero and covariance matrix D , such that $d_{i} \sim MVN (0, D)$ , where D has dimension $k \times k$ . The individual-specific (i.e., level 1) residuals, $e_{t i}$ , are also assumed to follow a normal distribution with mean zero and covariance matrix R _i with dimension ( $T_{i} \times T_{i}$ ). The covariance matrix for the level-1 residuals, R _i , is often assumed to be diagonal with constant variance; however, this assumption can be relaxed.

An example of a nonlinear mixed-effects model is the power model, which can be written as

y_{t i} = [(β_{0} + d_{0 i}) + (β_{1} + d_{1 i}) \cdot {day}_{t i}^{(β_{2} + d_{2 i})}] + e_{t i}

with fixed effects $β_{0}$ , $β_{1}$ , and $β_{2}$ and random effects $d_{0 i}$ , $d_{1 i}$ , and $d_{2 i}$ . Given the appearance of $d_{2 i}$ in the exponent of the model, this power model is nonlinear with respect to the random effects. In this model, $β_{0}$ is the mean of the intercept, which is centered at ${day}_{t i} = 0$ , $β_{1}$ is the mean amount of change for the sample from ${day}_{t i} = 0$ to ${day}_{t i} = 1$ , and $β_{2}$ is the mean power parameter, which controls the curvature of the function and usually restricted to be contained in the interval (0,1). Each fixed effect has an associated random effect to allow for individual differences in each aspect of the power model.

In the mixed-effects model, as with regression models, each observation of the outcome variable, $y_{t i}$ , counts equally toward the estimation of model parameters. In this article, our goal is to modify this assumption, such that each instance of $y_{t i}$ counts toward the estimation of model parameters relative to its reliability with more reliable scores getting more weight and less reliable scores getting less. To obtain an estimate of each score’s reliability, we turn to item response modeling and each estimated score’s standard error of measurement. We discuss this framework next.

Standard Errors of Measurement in IRT

IRT is a latent variable measurement framework whereby the items on a scale are assumed to be related to an underlying latent variable. Thus, an individual’s observed item responses are thought to be determined by the individual’s score on the latent variable, item parameters representing the strength of the association between the latent variable and the item and the difficulty of endorsing or correctly responding to the item, and random variability. The two-parameter logistic model (2PLM) is a commonly used item response model for binary data (items coded 0/1) and the model that we use in our empirical example. We discuss this model to demonstrate the features of item response models. The 2PLM can be written as

P (y_{p i} = 1 | α_{p}, β_{p}, θ_{i}) = \frac{exp (α_{p} (θ_{i} - β_{p}))}{1 + exp (α_{p} (θ_{i} - β_{p}))}

where $P (y_{p i} = 1 | α_{p}, β_{p}, θ_{i})$ is the probability of a correct response to item p by person i ( $y_{p i} = 1$ ) given the item parameters $α_{p}$ and $β_{p}$ , and latent variable score $θ_{i}$ ; $α_{p}$ is item p’s discrimination parameter representing the strength of the association between the item and the latent variable; $β_{p}$ is item p’s location or difficulty parameter representing the value on the latent variable distribution where the probability of a correct response is 0.50; and $θ_{i}$ is the latent variable score for person i.

To visualize the nature of the association between the probability of a correct response and the latent variable score, we have plotted this association given specific values of the discrimination and location parameters in Figure 1. In this plot, the probability of a correct response is on the y-axis and the latent variable score is on the x-axis. The item parameters are $α_{p} = 1.4$ and $β_{p} = 0.4$ . This plot is referred to as an item characteristic curve or a trace line. As seen in Figure 1, the probability of a correct response increases with increases in the latent variable score, which is due to having a positive discrimination parameter. A greater discrimination parameter (assuming $α_{p} > 0$ ) is related to a steeper trace line. More specifically, the slope of the tangent line to the trace line at $θ_{i} = β_{p}$ is equal to $\frac{a_{p}}{4}$ . The location of the trace line on the x-axis is due to the location parameter. Specifically, the trace line will obtain a value of 0.50 on the y-axis when $θ_{i} = β_{p}$ . Thus, as $β_{p}$ increases, the trace line will appear further to the right in the plot. The location parameter is often referred to as the difficulty parameter because items with a greater $β_{p}$ estimate have a lower probably of a positive response holding latent variable score and discrimination constant.

Figure 1.

Example of an ICC with $α_{p} = 1.4$ and $β_{p} = 0.4$ . ICC = item characteristic curve.

The trace line in Figure 1 highlights an important aspect of item response modeling regarding when an item provides information to help estimate an individual’s latent variable score. For example, the item whose trace line is contained in Figure 1 will provide little to no information regarding an individual’s latent variable score estimate if the individual’s true latent variable score is less than −1 or if the individual’s true latent variable score is greater than 2 (when $β_{p}$ is sufficiently different from $θ_{i}$ ). The reason why the item provides little to no information about an individual in these ranges ( $θ_{i} < - 1$ or $θ_{i} > 2$ ) is because there is little to no differences in the probability of correcting responding to the item in these ranges. When $θ_{i}$ is sufficiently larger than $β_{p}$ , it is akin to asking a college student a simple two-digit addition problem. The student is very likely to answer the question correctly, and answering the question correctly does not provide much (or any) information regarding how much mathematics the student knows. Thus, the item whose trace line is contained in Figure 1 will provide much more information for individuals whose true latent variable score is between −1 and 2.

The amount of information that an item provides is defined as

I_{p} (θ) = α_{p} [P (y_{p i} = 1 | α_{p}, β_{p}, θ)] [1 - P (y_{p i} = 1 | α_{p}, β_{p}, θ)]

where $I_{p} (θ)$ is item p’s information value at a latent variable score of $θ$ , $α_{p}$ is item p’s discrimination parameter, and $P (y_{p i} = 1 | α_{p}, β_{p}, θ)$ is the probability of correct response at latent variable score $θ$ . The information function for the item whose trace line is contained in Figure 1 is plotted in Figure 2. The information function in Figure 2 peaks when $θ = β_{p}$ , is low when $θ$ is sufficiently different from $β_{p}$ , and would be more peaked as $α_{p}$ increases.

Figure 2.

Example of an Information Curve for the Item with $α_{p} = 1.4$ and $β_{p} = 0.4$ .

The amount of information to estimate an individual’s latent variable score is simply the sum of item information functions for the items in which the individual responded. If all participants respond to every item on a measurement instrument, then the test information, $I (θ)$ , function can be defined as

I (θ) = \sum_{p = 1}^{P} I_{p} (θ)

where $I_{p} (θ)$ is the item information for item p, and P is the total number of items on the measurement instrument. An example test information function for a set of 5 items is plotted in Figure 3. From this figure, it is clear that the amount of information to estimate latent variable scores depends on the value of the latent variable score.

Figure 3.

Example of a Test Information Function for Five Items with $α_{p} = (1.4, 1.8, 0.7, 0.9, 1.2)$ and $β_{p} = (0.4, - 1.8, 1.7, - 1.2, 1.2)$ .

The variance of a latent variable score estimate is equal to the reciprocal of the amount of test information at the value of the latent variable score estimate, such that

var (\hat{θ}) = \frac{1}{I (\hat{θ})}

and this variance represents noise variance in the estimation of the latent variable score, which is related to error variance from CTT. Taking the square root of this variance yields the standard error of measurement of the latent variable score estimate and indicates how much the latent variable score estimate is likely to vary by chance. The magnitude of this uncertainty depends on the number and type (item parameters) of items to which the individual responded and the estimate of the individual’s latent variable score. And it is this uncertainty that we want to account for during the estimation of growth models.

Accounting for Standard Errors of Measurement in Estimation

When estimating a growth model using the mixed-effects modeling framework with maximum likelihood estimation, we attempt to maximize the log-likelihood (LL) function. For a continuous outcome, the LL function for each observation is defined as

{LL}_{t i} = - \frac{1}{2} [ln (2 π) + \frac{{(y_{t i} - {\hat{y}}_{t i})}^{2}}{σ_{e}^{2}} + ln (σ_{e}^{2})]

where $y_{t i}$ is the outcome measured at time t for person i, ${\hat{y}}_{t i}$ is the predicted value of the outcome at time t for person i, and $σ_{e}^{2}$ is the estimated residual (level-1) variance. The LL for the model is simply the sum of the ${LL}_{t i}$ values.

To account for the standard error of measurement for each observation, we propose to weight the ${LL}_{t i}$ (equation (10)) by a function of the standard error of measurement. The weight is defined as

w_{t i} = \frac{1}{s . e . m ({\hat{θ}}_{t i})}

where $s . e . m ({\hat{θ}}_{t i})$ is the standard error of measurement for the estimated latent variable score. These weights are then be normalized, such that

n w_{t i} = \frac{w_{t i}}{{\bar{w}}_{t i}}

where ${\bar{w}}_{t i}$ is the mean weight across all observations. Thus, $n w_{t i}$ will have a mean equal to 1 and all weights will be positive. The final weighted observation level ${LL}_{t i}$ is

Weighted {LL}_{t i} = n w_{t i} \cdot {LL}_{t i}

and summing these values over time and individuals will yield the final weighted LL. That is,

Weighted LL = \sum_{i = 1}^{N} \sum_{t = 1}^{T} n w_{t i} \cdot {LL}_{t i}

This type of weighting allows each observation to contribute to the LL depending on the relative size of the standard error of measurement for each observation, which is our preference when accounting for differential reliability in the outcome.

Implementation

This weighted approach to estimation can be implemented in PROC NLMIXED in SAS using the general LL approach and Mplus (Muthén & Muthén, 1998–2017). In NLMIXED, the general LL approach allows the user to specify the LL function. This approach has been used by researchers to specify the LL functions for distributions that are not embedded into NLMIXED (e.g., Sheu et al., 2005), such as when specifying a mixed-effects model for an outcome that is ordinal (i.e., ordered logit) or categorical (i.e., baseline logit). The weighted ${LL}_{t i}$ in Equation (14) can be specified directly in NLMIXED, and this approach follows work by Grilli and Pratesi (2004) in the utilization of level-1 weights in generalized multilevel models.

Mplus allows for different types of weights when estimating mixed-effects models (TYPE=TWOLEVEL). The WEIGHT option is used to identify a level-1 weight variable, and the WTSCALE option is used to specify how the weights should be scaled. Specifying $n w_{t i}$ in the WEIGHT option and setting WTSCALE to UNSCALED yields the weighted LL approach presented here. Mplus also allows the level-1 weights to be scaled so that they sum to the sample size in each cluster (WTSCALE=CLUSTER). This approach would differentially weight the observations for each person; however, each person would contribute to the LL equivalent to their number of observations. Mplus also allows for level-2 weights, which could be utilized to differentially weight data for each individual (with the BWEIGHT option).

Illustration

Data

Longitudinal data from the Applied Problems subtest of the Woodcock–Johnson Tests of Achievement III collected as part of the National Institute of Health and Human Development’s Study of Early Child Care and Youth Development (SECCYD; NICHD Early Child Care Research Network, 1997 is analyzed for illustration). The SECCYD is a longitudinal study of $N = 1, 364$ children measured from birth through high school. The Applied Problems subtest is designed to measure children’s ability to analyze and solve word problems. The Applied Problems subtest is a linear test with 60 items; however, participants are rarely asked all items. There are starting rules based on the characteristics of the participant (e.g., age), and stopping rules when the participant incorrectly answers six consecutive questions. In the SECCYD, the Applied Problems subtest was administered up to 5 times at approximately 54 months, first grade, third grade, fifth grade, and eighth grade, and these item-level data are analyzed here. The data contained 4,974 assessments from 1,158 children. Children were asked between 6 and 45 of the 60 items from the Applied Problems subtest. Thus, there is quite a bit of variability in the number of items administered.

Analytic Methods

Item response modeling

The 2PLM (equation (6)) was fit to the longitudinal item-level data to estimate each child’s ability (latent variable score) to analyze and solve word problems at each measurement occasion. Analyzing the longitudinal item-level data with an item response model is an appropriate approach to estimate latent variable scores and ensures that the measurement properties of the scale (e.g., item parameters) are the same over time (Davoudzadeh, 2017). This allows the latent variable estimates to capture change over time. As noted above, incomplete item-level data occurred because of the starting and stopping rules of the Applied Problems subtest and due to the participant-level missingness (e.g., participant was not assessed at a given grade). The latent variable estimates from the 2PLM were the expected a posteriori estimates and are based on the items asked and the child’s pattern of correct and incorrect responses. Standard errors of the latent variable scores were also obtained from fitting the 2PLM.

The estimated latent variable scores are plotted against age at testing in Figure 4. In this plot, dots connected by a line are scores belonging to the same individual, and the slope of this line represents the observed rate of change between the assessments. Based on these observed trajectories, children’s mathematical problem-solving appears to increase rapidly during the early elementary school years before leveling off. There are individual differences in mathematical problem-solving at all ages, and there appears to be individual differences in the rate of growth in mathematical problem-solving.

Figure 4.

Longitudinal Plot of Mathematical Problem-Solving Estimates from the 2PLM Against Age at Testing. 2PLM = two-parameter logistic model.

Growth modeling

Given the observed trajectories of mathematical problem-solving, a nonlinear change model is needed to capture the individual change process and the individual differences in the trajectories for mathematical problem-solving. Given the observed changes, a three-parameter exponential growth model was specified. This exponential growth model can be written as

{\hat{θ}}_{t i} = b_{0 i} + b_{1 i} \cdot (1 - exp (- β_{2} \cdot (a g e_{t i} - 4.5))) + e_{t i}

where ${\hat{θ}}_{t i}$ is the child’s mathematical ability latent variable estimate at time point t for child i, $b_{0 i}$ is the random intercept for child i representing the child’s mathematical problem-solving ability at 4.5 years of age, $b_{1 i}$ is the random slope for child i representing the child’s increase in mathematical problem-solving ability from 4.5 years of age to the child’s upper asymptote of mathematical problem-solving potential, $β_{2}$ is a fixed-effect parameter indicating the curvature or rate of approach to the asymptotic level for all children, ${age}_{t i}$ is child i’s age at time point t, and $e_{t i}$ is the time-dependent residual at time point t for child i. The random coefficients $b_{0 i}$ and $b_{1 i}$ are assumed to follow a multivariate normal distribution with an estimated mean vector and covariance matrix (i.e., ${[b_{0 i}, b_{1 i}]}^{'} \sim N (β, Ψ)$ ). The time-dependent residual is assumed to be normally distributed with a zero vector for the mean and an unknown variance (i.e., $e_{t i} \sim N (0, σ_{e}^{2})$ ).

Estimation

The 2PLM was estimated using Marginal Maximum Likelihood with the Multilog (Thissen, 1991) program. The growth model was estimated using PROC NLMIXED using the general LL approach with and without accounting for the standard errors of measurement of the mathematical problem-solving latent variable estimates. All of the statistical programming scripts can be found on the first author’s website. Additionally, programming scripts for linear mixed-effects models using this weighted LL approach in Mplus , PROC MIXED, and R are also contained on the first author’s website.

Results

Standard errors of measurement

The mean standard error of measurement was 0.162 and the standard errors of measurement ranged from 0.138 to 0.405. The standard errors of measurement are plotted against the latent variable estimates in Figure 5. From this figure, there is a flat u-shaped association between the latent variable estimates and their standard errors of measurement. This type of association is common because there is often less variability in the pattern of correct and incorrect responses (e.g., most responses were incorrect or correct) at the upper and lower ends of the ability range (e.g., participant got almost all items correct or incorrect). Additionally, it is also common that participants with low latent variable estimates responded to fewer questions on the Applied Problems subtest because of stopping rules.

Figure 5.

Plot of the Standard Error of Measurement Against Latent Variable Estimate Based on the 2PLM. 2PLM = two-parameter logistic model.

Growth modeling

The exponential growth model in Equation (15) was specified and fit to the latent variable estimates with and without using the standard error informed weight (equations (11) and (12)). Parameter estimates for the two models are contained in Table 1. Overall, the estimates are fairly similar, which is expected; however, certain estimates have noticeable differences with and without the standard error weights. The rate of approach to the asymptotic level and the correlation between the intercept and the total amount of change were sufficiently different. The estimate for the rate of approach to the asymptote was slightly smaller when the standard error weights were utilized (0.193 vs. 0.188), and the estimated correlation between the intercept and the total amount of change to the asymptotic level was slightly closer to zero (−0.294 vs. −0.207). These changes were caused by down weighting the ability scores at the lower and upper ends of the distribution, which had higher standard errors of measurement due to the low variability in the pattern of correct and incorrect responses.

Table 1.

Parameter Estimates from the Exponential Growth Model (a) With and (b) Without Standard Error of Measurement Weights.

Parameter	Estimate	Standard Error
(a)
Fixed-effects
$β_{0}$	−1.377	0.013
$β_{1}$	3.122	0.021
α	0.193	0.002
Random-effects
$\sqrt{ψ_{00}}$	0.344	0.010
$\sqrt{ψ_{11}}$	0.374	0.017
$\frac{ψ_{10}}{\sqrt{ψ_{00} \cdot ψ_{11}}}$	−0.294	0.045
$\sqrt{θ}$	0.047	0.001
(b)
Fixed-effects
$β_{0}$	−1.363	0.012
$β_{1}$	3.143	0.021
α	0.188	0.002
Random-effects
$\sqrt{ψ_{00}}$	0.325	0.010
$\sqrt{ψ_{11}}$	0.359	0.017
$\frac{ψ_{10}}{\sqrt{ψ_{00} \cdot ψ_{11}}}$	−0.207	0.051
$\sqrt{θ}$	0.047	0.001

Discussion

The reliability of individual change can be particularly low. Reliability of change worsens under a variety of circumstances. For example, change reliability is diminished when there are fewer measurement occasions (see Bryk & Raudenbush, 1987). Change reliability becomes poorer when observed scores are close to the lower or upper boundary of the scale. This is a particular issue with psychological scales that were designed to measure relatively large between-person differences in behavior at a particular point in time, as well as scales that were designed to classify people with relatively extreme behaviors (e.g., depression). In these cases, scores at the upper and lower end of the distribution can represent a wide variety of behaviors, which hampers the measurement of change. Given that scores at the extremes of the distribution are more unreliable, it can be beneficial to down weight those observations in order to get a better representation of the overall change process.

Mixed-effects modeling is one framework used to assess individual change and between-person differences in change and allows for the inclusion of observation-level weights, as opposed to person-level weights. Observation-level weighting in mixed-effects modeling has been discussed in the literature (see Zhou, 2009) and has been implemented in a variety of ways. For example, Zhou (2009) down-weighted observations with higher residuals to reduce the impact of outliers. Grilli and Pratesi (2004) discussed probability-weighted estimation in mixed-effects models when there are differential inclusion probabilities at each sampling stage. The approach of down weighting less reliable scores here follows the ideas of Zhou (2009), where observed scores are weighted based on characteristics of the scores themselves.

The weighted LL approach is straightforward to implement in PROC NLMIXED in SAS and in the Mplus (Muthén & Muthén, 1998–2017) program. Other commonly used mixed-effects modeling procedures, such as the lmer function from lme4 in R and PROC MIXED in SAS, allow for observation-level weights. Weights in PROC MIXED are not applied to the LL function, but applied to the design matrices defining the fixed and random effects or the residual covariance matrix, depending on the specification (SAS Institute, Inc., 2008). The weights statement in lmer and nlmer applies the weights to the residual covariance matrix (Bates et al., 2015). When the weights are normalized as discussed, the estimates from PROC MIXED and lme4 were the same as the estimates from PROC NLMIXED and Mplus . Thus, using these mixed-effects modeling programs are a viable way to implement the weighted LL approach. Work by Carle (2009) discusses how weights can be utilized in the MLwiN and GLLAMM programs, which suggests additional software options where the standard error weighting approach can be implemented.

Growth Mixture Modeling

Growth mixture modeling (Muthén & Shedden, 1999; Ram & Grimm, 2009) is an extension of growth models that can be used to explore whether there are groups of participants with different change trajectories, and this approach is common in the behavioral sciences. The issue of differential reliability, especially for extreme scores, may be a greater issue for growth mixture models because these models can account for non-Gaussian data through the incorporation of latent classes (Bauer & Curran, 2003). Extreme scores tend to be less reliable, and down weighting these scores may decrease the likelihood of growth mixture modeling identifying spurious classes (Bauer & Curran, 2003; K. Masyn, personal communication, September 27, 2018). Growth mixture models can be fit using NLMIXED and Mplus , which suggests the incorporation of standard error informed weights can be implemented with growth mixture models. Future work should consider whether the incorporation of observation-level weights can improve the performance of growth mixture models.

Second-Order Growth Models

Growth models can and have been specified with lower-order measurement models, such as confirmatory factor models (McArdle, 1988) and item response models (McArdle et al., 2009; Wang et al., 2016). When analyzing longitudinal item response data, the second-order growth model with a lower-order item response model is considered optimal for at least two reasons. First, latent variable estimation is unnecessary. Second, the approach inherently weighs observations given the number of item responses. That is, an observation from a participant who responded to 45 items counts more than an observation from a participant who only responded to 6 items. This second reason aligns with our goals to differentially weigh observations according to the standard error of measurement, which is partially related to the number of administered items.

The approach outlined here is a two-step approach with latent variable estimates obtained in the first step and longitudinal modeling comprising the second step. There are known limitations of latent variable estimation; however, the two-step approach is common because the combined estimation of longitudinal models and item response models is challenging, particularly when there is a large number of time points, a small sample, a large number of items, or a complex longitudinal model (e.g., nonlinear mixed-effects model). Moreover, the two-step approach is common when conducting integrative data analyses (Curran et al., 2008) because of the challenges with simultaneous estimation with a lack of consistent measurement protocols across studies. While simultaneous estimation of measurement and growth is optimal, two-step approaches remain common (e.g., every time a sum score is analyzed) and analyzing latent variable estimates from item response models is becoming more common. Furthermore, there are times where latent variable estimates and standard errors of measurement are available and the item-level data are not (e.g., Early Childhood Longitudinal Study–Kindergarten Cohort). Thus, we see many potential uses of this estimation approach.

Concluding Remarks

Psychological measurement is not often a priority when conducting longitudinal research even though it should be a top priority. In large-scale longitudinal studies, short forms are often implemented to minimize participant burden. While this is an important consideration, researchers should take advantage of item response models to quantify differences in the size of the standard error of measurement. Ideally, adaptive tests can be implemented with the goal of obtaining a specific standard error of measurement in an attempt to have equally reliable latent variable estimates. When this is not possible, we encourage researchers to give priority to scores that are known to have greater reliability whether in a change analysis or any statistical analysis (e.g., regression). This approach will help ensure that conclusions are not due to the least reliable scores.

Footnotes

Acknowledgments

The authors would like to thank Jack McArdle, Aki Hamagami, and Keith Widaman for their thoughtful comments on this work. This work was presented at the Developmental Methods Conference in Whitefish, MT, in September 2018.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Science Foundation Grant REAL-1252463 awarded to the University of Virginia, David Grissmer (Principal Investigator), and Christopher Hulleman (Co-Principal Investigator).

ORCID iD

Kevin J. Grimm

References

Bates

Maechler

Bolker

Walker

(2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67, 1–48.

Bauer

D. J.

Curran

P. J.

(2003). Distributional assumptions of growth mixture models: Implications for over-extraction of latent trajectory classes. Psychological Methods, 8, 338–363.

Blozis

S. A

Cudeck

(1999). Conditionally linear mixed-effects models with latent variable covariates. Journal of Educational and Behavioral Statistics, 24, 245–270.

Bryk

A. S.

Raudenbush

S. W.

(1987). Application of hierarchical linear models to assessing change. Psychological Bulletin, 101, 147–158.

Burchinal

Appelbaum

M. I.

(1991). Estimating individual developmental functions: Methods and their assumptions. Child Development, 62, 23–43.

Carle

A. C.

(2009). Fitting multilevel models in complex survey data with design weights: Recommendations. BMC Medical Research Methodology, 9, 49–61.

Curran

P. J.

Hussong

A. M.

Cai

Huang

Chassin

Sher

K. J.

Zucker

R. A.

(2008). Pooling data from multiple longitudinal studies: The role of item response theory in integrative data analysis. Developmental Psychology, 44, 365–380.

Davidian

Giltinan

D. M.

(1995). Nonlinear models for repeated measurement data. Chapman & Hall/CRC Press.

Davoudzadeh

(2017). A statistical evaluation of latent variable score estimation from simple and complex item response models: Implications for integrative data analysis (Doctoral dissertation). PsycINFO, AAI10165897.

10.

Grilli

Pratesi

(2004). Weighted estimation in multilevel ordinal and binary models in the presence of informative sampling designs. Survey Methodology, 30, 93–103.

11.

Grimm

K. J.

Ram

Estabrook

(2017). Growth modeling: Structural equation and multilevel modeling approaches. Guilford.

12.

Grimm

K. J.

Widaman

K. F.

(2010). Residual structures in latent growth curve analysis. Structural Equation Modeling: A Multidisciplinary Journal, 17, 424–442.

13.

Laird

N. M.

Ware

J. H.

(1982). Random effects models for longitudinal data. Biometrics, 38, 963–974.

14.

Lindstrom

M. J.

Bates

D. M.

(1990). Nonlinear mixed effects models for repeated measures data. Biometrics, 46, 673–687.

15.

McArdle

J. J.

(1988). Dynamic but structural equation modeling of repeated measures data. In Nesselroade

J. R.

Cattell

R. B.

(Eds.), Handbook of Multivariate Experimental Psychology (pp. 561–614). Springer, Boston, MA.

16.

McArdle

J. J.

Epstein

(1987). Latent growth curves within developmental structural equation models. Child Development, 58, 110–133.

17.

McArdle

J. J.

Grimm

K. J.

Hamagami

Bowles

R. P.

Meredith

(2009). Modeling life-span growth curves of cognition using longitudinal data with multiple samples and changing scales of measurement. Psychological Methods, 14, 126–149.

18.

Meredith

Tisak

(1984). Statistical considerations in Tuckerizing curves with emphasis on growth curves and cohort sequential analysis. Paper presented at the Annual Meeting of the Psychometric Society , Santa Barbara, CA, USA.

19.

Meredith

Tisak

(1990). Latent curve analysis. Psychometrika, 55, 107–122.

20.

Muthén

B. O.

Shedden

(1999). Finite mixture modeling with mixture outcomes using the EM algorithm. Biometrics, 55, 463–469.

21.

Muthén

L. K.

Muthén

B. O.

(1998–2017). Mplus user’s guide (8th ed.). Muthén & Muthén.

22.

NICHD Early Child Care Research Network. (1997). The effects of infant child care on infant-mother attachment security: Results of the NICHD study of early child care. Child Development, 68, 860–879.

23.

Ram

Grimm

K. J.

(2009). Growth mixture modeling: A method for identifying differences in longitudinal change among unobserved groups. International Journal of Behavioral Development, 33, 565–576.

24.

SAS Institute Inc. (2008). SAS/STAT® 9.2 user’s guide.

25.

Searle

S. R.

Casella

McCulloch

C. E.

(1992). Variance components. Wiley.

26.

Sheu

C. F.

Chen

C. T.

Y. H.

Wang

W. C.

(2005). Using SAS PROC NLMIXED to fit item response theory models. Behavior Research Methods, 37(2), 202–218.

27.

Singer

J. D.

Willett

J. B.

(2003). Applied longitudinal data analysis. Oxford University Press.

28.

Thissen

(1991). MULTILOG user’s guide: Multiple, categorical item analysis and test scoring using item response theory. Scientific Software International.

29.

Wang

Kohli

Henn

(2016). A second-order longitudinal model for binary outcomes: Item response theory versus structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 23, 455–465.

30.

Zhou

. (2009). Weighting method for a linear mixed model. Communications in Statistics—Theory and Methods, 39, 214–227.