Application of a Longitudinal IRTree Model: Response Style Changes Over Time

Abstract

Traditional psychometric models focus on studying observed categorical item responses, but these models often oversimplify the respondent cognitive response process, assuming responses are driven by a single substantive trait. A further weakness is that analysis of ordinal responses has been primarily limited to a single substantive trait at one time point. This study applies a significant expansion of this modeling framework to account for complex response processes across multiple waves of data collection using the item response tree (IRTree) framework. This study applies a novel model, the longitudinal IRTree, for response processes in longitudinal studies, and investigates whether the response style changes are proportional to changes in the substantive trait of interest. To do so, we present an empirical example using a six-item sexual knowledge scale from the National Longitudinal Study of Adolescent to Adult Health across two waves of data collection. Results show an increase in sexual knowledge from the first wave to the second wave and a decrease in midpoint and extreme response styles. Model validation revealed failure to account for response style can bias estimation of substantive trait growth. The longitudinal IRTree model captures midpoint and extreme response style, as well as the trait of interest, at both waves.

Keywords

response style longitudinal model item response tree Bayesian estimation

Researchers and measurement practitioners often assume the substantive trait of interest (TOI) drives responses to ordinal items, but response styles may also influence respondent option selection. Response styles describe a respondent’s propensity to select item response options in a systematic way, regardless of both the respondent’s TOI and the content of the item. One commonly studied response style, extreme response style (ERS; Cronbach, 1946), is the tendency to endorse the lowest and highest categories of a response scale. In contrast, midpoint response style (MRS) describes a respondent’s propensity to use the midpoint response category (Messick, 1968; Plieninger & Heck, 2018; Schuman & Presser, 1981). Other response styles include acquiescence response style (ARS), the propensity of a respondent to agree with or affirm conceptually distinct ordinal items (Martin, 1964; Ray, 1983) and disacquiescence response style (DRS), which is the tendency to disagree with conceptually distinct ordinal items.

The presence of response styles in self-reported measures is a concern, as the validity of inferences made from instruments using these measures is threatened by response styles that have not been accounted for. Respondents’ preference for extreme responses can bias measurements of traits of interest and can confound interpretations of scores, thus interfering with the psychometric study of an instrument. For example, items may appear to function differently in their measurement of the TOI when respondents vary only in response style (Bolt & Johnson, 2009; Park & Wu, 2019). The presence of response styles presents a host of other issues. For example, ERS can lead to spurious correlations between constructs as respondents may have inflated or deflated scores in unrelated constructs (Jeon & De Boeck, 2016; Moors, 2012; Park & Wu, 2019; Paulhus, 1991), which may lead to overestimation or underestimation of the construct of interest and score inflation (Park & Wu, 2019) or manifest as an issue in diagnostic surveys where a respondent’s score plays a role in a clinical diagnosis or access to services. Furthermore, response styles threaten dimensionality, construct validity, predictive validity, and the reliability of self-reported measures (Baumgartner & Steenkamp, 2001; Cheung & Rensvold, 2000; de Jong et al., 2008; van Herk et al., 2004; van Rosmalen et al., 2010). This can result in differential item functioning, a lack of measurement invariance, and present negative effects on parameter recovery (Bolt & Johnson, 2009; Liu et al., 2017). Going further, response styles threaten comparisons of diverse groups and international or cross-cultural assessment. There are many documented correlates (i.e., age, individual traits, or cultural background) to different types of response styles that often vary by culture or region, thus making accurate comparisons difficult (Harzing, 2006; Khorramdel & von Davier, 2014).

Stability in individuals’ response styles has led researchers to suggest that these tendencies should be conceptualized as trait-like individual difference variables of substantive interest (e.g., Böckenholt, 2017; Wetzel et al., 2016). Stability of response styles implies that individuals systematically use the response scale within the same questionnaire (e.g., ARS: Danner et al., 2015; ARS and ERS: Weijters et al., 2010b) and across scales (e.g., ERS: Aichholzer, 2013; Bolt & Newton, 2011; Wetzel et al., 2013; ARS & ERS: Weijters et al., 2010b).

Previous research has also indicated that response styles are stable over both short and long periods of time. Evidence of response style stability has been found over a 1-year period (ARS, ERS, DRS, and MRS: Weijters et al., 2010a), a 2-year period (ARS and ERS: Bachman & O’Malley, 1984), a 4-year period (ARS: Billiet & Davidov, 2008), and an 8-year period (ARS and ERS: Wetzel et al., 2016). The stability of personality and attitude constructs, such as self-esteem or perceived ethnic threat, may be ascribed to response style stability (e.g., Bachman & O’Malley, 1984; Billiet & Davidov, 2008), although this is not the only option for observed stability in constructs. However, due to the possibility and evidence of response style stability over constructs and time, Böckenholt (2017) stresses the use of models that account for individual differences due to response style tendencies, especially for longitudinal studies, as emphasized by Weijters et al. (2010b). The methods used to answer this call are still developing, particularly those to account for response styles over multiple time points using an item response framework. As such, this study describes and applies one such model.

Measuring Response Style

Because of potential response style contamination in surveys and assessments, researchers propose and use multiple approaches to measure the presence and impact of response styles. van Vaerenbergh and Thomas (2013) provide a useful review of approaches. Some of the earliest methods, often termed the classical methods, record the frequency of occurrence for which a participant selects a particular response category, such as the extreme categories, and use this frequency, or proportion, as a measure of the response style (e.g., Bachman & O’Malley, 1984; Greenleaf, 1992). The primary limitation of this approach is that it confounds measurement of response styles and the trait(s) of interest. That is, a person providing many responses in the lowest category, implying they could be an extreme responder, or the respondent could also be very low on the substantive TOI. Although this method may aid in quantifying a respondent’s response style, this classical approach cannot disentangle response style and the substantive trait; as such, there is no direct way of correcting the TOI measure for response style bias (Bolt et al., 2014).

In a promising approach that addresses issues with the classical approach, Wetzel et al (2016) use a trait-state-occasion model for ERS and ARS, building on Weijters et al. (2010a). They measure the stability of response styles as the amount of variance in the state response style factors that is explained by the trait response style factors. Wetzel et al. (2016) also use the latent-state model, which measures stability of response styles as the correlations between time-specific response style factors across waves. However, indicators of response style were based on sum scores for a random sample of items and do not utilize item-level responses.

Multidimensional item response models address the shortcoming of the classical approaches and those using sum scores by simultaneously modeling the TOI and response styles utilizing item-level information. One multidimensional approach, the item response tree (IRTree), models systemic response tendencies when an individual is assumed to respond to ordinal items in a multiple-stage decision-making process (Böckenholt, 2012). As an example, consider a five-category ordinal item. One three-process model hypothesizes that individuals respond to ordinal items using three distinct decisions, as illustrated in Figure 1, with each decision driven by a separate latent trait ( $θ_{i, M R S}, θ_{i, T O I}, a n d θ_{i, E R S}$ for MRS, TOI, and ERS, respectively; Böckenholt, 2012). The model has been adapted to fit different conceptualizations and different numbers of categories (Böckenholt, 2017). In responding to a five-category item, with response categories strongly disagree, disagree, neither agree nor disagree, agree, and strongly agree; respondent i may enter into the MRS stage of item j by endorsing the middle category, neither agree nor disagree with probability m_ij. That is, the respondent must decide whether they have an opinion regarding the ordinal survey item or not. If the respondent does not endorse the midpoint category (with probability 1 − m_ij), the TOI is endorsed with probability t_ij (or not endorsed with probability 1 − t_ij) in the second stage. In the third stage, the ERS stage, respondents endorse an extreme response with probability e_ij (or not endorse an extreme response with probability 1 − e_ij; Böckenholt, 2012; Plieninger & Heck, 2018).

Figure 1.

Item response tree (IRTree) model accounting for midpoint response style (MRS), the trait of interest (TOI), and extreme response style (ERS) for a 5-point ordinal item.

The probability of each Likert-type item response category can be specified by tracing the path in Figure 1, assuming the three latent traits are considered independent. For example, the probability of person i responding to item j in the highest category, strongly agree, is given by:

\begin{array}{l} \Pr (Y_{i j} = 5 | θ_{i, M R S}, θ_{i, T O I}, θ_{i, E R S}, ω_{j}) \\ = (1 - m_{i j}) * (t_{i j}) * (e_{i j}), \end{array}

(1)

where $ω_{j}$ represents the item parameters. That is, they have failed to endorse the midpoint (i.e., have an opinion of the item), endorsed the TOI, and endorsed the extreme option. Each decision stage in this model is therefore a dichotomy, as illustrated by the recoding of the original ordinal-scale data into three dichotomous pseudo-items, $Y_{i j k}^{*}$ , for the k^th decision stage (k = MRS, TOI, ERS). Returning to the example of category strongly agree, an individual would have a response profile of [011] for item j to represent a decision process of failing to endorse the midpoint pseudo-item, but endorsing the TOI and extreme response pseudo-items. Another example is the second-highest category, agree, for which an individual would have a response profile of [010] to represent a decision process of failing to endorse the midpoint pseudo-item, endorsing the TOI pseudo-item, and failing to endorse the extreme response pseudo-item. Other response profiles include [001] for endorsing strongly disagree, [000] for endorsing disagree, and [1—] for endorsing neither agree nor disagree, Once the midpoint pseudo-item is endorsed, the respondent indicates no opinion and does not have responses to the TOI pseudo-item or extreme response pseudo-item. Recoding j items in this way results in j*k pseudo-items.

Each $Y_{i j k}^{*}$ represents a dichotomous decision so that each pseudo-item in the IRTree in Figure 1 can be modeled using a two-parameter dichotomous IRT model, with item-specific discrimination and easiness parameters, for each $Y_{i j k}^{*}$ of the tree. The generalized IRTree model (Jeon & De Boeck, 2016), for person i endorsing the pseudo-item j in the k^th decision stage, is given by:

\Pr (Y_{i j k}^{*} = 1 | θ_{i k}, ω_{j k}) = g^{- 1} (a_{j k} (θ_{i k} + b_{j k})),

(2)

where $g^{- 1}$ is the inverse logit function and $ω_{j k}$ are the decision-stage specific parameters for item $j$ in decision stage k. The person parameter, $θ_{i k}$ , represents person i's latent trait at decision stage k. For example, person i’s latent propensity for ERS would be denoted as $θ_{i E R S}$ . The discrimination parameter, $a_{j k}$ , represents how well item j discriminates between those who are high on the latent trait at decision stage k and those who are low. The easiness parameter, $b_{j k}$ , represents how easy it is for respondents to endorse the pseudo-item j in decision stage k.

Multiple response styles, such as ERS and MRS, may be present in survey data, but previous multidimensional models have been limited by accounting for only one response style per survey (e.g., Bolt & Johnson, 2009). The IRTree provides flexibility to simultaneously account for, and parse out the effects of, multiple response styles, such as MRS and ERS, from the TOI. For example, Spratto et al. (2021) analyzed a 7-option response scale using two-competing IRTrees to model MRS and ERS. The first IRTree modeled four dichotomous hypothesized stages $(K = 4)$ with MRS, TOI, ERS, and an undetermined trait while the other had three hypothesized stages $(K = 3)$ with MRS and TOI dichotomous branches and ERS as a trichotomous split. MRS may not always be included, such as in a study by Leventhal and Stone (2018) where four response options with two stages ( $K = 2$ ; TOI and ERS) were analyzed. IRTrees also provide the flexibility to hypothesize multiple response processes and examine group differences (Plieninger, 2021) in addition to other response styles, such as ARS (Leventhal & Zigler, 2021; Park & Wu, 2019). These models may be used with scales without a neutral option or with scales where acquiescence (tendency to agree) is not possible, as in the case of frequency rating scales where positivity bias, not ARS, may be present. Thissen-Roe and Thissen (2013) and Plieninger (2021) outline generalized IRTree frameworks to be applied to for response styles for both odd numbered and even numbered response scales.

Despite the beneficial perspective of IRTrees, there has been limited research to date on the stability of response style over multiple time points within this valuable framework. The purpose of this study is to apply a novel model, the longitudinal IRTRee (LIRTree), to a real data set and evaluate the validity of inferences made from the model. The specific research questions are as follows: Does the LIRTree validly account for multiple response styles across multiple waves of data collection (i.e., multiple time points)? Are extreme and MRS traits stable across multiple waves of data collection in an empirical data set measuring sexual knowledge? How do extreme and MRS traits change in relation to the substantive TOI across multiple waves of data collection in an empirical data set measuring sexual knowledge? How do the substantive interpretations when accounting for response style differ from the interpretations made when the explicit modeling of response style is ignored?

Method

The longitudinal model for response processes builds on the longitudinal IRT model for dichotomous responses (Kim & Camilli, 2014), which leads into an extension for response styles, the longitudinal IRTree (LIRTree; Ames & Leventhal, 2021).

Longitudinal IRT

A traditional two-parameter logistic IRT model is defined via:

\Pr (Y_{i j}^{*} = 1 | θ_{i}) = g^{- 1} (a_{j} (θ_{i} + b_{j})),

(3)

with longitudinal extension, per specification in Kim and Camilli (2014):

\Pr (Y_{i j t}^{*} = 1 | θ_{i t}, ω_{j}, μ_{t}) = g^{- 1} (a_{j} (μ_{t} + θ_{i t} + b_{j})),

(4)

where $\Pr (Y_{i j t}^{*} = 1 | θ_{i t})$ is the probability of person i responding correctly at time t to item j. Alternatively, $\Pr (Y_{i j t}^{*} = 1 | θ_{i t})$ can be thought of as the probability of person endorsing item j at time t. The population parameter effects of individuals are µt and $θ_{i t}$ , where µt is average performance at time t across all i individuals and $θ_{i t}$ is the specific effect for individual i as a departure from µt. This model is multidimensional because each of the t time points provides a different measurement to the same, constant instrument (Kim & Camilli, 2014). The covariance structure of θ can be structured or unstructured. To set the scale in a frequentist framework, one common approach is to constrain $μ_{t = 1} = 0$ and $σ_{t = 1} = 1$ , freely estimating $μ_{t > 1}$ and $σ_{t > 1}$ . Thus, $μ_{t > 1}$ can be interpreted as the shift in the mean performance from Time 1 to Time t, with negative shifts representing a mean decrease in the trait and positive shifts representing a mean increase in the trait. The item parameters, $a_{j}$ and $b_{j}$ , represent the discrimination and easiness of item j, respectively. The item parameters for the model in Equation 4 are invariant across time points, as specified by the absence of a time-specific subscript.

Longitudinal IRTree

Following the longitudinal IRT framework, Ames and Leventhal (2021) extend the IRTree in Equation 2 to now include multiple time points, resulting in $K * T$ latent traits per respondent: $θ_{i k t}$ , for $k = 1, 2, \dots, K$ (e.g., 1 = $M R S,$ 2 = $E R S,$ and $3 = T O I$ for a three-process model) at time points $t = 1, 2, \dots, T$ :

\Pr (Y_{i j k t}^{*} = 1 | θ_{i k t}, μ_{k t}, ω_{j k}) = g^{- 1} (a_{j k} (μ_{k t} + θ_{i k t} + b_{j k})) .

(5)

In the LIRTree approach, the linear latent score for person i, at time t, in decision-stage k, is a function of the item effects $ω_{j k}$ ( $a_{j k}$ , $b_{j k}$ ) and respondent effects ( $μ_{k t}, θ_{i k t}$ ). The item parameter $b_{j k}$ represents the j^th item easiness, or intercept, at decision stage k. In the same way that easiness of a dichotomous item is related to its probability of eliciting a correct response, easiness of the dichotomous pseudo-item pertains the probability of eliciting an endorsement ( $Y_{i j k t}^{*} = 1$ ) at decision-stage k. An alternative interpretation of the $b_{j k}$ is that the parameter represents the easiness of endorsing the decision of the k^th decision stage. For example, $b_{j E R S}$ represents how easy it is for the dichotomous pseudo-item j to elicit an extreme response or how easy it is to endorse the extreme option. Large, positive values indicate the pseudo-item elicits more extreme responses and negative values indicate the pseudo-item elicits fewer extreme responses. The slope parameter, $a_{j k}$ , is interpreted as how well the item discriminates between individuals who are low on the latent trait in decision stage k and those who are high on the latent trait in decision stage k (de Jong et al., 2008). For example, the parameter $a_{j E R S}$ represents how well the item discriminates between individuals who are low in ERS and those high in ERS. One feature of this model is that the item parameters are time-independent. That is, there is measurement invariance across the time points. We assume there is no item parameter drift from Time 1 to Time $t$ of the item parameters in each of the k decision-stages and the item parameters do not have time-specific subscripts.

In the LIRTree, µkt and $θ_{i k t}$ are the population and individual respondent person effects. Specifically, µt is the average of the k^th trait at Time t and $θ_{i k t}$ is the specific effect for individual i as a departure from µt in decision-stage $k$ . For example, µERSt is the average level of the extreme response trait at time $t$ and $θ_{i E R S t} i s$ respondent $i$ 's extreme response trait deviation from the average extreme response trait. Positive $θ_{i k t}$ represent individuals with trait k above the mean at Time $t$ and negative $θ_{i k t}$ represent individuals with trait k below the mean at time t.

The covariance structure of θ allows for covariance among different times and traits using an unstructured covariance matrix, with the exception that the mean and variance of the latent traits at the first time point are fixed to 0 and 1, respectively. That is, $θ_{i k (t = 1)} ~ N o r m a l (0, 1)$ and $θ_{i k t} ~ N o r m a l (μ_{k (t = t)}, σ_{k (t = t)}^{2})$ where $t = 2, \dots, T$ . Accordingly, the latent means at times $t > 1$ represent the change, or shift, in average trait k from time $t = 1$ to time $t$ . For example, $μ_{E R S (t = 2)} = 0.5$ indicates an increase in average ERS from Time 1 to Time 2, and $μ_{T O I (t = 2)} = - 0.2$ indicates a decrease in the substantive TOI from Time 1 to Time 2, on average. Thus, an individuals’ latent TOI trajectory can be mapped using the TOI latent trait mean at time t and their own personal TOI effect at Time t. Spread of latent traits at Time t is represented by $σ_{k t}^{2}$ , which indicates whether respondents are becoming more or less homogenous in their response styles and TOI over time. The covariance between latent trait $k$ at Times $t_{1}$ and $t_{2}$ , $σ_{k (t = 1, t = 2)}$ , is freely estimated, across all k latent traits and t = 2 time points.

Empirical Example: Measurement Instrument

We present an empirical example to demonstrate the usefulness of the model and parameter interpretation. The empirical example uses a 6-item sexual knowledge scale from the National Longitudinal Study of Adolescent to Adult Health (Add Heath; Harris, 2009), measured across two time points (Wave I and Wave II of Add Health data collection). Wave I took place between 1994 and 1995. For Wave II, nearly 15,000 of the Wave I respondents were interviewed from April to August 1996, approximately 1 year after the Wave I interview. The six items are presented in Table 1, along with the original variable name from the Add Health data set. All items respond to the prompt: “Do you agree or disagree with the following statement?” Response options include strongly disagree, disagree, neither agree nor disagree, agree, and strongly agree, numerically represented by scores 1 to 5, respectively. A random sample of n = 3,000 respondents was selected from the full sample using the SURVEYSELECT procedure in SAS (SAS Institute Inc., 2008). All respondents in the random sample had complete data, such that they provided item responses to both waves of data collection at $t = 1$ and $t = 2$ .

Table 1.

Sexual Knowledge Scale of the Add Health Survey.

Add Health Survey Item	Wave I	Wave II
1. Your closest friends are quite knowledgeable about the withdrawal method of birth control.	H1PF9	H2PF39
2. Your closest friends are quite knowledgeable about the rhythm method of birth control and when it is a “safe” time during the month for a woman to have sex and not get pregnant.	H1PF17	H2PF41
3. Your closest friends are quite knowledgeable about how to use a condom correctly.	H1PF12	H2PF40
4. You are quite knowledgeable about the withdrawal method of birth control.	H1PF22	H2PF38
5. You are quite knowledgeable about the rhythm method of birth control and when it is a “safe” time during the month for a woman to have sex and not get pregnant.	H1PF11	H2PF37
6. You are quite knowledgeable about how to use a condom correctly.	H1PF6	H2PF36

Note. Wave I and Wave II contain the variable names in the Add Health public use data file.

Description of the Sample

Response percentages and item mean scores of the six items are provided in Table 2. Average item scores increased from Wave I to Wave II for all items, indicating an increase in average sexual knowledge scale scores for the sample. The largest increase in mean item score was for Item 2 ( $Δ = 0.16$ ).

Table 2.

Response Percentages.

Item	Wave	Strongly disagree, %	Disagree, %	Neither agree nor disagree, %	Agree, %	Strongly agree, %	M, %	SD, %
1	I	4.167	10.133	28.967	37.433	19.300	3.576	1.041
	II	3.300	10.400	29.400	37.133	19.767	3.597	1.021
2	I	4.233	13.667	33.633	35.467	13.000	3.393	1.013
	II	3.233	10.600	31.967	36.167	18.033	3.552	1.007
3	I	2.600	6.300	19.867	44.167	27.067	3.868	0.969
	II	1.733	4.700	19.433	46.167	27.967	3.939	0.903
4	I	4.300	12.767	20.767	41.933	20.233	3.610	1.075
	II	4.067	12.933	17.700	40.067	25.233	3.695	1.104
5	I	5.867	15.533	20.933	38.400	19.267	3.497	1.139
	II	4.400	15.000	17.933	38.267	24.400	3.633	1.134
6	I	3.067	5.500	9.233	39.600	42.600	4.132	0.999
	II	1.900	4.433	10.800	43.100	39.767	4.144	0.913

Note. Values in table represent the percentage of respondents in each category.

Estimation

All data analysis and model fitting was performed in Mplus, which relies on the Gibbs sampling algorithm. Details of the sampling algorithm can be found in Asparouhov and Muthén (2010) for item response models. Convergence was assessed using the potential scale reduction factor (PSRF; Gelman & Rubin, 1992), the default in Mplus. PSRF values for all model parameters were close to 1. As a rule of thumb, a PSRF lower than 1.2 would indicate convergence (Muthén & Asparouhov, 2012). Prior distributions for the item parameters were weakly informative (i.e., $b_{j k} ~ N (0, σ^{2} = 4); a_{j k} ~ l o g n o r m a l (0, σ^{2} = 0.5)$ ) and the prior on the vector of latent abilities was multivariate normal (i.e., $θ ~ M V N (μ, Σ)$ ). Priors on the individual elements of µ were specified as $μ_{k (t = 2)} ~ N (0, σ^{2} = 2)$ . For the latent trait correlation matrix, $Σ$ , the default prior in Mplus is inverse-Wishart ( $I W (I, k + 1)$ ), where $k$ is the number of latent traits and $I$ is an identity matrix. Under this specification, the marginal distribution for all correlations is uniform on the interval $(- 1, 1)$ , and the marginal distributions of the variance is $I G (1, 0.50)$ , for which the mode is $0.25$ (Asparouhov & Muthén, 2010).

Parameter Recovery

Previous simulation studies (Ames & Leventhal, 2021) demonstrated adequate parameter recovery of LIRTree parameters, manipulating conditions of survey length (10, 20 items); sample size (1,000, 2,000 people); time points (t = 2, 3); latent trait variances ( $σ_{k (t = t)}^{2}$ = 1.3, 0.7); shift (two mean shift conditions, representing small and large shifts); intertrait correlations, with a baseline condition and manipulation of $ρ_{θ M R S, θ E R S}$ , $ρ_{θ M R S, θ T O I}$ , and $ρ_{θ E R S, θ T O I}$ ; and prior distributions. The prior distributions included default Mplus priors compared with “matching” prior distributions, which mirror the generating item parameter distributions.

Outcomes in Ames and Leventhal (2021) included mean absolute bias (MAB), defined as the average absolute deviance between the posterior point estimate (i.e., expected a posteriori [EAP]) and the generating value, and coverage rate, defined as the percentage of replications in which the highest posterior density encompassed the generating value. Each condition was replicated 250 times, with estimation in Mplus version 8.3 (Muthén & Muthén, 1998-2017), which relies on the Gibbs sampler for MCMC estimation (Asparouhov & Muthén, 2010; Gelman et al., 2004). Convergence times ranged from 3 minutes, 44 seconds (i.e., for the condition with 1,000 people, 10 ordinal items, two time points) to 2 hours, 36 minutes and 27 seconds (i.e., for the condition with 2,000 people, 20 ordinal items, three time points) when run in parallel on two processors. The computer processor is an Intel® Xeon® W-2155 CPU with 10 cores and 20 threads. Times to run indicate that computational time for Bayesian estimation of tree-like models is not prohibitive.

Item parameter recovery results improved as sample size increased, as measured by MAB, a finding that agrees with previous simulation that investigated noncompensatory MIRT models. Adequate item parameter recovery was found for a sample size of 2,000 at both survey lengths. Larger correlations between latent traits, such as $ρ_{θ M R S, θ E R S}$ , resulted in worse item parameter recovery. The variance parameter for the TOI decision stage had poor recovery for two time points, seeming to affect recovery of the correlations involving the TOI latent trait (i.e., $ρ_{θ M R S, θ T O I}$ , $ρ_{θ E R S, θ T O I}$ ). In turn, this resulted in relatively worse recovery of the loading parameters for the $ρ_{θ M R S, θ T O I}$ and $ρ_{θ E R S, θ T O I}$ conditions.

Matching priors improved MAB for item loading and variance parameter recovery, and had minor improvement for shift parameter recovery. Specifically, Ames and Leventhal (2021) demonstrated recovery was better in the “matching” prior conditions than for the Mplus baseline conditions for item loadings, particularly with two time points. The same gains were not seen across conditions for the three time point model, indicating that multiple measurements of the same item improves parameter recovery for loadings. However, when the $ρ_{θ E R S, θ T O I}$ correlation was manipulated, using matching priors did improve MAB. Previous researchers have found response styles to be relatively stable over time (e.g., Wetzel et al., 2016), so prior information for the response style shift parameters could reflect this stability.

Results

Drift

To evaluate the longitudinal assumption that item parameters were invariant across the two waves of data collection, we undertook a systematic procedure using the same approach as Kim and Camilli (2014). To evaluate longitudinal invariance from an IRT Millsap (2010) suggests fitting a model that constrains the item parameters to be invariant across all measurement occasions. Then, compare, using a likelihood ratio test, the invariant model to one that allows for varying item parameters across measurement occasions. This is essentially the approach we have taken, as have Kim and Camilli (2014), but adapted to a Bayesian framework.

First, we allowed only $b_{j T O I}$ to vary across Waves I and II, rather than remaining fixed for both waves, for the j^th item at a time. Specifically, for item j, we allowed $b_{j T O I}$ to be wave-specific, estimating $b_{j T O I, t = 1}$ and $b_{j T O I, t = 2}$ separately, but keeping the other item easiness fixed (i.e., $b_{j E R S}$ and $b_{j M R S}$ did not vary). Following Kim and Camilli (2014), we did not vary item discrimination ( $a_{j k}$ ) across waves in this approach. The model with the wave-specific $b_{j T O I}$ was compared with the “base” model, which had no wave-specific $b_{j T O I}$ . Then, we allowed all three decision-stage easiness ( $b_{j T O I}, b_{j M R S}, b_{j E R S}$ ) to vary, for the j^th item at a time. For each model, we compared the deviance information criterion (DIC with the “base” model for which all item parameters were fixed at both waves (Spiegelhalter et al., 2002). The DIC for the base model was 61816.27 (effective number of parameters = 9979.80). Lunn et al. (2012) suggested DIC differences of more than 10 “might definitely rule out the model with the higher DIC,” whereas “differences between 5 and 10 are substantial” (p. 107). However, Plummer (2008) says that the effective number of parameters must be small in relation to the sample size for these rough heuristics to hold. When that criteria does not hold, DIC underpenalizes complex models. Thus, we chose a rough guideline of a change in DIC of greater than 1% over the base model to guide the evaluation of drift. The model comparison analysis favored the base model (see Table 3), for which all items are fixed across waves, computed via:

% Δ D I C = 100 * \frac{D I C_{v a r y} - D I C_{b a s e}}{D I C_{b a s e}}

(6)

where $D I C_{v a r y}$ is the model in which item easiness is allowed to vary and $D I C_{b a s e}$ is the base model for which all item easiness are time-invariant.

Table 3.

DIC Comparison.

Item	TOI only varies	ERS, MRS, TOI vary	LGRM
1	11.98 (0.019%)	22.26 (0.036%)	8.69 (0.014%)
2	1.42 (0.002%)	−1.94 (−0.003%)	18.97 (0.031%)
3	24.15 (0.039%)	24.64 (0.04%)	0.02 (0%)
4	1.04 (0.002%)	9.29 (0.015%)	2.14 (0.003%)
5	5.88 (0.01%)	9.76 (0.016%)	0.91 (0.001%)
6	18.34 (0.03%)	25.37 (0.041%)	22.17 (0.036%)

Note. Values in table represent change in DIC over baseline, with values in parentheses the percentage change from baseline. The column “TOI Only Varies” is the model for which only the item difficulty parameters of the TOI node were allowed to vary and the column “ERS, MRS, TOI Vary” is the model for which item difficulty parameters of all three nodes were allowed to vary. “LGRM” represents the longitudinal drift analysis for the LGRM. TOI = trait of interest; ERS = extreme response style; MRS = midpoint response style; LGRM = longitudinal graded response model; DIC = deviance information criterion.

Parameter Interpretation

Because the drift analysis indicated the base model is most appropriate, we present the results from the analysis with fixed item parameters across all k decision-stages and both waves.

Item Parameters

The results in Table 4 present the item parameter posterior means and standard deviations. Item easiness in the MRS decision stage indicate high levels of MRS are required to endorse the midpoint option. Item 6 was the most difficult for midpoint endorsement ( $b_{6 M R S} = - 1.743$ ). For all items, the item easiness for the TOI were positive, indicating the items were relatively easy for respondents to endorse. Item 3’s TOI decision stage was easiest to endorse ( $b_{3 T O I} = 2.345$ ). In the ERS decision stage, item easiness indicate moderate levels of ERS are required to endorse the extreme options. Item 6 was easiest for extreme endorsement ( $b_{6 E R S} = - 0.109$ ) and Item 2 was the most difficult for extreme endorsement ( $b_{2 E R S} = - 1.602$ ). Taken together, empirical results show that items for which it is easier to endorse the TOI, it tends to be more difficult to endorse the midpoint option, but easier to endorse the extreme option. Regarding item discrimination, Item 2 was best for distinguishing between low- and high-midpoint responders ( $a_{2 M R S} = 1.654$ ). Item 4 was best for distinguishing between low- and high-extreme responders ( $a_{4 E R S} = 1.975$ ) and those low and high on the TOI ( $a_{4 T O I} = 2.449$ ).

Table 4.

Item Parameter Estimates.

Decision-stage	Item	Item discrimination $(a_{j k})$		Item easiness $(b_{j k})$
Decision-stage	Item	M	SD	M	SD
ERS	1	1.455	0.093	−1.046	0.072
	2	1.844	0.145	−1.602	0.121
	3	1.637	0.107	−0.828	0.072
	4	1.975	0.143	−1.317	0.106
	5	1.64	0.112	−1.054	0.078
	6	1.136	0.064	−0.109	0.044
MRS	1	1.285	0.076	−0.862	0.057
	2	1.654	0.109	−0.817	0.072
	3	1.349	0.085	−1.447	0.073
	4	1.121	0.074	−1.308	0.061
	5	0.989	0.067	−1.182	0.055
	6	0.84	0.067	−1.743	0.066
TOI	1	1.755	0.128	1.444	0.094
	2	2.251	0.193	1.354	0.114
	3	1.74	0.159	2.345	0.143
	4	2.449	0.209	1.868	0.148
	5	2.017	0.166	1.279	0.106
	6	1.317	0.106	2.168	0.105

Note. Mean is the posterior mean (EAP estimate) and SD is posterior standard deviation. ERS = extreme response style; MRS = midpoint response style; TOI = trait of interest; EAP = expected a posteriori.

Person Parameters

Table 5 shows a decrease in levels of extreme ( $μ_{E R S (t = 2)} = - 0.138$ ) and midpoint ( $μ_{M R S (t = 2)} = - 0.094$ ) response styles from Wave I to Wave II, as indicated by the negative shift parameters at Wave II. The positive TOI mean at Wave II ( $μ_{T O I (t = 2)} = 0.240$ ) indicates an increase in the sexual knowledge trait from Wave I to Wave II. All variances at Wave II indicate the latent traits were more variable at Wave II than Wave I, particularly ERS ( $σ_{E R S (t = 2)}^{2} = 2.268$ ). Figures 2 to 4 illustrate individual respondents’ trajectories from Wave I to Wave II and the average trajectory for MRS, TOI, and ERS, respectively. The trajectories in Figures 2 to 4 were constructed using the EAP point estimates. Table 6 presents correlations among latent traits for the estimated model.

Table 5.

Latent Trait Parameter Estimates.

Parameter	Decision-stage	M	SD
$μ_{E R S (t = 2)}$	ERS	−0.138	0.05
$μ_{M R S (t = 2)}$	MRS	−0.094	0.046
$μ_{T O I (t = 2)}$	TOI	0.240	0.047
$σ_{E R S (t = 2)}^{2}$	ERS	2.268	0.213
$σ_{M R S (t = 2)}^{2}$	MRS	1.175	0.128
$σ_{T O I (t = 2)}^{2}$	TOI	0.996	0.098
$μ_{G R M, T O I (t = 2)}$	LGRM	0.140	0.028
$σ_{G R M, T O I (t = 2)}^{2}$	LGRM	1.126	0.065

Note. Mean is the posterior mean (EAP estimate) and SD is posterior standard deviation. µ is the shift, representing the latent trait mean at Wave II and $σ_{(t = 2)}^{2}$ is the variance of the latent trait at the second time point. TOI = trait of interest; MRS = midpoint response style; ERS = extreme response style; LGRM = longitudinal graded response model.

Figure 2.

Wave I to Wave II trajectories for midpoint response style (MRS).

Figure 3.

Wave I to Wave II trajectories for the sexual knowledge trait of interest (TOI).

Figure 4.

Wave I to Wave II trajectories for extreme response style (ERS).

Table 6.

Correlation Among Latent Traits.

	ERS t2	MRS t1	MRS t2	TOI t1	TOI t2	GRM t1	GRM t2	Coll t1	Coll t2
ERS t2	0.598
MRS t1	−0.442	−0.279
MRS t2	−0.317	−0.360	0.609
TOI t1	0.117	0.143	−0.433	−0.359
TOI t2	0.166	0.281	−0.426	−0.558	0.794
GRM t1	0.483	0.352	−0.563	−0.408	0.745	0.607
GRM t2	0.368	0.579	−0.423	−0.604	0.542	0.767	0.646
Coll t1	0.197	0.189	−0.582	−0.430	0.850	0.683	0.862	0.570
Coll t2	0.190	0.262	−0.452	−0.656	0.642	0.863	0.609	0.833	0.693

Note. Values represent Pearson’s correlations among latent trait point estimated. GRM is the latent trait from the LGRM; and Coll is the latent trait from the GRM with categories collapsed. T1 and T2 are Time 1 and Time 2, respectively. ERS = extreme response style; MRS = midpoint response style; TOI = trait of interest from the LIRTree; LGRM = longitudinal graded response model.

Benefits of LIRTree

To demonstrate the benefit of the LIRTree framework, we conducted a series of studies. The first was a comparison of the conclusions drawn from the LIRTree, as they relate to the substantive TOI, and those drawn from a model ignoring response style using the longitudinal graded response model (LGRM; GRM from Samejima, 1969). To determine category threshold drift, we used a parallel process and similar DIC cut criterion as with the LIRTree drift analysis described previously. We also used the same Bayesian MCMC framework for this analysis in Mplus. Similarly, no category threshold parameters indicated evidence of drift (see Table 3). Table 5 also contains the shift, variance, and covariance parameters of the LGRM for comparison with the LIRTree. Using the LGRM, which fails to account for response style, growth in the sexual knowledge substantive trait was smaller for the LGRM ( $μ_{G R M, T O I (t = 2)} = 0.140$ ) than that of the LIRTree ( $μ_{T O I (t = 2)} = 0.240$ ), demonstrating that failure to account for response style using a LGRM underestimates sexual knowledge growth. The standardized effect size between shift parameters is 2.59. The correlation between Wave I and Wave II sexual knowledge scores ( $σ_{T O I, (t = 1, t = 2)}$ ) is positive for both models, but smaller in magnitude for the LGRM (0.794 for the LIRTree and 0.646 for the LGRM), showing that failing to account for response style underestimates the relationship between traits at Wave I and Wave II for this empirical data. Variation in the sexual knowledge substantive trait at Wave II was larger for the LGRM ( $σ_{G R M, T O I (t = 2)}^{2} = 1.126$ ) than that of the LIRTree ( $σ_{T O I (t = 2)}^{2} = 0.996$ ), demonstrating that failure to account for response style using a LGRM overestimates variation in sexual knowledge. The standardized effect size between variance parameters is 1.56.

To understand shifts in traits based on response patterns, we present responses and latent trait estimates for four individuals in Table 7. Each person has the same response set at Wave I (i.e., responding with “5, Strongly Agree” to all items), but differing response sets at Wave II. Although they have the same response sets at Wave I, ${\hat{θ}}_{i, E R S (t = 1)}$ , ${\hat{θ}}_{i, M R S (t = 1)},$ and ${\hat{θ}}_{i, T O I (t = 1)}$ differ. Because all traits are estimated simultaneously, responses at Wave II inform trait estimates at Wave I, and vice versa. Therefore, to understand differences among the three traits at Wave I and Wave II, we must consider the full response set across both waves of data collection.

Table 7.

Illustration of Response Shifts.

ID	Wave, t	Response set	${\hat{θ}}_{i, E R S (t = t)}$	${\hat{θ}}_{i, M R S (t = t)}$	${\hat{θ}}_{i, T O I (t = t)}$	${\hat{θ}}_{i, L G R M, T O I (t = t)}$
1	I	555555	1.76	−0.87	0.71	2.17
	II	555555	2.19	−1.07	0.99	2.48
2	I	555555	1.44	−0.88	0.73	1.79
	II	444444	−1.03	−1.04	0.98	0.54
3	I	555555	1.63	−0.20	0.62	1.62
	II	333333	0.95	1.79	0.73	−0.53
4	I	555555	1.76	−0.50	−0.02	1.52
	II	111333	2.05	0.45	−1.25	−1.26

Note. ${\hat{θ}}_{k (t = t)}$ is the posterior mean (EAP estimate) for the kth latent trait, where k represents TOI = trait of interest; MRS = midpoint response style; ERS = extreme response style; LGRM = longitudinal graded response model.

ERS

First, we consider ERS tendencies. Each respondent selected 6 extreme responses at Wave I (all: “5, strongly agree”), but only Respondents 1 and 4 selected extreme responses at Wave II (six extreme responses vs. three, respectively). Therefore, when the entire set is considered, Respondents 1 and 4 have the greatest ${\hat{θ}}_{i, E R S (t = 1)}$ and ${\hat{θ}}_{i, E R S (t = 2)}$ because they contain extreme responses at both data collection points. Overall, the entire sample of respondents decreased in ERS from Wave I to Wave II ( $μ_{E R S (t = 2)} = - . 174$ ). Therefore, continuing to provide extreme responses at Wave II is likely due to higher extreme response tendencies than just exhibited at Wave I, resulting in ${\hat{θ}}_{i, E R S (t = 2)} > {\hat{θ}}_{i, E R S (t = 1)}$ for both Respondents 1 and 4.

Respondents 2 and 3 failed to endorse any extreme options at Wave II, resulting in ${\hat{θ}}_{i, E R S (t = 2)} < {\hat{θ}}_{i, E R S (t = 1)}$ for both. However, there are substantial differences in ${\hat{θ}}_{i, E R S (t = 1)}$ and ${\hat{θ}}_{i, E R S (t = 2)}$ between the two respondents ( ${\hat{θ}}_{2, E R S (t = 1)} = - 1.03$ , ${\hat{θ}}_{3, E R S (t = 1)} = . 95$ ), which can be explained by other notable patterns. Respondent 2 endorsed all agreeable responses at both waves; comparatively, Respondent 3 only endorsed agreeable responses at Wave I. Additionally, from Wave I to Wave II, the entire sample shifted toward higher TOI ( $μ_{T O I (t = 2)} = . 2687$ ). Therefore, it is unlikely for Respondent 3 to be high on TOI at Wave I and shift dramatically downward (as would be represented by total score) at Wave II against the upward trend in the sample. Rather, it is more likely that Respondent 3’s strongly agree endorsements at Wave I are due to higher ERS tendencies ( ${\hat{θ}}_{3, E R S (t = 1)} > {\hat{θ}}_{2, E R S (t = 1)}$ ), and a lower TOI ( ${\hat{θ}}_{3, T O I (t = 1)} < {\hat{θ}}_{2, T O I (t = 1)}$ ), as compared with Respondent 2 who did not have as dramatic of a shift. Although Respondent 2 consistently endorsed lower response options at Wave II as compared with Wave I, countering the trend of the sample, their scores remained agreeable. Therefore, their extreme responses at Wave I are more likely influenced by high TOI as compared with Respondent 3’s high scores. As compared with Wave I, at Wave II, we see a dramatic decrease in ${\hat{θ}}_{2, E R S (t = 2)}$ for Respondent 2, but not as nearly as substantial for Respondent 3. Although at Wave II Respondent 3 did not select an extreme response, we do not have information on whether that is because of adverse tendency to select extremes. Since Respondent 3 endorsed the middle response for all items, their response profile for each item is [1—], indicating missing information for all extreme response pseudo-items. This missing data results in ${\hat{θ}}_{i, E R S (t = 2)} < {\hat{θ}}_{i, E R S (t = 1)}$ as there is no collected information to help inform their extreme response tendencies at Wave II. Thus, the difference between ${\hat{θ}}_{i, E R S (t = 2)}$ and ${\hat{θ}}_{i, E R S (t = 1)}$ is lower for Respondent 3 than it is for Respondent 2.

MRS

For MRS, Respondents 1 and 2 failed to endorse the midpoint on items at both Wave I and Wave II. For these respondents, the difference between ${\hat{θ}}_{i, M R S (t = 2)}$ and ${\hat{θ}}_{i, M R S (t = 1)}$ is roughly equivalent to the shift in MRS tendencies in the sample ( $μ_{M R S (t = 2)} = - 0.1972$ ). On the other hand, Respondent 3 endorsed 6 midpoint responses at Wave II and Respondent 4 endorsed three midpoint responses at Wave II, resulting in greater ${\hat{θ}}_{i, M R S (t = 1)}$ for Respondents 3 and 4 as compared with Respondents 1 and 2. For Respondent 3, we see a dramatic increase from ${\hat{θ}}_{3, M R S (t = 1)}$ to ${\hat{θ}}_{3, M R S (t = 2)}$ and a rise from ${\hat{θ}}_{4, M R S (t = 1)}$ to ${\hat{θ}}_{4, M R S (t = 2)}$ , but it is not as large for Respondent 4 as Respondent 3 due to a lower frequency in midpoint selection at Wave II.

TOI

Due to these patterns in response style tendencies, ${\hat{θ}}_{i, T O I (t = 1)}$ and ${\hat{θ}}_{i, T O I (t = 2)}$ differed among respondents. Respondents 1 and 2 have similar TOI estimates at both waves. Since, the respondents both endorsed positive responses across waves, the difference in response sets at Wave II is explained through ${\hat{θ}}_{i, E R S (t = 2)}$ , and ${\hat{θ}}_{i, T O I (t = 2)}$ . This is because, in the model, TOI is only informed by positive or negative endorsement, not the extreme or nonextreme selection, akin to collapsing agree with strongly agree and disagree with strongly disagree. When compared with the LGRM where no response styles were considered, ${\hat{θ}}_{i, T O I (t = 1)}$ from the LIRTree for Respondents 1 and 2 have been adjusted down due to the presence of high ERS. For Respondent 2, ${\hat{θ}}_{2, T O I (t = 2)}$ has been adjusted downward from the LGRM to LIRTree due to high ${\hat{θ}}_{2, E R S (t = 2)}$ . In other words, without consideration of ERS, the trait is likely overestimated. On the other hand, for Respondent 3, ${\hat{θ}}_{3, T O I (t = 2)}$ has been adjusted upward from LGRM to LIRTree due to low ${\hat{θ}}_{3, E R S (t = 2)}$ . Without the consideration of ERS, Respondent 2’s TOI is likely underestimated because of the adverse nature toward selecting extremes at Wave II. In other words, due to the resistance toward selecting an extreme response, the consistent endorsement of agree across results in an underestimate of their true TOI.

Respondent 3 is estimated to have high ERS tendencies at Wave I and high MRS tendencies at Wave II. Therefore, as compared with TOI estimates using the LGRM, ${\hat{θ}}_{3, T O I (t = 1)}$ is adjusted downward and ${\hat{θ}}_{3, T O I (t = 2)}$ is adjusted upward. As seen in the table, Respondent 4’s responses pattern ranges considerably over the response scale across waves. Therefore, although Respondent 4’s response set at Wave I is the same as other individuals, their TOI is likely more moderate than other respondents and their high ERS tendencies likely have a more substantial influence on response selection. As a result, ${\hat{θ}}_{4, T O I (t = 1)}$ is lower as compared with the other respondents. At Wave II, their lower total score is a result of high extreme response tendencies and moderate MRS tendencies. Because their extreme responses are all consistently on the disagree side of the response scale, response style effects on ${\hat{θ}}_{4, T O I (t = 2)}$ are likely canceled out. In other words, as compared with TOI estimates from the LGRM, their TOI estimate from the LIRTree is likely raised due low extremes; but lowered due to MRS. Therefore, these adjustments effectively cancel each other out and ${\hat{θ}}_{4, T O I (t = 2)}$ is similar for the LIRTree and LGRM.

Model Assumption Efficacy

We conducted model assumption efficacy studies of the response process assumed by the LIRTree. To do so, we collapsed extreme and nonextreme categories and fit modified responses using a three-category LGRM. Namely, we collapsed Category 1 with Category 2 and collapsed Category 4 with Category 5. Figure 5 presents scatterplots among LIRTree TOI, ERS, and MRS latent trait estimates with three-category LGRM TOI and 5-category LGRM TOI at both waves. In Figure 5, the top two rows represent Waves I and II of the five-category LGRM and the bottom two rows represent Waves I and II of the three-category LGRM. The columns, from left to right, represent scatter plots between the MRS, TOI, and ERS LIRTree latent trait point estimates with the LGRM TOI latent trait point estimates. The triangular relationship between LIRTree MRS and LGRM TOI at both waves indicates that individuals with high LIRTree MRS estimates tend to have average GRM TOI for both the three- and five-category LGRM models. Those with low MRS tendencies tend to have considerable variance in GRM TOI estimates. However, respondents with high and low GRM TOI had low LIRTree MRS at both time points. This pattern provides evidence the LIRTree MRS is capturing a midpoint response tendency. That is, collapsing the extreme categories has little effect on the relationship of LGRM TOI trait with the LIRTree MRS trait.

Figure 5.

Comparison of TOI, ERS, MRS estimated using IRTree, TOI estimated using GRM 5 categories, and TOI estimated using GRM 3 categories from Wave I to Wave II.

There is a relationship between LIRTree TOI and LGRM TOI for three- and five-category models. There are trivial differences in the strength of the relationship between the three-category LGRM TOI and LIRTree TOI and the strength of the relationship between the five-category LGRM TOI and LIRTree TOI at both waves. This indicates that the hypothesized LIRTree TOI information is captured adequately with the three-category LGRM. Comparing three- and five-category LGRM TOI correlates indicates the LIRTree method captures ERS. At both waves, there is no relationship between three-category LGRM TOI and LIRTree ERS. There is, however, a relationship between LIRTree ERS and five-category LGRM TOI. This indicates that extreme response behavior is captured in having Categories 1 and 5; as proposed in the LIRTree. Together, these results indicate that LIRTree TOI and five-category LGRM TOI are capturing similar information, however, the LIRTree captures ERS and MRS tendencies that are confounding TOI trait estimates using the five-category LGRM.

Conclusions

This study has presented and evaluated a longitudinal response process model. Applied researchers will need to convert the ordinal survey items to dichotomous pseudo-items in software such as SAS or R. The appendix provides an example of converting 10 ordinal survey at two time points items to 60 dichotomous pseudo-items (i.e., 10 survey items represents 30 pseudo-items, and this must be replicated for two time points). The appendix provides generic Mplus code for the simple example with 10 items and two time points.

The study’s research questions explored LIRTree model specification, and model validation. We saw a decrease in response styles from Wave I to Wave II and an increase in the substantive sexual knowledge TOI. The correlation between Wave I and Wave II for ERS was similar in magnitude to those in Wetzel et al. (2016), who found a positive correlation of 0.678. This study found a correlation of 0.794 Despite the time frame differences examined (i.e., 1 year for this study compared with 2 years for Wetzel et al. [2016]), agreement of this model with another modeling approach validates use of the LIRTree for measuring changes in response styles over time. Similar to conclusions found in Wetzel et al. (2016) and Weijters et al. (2010a), we found stability of response styles across time points, echoing the call for response style effects to be considered in the analysis of questionnaire data obtained with ordinal rating scales. The approach in this study is novel in that it allows for modeling a mean change ( $μ_{k, t > 1})$ from across waves for both response styles and the substantive trait and makes use of categorical indicators rather than sum scores.

We demonstrated through four individuals in the sample that the latent traits of the LIRTree model are informed by response sets across all waves of data collection. To further evaluate the validity of the models, we undertook a series of studies. We found that failure to account for response style underestimated sexual knowledge trait growth from one wave to the next by comparing the growth from a unidimensional LGRM to the growth realized by the LIRTree. An alternative explanation is that the LIRTree overestimated growth. While validation of LIRTree inferences is still in its infancy, a series of studies in this article show that the LIRTree method captures ERS and MRS. The latent trait from the LIRTree TOI and the LGRM TOI showed a strong relationship, illustrating that the TOI from the LIRTree can be used as a refined measure of the substantive trait, mitigating the effects of response style contamination.

To strengthen the argument for use of the LIRTree, we present some validity evidence from other studies. Ames and Myers (2021) examined the consequences of fitting an IRTree to unidimensional data (i.e., data generated from a GRM without response styles). In brief, they reported that magnitude of the discrimination parameter is a useful tool for detecting the presence of response styles. With unidimensional data, estimated IRTree item discrimination parameters tend to be very small, near zero, for the response style traits, but large for the substantive TOI. Examining the discrimination parameters in Table 4, we see the discrimination parameters are nonzero, providing evidence of the presence of response styles in the data. Ames and Leventhal (2021) demonstrated accurate estimation of the shift parameter. Plieninger and Meiser (2014) report on convergent evidence of construct and criterion validity of the IRTree model. Taken together, the validity evidence for use of the IRTree is growing, but we recognize the need for more studies investigating the LIRTree.

While we only estimated changes across two waves, the model easily generalizes to incorporate multiple time points. $θ_{i}$ is a vector with $K * T$ rows. In our example, $(K = 3) * (T = 2) = 6$ , but the model is flexible enough to allow for $T > 2$ and $K > 3$ . However, as the number of time points and latent traits increases, it is important to assess parameter recovery in the presence of greater dimensionality. In addition, we did not freely estimate all of the off-diagonal elements of $Σ$ . Important substantive conclusions can be gained from investigating the structure of the latent trait covariance matrix.

Future Research

Model comparison indices, such as the DIC, were of limited use for this study because the effective number of parameters was over three times as large as the sample size. Plummer (2008) states,

[An] important assumption is that the effective number of parameters pD must be small in relation to the sample size n. When this assumption does not hold, 2pD is a poor approximation to the optimism of the plug-in deviance, and DIC under-penalizes complex models.

Effective use of the DIC in the context of drift analysis will help determine whether measurement invariance holds over multiple time points. In addition, model-data fit approaches should be developed to evaluate the appropriateness of the model. In the Bayesian framework, the primary approach is posterior predictive model-checking, but such approaches have not been investigated for IRTrees or their longitudinal extensions. Choosing adequate priors and assessing prior sensitivity is also an important consideration.

The IRTree model at time point t contains three unidimensional, dichotomous pseudo-items rather than one multidimensional polytomous item. The pseudo-items are modeled explicitly as independent items, akin to a simple structure factor structure. The result is explicitly modelling the pseudo-items as noncompensatory. That may be reasonable for some traits, such as MRS and TOI, but perhaps not for ERS and TOI. For example, a respondent selection strongly agree, under the current model, is considered an extreme response and cannot be obtained by high levels of the TOI alone. One alternative option is the proposed model from Myers and Ames (2020), which allows for compensatory response process between TOI and ERS trait (i.e., high levels of TOI can compensate for low levels of ERS). Thus, strongly agree responses may be obtained without high levels of ERS. However, more research is needed on the compensatory IRTree before it is extended to the LIRTree.

Despite the limitations and need for more research into model evaluation, the longitudinal IRTree is an important addition to the survey research methodology. Kim and Camilli (2014) describe a possible extension for nonlinear growth in a longitudinal IRT framework, which could easily be adapted to the LIRTree approach, further illustrating the flexibility of the model. Stability of personality and attitude constructs may be ascribed to response style stability (e.g., Bachman & O’Malley, 1984; Billiet & Davidov, 2008), as demonstrated by the longitudinal model comparison. Böckenholt (2017) stresses the use of models that account for individual differences due to response style tendencies, especially for longitudinal studies (emphasized by Weijters et al., 2010a; Wetzel et al., 2016). The LIRTree is one important tool to account for response style in survey responses in longitudinal contexts.

Footnotes

Appendix

/*SAS code for converting survey items to dichotomous pseudo-items:*/

data recode;

set original;

array t1item{10} t1item1-t1item10;

array t2item{10} t2item1-t2item10;

array MRS{20} MRS1-MRS20;

array TOI{20} TOI1-TOI20;

array ERS{20} ERS1-ERS20;

do j=1 to 10;

if t1item[j]=3 then do; MRS[j]=1; TOI[j]=-99; ERS[j]=-99; end;

if t1item[j]=1 then do; MRS[j]=0; TOI[j]=0; ERS[j]=1; end;

if t1item[j]=2 then do; MRS[j]=0; TOI[j]=0; ERS[j]=0; end;

if t1item[j]=4 then do; MRS[j]=0; TOI[j]=1; ERS[j]=0; end;

if t1item[j]=5 then do; MRS[j]=0; TOI[j]=1; ERS[j]=1; end;

end;

do j=1 to 10;

if t2item[j]=3 then do; MRS[j+10]=1; TOI[j+10]=-99; ERS[j+10]=-99; end;

if t2item[j]=1 then do; MRS[j+10]=0; TOI[j+10]=0; ERS[j+10]=1; end;

if t2item[j]=2 then do; MRS[j+10]=0; TOI[j+10]=0; ERS[j+10]=0; end;

if t2item[j]=4 then do; MRS[j+10]=0; TOI[j+10]=1; ERS[j+10]=0; end;

if t2item[j]=5 then do; MRS[j+10]=0; TOI[j+10]=1; ERS[j+10]=1; end;

end;

run;

##Mplus code for 10 items:

TITLE:

LIRTree: 10 ordinal items

DATA:

File=C:\Users\data.csv;

VARIABLE:

IDVARIABLE=ID;

NAMES=ID item1-item60;

USEVARIABLES=all;

CATEGORICAL= item1-item60;

MISSING ARE all (-99);

ANALYSIS:

Estimator=Bayes;

Chains=2;

Bseed=11;

Point=Mean;

Processors=1;

MODEL PRIORS:

amrs1-amrs10~N(0,2);

aers1-aers10~N(0,2);

atoi1-atoi10~N(0,2);

bmrs1-bmrs10~N(0,2);

bers1-bers10~N(0,2);

btoi1-btoi10~N(0,2);

shiftMRS~N(0,2); shiftERS~N(0,2); shiftTOI~N(0,2);

varMRSt2~N(1,2); varERSt2~N(1,2); varTOIt2~N(1,2);

MODEL:

MRS1 BY i1-i10* (amrs1-amrs10);

[i1$1-i10$1] (bmrs1-bmrs10);

MRS2 BY i31-i40* (amrs1-amrs10);

[i31$1-i40$1] (bmrs1-bmrs10);

TOI1 BY i11-i20* (aTOI1-aTOI10);

[i11$1-i20$1] (bTOI1-bTOI10);

TOI2 BY i41-i50* (aTOI1-aTOI10);

[i41$1-i50$1] (bTOI1-bTOI10);

ERS1 BY i21-i30* (aERS1-aERS10);

[i21$1-i30$1] (bERS1-bERS10);

ERS2 BY i51-i60* (aERS1-aERS10);

[i51$1-i60$1] (bERS1-bERS10);

MRS1@1 ERS1@1 TOI1@1;

[MRS1@0 ERS1@0 TOI1@0];

MRS2* ERS2* TOI2* (varMRSt2 varERSt2 varTOIt2);

[MRS2* ERS2* TOI2*] (shiftMRS shiftERS shiftTOI);

OUTPUT:

tech1 tech8 cinterval(hpd);

SAVEDATA:

results are C:\Users\OUT.dat;

bparameters=C:\Users\POST.dat;

save=fscores(100);

factors=mrs1 toi1 ers1 mrs2 toi2 ers2;

file=C:\Users\PERSONOUT.dat;

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Allison J. Ames

References

Aichholzer

(2013). Intra-individual variation of extreme response style in mixed-model panel studies. Social Science Research, 42(3), 957-970. https://www-sciencedirect-com-443.web.bisu.edu.cn/science/article/pii/S0049089X13000227

Ames

A. J.

Leventhal

B. C.

(2021). Modeling changes in response style with longitudinal IRTree models. Multivariate Behavioral Research. Advance online publication. https://doi.org/10.1080/00273171.2021.1920361

Ames

A. J.

Myers

(2021, June). Validity evidence for item response trees [Paper presentation]. Annual Meeting of the National Council on Measurement in Education (NCME), Virtual paper presentation.

Asparouhov

Muthén

(2010). Bayesian analysis using Mplus: Technical implementation. Muthén & Muthén. https://www.statmodel.com/download/Bayes3.pdf

Bachman

J. G.

O’Malley

P. M.

(1984). Yea-saying, nay-saying, and going to extremes: Black-white differences in response styles. Public Opinion Quarterly, 48(2), 491-509. https://doi.org/10.1086/268845

Baumgartner

Steenkamp

J.-B. E. M.

(2001). Response styles in marketing research: A cross-national investigation. Journal of Marketing Research, 38(2), 143–156. https://doi.org/10.1509/jmkr.38.2.143.18840

Billiet

J. B.

Davidov

(2008). Testing the stability of an acquiescence style factor behind two interrelated substantive variables in a panel design. Sociological Methods & Research, 36(4), 542-562. https://doi.org/10.1177/0049124107313901

Böckenholt

(2012). Modeling multiple response processes in judgment and choice. Psychological Methods, 17(4), 665-678. https://doi.org/10.1037/a0028111

Böckenholt

(2017). Measuring response styles in Likert items. Psychological Methods, 22(1), 69-83. https://doi.org/10.1037/met0000106

10.

Bolt

Newton

J. R.

(2011). Multiscale measurement of extreme response style. Educational and Psychological Measurement, 71(5), 814-833.

11.

Bolt

D. M.

Johnson

T. R.

(2009). Addressing score bias and differential item functioning due to individual differences in response style. Applied Psychological Measurement, 33(5), 335-352. https://doi.org/10.1177/0146621608329891

12.

Bolt

D. M.

Kim

J. S.

(2014). Measurement and control of response styles using anchoring vignettes: A model-based approach. Psychological Methods, 19(4), 528-541. https://doi.org/10.1037/met0000016

13.

Cheung

G. W.

Rensvold

R. B.

(2000). Assessing extreme and acquiescence response sets in cross-cultural research using structural equations modeling. Journal of Cross-Cultural Psychology, 31(2), 187-212. doi:10.1177/0022022100031002003

14.

Cronbach

L. J.

(1946). Response sets and test validity. Educational and Psychological Measurement, 6(4), 475-494. https://doi.org/10.1177/001316444600600405

15.

Danner

Aichholzer

Rammstedt

(2015). Acquiescence in personality questionnaires: Relevance, domain specificity, and stability. Journal of Research in Personality, 57(August), 119-130. https://doi.org/10.1016/j.jrp.2015.05.004

16.

de Jong

Steenkamp

J.-B.

Fox

J.-P.

Baumgartner

(2008). Using item response theory to measure extreme response style in marketing research: A global investigation. Journal of Marketing Research, 45(1), 104-115. https://www.researchgate.net/publication/247837132_Using_Item_Response_Theory_to_Measure_Extreme_Response_Style_in_Marketing_Research_A_Global_Investigation

17.

Gelman

Carlin

J. B.

Stern

H. S.

Rubin

D. B.

(2004). Bayesian Data Analysis. Chapman and Hall/CRC.

18.

Gelman

Rubin

D. B.

(1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 457-472. http://www.stat.columbia.edu/~gelman/research/published/itsim.pdf

19.

Greenleaf

E. A.

(1992). Measuring extreme response style. Public Opinion Quarterly, 56(3), 328-351. https://doi.org/10.1086/269326

20.

Harris

(2009). The National Longitudinal Study of Adolescent to Adult Health (Add Health), Waves I & II, 1994–1996; Wave III, 2001–2002; Wave IV, 2007-2009 [Machine-readable data file and documentation]. Carolina Population Center, University of North Carolina at Chapel Hill.

21.

Harzing

(2006). Response styles in cross-national survey research: A 26-country study. International Journal of Cross Cultural Management, 6(2), 243–266. DOI:10.1177/1470595806066332

22.

Jeon

De Boeck

(2016). A generalized item response tree model for psychological assessments. Behavioral Research Methods, 48(3), 1070-1085. https://link.springer.com/article/10.3758/s13428-015-0631-y

23.

Kim

Camilli

(2014). An item response theory approach to longitudinal analysis with application to summer setback in preschool language/literacy. Large-scale Assessments in Education, 2, Article 1. https://largescaleassessmentsineducation.springeropen.com/articles/10.1186/2196-0739-2-1

24.

Khorramdel

von Davier

(2014). Measuring response styles across the big five: A multiscale extension of an approach using multinomial processing trees. Multivariate Behavioral Research, 49(2), 161–177. DOI:10.1080/00273171.2013.866536

25.

Leventhal

B. C.

Stone

C. A.

(2018). Bayesian analysis of multidimensional item response theory models: A discussion and illustration of three response style models. Measurement: Interdisciplinary Research and Perspectives, 16(2), 114-128. https://doi.org/10.1080/15366367.2018.1437306

26.

Leventhal

B. C.

Zigler

C. K.

(2021, June 11). A tree-based approach to identifying response styles with anchoring vignettes [Symposium]. Annual Meeting of the National Council on Measurement in Education (NCME).

27.

Liu

Conrad

F. G.

Lee

(2017). Comparing acquiescent and extreme response styles in face-to-face and web surveys. Quality & Quantity, 51(2), 941–958. https://doi.org/10.1007/s11135-016-0320-7

28.

Lunn

Jackson

Best

Thomas

Spiegelhalter

(2012). The BUGS book: A practical introduction to Bayesian analysis. Chapman & Hall.

29.

Martin

(1964). Acquiescence: Measurement and theory. British Journal of Social and Clinical Psychology, 3(3), 216-225. https://doi.org/10.1111/j.2044-8260.1964.tb00430.x

30.

Messick

(1968). Response sets. In Sills

D. F.

(Ed.), International encyclopedia of the social sciences (Vol. 13, pp. 492-496). Macmillan.

31.

Millsap

R. E.

(2010). Testing measurement invariance using item response theory in longitudinal data: An introduction. Child Development Perspectives, 4(1), 5-9. https://doi.org/10.1111/j.1750-8606.2009.00109.x

32.

Moors

(2012). The effect of response style bias on the measurement of transformational, transactional, and laissez-faire leadership. European Journal of Work and Organizational Psychology, 21(2), 271-298. https://doi.org/10.1080/1359432X.2010.550680

33.

Muthén

Asparouhov

(2012). Bayesian structural equation modeling: A more flexible representation of substantive theory. Psychological Methods, 17(3), 313-335. https://doi.org/10.1037/a0026802

34.

Myers

A. J.

Ames

A. J.

(2020, July 17th-20th). A noncompensatory and compensatory MIRT model for modeling response styles [Paper presentation]. Virtual International Meeting of the Psychometric Society (IMPS).

35.

Park

A. D.

(2019). Item response tree models to investigate acquiescence and extreme response styles in Likert-type rating scales. Educational and Psychological Measurement, 79(5), 911-930. https://doi.org/10.1177/0013164419829855

36.

Paulhus

D. L.

(1991). Measurement and control of response bias. In Measures of personality and social psychological attitudes (Vol. 1). San Diego, CA: Academic Press.

37.

Plieninger

(2021). Developing and applying IR-Tree models: Guidelines, caveats, and an extension to multiple groups. Organizational Research Methods, 24(3), 654-670. https://doi.org/10.1177/1094428120911096

38.

Plieninger

Heck

(2018). A new model for acquiescence at the interface of psychometrics and cognitive psychology. Multivariate Behavioral Research, 53(5), 633-654. https://doi.org/10.1080/00273171.2018.1469966

39.

Plieninger

Meiser

(2014). Validity of multiprocess IRT models for separating content and response styles. Educational and Psychological Measurement, 74(5), 875-899. https://doi.org/10.1177/0013164413514998

40.

Plummer

(2008). Penalized loss functions for Bayesian model comparison. Biostatistics, 9(3), 523-539. https://doi.org/10.1093/biostatistics/kxm049

41.

Ray

J. J.

(1983). Reviving the problem of acquiescent response bias. Journal of Social Psychology, 121(1), 81-96. https://doi.org/10.1080/00224545.1983.9924470

42.

Samejima

(1969). Estimation of a latent ability using a response pattern of graded scores. Psychometrika, 34(Suppl. 4), 1-97. https://doi.org/10.1002/j.2333-8504.1968.tb00153.x

43.

SAS Institute Inc. (2008). SAS/STAT® 9.2 user’s guide. Author. http://support.sas.com/documentation/cdl/en/statugmcmc/63125/PDF/default/statugmcmc.pdf

44.

Schuman

Presser

(1981). Questions and answers in attitude surveys. Academic Press.

45.

Spiegelhalter

Best

Carlin

Van der Linde

(2002). Bayesian measures of model complexity and fit (with discussion). Journal of the Royal Statistical Methodology: Series B. 64(Pt. 4), 583-639. https://rss.onlinelibrary.wiley.com/doi/pdf/10.1111/1467-9868.00353

46.

Spratto

E. M.

Leventhal

B. C.

Bandalos

D. L.

(2021). Seeing the forest and the trees: Comparison of two IRTree models to investigate the impact of full versus endpoint-only response option labeling. Educational and Psychological Measurement, 81(1), 39-60. https://doi.org/10.1177/0013164420918655

47.

Thissen-Roe

Thissen

(2013). A two-decision model for responses to Likert-type items. Journal of Educational and Behavioral Statistics, 38(5), 522-547. https://doi.org/10.3102/1076998613481500

48.

van Herk

Poortinga

Y. H.

Verhallen

T. M. M

. (2004). Response Styles in Rating Scales: Evidence of Method Bias in Data From Six EU Countries. Journal of Cross-Cultural Psychology, 35(3), 346–360. https://doi.org/10.1177/0022022104264126

49.

Van Rosmalen

Van Herk

Groenen

P. J. F

. (2010). Identifying response styles: A latent-lass bilinear multinomial logit model. Journal of Marketing Research, 47(1), 157–172. https://doi.org/10.1509/jmkr.47.1.157

50.

van Vaerenbergh

Thomas

T. D.

(2013). Response styles in survey research: A literature review of antecedents, consequences, and remedies. International Journal of Public Opinion Research, 25(2), 195-217. https://doi.org/10.1093/ijpor/eds021

51.

Weijters

Geuens

Schillewaert

(2010a). The individual consistency of acquiescence and extreme response style in self-report questionnaires. Applied Psychological Measurement, 34(2), 105-121. https://doi.org/10.1177/0146621609338593

52.

Weijters

Geuens

Schillewaert

(2010b). The stability of individual response styles. Psychological Methods, 15(1), 96-110. https://doi.org/10.1037/a0018721

53.

Wetzel

Carstensen

C. H.

Böhnke

J. R.

(2013). Consistency of extreme response style and non-extreme response style across traits. Journal of Research in Personality, 47(2), 178-189. https://doi.org/10.1016/j.jrp.2012.10.010

54.

Wetzel

Ludtke

Zettler

Böhnke

J. R.

(2016). The stability of extreme response style and acquiescence over 8 years. Assessment, 23(3), 279-291. https://doi.org/10.1177/1073191115583714