Measuring Response Style Stability Across Constructs With Item Response Trees

Abstract

Individual response style behaviors, unrelated to the latent trait of interest, may influence responses to ordinal survey items. Response style can introduce bias in the total score with respect to the trait of interest, threatening valid interpretation of scores. Despite claims of response style stability across scales, there has been little research into stability across multiple scales from the beneficial perspective of item response trees. This study examines an extension of the IRTree methodology to include mixed item formats, providing an empirical example of responses to three scales measuring perceptions of social media, climate change, and medical marijuana use. Results show extreme and midpoint response styles were not stable across scales within a single administration and 5-point Likert-type items elicited higher levels of extreme response style than the 4-point items. Latent trait of interest estimation varied, particularly at the lower end of the score distribution, across response style models, demonstrating as appropriate response style model is important for adequate trait estimation using Bayesian Markov chain Monte Carlo estimation.

Keywords

response style item response tree MCMC Markov chain Monte Carlo

Self-reported measures are common in education, the social sciences, and psychology, with instruments often using Likert-type items to measure attitudes and personalities of respondents (Böckenholt, 2017). Self-report Likert items vary in their format; some have midpoint responses (i.e., odd numbers of response categories) and some do not (i.e., even numbers of response categories). Despite their prevalence of use, Likert items and instrument scores using Likert items are prone to response style bias. A response style can be thought of as the tendency to respond to items in some systematic way other than the trait of interest (TOI) or as an interaction of personal disposition and situational factors (Plieninger & Meiser, 2014).

Bolt and Johnson (2009) have argued that response style may reflect a respondent’s attempt to reduce the cognitive demand of distinguishing between differing levels of agreement or disagreement. For example, respondents may have differing opinions over what constitutes strong agreement—it is possible that they endorse a construct equally, but only differ in their interpretation of the response anchors. Because response option anchors are not on a continuum, respondents can also vary in their interpretation of the distance between response categories. This further obfuscates accurate interpretations of survey results as differing scale scores may be due, in part, to response style bias.

There are varying kinds of response styles that manifest depending on item format or the personality and background of respondents (Clarke, 2001; Harzing, 2006; Paulhus, 1991). Some of the response styles commonly cited in the literature are acquiescence response style (ARS), disacquiescence response style (DRS), extreme response style (ERS), and midpoint response style (MRS). ARS and DRS are characterized by the propensity to agree or disagree, respectively, without regard to the construct being measured. MRS is the tendency of respondents to pick the midpoint or neutral response on a scale with an odd number of categories while ERS is characterized by respondents endorsing the extreme options at the endpoints of scales. These response styles are considered content-independent, trait-like constructs that are stable over time and across subscales during a single test administration (Austin et al., 2006; Baumgartner & Steenkamp, 2001; Jin & Wang, 2014; Wetzel et al., 2013).

The presence of response styles in self-reported measures is a concern, as the validity of instruments using these measures is threatened by the unaccounted for response styles. Respondents’ preference for extreme responses can bias measurements of traits of interest and can confound interpretations of scores, thus interfering with the psychometric study of an instrument. For example, items may appear to function differently in their measurement of the TOI when respondents vary only in response style (Bolt & Johnson, 2009; Park & Wu, 2019). The presence of response styles presents a host of other issues. ERS can lead to spurious correlations between constructs as respondents may have inflated or deflated scores in unrelated constructs (Jeon & De Boeck, 2019; Park & Wu, 2019; Paulhus, 1991), which may lead to overestimation or underestimation of the construct of interest and score inflation (Park & Wu, 2019) or manifest as an issue is diagnostic surveys where a respondent’s score plays a role in a clinical diagnoses or access to services. Furthermore, response styles threaten dimensionality, construct validity, predictive validity, and the reliability of self-reported measures (Baumgartner & Steenkamp, 2001; Cheung & Rensvold, 2000; De Jong et al., 2008; van Herk et al., 2004; van Rosmalen et al., 2010). This can result in differential item functioning, a lack of measurement invariance, and present negative effects on parameter recovery (Bolt & Johnson, 2009; Liu et al., 2017). Going further, response styles threaten comparisons of diverse groups and international or cross-cultural assessment. There are many documented correlates (i.e., age, individual traits, or cultural background) to different types of response styles that often vary by culture or region, thus making accurate comparisons difficult (Harzing, 2006; Khorramdel & von Davier, 2014).

Measuring Response Style

Researchers have developed several methods to measure response style and response style bias. More simplistic methods entail counting the number of extreme or midpoint responses (e.g., Harzing, 2006). However, these methods do not account for the influence of the target trait on responses (Bolt & Newton, 2011). For example, a person providing many very low scores on a set of items could be an extreme responder, tending to select the most extreme option on the low end of the scale, but could also be very low on the substantive TOI. The classical approach does not address the confounding of the substantive TOI and, in this example, ERS.

Providing a promising approach to address issues with the classical approach, Wetzel et al. (2016) use a trait-state-occasion model for ERS and ARS, building on Weijters, Geuens, and Schillewaert (2010b). They measure the stability of response styles as the amount of variance in the state response style factors that is explained by the trait response style factors. Wetzel et al. (2016) also use the latent-state model, which measures stability of response styles as the correlations between time-specific response style factors across waves. However, indicators of response style were based on sum scores for a random sample of items and do not utilize item-level responses.

Item-level approaches utilize various versions of item response models. Jin and Wang (2013) provide a thorough overview of item response theory (IRT) models for measuring ERS. They describe how the Partial Credit Model (PCM; Masters, 1982) can be extended into a latent class PCM, as done in Moors (2008) and van Rosmalen et al. (2010), identifying latent classes of extreme and non-extreme responders. Others (e.g., Bolt & Johnson, 2009; Bolt & Newton, 2011; Johnson & Bolt, 2010) used multidimensional IRT to simultaneously model one or two substantive traits of interest and one trait corresponding to ERS using a multidimensional nominal response model. This approach ignores the ordinal nature of Likert-type items. In a method that does account for ordinal items, Johnson (2003) uses random thresholds, allowing for individual respondents to have their own thresholds in a probit model for ordinal responses. A respondent’s distance between adjacent thresholds is used as an indicator of ERS propensity, with larger distances indicating use of ERS more frequently. Johnson’s (2003) approach does not model response style as a separate latent trait, one drawback of the approach.

A recent class of approaches use multiprocess, multidimensional item response trees (IRTrees) to model hypothesized response subprocesses (e.g., Böckenholt, 2012; Böckenholt & Meiser, 2017; De Boeck & Partchev, 2012; Khorramdel & von Davier, 2014). While the modeling has been developed in the last decade, early efforts (Hart, 1923) theorized respondents used a decision-making series of steps, or subprocesses, when responding and developed a questionnaire in order to assess this decision based approach. These subprocess models combine processing trees of binary decision nodes with IRT parameterizations of node probabilities (Meiser et al., 2019). The decision nodes represent response subprocesses and the continuous latent variables related to those processes (i.e., target traits and response styles; Khorramdel et al., 2019). In practice, these models are used to disentangle ordinal item responses into binary pseudo-items based on the decision tree specification (Khorramdel & von Davier, 2014) and the resulting pseudo-items are used to measure response styles and target traits. Thus, the IRTree addresses many of the shortcomings of previous approaches in that they use item-level responses, preserve the ordinal nature of items, and treat response styles separate from the latent TOI to avoid confounding.

Consider a five-category Likert item, with response options “strongly disagree (1),”“disagree (2),”“neither agree nor disagree (3),”“agree (4),” and “strongly agree (5).” This item can be decomposed into three subprocesses (i.e., three decision stages), with each subprocess driven by a separate latent trait. The first subprocess for items on these scales represents whether a respondent has an opinion on the construct or not. That is, the respondent either selects “neither agree nor disagree (3),” the midpoint, or one of the other options, either in agreement with the item’s content or not. Midpoint responses resulted in MRS pseudo-items coded as 1 and the remaining TOI and ERS pseudo-items coded as missing because a respondent cannot simultaneously have an opinion and not have an opinion on the item’s content.

The second subprocess represents a respondent’s decision of whether they are high or low on the target trait (i.e., choosing between agreement or disagreement). A respondent choosing “agree (4)” or “strongly agree (5)” has the TOI pseudo-items coded as 1 with responses of “disagree (2)” or “strongly disagree (1)” coded as 0 to indicate endorsement (or not) of the TOI.

The third subprocess constitutes respondents’ decision to choose the extreme response option or not (e.g., choosing between “strongly agree (5)” and “agree (4)” or “strongly disagree (2)” and “disagree (2)”). Respondents selecting “strongly disagree (1)” or “strongly agree (5)” had ERS pseudo-items coded as 1 while respondents selecting “disagree (2)” or “agree (4)” had ERS pseudo-items coded as 0. Hence, a respondent selecting “agree” on an item has pseudo-item responses [010] reflecting failure to endorse the midpoint response, agreement with the target trait, and failure to endorse the extreme response option, respectively. A respondent selecting “neither agree nor disagree” would have pseudo-item responses [1-missing-missing]. Figure 1 and Table 2 present this subprocess and pseudo-item coding.

Figure 1.

Three-process IRTree.

For the three-decision IRTree (i.e., Figure 1), probabilities of each observed response option to item j are computed for respondent i by tracing the path from the MRS-driven subprocess through the second (TOI) and third (ERS) subprocesses. The respondent either endorses the middle category, “Neither Agree nor Disagree,” with probability $m_{ij}$ or with probability $1 - m_{ij}$ . If the respondent does not endorse the middle category, the TOI is endorsed with probability $t_{ij}$ , or not endorsed with probability $1 - t_{ij}$ . Finally, regardless of their agreement or disagreement in the second stage, the extreme response is endorsed with probability $e_{ij}$ , or not endorsed with probability $1 - e_{ij}$ (Böckenholt, 2012; Plieninger & Heck, 2018). More formally, the observed response Y_ij is modeled as the product of individual probabilities of each pseudo-item response. For example, the observed response “disagree” (i.e., Y_ij = 2) is given by

\Pr (Y_{ij} = 2 | θ_{MRS}, θ_{TOI}, θ_{ERS}, ω_{j}) = (1 - m_{ij}) * (1 - t_{ij}) * (1 - e_{ij}),

(1)

where $ω_{j}$ represents the item parameters. This represents the failure to endorse the midpoint, TOI, and ERS pseudo-items of each subprocess.

The probability of endorsing each pseudo-item is parameterized using an IRT two-parameter logistic (2PL) model via:

P (Y_{ijk} = 1 | a_{ik}, b_{ik}, θ_{jk}) = \frac{e^{a_{ik} (θ_{jk} - b_{ik})}}{1 + e^{a_{ik} (θ_{jk} - b_{ik})}},

(2)

where $Y_{ijk}$ is the response on binary pseudo-item response to item i by person j in decision stage k; $a_{ik}$ is the pseudo-item discrimination parameter in decision stage k; $b_{ik}$ is the pseudo-item difficulty parameter; and $θ_{jk}$ is the respondent latent trait parameter for the trait in the kth decision stage (i.e., k = ERS, MRS, or TOI). The loading parameter, $a_{jk}$ , represents how well pseudo-item j discriminates between those who are high on the kth latent trait ( $θ_{ik}$ ) and those who are low on the kth latent trait. The difficulty parameter, $b_{jk}$ , represents how difficult it is for respondents to endorse pseudo-item j of the kth subprocess. Böckenholt (2012) indicates other IRT models can be used to model each subprocess. For example, Leventhal (2017) presents a six-category IRTree, in which the TOI subprocess is modeled using a 2PL and the ERS subprocess is modeled using a modified Graded Response Model (Samejima, 1969).

Stability

Measurement of response styles in this way are important because unaccounted for response styles are a threat to instrument validity and the validity of comparisons made across groups. However, there has been little research into response style stability from the beneficial perspective of IRTrees. Stability in individuals’ response styles can be considered across time, across instruments, and across items within the same instrument. Across time, previous research indicates response styles are stable over both short and long periods. Stability has been found over a 1-year period (ARS, ERS, DRS, and MRS: Weijters, Geuens, & Schillewaert, 2010b), a 2-year period (ARS and ERS: Bachman & O’Malley, 1984), a 4-year period (ARS: Billiet & Davidov, 2008), and an 8-year period (ARS and ERS: Wetzel et al., 2016). Across-time stability suggests that the perceived stability of personality and attitude constructs may be ascribed to response style stability. Besides the methods just referenced, Ames and Leventhal (2019) present a longitudinal IRTree for measuring changes in both TOI and response style across multiple time points. Within the same instrument, across item response style stability implies that individuals systematically use the response scale in a similar manner across items (e.g., ARS and ERS: Weijters, Geuens, & Schillewaert, 2010a; ARS: Danner et al., 2015).

Across-scale stability implies respondents systematically use the response scale in a similar manner across scales (e.g., ERS: Aichholzer, 2013;Bolt & Newton, 2011; Wetzel et al., 2013; ARS & ERS: Weijters, Geuens, & Schillewaert, 2010a). However, many previous across-scale studies used related constructs. For example, Bolt and Newton (2011) examined two science subscales from the Program for International Student Assessment (PISA; OCED, 2006), measuring a respondent’s “Enjoyment of Science” (ENJ) and “Value of Science” (VAL), each item a forced-choice, 4-point Likert-style item. The ENJ and VAL scales had a positive correlation of .580. Park and Wu (2019) use only Rosenberg’s Self Esteem Scale to estimate ERS, ARS, and DRS. Another two studies measure ERS and MRS using the Personal Need for Structure questionnaire containing two substantive dimensions (Böckenholt & Meiser, 2017; Meiser et al., 2019). In the latter study, the estimated correlation between the two substantive dimensions was large (r = .75). Jeon and De Boeck (2019) measure ERS in three multiple subscale assessment data samples using the Big Five personality assessment (Goldberg, 1992), the DISC personality assessment (Rosenberg & Silvert, 2013), and the Humor Styles Questionnaire (Martin et al., 2003). In Jeon and De Boeck’s analyses, the magnitudes of correlations between subscales averaged at .22 for the Big Five assessment, .39 for the DISC assessment, and .25 for the Humor Styles Questionnaire. Khorramdel and von Davier’s (2014) analysis of Big Five personality traits from the International Personality Item Pool also showed a moderate relationship between subscales, with magnitudes of correlations between the subscales averaging at .33. In another study, Khorramdel et al. (2019) used the PIAAC with four subscales measuring work-related tasks—the average magnitude of correlations between subscales was .64.

Examining across-scale studies from unrelated constructs, and different item types, will improve the field’s understanding of response style stability and how to accurately account for response styles when simultaneously measured with substantive traits. Bolt and Newton (2011) found improved measurement of ERS when using simultaneous analysis of multiple scales, but that improved measurement of TOI latent traits was smaller. This pattern may not hold for unrelated constructs, or those with lower intertrait correlations. Baumgartner and Steenkamp (2001) recommend heterogeneous items to measure response style—either different content than the substantive TOI or to include both positive and negatively worded items. The different-content approach has not been evaluated using IRTrees; the result of this research may help guide future measurement of response styles and designing questionnaires to account for response styles such as ERS and MRS. To address the gap in the current literature around response styles, this study addresses the following research questions using an empirical data set:

Research Question 1: Using IRTrees to simultaneously measure response style and the substantive traits of interest, is there evidence that ERS and MRS estimates are stable across scales within a single survey administration?

Research Question 2: Is there evidence that ERS estimates are stable across varying item types?

Research Question 3: What is the impact on TOI estimation of assuming stable ERS and MRS when the structure of the response style latent trait suggests otherwise?

Method

As detailed, previous studies of response styles used either a single scale, or scales with correlated subscales to estimate response style. This study estimates response styles using three attitudinal subscales measuring opinions on medicinal cannabis, climate change, and social media. The current study uses multiple IRTree models to determine the stability of response styles across responses to three subscales. This novel approach to the estimation of response styles will lend evidence in support of, or against, the hypothesized content independence and stability of response styles. Validating a model for disentangling response styles from substantive traits of interest will help future researchers to draw valid conclusions from surveys using Likert scales.

Instrumentation

The medicinal cannabis subscale intends to measure an individual’s perception on the appropriateness of using cannabis for medicinal purposes and contains 11, 5-point Likert items. The climate change subscale intends to measure an individual’s perception on the change in global or regional climate patterns and contains 10, 5-point Likert items. For the medicinal cannabis and climate change subscales, higher scores indicate greater endorsement of medicinal cannabis and stronger belief in human impact on climate change and its negative consequences. The Likert social media scale contains 13, 4-point Likert items intended to measure the influence of engagement in social media on one’s perceptions of their image portrayed in the social environment and self-image. Higher scores on the social media subscale indicate less authenticity in one’s online persona. The 5-point Likert scales contained items with responses ranging from “strongly disagree” to “strongly agree.” The 4-point Likert scale contained items with responses ranging from “strongly disagree” to “strongly agree.” Four items on the climate change subscale and five items on the social media subscale that were negatively worded were recoded before the subsequent analysis. The relationships among study scales are presented in Table 1, with reliability coefficients on the diagonal of the matrix. Reliability of each subscale was assessed using Cronbach’s (1951)coefficient alpha. Results generally supported the reliability of each subscale, with each of the medicinal cannabis and climate change subscales having coefficient alpha≥ 0.90 (Nunnally & Bernstein, 1994) and the social media subscale had a coefficient alpha value of 0.80.

Table 1.

Relationship Among Study Scales.

	N items	Likert scale response options	Midpt.	Ext.	Medicinal cannabis	Climate change	Social media
Medicinal cannabis	11	5	0.10	0.46	(0.96)
Climate change	10	5	0.22	0.25	0.46	(0.92)
Social media	13	4	—	0.08	0.01	0.13	(0.80)

Note. Reliability coefficients are enclosed by parenthesis. Midpt. represents the proportion of midpoint responses for the scale, and Ext. represents the proportion of extreme responses.

In the current study, the scales are from different constructs and two uncorrelated scales (i.e., medicinal cannabis and social media are uncorrelated; r = .01). Unrelated, uncorrelated scales have yet to be examined in the study of response style stability. Table 1 presents the relationship among raw scores of the scales, as well as other scale information. If response styles are truly stable traits, then individuals with high levels of response tendencies on one scale should also show high levels of the same response tendency on another scale. That is, we would expect moderate to large positive correlations among response style traits across scales.

Sample

The data used in the study come from responses collected in fall 2018 (n = 1199) collected via the online survey platform Qualtrics. Of respondents that gave their gender, 27% were male, 72% were female, and the remainder specified another gender identity. The majority of respondents completed some form of higher education, with 41% holding a master’s degree or higher, 35% holding a 2-year or 4-year degree, and only 23% reporting their level of education as some college or less. The median age of respondents was 33.

IRTrees and Binary Pseudo-Items

Responses to the medicinal cannabis and climate change scales were recoded into binary pseudo-items representing ERS, TOI, and MRS, depicted in Figure 1 and Table 2. Responses to the social media subscale, with no midpoint for the items and four response options, were coded into pseudo-items representing TOI and ERS, without the MRS subprocess. Hence, a respondent responding “strongly disagree” on a social media item would have pseudo-item responses [01] indicating disagreement with the social media TOI and selection of the extreme response option, respectively. This 4-point Likert item is presented in Figure 2.

Table 2.

Pseudo-Item Coding Matrix for 5-Point Likert Items.

Original scale (numeric response)	$Y_{MRS}^{*}$	$Y_{TOI}^{*}$	$Y_{ERS}^{*}$
Strongly agree (5)	0	1	1
Agree (4)	0	1	0
Neither agree nor disagree (3)	1	—	—
Disagree (2)	0	0	0
Strongly disagree (1)	0	0	1

Note. $Y_{k}^{*}$ is the pseudo-item for the kth decision stage, where k = MRS (midpoint response style), ERS (extreme response style), or TOI (trait of interest).

Figure 2.

Two-process IRTree.

Modeling Approach

To answer the three research questions, a series of models were fit to the data. Model 1, the “Full” model, estimates all traits simultaneously. That is, the model has two MRS traits, one per each 5-point scale; three ERS traits, one per each scale; and three TOI traits, one per each scale, for a total of 8 latent traits (see Figure 3). Each latent trait in the Full model is informed only by items in the specific scale. For example, while the MRS trait for climate change and medicinal cannabis are estimated simultaneously in the Full model, the MRS pseudo-items from the climate change scale only inform the climate change MRS trait. Those pseudo-items do not inform the medicinal cannabis MRS trait.

Figure 3.

Full model.

Model 2, the “Single Response Style” model, estimates one trait for MRS and one trait for ERS to represent the case of response style stability. Specifically, there is one MRS trait, estimated across all 5-point scales. Specifically, all MRS pseudo-items from the climate change and medicinal cannabis inform the single MRS trait. There is one ERS trait, estimated across all scales, and all ERS pseudo-items inform the single ERS trait. Finally, there are three TOI traits, one per each scale, for a total of 5 latent traits (see Figure 4).

Figure 4.

Single response style model.

Models 3 to 5 fit an IRTree to each scale separately. That is, a three-process model like in Figure 1 was fit separately to the medical cannabis scale and to the climate change scale. Models 3 and 4 have three latent traits each (MRS, TOI, and ERS). A two-process model like, in Figure 2, was fit to the social media scale. Model 5 has two latent traits (TOI, ERS).

Stable across-scale response styles (research question 1) will be evidenced by a consistent ranking of respondents’ estimated response styles across Model 1 and Model 2. Furthermore, stable response styles will favor Model 2. Stable across-item response styles (Research Question 2) will be evidenced by a consistent ranking of respondents’ estimated ERS across Model 1. The impact of model specification on TOI estimation (Research Question 3) will be examined by comparing TOI estimates across all models.

Estimation

Model estimation occurred in Mplus using Markov chain Monte Carlo (MCMC) estimation with the Gibbs sampler (Muthén & Muthén, 1998-2017). Two chains were estimated with the first 25,000 iterations from each chain discarded as burn-ins and the posterior distribution consisted of the last 25,000 iterations from each chain. Weakly informative priors were used for the item parameters: $a_{jk}$ and $b_{jk} ~ N (0, σ^{2} = 2) .$ The latent trait vector was assumed multivariate standard normal, and an inverse-Wishart prior was placed on the covariance structure: $Σ ~ IW (0, 4) .$ There is, generally, a lack of research into the appropriate priors for IRTree models. The priors were chosen from findings in other IRTree parameter recovery studies. For instance, Ames and Leventhal (2021) found that matching priors had consistently higher coverage rates and lower mean absolute bias than the non-informative default priors from Mplus. The matching priors in their study matched the generating prior distribution. With empirical data, the “true” data-generating distribution is unknown. Therefore, this study used somewhat more informative priors than the default Mplus priors.

Convergence was assessed via visualization of trace plots and evaluation of the potential scale reduction factor (PSRF) convergence criteria (Gelman & Rubin, 1992). The minimum PSRF convergence criteria was set to 1.02. From each of the five models, the latent trait expected a priori (EAP) estimate was saved. Marginal DIC used to compare full, single response style, and individual models.

Several previous studies provided guidelines for parameter recovery of IRTrees. For example, Ames and Myers (2020) found mean bias of near zero for item difficulty parameters in IRTrees at the lowest sample size of their simulation (N = 1,000). Wang and Nydick (2015) evaluated a similar noncompensatory multidimensional IRT model, suggesting sample sizes of 1,000 are required for adequate parameter recovery without missing data (Wang & Nydick, 2015). As such, the sample size of 1,199 was deemed adequate.

Results

Model Comparison and Evaluation

Marginal DIC favored the “full” model (DIC = 19910.92) over the “single response style” model (DIC = 25344.16), indicating that the model with separate MRS and ERS per scale provided a better fit to the data than a model that estimated a single MRS and single ERS trait across all scales. This provides the first piece of evidence that scale-specific response styles provide a better fit to the data than a single estimate of response style across models.

Further model evaluation was performed using Posterior Predictive Model Checking (PPMC). PPMC methods are useful in evaluating absolute model fit for IRT models (see Sinharay, 2005, 2006, for an exploration and description of PPMC for IRT). PPMC is based on the simple, intuitive principle that the data generated from the posterior distribution(s) should resemble the observed data if the model provides adequate fit for the data. The posterior predictive distribution (PPD) for replicated data is a distribution of future observable data conditioned on the observed values (Rubin, 1984). That is, PPD is the distribution of simulated, or “replicated,” data that would arise from the model and parameters estimated from observed data. Computing the replicated samples is accomplished by randomly drawing values of the parameter(s) from the posterior distribution, which are then used to simulate the data using the model of interest. This process is repeated many times—for this study, 1,000 times. A discrepancy statistic is computed for each of the 1,000 replicated data sets, which is compared to the same discrepancy measure computed from the observed data. A posterior predictive probability (PPP) value is the proportion of replicated discrepancy statistics larger than the same discrepancy statistic calculated under the observed data. PPP values near 0.5 indicate that no systematic differences exist between replicated data and observed data, of the discrepancy statistic. Thus, PPP values near 0.5 indicate model fit and PPP values near 0 or 1 indicate systematic differences between replicated data and observed data.

Leventhal (2017) describes four approaches to PPMC for IRTrees. Because the purpose of this study is on person-fit of the ERS and MRS traits, Leventhal’s fourth approach, exploration of the proportion of individuals who selected extreme response options and midpoint response options as discrepancy measures, is used in this study. Specifically, 1,000 replicated data sets for the “full” model (Model 1) and the “single response style” model (Model 2) were generated. Across each respondent, the proportions of extreme and midpoint responses were computed for the 1,000 replicated data sets and the observed data, resulting in four PPP values per respondent: one PPP value per model, per trait.

PPP values were averaged across all respondents with the results presented in Figures 5 and 6 for ERS and MRS, respectively. Respondents were “binned” based on their average proportion of extreme responses and midpoint responses in the observed data. For example, a respondent who provided 11 extreme responses across the 34 items of the three scales provided a proportion of .32 extreme responses. They were included in the bin .3. Figure 5 shows that for Model 1, the full model, average PPP values are near .5 for most bins of extreme responders. For respondents with a low level of extreme responses (i.e., extreme proportions of .2 and below), the single response style model (Model 2) does a poor job of modeling their extreme responses, with average PPP values above the .95 level. The high average PPP values indicate Model 2 tends to overestimate the extreme responses in the replicated data for those providing an initially low level of extreme responses. As the proportion of extreme responses increases, the average PPP values decrease for Model 2, approaching the average PPP value for Model 1 for high extreme responders.

Figure 5.

Average PPP values: Proportion ERS as discrepancy value.

Figure 6.

Average PPP values: Proportion MRS as discrepancy value.

Figure 6 shows that for Model 1, the full model, average PPP values are between .2 and .5 for most bins of midpoint responders. For respondents with a low level of midpoint responses (i.e., midpoint proportions of .1 and below), the single response style model (Model 2) does a poor job of modeling their midpoint responses, with average PPP values above the .95 level. The high average PPP values indicate Model 2 tends to overestimate the midpoint responses in the replicated data. As the proportion of midpoint responses increases, the average PPP values decrease.

Midpoint Response Style

Table 3 provides the Pearson correlations for EAP estimates of MRS for Models 1 to 4 in the upper triangle. The social media scale is a 4-point scale that does not include a midpoint. Thus, no midpoint pseudo-item is created and no MRS trait is estimated for the social media scale. In addition, EAP estimates were rank-ordered into 5 groups and Table 3 also includes Spearman rank-order correlations for Models 1 to 4 in the lower triangle. All correlations were positive and statistically significant. However, the magnitude of correlations varied considerably.

Table 3.

Pearson and Spearman Rank-Order Correlations for MRS Trait Among Models.

	MC full (Model 1)	CC full (Model 1)	Single RS (Model 2)	MC single (Model 3)	CC single (Model 4)
MC full	1	0.424	0.691	0.937	0.394
CC full	0.409	1	0.893	0.355	0.983
Single RS	0.645	0.880	1	0.639	0.894
MC single	0.931	0.354	0.590	1	0.331
CC single	0.399	0.963	0.884	0.346	1

Note. MC is medicinal cannabis; CC is climate change; Full is the “full” model (Model 1); Single RS is Model 2, the “Single Response Style” model; and Alone is Model 3 or 4 in which a single scale is fit to the IRTree.

The largest Pearson correlation was between the MRS EAP estimates for climate change in the full model (Model 1) and the single-scale climate change model (Model 4; r = .983), representing a large effect size (Cohen, 1992). The smallest correlation was between the single-scale medicinal cannabis (Model 3) and single-scale climate change (Model 4; r = .331), representing a small to moderate effect size (Cohen, 1992). When comparing the MRS EAP estimates from the full model (Model 1) to the single response style model (Model 2) the correlations were large and positive (r = .691, .893 for the medicinal cannabis and climate change scales, respectively), but not as strong as the correlations from the full model (Model 1) with the single-scale models (r = .937, .983 for the medicinal cannabis and climate change scales, respectively). Taken together, evidence implies a positive relationship between MRS estimates across traits, but that the single response style model (Model 2) does not measure an individual respondent’s level of the MRS trait as well as when the trait is estimated separately by scale.

Spearman rank-order correlations are found in the lower triangle of Table 3, revealing correlations similar in magnitude the Pearson correlations. Rankings are especially revealing in visualizing the differences among the model MRS estimates, presented in Figure 7. Ranks for the single response style model (Model 2) are given different shapes and shadings. For example, the open triangle is the top-ranked group from Model 2. For the most part, rankings are preserved across models. However, rankings were inconsistent between the full model and single response style model (see Figure 7 and Table 4) for both medicinal cannabis and climate change. For example, only 57.32% of individuals ranked in the lowest group for MRS from the medicinal cannabis in Model 1 were ranked in the lowest group in Model 2. Of those ranked in the second-lowest group for MRS from the medicinal cannabis in Model 1, only 28.75% were ranked in the second-lowest group from the single response scale (Model 2). Rankings were somewhat more consistent for climate change MRS estimates (e.g., 74.9% of individuals ranked in the lowest group for MRS from the climate change in Model 1 were ranked in the lowest group in Model 2). These findings illustrate that the single response style model does not adequately represent MRS traits across multiple scales, providing further evidence for Research Question 1 that response styles are not stable across unrelated scales.

Figure 7.

Midpoint response style relationship across models.

Table 4.

Crosstabs of MRS Ranks (Percentage).

Single RS (Model 2)	MC full (Model 1)					CC full (Model 1)
	1	2	3	4	5	1	2	3	4	5
1	57.32	31.67	10.42	0.42	0	74.9	23.65	1.26	0	0
2	22.59	28.75	30.83	13.75	4.17	19.25	51.87	28.03	0.42	0.42
3	12.55	20.83	21.25	31.67	13.75	4.18	18.67	55.23	21.25	0.83
4	4.6	12.08	24.17	29.58	29.58	1.26	5.39	12.97	62.92	17.5
5	2.93	6.67	13.33	24.58	52.5	0.42	0.41	2.51	15.42	81.25

Extreme Response Style

Table 5 provides the Pearson correlations for EAP estimates of ERS for Models 1 to 5 in the upper triangle. In addition, EAP estimates were rank-ordered into 5 groups, and Table 5 also includes Spearman rank-order correlations for Models 1 to 5 in the lower triangle. Similar to measurement of MRS, all correlations were positive and statistically significant. The largest Pearson correlation was between the ERS EAP estimates for social media in the full model (Model 1) and the single-scale social media model (Model 5; r = .966), representing a large effect size (Cohen, 1992). The smallest correlation was between the single-scale medicinal cannabis (Model 3) and single-scale social media (Model 4; r = .213), representing a small to moderate effect size (Cohen, 1992). When comparing the ERS EAP estimates from the full model (Model 1) to the single response style model (Model 2) the correlations were large and positive (r = .909, .721, .491 for the medicinal cannabis, climate change, and social media scales, respectively), but not as strong as the correlations from the full model (Model 1) with the single-scale models (Models 3-5; r = .939, .961, .966 for the medicinal cannabis, climate change, and social media scales, respectively). Similar to measurement of MRS, evidence implies a positive relationship between ERS measures across traits, but that the single response style model (Model 2) does not measure an individual respondent’s level of the ERS trait as well as when the trait is estimated separately by scale.

Table 5.

Pearson and Spearman Rank-Order Correlations for ERS Trait Among Models.

	MC full	CC full	SM full	Single RS	MC alone	CC alone	SM alone
MC full	1	0.575	0.381	0.909	0.939	0.483	0.281
CC full	0.521	1	0.439	0.721	0.494	0.961	0.344
SM full	0.334	0.401	1	0.491	0.307	0.330	0.966
Single RS	0.939	0.639	0.386	1	0.854	0.646	0.397
MC alone	0.950	0.468	0.291	0.915	1	0.410	0.213
CC alone	0.454	0.947	0.309	0.579	0.398	1	0.243
SM alone	0.261	0.323	0.950	0.304	0.222	0.234	1

Note. MC is medicinal cannabis; CC is climate change; Full is the “full” model (Model 1); Single RS is Model 2, the “Single Response Style” model; and Alone represents Models 3 to 5 in which a single scale is fit to the IRTree.

Spearman rank-order correlations are found in the lower triangle of Table 5, revealing correlations similar in magnitude the Pearson correlations. Rankings were inconsistent between the full model and single response style model (see Figure 8 and Table 6), especially for the social media scale estimates. For the medicinal cannabis scale, with a large proportion of extreme responses (Table 1; 0.46), the single ERS trait adequately captures the rankings. Not so for the social media scale, which had only 0.08 proportion of extreme responses. For example, only 45.19% of individuals ranked in the lowest group for ERS from the social medial scale in Model 1 were ranked in the lowest group in Model 2. Of those ranked in the second-lowest group for ERS from the social medial scale in Model 1, only 20.33% were ranked in the second-lowest group from the single response scale (Model 2). Rankings were more consistent for medicinal cannabis ERS estimates (e.g., 85.77% of individuals ranked in the lowest group for ERS from the medicinal cannabis in Model 1 were ranked in the lowest group in Model 2). These findings illustrate that the single response style model does not adequately represent ERS traits across multiple scales, again providing further evidence for Research Question 1 and that response styles are not stable across unrelated scales and different item types.

Figure 8.

Extreme response style relationship across models.

Table 6.

Crosstabs of ERS Ranks (Percentage).

	Single RS (Model 2)
		1	2	3	4	5
MC full (Model 1)	1	85.77	13.28	0	0.42	0.42
	2	14.23	74.69	10.46	0	0.42
	3	0	12.03	82.85	5.83	0
	4	0	0	6.69	70.42	22.5
	5	0	0	0	23.33	76.67
CC full (Model 1)	1	59.41	20.75	10.88	8.33	0.42
	2	27.2	26.14	21.34	23.33	2.08
	3	12.55	26.97	24.27	30.83	5.42
	4	0.84	19.5	25.94	23.75	30
	5	0	6.64	17.57	13.75	62.08
SM full (Model 1)	1	45.19	22.41	14.64	15	2.5
	2	27.2	20.33	17.57	20.42	14.58
	3	14.64	23.65	20.92	22.5	18.33
	4	9.62	17.01	21.34	20.42	31.67
	5	3.35	16.6	25.52	21.67	32.92

Trait of Interest

While stability of response style is the focus of the study, estimation of response style will impact TOI estimation in the multidimensional model. Figure 9 illustrates a scatter plot of the EAP estimates across all 5 models. Similar to response styles, the same-trait, different model estimates are strong and positive. However, for medicinal cannabis trait estimation, the model for response style also seems to impact estimation of TOI. For example, comparing the medicinal cannabis EAP estimates from the Full model (Model 1) to the single response style model (Model 2), particularly at the lower end of the scale, showed a distinct fan shape at the lower end of the scale. The same shape is seen for the climate change scale, but not for the social media scale. In fact, the estimation of the social media scale was remarkably stable across models, as indicated by the very tight scatterplot between model estimates.

Figure 9.

Trait of interest relationship across models.

Discussion

Using an empirical example, exploring research question 1 showed extreme and midpoint response styles were not stable across scales within a single testing administration, challenging the notion that these systematic response tendencies such as ERS and MRS are stable across scales (e.g., for ERS: Aichholzer, 2013;Bolt & Newton, 2011; Wetzel et al., 2013). The single response style model is not appropriate for response style measurement across multiple, distinct scales. In support of this notion, marginal DIC favored the “full” model over the “single response style” model, indicating that the model with separate MRS and ERS per scale provided a better fit to the data than a model that estimated a single MRS and single ERS trait across all scales. This was supported by the PPMC approach, which indicated that proportions of extreme and midpoint responses were not adequately captured using the “single response style model,” tending to overestimate extreme and midpoint responses in the replicated data for respondents with a low level of observed extreme and midpoint responses. On average these proportions tended to be captured using the “full” model.

There were positive correlations in measures of response style across models and traits. These finds agree with previous attempts to validate the use of IRTrees. Pleininger and Meiser (2014) examined responses to nine, 7-point ordinal items measuring self-confidence items that were recoded into dichotomous pseudo-items, reflecting MRS, a content-related process (i.e., TOI), and ERS. To evaluate the validity of the response style processes, 60 heterogeneous items were used as an extraneous response style measure, which was correlated with the ERS process. Pleininger and Meiser (2014) found convergent evidence of construct and criterion validity of the multiprocess model, citing a strong correlation between the peripheral measure of ERS and ERS from the IRTree, and between the peripheral measure of MRS and MRS from the IRTree. This study extends beyond Pleininger and Meiser (2014) by examining interrelationships of multiple scales and measures of response style across scales.

To answer Research Question 2, across item types, ERS traits tended to be positively correlated (Table 5). However, the correlations were stronger between items types that had the same number of Likert response options (i.e., 5) than between item types that had a different number of response options. This agrees with Weijters, Cabooter, and Schillewaert (2010), who reported that including a midpoint (i.e., 5-point Likert items) resulted in different levels of extreme responses than items without a midpoint (i.e., 4-point Likert items). However, in this study the inclusion of a midpoint resulted in more extreme responses. The 5-point items (medicinal cannabis, climate change) contained a higher proportion of extreme responses (.46 and .25, respectively) than the 4-point items (social media; .08). The large proportion of extreme responses for medicinal cannabis is the likely reason for the strong correlation between the single response style model measure of ERS (Model 2) and the measurement of a medicinal cannabis-specific measure of ERS in Model 1 (r = .909) and Model 3 (r = .854) and consistent rank-ordering of individuals across models in terms of their ERS estimates. In contrast, the low proportion of extreme responses for social media resulted in a much weaker correlation between the single response style model measure of ERS (Model 2) and the measurement of a social media-specific measure of ERS in Model 1 (r = .491) and Model 3 (r = .397). Of particular note is the magnitude of the correlation for social media, a 4-point Likert scale, between ERS EAP estimates of Model 1 and Model 2 (r = .491), which is much weaker in comparison to the other scales. Because the scales contain different content, it is unclear whether the difference in proportion of extreme responses is due to the item format or content.

The climate change scale contained a larger proportion of midpoint responses (.22) than the medicinal cannabis scale (.10). Similar to ERS, the larger proportion of midpoint responses for climate change is the likely reason for the stronger correlation between the single response style model measure of ERS (Model 2) and the measurement of a climate change-specific measure of MRS in Model 1 (r = .893) and Model 4 (r = .894). However, neither scale had a consistent rank-ordering of individuals across models, particularly for the middle three of five ranked groups. This is in contrast to the findings of Jeon and De Boeck (2019) who state that it is “reasonable to assume a general extreme responding latent variable that overarches the extreme responding latent variables across multiple sub-scales pp. 533.” They did not, however, compare across different traits, but across sub-scales representing similar traits. In agreement with that study, this study supports the use of generalized IRTree models that are flexible in the structure of the latent traits.

To aid in answering Research Question 3, Table 7 presents the correlations of TOI estimates across models. Correlations among TOI estimates tended to be highest within the full model, and weakest when comparing estimates among estimates of the single-scale models. However, the differences were negligible. Examining EAP estimates across models revealed important results that an appropriate response style model is important for adequate TOI estimation using MCMC estimation. Examining the scatterplot (Figure 5) revealed that measurement of TOI varied at the lower end of the score distribution, across response style models for two of the three scales (i.e., medicinal cannabis and climate change), but not for the third scale (i.e., social media). The medicinal cannabis and climate change are both 5-point Likert scales and had high proportions of extreme responses, which could be one reason for varying estimation of TOI across response style models.

Table 7.

Correlations Among TOI Estimates.

		Full (Model 1)			Single RS (Model 2)			Alone (Models 3–5)
		MC	CC	SM	MC	CC	SM	MC	CC	SM
Full	MC	1	0.53	0.04	0.87	0.49	0.02	0.86	0.43	0.02
	CC	0.53	1	0.21	0.48	0.89	0.21	0.41	0.91	0.17
	SM	0.04	0.21	1	0.00	0.23	0.97	−0.01	0.16	0.99
Single RS	MC	0.87	0.48	0.00	1	0.49	−0.02	0.84	0.41	−0.01
	CC	0.49	0.89	0.23	0.49	1	0.25	0.42	0.90	0.20
	SM	0.02	0.21	0.97	−0.02	0.25	1	−0.02	0.17	0.99
Alone	MC	0.86	0.41	−0.01	0.84	0.42	−0.02	1	0.35	−0.02
	CC	0.43	0.91	0.16	0.41	0.90	0.17	0.35	1	0.13
	SM	0.02	0.17	0.99	−0.01	0.20	0.99	−0.02	0.13	1

Limitations

Even though relative fit using marginal DIC was used to compare models, absolute fit of each model was not evaluated. Mplus provides posterior predictive model checking (PPMC) using the chi-square discrepancy statistic, but this discrepancy statistic has not been evaluated for use with IRTree models. Future research should examine model-data fit evaluation using additional discrepancy measures for PPMC for IRTrees. This study used the proportion of midpoint and extreme responders as discrepancy measures, but their use could be further examined with a simulation study. In addition, the choice of weakly informative priors on item parameters requires more investigation, not only in the specific models presented here, but for IRTrees in general.

Order of the scales remained constant across respondents: first, medicinal cannabis; second, climate change; and last, social media. There is limited research into item order, scale length, and whether the cognitive load of responding to multiple surveys impacts how respondents rely on response styles in providing responses. In addition, the presence of negatively worded items warrants more consideration. Park and Wu (2019) found that individuals tended to avoid using the lower extreme category, regardless of whether the item was positively or negatively worded, but future research could examine whether scales without negatively worded items showed more stability in response styles. Despite the need for more research, this study provides an important contribution to research on the accurate estimation and conceptualization of response style latent traits.

Footnotes

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Allison J. Ames

References

Aichholzer

(2013). Intra-individual variation of extreme response style in mixed-model panel studies. Social Science Research, 42, 957-970. https://doi.org/10.1016/j.ssresearch.2013.01.002

Ames

A. J.

Leventhal

(2019, July 15-19). Modeling changes in response style with longitudinal IRTree models. International Meeting of the Psychometric Society 2019.

Ames

A. J.

Myers

A. J.

(2020). Explaining variability in response style traits: A covariate-adjusted IRTree. Educational and Psychological Measurement. Advance online publication. https://doi.org/10.1177/0013164420969780

Austin

E. J.

Deary

I. J.

Egan

(2006). Individual differences in response scale use: Mixed Rasch modelling of responses to NEO-FFI items. Personality and Individual Differences, 40(6), 1235-1245. https://doi.org/10.1016/j.paid.2005.10.018

Bachman

J. G

O…Malley

P. M

. (1984). Yea-saying, nay-saying, and going to extremes: Black-White differences in response styles, Public Opinion Quarterly, 48(2), 491-509, https://doi.org/10.1086/268845

Baumgartner

Steenkamp

J.-B. E. M

. (2001). Response styles in marketing research: A cross-national investigation. Journal of Marketing Research, 38(2), 143-156. https://doi.org/10.1509/jmkr.38.2.143.18840

Billiet

J. B.

Davidov

(2008). Testing the Stability of an Acquiescence Style Factor Behind Two Interrelated Substantive Variables in a Panel Design. Sociological Methods & Research, 36(4), 542-562. https://doi.org/10.1177/0049124107313901

Böckenholt

(2012). Modeling multiple response processes in judgment and choice. Psychological Methods, 17(4), 665-678. https://doi.org/10.1037/a0028111

Böckenholt

(2017). Measuring response styles in Likert items. Psychological Methods, 22(1), 69-83. https://doi.org/10.1037/met0000106

10.

Böckenholt

Meiser

(2017). Response style analysis with threshold and multi-process IRT models: A review and tutorial. British Journal of Mathematical and Statistical Psychology, 70(1), 159-181. https://doi.org/10.1111/bmsp.12086

11.

Bolt

D. M.

Johnson

T. R.

(2009). Addressing score bias and differential item functioning due to individual differences in response style. Applied Psychological Measurement, 33(5), 335-352. https://doi.org/10.1177/0146621608329891

12.

Bolt

D. M.

Newton

J. R.

(2011). Multiscale measurement of extreme response style. Educational and Psychological Measurement, 71(5), 814-833. https://doi.org/10.1177/0013164410388411

13.

Cheung

G. W.

Rensvold

R. B.

(2000). Assessing extreme and acquiescence response sets in cross-cultural research using structural equations modeling. Journal of Cross-Cultural Psychology, 31(2), 187–212. https://doi.org/10.1177/0022022100031002003

14.

Clarke

(2001). Extreme response style in cross-cultural research. International Marketing Review, 18(3), 301-324. https://doi.org/10.1108/02651330110396488

15.

Cohen

(1992). A power primer. Psychological Bulletin, 112(1), 155-159. https://doi.org/10.1037/0033-2909.112.1.155

16.

Cronbach

L. J.

(1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. https://doi.org/10.1007/BF02310555

17.

Danner

Aichholzer

Rammstedt

(2015). Acquiescence in personality questionnaires: Relevance, domain specificity, and stability. Journal of Research in Personality, 57, 119-130. https://doi.org/10.1016/j.jrp.2015.05.004

18.

De Boeck

Partchev

. (2012). IRTrees: Tree-based item response models of the GLMM family. Journal of Statistical Software, 48, 1-28. https://doi.org/10.18637/jss.v048.c01

19.

De Jong

M. G.

Steenkamp

J.-B. E. M.

Fox

J.-P.

Baumgartner

. (2008). Using item response theory to measure extreme response style in marketing research: A global investigation. Journal of Marketing Research, 45(1), 104-115. https://doi.org/10.1509/jmkr.45.1.104

20.

Gelman

Rubin

D. B.

(1992). Inference from iterative simulation using multiple sequences (with discussion). Statistical Science, 7, 457-511. https://doi.org/10.1214/ss/1177011136

21.

Goldberg

L. R.

(1992). The development of markers for the big-five factor structure. Psychological Assessment, 4(1), 26-42. https://doi.org/10.1037/1040-3590.4.1.26

22.

Hart

H. N.

(1923). Progress report on a test of social attitudes and interests. The University.

23.

Harzing

(2006). Response styles in cross-national survey research: A 26-country study. International Journal of Cross Cultural Management, 6(2), 243-266. https://doi.org/10.1177/1470595806066332

24.

Jeon

De Boeck

(2019). Evaluation on types of invariance in studying extreme response bias with an IRTree approach. British Journal of Mathematical and Statistical Psychology, 72, 517-537. https://doi.org/10.1111/bmsp.12182

25.

Jin

Wang

(2014). Generalized IRT models for extreme response style. Educational and Psychological Measurement, 74(1), 116-138. https://doi.org/10.1177/0013164413498876

26.

Johnson

T. R.

(2003). On the use of heterogeneous thresholds ordinal regression models to account for individual differences in response style. Psychometrika, 68, 563-583. https://doi.org/10.1007/BF02295612

27.

Johnson

T. R.

Bolt

D. M.

(2010). On the use of factor-analytic multinomial logit item response models to account for individual differences in response style. Journal of Educational and Behavioral Statistics, 35, 92-114. https://doi.org/10.3102/1076998609340529

28.

Khorramdel

von Davier

(2014). Measuring response styles across the big five: A multiscale extension of an approach using multinomial processing trees. Multivariate Behavioral Research, 49(2), 161-177. https://doi.org/10.1080/00273171.2013.866536

29.

Khorramdel

von Davier

Pokropek

(2019). Combining mixture distribution and multidimensional IRTree models for the measurement of extreme response styles. British Journal of Mathematical and Statistical Psychology, 72(3), 538-559. https://doi.org/10.1111/bmsp.12179

30.

Leventhal

(2017). Extreme response style: Which model is best? [Unpublished doctoral dissertation, University of Pittsburgh]

31.

Liu

Wang

(2019). A general unfolding IRT model for multiple response styles. Applied Psychological Measurement, 43(3), 195-210. https://doi.org/10.1177/0146621618762743

32.

Liu

Harbaugh

A. G.

Harring

J. R.

Hancock

G. R.

(2017). The effect of extreme response and non-extreme response styles on testing measurement invariance. Frontiers in Psychology, 8, 726. https://doi.org/10.3389/fpsyg.2017.00726

33.

Martin

R. A.

Puhlik-Doris

Larsen

Gray

Weir

(2003). Individual differences in uses of humor and their relation to psychological well-being: Development of the humor styles questionnaire. Journal of Research in Personality, 37(1), 48-75. https://doi.org/10.1016/S0092-6566(02)00534-2

34.

Masters

G. N.

(1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149-174. https://doi.org/10.1007/BF02296272

35.

Meiser

Plieninger

Henninger

(2019). IRTree models with ordinal and multidimensional decision nodes for response styles and trait-based rating responses. British Journal of Mathematical and Statistical Psychology, 72(3), 501-516. https://doi.org/10.1111/bmsp.12158

36.

Moors

(2008). Exploring the effect of a middle response category on response style in attitude measurement. Quality & Quantity, 42, 779-794. https://doi.org/10.1007/s11135-006-9067-x

37.

Muthén

L. K.

Muthén

B. O.

(1998-2017). Mplus user’s guide (8th ed.). Muthén & Muthén.

38.

Nunnally

J. C.

Bernstein

I. H.

(1994). Psychometric theory. McGraw-Hill.

39.

OECD (2009), PISA 2006 Technical Report, PISA, OECD Publishing, Paris, https://doi.org/10.1787/9789264048096-en.

40.

Park

A. D.

(2019). Item response tree models to investigate acquiescence and extreme response styles in Likert-type rating scales. Educational and Psychological Measurement, 79(5), 911-930. https://doi.org/10.1177/0013164419829855

41.

Paulhus

D. L.

(1991). Measurement and control of response bias. In Robinson

J. P.

Shaver

P. R.

Wrightsman

L. S.

(Eds.), Measures of social psychological attitudes, Vol. 1. Measures of personality and social psychological attitudes (p. 17-59). Academic Press. https://doi.org/10.1016/B978-0-12-590241-0.50006-X

42.

Plieninger

Heck

D. W.

(2018). A new model for acquiescence at the interface of psychometrics and cognitive psychology. Multivariate Behavioral Research, 53(5), 633-654. https://doi.org/10.1080/00273171.2018.1469966

43.

Plieninger

Meiser

(2014). Validity of Multiprocess IRT Models for Separating Content and Response Styles. Educational and Psychological Measurement, 74(5), 875-899. https://doi.org/10.1177/0013164413514998

44.

Rosenberg

Silvert

(2013). Taking flight! Master the DISC styles to transform your career, your relationships . . . your life. FT Press.

45.

Rubin

D. B.

(1984). Bayesianly justifiable and relevant frequency calculations for the applied statistician. The Annals of Statistics, 12(4), 1151-1172. https://doi.org/10.1214/aos/1176346785

46.

Samejima

(1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement, 34(1), 100-114. https://doi.org/10.1002/j.2333-8504.1968.tb00153.x

47.

Sinharay

(2005). Assessing fit of unidimensional item response theory models using a Bayesian approach. Journal of Educational Measurement, 42, 375-394. 10.1111/j.1745-3984.2005.00021.x.

48.

Sinharay

(2006). Bayesian item fit analysis for unidimensional item response theory models. British Journal of Mathematical and Statistical Psychology, 59, 429-449. https://doi.org/10.1348/000711005X66888

49.

Van Herk

Poortinga

Y. H.

Verhallen

T. M. M

. (2004). Response styles in rating scales: Evidence of method bias in data from six EU countries. Journal of Cross-Cultural Psychology, 35(3), 346–360. https://doi.org/10.1177/0022022104264126

50.

Van Rosmalen

Van Herk

Groenen

P. J. F

. (2010). Identifying response styles: A latent-class bilinear multinomial logit model. Journal of Marketing Research, 47(1), 157-172. https://doi.org/10.1509/jmkr.47.1.157

51.

Wang

Nydick

S. W.

(2015). Comparing two algorithms for calibrating the restricted non-compensatory multidimensional IRT model. Applied Psychological Measurement, 39(2), 119-134. https://doi.org/10.1177/0146621614545983

52.

Weijters

Cabooter

Schillewaert

(2010). The effect of rating scale format on response styles: The number of response categories and response category labels. International Journal of Research in Marketing, 27, 236-247. https://doi.org/10.1016/j.ijresmar.2010.02.004

53.

Weijters

Geuens

Schillewaert

(2010a). The stability of individual response styles. Psychological Methods, 15(1), 96-110. https://doi.org/10.1037/a0018721

54.

Weijters

Geuens

Schillewaert

(2010b). The individual consistency of acquiescence and extreme response style in self-report questionnaires. Applied Psychological Measurement, 34(2), 105-121. https://doi.org/10.1177/0146621609338593

55.

Wetzel

Carstensen

C. H.

Böhnke

J. R.

(2013). Consistency of extreme response style and non-extreme response style across traits. Journal of Research in Personality, 47(2), 178-189. https://doi.org/10.1016/j.jrp.2012.10.010

56.

Wetzel

Lüdtke

Zettler

Böhnke

J. R.

(2016). The stability of extreme response style and acquiescence over 8 years. Assessment, 23(3). 279-291. https://doi.org/10.1177/1073191115583714