Measurement of Latent Variables With Different Rating Scales

Abstract

Effects of rating scale forms on cross-sectional reliability and measurement equivalence were investigated. A randomized experimental design was implemented, varying category labels and number of categories. The participants were 800 students at two German universities. In contrast to previous research, reliability assessment method was used, which relies on the congeneric measurement model. The experimental manipulation had differential effects on the reliability scores and measurement equivalence. Attitude strength seems to be a relevant moderator variable, which influences measurement equivalence. Overall, the results show that measurement quality is influenced by rating scale forms. Results are discussed in terms of their implications for latent variables measurement.

Keywords

labeling number of categories attitude strength measurement invariance reliability

Introduction

Multi-item instruments have often been used in surveys to address concepts of interest, assuming that the items (observed variables) provide an appropriate representation of the latent construct. The result of such data collection with an instrument is referred to as measurement of latent, not observed variables, with a number of observed reactions to the items. Such latent variable measurement is based on a measurement theory, for instance, the Classical Test Theory (Lord and Novick 1968), which allows for an evaluation of measurement quality in terms of reliability. Reliability is defined as the ratio of true variance to observed variance (Lord and Novick 1968). In this article, we address cross-sectional reliability, which relies on measurements that use more than two items and are carried out at one point in time (henceforth referred to as reliability).

An important prerequisite for measuring a latent variable and evaluating reliability is one-dimensionality, meaning that observed variables measure only one latent variable (Graham 2006; Raykov and Marcoulides 2011). The estimation of the true score of the latent variable and reliability evaluation also rely on assumptions, which are referred to as measurement model assumptions (Raykov and Marcoulides 2011). Three kinds of measurement models have been established, namely, the parallel model (PM), the tau equivalent model (TSEM), and the congeneric measurement model (CMM). Key assumption of PM is that the items measure the same true score with the same level of measurement precision, which is rarely the case (e.g., Raykov and Marcoulides 2011). Therefore, we do not consider PM in this article. TSEM assumes a uniform linear relationship between the observed variables and the latent variable. The least restrictive measurement model is CMM, in which the items’ linear relationships with the latent variable are allowed to differ. Testing the assumptions of measurement models is crucial when testing reliability. For example, Cronbach’s α (Cronbach 1951), which is a commonly used reliability measure, is based on the TSEM (Lord and Novick 1968). Because the assumptions of TSEM have often been violated in social science measurements, reliability coefficients based on CMM have been recommended as an alternative (Schweizer 2011).

Reliability can be affected by the choice of rating scales used with the items (e.g., Krosnick and Fabrigar 1997). A rating scale presents a measurement continuum that extends from one extreme to the other (e.g., strongly agree–strongly disagree). Rating scales can affect reliability because of differences in the interpretation of categories (Andrews 1984) or difficulties in administration, which could lead to satisficing behavior, such as superficial information processing (Krosnick and Alwin 1987). The number of categories and the labeling of a rating scale are basic cues that respondents use to understand the measurement continuum (Parducci 1983).

With respect to labeling, it is possible to differentiate between rating scales with verbal labels for each category (ALL form) and those with verbal labels for the end categories only (END form). Previous research has reported mixed results regarding the effects of verbal labeling on reliability: Whereas some studies found that the ALL form increases reliability (Alwin and Krosnick 1991; Menold et al. 2014), others did not (e.g., see a meta-analysis by Churchill and Peter 1984). The END form has also often been used with numeric labels (ENDN form). Numeric labels can increase satisficing because numbers are rarely used for self-description in everyday communication (Krosnick and Fabrigar 1997).

With respect to the number of categories, rating scales with five to seven categories often maximize reliability in the case of attitude measurement (e.g., Maitland 2009). Newer simulation studies found an increase in reliability when scales included five categories, compared to scales with three or four categories, whereas reliability coefficients remained stable for scales with five and seven categories (Lozano, García-Cueto, and Muñiz 2008; Parker, Vannest, and Davis 2013). However, Leung (2011) did not find differences in Cronbach’s α for scales with 4–11 categories. Weng (2004) analyzed the effects of the number of categories (varying between three and nine categories) and verbal labeling and did not find that reliability was affected. However, few studies address both numbers of categories and labeling, and the choice between five and seven categories is still under discussion in the literature (Toepoel and Dillman 2011).

Furthermore, there is a need for research on how rating scale forms affect reliability, because Cronbach’s α has been used in previous studies, although the assumptions of TSEM, upon which Cronbach’s α is based, have not been addressed (Leung 2011; Preston and Colman 2000; Weng 2004). The conclusions of these studies may, therefore, have been erroneous. Our first research question addresses the effect of rating scales on cross-sectional reliability: How do five versus seven categories, using ALL or END forms in both cases, affect reliability? In contrast to previous research, we will choose a reliability evaluation method upon the results of testing measurement model assumptions.

We also focus on measurement equivalence between different rating scales. When measurements that use different rating scales have to be compared, for example, between different waves of a survey, measurement equivalence is an important concern. Measurement equivalence can be assessed with the multigroup confirmatory factor analysis (MGCFA; e.g., Byrne 2011; Davidov, Schmidt, and Schwartz 2008). One can conclude that the same latent dimension is measured only if the factor loadings (metric invariance) and intercepts (scalar invariance) are equivalent (e.g., Byrne 2011; Davidov et al. 2008). Nonequivalent factor loadings between different rating scales may reflect variations in the understanding of an item’s meaning (e.g., Bollen 1989; Steinmetz 2013). Differences in items’ intercepts can be interpreted as different or biased reactions to the items, which are independent of the item content (Steinmetz 2013). Only a few studies address the measurement equivalence of different rating scales. Krebs and Hoffmeyer-Zlotnik (2010) found that rating scale orientation affected factor loadings, whereas item intercepts were not affected. However, we do not know whether varying the number of categories and the category labeling affects measurement equivalence. Accordingly, with our second research question, we investigate whether the measurements of one construct obtained with the same items but with different rating scales are equivalent with respect to (1) factor loadings and (2) items’ intercepts.

Little is known about the situational context of measurements. In particular, attitudes can differ in strength, a characteristic that may be a relevant factor when comparing the effects of different rating scales. While strong attitudes are highly pronounced and have significant impacts on cognition and behavior, weak attitudes do not (Bassili and Krosnick 2000). In addition, weak attitudes are more difficult to report, because respondents may be less motivated to report them (since these would be associated with a minor interest in the topic). Thus, the effect of rating scales may be more highly pronounced in the case of weak attitudes (Krosnick and Schuman 1988). Therefore, our third research question is: How does attitude strength (AS) affect measurement results, when different rating scales are used? Specifically, can the consideration of AS as an explanatory variable improve measurement equivalence between different rating scales?

Method

Participants

The data (N = 800) were collected at two German universities in 2011 and 2012. The study was a paper-and-pencil self-administered survey. Approximately 46 percent of respondents were male, and the respondents’ average age was 23 years (SD = 4.39 years). Thirty-two percent of the participants studied social sciences, 27 percent studied economics, 8 percent studied psychology, and 6 percent studied medicine; the remaining students studied other disciplines. There were no significant differences between the experimental groups with respect to gender (χ ² _(4,N=800) = .95; p > .10), age (F_(4,783) = 1.49; p > .10), or discipline of study (four experimental groups: χ ² _{(45,N= 690)} = 56.3; p > .10; five experimental groups: χ ² _(16,N=500) = 19.79; p > .10) (see Experimental Design).

Concepts

Two concepts were analyzed. The first concept—opinions about the European Union (EU)—represents an attitude toward an issue, whereas the second concept—studying effort—refers to attitudes toward oneself. Two types of concepts were used to examine whether the results could be generalized across a variety of concepts.

To measure opinions about the EU, six items (Table 1) were used that were part of a battery from the German Longitudinal Election Study (GLES; Rattinger et al. 2011) and that had been shown, via Principal Component Analysis (PCA), to cover one dimension (Menold et al. 2014). These items address negative aspects of the EU’s impact on German life and the economy and the preference for retaining state sovereignty within the EU. In the GLES, the items were used with a five-category agree–disagree ENDN rating scale.

Table 1.

Items Used in the Study.

Opinions about the EU

1. In Germany, social security is weakened by EU regulations

2. The regions in Europe should preserve their sovereignty

3. A member state should be able to quit the EU of its own accord

4. Eastward expansion has endangered job security in Germany

5. All EU citizens should be able to decide on EU contracts by referendum

6. Eastward expansion has led to an increase in criminal activities in Germany

Studying effort

1. If I have set myself a certain learning task, I make an effort to complete it

2. I make an effort to learn/study, even when I do not really like the subject

3. I do not give up on learning, even if the material is very difficult or complicated

4. If it is necessary, I study late into the evening and on the weekend

5. Normally, it does not take me much time to decide to start studying

6. Prior to an exam, I take a sufficient amount of time to go through all of the material again

7. I spend more time learning than most of my fellow students do

8. Before an exam, I study long enough to be sure to pass it

Note: EU = European Union. Source for the EU items: Rattinger et al. (2011), Menold et al. (2014). Source for the studying effort items: Wild and Schiefele (1994). Authors’ translation.

Studying effort was addressed by using eight items (Table 1) that also originated from a battery (students’ learning strategies; Wild and Schiefele 1994). Originally, a fully verbalized five-category frequency rating scale with numeric labels was used; according to the authors’ (Wild and Schiefele 1994) PCA, the eight items represent one dimension (Cronbach’s α = .74).

Two items (cf. Bassili and Krosnick 2000) were used to measure AS: (1) How important is the topic (studying effort vs. European politics) for a respondent (importance)? (2) How certain was a respondent when answering the items related to the topic (certainty)? These questions were placed after the items for each concept. Both items were rated on a five-category ALL rating scale, ranging from not at all important/not sure at all to very important/very sure. The two items were aggregated into a multiplicative index variable because low values of importance are supposed to not be compensated for by high values of certainty, and vice versa (cf., e.g., Trautwein et al. 2012).

Experimental Design

A randomized, experimental, between-subjects (split-ballot) design was used. This design is regarded as more accurate with respect to causal references of treatment manipulation than a within-design approach, such as a multitrait-multimethod design (Krosnick 2011). A systematic method with random start was applied for the randomization.

We used four experimental groups: a five-category fully verbalized rating scale (5ALL); a five-category end-points-only verbalized (5END) rating scale; and the same forms with seven categories (7ALL and 7END; Figure 1). We used verbal labels developed by Rohrmann (1978) and asked respondents to evaluate the degree to which an item applied. In addition, a fifth experimental group with the five-category ENDN (5ENDN) form from the GLES was used with some of the data (Figure 1). This allowed for a comparison of the previously described forms with this 5ENDN form, which has been used with the EU items in a large population survey. For the 5ENDN form, we retained the agreement dimension from GLES (“do not agree at all” to “fully agree”). However, because such “agree” scales are associated with acquiescence (e.g., Billiet and McClendon 2000), which we strove to avoid, we did not use the agreement dimension for the other four experimental groups.

Figure 1.

Rating scales in experimental groups. Note: Authors’ translation of German category labels.

Data Analysis

Data were first z-standardized for each treatment group separately, using the SPSS 20 software, to obtain comparability between the groups with differing numbers of rating scale categories (Krosnick 2011). Thus, the resulting variable values were z-distributed, ranging from −3 to +3, with a mean of 0 and a standard deviation of 1, regardless of the number of categories. To retain better comparability with the original variables, integers were used. To avoid effects of outliers, z-values with n ≤ 8 were summarized with the neighboring values (e.g., 3 with 2 or −3 with −2) in each experimental condition.

Tests of measurement model assumptions and reliability scores were analyzed for each experimental group, using confirmatory factor analyses (CFAs) as described by Raykov and Marcoulides (2011). While the assumptions of the CMM can be confirmed if the items have at least significant loadings (>.3) on the corresponding factor, the TSEM additionally requires an equality of factor loadings (e.g., Graham 2006; Raykov and Marcoulides 2011). Depending on the results of the measurement model test, a specific reliability method was then chosen. Whereas Cronbach’s α requires that assumptions of the TSEM must be met, McDonald’s Ω (McDonald 1999) relies on the CMM (e.g., Raykov and Marcoulides 2011).

Subsequently, various hierarchical MGCFA models were compared to address measurement equivalence among different rating scales. The first (baseline) model assumes the same one-dimensional structure of measurements in all groups (configural invariance), the second restricts the factor loadings to being equal (metric invariance), and the third model restricts the items’ intercepts to being equal across groups (scalar invariance; e.g., Byrne 2011). A second set of MGCFA models was tested to address the impact of AS on measurement equivalence by including AS as an observed covariate variable in the models. Linear regression paths of AS on observed variables were thereby included. Nonnested models with and without AS were compared by using the Bayesian Information Criterion (BIC) (Raftery 1995); lower BIC values represent models with a better fit to the data, and differences higher than ΔBIC = 10 were considered to be statistically significant (Raftery 1995).

Because normality was violated in all items and for all experimental groups (demonstrated by Kolmogorow–Smirnow statistics, with p < .001), the maximum likelihood parameter estimator that is robust with respect to violation of nonnormality (MLR; Muthén and Muthén 2010) was used in CFAs and MGCFAs. MLR has also been recommended for rating scales with more than four categories (Muthén and Muthén 2010; Raykov and Marcoulides 2011). Additional analyses for categorical data (weighted least squares (WLSMV) estimator; mixture factor analysis; Muthén and Muthén 2010) yielded results that were comparable with those obtained with the MLR. The model fit of CFA and MGCFAs was obtained using the chi-square (χ ² ) test, the root mean square error of approximation (RMSEA), and the comparative fit index (CFI; Beauducel and Wittmann 2005). The CFI should be .95 or higher (Hu and Bentler 1999), while an RMSEA of .08 or less indicates an acceptable fit (Raykov 1998). With respect to the comparison of nested MGCFA models, a significant change in χ² or a change in ΔCFI ≥ |.01| and ΔRMSEA ≥ |.015| indicates reasonable differences (Byrne 2011; Chen 2007).

Results Related to the EU Opinion Concept

Tests of Measurement Models

The goodness-of-fit statistics of the CFA for the six EU items, in which no equality restrictions for single items within a condition were modeled, are presented in Table 2 (initial models). Acceptable RMSEA and CFI values were observed in the 5END and 5ENDN groups, but the factor loadings of two items (EU3 and EU5) were not significant. In the other treatment groups, no goodness-of-fit statistics reached the cutoff values, and nonsignificant standardized factor loadings or loadings much lower than λ = .30 were obtained. A relatively tenable result was found for the 7ALL group, in which only one item (EU3) failed to have a significant loading, and goodness-of-fit statistics were near the cutoff criteria. In total, the assumptions of the CMM—all EU items have at least a linear relationship with the latent variable—were not supported by the data.

Table 2.

Measurement Model Test (CFA) Results for the EU Items.

	5ALL	5END	7ALL	7END	5ENDN
Initial models
Loadings λ of items
EU1	.46***	.54***	.52***	.39***	.49**
EU2	.27*	.17**	.47***	.30**	.26*
EU3	.07	.20*	.11	.21*	−.22
EU4	.75***	.57***	.67***	.58***	.61***
EU5	.02	.05	.40***	.06	.17
EU6	.70***	.44***	.65***	.67***	.73***
Goodness-of-fit
χ²(df = 9)	38.34***	3.56	16.88*	17.87*	5.75
CFI	0.74	1.00	.91	.77	1.00
RMSEA	0.13	0.00	.07	.08	0.00
90 percent CI of RMSEA	[.09, .17]	[.00, .02]	[.00, .12]	[.02, .13]	[.00, .08]
n	191	161	163	153	94
Re-specified models
Loadings λ of items
EU1	.50***	.52***	.51***	.38***	.42***
EU2	—	—	.46***	—	—
EU3	—	—	—	—	—
EU4	.70***	.46***	.67***	.54***	.58***
EU5	—	—	.39***	—	—
EU6	.65***	.45***	.66***	.58***	.83***
Goodness-of-fit
χ²(df)	0.00 (0)	0.00 (0)	9.19 (5)	0.00 (0)	0.00 (0)
CFI	1.00	1.00	0.95	1.00	1.00
RMSEA	0.00	0.00	0.07	0.00	0.00
90 percent CI of RMSEA	[.00, .00]	[.00, .00]	[.00, .14]	[.00, .00]	[.00, .00]
n	193	162	163	155	95

Note: CFA = confirmatory factor analysis; CI = confidence interval; EU = European Union; RMSEA = root mean square error of approximation; df = degrees of freedom. Experimental groups: 5ALL (five categories, fully verbalized), 5END (five categories, only end categories verbalized), 7ALL (seven categories, fully verbalized), 7END (seven categories, only end categories verbalized), and 5ENDN (five categories, end categories verbalized, numeric labels for each category). Differences in n (number of cases) because of data missing.

*p < .05.

**p < .01.

***p < .001.

To reach the CMM level, three items had to be deleted in the groups 5ALL, 5END, 7END, and 5END. The three remaining items (EU1, EU4, and EU6) in these groups describe negative effects of the EU on German life and the economy; thus, the concept to be measured is much narrower. Use of only three items leaves df equal to 0, so that the CFA model fit cannot be evaluated. Hence, in the 5ALL, 5END, 7END, and 5ENDN conditions, items had sufficient factor loadings and no correlated error terms (Table 2, re-specified models). In the condition 7ALL, one item had to be deleted (EU3, quitting of the EU by a state). The CMM can be assumed for five items in the 7ALL group, as shown by tenable goodness-of-fit for the re-specified model.

The TSEM assumptions (factor loadings are equal within one rating scale group) could not be confirmed, either for the initial or for the re-specified models, with the exception of the re-specified model in the 7END group. The detailed results of this test are available from the corresponding author on request.

Reliability

We calculated McDonald’s Ω because, overall, the assumptions of the TSEM were not met and thus Cronbach’s α was inappropriate. Table 3 shows the reliability coefficients for both the initial six items and the re-specified models. Notably, the inclusion of some error covariances improved the goodness-of-fit of the initial models in the 5ALL, 7ALL, and 7END groups (see note in Table 3); thus, the error covariances were considered as a part of the error variance when obtaining reliability (cf. Raykov and Marcoulides 2011). With respect to the initial models, an acceptable reliability coefficient could be found only in the 7ALL group. In all other groups, reliability coefficients were unacceptably low (i.e., with a reliability of ∼.50; the measurement results were inflated by a nonsystematic error of up to 50 percent). The deletion of items for the re-specified models, in which the CMM measurement level could be assumed, led to a remarkable increase in reliability in the 5ALL and 5ENDN conditions. As a result, for the re-specified models, acceptable reliability coefficients with highly overlapping confidence intervals were found for the 5ALL, 7ALL, and 5ENDN groups.

Table 3.

Reliability (r; McDonald’s Ω) with Standard Errors (SE) in Rating Scale Groups.

Model		5ALL	5END	7ALL	7END	5ENDN
EU
Initial	r (SE)	0.47 (0.057)	0.47 (0.079)	0.65 (0.05)	0.50 (0.08)	0.45 (0.07)
	95 percent CI	[0.36, 0.58]	[0.32, 0.62]	[0.56, 0.73]	[0.36, 0.65]	[0.32, 0.59]
	n (items)	6	6	6	6	6
Re-specified	r (SE)	0.65 (0.04)	0.49 (0.07)	0.65 (0.04)	0.54 (0.06)	0.62 (0.06)
	95 percent CI	[0.57, −0.72]	[0.36, −0.62]	[0.56, −0.73]	[0.42, −0.65]	[0.50, −0.73]
	n (items)	3	3	5	3	3
Studying effort
Initial	r (SE)	0.72 (0.03)	0.81 (0.02)	0.74 (0.03)	0.71 (0.04)	0.66 (0.05)
	95 percent CI	[0.67, −0.77]	[0.77, −0.85]	[0.68, −0.79]	[0.63, −0.78]	[0.57, −0.74]
	n (items)	8	8	8	8	8
Re-specified	r (SE)	0.77 (0.03)	0.81 (0.02)	0.83 (0.02)	0.77 (0.03)	0.68 (0.05)
	95 percent CI	[0.72, −0.82]	[0.76, −0.85]	[0.78, −0.87]	[0.70, −0.82]	[0.58, −0.77]
	n (items)	6	7	6	5	4

Note: CI = confidence interval; EU = European Union. Experimental groups: 5ALL (five categories, fully verbalized), 5END (five categories, only end categories verbalized), 7ALL (seven categories fully verbalized), 7END (seven categories, only end categories verbalized), and 5ENDN (five categories, end categories verbalized, numeric labels for each category). Initial model EU items: Two error covariances (ECs) are considered in the 5ALL group and one is considered in the 7ALL and 7END groups, to reach a tenable model fit. Initial model studying effort: four ECs are included in the 5ALL group, one EC in the 5END group, three ECs in the 7ALL group, four ECs in the 7END group, and five ECs in the 5ENDN group.

Measurement Equivalence between Rating Scales

With the help of MGCFAs, we compared the goodness-of-fit statistics between the unrestricted model (configural invariance) and models in which either the factor loadings (metric invariance) or the item intercepts were restricted to being equal (scalar invariance) among the groups. The configural model for the six items (initial model) did not yield a reasonable goodness-of-fit, χ²(45) = 83.75, p < .001; RMSEA = .07; 95 percent confidence interval, CI [.05, .10]; CFI = .88). To improve the model fit, two error covariances were allowed for (between EU2 and EU3 and between EU4 and EU6, Table 1). The error covariances were restricted to being equal in different rating scale groups to avoid an effect on the comparison of the MGCFA models. With these modifications, a tenable goodness-of-fit could be reached for the initial model (see Table 4: Configural). Restricting factor loadings to being equal significantly increased χ², and decreased the value of CFI (Table 4, Metric), so that equality of loadings among the groups could not be confirmed. Restricting the item intercepts to being equal led to a very large and significant decrease in model fit for all model-fit statistics (Table 4: EU initial model, Scalar), so that the equality of items’ intercepts between different rating scale groups was not given.

Table 4.

Measurement Equivalence: Comparison of MGCFA Models.

Model	χ² (df)	Δχ²	RMSEA	ΔRMSEA	CFI	ΔCFI	Sample size adjusted BIC
EU
Initial model
Configural	53.68 (43)	—	.04	—	.97	—	12,780.62
Metric	95.45 (67)*	41.02**	.05	.01	.91	.06	12,742.76
Scalar	166.64 (67)***	114.00***	.10	.06	.69	.28	12,810.50
Three items
Configural	0.00 (0)	—	.00	—	1.00	—	6,448.64
Metric	9.89 (12)	9.89	.00	.00	1.00	.00	6,416.71
Scalar	98.17 (12)***	87.95***	.21	.21	.59	.41	6,502.77
Initial model with AS
Configural	58.44 (43)	—	.05	—	.95	—	12,826.24
Metric	98.54 (67)**	39.57*	.06	.01	.90	.05	12,787.11
Scalar	104.11 (67)**	45.86**	.06	.01	.89	.06	12,787.45
Three items with AS
Configural	0.00 (0)	—	.00	—	1.00	—	6,468.63
Metric	9.74 (12)	9.74	.00	.00	1.00	.00	6,463.67
Scalar	33.26 (12)***	33.26***	.11	.11	.90	.10	6,459.23
Studying effort
Initial model
Configural	168.87 (95)***	—	.07	—	.95	—	15,829.16
Metric	260.89 (127)***	93.99***	.08	.01	.90	.05	15,814.49
Scalar	380.70 (127)***	190.56***	.11	.04	.84	.09	15,911.10
Four items
Configural	10.39 (10)	—	.02	—	1.00	—	8,159.51
Metric	82.65 (26)***	71.37***	.12	.10	.89	.11	8,195.75
Scalar	83.91 (26)***	71.89***	.12	.10	.89	.11	8,198.85
Initial model with AS
Configural	165.34 (95)***	—	.07	—	.95	—	15,726.92
Metric	256.37 (127)***	93.67***	.08	.01	.91	.04	15,711.54
Scalar	234.81 (127)***	69.32**	.07	.00	.93	.02	15,695.38
Four items with AS
Configural	11.23 (10)	—	.03	—	1.00	—	8,074.85
Metric	76.41 (26)***	63.63***	.11	.08	.91	.09	8,085.78
Scalar	58.05 (26)***	45.15***	.09	.06	.95	.05	8,070.08

Note: AS = attitude strength; BIC = Bayesian information criterion; CFI = comparative fit index; EU = European Union; MGCFA = multigroup confirmatory factor analysis; RMSEA = root mean square error of approximation. Configural is baseline; initial models with correlated error covariances (equal across groups): EU2 with EU3 and EU4 with EU6. Studying effort: 1 with 2; 2 with 3; 3 with 4; 4 with 8; and 6 with 8. We ran the same set of models (without AS), to compare experimental groups with only five and only seven categories. We obtained a lack of metric and scalar invariance for five category groups for both concepts that we addressed. For seven category groups, metric and scalar invariance were given for the model with only three EU items, but not for the models with all EU items. For the studying effort, the MGCFA analysis revealed a lack of metric and scalar invariance in the case of five categories, which was valid for both eight and four items. In the case of seven categories, metric and scalar invariance were given for eight items, whereas metric invariance was violated for four items. Therefore, violations of invariance could also be found in the case of comparisons with only five or only seven categories. Detailed results of this analysis are available from the corresponding author on request.

*p < .05.

**p < .01.

***p < .001.

However, in the case of the six items, the CMM assumptions could not be regarded as fulfilled. Significant factor loadings and an absence of correlated error terms were found with an MGCFA using the items EU1, EU4, and EU6. Restricting the factor loadings to being equal in these three items did not significantly change the goodness-of-fit statistics (Table 4, EU, three items, metric), which demonstrates that the factor loadings did not significantly differ between the five rating scale groups. However, restricting the items’ intercepts to being equal remarkably worsened the model fit for all goodness-of-fit statistics (Table 4, EU, three items, scalar). Thus, rating scales had an impact on the items’ intercepts, which was demonstrated for models with six and three items.

The nonnested MGCFA models with and without AS were compared in terms of BIC; the results are shown in Table 4. It can be seen that, for the configural and metric models, BIC is higher with AS than without AS. However, BIC is much lower for scalar models with AS than without AS; this holds true for both the initial and the three-item models. Therefore, AS moderated the effect of rating scales on intercepts.

Results Related to the Measurement of Studying Effort

Tests of Measurement Models

The goodness-of-fit statistics of the CFAs for the eight studying effort items, in which no equality of factor loadings were modeled within a rating scale group, are presented in Table 5 (initial model). In all groups, standardized factor loadings were significant and equal to or higher than λ = .30. Nevertheless, a relatively reasonable model fit with respect to RMSEA and CFI was found only for the 5END group (with the CFI near the cutoff value).

Table 5.

Measurement Model Test (CFA) Results for Studying Effort.

	5ALL	5END	7ALL	7END	5ENDN
Initial model
Loadings λ of items
1	.55***	.67***	.53***	.52***	.68***
2	.74***	.83***	.58***	.46***	.75***
3	.58***	.62***	.46***	.36**	.67***
4	.32***	.36***	.47***	.29**	.36**
5	.55***	.56***	.64***	.66***	.32*
6	.59***	.54***	.76***	.64***	.32*
7	.50***	.64***	.66***	.70***	.32*
8	.52***	.59***	.55***	.59***	.66***
Goodness-of-fit
χ²(df = 21)	77.03***	40.12**	95.99***	80.52***	53.61***
CFI	.87	.94	.81	.74	.69
RMSEA	.12	.07	.15	.13	.13
90 percent CI of RMSEA	[.09, .15]	[.04, .11]	[.12, .18]	[.10, .17]	[.09, .17]
n	192	165	166	155	96
Re-specified model
Loadings λ of items
1	.55***	.67***	—	.44***	.51***
2	.74***	.86***	.49***	—	.86***
3	.57***	.63***	—	—	.65***
4	.36***	.37***	.44***	—	—
5	.54***	.53***	.66***	.69***	—
6	—	—	.83***	.66***	—
7	—	.63***	.74***	.75***	—
8	.74***	.58***	.69***	.59***	.45***
Goodness-of-fit
χ²(df)	12.35 (9)	17.04 (14)	16.08 (9)	12.56 (5)	0.03 (2)
CFI	.98	.99	.97	.95	1.00
RMSEA	.04	.04	.07	.10	.00
90 percent CI of RMSEA	[.00, .10]	[.00, .09]	[.00, .12]	[.03, .17]	[.00, .00]
n	194	165	166	155	97

Note: CFA = confirmatory factor analysis; CFI = comparative fit index; CI = confidence interval; RMSEA = root mean square error of approximation. Experimental groups: 5ALL (five categories, fully verbalized), 5END (five categories, only end categories verbalized), 7ALL (seven categories, fully verbalized), 7END (seven categories, only end categories verbalized), 5ENDN (five categories, end categories verbalized, numeric labels for each category). Differences in n (number of cases) because of data missing.

*p < .05.

**p < .01.

***p < .001.

According to modification indices (MIs), the goodness-of-fit can be improved when error covariances between items are included. However, error covariances may mean that one-dimensionality is violated (e.g., Raykov and Marcoulides 2011). Therefore, with re-specifications, we excluded items that led to error covariances. One or two items were deleted in the 5ALL, 5END, and 7ALL groups, while three or even four items had to be excluded in the 7END and 5ENDN groups. Following deletion of items, we obtained a tenable model fit with respect to χ² and CFI statistics in each rating scale group (Table 5). We conclude that the CMM level could be reached, in terms of the deletion of certain items; this was associated with differing amounts of information loss in the various rating scale groups.

Restricting the factor loadings to being equal within each rating scale group demonstrated that TSEM cannot be assumed in the initial and re-specified models. Detailed results are available on request.

Reliability

For the studying effort items, McDonald’s Ω was also the appropriate measure for reliability evaluation. To reach a tenable model fit in the initial model, correlated error terms were considered (note in Table 3). The results are presented in Table 3 (initial). A significantly higher reliability was found in the 5END group, compared to the other groups. The reliability also tended to be higher in the 7ALL group than in the 5ENDN group. Using a reduced number of items for which a CMM level is given (Table 3, re-specified) increased reliability in the 5ALL, 7ALL, and 7END groups but not in the 5ENDN group. Overall, reliability tended to be higher in the 5END and 7ALL groups than in the other groups.

Measurement Equivalence between Rating Scales

A one-factor MGCFA model of eight studying effort items reached inadequate goodness-of-fit: χ² ₍₁₀₀₎ = 309.33 (p < .001), CFI = .85, RMSEA = .12 (90 percent CI = [.10, .13]). An improvement of goodness-of-fit could be achieved by including numerous error covariances (see note in Table 4) that were modeled to be equal in all rating scale groups (Table 4). Restricting factor loadings to being equal among the groups remarkably increased χ² and decreased CFI (Table 4, initial model, metric). Restricting the items’ intercepts to being equal led to a very large and significant decrease of model fit for all goodness-of-fit statistics (see Table 4, initial model, scalar), so that neither the equality of items’ loadings nor that of intercepts can be assumed for the eight studying effort items.

We found a four-item model (with items 2, 5, 6, and 7; see Table 1) in which no correlated error terms were observable (CMM). The tenable goodness-of-fit statistics for the corresponding configural model are presented in Table 4 (four items). With these four items, the equivalence of factor loadings and items’ intercepts could not be confirmed, as can be seen in Table 4. According to BIC, models with AS suited the data better than those without AS (Table 4).

Discussion

With reference to the first research question, the forms of rating scales were found to influence reliability for both concepts. Where a CMM measurement level could be assumed, reliability was somewhat higher in the case of the EU items when either verbal or numerical labels were used for all categories than for the forms with verbal labels only for the end categories. In the case of studying effort, the 5END and the 7ALL forms tended to have higher reliability than the other forms. The results are only partly in line with other studies, which reported that the ALL form increases reliability (e.g., Krosnick and Fabrigar 1997; Menold et al. 2014). In contrast to previous research, we used a reliability evaluation method that did not require the TSEM level. For both concepts, obtaining CMM was associated with less information loss for the 7ALL rating scale than for most other scales. The 7ALL form also reached an acceptable level of reliability for both concepts. The 7ALL scale seems to decrease heterogeneity and thereby to increase the measurement consistency of items. An explanation for this result could be that fully verbally labeled response categories improve the clarity of measurement dimension. Furthermore, seven categories allow for a higher differentiation of responses than do five categories.

Regarding the second research question, we found, for both concepts, that measurement equivalence between different rating scales was violated with respect to metric and scalar equivalence. Therefore, measurement results obtained with different rating scales are not comparable. This is consistent with the results of Krebs and Hoffmeyer-Zlotnik (2010), who addressed rating scale orientation. Respondents seemed to understand and handle varying rating scales differently; therefore, it cannot be assumed that the same latent variable is measured with the same items when different rating scales are used. We therefore advise researchers to be cautious when using nonuniform rating scales for the measurement of a construct. When rating scales are changed from wave to wave in a survey, it is important to check measurement equivalence between these waves.

With regard to the third research question, AS was a significant explanatory variable, capable of increasing measurement equivalence between different rating scales. However, even including this variable did not lead to the achievement of metric or scalar invariance. Further research could address additional explanatory variables, for example, the need for cognition (Cacioppo and Petty 1982), or other personal variables (e.g., Rammstedt and John 2007) that may affect respondents’ accuracy or motivation when responding to survey questions.

One limitation of this study is that a student sample was used. Generalization of the results to other respondent groups must therefore be undertaken with caution. In addition, numeric labels were used for only one rating scale, to enable a comparison with original survey data containing the EU items and, thus, the number of categories was not varied in this case. Because numeric labels are widely used in surveys, more research is needed to test their impact on reliability and measurement equivalence.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Alwin

Duane F.

Krosnick

Jon A.

. 1991. “The Reliability of Survey Attitude Measurement: The Influence of Question and Respondent Attributes.” Sociological Methods & Research 20:139–81.

Andrews

Frank M.

1984. “Construct Validity and Error Components of Survey Measures: A Structural Modeling Approach.” Public Opinion Quarterly 48:409–48.

Bassili

John N.

Krosnick

Jon A.

. 2000. “Do Strength-related Attitude Properties Determine Susceptibility to Response Effects? New Evidence from Response Latency, Attitude Extremity, and Aggregate Indices.” Political Psychology 21:107–32.

Beauducel

Andre

Wittmann

Werner W.

. 2005. “Simulation Study on Fit Indexes in CFA Based on Data with Slightly Distorted Simple Structure.” Structural Equation Modeling: A Multidisciplinary Journal 12:41–75.

Billiet

Jaak B.

McClendon

McKee J.

. 2000. “Modeling Acquiescence in Measurement Models for Two Balanced Sets of Items.” Structural Equation Modeling: A Multidisciplinary Journal 7:608–28.

Bollen

Kenneth A.

1989. Structural Equations with Latent Variables. New York: Wiley.

Byrne

Barbara

. 2011. Structural Equation Modeling with Mplus: Basic Concepts, Applications, and Programming (Multivariate Applications). London, UK: Taylor & Francis.

Cacioppo

John T.

Petty

Richard E.

. 1982. “The Need for Cognition.” Journal of Personality and Social Psychology 42:116–31.

Chen

Fang F.

2007. “Sensitivity of Goodness of Fit Indexes to Lack of Measurement Invariance.” Structural Equation Modeling: A Multidisciplinary Journal 14:464–504. doi:10.1080/10705510701301834

10.

Churchill

Gilbert A.

Peter

J. P.

. 1984. “Research Design Effects on the Reliability of Rating Scales: A Meta-analysis.” Journal of Marketing Research 21:360–75.

11.

Cronbach

Lee J.

1951. “Coefficient Alpha and the Internal Structure of Tests.” Psychometrika 16:297–334.

12.

Davidov

Schmidt

Schwartz

S. H.

. 2008. “Bringing Values Back in: The Adequacy of the European Social Survey to Measure Values in 20 Countries.” Public Opinion Quarterly 72:420–45.

13.

Graham

James M.

2006. “Congeneric and (Essentially) Tau-equivalent Estimates of Score Reliability: What They Are and How to Use Them.” Educational and Psychological Measurement 66:930–44. doi:10.1177/0013164406288165.

14.

Li-tze

Bentler

Peter M.

. 1999. “Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria Versus New Alternatives.” Structural Equation Modeling: A Multidisciplinary Journal 6:1–55.

15.

Krebs

Dagmar

Hoffmeyer-Zlotnik

Juergen H.

. 2010. “Positive First or Negative First?” Methodology: European Journal of Research Methods for the Behavioral and Social Sciences 6:118–27. doi:10.1027/1614-2241/a000013.

16.

Krosnick

Jon A.

2011. “Experiments for Evaluating Survey Questions.” Pp. 215–38 in Question Evaluation Methods. Contribution to the Science of Data Quality, edited by Madans

Miller

Maitland

Willis

. Hoboken, NJ: Wiley.

17.

Krosnick

Jon A.

Alwin

. 1987. “An Evaluation of a Cognitive Theory of Response-order Effects in Survey Measurement.” Public Opinion Quarterly 51:201–19.

18.

Krosnick

Jon A.

Fabrigar

Leandre R.

. 1997. “Designing Rating Scales for Effective Measurement in Surveys.” Pp. 141–64 in Survey Measurement and Process Quality, edited by Lyberg

Biemer

Collins

de Leeuw

Dippo

Schwarz

Trewin

. Hoboken, NJ: John Wiley & Sons, Inc.

19.

Krosnick

Jon A.

Schuman

Howard

. 1988. “Attitude Intensity, Importance, and Certainty and Susceptibility to Response Effects.” Journal of Personality and Social Psychology 54:940–52.

20.

Leung

Shing-On

. 2011. “A Comparison of Psychometric Properties and Normality in 4-, 5-, 6-, and 11-Point Likert Scales.” Journal of Social Service Research 37:412–21.

21.

Lord

Frederic M.

Novick

Melvin R.

. 1968. Statistical Theories of Mental Test Scores. Reading, MA: Addison-Wesley.

22.

Lozano

Luis M.

García-Cueto

Eduardo

Muñiz

José

. 2008. “Effect of the Number of Response Categories on the Reliability and Validity of Rating Scales.” Methodology: European Journal of Research Methods for the Behavioral and Social Sciences 4:73–79.

23.

Maitland

Aaron

. 2009. “How many scale points should I include for attitudinal questions?” Survey Practice 5. Retrieved from http://www.surveypractice.org/index.php/SurveyPractice/article/viewFile/179/pdf at 20.04.2015.

24.

McDonald

R. P.

1999. Test Theory: A Unified Treatment. Mahwah, NJ: Erlbaum.

25.

Menold

Natalja

Kaczmirek

Lars

Lenzner

Timo

Neusar

Aleš

. 2014. “How Do Respondents Attend to Verbal Labels in Rating Scales?” Field Methods 26:21–39. doi:10.1177/1525822X13508270.

26.

Muthén

Linda K.

Muthén

Bengt O.

. 2010. Mplus User’s Guide. 6th ed. Los Angeles, CA: Muthén & Muthén.

27.

Parducci

Allen

. 1983. “Category Ratings and the Relational Character of Judgment.” Pp. 262–82 in Modern Issues in Perception, edited by H. G. Geissler, H. F. J. M. Bulfart, E. L. H. Leeuwenberg, and Sarris V. Berlin: VEB Deutscher Verlag der Wissenschaften.

28.

Parker

Richard I.

Vannest

Kimberly J.

Davis

John L.

. 2013. “Reliability of Multi-category Rating Scales.” Journal of School Psychology 51:217–29.

29.

Preston

Carolyn C.

Colman

Andrew M.

. 2000. “Optimal Number of Response Categories in Rating Scales: Reliability, Validity, Discriminating Power, and Respondent Preferences.” Acta Psychologica 104:1–15.

30.

Raftery

Adrian E.

1995. “Bayesian Model Selection in Social Research.” Sociological Methodology 25:111–64.

31.

Rammstedt

Beatrice

John

Oliver P.

. 2007. “Measuring Personality in One Minute or Less: A 010-item Short Version of the Big Five Inventory in English and German.” Journal of Research in Personality 41:203–12.

32.

Rattinger

Hans

Rüdiger

Schmitt-Beck

Sigrid

Roßteutscher

Weßels

Bernhard

. 2011. Long-term Online Tracking T17 (GLES), Köln. Retrieved from https://dbk.gesis.org/dbksearch/sdesc2.asp?no=5347&db=e&doi=10.4232/1.10763 at 20.04.2015.

33.

Raykov

Tenko

. 1998. “On the Use of Confirmatory Factor Analysis in Personality Research.” Personality and Individual Differences 24:291–93.

34.

Raykov

Tenko.

Marcoulides

George A.

. 2011. Introduction to Psychometric Theory. New York: Routledge.

35.

Rohrmann

Bernd

. 1978. “Empirische Studien zur Entwicklung von Antwortskalen für die sozialwissenschaftliche Forschung.” Zeitschrift für Sozialpsychologie 9:222–45.

36.

Schweizer

Karl

. 2011. “On the Changing Role of Cronbach’s α in the Evaluation of the Quality of a Measure.” European Journal of Psychological Assessment 27:143–44. doi:10.1027/1015-5759/a000069.

37.

Steinmetz

Holger

. 2013. “Analyzing Observed Composite Differences across Groups.” Methodology: European Journal of Research Methods for the Behavioral and Social Sciences 9:1–12. doi:10.1027/1614-2241/a000049.

38.

Toepoel

Vera

Dillman

Don A.

. 2011. “Words, Numbers, and Visual Heuristics in Web Surveys: Is There a Hierarchy of Importance?” Social Science Computer Review 29:193–207. doi:10.1177/0894439310370070.

39.

Trautwein

Ulrich

Marsh

Herbert W.

Nagengast

Benjamin

Lüdtke

Oliver

Nagy

Gabriel

Jonkmann

Kathrin

. 2012. “Probing for the Multiplicative Term in Modern Expectancy–value Theory: A Latent Interaction Modeling Study.” Journal of Educational Psychology 104:763–77. doi:10.1037/a0027470.

40.

Weng

Li-Jen

. 2004. “Impact of the Number of Response Categories and Anchor Labels on Coefficient Alpha and Test-retest Reliability.” Educational and Psychological Measurement 64:956–97.

41.

Wild

Klaus P.

Schiefele

Ulrich

. 1994. “Lernstrategien im Studium. Ergebnisse zur Faktorenstruktur und Reliabilität eines neuen Fragebogens.” Zeitschrift für Differentielle und Diagnostische Psychologie 15:185–200.