Scaling Sensitive Factorial Survey Analysis

Abstract

In recent decades, factorial survey experiments (FSEs) have become increasingly widespread and successful for analyzing attitudes and behavioral intentions. FSEs measure the ratings of multidimensional treatments embedded in textual scenarios, which are called vignettes. Analyses of FSEs often assume that these ratings are interval scaled. Past research indicates that this assumption is problematic. Therefore, the following article develops a design for interval scaling tests in FSEs and a method for analyzing FSEs which is sensitive to the scaling level of ratings. An exemplary application of scaling sensitive factorial survey analysis in comparison to standard methods yields effect sizes, which are about a sixth larger with regard to the treatments used in the FSE and about a third larger with respect to influences of between respondent differences. The new method also enables the evaluation of interval in contrast to noninterval scaled rating behavior.

Keywords

factorial survey experiment interval scale experimental design survey data quality structural equation modeling

In recent decades, factorial survey experiments (FSEs, Jasso and Rossi 1977; Rossi and Anderson 1982) have become an increasingly widespread and successful method for measuring and analyzing attitudes, judgments, beliefs, opinions, preferences, and behavioral intentions (Wallander 2009). An FSE is a type of survey experiment consisting of—typical textual—scenarios (called vignettes) combining several treatments (called dimensions) with controlled varying doses (called levels). The vignettes are used as stimuli which are singly evaluated by respondents (Auspurg and Hinz 2015). The review of Wallander (2009) shows that in 71 percent of the FSEs, the responses to these evaluation tasks were analyzed with methods that assume an observed interval or log-interval scale like linear regression or analysis of variance, that is, it was assumed that not only the ranking of the vignette evaluations but also the distance between these rankings is informative. With regard to the response instruments used for the evaluation tasks, 46 percent of the reviewed FSEs used ordered and 15 percent unordered category rating instruments (Wallander 2009).¹ Hence, while interval scales are not necessarily implied by the response instruments commonly used in FSEs, the methods of analysis typically implemented require the assumption of an interval scale. Such methods of analysis are also most widely discussed in the literature on analyzing FSEs (Hox, Kreft, and Hermkens 1991, Jasso 2006). Past research demonstrates that the match between responses and the interval scaling assumption varies between individuals and study settings (Wegener 1982). Thus, there is an interval scaled and a noninterval scaled evaluation mode in empirical applications.

These results indicate that assuming interval scaled vignette ratings, in general, is problematic since for some cases, the distance between the ratings is not informative for the analyses. Therefore, it is plausible to expect that utilizing information on the scaling level of vignette ratings improves the effect sizes, that is, standardized mean differences, resulting from factorial survey analyses and furthermore, that factors influencing vignette ratings can vary between an interval and a noninterval scaled response mode. Based on these expectations, the following article adds to the methodological research on FSEs and survey response quality by answering two related questions: First, how is the interval scaling assumption testable in FSEs? Second, how can one analyze FSEs in a way which is sensitive to the scaling level of vignette ratings, that is, with a method which does not mistakenly treat all vignette evaluations in an FSE as interval scaled but also does not disregard the truly interval scaled information measured in a FSE?

This article is structured as follows: First, I introduce methods for testing the interval scaling assumption based on category ratings (Orth 1982; Orth and Wegener 1983), discuss expectations regarding this assumption in FSEs, and introduce a design implementing such a test in a FSE. Then, I develop a method for scaling sensitive factorial survey analysis (SSFSA) using the previously introduced design and generalized multilevel structural equation modeling (GSEM; Rabe-Hesketh, Skrondal, and Pickles 2004). Afterward, I present an exemplary application of the previously proposed methods. In the application, I investigate the influences of tertiary students’ educational motivations on their preferences regarding internships. The final section concludes and discusses the implications of this study for future research using FSEs.

Testing the Interval Scaling Assumption in FSEs

Methods for Testing the Interval Scaling Assumption

Measurements are interval scaled if not only their ranking but also their distance on a scale is meaningful. “Commonly,…category ratings are supposed to lead to interval scales…based on the assumption that respondents judge according to perceived stimulus differences but there is no direct approach to testing whether these differences allow the construction of an interval scale” (Orth and Wegener 1983, p. 418f). To address this problem, tests of the interval scaling assumption use a rank order of subjective paired stimuli differences to be measured empirically in addition to the subjective single stimulus category ratings (Krantz et al. 1971; Wegener 1983). Thus, in the case of FSEs, not only category ratings C of single vignettes are measured (e.g., vignettes called a, b, c, and d) but also ratings D of differences between pairings of these single vignettes (e.g., vignette pairings called ab, ac, bc). In general, techniques using such an additional set of observations for interval scaling tests are called conjoint measurement.²

If the rank orders of the single vignette ratings C and paired vignette ratings D satisfy the two conditions of an algebraic difference structure, that is, the quadruple and the compatibility condition, then the category ratings form an interval scale (Krantz 1972; Krantz et al. 1971; Orth 1982).

First, the quadruple condition implicates weak monotonicity and states that,

if D (a b) ≿_{R} D (c d), then D (a c) ≿_{R} D (b d) .

The relation $≿_{R}$ denotes that the ranking of the prior mentioned rating is subjectively greater than or equal to the subsequently mentioned rating. For example, $D (a b) ≿_{R} D (c d)$ states that the ranking of the rating of the differences between vignettes a and b is greater than or equal to the ranking of the rating of the differences between vignettes c and d. In consequence, if the quadruple condition (1) holds for all ratings D of differences between vignette pairings, then these pairings can be ranked without contradictions.

Second, the compatibility condition postulates:

D (a b) ≿_{R} D (c d) if and only if C (a) - C (b) \geq C (c) - C (d) .

Here, $C (a) - C (b) = \hat{C} (a b)$ denotes the calculated numerical difference between the two ratings of the single vignettes a and b. Therefore, if the compatibility condition (2) holds for all paired ratings D and calculated numerical differences $\hat{C}$ based on a set of single vignette ratings C, then the ranking of the ratings of the differences between the vignettes coincides with the ordering of the calculated numerical differences between the single vignette ratings.

If conditions (1) and (2) hold for all stimuli, then the ratings C and D form an algebraic difference structure which implies that these measurements are unique up to positive linear transformations and, thus, are interval scaled. Specifically, the interval scaling property states that the calculated numerical differences $\hat{C}$ of the single vignettes and the ratings D of the differences between the vignette pairings are linear additive related by a scaling parameter $u > 0 \in ℝ$ and a shifting parameter $v \in ℝ$ , that is:

\hat{C} = u \times D + v .

u > 0 implicates that the rated differences between the vignette pairings D are predictive of the distances $\hat{C}$ between the category ratings of the respective vignettes. So, these distances are informative, and therefore, the category ratings C constitute an interval scale.

Given respondents’ additional ratings of differences between vignette pairings D, both, conditions (1) and (2), as well as property (3), are empirical testable on the respondent level i. The quadruple condition (1) is typically assessed by calculating the proportion of violations qⁱ ∈ℝ[0; 1] over the set of all vignette pairings. If the quadruple condition (1) is satisfied, then qⁱ = 0. The compatibility condition (2) is evaluated by calculating the Kendall’s (1938) $τ_{b}^{i}$ rank correlation coefficient (Kendall’s τ corrected for ties) ∈ ℝ[−1; 1] between the ranking of the paired vignette difference ratings D and the ranking of the calculated numerical differences $\hat{C}$ between the single vignette ratings.³ If the compatibility condition (2) holds, then $τ_{b}^{i}$ = 1. Also, the quadruple condition (1) is a precondition for the compatibility condition (2) to hold, thus, if qⁱ > 0, then $τ_{b}^{i}$ < 1. The interval scaling property (3) is indicated by the Pearson’s correlation coefficient rⁱ between $\hat{C}$ and D, which is a standardized estimate of u. If D predicts $\hat{C}$ perfectly, then rⁱ = 1. Moreover, $τ_{b}^{i}$ < 0 or rⁱ < 0 shows that the differences in $\hat{C}$ and ratings in D tend to be counterrotating, that is, are not even similar in direction. Since $\hat{C}$ and D are based on the same stimuli, they should be similar directed, and thus, $τ_{b}^{i}$ < 0 or rⁱ < 0 is an indication that these measurements are not valid. Next, I discuss expectations regarding tests of compliance with the interval scaling assumption in FSEs based on these indicators.

Expectations Regarding Interval Scaled Response Behavior in Factorial Surveys

In social scientific research applications such as FSEs, it is likely that the measurement of qⁱ , $τ_{b}^{i}$ , and rⁱ is affected by systematic nonsampling survey error as well as by random error.⁴ Specifically, in such applications, qⁱ , $τ_{b}^{i}$ , and rⁱ are influenced not only by study design features leading to measurement error but also by respondent characteristics and attributes of the measurement situation which can produce response error. Formally, qⁱ , $τ_{b}^{i}$ , and rⁱ are functions of the study design, the characteristics of respondents, the measurement situation, and a respondent specific random error component. All of these factors cause qⁱ , $τ_{b}^{i}$ , and rⁱ to deviate from their ideal limiting values of 0 and 1, respectively. In consequence, it has to be expected that the average $τ_{b}^{i}$ and rⁱ are smaller than 1 and the average qⁱ is larger than 0 in FSEs and, thus, that the interval scaling assumption is violated. Figure 1 illustrates the stylized cumulative distribution function (CDF) of rⁱ or $τ_{b}^{i}$ for a FSE resulting from these expectations.

Figure 1.

Stylized cumulative distribution function of rⁱ or $τ_{b}^{i}$ for a factorial survey experiment.

Past research supports these expectations regarding qⁱ , $τ_{b}^{i}$ , and rⁱ . Two laboratory studies on evaluations of line sizes with 64 subjects (Orth 1982) and 18 subjects (Wegener 1983) both resulted in a mean rⁱ of .97 (SD = .02) and a mean $τ_{b}^{i}$ of .75 (SD = .10) with small deviations depending on the number of categories of the response instrument used. The related qⁱ ’s were estimated to be 3 percent and 9 percent (SD = 2 percent and 3 percent). Furthermore, an interviewer-assisted test on evaluations of occupational prestige with 46 participants (Orth and Wegener 1983; Wegener 1983) showed a mean rⁱ of .82 (SD = .28) and a mean $τ_{b}^{i}$ of .55 (SD = .20) for a rating instrument with 20 categories as well as a mean rⁱ of .87 (SD = .19) and a mean $τ_{b}^{i}$ of .63 (SD = .15) for a rating instrument with 9 categories. Here, the respective qⁱ was estimated to be 8 percent (SD = 5 percent).

I expect a lower level of compliance with the interval scaling assumption for FSEs in comparison to the studies cited above for two reasons. First, compared to the one-dimensional evaluation task of line sizes and occupational prestige in previous applications, FSEs are multi-dimensional, more complex evaluation tasks. Second, in comparison to these previous studies, FSEs are mostly conducted without noteworthy prior training for the rating tasks and often also in field settings without interviewer assistance, for example, as an online study (Weinberg, Freese, and McElhatten 2014). In the following subsection, I outline a design to test these expectations regarding the interval scaling assumption in an FSE.

Implementing an Interval Scaling Test in a Factorial Survey

In principal, an interval scaling test for a FSE can be set up by constructing vignette pairings based on the vignettes used as stimuli for the experiment and have respondents rate the differences between these vignette pairings in addition to the single vignettes. Practically, the large number of vignette pairings per respondent needed for an assessment of the quadruple condition (1) makes testing it barely feasible for most survey designs.⁵ Formally, given a set of vignettes per respondent of size A, the related set of vignette pairings B is constituted by the lower triangle of the squared matrix given by the Cartesian product of A. Hence, the size of B is (A ² − A)/2. For example, B consists of 45 vignette pairings if A contains 10 vignettes. Moreover, the set of vignette pairings B contains many repetitions of the same single vignettes which can annoy or irritate respondents.

The structure of the interval scaling test described above indicates a possible solution. The quadruple condition (1) is a necessary precondition for the compatibility condition (2) and the interval scaling property (3) to hold (Orth 1982). Therefore, compared to the interval scaling assumption—stating that both conditions (1) and (2) are satisfied—presuming consistency with only the quadruple condition (1) is a weaker postulate. Assuming consistency with the quadruple condition (1), drastically reduces the workload of an interval scaling test for respondents. For a test of (2) and (3) under this assumption, a subset of paired vignette difference ratings which is large enough to calculate $τ_{b}^{i}$ and rⁱ is sufficient. For this study, I constructed 8 vignette pairings based on 16 single vignettes per respondent (see Design and Data Used subsection). Furthermore, I only used each of the single vignettes once for the construction of the vignette pairings to minimize annoying or irritating repetitions within respondents’ vignette sets. Since respondents have to rate the differences between the vignette pairings in addition to the single vignettes, I call this type of factorial survey setup a paired differences rating (PDR) design.

In contrast to other FSE designs, the PDR design provides information regarding the degree to which vignette ratings match the interval scaling assumption on the respondent level. The results of the other applications described above suggest that this assumption will be violated in FSEs. To address this problem, I develop a method to analyze FSE data that incorporate information about the scaling level of vignette ratings in the next section. Utilizing such information facilitates more robust and differentiated analyses of FSEs.

A Model for Scaling Sensitive Factorial Survey Analysis

Standard and Ordinal Factorial Survey Analysis

Nowadays, it is routine practice to analyze FSE data in which respondents rate several vignettes y^v using linear random intercept models, that is, a linear regression model with a normal distributed random parameter ε ⁱ ∼ N(0, σ²) and cluster-robust standard errors on the respondent level (Auspurg and Hinz 2015; Hox et al. 1991; Jasso 2006).⁶ Therefore, I call this approach standard factorial survey analysis (SFSA). The cluster-robust standard errors account for the within respondent dependence between ratings in statistical tests. This dependence is induced by the hierarchical data structure in which vignettes are nested within respondents, that is, the cluster-robust standard errors address the violation of the assumption that the vignette ratings are independent and identical distributed (iid). The respondent-level random component controls for normal distributed unobserved heterogeneity in the mean vignette ratings between respondents.⁷ Accounting for such unobserved heterogeneity in FSEs is important if not only the effects of the experimental treatments implemented in the vignettes on the ratings but also the effects of differences between respondents—or of interactions between such differences and the treatments—on the vignette ratings are the focus of the analyses. A model for SFSA is given by:

y^{v} = α + X^{v, i} β + ε^{i} + ε^{v} .

Here, α is a fixed overall intercept parameter, X^v ^{, i} is a matrix of exogenous variables on the vignette level v and the respondent level i, β is a vector of the respective fixed parameters, and ε ^v ∼ F(0, σ²) is a random error component with a fixed variance parameter on the vignette level.

A central assumption of SFSA is that vignette ratings are interval scaled. Violations of this assumption call into question the linear additive structure of equation (4) and, hence, the use of SFSA. A way to handle FSE data which violate the interval scaling assumption, that is, data including a relevant share of respondent with $τ_{b}^{i}$ < 1 or rⁱ < 1, is to analyze it using an ordered logistic regression model with a normal distributed random error parameter ζⁱ ∼ N(0, σ²) and cluster-robust standard errors on the respondent level (Hedeker 2008; Hox 2002). As this approach assumes that the vignette ratings are ordinal scaled, I call it ordinal factorial survey analysis (OFSA). A model for OFSA is specified as:

y^{v} \leq invlogit (α_{c} - (y * = X^{v, i} γ + ζ^{i} + ζ^{v})) .

Here, α _c is a vector of fixed parameters for the c − 1 thresholds separating the c ordered outcome categories of y^v , y* are the values of y^v on the logit scale estimated by the model, γ is a vector of the exogenous variables’ fixed parameters, and ζ ^v ∼ SL(0, π²/3) is a standard logistic distributed random error component on the vignette level. OFSA renders the interval scaling assumption unnecessary for analyzing FSE data and it can be implemented as robustness check for SFSA even if information on the scaling level of the vignette ratings is unavailable (Auspurg and Hinz 2015). But with knowledge about the degree to which the interval scaling assumption holds, for example, through the calculation of rⁱ and $τ_{b}^{i}$ based on a PDR design, OFSA is not an efficient modeling strategy. It ignores the information on the distances between the sufficiently interval scaled vignette ratings and, similar to SFSA, potentially includes nonvalid, that is, in this case not even ordinal scaled, ratings in the analysis. Thus, a model incorporating information on the interval scaling level of vignette ratings is desirable.

Scaling Sensitive Factorial Survey Analysis

Given indicators of the interval scaling level of vignette ratings like rⁱ or $τ_{b}^{i}$ , implementing such a model is possible. In a first step, a minimum level r _min or τ_min for rⁱ or $τ_{b}^{i}$ to consider the ratings of respondents valid is defined. Regarding the specification of r _min or τ_min, it is plausible to set the minimum level to zero as rⁱ < 0 or $τ_{b}^{i}$ < 0 indicates that some of the vignette ratings are not even similarly directed and, hence, are not valid. After defining r _min or τ_min, one can estimate equation (4) or (5) conditional on this threshold for rⁱ or $τ_{b}^{i}$ . Despite reducing the respective FSEs’ sample size, such a conditional analysis can be expected to improve inferences because it excludes vignette ratings which are not valid. But such a SFSA or OFSA conditional on r _min or τ_min still disregards the information on the scaling level of the vignette ratings included in the analysis.

To address this problem, I extend the afore-described conditional modeling strategy by conceptualizing a weighting scheme for FSE data including information on the interval scaling level of vignette ratings. For clarity, I restrict the following part of the presentation to analysis using rⁱ as an indicator of the interval scaling level of ratings, but an identical setup using $τ_{b}^{i}$ instead is indeed possible. Specifically, a model conditioned on r _min assigns a weight of zero to the vignette ratings with rⁱ < r _min and a weight of one to the vignette ratings with rⁱ ≥ r _min. Next, the additional information on the degree to which vignette ratings match the interval scaling assumption contained in r_i is incorporated in this weighting scheme. Therefore, I specify the rⁱ ≥ r _min part of the weighting scheme dependent on rⁱ in a way which assigns an increasing weight to ratings with a larger rⁱ . For example, such a specification is given by (rⁱ − r _min)/(1 − r _min). Hence, the complete weighting function w_r for this weighting scheme is defined by:

w_{r} = (r^{i} < r_{min} : 0, r^{i} \geq r_{min} : (r^{i} - r_{min}) / (1 - r_{min})) .

Given equation (6), a GSEM (Rabe-Hesketh et al. 2004) based on equations (4) and (5) can be set up. In such a GSEM, a weight of w_r that increases with the interval scaling level of the vignette ratings, indicated by rⁱ , is assigned to equation (4) and a weight of 1 − w_r that decreases with rⁱ is assigned to equation (5). Formally, such a GSEM is specified as:

y^{v} [w_{r}] | r^{i} \geq r_{min} = α + X^{v, i} [w_{r}] β + ε^{i} + ε^{v} and

y^{v} [1 - w_{r}] | r^{i} \geq r_{min} \leq invlogit (α_{c} - (y * [1 - w_{r}] | r^{i} \geq r_{min} = X^{v, i} [1 - w_{r}] γ + ζ^{i} + ζ^{v})) .

with [·] denoting a function assigning the respective weight to y^v , y*, and X^v ^, i.

A GSEM based on equations (7) and (8) facilitates the simultaneous estimation of a SFSA and an OFSA in a way that makes gradual use of the distance information contained in the vignette ratings dependent on the interval scaling level, indicated by rⁱ . Therefore, I call such a GSEM a model for scaling sensitive factorial survey analysis (SSFSA). GSEMs are typically estimated using a maximum likelihood algorithm. For the maximum likelihood estimation, ε ^v and ζ ^v are assumed to be normally distributed.

Importantly, over equations (7) and (8), the weight assigned to each vignette rating by equation (6) given rⁱ ≥ r _min sums to 1, that is, for each observation weight (7) + weight (8) = w_r + (1 − w_r ) = 1 holds. Thus, the sample size used for a SSFSA is the same as in the corresponding SFSA or OFSA conditional on rⁱ ≥ r _min. Additionally, specifying r _min = 0 gives a simple formulation for the weighting scheme: If r _min = 0, then equation (6) results in w_r = rⁱ for respondents with rⁱ ≥ 0. Moreover, if instead of defining w_r based on equation (6), w_r is set to 1, then equations (7) and (8) reduce to equation (4), and if w_r is set to 0, then equations (7) and (8) reduce to equation (5). In consequence, SSFSA is a generalization containing the models typically used to analyze FSE data, SFSA or OFSA, as borderline cases.

Implementation of a SSFSA

A central advantage of SSFSA is its’ ability to model comparisons of parameters determining the vignette evaluation process between an interval scaled and a noninterval scaled rating mode, that is, between β in equation (7) and γ in equation (8). This can be achieved by rescaling γ in equation (8) based on the ratio of σ, the residual standard deviation of ε ^v estimated in equation (7), and the residual standard deviation of ζ ^v in equation (8), which is fixed at π/√3. This ratio defines a scaling parameter λ = σ/(π/√3) = σ√3/π mapping the scale of equation (7) on the scale of equation (8). Using λ, z tests of hypotheses such as

β−λγ \neq 0,

can be conducted. Similar methods for testing parameter differences are used in comparing the results of different discrete choice experiments or combining stated and reveal preference data on choice processes (Louviere, Hensher, and Swait 2000; Swait and Louviere 1993).

In consequence, equation (9) can be used to test possible restrictions regarding the parameters to estimate in a SSFSA. Given that equation (9) is rejected for certain parameters, related constraints can be implemented by specifying β = λγ (Louviere et al. 2000) and estimating the GSEM based on equations (7) and (8) again including these constraints. To differentiate the two steps of this approach, I call the first estimation without parameter restrictions unconstrained SSFSA and the second estimation with parameter restrictions constrained SSFSA. To account for the uncertainty in the first step estimate of λ, I use a plausible value estimator for the second estimation step (Mislevy 1991).⁸

Furthermore, differences in the respondent-level random error parameters, ε ⁱ and ζ ⁱ , between the interval and the noninterval scaled rating mode can also be tested using equation (9). Therefore, the random error parameters are specified as ε ⁱ = βeⁱ and ζ ⁱ = γeⁱ with eⁱ ∼ N(0, 1) in the estimation of the GSEM based on equations (7) and (8). Afterward, equation (9) is calculated for the β and γ parameters corresponding to eⁱ . Moreover, the scaling parameter λ can be used to compare the threshold parameters α _c of the logit scale estimated by equation (8) with the respective fixed values implied by the interval scale in the estimation of equation (7). Such a threshold parameter is given by calculating α + λα _c .

Finally, since SSFSA is based on GSEM extensions using random slopes or mapping additional, hierarchical data structures can be implemented. Furthermore, different specifications for the link functions of y^v and related error components within the overdispersed exponential family of probability distributions are possible. For example, y^v in equation (8) can be implemented as an ordered probit regression or y^v in equation (7) can be specified as a tobit regression (Tobin 1958). Thus, SSFSA is also compatible with other link functions used to address censoring in FSE data (Auspurg and Gundert 2015; Groß and Lang 2018).

In the following section, I present an exemplary application of a SSFSA based on a PDR design. I show the implementation of a SSFSA step-by-step and demonstrate that SSFSA results in improved inferences, that is, increased effect sizes, compared to SFSA.

Application: Students’ Internship Preferences and Educational Motivations

Design and Data Used

The substantive interest of the FSE presented in the following section is tertiary students’ preferences regarding internships, that is, work placements with a strong practical training aspect conducted while enrolled in tertiary education. Since the FSE was set up with the primary aim to test the PDR design and the model for SSFSA, a simple experimental set up was implemented: a full factorial set up with two 4-level and two 2-level dimensions.⁹ In a full factorial set up, all treatment factors (i.e., levels of dimensions) are covered by the measured stimuli (i.e., vignettes). Therefore, all main effects of the treatments, as well as all effects of interactions between them on the vignette ratings, are identified and the number of vignettes which cover all possible combinations of dimensions and levels (called vignette population or universe) is equal to the number of vignettes measured (called vignette sample). For the case of two 4-level and two 2-level dimensions, these are 4² × 2² = 64 vignettes. This vignette sample was randomly split into four groups (called decks) of 16 vignettes each for respondents to rate. To construct the additional paired vignettes for the PDR design, all single vignettes were used only once, resulting in four different decks of eight vignette pairings each. This strategy avoided additional repetitions of vignettes which could annoy or irritate respondents. Specifically, the vignette pairs were created by constrained random matching without replacement of the single vignettes in each deck.¹⁰

Study participants were randomly assigned to one of the decks and instructed to evaluate the single vignettes and also the vignette pairs for the PDR design in comparison to each other. The choice of the response instruments for the evaluations was guided by the research interest to test the interval scaling assumption in the context of a FSE. First, I picked the instrument most often used in FSEs: an 11-category rating instrument with range ℕ[−5, 5]. Eleven is the mode of the number of categories for rating instruments used in FSEs (Wallander 2009). Second, I used a 21-category rating instrument with range ℕ[−10, 10] which is the maximum number of categories used for rating instruments in FSEs, so far (Wallander 2009). The idea motivating the use of rating instruments with more categories is to enable respondents to make more fine-grained evaluations, potentially also facilitating a better match of these evaluations with the interval scaling assumption. I wanted to take this idea even further and hence, third, I implemented a response instrument allowing very fine-grained evaluations: a slider rating instrument with range ℕ[−1,000, 1,000]. All three instruments were bipolar and had a neutral midpoint at zero between the extremes of “very unattractive” and “very attractive” since this is a category rating design which is suitable to map a continuous preference which can also involve dislike of alternatives (i.e., internship vignettes). In such cases, using a unipolar response instrument mapping categories of “attractiveness” can result in skewed rating distributions because respondents cannot express a part of their preference continuum. Moreover, bipolar rating instruments are more readily applicable to the vignette pairings used in the PDR design. Here, participants have to calculate differences in “attractiveness” between two internships. A task which is easier if clearly preferring one of the two alternatives sets the poles of the scales. Respondents were randomly assigned to one of the three different response instruments. In addition, the ordering of the single vignette and paired vignette rating tasks was exchanged for a third of the respondents assigned to a 21-category or a slider rating instrument. Overall, this combination of an FSE with a split-ballot experiment resulted in five comparison groups (see Table 1). Moreover, the presentation order of the vignettes and vignette pairs, as well as the text order of the dimensions within a vignette, was randomized between respondents. Figure A1 in the Online Supplemental Material shows an example of the single vignettes and Figure A2 in the Online Supplemental Material is an example of the vignette pairings used for the PDR design. In addition, survey participants had to answer several psychological short scales as well as some questions regarding their studies, internship experiences, and social background.

Table 1.

Overview of the Experimental Design and Related Sample Statistics.

Instrument Used	Order of Rating Tasks		Excluded Cases	Number of Respondents	Mean \|r ^dim\|	Max. \|r ^dim\|	Mean y^v
11-Category rating	Single vignettes	Vignette pairings	4	37	.01	.02	104
21-Category rating	Single vignettes	Vignette pairings	14	63	.01	.02	143
Slider rating	Single vignettes	Vignette pairings	12	67	.01	.01	16
21-Category rating	Vignette pairings	Single vignettes	5	28	.01	.05	103
Slider rating	Vignette pairings	Single vignettes	3	30	∼0	.09	84
	Overall		38	225	.01	.02	86

Source: Own calculations.

Between April 15 and 26, 2015, 1,920 randomly selected tertiary students from a commercial sampling pool of about 10,000 participants covering different German tertiary institutions and fields of study were successively invited to participate in the study. About every other day—overall five times—384 invitations were posted. Each student was only invited once and no reminders were sent. The gross sample was restricted to tertiary students aged 35 or younger. Three hundred and fifty-three students started answering the questionnaire resulting in a response rate of about 20 percent. Forty-one of these starters were screened out for speeding while filling out the questionnaire. Furthermore, 49 of the remaining respondents dropped out before finishing the survey, thus resulting in a sample of 263 complete cases and a completion rate of about 85 percent, excluding the speeders. Thirty-eight of the complete cases were excluded from further analyses due to straight lining.¹¹ The median response time for the 225 remaining respondents was about 27 min. The sample statistics described in Table A1 in the Online Supplemental Material (Statistisches Bundesamt, 2014) show that this net sample is comparable to the population of tertiary students in Germany with respect to the type of tertiary institution attended, fields and years of study, and age.¹²

Table 1 gives an overview of the experimental design and the sample used for this study differentiated by instruments and rating task order. The maximum absolute correlations between the vignette dimensions (max. |r ^dim|) are .02 in the overall sample and .09 in the sample split by the different design specifications. The respective mean absolute correlations (mean |r ^dim|) are .01 in both cases. These correlations indicate that the randomization of the experimental conditions worked out correctly. To enhance comparability, the ratings of the instruments with 11 and 21 categories were rescaled to the range of the slider instrument (ℕ[−1,000, 1,000]). Figure A3 in the Online Supplemental Material shows the related vignette rating distributions differentiated by design conditions. The response range is covered well in all conditions and the distributions are only weakly negative skewed. The boundary values of the response range are used about 5–9 percent in all design conditions.¹³ The mean of the vignette ratings y^v is 86. In contrast, the mean y^v of 16 in the design condition with the slider rating instrument and the single vignette rating task first is significantly lower (z-value −3.24***). This difference persists in multivariate analyses controlling for other design factors, treatments of the FSE, and heterogeneity between respondents. It would be interesting to test whether this design effect can be replicated in other FSEs.

Operationalization and Methods Used

In the next subsection, I present the results of the interval scaling test based on PDR design using the respondent-level indicators rⁱ and $τ_{b}^{i}$ (see Testing the Interval Scaling Assumption in FSEs section). Afterward, in the next but one subsection, I present the results of a step-by-step implementation of a SSFSA. First, I carry out a SFSA and an OFSA. For the OFSA (equation (5)) and the ordinal logit model part of the SSFSA (equation (8)), the vignette ratings based on the slider and the 21-category rating instruments are combined into 11 categories similar to the 11-category rating instrument.¹⁴ Then, I estimate SFSAs restricted to rⁱ > 0 and $τ_{b}^{i}$ > 0. Next, I implement unconstrained SSFSAs with r _min = 0 and τ_min = 0 and w_r or w _τ calculated according to equation (6). Thereon, I test the differences between the parameters of equations (7) and (8) according to equation (9) on the 95 percent confidence level for both unconstrained SSFSAs and setup a related constrained SSFSA, which I estimate based on 20 plausible values for λ (see A Model for SSFSA section). In addition, I calculate the changes in effect sizes (Kelley and Preacher 2012) between the SSFSA and the SFSA. Here, I use the difference in standardized mean effects (Δz-value) between two models x and y scaled by the standardized mean effect (z-value) of model y in percentage as indicator of effect size changes: Δz-value = (z-value _x − z-value _y )/z-value _y × 100. Finally, I compare the threshold parameters α _c for the vignette rating scale in the ordinal logit model part of the SSFSA (equation (8)) with the respective assigned values of the interval rating scale used in the estimation of equation (7).

In all the models, I implement dichotomous indicators for the within respondent effects, that is, the treatment levels of the FSE.¹⁵ On the between respondent level, I use z-standardized sum indexes of external, internal, and altruistic motivations for tertiary studies based on two-item instruments each (Simeaner, Ramm, and Kolbert-Ramm 2014) as further explanatory variables.¹⁶ In addition, I incorporate different factors related to the interval scaling level of vignette ratings based on previous research (Lang 2016): An indicator of being a first-generation migrant and z-standardized sum indexes for the psychological constructs need for cognition, conscientiousness, and expressiveness (Rammstedt and Beierlein 2014) based on a four-, a two-, and a five-item instrument, respectively. Figure A4 in the Online Supplemental Material shows descriptive statistics for the variables used. Finally, I include indicators controlling the described design splits, vignette decks, and the random ordering of the vignettes as well as the vignette texts.

Results of the Interval Scaling Test

Now, I summarize the results of the interval scaling test for the FSE on tertiary students’ internship preferences described above. Figure 2 displays the CDFs of the indicators for the interval scaling level rⁱ and $τ_{b}^{i}$ in the sample of this FSE. $τ_{b}^{i}$ tests the compatibility condition (2) on the respondent level and rⁱ indicates the fit of the related vignette ratings with a metric scale based on equation (3). The sample mean of $τ_{b}^{i}$ is .34 (SD = .27) and that of rⁱ is .45 (SD = .33). The correlation between rⁱ and $τ_{b}^{i}$ is .83. Lang (2016) presents more detailed analyses of factors influencing the interval scaling level of respondents in this FSE. The shapes of the CDFs in Figure 2 and related sample statistics are in line with the expectations regarding interval scaling tests for FSEs summarized in Figure 1. The substantially smaller values of rⁱ and $τ_{b}^{i}$ than one indicate violations of the interval scaling assumption. In a standard FSE design, this problem would go unnoticed. Also as expected, the means of rⁱ and $τ_{b}^{i}$ are lower than in previous studies (Orth 1982; Wegener 1983). Plausible causes are the more complex rating task compared to the former applications but also the lack of prior training for the rating and the less standardized field situation of an online survey.

Figure 2.

Cumulative distribution functions of the interval scaling level indicated by rⁱ or $τ_{b}^{i}$ in the sample. Source: Own calculations (n _individuals = 225).

The results of this interval scaling test point out the potential utility of analyzing this FSE with a SSFSA. Related, 24 respondents in the sample have a rⁱ < 0 and 25 respondents a $τ_{b}^{i}$ < 0, that is, based on both interval scaling-level indicators, 11 percent of the sample are excluded in a SSFSA because their vignette ratings are not valid. Table A4 in the Online Supplemental Material shows that excluding these cases is not selective with regard to the variables used in the analysis. Conditional on rⁱ ≥ 0, the sample mean of rⁱ is .53 (SD = .23) and the respective statistic for $τ_{b}^{i}$ is .41 (SD = .22). Hence, given r _min = 0, the mean w_r assigned using equation (6) in the SSFSA based on r_i is .53. If the SSFSA is based on $τ_{b}^{i}$ instead, then the mean w _τ assigned given τ_min = 0 is .41.

Results of the SSFSA

In the following subsection, I describe the implementation of an SSFSA for the FSE on tertiary students’ internship preferences step-by-step and present related results. The results of the different analyses are summarized in Tables 2 and 3. The first column of Table 2 shows the results of the SFSA (equation (4)). First, we look at the effects of the experimental factors on the vignette evaluations. The indicators for a match of the internship with the content of the respondents’ tertiary studies and a hiring option subsequent to the internship, as well as all three payment related indicators, increase the ratings, are highly significant, and reach z-values > 10. The indicator for a nonprofit foundation as an employer also has a significant positive effect. Regarding the influences of between respondent differences on the vignette ratings, I find significantly higher ratings with z-values > 2 for students with a more altruistic motivation. In addition, I obtain comparable results, given vignettes with additional internship payment for students with more external motivation and given vignettes indicating a match between the studies and the internship for students with more internal motivation. Furthermore, the ratings are significantly lower for respondents with more need for cognition. In addition, the highly significant respondent-level intercept points to relevant unobserved heterogeneity between respondents with regard to the vignette ratings. The residual variance of 135,306 implicates an explained variance of 64 percent for the SFSA, indicating an overall good fit of the model with this FSE data.

Table 2.

Analyses of Tertiary Students’ Internship Preferences.

Model Type	SFSA	SFSA	Unconstrained SSFSA Based on rⁱ			Constrained SSFSA Based on rⁱ
Equation	(4)	(4)\|rⁱ ≥ 0	(7) Interval	(8) Ordinal	(9)	(7) and (8)	vs. (4); (4)\|rⁱ ≥ 0
Variable^a	β; z-value	β; z-value	β; z-value	β; z-value	β; z-value	β; z-value	Δz-value in %
Within respondent indicators:							Mean: 18; 23
p1: commuting exp. (ce)	155; 10.08***	149; 9.07***	138; 7.80***	137; 8.04***	1; 0.05	139; 11.22***	11; 24
p2: ce and €500 per month	581; 25.27***	577; 24.00***	550; 21.85***	552; 19.17***	−2; −.06	554; 29.99***	19; 25
p3: ce and €1,500 per month	861; 30.25***	868; 29.71***	848; 27.76***	868; 22.04***	−19; −0.39	861; 36.48***	21; 23
Match with tertiary studies	568; 22.66***	590; 23.74***	612; 22.60***	541; 16.56***	71; 1.64	582; 27.58***	22; 16
Hiring option	175; 11.97***	177; 11.20***	177; 10.44***	175; 9.75***	1; 0.06	177; 12.39***	19; 27
e1: private small business	9; 0.42	11; 0.45	−2; −0.10	14; 0.50	−16; −0.45	6; 0.31
e2: public agency	22; 1.03	8; 0.38	−15; −0.67	22; 0.90	−37; −1.12	2; 0.09
e3: nonprofit foundation	64; 2.70***	62; 2.58**	48; 2.12**	69; 2.45**	−20; −0.57	58; 3.18***	18; 23
Between respondent indicators:							Mean: 37; 34
External motivation	−33; −1.62	−34; −1.58	−33; −1.55	−39; −1.74*	6; 0.18	−36; −2.27**	41; 44
Internal motivation	−8; −.35	−8; −.35	−16; −.76	∼0; .01	−16; −.50	−8; −0.47
Altruistic motivation	39; 2.27**	32; 1.73*	46; 2.38**	11; 0.65	35; 1.37	30; 2.22**	−2; 28
External motivation × p2	49; 2.52**	48; 2.36**	44; 2.08**	59; 2.67***	−15; −0.49	51; 3.29***	31; 39
External motivation × p3	52; 2.08**	50; 1.87*	54; 1.93*	68; 2.05**	−14; −0.32	59; 2.76***	33; 48
Altruistic motivation × p3	−35; −1.60	−26; −1.19	−38; −1.90*	−7; −0.24	−29; −0.83	−23; −1.33
Internal motivation × match	59; 2.59**	55; 2.22**	48; 1.83*	59; 2.18**	−11; −0.28	56; 2.92***	13; 32
External motivation × e3	−33; −1.75*	−35; −1.78*	−38; −1.90*	−24; −1.10	−14; −0.47	−32; −2.11**	20; 18
First-generation migrant	−23; −0.29	34; 0.36	58; 0.59	37; 0.36	21; 0.15	48; 0.66
Need for cognition	−38; −2.47**	−39; −2.39**	−31; −2.28**	−50; −2.69***	19; 0.82	−41; −3.32***	34; 39
Conscientiousness	−8; −0.61	−7; −0.54	−7; −0.53	−5; −0.42	−2; −0.12	−7; −0.69
Expressiveness	19; 1.25	35; 2.23**	27; 1.79*	41; 2.30**	−14; −0.59	33; 2.81***	126; 26
Respondent-level intercept	173; 12.04***	170; 10.81***	136; 10.77***	141; 4.68***	−5; −0.15	144; 8.73***
Reference category^b	−711; −11.16***	−725; −11.15***	−735; −11.13***			−702; −13.26***
Residual variance	135,306	126,504	127,420	π²/3 × λ²		133,195
Scaling parameter λ			197; 46.21***			197; 46.21***
n _individuals; n _ratings	225; 3,600	201; 3,216	201; 3,216			201; 3,216

Source: own calculations.

Note: SFSA = standard factorial survey analysis; SSFSA = scaling sensitive factorial survey analysis. ^aControl variables: instrument, task order, deck, vignette order and text order indicators; ^bReference category: no payment, no hiring option, no match with studies, employer is a private large enterprise, no first-generation migrant, and all other indicators are fixed at their mean level rescaled to zero.

*P(Z > |z|) < .10.

**P(Z > |z|) < .05.

***P(Z > |z|) < .01.

The first column of Table 3 displays the results of an OFSA (equation (5)) based on the same data. This model implies no interval scaling assumption with respect to the vignette ratings. The estimated effects of the experimental treatments are quite similar to the SFSA and the related z-values are consistently slightly lower. Regarding the influences of between respondent differences, I do not find this general similarity. In contrast to the SFSA, the effect of students’ external motivation on the vignette evaluations is significantly negative and the positive effect of students’ altruistic motivation is only weakly significant. The other effects of between respondent differences mentioned above are robust across both specifications. Overall, the comparison between SFSA and OFSA demonstrates that the same conclusions regarding the influences of the experimental treatments can be drawn without the interval scaling assumption, but uncertainty exists regarding the main effects of students’ motivations on the vignette ratings.

Table 3.

Analyses of Tertiary Students’ Internship Preferences.

Model Type	OFSA	SFSA	Unconstrained SSFSA Based on $τ_{b}^{i}$			Constrained SSFSA Based on $τ_{b}^{i}$
Equation	(5)	(4)\| $τ_{b}^{i}$ ≥ 0	(7) Interval	(8) Ordinal	(9)	(7) and (8)	vs. (4); (4)\| $τ_{b}^{i}$ ≥ 0
Variable^a	β; z-value	β; z-value	β; z-value	β; z-value	β; z-value	β; z-value	Δz-value in %
Within respondent indicators:							mean: 14; 20
p1: commuting exp. (ce)	0.69; 9.01***	142; 8.74***	131; 6.97***	126; 7.75***	5; 0.20	129; 10.46***	4; 20
p2: ce and €500 per month	2.76; 21.68***	574; 24.12***	548; 20.37***	552; 19.82***	−4; −0.11	550; 29.31***	16; 21
p3: ce and €1,500 per month	4.39; 25.41***	862; 29.45***	858; 26.51***	866; 21.71***	−9; −0.17	863; 34.31***	13; 17
Match with tertiary studies	2.87; 19.51***	595; 23.65***	604; 21.73***	579; 16.93***	25; 0.55	589; 27.07***	19; 14
Hiring option	0.90; 11.94***	174; 11.11***	172; 10.43***	175; 9.82***	−3; −0.11	173; 13.80***	15; 24
e1: private small business	−0.02; −0.21	12; .51	3; 0.12	3; 0.12	∼0; −0.01	3; 0.18
e2: public agency	0.06; 0.55	16; 0.73	−14; −0.59	23; 1.00	−37; −1.12	7; 0.44
e3: nonprofit foundation	0.26; 2.24**	61; 2.58**	63; 2.71***	53; 2.09**	9; 0.28	57; 3.18***	18; 23
Between respondent indicators:							Mean: 37; 43
External motivation	−0.20; −1.97**	−27; −1.29	−40; −1.84*	−27; −1.28	−13; −0.42	−31; −2.01**	24; 56
Internal motivation	−0.01; −0.06	−23; −1.15	−21; −1.06	−23; −1.15	2; 0.06	−21; −1.46
Altruistic motivation	0.17; 1.87*	39; 2.24**	61; 3.36***	18; 1.13	44; 1.80*	37; 2.99***	32; 33
External motivation × p2	0.29; 2.94***	50; 2.44**	42; 1.90*	62; 2.90***	−20; −0.65	53; 3.42***	36; 40
External motivation × p3	0.36; 2.56**	50; 1.87*	62; 2.23**	59; 1.92*	3; −0.07	60; 2.77***	33; 48
Altruistic motivation × p3	−0.13; −1.03	−16; −0.73	−47; −2.07**	13; 0.46	−59; −1.67*	−13; −0.72
Internal motivation × match	0.30; 2.58**	52; 2.13**	53; 2.06**	55; 2.08**	−2; −0.06	55; 2.87***	11; 35
External motivation × e3	−0.16; −1.72*	−43; −2.19**	−35; −1.71*	−46; −2.09**	11; 0.35	−42; −2.70***	54; 24
First-generation migrant	0.08; 0.19	−81; −1.00	23; 0.27	−78; −1.04	101; 0.89	−41; −0.69
Need for cognition	−0.21; −2.50**	−30; −2.29**	−27; −1.89*	−33; −2.52***	6; 0.31	−32; −3.18***	29; 39
Conscientiousness	−0.06; −0.94	−6; −0.43	−10; −0.63	−3; −0.20	−7; −0.35	−6; −0.57
Expressiveness	0.12; 1.47	19; 1.31	28; 1.88*	22; 1.53	6; 0.31	23; 2.18**	75; 67
Respondent-level intercept	0.89; 9.14***	158; 15.33***	115; 8.25***	129; 9.18***	−14; −0.71	129; 13.35***
Reference category^b		−755; −11.72***	−752; −10.95***			−732; −14.00***
Residual variance	π²/3	129,145	133,488	π²/3 × λ²		133,278
Scaling parameter λ			201; 43.25***			201; 43.25***
n _individuals; n _ratings	225; 3,600	200 3,200	200; 3,200			200; 3,200

Source: Own calculation.

Note: OFSA = ordinal factorial survey analysis; SFSA = standard factorial survey analysis; SSFSA = scaling sensitive factorial survey analysis. ^aControl variables: instrument, task order, deck, vignette order, and text order indicators; ^bReference category: no payment, no hiring option, no match with studies, employer is a private large enterprise, no first-generation migrant, and all other indicators are fixed at their mean level rescaled to zero.

*P(Z > |z|) < .10.

**P(Z > |z|) < .05.

***P(Z > |z|) < .01.

Next, the second columns of Tables 2 and 3 show the results of SFSAs restricted to rⁱ ≥ 0 and $τ_{b}^{i}$ ≥ 0. For both restricted specifications, the effects of the experimental factors and the between respondent differences are comparable to the unrestricted SFSA. Despite the 11 percent smaller sample used for the restricted SFSAs, the z-values are similar in size. In contrast to the unrestricted SFSA, the influence of the interaction between students’ external motivation and vignettes with a high additional internship payment is only weakly significant for both restricted specifications, and that of students’ altruistic motivation is only weakly significant for the SFSA restricted to rⁱ ≥ 0. Furthermore, the effect of respondents’ expressiveness on the vignette ratings is significant in the restricted SFSA based on rⁱ ≥ 0. The overall stability of the effect sizes in the restricted specifications compared to the unrestricted SFSA confirms the expectation that the vignette ratings of respondents with rⁱ < 0 or $τ_{b}^{i}$ < 0 are not valid, that is, excluding these cases removes noisy data from the analyses. This is also reflected in the lower residual variances for the restricted SFSAs of 126,504 and 129,145 which imply an explained variance of 66 percent and 67 percent for these models, respectively.

Now, I come to the SSFSA. Columns 3–5 of Table 2 show the results of an unconstrained SSFSA based on rⁱ ≥ 0. Column 3 displays the results of the linear regression based on equation (7), column 4 those of the ordered logit regression based on equation (8), and column 5 those of tests of parameter differences between the two model parts based on equation (9). With two exceptions, the coefficients are quite similar across both model parts of this SSFSA. The exceptions are the influence of students’ altruistic motivation on the vignette evaluations and its’ interaction with vignettes including a high additional internship payment which are both only significant in the linear regression part. The first effect is also found in the restricted SFSA based on rⁱ ≥ 0 in column 2 of Table 2, while the second is not. For the latter, the not significant result in the ordered logit model part of the SSFSA is consistent with the OFSA in column 1 of Table 3. The overall stability across both parts of this SSFSA is confirmed by the tests of parameter differences. They do not indicate significant discrepancies between the coefficients. The scaling parameter λ mapping the parameters of the linear regression part on the ordinal logit model part is 197, that is, a change of 197 units on the interval rating scale of equation (7) equals a change of one unit on the logit scale estimated by equation (8). Columns 3–5 of Table 3 display the results of the related unconstrained SSFSA based on $τ_{b}^{i}$ ≥ 0 which are comparable to those just presented. The tests of parameter differences for this SSFSA result in only two weakly significant differing parameters between both parts of the model. One is the effect of students’ altruistic motivation and the other is this effects’ interaction with vignettes describing a high additional internship payment. Furthermore, the scaling parameter for the SSFSA based on $τ_{b}^{i}$ ≥ 0 is 201.

Given the similarity of parameters across model parts for both unconstrained SSFSA specifications, I fitted SSFSAs with equality constraints on all common parameters using a plausible value estimator. The results of these constrained SSFSAs are shown in the sixth columns of Tables 2 and 3. Related calculations of effect size changes in comparison to the SFSA (equation (4)) and the respective restricted SFSA specifications are displayed in the seventh columns. Regarding the effects of the experimental treatments on the vignette ratings, the conclusions drawn are comparable to the SFSA and restricted SFSAs for both constrained specifications. But the effect sizes of the significant coefficients in the constrained SSFSAs are consistently larger. For the constrained SSFSA based on rⁱ ≥ 0, the z-values increase on average by 18 percent compared to the SFSA and by 23 percent compared to the restricted SFSA. For the constrained SSFSA based on $τ_{b}^{i}$ ≥ 0, the respective mean increases are 14 percent and 20 percent. With regard to the influences of between respondent differences on the vignette evaluations, the conclusions drawn for some coefficients differ between the SSFSA specifications and the SFSA. In the SSFSA specifications, the negative effect of students’ external motivation is significant. Thus, in contrast to the SFSA, the SSFSA specifications enable more clear-cut conclusions regarding the influence of students’ motivations on the vignette ratings. Furthermore, the positive effect of respondents’ expressiveness on the ratings is significant in both SSFSA specifications compared to the SFSA. The average increases in effect sizes for the SSFSA specifications related to the coefficients of between respondent differences are larger than those associated with the experimental factors. Here, the z-values increase on average by 37 percent compared to the SFSA and by 43 percent compared to the restricted SFSA for the constrained SSFSA based on rⁱ ≥ 0. For the constrained SSFSA based on $τ_{b}^{i}$ ≥ 0, the respective mean increases are 37 percent and 34 percent. These larger z-values are again consistently found across all coefficients with one exception, the influence of students’ altruistic motivation in the SSFSA based on rⁱ ≥ 0 compared to the SFSA. Thus, the constrained SSFSA specifications lead overall to more robust conclusions regarding this FSE.

Finally, Figure 3 displays the estimated threshold parameters α _c in the ordinal logit part of the constrained SSFSA specifications in comparison to the respective values of the interval vignette rating scale. The results are quite similar for both constrained SSFSA specifications. The threshold parameters α _c are substantially larger than implied by the interval scale in the lower part of the scales’ range and a bit smaller than assumed in the upper part. Thus, the noninterval scaled vignette evaluations are less dispersed compared to the interval scaled vignette ratings. Especially, the strongly negative noninterval scaled evaluations are less negative than implied by the interval scale. Hence, in this application, violations of the interval scaling assumption in a SFSA mainly results in a misfit with the implied scale in the lower part of the range.

Figure 3.

Threshold parameters α _c for constrained scaling sensitive factorial survey analyses based on rⁱ and $τ_{b}^{i}$ . Source: Own calculations.

Discussion and Conclusion

In this article, I introduced a FSE design which enables testing the interval scaling assumption for vignette ratings, the PDR design, and a new model for FSE data, SSFSA. The PDR design conjointly measures single vignette ratings and evaluations of vignette pairings constructed using the same set of vignettes. The strength of positive correlation between the single and the paired vignette ratings indicates the fit of the responses with the interval scaling assumption. SSFSA uses these correlations as indicators for interval scaled response behavior. This new method has two advantages: First, it does not generally assume interval scaled vignette ratings and, second, it utilizes the respondent-level information on the match of the ratings with the interval scaling assumption to gain efficiency. I tested the PDR design and the SSFSA using a FSE on the influence of tertiary students’ motivations on their internship preferences. Regarding the interval scaling test, the results clearly demonstrate that the interval scaling assumption for the vignette ratings is violated in this application. Previous findings and theories about survey response behavior suggest that this outcome has to be expected for other FSEs, too. With respect to the SSFSA based on the PDR design, the results show that this method addresses this problem and, thus, facilitates more robust analyses of FSEs. The SSFSA yield on average 14–18 percent larger treatment effects than standard methods depending on the specification used. For the influences of between respondent differences on the vignette ratings, the gain in effect sizes of on average 37 percent is even larger. Moreover, based on the SSFSA, I compare the interval rating scale with the threshold parameters between the response categories estimated for the noninterval scaled vignette evaluations. Here, the results indicate that these evaluations mainly deviate from the interval rating scale in the lower part of the scale range.

In summary, the study shows that the efficiency gains achieved by SSFSA and differences in conclusions drawn based on SSFSA are larger for the effects of between respondent differences than for the influences of treatments. While replications have to demonstrate that the stability of treatment effects is not due to the comparatively simple FSE design used or the respondent population focused on in this study (see the following paragraphs), the results so far indicate that one has to be less concerned about violations of the interval scaling assumption in FSEs focusing solely on the treatment effects. In these applications, the additional efforts regarding data collection using a PDR design to implement a SSFSA might not be justified. But Wallander’s (2009) review shows that 82 percent of the FSEs also analyzed influences of between respondent differences, and additionally, 30 percent estimated interaction effects of treatments with between respondent differences. In this majority of cases, a diligent assessment of the interval scaling assumption based on a PDR design could be insightful. If such a design is considered too costly (e.g., due to the burden for respondents incurred by evaluating the paired vignettes), it can alternatively be implemented in a pretest or a subsample. Such an approach confines PDR designs to applications where they are necessary and at the same time advances the understanding of interval scaled response behavior. Specifically, future research could address the following limitations of this study.

The PDR design implemented in this study is restricted to an interval scaling test based on category rating instruments. This focus is based on common practice since magnitude estimation instruments are not as often used in FSEs (Auspurg and Hinz 2015). To widen this focus and apply the methods developed here to a FSE design using magnitude estimation instruments, especially in comparison to category rating instruments, would be a useful extension of this research. Not only the interval scaling test of Orth and Wegener (1983) but also the model for SSFSA developed in this article is readily transferable to FSEs using magnitude estimation instruments. Research in this direction could help to solve unsettled issues with respect to survey response behavior, at least for FSEs (Birnbaum 1980; Cools, Hofmans, and Theuns 2006; Hardin and Birnbaum 1990; Jasso and Wegener 1997; Liebig, Sauer, and Friedhoff 2015; Schaeffer and Bradburn 1989): To which degree are difference, ratio, and noninterval scaled rating behavior characteristics of respondents in comparison to study design features? And hence, to what extent can certain response modes be stimulated by instruction? Moreover, the development of a PDR design overcoming the practical hurdles to testing the quadruple condition assumption in FSEs would be a valuable extension of the current study.

Furthermore, the application in this study uses a comparably simple FSE, a full factorial with a vignette universe consisting of 64 vignettes. The full factorial setup excludes vignette sampling variably from the experimental design and thus enables this study to focus the analyses on testing the interval scaling assumption and evaluating the SSFSA. Also, the respondents used for this research were recruited using a sampling pool of tertiary students for online studies. Given this combination of a simple design with a comparably trained and educated sample of respondents, this application probably indicates fewer violations of the interval scaling assumption than typical FSEs. In a next step, it would be useful to test this expectation. Therefore, it would be necessary to implement a PDR design for an FSE with a vignette universe of the size used in typical applications, consisting of several thousand vignettes which necessitate the sampling of vignettes. Here, deliberate methods which facilitate the efficient construction of a vignette sample according to experimental design criteria like D-efficient sampling should be implemented (Dülmer 2007; Kuhfeld, Tobias, and Garratt 1994).

In addition, it would be informative to choose an application for this extension which could be applied to a general population sample. Using such a sample, it would be especially interesting to assess which kind of between respondent differences lead to different effect sizes and conclusions in an SSFSA compared to other methods of analysis. Are these only the mainly psychological factors focused on in this study which are partly also determining interval scaled response behavior? Or are dimensions of social stratification like age, education, and gender or cultural context involved? Methodological research on rating scales not explicitly addressing the scaling level of responses suggests that there are group differences in response styles by cultural contexts (e.g., Chen, Lee, and Stevenson 1995). From the perspective of many FSEs focusing on differences between groups of respondents’ knowledge about the effects of the scaling level of responses on the analysis, such group differences would be pertinent to assess the relevance of complying with the interval scaling assumption in their applications.

Related, in this study, the different SSFSA specifications indicate no or only a few weakly significant differences between the coefficients of factors affecting vignette ratings. But research on combined analyses of different discrete choice experiments suggests that this is not common (Louviere et al. 2000). Without replications, this result could also be a consequence of the specific FSE or sample of respondents used (see previous paragraph). A more general expectation regarding the results of an SSFSA is to find more substantially relevant determinants of vignette evaluations in an interval scaled compared to a noninterval scaled response mode since the violations of the interval scaling assumption in the latter hint to a cognitive overload regarding the vignette rating task. Therefore, I call this expectation overload hypothesis. Extensions of the current research like those described in the last paragraph could also test this hypothesis. Since constraints can be applied separately to each coefficient, the SSFSA proposed in this article is suitable for handling applications in which some parameters differ between an interval and a noninterval scaled response mode.

Moreover, the method of SSFSA can in principle be implemented using any indicator of the responses’ interval scaling level. For the interval scaling test used in this study, two respondent-level indicators based on correlation coefficients are plausible: first, rⁱ which indicates the fit of the distances between the measurements in the two sets of ratings used for the test and second, $τ_{b}^{i}$ which assesses the agreement between these two sets of ratings with the compatibility condition. rⁱ and $τ_{b}^{i}$ correlate .83, that is, they are strongly but not perfectly correlated. Thus, it would be helpful to develop criteria to decide which indicator to use in which situation.

Finally, the GSEM-based scaling sensitive analyses methods introduced in this study aid deal with noninterval scaled responses not only in FSEs but also in other applications using interval response scales. The only precondition is that information on the interval scaling level of responses is available. Therefore, I hope that this research draws the attention of social scientists to the potential gains of accounting more explicitly for the scaling level of survey and test responses in their analyses.

Supplemental Material

Supplemental Material, Online_Supplement_SMR799382 - Scaling Sensitive Factorial Survey Analysis

Supplemental Material, Online_Supplement_SMR799382 for Scaling Sensitive Factorial Survey Analysis by Volker Lang in Sociological Methods & Research

Footnotes

Acknowledgments

The author would like to thank Martin Groß, Stefan Liebig, Meike Han, Knut Petzold, and Marc Schwenzer, as well as the participants and organizers of the “Colloquium on Empirical Social Research” at the University of Konstanz in February 2016 and the participants and organizers of the “Colloquium on Macro-sociology” at the University of Tübingen in January 2016 for their helpful comments on draft versions of this article.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Supplemental Material

Supplemental material for this article is available online.

Notes

References

Allison

Paul D.

2001. “Multiple Imputation for Missing Data: A Cautionary Tale.” Sociological Methods and Research 28(3):301–09.

Anderson

Norman H.

1976. “How Functional Measurement Can Yield Validated Interval Scales of Mental Quantities.” Journal of Applied Psychology 61(6):677–92.

Atzmüller

Christiane

Steiner

Peter M.

. 2010. “Experimental Vignette Studies in Survey Research.” Methodology: European Journal of Research Methods for the Behavioral and Social Sciences 6(3):128–38.

Auspurg

Katrin

Gundert

Stefanie

. 2015. “Precarious Employment and Bargaining Power: Results of a Factorial Survey Analysis.” Zeitschrift für Soziologie 44(2):99–117.

Auspurg

Katrin

Hinz

Thomas

. 2015. Factorial Survey Experiments. Quantitative Applications in the Social Sciences 175. Thousand Oaks, CA: Sage.

Birnbaum

Michael H.

1980. “Comparison of Two Theories of “Ratio” and “Difference” Judgments.” Journal of Experimental Psychology: General 109(3):304–19.

Chen

Chuansheng

Lee

Shin-ying

Stevenson

Harold W.

. 1995. “Response Style and Cross-cultural Comparisons of Ranting Scales among East Asian and North American Students.” Psychological Science 6(3):170–75.

Cools

Walentina

Hofmans

Joeri

Theuns

Peter

. 2006. “Context in Category Scales: Is “Fully Agree” Equal to Twice Agree?” European Review of Applied Psychology 56(4):223–29.

Dülmer

Hermann

. 2007. “Experimental Plans in Factorial Surveys: Random or Quota Design?” Sociological Methods and Research 35(3):382–409.

10.

Groß

Martin

Lang

Volker

. 2018. “Why Citizens Oppose Inheritance Taxes – Even if They Do Not Have to Pay Them: Results of a Factorial Survey.” Zeitschrift für Soziologie (Journal of Sociology) 47(3):200–207.

11.

Hardin

Curtis

Birnbaum

Michael H.

. 1990. “Malleability of “Ratio” Judgments of Occupational Prestige.” American Journal of Psychology 103(1):1–20.

12.

Hedeker

Donald

. 2008. “Multilevel Models for Ordinal and Nominal Variables.” Pp. 237–74 in Handbook of Multilevel Analysis, edited by Leeuw

Jan de

Meijer

Erik

. New York: Springer.

13.

Hofmans

Joeri

Theuns

Perter

Mairesse

Oliver

. 2007. “Impact of the Number of Response Categories on Linearity and Sensitivity of Self-anchoring Scales: A Functional Measurement Approach.” Methodology 3(4):160–69.

14.

Hox

Joop J.

2002. Multilevel Analysis: Techniques and Applications. New York: Routledge.

15.

Hox

Joop J.

Kreft

Ita G. G.

Hermkens

Piet L. J.

. 1991. “The Analysis of Factorial Surveys.” Sociological Methods and Research 19(4):493–510.

16.

Jasso

Guillermina

. 2006. “Factorial Survey Methods for Studying Beliefs and Judgments.” Sociological Methods and Research 34(3):334–423.

17.

Jasso

Guillermina

. 1996. “Exploring the Reciprocal Relations between Theoretical and Empirical Work: The Case of the Justice Evaluation Function.” Sociological Methods and Research 24(3):253–303.

18.

Jasso

Guillermina

. 1978. “On the Justice of Earnings: A New Specification of the Justice Evaluation Function.” American Journal of Sociology 83(6):1398–419.

19.

Jasso

Guillermina

Rossi

Peter H.

. 1977. “Distributive Justice and Earned Income.” American Sociological Review 42(4):639–51.

20.

Jasso

Guillermina

Wegener

Bernd

. 1997. “Methods for Empirical Justice Analysis: Part 1. Framework, Models, and Quantities.” Social Justice Research 10(4):393–430.

21.

Johnson

F. Reed

Kanninen

Barbara

Bingham

Matthew

Özdemir

Semra

. 2006. “Experimental Design for Stated-choice Studies.” Pp. 159–202 in Valuing Environmental Amenities Using Stated Choice Studies: A Common Sense Approach to Theory and Practice, edited by Kanninen

Barbara

. Dordrecht, the Netherlands: Springer.

22.

Kelley

Ken

Preacher

Kristopher J.

. 2012. “On Effect Size.” Psychological Methods 17(2):137–52.

23.

Kendall

Maurice G.

1938. “A New Measure of Rank Correlation.” Biometrika 30(1/2):81–93.

24.

Krantz

David H.

1972. “Measurement Structures and Psychological Laws.” Science 175(4029):1427–35.

25.

Krantz

David H.

Duncan Luce

Suppes

Patrick

Tversky

Amos

. 1971. Foundations of Measurement: Volume I: Additive and Polynomial Representations. London, England: Academic Press.

26.

Kuhfeld

Warren F.

Tobias

Randall D.

Garratt

Mark

. 1994. “Efficient Experimental Design with Marketing Research Applications.” Journal of Marketing Research 31(4):545–57.

27.

Lang

Volker

. 2016. “Determinants of Interval Scaled Response Behavior in Factorial Surveys.” Bielefeld, Germany: Bielefeld University, Working Paper.

28.

Liebig

Stefan

Sauer

Carsten

Friedhoff

Stefan

. 2015. “Using Factorial Surveys to Study Justice Perceptions: Five Methodological Problems of Attitudinal Justice Research.” Social Justice Research 28(4):415–34.

29.

Little

Roderick J. A.

Rubin

Donald B.

. 1989. “The Analysis of Social Science Data with Missing Values.” Sociological Methods and Research 18(2-3):292–326.

30.

Louviere

Jordan J.

Hensher

David A.

Swait

Joffre D.

. 2000. Stated Choice Methods: Analysis and Applications. Cambridge, England: Cambridge University Press.

31.

Manski

Charles F.

1977. “The Structure of Random Utility Models.” Theory and Decision 8(3):229–54.

32.

McFadden

Daniel

. 1973. “Conditional Logit Analysis of Qualitative Choice Behavior.” Pp. 105–42 in Frontiers in Econometrics, edited by Zarembka

Paul

. New York: Academic Press.

33.

Mislevy

Robert J.

1991. “Randomization-based Inference about Latent Variables from Complex Samples.” Psychometrika 56(2):177–96.

34.

Orth

Bernhard

. 1982. “A Theoretical and Empirical Study of Scale Properties of Magnitude-estimation and Category-rating Scales.” Pp. 351–77 in Social Attitudes and Psychophysical Measurement, edited by Wegener

Bernd

. Hillsdale, NJ: Lawrence Erlbaum.

35.

Orth

Bernhard

Wegener

Bernd

. 1983. “Scaling Occupational Prestige by Magnitude Estimation and Category Rating Methods: A Comparison with the Sensory Domain.” European Journal of Social Psychology 13(4):417–31.

36.

Rabe-Hesketh

Sophia

Skrondal

Anders

Pickles

Andrew

. 2004. “Generalized Multilevel Structural Equation Modeling.” Psychometrika 69(2):167–90.

37.

Rammstedt

Beatrice

Beierlein

Constanze

. 2014. “Can’t We Make It Any Shorter? The Limits of Personality Assessment and Ways to Overcome Them.” Journal of Individual Differences 35(4):212–20.

38.

Rossi

Peter H.

Anderson

Andy B.

. 1982. “The Factorial Survey Approach: An Introduction.” Pp. 15–67 in Measuring Social Judgments: The Factorial Survey Approach, edited by Rossi

Peter H.

Nock

Steven L.

. Beverly Hills, CA: Sage.

39.

Schaeffer

Nora C.

Bradburn

Norman M.

. 1989. “Respondent Behavior in Magnitude Estimation.” Journal of the American Statistical Association 84(406):402–13.

40.

Simeaner

Hans

Ramm

Michael

Kolbert-Ramm

Christa

. 2014. Datenalmanach Studierendensurvey 1993–2010: Studiensituation und Studierende an Universitäten und Fachhochschulen nach Geschlecht. Arbeitsgruppe Hochschulforschung Heft 75. Konstanz, Germany: University of Konstanz.

41.

Statistisches Bundesamt. 2014. Bildung und Kultur: Studierende an Hochschulen: Wintersemester 2013/2014. Fachserie 11, Reihe 4.1. Wiesbaden, Germany: Statistisches Bundesamt (Federal Office of Statistics).

42.

Swait

Joffre D.

Louviere

Jordan J.

. 1993. “The Role of the Scale Parameter in the Estimation and Use of Multinomial Logit Models.” Journal of Marketing Research 30(3):305–14.

43.

Tobin

James

. 1958. “Estimation of Relationships for Limited Dependent Variables.” Econometrica 26(1):24–36.

44.

Wallander

Lisa

. 2009. “25 Years of Factorial Surveys in Sociology: A Review.” Social Science Research 38(3):505–20.

45.

Wegener

Bernd

, ed. 1982. Social Attitudes and Psychophysical Measurement. Hillsdale, NJ: Lawrence Erlbaum.

46.

Wegener

Bernd

. 1983. “Category-rating and Magnitude Estimation Scaling Techniques: An Empirical Comparison.” Sociological Methods and Research 12(1):31–75.

47.

Weinberg

Jill D.

Freese

Jeremy

McElhatten

David

. 2014. “Comparing Data Characteristics of an Online Factorial Survey between a Population-based and a Crowdsource-recruited Sample.” Sociological Science 1:292–310.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.31 MB

0.02 MB

0.28 MB