The Factorial Survey

Random Designs

In the literature, basically two proposals for generating simple random designs can be distinguished: In their introduction to the factorial survey, Rossi and Anderson (1982:40-41) recommend using a computer routine that generates the needed random samples from a variable list. The computer routine picks out a value (level) at random from the first vignette variable (characteristic), then a value of the second vignette variable, and so on through the last vignette variable. Each such cycle produces an additional vignette. The cycle is repeated until the number of vignettes for the specified set size has been reached. Thereafter, the whole routine has to be repeated until a vignette sample of the same set size has been produced for each respondent (simple random design with replacement). In a more recent introduction to the factorial survey, Jasso (2006:342-43) recommends drawing each vignette set randomly out of the fully crossed vignette universe, which makes it easy to draw simple random designs without replacement. Drawing without replacement might be preferable, especially for small vignette populations, since it guarantees that no respondent has to judge a vignette more than once.

The basic idea behind using simple random designs is to represent the vignette universe as accurately as possible with different vignette samples of the same set size. Each vignette set is, within the limits of sampling error, a reduced representative random sample of the complete vignette universe. Merging such random vignette samples again produces a random sample of the whole vignette universe. The combined vignette sample, however, has a much smaller sampling error than each individual vignette set (cf. Rossi and Anderson 1982:29-30). The higher the number of vignette sets, the closer the combined vignette sample approximates the central features of the fully crossed vignette universe: All main effects and interaction effects within the limits of a decreasing sampling error become asymptotically more orthogonal, and the levels within each vignette factor become asymptotically more balanced.

To what extent the estimators of a factorial survey can benefit from the smaller sampling error of the combined vignette sample, however, depends on the heterogeneity of the respondents’ answer behavior. Since respondents in general have to answer more than one vignette, the collected data are nested hierarchically: The answer behavior is embedded in the personal context of each individual respondent. For such hierarchically structured data, multilevel regression analysis (cf. Hox 2010; Raudenbush and Bryk 2002; Snijders and Bosker 2012) is the recommended choice. The mathematical equation system for a multilevel model with four vignette variables, no interaction effects, and completely heterogeneous answer behavior is given in the following (cf. also Hox 2010:11-13; Raudenbush and Bryk 2002:35; Snijders and Bosker 2012:74-75):

Vignette level (Level 1):

Y i j = β 0 j + β 1 j X 1 i j + β 2 j X 2 i j + β 3 j X 3 i j + β 4 j X 4 i j + r i j

1

Respondent level (Level 2):

β 0 j = γ 00 + u 0 j ,

β 1 j = γ 10 + u 1 j ,

β 2 j = γ 20 + u 2 j ,

β 3 j = γ 30 + u 3 j ,

β 4 j = γ 40 + u 4 j ,

where

From the equation for the respondent level (level 2), it becomes clear that by including the u-terms in the regression equation each respondent can have his or her own regression equation, that is, an individual β for the intercept as well as for each of the four slopes. These respondent-specific βs deviate by a respondent-specific u-term from their respective grand mean (the reported “grand mean” γ ˆ is a so-called empirical Bayes’s estimate or posterior mean). ⁵ Whether or not the estimated variance, denoted by τ ˆ , of an estimated u-term becomes significant is an empirical question (cf. also Snijders and Bosker 2012:43-44). The lower the differences between the estimators for the respondent-specific βs (i.e., the β ˆ s) and the estimator for a respective grand mean (i.e., the γ ˆ ) as captured by an estimated u-term, the lower the likelihood that the estimated variance component will become significant. ⁶ Now, the less variance components become significant, that is, the more homogeneous the answer behavior of the respondents, the more the estimators for a simple random design can benefit from the fact that the design is becoming asymptotically more orthogonal and more balanced within the limits of an increasingly smaller sampling error (cf. also Dülmer 2007:393-94). Only in the extreme case that none of the estimated variance components becomes significant and all u-terms can be dropped (fixed), that is, in cases where the multilevel regression is reduced to a simple ordinary least squares (OLS) regression, can the estimators for the simple random design fully benefit from the fact that the design is becoming asymptotically more orthogonal and more balanced (in contrast to multilevel analyses, OLS regression is estimated exclusively on the basis of the pooled data matrix).

At the other extreme, that is, in cases where the answer behavior of the respondents is completely heterogeneous as in equation (1), the grand mean γ ˆ (posterior mean) of the intercept as well as of all slopes is (strongly) based on the average of the respective βs estimated for each individual respondent on the basis of individual vignette sets (separate respondent-specific data matrices). Based on these considerations, it becomes clear that the statistical characteristics of each individual vignette set cannot be ignored. Randomly sampled vignette sets of relatively small sample size in particular, within the limits of the sampling error, sometimes not only show very high mutual correlations between vignette variables, but the levels of each vignette variable can also be very unbalanced. Based on such vignette sets, respondent-specific βs can only be estimated with a high standard error. In extreme cases, a vignette set might even contain a constant or a vignette variable (or an interaction term), which is a linear combination of other vignette variables. Such unsuitable vignette sets have to be excluded before the survey is carried out since otherwise it would no longer be at all possible to estimate all theoretically relevant βs for each individual respondent. Based on such considerations, it would seem to be desirable to search for designs that are restricted to vignette sets that try to optimize the precision (i.e., minimize the standard error) with which respondent-specific βs can be estimated. Quota designs offer such an alternative, compared with simple random designs.

Quota Designs

The basic idea behind quota designs is to represent the whole vignette universe with only one or relatively few different vignette samples which cover the central features of the vignette universe as accurately as possible. For this purpose, a complete knowledge of the statistical properties of the vignette universe is needed. In general, quota designs can be divided into the classical fractional factorial designs and the less well-known D-efficient designs (cf. Dülmer 2007:386). Both approaches are restricted to using the same vignette set for each participant. Fractional factorial designs are constructed by aliasing (cf. Alexander and Becker 1978:96-97; Gunst and Mason 1991:48; Steiner and Atzmüller 2006:126), that is, confounding main effects with higher order interaction effects within one vignette set. Since a perfectly aliased higher order interaction effect can no longer be separated statistically from the main effect (the correlation between the vignette variable and the interaction term is 1 or −1), one has to assume that the aliased interaction effect is negligible. If this is not the case, then the estimator for the main effect will be systematically biased. One of the main differences between fractional factorial design and D-efficient designs consists in the fact that by somewhat relaxing the classical requirement of orthogonality of vignette factors (including interaction terms that are assumed to have a nonnegligible impact), a D-efficient design can be found for every set size. Relaxing the requirement for orthogonality frequently allows the balance of a design to be improved, an aspect that is sometimes sacrificed by fractional factorial designs in order to preserve orthogonality (mutual uncorrelatedness between vignette variables of different vignette factors including nonnegligible interaction terms). Balance and orthogonality, however, are both important for optimizing the precision with which βs can be estimated (cf. Kuhfeld et al. 1994:545).

Further possibilities arise if different quota designs are combined. A confounded factorial design (cf. Atzmüller and Steiner 2010:132; Kirk 1995:587-664; Steiner and Atzmüller 2006:132-33) is a design where different fractional factorial designs are used for a survey. An advantage of using a confounded factorial design over a simple fractional factorial design is that by using more than one fractional factorial design, confounding interaction effects with set effects becomes possible. A higher order interaction term that has no variation within a vignette set of a fractional factorial design is perfectly aliased with a set effect. A set effect is the specific influence which can be caused by the context of the vignette set as a whole. The common effect of the higher order interaction effect and the set effect is captured by the intercept of a regression model estimated for the fractional factorial design. If the higher order interaction effect, for instance, of dichotomous effect-coded vignette variables is a positive constant (i.e., 1) within one fractional factorial design and a negative constant (i.e., −1) within another fractional factorial design, then including a set variable allows the common effect of the higher order interaction effect and the set effect to be captured across the two fractional factorial designs. Disentangling both effects statistically, however, is impossible because in this case the higher order interaction effect is perfectly confounded with the set effect. Furthermore, if a main effect of a vignette variable is perfectly positively aliased with an interaction effect between different vignette variables within one fractional factorial design, and perfectly negatively aliased with the same interaction effect within another fractional factorial design, then the common impact of the perfectly confounded interaction effect and the set effect can be captured by estimating a two-way interaction effect between the respective vignette variable on the one hand and the set variable on the other hand. By controlling for the set variable and the two-way interaction term between the vignette variable and the set variable, the estimated intercept, as well as the estimated main effect of the vignette variable are protected against potential systematic bias that otherwise might result from a chosen design. Finally, one might also opt for using a confounded D-efficient design. Choosing a simple D-efficient design over a simple random design is done with the intention to optimize the precision (by preserving the unbiasedness) with which respondent-specific βs of the vignette variables (including nonnegligible interaction effects) can be estimated. Combining the vignette sets of a simple D-efficient design, in contrast to a simple random design however, neither improves the balance of the levels of each vignette variable nor does it reduce mutual correlations between vignette variables (cf. also Dülmer 2007:393). This weakness of simple D-efficient designs can be overcome by using a confounded D-efficient design, which consists of different simple D-efficient designs of the same D-efficiency. Existing imbalances as well as correlations among vignette variables (including nonnegligible interaction terms) are leveled off for the confounded D-efficient design across different vignette sets. In this way, the combined confounded D-efficient design covers the central features of the fully crossed vignette universe better than the simple D-efficient design: The D-efficiency of the combined confounded D-efficient design is higher than the D-efficiency of a comparable, simple D-efficient design and it is at least as D-efficient as a combined simple random design. To what extent the estimators of a confounded D-efficient design can be improved by the comparably higher D-efficiency of the combined design, however, depends again on the heterogeneity of the respondents’ answer behavior: The higher the differences between respondent-specific β ˆ s, that is, the more heterogeneous the answer behavior of the respondents (the more estimated u-terms become significant), the less the estimators of a confounded D-efficient design benefit from the fact that the combined confounded D-efficient design better covers the central features of the fully crossed vignette universe than a simple D-efficient design. Heterogeneous answer behavior does on the other hand affect the estimators of the simple random design, mostly since in this case the individual vignette set has on average the lowest D-efficiency of all designs.

D-efficient designs

The use of quota designs that have been discussed up to now is restricted by mathematical rules of divisibility applied to the number of levels per vignette variable. Sometimes no fractional factorial design might exist for a reasonable maximum number of vignettes per respondent. By relaxing the classical requirement of perfect orthogonality, a D-efficient design can be generated for every set size.

The reason for modifying the classical criteria for constructing quota designs is, according to Kuhfeld et al. (1994:545), that orthogonality is only a secondary goal that has to be subordinated to the primary goal of minimizing the standard error of the parameter estimates. Since balanced orthogonal designs allow all effects (intercept, main effects, and nonnegligible interaction effects) to be estimated uncorrelated, they represent the vignette universe most adequately. With such designs as reference, D-efficiency is a standardized measure that takes into account orthogonality and balance. The formula for D-efficiency (cf. also Kuhfeld et al. 1994:547) is given by:

D − e f f i c i e n c y = 100 ⋅ 1 N D ⋅ ( X ′ ⋅ X ) − 1 1 p = 100 ⋅ 1 N D ⋅ X ′ ⋅ X 1 p ,

2

where N_D denotes a design’s set size, X ′ ⋅ X denotes the information matrix of the vignette variables including the intercept, and p denotes the number of βs including the intercept that will be estimated. If all vignette variables are standardized orthogonally contrast coded (cf. Kuhfeld 2010:74), then D-efficiency is in the range from 0 to 100 (cf. Kuhfeld et al. 1994:547, 549). Every deviation from balance or from orthogonality produces values lower than 100. Classical fractional factorial designs sacrifice balance to preserve orthogonality in cases where no balanced orthogonal design exists. Under the same conditions, search algorithms for D-efficient designs try to find an optimally efficient solution between perfect orthogonality and balance. For this reason, D-efficient designs most of the time deviate at least slightly from orthogonality.

D-efficiency measures the goodness of a selected design relative to a balanced orthogonal design. For designs where only qualitative variables are included, a D-efficiency of 100 at least provides a rough reference for the generated design, even if such a design may be far from being possible for a given research question. If quantitative variables that are not standardized orthogonally contrast coded have to be included, then D-efficiency is no longer restricted to a maximum of 100. A design comparison, however, is possible as long as the same coding is used (cf. Kuhfeld et al. 1994:548-49). Since increasing the set size does not necessarily result in higher or at least the same D-efficiency (i.e., D-efficiency does not follow a monotone rising function), it is recommended to compare designs with different set sizes (cf. Dülmer 2007:394).

Generating D-efficient designs requires computer programs like SAS (ADX “Optimal Design,” PROC OPTEX, or the Macro %MktEx) or the conjoint value analysis (CVA) module of Sawtooth Software. ¹⁰ Since nonexhaustive search algorithms are used, a computer program may fail to find the optimal design, even if the search algorithm is carried out several times (cf. Kuhfeld et al. 1994:547; Sawtooth Software 1997–2002:7-10). Repeated searches are therefore recommended, especially for complex designs. D-efficiency will be computed automatically by the programs mentioned above.

Applicability

After introducing the basic ideas and main variants of random and quota designs, it is important to discuss, at least in short, the applicability of quota designs in cases where the vignette universe includes logically impossible combinations of vignette characteristics. Logically impossible combinations, for instance, will generally exist when education and occupation are important characteristics of a described fictitious vignette person (cf. Alves and Rossi 1978:545; Jasso and Rossi 1977:642; Nock 1982:104). Since a minimum level of education is required to be qualified for certain occupations, one has to remove vignettes with logically impossible combinations from the fully crossed vignette universe (cf. Jasso 2006:343). Excluding such vignettes from simple random designs in general only increases the correlations between affected variables, but by doing so, fractional factorial designs generally also lose orthogonality between affected variables as well as frequently between affected and unaffected variables (cf. Dülmer 2007:390). Hence, fractional factorial designs and confounded fractional factorial designs would not be applicable any longer. Choosing a D-efficient design might under such conditions be a viable alternative. Although excluding logically impossible combinations will always reduce the D-efficiency of a chosen design, the loss will sometimes be very small for D-efficient designs (cf. Kuhfeld et al. 1994:551). Furthermore, the existence of logically impossible combinations might frequently also prevent a suitable confounded D-efficient design from being found.

Reliability

Designs that allow unstandardized regression coefficients to be estimated with a lower standard error and for that reason with a higher precision produce ceteris paribus more reliable results than other designs. Therefore, the key to understanding the reliability of the results is the formula for estimating the standard error of β ˆ s in multiple OLS regression analysis (cf. Fox 1991:7-8; Thome 1990:166-67):

σ ˆ β ˆ 1 = ∑ i = 1 n e i 2 ∑ i = 1 n e i 2 n − k − 1 n − k − 1 ∑ i = 1 n X 1 i − X ˉ 1 2 ⋅ 1 − R X 1 ; X 2 , X 3 … X k 2 ,

3

where

σ ˆ β ˆ 1 is the estimated standard error of the estimated unstandardized regression coefficient of X ₁,

Besides the reliability of a respondent’s answer behavior (observed error variance), a β ˆ ’s estimated standard error depends on three components that are under the control of a researcher: These are the number of the remaining degrees of freedom for estimating the respondent-specific error variance, the variation of the vignette variables, and the multiple coefficient of determination among the vignette variables.

The higher the set size, the higher the number of degrees of freedom for estimating the error variance for a given number of vignette variables and the lower the estimated standard error for a respondent-specific β ˆ . So a higher set size contributes to increased reliability. On the other hand, it might also contribute to fatigue in respondents (cf. also Jasso 2006:349), which reduces the reliability of the answers and as a result increases the estimated error variance. This trade-off relationship has to be taken into account before generating the vignette samples. An advantage of simple random designs and D-efficient designs over fractional factorial designs and confounded factorial designs is that they exist for every set size. ¹¹

The remaining two components are functions of the two factors in the denominator of equation (3): The higher the variation of the vignette variables and the lower the coefficient of multiple determination among the vignette variables, the lower ceteris paribus the estimated standard error of a respective β ˆ . For the simple random design, the variation of the vignette variables as well as the coefficient of multiple determination differ from vignette set to vignette set. Hence, the β of a vignette variable can be estimated for each vignette set only with different standard errors and for that reason only with different reliability. As a consequence, part of the β ˆ ’s observed variation across respondents results exclusively from using different vignette sets (sampling variation) and cannot be traced back to substantially different answer behavior (cf. Hox, Kreft, and Hermkens 1991:501). Standardizing the vignette samples, as in fractional factorial designs or D-efficient designs, has the advantage that it eliminates this source of random error. This is especially important for small set sizes where the βs will generally be estimated with high standard errors.

If for a given set size no balanced orthogonal design exists, then D-efficiency will be maximized for fractional factorial designs exclusively with respect to orthogonality. Perfect orthogonality reduces the multiple coefficient of determination among the vignette variables to zero, whereby the right-hand factor in the denominator of equation (3) reaches its maximum of 1. Under the same conditions, search algorithms for D-efficient designs try to optimize both factors of the denominator simultaneously, whereby the request for perfect orthogonality is relaxed. Consequently, it becomes clear that for a given set size fractional factorial designs as well as D-efficient designs allow the individual βs to be estimated with higher reliability than an average set of a simple random design.

The comparisons up to now have focused on individual vignette samples and not on the combined vignette sample. Adding the same fractional factorial designs or the same D-efficient designs changes neither the degree of balance nor the correlations among the vignette variables. Hence, each quota set will show the same D-efficiency as the combined sample of all used vignette sets. The situation changes if different unbalanced fractional factorial designs, different D-efficient designs, or different vignette sets of a simple random design are added. If for the unbalanced variables (factors) of an unbalanced fractional factorial design the assignment between a vignette characteristic and its numerical representation (indicator variables for the nominal scaled variables) is permuted in order to generate further designs, then the combined vignette sample can also reach the maximal D-efficiency of 100 (for designs exclusively including qualitative variables). This requires, however, that each vignette set is judged by the same number of respondents. The same procedure can also be applied to D-efficient designs, although the correlations among the vignette variables make the task in this case somewhat more demanding. Furthermore, adding different vignette sets of a simple random design at least asymptotically increases the balance and uncorrelatedness of the combined sample. Therefore, with an increasing respondent number, the D-efficiency of the combined vignette sets asymptotically approaches 100 within the limits of the sampling error. The only design that for each individual vignette set, as well as for the combined sample of all completely judged vignette sets, possesses the maximal D-efficiency of 100 remains, however, the balanced fractional factorial design (the same, of course, also applies to a confounded factorial design consisting of different balanced fractional factorial designs). From this point of view it is an ideal design that produces the most reliable results.

How much the estimators of a factorial survey can benefit from the fact that the combined vignette sample (pooled data matrix) of a random or a confounded design have a higher D-efficiency than a single vignette set depends on the heterogeneity of the respondents’ answer behavior. Heterogeneity means that the respondent-specific estimated βs significantly differ across the respondents. Such unexplained context effects are modeled in multilevel analysis by including random terms that capture respondent-specific deviations from the grand mean of the intercept or the slope of a vignette variable. If no significant context effects exist, then multilevel analysis ends up with conventional OLS-regression. Under this condition, the unstandardized regression coefficients are estimated on the basis of the pooled data matrix where the D-efficiency of the factorial survey exclusively corresponds to the D-efficiency of the combined sample of all vignette sets. In this case, simple random designs with an increasing number of vignette sets within the limits of a decreasing sampling error easily produce more reliable results than unbalanced fractional factorial designs and D-efficient designs. This especially applies to factorial surveys where a relatively low set size is used. If on the other hand the answer behavior of different respondents is very heterogeneous, then the estimated γs (posterior means) are much more strongly based on the D-efficiency of the single vignette sets. Since unexplained context effects caused by unmeasured respondent-level variables are very likely to occur in factorial surveys, the estimators of simple random designs can only partly benefit from the fact that the design is becoming asymptotically more orthogonal and more balanced. Especially for small set sizes with a high D-efficiency and a relatively heterogeneous answer behavior, both fractional factorial designs and D-efficient designs will allow respondent-specific βs to be estimated with higher reliability than simple random designs (cf. Dülmer 2007:395-96, 405). The same also applies to a comparison between confounded factorial designs or confounded D-efficient designs on the one hand and simple random designs on the other.

Internal Validity

High reliability is necessary but not sufficient for high validity (cf. for instance, Neuman 2006:196-97). This at least applies to the reliability of the measuring instrument (i.e., the chosen design), the focus of the present article, and to internal validity. Accordingly, even a highly reliable estimator might suffer from a high systematic bias and for that reason might have low internal validity. Although each individual vignette set of a quota sample reaches a higher D-efficiency and therefore the respondent-specific βs can be estimated with a higher reliability than an average set of a random vignette sample, both fractional factorial designs and D-efficient designs have in general a somewhat higher susceptibility to systematic biases caused by interaction effects between vignette variables that were not expected in theory and so were not included when the design was generated. If for a fractional factorial design a main effect is perfectly aliased with a higher order interaction effect, then the estimated main effect captures the whole influence of the interaction effect. So, if an unexpected interaction effect should have a nonnegligible effect (due to perfect aliasing this cannot be tested afterward), then the main effect with which the interaction effect is perfectly aliased will suffer from a corresponding bias. The internal validity of the results is endangered. For a D-efficient design, the influence of a not-included but nonnegligible interaction effect will be distributed corresponding to its correlation with the vignette variables across several main effects. Therefore, the bias that results for an individual main effect is less severe for a D-efficient design than for a fractional factorial design. ¹² The fact that at least subsequent testing for unexpected higher order interaction effects is frequently impossible for fractional factorial (perfect aliasing) and D-efficient designs probably explains why such designs are sometimes seen as having a somewhat lower internal validity than random designs. An advantage of simple random designs and confounded designs is that they at least allow for the impact of such interaction effects to be tested across different vignette sets. But such tests—although at least possible—also have certain limitations.

For simple random designs, all main and interaction effects of a vignette set are randomly aliased. If a nonnegligible interaction term is not included when the vignette sets are generated, then the respondent-specific β of a vignette variable will, depending on the set-specific correlation with the interaction term, be either under- or overestimated, or almost correctly estimated. Since the aliasing structure of each vignette set of simple random designs is generated randomly, with an increasing number of respondents, the set-specific error across all included vignette sets and within the limits of the sampling error will asymptotically approach zero. Therefore, a simple random design allows an unbiased estimation of the γs of all included vignette variables. So, although the differences in the aliasing structure of the vignette sets of a simple random design contribute to an increase in random errors, they will not cause a systematic bias of the estimated γs. ¹³

However, if an unanticipated interaction effect is nonnegligible in comparison to a respective main effect, then it becomes impossible to interpret conditional effects (a conditional effect is the main effect of one vignette variable under the condition that the other vignette variable of the interaction term reaches a specific value that the researcher is interested in, cf. Friedrich 1982; Thome 1991). In contrast to fractional factorial designs and D-efficient designs, with simple random designs it is possible, at least in principle, to test afterward for virtually all interaction effects. This, however, strictly speaking only applies to situations where the answer behavior of the respondents empirically justifies fixing the respective random components. The test of whether such a random component is needed or not (cf. Hox 2010:47) is based on the number of individual respondents for which the randomly selected vignette set allows a respective interaction effect to be estimated. For simple random designs, this is generally only possible for a part of all respondents. It follows that, the lower the number of such respondents, the less reliable the significance test will be for all between-respondent variance components included (listwise exclusion). If none of the individual vignette sets allows an interaction effect to be estimated, then the γ can only be estimated without a variance component, that is, across vignette sets, by assuming that the β ˆ is the same for all respondents. As a consequence, the t value for the common γ ˆ is based on the number of vignettes instead of on the much lower number of respondents (in the former case minus the degrees of freedom needed for estimating the γs of the whole multilevel model, in the latter case minus the degrees of freedom needed for estimating the γs that predict a random β ˆ ). Although the γ ˆ of an unanticipated interaction term (as well as the conditional effects) will not suffer from systematic bias, the t value will be too high in cases where the variance component would be needed but cannot be estimated. This is a limitation of simple random designs.

From the start, the confounded factorial design guarantees that the estimated γs of the vignette variables (including the intercept) are protected against a potential systematic bias that otherwise might be caused by a chosen design. If a suitable confounded factorial design exists which allows all important main and interaction effects to be estimated without any confounding, then respondent-specific β ˆ s will show no design-related systematic biases. This applies as long as all set variables are controlled for. Hence, the protection against potential biases, which might result in a higher internal validity, is an advantage of confounded fractional designs over fractional factorial designs and D-efficient designs.

However, if an unanticipated interaction effect captured across vignette sets via a set variable turns out to have a nonnegligible size in comparison to a respective main effect, then the unanticipated interaction effect ¹⁴ (as well as the conditional effects) will not be biased (provided that no or only a negligible set effect exists) but the t value again might be misleading. Depending on whether the γ of a main effect is estimated with a random component or not, the t value of the γ ˆ of the set variable that captures the impact of a respective interaction effect is based on the number of respondents or on the much higher number of vignettes (minus the degrees of freedom needed for estimating the γs that predict a random β ˆ or for estimating the γs of the whole multilevel model, respectively). An independent test of whether or not the γ of the interaction term has to be estimated with its own variance component cannot be carried out (set variables have no within respondent variability). These considerations show the limitations of confounded fractional factorial designs.

The gain in internal validity of confounded factorial designs over simple random designs consists in the higher D-efficiency of each individual fractional factorial design of which a confounded factorial design consists (this applies to each individual set comparison except the one where a fractional factorial design or sometimes even a D-efficient design is part of the simple random design) ¹⁵ ; at the same time, including all set variables needed to protect the estimated γs of the vignette variables of a confounded fractional factorial design against design-related potential systematic biases also reduces the variance components of the respondent-specific β ˆ s. Therefore, respondent-specific βs can be estimated with higher reliability and for this reason also with higher internal validity for the confounded factorial design than for a simple random design.

If no suitable confounded factorial design can be found, then a confounded D-efficient design may be used instead. The individual vignette set of such designs show—like D-efficient designs, fractional factorial designs, and confounded factorial designs—a higher D-efficiency than an average vignette set of simple random designs. The gain in internal validity of the confounded D-efficient design over D-efficient designs and fractional factorial designs is that the risk of design-related systematic biases of the reported γ ˆ s across the vignette sets is zero. This requires, however, that the number of respondents is the same for each D-efficient design of which the confounded D-efficient design consists (this requirement is not needed for confounded factorial designs as long as all set variables are included in the estimated multilevel model). The less this requirement is fulfilled and the higher the correlations between vignette variables and unanticipated nonnegligible interaction terms not included in a chosen D-efficient design, the higher the design-related systematic bias of the reported γ ˆ s will be. As with simple random designs, unanticipated higher order interaction effects can in general only be estimated without a variance component, that is, across different D-efficient designs, by assuming at the same time that the respondent-specific β ˆ s of an interaction effect do not differ significantly (homogeneous answer behavior). Here again, the t value will be too high in cases where a variance component would be needed but cannot be estimated. All in all, these considerations underline again how important it is that nonnegligible interaction effects can be estimated for each individual vignette set.

Operationalizations and Data

In order to illustrate the expected differences in the reliability and internal validity between different designs, a parsimonious design is most suitable since each respondent can be presented with the fully crossed vignette population (full factorial design). One of the main advantages of this procedure is that the estimates from the full factorial design can be used as the benchmark for the intended design comparison. Based on the four items that are used to measure Ingleharts’s materialist and postmaterialist value orientations, such a parsimonious full factorial design can easily be generated.

According to Inglehart (1994:290-91), materialist and postmaterialist value priorities can only be measured adequately via a ranking procedure. This assumption has been criticized by authors like Bürklin, Klein, and Ruß (1996) as theoretically inadequate. If a political system is not generally assumed to have limited capacities for problem solving, then there is no trade-off relationship between the two items “protecting freedom of speech” and “fighting rising prices” that would justify a ranking procedure. The factorial survey makes it possible to avoid such restrictions by presenting respondents with different vignettes with a list of the four Inglehart items. The task of the respondents could be to indicate how much they would like to be governed by a party for which the listed items are either not so important or very important. The factorial survey also reduces the problem of response sets (in many cases response sets can be easily identified), frequently observed in simple rating procedures, which have been criticized also for this reason by Inglehart (1997:116-17) as an unsuitable alternative for measuring both value orientations. ¹⁶ For illustration purposes, an example vignette with an introduction to the task is given in Table 1.

OPEN IN VIEWER

Table 1.

Example Vignette for Measuring Inglehart’s Value Orientations with Introduction.

The vignette universe of four items with each having two levels (“important” and “not so important”) consists of 16 (= 2⁴) vignettes. To prevent possible order effects from causing systematic bias to the estimators, the vignette order of all questionnaires was randomly selected (cf. also Jasso 2006:343; Rossi and Anderson 1982:33). The paper-and-pencil interviews with the full factorial survey (completely crossed vignette universe), which was presented to each participant were carried out on October 16, 2006, during the first session of two identical methodological courses designed for students of the Faculty of Economic and Social Sciences at the University of Cologne. In total, 137 students participated in the interviews. Five questionnaires were not sufficiently filled out and had to be excluded from analysis. Of the remaining 132 participants, 72 were female and 60 male.

The empirical comparison includes, besides the full factorial design, two random and two quota designs. The required random and quota designs were produced by dropping vignettes from the full factorial design. For analyzing random and quota designs, these subsamples/fractions of the vignette universe, including the corresponding fraction of respondent-specific answers from the full factorial design, were used. Therefore, the statistical comparison of random and quota designs is based on these fractions of the complete data set. The complete knowledge about the answers from the full factorial survey has the advantage that it allows an empirically based yardstick to be established for the empirical comparison of the reliability and internal validity of different designs. By using this setting, all possible other sources of disturbances not related to a selected design itself are eliminated.

Since Inglehart’s (1977:28-29) value types are assigned on the basis of the two items that a respondent ranked highest out of the four items, it is assumed that interaction effects are negligible. Therefore, all reduced designs included in the comparison are generated as main effect only designs. The reported D-efficiency in the following will refer to this model. Nonetheless, only those designs that should be especially robust against the influence of not included interaction effects between vignette variables will be compared. In order to reduce the likelihood of comparing outliers and at the same time to get a measurement for the stability of the results, each of the four reduced designs was produced 50 times, whereby the required vignette sets were each time randomly assigned to the respondents. Therefore, the reported results for each reduced design will be based on 50 multilevel regressions.

The first quota design included in the comparison is a confounded factorial design. This design was constructed by dividing the vignette universe along the value of the highest interaction term (X ₁·X ₂·X ₃·X ₄) into two sets of eight vignettes. Since, with regard to the main effects, each of the resulting half-fractional factorial designs is orthogonal and balanced, both set 1 and set 2 already possess a D-efficiency of 100. Dividing the vignette universe along the product term of the four vignette variables ensures that both vignette sets are of Resolution IV (cf. Ryan 2007:170). Table 2 gives an overview of the confounding pattern of the confounded factorial design.

OPEN IN VIEWER

Table 2.

Overview of the Aliasing and Confounding Pattern of the Confounded Factorial Design.

	Independent Variables				Aliasing/Confounding Pattern
Vignette Number	X 1	X 2	X 3	X 4	X 1 X 2 = X 3 X 4	X 1 X 3 = X 2 X 4	X 1 X 4 = X 2 X 3	X 1 X 2 X 3 = X 4	X 1 X 2 X 4 = X 3	X 1 X 3 X 4 = X 2	X 2 X 3 X 4 = X 1	X 1 X 2 X 3 X 4 = I a
Set 1
1	−	−	−	−	+	+	+	−	−	−	−	+
2	−	−	+	+	+	−	−	+	+	−	−	+
3	−	+	−	+	−	+	−	+	−	+	−	+
4	−	+	+	−	−	−	+	−	+	+	−	+
5	+	−	−	+	−	−	+	+	−	−	+	+
6	+	−	+	−	−	+	−	−	+	−	+	+
7	+	+	−	−	+	−	−	−	−	+	+	+
8	+	+	+	+	+	+	+	+	+	+	+	+
Vignette Number	X 1	X 2	X 3	X 4	X 1 X 2 = −X 3 X 4	X 1 X 3 = −X 2 X 4	X 1 X 4 = −X 2 X 3	X 1 X 2 X 3 = −X 4	X 1 X 2 X 4 = −X 3	X 1 X 3 X 4 = −X 2	X 2 X 3 X 4 = −X 1	X 1 X 2 X 3 X 4 = −I a
Set 2
1	−	−	−	+	+	+	−	−	+	+	+	−
2	−	−	+	−	+	−	+	+	−	+	+	−
3	−	+	−	−	−	+	+	+	+	−	+	−
4	−	+	+	+	−	−	−	−	−	−	+	−
5	+	−	−	−	−	−	−	+	+	+	−	−
Vignette Number	X 1	X 2	X 3	X 4	X 1 X 2 = −X 3 X 4	X 1 X 3 = −X 2 X 4	X 1 X 4 = −X 2 X 3	X 1 X 2 X 3 = −X 4	X 1 X 2 X 4 = −X 3	X 1 X 3 X 4 = −X 2	X 2 X 3 X 4 = −X 1	X 1 X 2 X 3 X 4 = −I a
6	+	−	+	+	−	+	+	−	−	+	−	−
7	+	+	−	+	+	−	+	−	+	−	−	−
8	+	+	+	−	+	+	−	+	−	−	−	−

Note: The equals sign in the table indicates which interaction terms are perfectly aliased with an X variable or another interaction term within a respective vignette set. If an interaction term is perfectly positively aliased with an X variable or another interaction term within vignette set 1 (for instance, X 1 ⋅ X 2 ⋅ X 3 is perfectly positive aliased with X ₄ within set 1) and perfectly negatively aliased with the same X variable or interaction term within vignette set 2 ( X 1 ⋅ X 2 ⋅ X 3 is perfectly negative aliased with X ₄ within set 2), then it is possible to capture the common influence of the perfectly confounded interaction term and the set variable across the two different vignette sets by including a respective set variable at the respondent level (i.e., the interaction term between X ₄ and the set variable; including this two-way interaction term allows the common influence of the interaction term X 1 ⋅ X 2 ⋅ X 3 and the set variable with which the interaction effect is perfectly confounded to be captured across both vignette sets). The same applies to a perfectly aliased interaction term that is at the same time also perfectly confounded with a set effect ( X 1 ⋅ X 2 ⋅ X 3 ⋅ X 4 is perfectly aliased with the intercept within a vignette set and perfectly confounded with the set effect). If and as long as the set effect is at least negligible, the magnitude of the estimated γ for the two-way interaction term captures the estimated magnitude of the three-way interaction effect with which the main effect is perfectly aliased (in our case, set effects are implausible because every respondent judged the same full factorial design).

^a I = intercept; “−” stands for “−1,” “+” stands for “+1.” In order to construct both half-fractional factorial designs (set 1 and set 2), the vignette universe has to be divided along the highest order interaction term X 1 ⋅ X 2 ⋅ X 3 ⋅ X 4 . Since the vignette universe is divided along a four-way interaction term, the resulting design is of Resolution IV (by choosing a three-way interaction term, one would produce a half-fractional factorial design of Resolution III). The formula for generating both vignette sets (only applicable to effect-coded dichotomous variables) is:

X 1 ⋅ X 2 ⋅ X 3 ⋅ X 4 = − and X 1 ⋅ X 2 ⋅ X 3 ⋅ X 4 = + . The same equation system can be written for 0–1 dummy coding (division with integer remainder) as:

X 1 + X 2 + X 3 + X 4 = 0 , m o d u l o 2 and X 1 + X 2 + X 3 + X 4 = 1 , m o d u l o 2 .

The first vignette set was assigned randomly to half of the 132 respondents; the second vignette set was assigned to the remaining respondents. ¹⁷ To get different assignments, this procedure was repeated 49 times. Since for a half-fractional factorial design only 8 instead of 16 vignettes are needed, the remaining 8 vignettes were excluded from the respective analysis. In this way, the number of vignettes included in a single multilevel analysis was reduced to 1,056 (132 respondents × 8 vignettes).

If for a given set size no suitable confounded factorial designs can be found, then reliability and internal validity can be optimized instead via a confounded D-efficient design. This condition is met for a given set size of 10 vignettes. For this set size, the maximal D-efficiency for a D-efficient design is 97.032 (cf. also Dülmer 2007:394, 398). Two different examples of such designs are presented in Table 3.

OPEN IN VIEWER

Table 3.

Balance and Correlations of Two Possible D-efficient Designs of Set Size 10.

		D-efficient Design 1					D-efficient Design 2
Variable		X 1	X 2	X 3	X 4		X 1	X 2	X 3	X 4
Balance		4/6	4/6	5/5	5/5		4/6	5/5	5/5	5/5
Correlations	X 1		0.167	0.000	0.000	X 1		0.000	0.000	0.000
	X 2			0.000	0.000	X 2			0.200	0.200
	X 3				−0.200	X 3				0.200

Note: The D-efficiency of both designs is 97.032 (search algorithm: Modified Federov which is the most reliable, cf. Kuhfeld et al. 1994:548). Balance refers to the ratio of the levels of the dichotomous vignette variables (i.e., the ratio of “−1” to “1”). Design 1 consists of the vignettes 1, 2, 3, 4, 5, 6, 7, 8 of set 1 and 7, 8 of set 2. Design 2 consists of the vignettes 1, 4, 6, 7, 8 of set 1 and 1, 4, 5, 6, 7 of set 2 (the numbering refers to Table 2, first column). The confounded D-efficient design (D1–D8) used for design comparison consists of the following vignettes (numbering again refers to Table 2, first column):

D1: Set 1: vignettes 1–8, Set 2: vignettes 7 and 8;

D2: Set 1: vignettes 1–8, Set 2: vignettes 3 and 4;

D3: Set 1: vignettes 1–8, Set 2: vignettes 5 and 6;

D4: Set 1: vignettes 1–8, Set 2: vignettes 1 and 2;

D5: Set 1: vignettes 3 and 4, Set 2: vignettes 1–8;

D6: Set 1: vignettes 5 and 6, Set 2: vignettes 1–8;

D7: Set 1: vignettes 7 and 8, Set 2: vignettes 1–8;

D8: Set 1: vignettes 1 and 2, Set 2: vignettes 1–8.

A main effect only model for a confounded D-efficient designs consisting of D1–D4 (or alternatively for D5–D8) would already reach a D-efficiency of 100 across the four selected designs. By permuting the assignment between the four Inglehart items and the variables X ₁ to X ₄, one could further increase the number of designs used for a confounded D-efficient design.

D-efficient design 1 is less balanced than D-efficient design 2 (X ₁ is unbalanced for both designs, X ₂ only for design 2) but compensates for higher imbalance by having lower correlations between the vignette variables. In accordance with classical fractional factorial designs, the D-efficient design with the lower correlations was given priority over the more balanced design. Based on D-efficient design 1, seven further D-efficient designs of the same efficiency were constructed by switching the coding for the two levels of one or more vignette variables. Therefore, the combined confounded D-efficient design consists of eight D-efficient designs of the same D-efficiency. For the combined confounded D-efficient design consisting of eight D-efficient designs, not only does the main effect only model have a D-efficiency of 100 but so does the model that includes all possible interaction effects besides the main effects. Therefore, the confounded D-efficient design consisting of eight D-efficient designs is, like the vignette universe, fully balanced and orthogonal. Exactly the same applies to the collected data only as long as each of the eight designs is answered equally often. To distribute the eight D-efficient designs as evenly as possible across the 132 respondents, four of the eight D-efficient designs were used 16 times, and the remaining four 17 times ( 4 ⋅ 16 + 4 ⋅ 17 = 132 vignette samples). The decision on which of the D-efficient designs should be included 17 times was determined randomly. This data set (ordered by the design and vignette number) was thereafter merged to the data set of the respondents, which was itself first ordered randomly by identification number (the order of the vignettes remained unchanged). Since for a set size of 10 only the answers for these 10 vignettes are needed, the answers for the six spare vignettes were replaced by a missing value and excluded from analyses. In this way, the number of vignettes used for a single multilevel analysis was reduced to 1,320 (132 respondents × 10 vignettes). Since for simulation purposes it was decided to have 50 multilevel regressions, the described procedure was repeated 49 times. This ensures that both the decision about which D-efficient designs will be used 17 times and the assignment between respondents and D-efficient designs vary randomly. Finally, for both set sizes (8 and 10 vignettes), a simple random design without replacement was also generated 50 times. This enables a comparison of quota and random designs on the basis of the same number of respondents.

The only respondent characteristic that is included in the multilevel analysis is gender. The decision to include a respondent characteristic essentially goes back to an assumption of Steiner and Atzmüller (2006:123-24): Confounding set effects (caused by differences between vignette sets) with respondent effects, as is the case for random designs, would have a negative impact on the estimators for respondent characteristics. Since vignette sets are randomly assigned to respondents, this fear seems to be unwarranted for the reason that, with an increasing number of respondents, both effects will, within the limits of sampling error, be asymptotically uncorrelated. Hence, the estimators for the respondent level should at least not suffer from a systematic bias.

All predictor variables included in the analyses were effect coded. ¹⁸ Code −1 was used if a political goal was “not so important” for a fictitious governing party, code 1 if the goal was “very important” for the party. A respondent’s gender was coded −1 for males and 1 for females. The rescaled answer scale from the vignettes ranges from 0 (not at all) to 8 (very strongly). ¹⁹ The set variable included in the confounded factorial design was coded −1 for the first half-fractional factorial design and 1 for the second half-fractional factorial design. In our case, systematic set effects can be excluded because all respondents answered the fully crossed vignette universe. Hence, the effect that is captured by a respective set variable exclusively measures the impact of the interaction effects between vignette variables that are perfectly confounded with this set effect. All multilevel analyses were carried out with the multilevel program HLM 6.

Empirical Results of the Example Comparison

Each multilevel regression model is based on equation (4), except for the predictor variables the same terminology is used as in equation (1), whereby the set variable (Set) was only included in the multilevel regression model for the confounded factorial design. Since all five estimated variance components turned out to be highly significant, none of them could be fixed (i.e., dropped from the equation for the multilevel model).

Vignette level (Level 1):

Y i j = β 0 j + β 1 j O r d e r i j + β 2 j G o v e r n m e n t i j + β 3 j P r i c e i j + β 4 j S p e e c h i j + r i j .

4

Respondent level (Level 2):

β 0 j = γ 00 + γ 01 G e n d e r j + γ 02 S e t j + u 0 j ,

β 1 j = γ 10 + γ 11 S e t j + u 1 j ,

β 2 j = γ 20 + γ 21 S e t j + u 2 j ,

β 3 j = γ 30 + γ 31 S e t j + u 3 j ,

β 4 j = γ 40 + γ 41 S e t j + u 4 j .

The empirical results of the design comparison are presented in Table 4. The first column contains the estimators for the main effect model of the full factorial design. Since 50 separate multilevel analyses were estimated for each of both random designs and for each of both quota designs, the reported results for these cases also include the respective standard deviations. Two further columns were needed for the confounded factorial design in order to document the estimators of the main effect of the set variable and its interaction effect with each of the four vignette variables (mean effect: set variable) as well as their respective standard deviations.

OPEN IN VIEWER

Table 4.

Empirical Comparison of the Estimated Coefficients of Different Designs.

	Full Factorial Design	Simple Random Design (50 Samples)		Confounded Factorial Design (50 Samples)							Simple Random Design (50 Samples)		Confounded D-efficient Design (50 Samples)
Set Size (n)	16	8		8							10		10
Respondents (n)	132	132		132							132		132
Vignettes (n)	2,112	1,056		1,056							1,320		1,320
		Mean	Standard Deviation	Mean	Standard Deviation		Mean Effect: Set Variable		Standard Deviation		Mean	Standard Deviation	Mean	Standard Deviation
γ ˆ Intercept	3.515	3.514	0.029	3.514	0.022		−0.110		0.082		3.511	0.025	3.519	0.014
γ ˆ Gender	0.105	0.109	0.038	0.114	0.026						0.113	0.023	0.109	0.023
γ ˆ Order	0.428	0.430	0.031	0.430	0.023		−0.095		0.065		0.429	0.032	0.427	0.022
γ ˆ Government	0.668	0.671	0.027	0.674	0.026		−0.132		0.043		0.671	0.024	0.668	0.020
γ ˆ Price	0.320	0.321	0.031	0.321	0.018		−0.078		0.038		0.320	0.028	0.317	0.019
γ ˆ Speech	1.107	1.113	0.033	1.106	0.025		−0.045		0.058		1.102	0.023	1.104	0.019
σ ˆ Intercept	0.078	0.085	0.004	0.081	0.002		0.081		0.002		0.082	0.003	0.080	0.002
σ ˆ Gender	0.077	0.083	0.004	0.080	0.003						0.080	0.003	0.079	0.002
σ ˆ Order	0.068	0.074	0.003	0.072	0.002		0.072		0.002		0.072	0.002	0.072	0.002
σ ˆ Government	0.049	0.058	0.002	0.055	0.003		0.055		0.003		0.054	0.002	0.053	0.002
σ ˆ Price	0.044	0.054	0.002	0.051	0.002		0.051		0.002		0.049	0.002	0.049	0.002
σ ˆ Speech	0.054	0.062	0.003	0.058	0.002		0.058		0.002		0.059	0.002	0.058	0.002
t Intercept	44.924	41.624	1.885	43.423	1.254		−1.357		1.016		43.027	1.322	43.861	1.137
t Gender	1.372	1.322	0.482	1.424	0.328						1.416	0.297	1.384	0.300
t Order	6.307	5.800	0.518	5.960	0.376		−1.318		0.906		5.959	0.490	5.971	0.357
t Government	13.767	11.671	0.554	12.327	0.908		−2.419		0.786		12.341	0.675	12.561	0.648
t Price	7.313	5.935	0.589	6.362	0.381		−1.552		0.778		6.478	0.579	6.547	0.433
t Speech	20.646	18.038	1.118	19.021	0.758		−0.762		0.991		18.777	0.789	19.188	0.732
τ ˆ 00 Intercept	0.708	0.715	0.088	0.661		0.049					0.707	0.056	0.683	0.045
τ ˆ 11 Order	0.513	0.504	0.059	0.491		0.040					0.517	0.038	0.515	0.042
τ ˆ 22 Government	0.218	0.209	0.031	0.200		0.038					0.222	0.030	0.212	0.029
τ ˆ 33 Price	0.159	0.157	0.033	0.139		0.026					0.152	0.023	0.154	0.023
τ ˆ 44 Speech	0.286	0.278	0.046	0.250		0.027					0.287	0.034	0.282	0.027
σ ˆ R 2	1.492	1.494	0.117	1.590		0.091					1.484	0.059	1.537	0.060
χ U 00 2 Intercept	1,117.399	480.779	62.105	561.470		46.023					631.731	58.515	670.856	44.576
χ U 11 2 Order	852.117	372.845	35.095	452.881		33.059					496.000	35.679	541.872	40.957
χ U 22 2 Government	436.589	228.891	18.550	261.856		27.865					289.293	27.840	300.424	26.114
χ U 33 2 Price	354.125	207.711	22.790	221.675		18.980					240.638	20.305	254.795	21.400
χ U 44 2 Speech	532.550	267.545	33.554	294.477		22.641					333.976	34.769	358.390	25.550
R 1 2	.368	.372	.018	.378		.014					.368	.016	.366	.011
R 2 2	.013	.132	.050	.025		.024					.097	.035	.033	.028
Iterations	5.000	26.880	11.508	6.320		2.938					16.820	4.241	13.080	2.656

Note: Coding of gender: male = −1, female = 1. The effect of the set variable for the intercept is the main effect of the set variable. The effect of the set variable of the vignette variables is the interaction effect between a respective vignette variable and the set variable. The (Pseudo-) R ² for each level has been calculated according to the simplified formula of Snijders and Bosker (1994:350-54). All multilevel models were estimated via restricted maximum likelihood.

The estimated γs are documented in the first block (rows 1–6) of the table. Since for the full factorial design all correlations between the vignette variables and their interaction terms are exactly zero and since only respondents who judged all 16 vignettes were included, the reported γ ˆ s of the vignette variables will remain unchanged even if interaction terms are entered into the multilevel regression analyses. Therefore, the γ ˆ s of the full factorial design offer a suitable yardstick for identifying possible systematic biases of the estimators of the reduced designs. The biggest average deviation between the γ ˆ of a vignette characteristic of one of the four reduced designs on the one hand, and the respective coefficient for the full factorial design on the other amounts to .006 scale points. This deviation is observed for the simple random design of set size 8 ( γ ˆ Speech) and for the confounded factorial design ( γ ˆ Government). Furthermore, the average γ ˆ s for the intercept as well as for the respondents’ gender also show only negligible deviations from their respective yardstick. Hence, all four reduced designs allow the γs to be estimated without systematic bias.

The confounded factorial design is the only reduced design for which the magnitude of the interaction effects that are perfectly aliased with the main effects can be estimated via respective set variables across respondents (this applies as long as no or only a negligible set effect exists). Such set variables prevent the estimated main effects from being systematically biased by potentially nonnegligible interaction effects in cases where the response rate for different vignette sets (i.e., different fractional factorial designs) somewhat differs.

According to Table 2, each of the four Inglehart items is perfectly aliased within a vignette set with a three-way interaction term between the remaining three Inglehart items, in vignette set 1 positively (for instance, X 1 = X 2 ⋅ X 3 ⋅ X 4 ) and in vignette set 2 negatively ( − X 1 = X 2 ⋅ X 3 ⋅ X 4 ). This difference in the aliasing pattern allows the common magnitude of the perfectly confounded three-way interaction effect and the set effect to be estimated across respondents via a set variable (more precisely, via a two-way interaction term between the vignette variable and the set variable). Since in our case systematic set effects can be excluded, the magnitude of the estimated γ for the two-way interaction term between the vignette variable and the set variable (to be found in the column “mean effect: set variable”) measures the estimated magnitude of the three-way interaction effect with which the main effect is perfectly aliased within separate vignette sets. Under this condition, the estimated two-way interaction effect between the set variable and the item “order” means that the γ of this item is, due to its perfect aliasing with the three-way interaction between the items “government,” “price,” and “speech,” overestimated on average by .095 scale points for the first half-fractional factorial design and underestimated on average by the same amount for the second half-fractional factorial design (the polarity of the systematic bias is given by the coding of the set variable, i.e., a −1 for set 1 and a +1 for set 2). Hence, the γ ˆ for the three-way interaction between “government,” “price,” and “speech” amounts to .095 scale points. The γ for the items “government,” “price,” and “speech” is on average overestimated by .132, .078, and .045 scale points, respectively, for the first half-fractional factorial design and underestimated by the same amount for the second half-fractional factorial design. The average bias of the intercept which is perfectly aliased within each vignette set with the four-way interaction effect between all four Inglehart items is .110 for the first set and −.110 for the second one. Therefore, due to its aliasing pattern, using a half-fractional factorial design instead of a confounded fractional design would have resulted on average in slightly biased estimators for the γs. For the confounded factorial design, this systematic bias is captured by including set variables. The results in Table 4 show furthermore that both simple random designs also allow the γs to be estimated without systematic biases—despite the fact that there are small interaction effects present in the student sample. ²⁰

The first systematic differences between the four reduced designs can be observed with respect to the standard deviations for the γ ˆ s which show the average distance between the estimated γs and their mean: The lower the standard deviation, the higher the reliability of a respective design. The standard deviation for the γ ˆ s of quota designs, because of the higher D-efficiency of both quota designs in comparison to simple random designs, is almost always smaller than the standard deviation of the simple random design of the same set size (the only exception is the γ ˆ for gender which has a standard deviation of .023 for both designs of set size 10). The factor by which the standard deviation for a γ ˆ of the simple random designs is higher than that of the quota design of the same set size reaches a maximal value of 1.72 for the comparison between the confounded factorial design and the respective simple random design (.018 vs. .031 for “price”) and a maximal value of 1.79 for the comparison between the confounded D-efficient design and the simple random design of set size 10 (.014 vs. .025 for the intercept). Therefore, both quota designs, for the same set size, show the higher reliability. Although the γs can be estimated for all four reduced designs without systematic bias, simple random designs, due to their lower reliability, also possess a lower internal validity than the respective quota design. This result is also corroborated by the estimated standard errors, σ ˆ , of the γ ˆ s (cf. block 2), although the observed pattern in this case is somewhat less pronounced.

If the same γ is estimated with a lower standard error, then the t value as the quotient of both estimators will increase. As a consequence, a γ ˆ will become significant more easily. The comparison of the means of the t values (block 3) for single vignette characteristics documents that with one exception (t value of gender for the confounded D-efficient design) they are higher for quota designs than for simple random designs of the same set size. In our case, not until the set size of the simple random design is increased by two vignettes do its t values reach nearly the same level as the respective t values of the confounded factorial design.

In multilevel analysis, the estimated sum of the squared deviations of the β ˆ of each context (here each respondent) from that unstandardized regression coefficient’s grand mean γ ˆ is called variance component τ ˆ , the estimated variance of the estimated residuals r_ij at level 1 (here the vignette level) is called σ ˆ 2 . These estimators can be found in the second part of Table 4 (first block). The mean of the variance components of the confounded factorial design is always somewhat smaller than the respective mean of the variance components of both simple random designs. The respective χ² values (block 2) are, however, on average somewhat smaller for the simple random design than for the quota design of the same set size. Whether an estimated variance component τ becomes significant or not is tested in HLM via a χ² test. A χ² value is computed by summing up across all respondents the squared deviation of a respondent-specific estimated β from its overall estimator γ computed across all respondents, divided by the respondent-specific estimated sampling variance of that β ˆ (i.e., by the square of the respondent-specific estimated standard error of that β ˆ ; cf. also Hox 2010:47). As long as a β is estimated ceteris paribus with a smaller standard error (higher reliability) for a quota design than for a simple random design, the respective χ² value will be higher. Therefore, for a given set size, quota designs are better able to detect unexplained heterogeneity between respondents. Increasing the set size reduces the estimated standard error of the β ˆ s and a respective χ² value will increase. This consideration also explains why the simple random design of set size 8 shows the lowest χ² values and the confounded D-efficient design the highest ones.

The average coefficients of multiple determination computed according to the simplified formulas proposed by Snijders and Bosker (1994:350-54) can be found immediately below the χ² values. Including the four vignette variables in the multilevel model explains 36.8 percent of the vignette level variance of the full factorial design. All reduced designs show very similar values. Big differences between the designs become visible for the respondent level: According to the full factorial design, gender accounts for 1.3 percent of the level 2 variance. With an average of 2.5 percent explained variance, the confounded factorial design comes closest to this value. This small difference, however, can be traced back to the additional inclusion of the set variables at level 2. The remaining respondent level R ²s are on average 3.3 percent for the confounded D-efficient design, 9.7 percent for the simple random design of set size 10, and 13.2 percent for the simple random design of set size 8. The reason for the higher percentage of explained variance is that differences between the means of the vignette variables of different vignette sets lead to variation in the mean of the dependent variable between respondents. Including the vignette variables in regression analysis explains these differences between respondents—the level 2 coefficient of multiple determination increases. Since both half-fractional factorial designs of the confounded factorial design are balanced, no differences between the means of the vignette variables exist across either set. Although such mean differences also exist for the confounded D-efficient design, optimizing D-efficiency reduces these differences in comparison to simple random designs. Hence, including all vignette variables in multilevel analyses of quota designs either cannot increase the R ² values of the respondent level or only to a lower degree than for a simple random design.

A final look at the number of iterations shows that due to their more complex error structure where unique set effects are confounded with unique respondent effects, random designs need on average more iterations than quota designs until the maximum likelihood function for estimating multilevel regression converges.