Exploring the Response Process of Culturally Differing Survey Respondents with a Response Style

Latent Variable Modeling to Assess Cross-Cultural Comparability

Within the latent variable modeling approach, attitudes are approached as complex theoretical constructs for which multiple empirical indicators reflecting several important aspects of the attitude are needed for a valid and reliable measurement (Bollen 2002). An attitude is modeled as a latent—unobserved—variable (also called factor or trait) for which survey questions (hereafter called items) are the indicators or observed variables. Covariates are included in this basic modeling approach to investigate differences between groups, as shown in Figure 1.

OPEN IN VIEWER

Figure 1.

The latent variable model and measurement equivalence.

Here, Y ₁ to Y ₁₅ represent the item responses that are directly related to the latent variables measuring the attitudes F ₁, F ₂, and F ₃, and indirectly related to the observed group variable G. The systematic variance among the item responses is captured by the factor loadings β_1j , β_2j , and β_3j (index j refers to an item). It is assumed that the five items in the first item subset do not relate directly to the second and third attitude; that is, the item parameters β_2j and β_3j are fixed to zero. Likewise, the parameters β_1j and β_3j are fixed to zero for items Y ₆ to Y ₁₀ and parameters β_1j and β_2j are fixed to zero for items Y ₁₁ to Y ₁₅.

If respondents with similar attitude levels respond differently to the items given his or her group membership, the measurements are inequivalent (model A) or partially equivalent (model B). Measurements inferred from respondents whose responses are similar irrespective of their group membership are scalar equivalent (model C). Note that measurement equivalence allows for group differences in attitudes but not for group differences in the measurement of these attitudes. Formally, model A allows for group differences in both factor loadings and intercepts, model B for group differences in intercepts, and model C restricts the responses to be equivalently related to the attitudes across groups. In Figure 1, violation of scalar equivalence is indicated by the direct effect β_{05 g} (dashed arrow) of G on item Y ₅, and a violation of metric equivalence is indicated by the direct effect β_{05 g} and the interaction effect on the relationship β_{15 g} between item Y ₅ and factor F ₁, meaning that G moderates the association between the fifth item and the first factor. We test for scalar equivalence by fixing the group-specific parameters β_05 g and β_15 g to zero; to test for metric equivalence, only β_15g is fixed to zero. Even though in Figure 1, the group parameters are only indicated for item Y ₅, the group variable G is usually assumed to affect all items simultaneously.

We applied study 1 to a secondary data set collected in 2002 among heads of the households in the four largest ethnic minorities in the Netherlands, namely Turkish, Moroccan, Surinamese, and Antillean people. While most people immigrated for economical and educational reasons, Surinamese and Antilleans came from Dutch colonies and were therefore more acquainted with the Dutch language, culture, and heritage than the Turks or Moroccans (CBS 2005). We include 15 items from this Social Position and Utility Use of Ethnic Minorities (SPVA) survey, each having five ordered response categories that range from totally agree to totally disagree. These items operationally define three attitudes, namely attitude toward the Netherlands, endorsement of traditional family values, and autonomy of children within the family. Given the substantive nature of these constructs, the groups are expected to be subject to culturally specific sources of measurement error and possibly to derive their answers from diverse response processes.

To investigate explanations for violations of measurement equivalence, we control for the following covariates: language use during the interview to measure meaningfulness of language (Gibbons et al. 1999); generation of immigration to measure acculturation (Marin et al. 1992); age, gender, level of income, and education (Light et al. 1965). Following Moors (2003), we also include a latent response style factor to control for heterogeneous response behavior (i.e., the respondents’ tendency to answer differently than expected based on their attitudes).

In all models, the effect of attitudes on item responses is distinguished from the effects of the control variables by estimating models in which only one item subset is affected by one attitude, while all items are simultaneously affected by the control variables. By comparing the statistical fit of models with and without these covariates, we investigate to what extent the covariates form a reasonable explanation of measurement inequivalence. To compare between models, we use the Bayesian Information Criterion (BIC), which introduces a penalty for the sample size and the number of parameters (Burnham and Anderson 2004) and is the most widely used measure for model selection in latent class analysis (Magidson and Vermunt 2004).

The best model in terms of fit and parsimony has the lowest value of BIC. According to the BIC, only the model including the response style factor explains part of the measurement inequivalence; the models including the other control variables remain measurement inequivalent. For an extensive description of these models and the results, see the web appendix (available at http://fmx.sagepub.com/).

First, without controlling for a response style, the ethnic minorities interpret the items differently given their attitude; close inspection of the parameters shows that these differences are visible in the intercepts as well as the factor loadings that indicates full measurement inequivalence (also indicated by model A having the lowest BIC value). Furthermore, we find that the style factor accounts for the group differences in the factor loadings: The BIC values decrease between model A_RS and model B_RS, indicating that the model assuming metric equivalence should be preferred. Even though group differences in the intercepts (not reported here) also decrease substantially once a control for response style is included, scalar equivalence is not attained. Note that the cultural group membership also influences response style directly: The Surinamese respond significantly less extremely than the other groups; the other groups do not differ significantly in their style of responding (see web appendix, available at http://fmx.sagepub.com/).

Based on a post hoc interpretation of the parameters of the response style factor, we conclude that this style factor measures two related types of response style. First, a person who scores high on the style factor is more likely to select the extreme categories than other respondents with similar attitudes. We refer to this response style as the Extreme Response Style (ERS), defined as the tendency to select extreme categories irrespective of the item content across a large range of unrelated or weakly related survey questions. Second, close inspection of the parameters shows that a person who scores low on the style factor avoids the extreme categories and selects the adjacent categories agree and disagree. Here, this response style is referred to as the Adjacent Response Style (AdjRS) and should not be confused with the tendency to select the midpoint category (midpoint responding) or to select the categories around the midpoint category (response range; see Baumgartner and Steenkamp 2001).

The style factor is specified as a latent variable with three ordered classes, of which class 3 represents ERS and the class at the other endpoint (class 1) represents AdjRS. The middle class (class 2) defines a position in between these two endpoints. We interpret this position as a mild response style and treat these respondents as not endorsing any response style. Based on the item responses of a respondent, we can compute the probability of belonging to each of these three classes. Typically, we would assign a respondent to the most likely class. Practically, this means that respondents who preferably select extreme categories are assigned to class 3 (12%), and those who prefer to select adjacent categories are assigned to class 1 (46%). The remaining respondents are assigned to class 2. To validate the ERS (AdjRS) class from model B_RS, we related these classes with an ERS (AdjRS) index based on all 55 five-point Likert-type items from the SPVA survey. The interitem correlation is .066 (variance = .026), indicating no substantive overall relationship between the items. As suggested by Greenleaf (1992), an ERS index is obtained as the proportion of items answered in the extreme categories (one or five). Moreover, the AdjRS index is computed as the proportion of items answered in the adjacent categories (two or four). The ERS and AdjRS indices correlate highly with the style factor, that is .85 and –.84 (both significant at p < .001) respectively, demonstrating the validity of our interpretation of the factor.

Data Collection

Cognitive interviews were held from December 2009 until May 2010 among 24 interviewees, recruited via different contact persons, organizations, and personal contacts to avoid overlap in social background. Due to privacy restrictions, it was not possible to contact the original respondents from the large-scale survey. We aimed for a heterogeneous sample (see Table 2), as we found from study 1 that the style of responding was not systematically related to socioeconomic characteristics. The interviews were conducted in Dutch and lasted 45 minutes on average. During eight interviews, a research assistant was present who also helped with transcribing and coding. Five interviews were excluded from further analyses because two were used as a pilot, two were conducted within the same household, and one was conducted in another language using a translator.

We first ask the interviewees to answer the survey questions used in study 1 in a similar fashion of a regular survey interview: The respondent is asked to select one answer category that reflects his or her opinion best without providing further explanation. Based on the responses that interviewees give to the survey questions, we are able to apply the results obtained in study 1 to determine whether they use a particular response style. After the interviewees have given their initial response, we probe them retrospectively about a selection of questions proven to elicit inequivalent answers among minorities in study 1. We also probe the questions to which the interviewees show problems in the first interview phase. In general, less-educated respondents had difficulty understanding the purpose of the cognitive interviews. More specifically, they could not see why they should explain their answers given during the first interview phase. Presumably, a straightforward think-aloud interview would have been more suitable, but we persisted with the two-phase interview to maintain comparability with the other interviews.

Coding Scheme

All interviews were recorded and transcribed by the first investigator and a trained research assistant. To analyze the qualitative data from the interviews, we developed a code list by integrating data- and theory-driven codes. First, we approached the interview transcripts by open coding (Miles and Huberman 1994). However, the resulting coding scheme was not helpful in answering the research questions posed earlier. Therefore, we coded the interviews again, now including thematic codes based on the theory of psychology of the survey response (Tourangeau et al. 2000). We paid special attention to elements indicated by previous research to the widespread occurrence of ERS in cross-cultural research: (1) sociocultural characteristics of respondents such as language use, acculturation (Gibbons et al. 1999; Marin et al. 1992); and (2) the type and content of questions (i.e., number of categories, labeling, language; Hui and Triandis 1989; Ralston et al. 1995). The integration of both coding schemes was considered satisfactory and after several debriefings among members of the research team, this led to the final version of the code list.

Classification of the Interviewees in the Cognitive Interviews

We integrate quantitative and qualitative methods by calculating individual latent scores for the interviewees in the cognitive interviews based on their responses to the 15 items in Table 1 and the posterior probabilities (i.e., the probability belonging to a class given these responses) inferred from the analysis of the survey data. For two important reasons, we do not use the best-fitting model B_RS for obtaining the posterior probabilities but instead use the most parsimonious model model C_RS that assumes scalar equivalence.

OPEN IN VIEWER

Table 1.

Mean Observed Item Response per Ethnic Group (N = 3,549).

		Turks	Moroccans	Surinamese	Antilleans
Factor 1	Attitude toward the Netherlands
Item 1	In the Netherlands immigrants get many opportunities	2.48 (1.06)	2.58 (1.07)	2.74 (1.11)	2.75 (1.15)
Item 2	The Netherlands is hostile to immigrantsa	3.20 (1.02)	3.53 (0.88)	3.60 (0.88)	3.48 (0.91)
Item 3	In the Netherlands your civil rights as an immigrant are respected	2.97 (0.97)	2.53 (0.91)	2.31 (0.89)	2.40 (0.91)
Item 4	The Netherlands is a hospitable country for immigrants	2.60 (0.90)	2.44 (0.85)	2.48 (0.86)	2.56 (0.84)
Item 5	The Netherlands is tolerant toward foreign cultures	2.17 (0.91)	2.43 (0.87)	2.16 (0.82)	2.31 (0.82)
Factor 2	Autonomy of children
Item 6	Children should live at home until marriage	2.30 (1.04)	2.24 (1.12)	3.07 (1.27)	3.41 (1.23)
Item 7	Elderly must be able to move in with their children	2.88 (1.13)	2.21 (0.96)	2.90 (1.15)	2.99 (1.17)
Item 8	Adult children should be able to move in with their parents	2.11 (0.88)	2.06 (0.85)	2.68 (1.08)	2.87 (1.11)
Item 9	Parents always have to be respected, even if they do not deserve it based on their behavior or attitude	1.89 (0.83)	1.80 (0.89)	2.39 (1.11)	2.30 (1.09)
Item 10	Older family members should have more influence in important decisions (for instance, about moving) than younger ones	2.89 (1.15)	2.63 (1.12)	3.15 (1.12)	3.12 (1.09)
Factor 3	Family values
Item 11	A man and woman are allowed to live together without being married	3.46 (1.25)	3.88 (1.11)	2.10 (1.04)	2.05 (1.04)
Item 12	Married people with children should not be allowed to divorcea	2.88 (1.19)	3.33 (1.14)	3.36 (1.09)	3.59 (1.05)
Item 13	The best family remains: Two married parents with childrena	2.46 (1.09)	1.94 (0.93)	2.53 (1.22)	2.60 (1.21)
Item 14	A daughter aged 17 is allowed to live by herself	4.00 (0.93)	4.06 (0.95)	3.59 (1.04)	3.40 (1.14)
Item 15	The opinion of the parents has to be important in the choice of a partner for their childa	2.54 (1.04)	2.52 (1.09)	3.36 (1.11)	3.53 (1.07)
n (unweighted)		905	854	1,014	776
Response rate (%)		52	52	44	51

Note: ^aItems 2, 12, 13, and 15 are formulated in reverse, where a positive answer indicates a conservative attitude. For the other items, a positive answer indicates a modern attitude. The standard deviations are in parentheses. Totally agree is indicated by 1, agree by 2, neither agree nor disagree by 3, disagree by 4, totally disagree by 5.

OPEN IN VIEWER

Table 2.

The Classification of a Selection of Interviewees in the Cognitive Interviews.

	Demographics					Response Pattern
Interviewee	Minority	Education	Age	Immigration	Gender	# Extr.	# Adj.	Response Style
M3	Moroccan	Intermediate	20	2	F	7	1	ERS
A10	Antillean	Low	65	1	F	10	5	ERS
A9	Antillean	Low	49	1	F	7	4	ERS
A11	Antillean	Intermediate	53	2	M	5	5	None
T7	Turkish	High	29	2	M	7	6	None
S14	Surinamese	High	32	2	F	5	7	None
M4	Moroccan	High	29	1	M	5	7	None
T6	Turkish	High	21	2	M	4	10	None
S15	Surinamese	High	27	1	F	3	7	None
M5	Moroccan	High	29	2	M	1	7	None
T8	Turkish	High	22	2	F	1	7	None
M2	Moroccan	High	26	2	F	2	11	None
M1	Moroccan	Low	26	1	F	2	13	None
S16	Surinamese	High	25	2	M	2	12	None
T17	Turkish	High	27	1	F	0	8	AdjRS
S22	Surinamese	High	37	2	F	0	12	AdjRS
S20	Surinamese	Intermediate	53	1	F	0	13	AdjRS
S21	Surinamese	Intermediate	55	1	F	0	12	AdjRS
A12	Antillean	Intermediate	42	1	M	0	14	AdjRS

Note: Interviewees are labeled by the first letter of their ethnicity and a number. Lower education includes primary education, the lower level of high school, and the lower level of professional education. Intermediate education includes intermediate level of high school and intermediate level of professional education. High education includes high level of high school and higher professional and academic education. Immigration stands for the generation that immigrated, # stands for the number of extreme or midpoint responses to the 15 items.

First, the number of parameters increases considerably when allowing for group differences in models A_RS and B_RS. This makes the classification of respondents according to model B_RS more uncertain than the classification of respondents according to the more simple model C_RS. Second, great uncertainty exists with respect to the nature of the resulting sample as we use purposive sampling in the qualitative part of the study. This means that using a relatively simple and parsimonious model may be more appropriate. Based on these considerations, we decided to use the parameters from model C_RS for the classification of respondents in study 2. The class assignments for the response style factor serve as the basis for the comparison among the respondents’ explanations for their answers.

Table 2 reports the estimated response style of the interviewees, the number of extreme and adjacent responses in their response patterns, and demographic information. The interviewees are presented in the order of the assigned positions on the RS style factor.

Extreme and Adjacent Responders

To understand why some respondents employ a response style and others do not, a clear understanding of the response process is crucial. Presumably, many respondents have problems with the attitude statements because they leave a lot of space for interpretation. We observe that respondents narrow down the possible interpretations to answer these items by: (1) having a critical attitude toward the item wording or survey process in general; (2) carefully weighing arguments in mapping the answers; or (3) excluding personal information in the arguments. Apparently, these response strategies are used as heuristic tools to solve a problem by finding the best possible answer (Krosnick 1991), also referred to as an educated guess (Tversky and Kahneman 1974). The most important difference between the respondents with and without a response style was that the first group applies these response strategies systematically to all questions. The use of language proves to be a complicating factor: Some respondents do not clearly distinguish between extreme and adjacent categories. Below we describe the response strategies.

Critical Attitude

Findings indicate that adjacent responders are very critical toward the items and the survey process. Interviewees T17 and S22 focus on the auxiliary verbs used in each item; interviewee A12 repeatedly expresses distrust toward the survey process in general and to the formulation of the items specifically: “All these questions are trick questions.” Interviewee S20 persistently argues throughout the interview that she does not want to exclude the possibility of other norms and values. Compared to other interviewees, adjacent responders are more sensitive and extreme responders are less sensitive toward the auxiliary verbs that determine the grammatical mood of the item. The problem addressed about “have to” is that—although a person can relate to the item content—it is possible to disagree with the obligation expressed by this auxiliary verb. The verbs “should” or “ought to” are problematic for some interviewees because they feel that they cannot agree or disagree with a recommendation expressed by this auxiliary verb: “Who are we to judge other people’s lives?” Finally, the auxiliary verb “is allowed to” is viewed by some interviewees as a limitation to a person’s freedom to act according to his or her own standards. The differences in sensitivity toward these auxiliary verbs between adjacent and extreme responders and the other interviewees are illustrated in Table 3.

OPEN IN VIEWER

Table 3.

Interviewees’ Inconvenience, with the Auxiliary Verbs Used in the Questions (Expressed in %).

	Auxiliary Words
Interviewee	Obligationa	Recommendationb	Possibilityc
Adjacent responders
T17	10	7	3
S22	14	5	5
S20	8	0	0
Extreme responders
M3	3	0	0
A9	2	0	0
A10	0	0	0
Without response style
T6	1	2	0
T7	4	0	0
A11	6	0	1

Note: The numbers are obtained by dividing the words used to address the auxiliary verbs in the survey questions by the total number of words used during the second phase of the interview (to correct for the individual differences of the total number of words).

^aObligation is expressed by the auxiliary “have to” or “must.”

^bRecommendation is expressed by the auxiliaries “should” or “ought to.”

^cPossibility is expressed by “could,” “might,” or “is allowed to.”

Table 3 reports the percentage of words used by nine interviewees to address their inconvenience with the auxiliary words in the question. On average, the extreme responders spent 1.9% of the interview on explaining the influence of auxiliary verbs; adjacent responders spent 15.2%; and the other interviewees spent 4.7%. Although Gibbons et al. (1999) find that meaningfulness of item wording is associated with the language in which the interview is conducted, we observe no systematic relation between language proficiency and sensitivity to auxiliary verbs.

Careful versus Abrupt Argumentation

In line with our previous findings, adjacent responders respond more carefully than extreme responders: They contemplate the multiple meanings of words and select a category, often reluctantly, after weighing their arguments. Two adjacent responders in our sample feel ambiguous toward the items and find it difficult to select a response category that reflects their opinion. Interviewee T17 has doubts about selecting categories because she feels that many answers are possible. For example, she discusses her answer to item 2 as follows: “If you take everything in consideration, I believe that—during the last few years—we have evolved in that direction.” After hesitation, she selects the midpoint category as she cannot know how this process will proceed. Interviewee A12 is annoyed with the vague formulation of the questions and responds to these questions by emphasizing that not all minorities are treated the same: “Some cultures yes, other cultures no.” However, many interviewees without a response style express similar problems with the item wording and also elaborate on the precise meaning they attach to the words. But, unlike the extreme and adjacent responders, they do not systematically select the same response categories.

The extreme responders react rather emotionally and abruptly instead of carefully weighing their arguments. For instance, in response to item 2, an extreme responder (M3) becomes agitated about the aggressive language used addressed to minorities by people in the street and at a national level by certain Dutch politicians. Interviewee A9 argues differently but with equal intensity to the same item: “If the Netherlands would be hostile toward foreigners, do you think I would get a house? No! Do you think I would get an allowance? No!” In general, the extreme responders give strong reactions to the item content without considering the precise item wording.

Personal Argumentation

Even though respondents with AdjRS and without a response style display sensitivity toward the wording of the questions, the findings indicate that they respond differently to the item content. Throughout the interview, one adjacent responder shuns giving any personal information, while another adjacent responder only once relates to his personal situation but in a rather hypothetical way: He argues that he would take care of his parents if they didn't live in Curaçao (A12). On the other hand, the other interviewees often explain their answers using personal experiences. Below we compare two interviewees’ answers to item 15:

Although these answers look very similar, interviewee M2 thinks of people in her surroundings and gives several examples subsequently (not shown here). In contrast, interviewee T17 (subject to AdjRS) refers to her personal surroundings in a more abstract way; she does not—explicitly—integrate her personal experiences into her judgment. We argue that interviewee T17 simplifies the response process by systematically excluding her personal experiences from her answers. As interviewee M2 thinks about her personal experiences, she makes an effort to regard each question separately. These findings may be generalized to the other interviewees, as the percentages in Table 4 illustrate.

OPEN IN VIEWER

Table 4.

Interviewees’ Personal and General Arguments (Expressed in %).

	Argumentation
Interviewee	Personala	Generalb
Adjacent responders
T17	8	50
A12	2	91
S20	10	49
S21	25	25
A22	12	26
Without response style
S14	13	52
S16	13	54
M4	17	58
T7	19	56
M1	25	33
S15	26	49
M2	32	61
A11	35	11
T6	36	22
T8	37	54
M5	56	34

Note: The numbers are obtained by dividing the words used to address the general or personal arguments by the total number of words used during the second phase of the interview.

^aPersonal arguments include comments about the personal situation or the cultural background.

^bGeneral arguments refer to comments about abstract situations.

As seen in Table 4, adjacent responders spend the smallest amount of words on personal arguments. This leads us to suggest that including personal arguments in the answers impedes AdjRS, which is illustrated by interviewee M4 who retrospectively changes his answer to “totally agree”: “I didn’t think of a family situation at first, [. . .] that is the difference I guess. So earlier I answered without considering my own family and when I think about it now, I come to a different conclusion.” Adjacent responders S20 and S21 do not fit into the pattern by equally using personal and general arguments while avoiding extreme categories altogether (see language). Extreme responders are excluded from Table 4 because we could not detect a general pattern: Some extreme responders use more personal arguments; others use more general arguments.

Mapping

When we probe about mapping judgments to the response scale (i.e., the selection of response categories that best reflect the judgment), extreme responders react differently than adjacent responders and interviewees without a response style:

As can be seen in Box 1, interviewees mention heuristics in how they deal with response categories: Extreme responders have rather fixed ideas about their response pattern, while adjacent responders use a more nuanced language. These results suggest that extreme responders reflect less often on their own response behavior than other interviewees who select extreme categories occasionally, although they prefer nuance.

Adjacent responders claim they do not like extreme categories because they do not want to exclude contradicting situations, arguments, or people’s behavior. Six interviewees without a response style argue similarly but nonetheless occasionally select extreme categories, mostly to family-related questions (10 of the 12) or in addressing personal situations (8 of the 10). Other aspects of the response process also seem to influence the tendency to select or to avoid extreme categories. Extreme responders are less articulate about the reasons for selecting categories, and, after probing, two of the three extreme responders are unable to explain their extreme answers. In contrast, adjacent responders consistently avoid the extreme categories.

Language

Among the less-educated ethnic Antillean and Surinamese interviewees, seven of the nine interviewees do not clearly differentiate between extreme and adjacent categories. Presumably, this can be explained by cultural differences in language usage. Van de Vijver and Phalet (2004) suggest that—similar to African American respondents (Bachman and O’Malley 1984)—Surinamese respondents may have developed their own variation on the Dutch language, which has stronger connotations. The languages of these groups, Papiamento and Surinaams (Sranantongo), are rather expressive. Being used to expressing themselves more extremely in their mother tongue may lead interviewees to regard the difference between “totally agree” and “agree” less strongly than the other minorities. For example, interviewee S16 argues: “Actually, I don’t really care whether I agree or totally agree.” Interviewees A10 and A9 do not differentiate between the extreme and adjacent categories, which could also be connected to their response style, as both are extreme responders. In short, 8 of the 10 ethnic Surinamese and Antillean respondents either explain vaguely or do not explain at all how they perceive the difference between extreme and adjacent categories.

In line with Gibbons et al. (1999) and Hui and Triandis (1989), we suggest that cultural background influences susceptibility to a response style via the meaningfulness of language. In the Antillean and Surinamese cultures, language is used in a more extreme way than in the Dutch culture. This may induce these respondents to overlook the difference between extreme and adjacent categories. In contrast, of the 10 ethnic Moroccans and Turkish interviewees, only 1 remarks that he does not perceive a clear difference between the categories.