Exploring the Response Process of Culturally Differing Survey Respondents with a Response Style

Abstract

This article presents a mixed methods approach that integrates quantitative and qualitative methods to analyze why the four largest minorities in the Netherlands—Turks, Moroccans, Antilleans, and Surinamese—respond differently to items treating cultural topics. First, we conducted latent class factor analyses on a large data set to assess whether these minorities respond differently to the items and to distinguish respondents with different types of response styles. These analyses indicate that group differences between minorities can partially be explained by response styles and that some respondents prefer to select the extreme categories whereas others prefer adjacent categories. Then, we selected interviewees from the same cultural groups and classified them according to the response profiles derived from the quantitative study. The qualitative study shows that extreme responders use different strategies to answer survey questions than do adjacent responders.

Keywords

mixed methods latent class factor analysis cognitive interviews response styles

Studies on people’s values, attitudes, and opinions use surveys as the key instrument to measure these characteristics. Survey instruments intended for making comparisons among national cultures or cultural groups within nations should be equally applicable across cultures to validly compare attitudes. Unfortunately, it has been shown repeatedly that survey results are not always comparable across cultures (e.g., Van de Vijver and Leung 1997). Many studies illustrate that response differences to attitude questions—particularly those dealing with culturally sensitive topics—can be attributed to cultural differences in language (Gibbons et al. 1999), acculturation (Marin et al. 1992), or response styles (e.g., Harzing 2006). For instance, Moors (2004) has demonstrated that due to a heterogeneous response behavior, survey responses among the Moroccan and Turkish minorities in Belgium are incomparable.

To explore such group differences in response behavior, one ideally needs statistical methods that simultaneously measure the underlying attitudes and detect the group specific response behavior. In this article, we use the latent class factor approach (LCFA) to take into account group differences in response styles as well as in attitudes (Moors 2003; Morren et al. 2011). Other similar approaches can also detect and correct for these group differences (Meade and Lautenschlager 2004; Steenkamp and Baumgartner 1998). Strong points of LCFA are that strict statistical assumptions need not be met for its application and variables of different measurement levels to be included in a model. For detailed information about the statistical properties, we refer to the web appendix (available at http://fmx.sagepub.com/). Although LCFA provides insight into the magnitude and the correlates of this differential response behavior, it does not clarify how the response processes underlying the differential response behavior operate. Therefore, we integrate statistical modeling and cognitive interviewing: In study 1, we estimate a statistical model on data from a large-scale nationally representative survey to detect the magnitude of group differences in responding. In study 2, we conduct cognitive interviews in a small purposive sample of respondents to study the response process in more detail. We integrate both data sets by inferring the response style of the interviewees from the statistical model and comparing the interviewees’ explanations for their response behavior in the cognitive interview accordingly. In this particular application of the mixed method approach, we aim to find an answer to the following research questions:

Which respondents endorse what type of response style, and, specifically, is response style usage systematically related to cultural background?

Is the response process for respondents with a particular response style systematically different from the response process of other respondents?

Study 1: Analysis of Large-Scale Survey Data to Assess Measurement Equivalence and Detect Response Style

Latent Variable Modeling to Assess Cross-Cultural Comparability

Within the latent variable modeling approach, attitudes are approached as complex theoretical constructs for which multiple empirical indicators reflecting several important aspects of the attitude are needed for a valid and reliable measurement (Bollen 2002). An attitude is modeled as a latent—unobserved—variable (also called factor or trait) for which survey questions (hereafter called items) are the indicators or observed variables. Covariates are included in this basic modeling approach to investigate differences between groups, as shown in Figure 1.

Figure 1.

The latent variable model and measurement equivalence.

Here, Y ₁ to Y ₁₅ represent the item responses that are directly related to the latent variables measuring the attitudes F ₁, F ₂, and F ₃, and indirectly related to the observed group variable G. The systematic variance among the item responses is captured by the factor loadings β_1j, β_2j, and β_3j (index j refers to an item). It is assumed that the five items in the first item subset do not relate directly to the second and third attitude; that is, the item parameters β_2j and β_3j are fixed to zero. Likewise, the parameters β_1j and β_3j are fixed to zero for items Y ₆ to Y ₁₀ and parameters β_1j and β_2j are fixed to zero for items Y ₁₁ to Y ₁₅.

If respondents with similar attitude levels respond differently to the items given his or her group membership, the measurements are inequivalent (model A) or partially equivalent (model B). Measurements inferred from respondents whose responses are similar irrespective of their group membership are scalar equivalent (model C). Note that measurement equivalence allows for group differences in attitudes but not for group differences in the measurement of these attitudes. Formally, model A allows for group differences in both factor loadings and intercepts, model B for group differences in intercepts, and model C restricts the responses to be equivalently related to the attitudes across groups. In Figure 1, violation of scalar equivalence is indicated by the direct effect β_{05 g} (dashed arrow) of G on item Y ₅, and a violation of metric equivalence is indicated by the direct effect β_{05 g} and the interaction effect on the relationship β_{15 g} between item Y ₅ and factor F ₁, meaning that G moderates the association between the fifth item and the first factor. We test for scalar equivalence by fixing the group-specific parameters β_05 g and β_15 g to zero; to test for metric equivalence, only β_15g is fixed to zero. Even though in Figure 1, the group parameters are only indicated for item Y ₅, the group variable G is usually assumed to affect all items simultaneously.

We applied study 1 to a secondary data set collected in 2002 among heads of the households in the four largest ethnic minorities in the Netherlands, namely Turkish, Moroccan, Surinamese, and Antillean people. While most people immigrated for economical and educational reasons, Surinamese and Antilleans came from Dutch colonies and were therefore more acquainted with the Dutch language, culture, and heritage than the Turks or Moroccans (CBS 2005). We include 15 items from this Social Position and Utility Use of Ethnic Minorities (SPVA) survey, each having five ordered response categories that range from totally agree to totally disagree. These items operationally define three attitudes, namely attitude toward the Netherlands, endorsement of traditional family values, and autonomy of children within the family. Given the substantive nature of these constructs, the groups are expected to be subject to culturally specific sources of measurement error and possibly to derive their answers from diverse response processes.

To investigate explanations for violations of measurement equivalence, we control for the following covariates: language use during the interview to measure meaningfulness of language (Gibbons et al. 1999); generation of immigration to measure acculturation (Marin et al. 1992); age, gender, level of income, and education (Light et al. 1965). Following Moors (2003), we also include a latent response style factor to control for heterogeneous response behavior (i.e., the respondents’ tendency to answer differently than expected based on their attitudes).

In all models, the effect of attitudes on item responses is distinguished from the effects of the control variables by estimating models in which only one item subset is affected by one attitude, while all items are simultaneously affected by the control variables. By comparing the statistical fit of models with and without these covariates, we investigate to what extent the covariates form a reasonable explanation of measurement inequivalence. To compare between models, we use the Bayesian Information Criterion (BIC), which introduces a penalty for the sample size and the number of parameters (Burnham and Anderson 2004) and is the most widely used measure for model selection in latent class analysis (Magidson and Vermunt 2004).

The best model in terms of fit and parsimony has the lowest value of BIC. According to the BIC, only the model including the response style factor explains part of the measurement inequivalence; the models including the other control variables remain measurement inequivalent. For an extensive description of these models and the results, see the web appendix (available at http://fmx.sagepub.com/).

First, without controlling for a response style, the ethnic minorities interpret the items differently given their attitude; close inspection of the parameters shows that these differences are visible in the intercepts as well as the factor loadings that indicates full measurement inequivalence (also indicated by model A having the lowest BIC value). Furthermore, we find that the style factor accounts for the group differences in the factor loadings: The BIC values decrease between model A_RS and model B_RS, indicating that the model assuming metric equivalence should be preferred. Even though group differences in the intercepts (not reported here) also decrease substantially once a control for response style is included, scalar equivalence is not attained. Note that the cultural group membership also influences response style directly: The Surinamese respond significantly less extremely than the other groups; the other groups do not differ significantly in their style of responding (see web appendix, available at http://fmx.sagepub.com/).

Based on a post hoc interpretation of the parameters of the response style factor, we conclude that this style factor measures two related types of response style. First, a person who scores high on the style factor is more likely to select the extreme categories than other respondents with similar attitudes. We refer to this response style as the Extreme Response Style (ERS), defined as the tendency to select extreme categories irrespective of the item content across a large range of unrelated or weakly related survey questions. Second, close inspection of the parameters shows that a person who scores low on the style factor avoids the extreme categories and selects the adjacent categories agree and disagree. Here, this response style is referred to as the Adjacent Response Style (AdjRS) and should not be confused with the tendency to select the midpoint category (midpoint responding) or to select the categories around the midpoint category (response range; see Baumgartner and Steenkamp 2001).

The style factor is specified as a latent variable with three ordered classes, of which class 3 represents ERS and the class at the other endpoint (class 1) represents AdjRS. The middle class (class 2) defines a position in between these two endpoints. We interpret this position as a mild response style and treat these respondents as not endorsing any response style. Based on the item responses of a respondent, we can compute the probability of belonging to each of these three classes. Typically, we would assign a respondent to the most likely class. Practically, this means that respondents who preferably select extreme categories are assigned to class 3 (12%), and those who prefer to select adjacent categories are assigned to class 1 (46%). The remaining respondents are assigned to class 2. To validate the ERS (AdjRS) class from model B_RS, we related these classes with an ERS (AdjRS) index based on all 55 five-point Likert-type items from the SPVA survey. The interitem correlation is .066 (variance = .026), indicating no substantive overall relationship between the items. As suggested by Greenleaf (1992), an ERS index is obtained as the proportion of items answered in the extreme categories (one or five). Moreover, the AdjRS index is computed as the proportion of items answered in the adjacent categories (two or four). The ERS and AdjRS indices correlate highly with the style factor, that is .85 and –.84 (both significant at p < .001) respectively, demonstrating the validity of our interpretation of the factor.

Study 2: Cognitive Interviewing to Explore the Response Process of Individuals Who Use a Response Style

Data Collection

Cognitive interviews were held from December 2009 until May 2010 among 24 interviewees, recruited via different contact persons, organizations, and personal contacts to avoid overlap in social background. Due to privacy restrictions, it was not possible to contact the original respondents from the large-scale survey. We aimed for a heterogeneous sample (see Table 2), as we found from study 1 that the style of responding was not systematically related to socioeconomic characteristics. The interviews were conducted in Dutch and lasted 45 minutes on average. During eight interviews, a research assistant was present who also helped with transcribing and coding. Five interviews were excluded from further analyses because two were used as a pilot, two were conducted within the same household, and one was conducted in another language using a translator.

We first ask the interviewees to answer the survey questions used in study 1 in a similar fashion of a regular survey interview: The respondent is asked to select one answer category that reflects his or her opinion best without providing further explanation. Based on the responses that interviewees give to the survey questions, we are able to apply the results obtained in study 1 to determine whether they use a particular response style. After the interviewees have given their initial response, we probe them retrospectively about a selection of questions proven to elicit inequivalent answers among minorities in study 1. We also probe the questions to which the interviewees show problems in the first interview phase. In general, less-educated respondents had difficulty understanding the purpose of the cognitive interviews. More specifically, they could not see why they should explain their answers given during the first interview phase. Presumably, a straightforward think-aloud interview would have been more suitable, but we persisted with the two-phase interview to maintain comparability with the other interviews.

Coding Scheme

All interviews were recorded and transcribed by the first investigator and a trained research assistant. To analyze the qualitative data from the interviews, we developed a code list by integrating data- and theory-driven codes. First, we approached the interview transcripts by open coding (Miles and Huberman 1994). However, the resulting coding scheme was not helpful in answering the research questions posed earlier. Therefore, we coded the interviews again, now including thematic codes based on the theory of psychology of the survey response (Tourangeau et al. 2000). We paid special attention to elements indicated by previous research to the widespread occurrence of ERS in cross-cultural research: (1) sociocultural characteristics of respondents such as language use, acculturation (Gibbons et al. 1999; Marin et al. 1992); and (2) the type and content of questions (i.e., number of categories, labeling, language; Hui and Triandis 1989; Ralston et al. 1995). The integration of both coding schemes was considered satisfactory and after several debriefings among members of the research team, this led to the final version of the code list.

Classification of the Interviewees in the Cognitive Interviews

We integrate quantitative and qualitative methods by calculating individual latent scores for the interviewees in the cognitive interviews based on their responses to the 15 items in Table 1 and the posterior probabilities (i.e., the probability belonging to a class given these responses) inferred from the analysis of the survey data. For two important reasons, we do not use the best-fitting model B_RS for obtaining the posterior probabilities but instead use the most parsimonious model model C_RS that assumes scalar equivalence.

Table 1.

Mean Observed Item Response per Ethnic Group (N = 3,549).

		Turks	Moroccans	Surinamese	Antilleans
Factor 1	Attitude toward the Netherlands
Item 1	In the Netherlands immigrants get many opportunities	2.48 (1.06)	2.58 (1.07)	2.74 (1.11)	2.75 (1.15)
Item 2	The Netherlands is hostile to immigrants^a	3.20 (1.02)	3.53 (0.88)	3.60 (0.88)	3.48 (0.91)
Item 3	In the Netherlands your civil rights as an immigrant are respected	2.97 (0.97)	2.53 (0.91)	2.31 (0.89)	2.40 (0.91)
Item 4	The Netherlands is a hospitable country for immigrants	2.60 (0.90)	2.44 (0.85)	2.48 (0.86)	2.56 (0.84)
Item 5	The Netherlands is tolerant toward foreign cultures	2.17 (0.91)	2.43 (0.87)	2.16 (0.82)	2.31 (0.82)
Factor 2	Autonomy of children
Item 6	Children should live at home until marriage	2.30 (1.04)	2.24 (1.12)	3.07 (1.27)	3.41 (1.23)
Item 7	Elderly must be able to move in with their children	2.88 (1.13)	2.21 (0.96)	2.90 (1.15)	2.99 (1.17)
Item 8	Adult children should be able to move in with their parents	2.11 (0.88)	2.06 (0.85)	2.68 (1.08)	2.87 (1.11)
Item 9	Parents always have to be respected, even if they do not deserve it based on their behavior or attitude	1.89 (0.83)	1.80 (0.89)	2.39 (1.11)	2.30 (1.09)
Item 10	Older family members should have more influence in important decisions (for instance, about moving) than younger ones	2.89 (1.15)	2.63 (1.12)	3.15 (1.12)	3.12 (1.09)
Factor 3	Family values
Item 11	A man and woman are allowed to live together without being married	3.46 (1.25)	3.88 (1.11)	2.10 (1.04)	2.05 (1.04)
Item 12	Married people with children should not be allowed to divorce^a	2.88 (1.19)	3.33 (1.14)	3.36 (1.09)	3.59 (1.05)
Item 13	The best family remains: Two married parents with children^a	2.46 (1.09)	1.94 (0.93)	2.53 (1.22)	2.60 (1.21)
Item 14	A daughter aged 17 is allowed to live by herself	4.00 (0.93)	4.06 (0.95)	3.59 (1.04)	3.40 (1.14)
Item 15	The opinion of the parents has to be important in the choice of a partner for their child^a	2.54 (1.04)	2.52 (1.09)	3.36 (1.11)	3.53 (1.07)
n (unweighted)		905	854	1,014	776
Response rate (%)		52	52	44	51

Note: ^aItems 2, 12, 13, and 15 are formulated in reverse, where a positive answer indicates a conservative attitude. For the other items, a positive answer indicates a modern attitude. The standard deviations are in parentheses. Totally agree is indicated by 1, agree by 2, neither agree nor disagree by 3, disagree by 4, totally disagree by 5.

Table 2.

The Classification of a Selection of Interviewees in the Cognitive Interviews.

	Demographics					Response Pattern
Interviewee	Minority	Education	Age	Immigration	Gender	# Extr.	# Adj.	Response Style
M3	Moroccan	Intermediate	20	2	F	7	1	ERS
A10	Antillean	Low	65	1	F	10	5	ERS
A9	Antillean	Low	49	1	F	7	4	ERS
A11	Antillean	Intermediate	53	2	M	5	5	None
T7	Turkish	High	29	2	M	7	6	None
S14	Surinamese	High	32	2	F	5	7	None
M4	Moroccan	High	29	1	M	5	7	None
T6	Turkish	High	21	2	M	4	10	None
S15	Surinamese	High	27	1	F	3	7	None
M5	Moroccan	High	29	2	M	1	7	None
T8	Turkish	High	22	2	F	1	7	None
M2	Moroccan	High	26	2	F	2	11	None
M1	Moroccan	Low	26	1	F	2	13	None
S16	Surinamese	High	25	2	M	2	12	None
T17	Turkish	High	27	1	F	0	8	AdjRS
S22	Surinamese	High	37	2	F	0	12	AdjRS
S20	Surinamese	Intermediate	53	1	F	0	13	AdjRS
S21	Surinamese	Intermediate	55	1	F	0	12	AdjRS
A12	Antillean	Intermediate	42	1	M	0	14	AdjRS

Note: Interviewees are labeled by the first letter of their ethnicity and a number. Lower education includes primary education, the lower level of high school, and the lower level of professional education. Intermediate education includes intermediate level of high school and intermediate level of professional education. High education includes high level of high school and higher professional and academic education. Immigration stands for the generation that immigrated, # stands for the number of extreme or midpoint responses to the 15 items.

First, the number of parameters increases considerably when allowing for group differences in models A_RS and B_RS. This makes the classification of respondents according to model B_RS more uncertain than the classification of respondents according to the more simple model C_RS. Second, great uncertainty exists with respect to the nature of the resulting sample as we use purposive sampling in the qualitative part of the study. This means that using a relatively simple and parsimonious model may be more appropriate. Based on these considerations, we decided to use the parameters from model C_RS for the classification of respondents in study 2. The class assignments for the response style factor serve as the basis for the comparison among the respondents’ explanations for their answers.

Table 2 reports the estimated response style of the interviewees, the number of extreme and adjacent responses in their response patterns, and demographic information. The interviewees are presented in the order of the assigned positions on the RS style factor.

Results

Extreme and Adjacent Responders

To understand why some respondents employ a response style and others do not, a clear understanding of the response process is crucial. Presumably, many respondents have problems with the attitude statements because they leave a lot of space for interpretation. We observe that respondents narrow down the possible interpretations to answer these items by: (1) having a critical attitude toward the item wording or survey process in general; (2) carefully weighing arguments in mapping the answers; or (3) excluding personal information in the arguments. Apparently, these response strategies are used as heuristic tools to solve a problem by finding the best possible answer (Krosnick 1991), also referred to as an educated guess (Tversky and Kahneman 1974). The most important difference between the respondents with and without a response style was that the first group applies these response strategies systematically to all questions. The use of language proves to be a complicating factor: Some respondents do not clearly distinguish between extreme and adjacent categories. Below we describe the response strategies.

Critical Attitude

Findings indicate that adjacent responders are very critical toward the items and the survey process. Interviewees T17 and S22 focus on the auxiliary verbs used in each item; interviewee A12 repeatedly expresses distrust toward the survey process in general and to the formulation of the items specifically: “All these questions are trick questions.” Interviewee S20 persistently argues throughout the interview that she does not want to exclude the possibility of other norms and values. Compared to other interviewees, adjacent responders are more sensitive and extreme responders are less sensitive toward the auxiliary verbs that determine the grammatical mood of the item. The problem addressed about “have to” is that—although a person can relate to the item content—it is possible to disagree with the obligation expressed by this auxiliary verb. The verbs “should” or “ought to” are problematic for some interviewees because they feel that they cannot agree or disagree with a recommendation expressed by this auxiliary verb: “Who are we to judge other people’s lives?” Finally, the auxiliary verb “is allowed to” is viewed by some interviewees as a limitation to a person’s freedom to act according to his or her own standards. The differences in sensitivity toward these auxiliary verbs between adjacent and extreme responders and the other interviewees are illustrated in Table 3.

Table 3.

Interviewees’ Inconvenience, with the Auxiliary Verbs Used in the Questions (Expressed in %).

	Auxiliary Words
Interviewee	Obligation^a	Recommendation^b	Possibility^c
Adjacent responders
T17	10	7	3
S22	14	5	5
S20	8	0	0
Extreme responders
M3	3	0	0
A9	2	0	0
A10	0	0	0
Without response style
T6	1	2	0
T7	4	0	0
A11	6	0	1

Note: The numbers are obtained by dividing the words used to address the auxiliary verbs in the survey questions by the total number of words used during the second phase of the interview (to correct for the individual differences of the total number of words).

^aObligation is expressed by the auxiliary “have to” or “must.”

^bRecommendation is expressed by the auxiliaries “should” or “ought to.”

^cPossibility is expressed by “could,” “might,” or “is allowed to.”

Table 3 reports the percentage of words used by nine interviewees to address their inconvenience with the auxiliary words in the question. On average, the extreme responders spent 1.9% of the interview on explaining the influence of auxiliary verbs; adjacent responders spent 15.2%; and the other interviewees spent 4.7%. Although Gibbons et al. (1999) find that meaningfulness of item wording is associated with the language in which the interview is conducted, we observe no systematic relation between language proficiency and sensitivity to auxiliary verbs.

Careful versus Abrupt Argumentation

In line with our previous findings, adjacent responders respond more carefully than extreme responders: They contemplate the multiple meanings of words and select a category, often reluctantly, after weighing their arguments. Two adjacent responders in our sample feel ambiguous toward the items and find it difficult to select a response category that reflects their opinion. Interviewee T17 has doubts about selecting categories because she feels that many answers are possible. For example, she discusses her answer to item 2 as follows: “If you take everything in consideration, I believe that—during the last few years—we have evolved in that direction.” After hesitation, she selects the midpoint category as she cannot know how this process will proceed. Interviewee A12 is annoyed with the vague formulation of the questions and responds to these questions by emphasizing that not all minorities are treated the same: “Some cultures yes, other cultures no.” However, many interviewees without a response style express similar problems with the item wording and also elaborate on the precise meaning they attach to the words. But, unlike the extreme and adjacent responders, they do not systematically select the same response categories.

The extreme responders react rather emotionally and abruptly instead of carefully weighing their arguments. For instance, in response to item 2, an extreme responder (M3) becomes agitated about the aggressive language used addressed to minorities by people in the street and at a national level by certain Dutch politicians. Interviewee A9 argues differently but with equal intensity to the same item: “If the Netherlands would be hostile toward foreigners, do you think I would get a house? No! Do you think I would get an allowance? No!” In general, the extreme responders give strong reactions to the item content without considering the precise item wording.

Personal Argumentation

Even though respondents with AdjRS and without a response style display sensitivity toward the wording of the questions, the findings indicate that they respond differently to the item content. Throughout the interview, one adjacent responder shuns giving any personal information, while another adjacent responder only once relates to his personal situation but in a rather hypothetical way: He argues that he would take care of his parents if they didn't live in Curaçao (A12). On the other hand, the other interviewees often explain their answers using personal experiences. Below we compare two interviewees’ answers to item 15:

Many questions you ask concern issues in our personal surroundings, parents often have a very, um, . . . well, yes, an opinion that interferes with their children’s opinion, and therefore I believe that [the influence of parents in the children’s choice] should not be important. (M2)

It plays a great role in our community, but I do not believe that that should be a leading role, so, um, . . . so, yes in my view they should have a voice but, umm, . . . not the choice itself [. . .], given my reasoning I think I agree. (T17)

Although these answers look very similar, interviewee M2 thinks of people in her surroundings and gives several examples subsequently (not shown here). In contrast, interviewee T17 (subject to AdjRS) refers to her personal surroundings in a more abstract way; she does not—explicitly—integrate her personal experiences into her judgment. We argue that interviewee T17 simplifies the response process by systematically excluding her personal experiences from her answers. As interviewee M2 thinks about her personal experiences, she makes an effort to regard each question separately. These findings may be generalized to the other interviewees, as the percentages in Table 4 illustrate.

Table 4.

Interviewees’ Personal and General Arguments (Expressed in %).

	Argumentation
Interviewee	Personal^a	General^b
Adjacent responders
T17	8	50
A12	2	91
S20	10	49
S21	25	25
A22	12	26
Without response style
S14	13	52
S16	13	54
M4	17	58
T7	19	56
M1	25	33
S15	26	49
M2	32	61
A11	35	11
T6	36	22
T8	37	54
M5	56	34

Note: The numbers are obtained by dividing the words used to address the general or personal arguments by the total number of words used during the second phase of the interview.

^aPersonal arguments include comments about the personal situation or the cultural background.

^bGeneral arguments refer to comments about abstract situations.

As seen in Table 4, adjacent responders spend the smallest amount of words on personal arguments. This leads us to suggest that including personal arguments in the answers impedes AdjRS, which is illustrated by interviewee M4 who retrospectively changes his answer to “totally agree”: “I didn’t think of a family situation at first, [. . .] that is the difference I guess. So earlier I answered without considering my own family and when I think about it now, I come to a different conclusion.” Adjacent responders S20 and S21 do not fit into the pattern by equally using personal and general arguments while avoiding extreme categories altogether (see language). Extreme responders are excluded from Table 4 because we could not detect a general pattern: Some extreme responders use more personal arguments; others use more general arguments.

Mapping

When we probe about mapping judgments to the response scale (i.e., the selection of response categories that best reflect the judgment), extreme responders react differently than adjacent responders and interviewees without a response style:

As can be seen in Box 1, interviewees mention heuristics in how they deal with response categories: Extreme responders have rather fixed ideas about their response pattern, while adjacent responders use a more nuanced language. These results suggest that extreme responders reflect less often on their own response behavior than other interviewees who select extreme categories occasionally, although they prefer nuance.

Box 1.

Interviewees Reflect on Their Own Response behavior

As Muslim [. . .], I always totally agree [with what Gods prescribes or forbids]” and “God created me [. . .] so if he gives you advice and you say no, I don’t agree, yes, that sounds funny. (M3)

I never choose extreme categories [. . .] I also think it is my character that I don't like to totally exclude things. (T17)

I am a free-thinking man. (A12)

I decide not to use totally agree or totally disagree because this leaves room for discussion, or debate or eh . . . interpretation so to say. (M5)

Adjacent responders claim they do not like extreme categories because they do not want to exclude contradicting situations, arguments, or people’s behavior. Six interviewees without a response style argue similarly but nonetheless occasionally select extreme categories, mostly to family-related questions (10 of the 12) or in addressing personal situations (8 of the 10). Other aspects of the response process also seem to influence the tendency to select or to avoid extreme categories. Extreme responders are less articulate about the reasons for selecting categories, and, after probing, two of the three extreme responders are unable to explain their extreme answers. In contrast, adjacent responders consistently avoid the extreme categories.

Language

Among the less-educated ethnic Antillean and Surinamese interviewees, seven of the nine interviewees do not clearly differentiate between extreme and adjacent categories. Presumably, this can be explained by cultural differences in language usage. Van de Vijver and Phalet (2004) suggest that—similar to African American respondents (Bachman and O’Malley 1984)—Surinamese respondents may have developed their own variation on the Dutch language, which has stronger connotations. The languages of these groups, Papiamento and Surinaams (Sranantongo), are rather expressive. Being used to expressing themselves more extremely in their mother tongue may lead interviewees to regard the difference between “totally agree” and “agree” less strongly than the other minorities. For example, interviewee S16 argues: “Actually, I don’t really care whether I agree or totally agree.” Interviewees A10 and A9 do not differentiate between the extreme and adjacent categories, which could also be connected to their response style, as both are extreme responders. In short, 8 of the 10 ethnic Surinamese and Antillean respondents either explain vaguely or do not explain at all how they perceive the difference between extreme and adjacent categories.

In line with Gibbons et al. (1999) and Hui and Triandis (1989), we suggest that cultural background influences susceptibility to a response style via the meaningfulness of language. In the Antillean and Surinamese cultures, language is used in a more extreme way than in the Dutch culture. This may induce these respondents to overlook the difference between extreme and adjacent categories. In contrast, of the 10 ethnic Moroccans and Turkish interviewees, only 1 remarks that he does not perceive a clear difference between the categories.

Conclusions

In this article, we have found that the incomparability of responses across culturally diverse groups can be partially explained by a response style. The response style itself was found to be systematically related to cultural groups. In study 1, we showed by analyzing a large data set quantitatively that this style factor measures a synthesis of avoiding extreme categories, on one hand, and a preference for extreme categories, on the other. As this response style could explain much of the variance in responding across the cultural groups, we further investigated this response style in study 2, using semistructured interviews in which we focused on the cognitive aspects of the response process. We integrated the quantitative and qualitative results by classifying the participants in the cognitive interviews according to the types of responders inferred from a model estimated on the large data set. Subsequently, we compared the arguments given in the cognitive interviews for the interviewees who exhibited a preference for extreme categories—the extreme responders—for adjacent categories—the adjacent responders—or without a clear preference.

Using this sequential mixed method design, we found that the response style is the outcome of different response strategies. First, extreme responders less precisely considered the item wording than adjacent responders and interviewees without a response style. In contrast, adjacent responders more carefully considered the precise item formulation than the other respondents. Second, adjacent responders carefully weighed arguments for and against when responding. In contrast, extreme responders and interviewees without a response style weighed their arguments to a lesser degree. Third, some adjacent responders persistently shunned personal information in their argumentation. Fourth, and most importantly, even though extreme responders, adjacent responders, and interviewees without a response style all displayed these response strategies to some extent, extreme and adjacent responders systematically translated these response strategies into selecting certain categories across a diverse range of questions. Finally, the findings suggest that extreme responding might be related to cultural heritage. Respondents with a Surinamese and Antillean ethnic background especially seem to disregard the difference between extreme and adjacent categories. However, this finding should be studied further as these conclusions are based on a small sample and must be interpreted with care.

This article illustrates that latent class factor models is a powerful and straightforward method to derive response profiles from large-scale surveys, which can subsequently be used to classify respondents who were not initially included in the survey but purposely selected. Furthermore, cognitive interviews can be used to evaluate items after the survey has been conducted. In this case, we recommend integrating the findings from the cognitive interviews with the survey findings by classifying the interviewees based on the model estimates.

We have shown that interviewees have problems with the vague formulation of the attitude statements. Complex syntaxes and auxiliary verbs cause special problems as they increase the number of meanings that can be attached to the questions. Interviewees can agree with one meaning and simultaneously disagree with another meaning. In the design of attitude statements, we recommend avoiding auxiliary verbs, as these were repeatedly mentioned by interviewees as creating problems in interpreting the entire sentence. We also recommend avoiding examples, as respondents tend to focus on these examples and do not respond to other possible fields to which the topic could apply. Next, make sure that the words in the questions are understood in the way that the researcher intends, as some words have double meanings, especially for the ethnic minorities. Finally, we recommend formulating the sentence as simply as possible to avoid using auxiliary verbs, examples, and subordinate clauses, as the use of subordinate clauses is confusing to many respondents.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This research was supported by a grant from the Netherlands Organization for Scientific Research (NWO), grant number 400-05-164.

References

Bachman

J. G.,

O’Malley

P. M.

. 1984. Yea-saying, nay-saying, and going to extremes: Black-white differences in response styles. Public Opinion Quarterly 48:491–509.

Baumgartner

H.,

Steenkamp

J.-B. E. M.

. 2001. Response styles in marketing research: A cross-national investigation. Journal of Marketing Research 38:143–56.

Bollen

K. A.

2002. Latent variables in psychology and the social sciences. Annual Review Psychology 53:605–34.

Burnham

K. P.,

Anderson

D. R.

. 2004. Multi-model inference. Understanding AIC and BIC in model selection. Sociological Methods and Research 33:261–304.

Central Bureau voor Statistiek (CBS). 2005. Jaarrapport Integratie 2005 (Annual report integration 2005). http://www.cbs.nl/nl-NL/menu/themas/dossiers/allochtonen/publicaties/publicaties/archief/2005/2005-jaarrapport-integratie.htm (accessed September 2010).

Gibbons

J. L.

Zellner

J. A.

Rudek

D. J.

. 1999. Effects of language and meaningfulness on the use of extreme response style by Spanish-English bilinguals. Cross-Cultural Research 33:369–81.

Greenleaf

E. A.

1992. Measuring extreme response style. Public Opinion Quarterly 56:328–51.

Harzing

A.-W.

2006. Response styles in cross-national survey research. International Journal of Cross-Cultural Management 6:243–66.

Hui

H. C.,

Triandis

H. C.

. 1989. Effects of culture and response format on extreme response style. Journal of Cross-Cultural Psychology 20:296–309.

10.

Krosnick

J. A.

1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology 5:213–36.

11.

Light

C. S.

Zax

Gardiner

D. H.

. 1965. Relationship of age, sex, and intelligence level to extreme response style. Journal of Personality and Social Psychology 2:907–9.

12.

Magidson

J.,

Vermunt

J. K.

. 2004. Latent class models. In The Sage handbook of quantitative methodology for the social sciences, ed. Kaplan

, 175–98. Thousand Oaks, CA: Sage.

13.

Marin

Gamba

R. J.

Marin

B. V.

. 1992. Extreme response style and acquiescence among Hispanics: The role of acculturation and education. Journal of Cross-Cultural Psychology 23:498–509.

14.

Meade

A. W.,

Lautenschlager

G. J.

. 2004. A comparison of item response theory and confirmatory factor analytic methodologies for establishing measurement equivalence/invariance. Organizational Research Methods 7:361–88.

15.

Miles

M. B.,

Huberman

M. A.

. 1994. An expanded sourcebook. Qualitative data analysis. Thousand Oaks, CA: Sage.

16.

Moors

2003. Diagnosing response style behavior by means of a latent-class factor approach. Socio-demographic correlates of gender role attitudes and perceptions of ethnic discrimination reexamined. Quality and Quantity 37:277–302.

17.

Moors

2004. Facts and artifacts in the comparison of attitudes among ethnic minorities. A multi-group latent class structure model with adjustment for response style behavior. European Sociological Review 20:303–20.

18.

Morren

Gelissen

J. P. T. M.

Vermunt

J. K.

. 2011. Dealing with extreme response style in cross-cultural research: A restricted latent class factor approach. Sociological Methodology 41:13–47.

19.

Ralston

D. A.

Cunniff

M. K.

Gustafson

D. J.

. 1995. Cultural accommodation. The effect of language on responses of bilingual Hong Kong Chinese managers. Journal of Cross-Cultural Psychology 26:714–27.

20.

Steenkamp

J.-B. B. E. M.,

Baumgartner

. 1998. Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research 25:78–90.

21.

Tourangeau

Rips

L. J.

Rasinski

K. A.

. 2000. The psychology of survey response. Cambridge: Cambridge University Press.

22.

Tversky

A.,

Kahneman

. 1974. Judgment under uncertainty: Heuristics and biases. Science 185:1124–31.

23.

Van de Vijver

F. J. R.,

Leung

. 1997. Methods and data analysis for cross-cultural research, vol. 1, eds. Lonner

W. J.

Berry

J. W.

. Thousand Oaks, CA: Sage.

24.

Van de Vijver

F. J. R.,

Phalet

. 2004. Assessment in multicultural groups: The role of acculturation. Applied Psychology: An International Review 53:215–36.

25.

Vermunt

J. K.,

Magidson

. 2008. LG-SyntaxTM user’s guide: Manual for Latent GOLD 4.5 syntax module. Belmont, MA: Statistical Innovations.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.06 MB