On Enhancing the Cross–Cultural Comparability of Likert–Scale Personality and Value Measures: A Comparison of Common Procedures

Abstract

This study aims to evaluate a number of procedures that have been proposed to enhance cross–cultural comparability of personality and value data. A priori procedures (anchoring vignettes and direct measures of response styles (i.e. acquiescence, extremity, midpoint responding, and social desirability), a posteriori procedures focusing on data transformations prior to analysis (ipsatization and item parcelling), and two data modelling procedures (treating data as continuous vs as ordered categories) were compared using data collected from university students in 16 countries. We found that (i) anchoring vignettes showed lack of invariance, so they were not bias–free; (ii) anchoring vignettes showed higher internal consistencies than raw scores where all other correction procedures, notably ipsatization, showed lower internal consistencies; (iii) in measurement invariance testing, no procedure yielded scalar invariance; anchoring vignettes and item parcelling slightly improved comparability, response style correction did not affect it, and ipsatization resulted in lower comparability; (iv) treating Likert–scale data as categorical resulted in higher levels of comparability; (v) factor scores of scales extracted from different procedures showed similar correlational patterning; and (vi) response style correction was the only procedure that suggested improvement in external validity of country–level conscientiousness. We conclude that, although no procedure resolves all comparability issues, anchoring vignettes, parcelling, and treating data as ordered categories seem promising to alleviate incomparability. We advise caution in uncritically applying any of these procedures. Copyright © 2017 European Association of Personality Psychology

Keywords

personality values anchoring vignettes response styles score standardization parcelling

Get full access to this article

View all access options for this article.

References

Bäckström

, & Björklund

(2014). Social desirability in personality inventories: The nature of the evaluative factor. Journal of Individual Differences, 35, 144–157 https://doi.org/10.1027/1614–0001/a000138.

Bäckström

, Björklund

, & Larsson

M. R.

(2014). Criterion validity is maintained when items are evaluatively neutralized: Evidence from a full–scale five–factor model inventory. European Journal of Personality, 28, 620–633 https://doi.org/10.1002/per.1960.

Bandalos

D. L.

(2002). The effects of item parceling on goodness–of–fit and parameter estimate bias in structural equation modeling. Structural Equation Modeling, 9, 78–102 https://doi.org/10.1207/S15328007SEM0901_5.

Barrett

P. T.

, Petrides

, Eysenck

S. B. G.

, & Eysenck

H. J.

(1998). The Eysenck personality questionnaire: An examination of the factorial similarity of P, E, N, and L across 34 countries. Personality and Individual Differences, 25, 805–819 https://doi.org/10.1016/S0191–8869(98)00026–9.

Billiet

J. B.

, & Davidov

(2008). Testing the stability of an acquiescence style factor behind two interrelated substantive variables in a panel design. Sociological Methods & Research, 36, 542–562 https://doi.org/10.1177/0049124107313901.

Bilsky

, Janik

, & Schwartz

S. H.

(2011). The structural organization of human values—Evidence from three rounds of the European social survey (ESS). Journal of Cross–Cultural Psychology, 42, 759–776 https://doi.org/10.1177/0022022110362757.

Bou Malham

, & Saucier

(2016). The conceptual link between social desirability and cultural normativity. International Journal of Psychology, 51, 474–480 https://doi.org/10.1002/ijop.12261.

Brown

T. A.

(2006). Confirmatory factor analysis for applied research. New York, NY: Guilford.

Byrne

B. M.

(2001). Structural equation modeling with AMOS: Basic concepts, applications, and programming. Mahwah, NJ: Lawrence Erlbaum Associates.

10.

Cheung

G. W.

, & Rensvold

R. B.

(2002). Evaluating goodness–of–fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233–255 https://doi.org/10.1207/s15328007sem0902_5.

11.

Church

A. T.

, Anderson–Harumi

C. A.

, del Prado

A. M.

, Curtis

G. J.

, Tanaka–Matsumi

, Valdez Medina

J. L.

, … White

F. A.

(2008). Culture, cross–role consistency, and adjustment: Testing trait and cultural psychology perspectives. Journal of Personality and Social Psychology, 95, 739–755 https://doi.org/10.1037/0022–3514.95.3.739.

12.

Church

T. A.

, Alvarez

J. M.

, Mai

N. T. Q.

, French

B. F.

, Katigbak

M. S.

, & Ortiz

F. A.

(2011). Are cross–cultural comparisons of personality profiles meaningful? Differential item and facet functioning in the Revised NEO Personality Inventory. Journal of Personality and Social Psychology, 101, 1068–1089. https://dx-doi-org.web.bisu.edu.cn/10.1037/a0025290.

13.

Crowne

D. P.

, & Marlowe

(1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24, 349–354. https://doi.org/10.1037/h0047358.

14.

Davidov

, Schmidt

, & Schwartz

S. H.

(2008). Bringing values back in: The adequacy of the European social survey to measure values in 20 countries. Public Opinion Quarterly, 72, 420–445 https://doi.org/10.1093/poq/nfn035.

15.

De Raad

, Barelds

D. P. H.

, Timmerman

M. E.

, De Roover

, Mlačić

, & Church

A. T.

(2014). Towards a pan–cultural personality structure: Input from 11 psycholexical studies. European Journal of Personality, 28, 497–510 https://doi.org/10.1002/per.1953.

16.

Eigenhuis

, Kamphuis

J. H.

, & Noordhof

(2015). Personality differences between the United States and the Netherlands: The influence of violations of measurement invariance. Journal of Cross–Cultural Psychology, 46, 549–564 https://doi.org/10.1177/0022022115570671.

17.

Ferrer–i–Carbonell

, van Praag

B. M. S.

, & Theodossiou

(2011). Vignette equivalence and response consistency: The case of job satisfaction. IZA Discussion Paper (No. 6174).

18.

Fischer

(2004). Standardization to account for cross–cultural response bias: A classification of score adjustment procedures and review of research in JCCP. Journal of Cross–Cultural Psychology, 35, 263–282 https://doi.org/10.1177/0022022104264122.

19.

Fischer

(2014). What values can (and cannot) tell us about individuals, society and culture. In Gelfand

M. J.

, Chiu

, & Hong

(Eds.), Advances in culture and psychology (pp. 218–272), 4. Oxford, United Kingdom: Oxford University Press.

20.

Friborg

, Martinussen

, & Rosenvinge

J. H.

(2006). Likert–based vs. semantic differential–based scorings of positive psychological constructs: A psychometric comparison of two versions of a scale measuring resilience. Personality and Individual Differences, 40, 873–884 https://doi.org/10.1016/j.paid.2005.08.015.

21.

van de Gaer

, Grisay

, Schulz

, & Gebhardt

(2012). The reference group effect: An explanation of the paradoxical relationship between academic achievement and self–confidence across countries. Journal of Cross–Cultural Psychology, 43, 1205–1228 https://doi.org/10.1177/0022022111428083.

22.

Goldberg

L. R.

, Johnson

J. A.

, Eber

H. W.

, Hogan

, Ashton

M. C.

, Cloninger

C. R.

, & Gough

H. G.

(2006). The international personality item pool and the future of public–domain personality measures. Journal of Research in Personality, 40, 84–96 https://doi.org/10.1016/j.jrp.2005.08.007.

23.

, & van de Vijver

F. J. R.

(2013). A general response style factor: Evidence from a multi–ethnic study in the Netherlands. Personality and Individual Differences, 55, 794–800 https://doi.org/10.1016/j.paid.2013.06.017.

24.

, & van de Vijver

F. J. R.

(2015a). Effects of a general response style on cross–cultural comparisons: Evidence from the teaching and learning international survey. Public Opinion Quarterly, 79, 267–290 https://doi.org/10.1093/poq/nfv006.

25.

, & van de Vijver

F. J. R.

(2015b). Self–presentation styles in self–reports: Linking the general factors of response styles, personality traits, and values in a longitudinal study. Personality and Individual Differences, 31, 129–134 https://doi.org/10.1016/j.paid.2014.09.009.

26.

, van de Vijver

F. J. R.

, Domínguez Espinosa

, Abubakar

, Dimitrova

, Adams

, … Villieux

(2015). Socially desirable responding: Enhancement and denial in 20 countries. Cross–Cultural Research, 49, 227–249. https://doi.org/10.1177/1069397114552781.

27.

Hopkins

D. J.

, & King

(2010). Improving anchoring vignettes: Designing surveys to correct interpersonal incomparability. Public Opinion Quarterly, 74, 201–222 https://doi.org/10.1093/poq/nfq011.

28.

Hopwood

C. J.

, & Donnellan

M. B.

(2010). How should the internal structure of personality inventories be evaluated? Personality and Social Psychology Review, 14, 332–346 https://doi.org/10.1177/1088868310361240.

29.

Jürges

, & Winter

(2013). Are anchoring vignettes ratings sensitive to vignette age and sex? Health Economics, 22, 1–13 https://doi.org/10.1002/hec.1806.

30.

Kapteyn

, Smith

J. P.

, Van Soest

, & Vonkova

(2011) Anchoring vignettes and response consistency. Vol. RAND working paper WR–840: RAND.

31.

King

, Murray

C. J. L.

, Salomon

J. A.

, & Tandon

(2004). Enhancing the validity and cross–cultural comparability of measurement in survey research. American Political Science Review, 98, 191–207 https://doi.org/10.1017/S000305540400108X.

32.

Kyllonen

P. C.

, & Bertling

J. J.

(2014). Innovative questionnaire assessment methods to increase cross–country comparability. In Rutkowski

, von Davier

, & Rutkowski

(Eds.), Handbook of international large–scale assessment: Background, technical issues, and methods of data analysis (pp. 277–286). Boca Raton, FL: CRC Press.

33.

Leung

, Au

, Huang

, Kurman

, Niit

, & Niit

K.–K.

(2007). Social axioms and values: A cross–cultural examination. European Journal of Personality, 21, 91–111 https://doi.org/10.1002/per.615.

34.

Leung

, Lam

B. C. P.

, Bond

M. H.

, Conway

L. G.

III , Gornick

L. J.

, Amponsah

, … Zhou

(2012). Developing and evaluating the social axioms survey in eleven countries: Its relationship with the five–factor model of personality. Journal of Cross–Cultural Psychology, 43, 833–857 https://doi.org/10.1177/0022022111416361.

35.

Marsh

H. W.

, Lüdtke

, Nagengast

, Morin

A. J. S.

, & Von Davier

(2013). Why item parcels are (almost) never appropriate: Two wrongs do not make a right: Camouflaging misspecification with item parcels in CFA models. Psychological Methods, 18, 257–284 https://doi.org/10.1037/a0032773.

36.

McCrae

R. R.

(2015). A more nuanced view of internal consistency. Personality and Social Psychology Review, 19, 97–112 https://doi.org/10.1177/1088868314541857.

37.

McCrae

R. R.

, & Costa

P. T.

Jr. (1983). Social desirability scales: More substance than style. Journal of Consulting and Clinical Psychology, 51, 882–888 https://doi.org/10.1037/0022–006X.51.6.882.

38.

McCrae

R. R.

, Terracciano

, & 78 Members of the Personality Profiles of Cultures Project (2005). Universal features of personality traits from the observer's perspective: Data from 50 cultures. Journal of Personality and Social Psychology, 88, 547–561 https://doi.org/10.1037/0022–3514.88.3.547.

39.

McGrath

R. E.

, Mitchell

, Kim

B. H.

, & Hough

(2010). Evidence for response bias as a source of error variance in applied assessment. Psychological Bulletin, 136, 450–470 https://doi.org/10.1037/a0019216.

40.

Mõttus

, Allik

, Realo

, Pullmann

, Rossier

, Zecca

, … Tseung

C. N.

(2012). Comparability of self–reported conscientiousness across 21 countries. European Journal of Personality, 26, 303–317 https://doi.org/10.1002/per.840.

41.

Mõttus

, Allik

, Realo

, Rossier

, Zecca

, Ah–Kion

, … Johnson

(2012). The effect of response style on self–reported conscientiousness across 20 countries. Personality and Social Psychology Bulletin, 38, 1423–1436 https://doi.org/10.1177/0146167212451275.

42.

Muthen

L. K.

, & Muthen

B. O.

(1998–2012). Mplus user's guide (7th ed.). Los Angeles, CA: Muthen & Muthen.

43.

Nye

C. D.

, Roberts

B. W.

, Saucier

, & Zhou

(2008). Testing the measurement equivalence of personality adjective items across cultures. Journal of Research in Personality, 42, 1524–1536 https://doi.org/10.1016/j.jrp.2008.07.004.

44.

Organisation for Economic Co–operation and Development. (2014). TALIS 2013 technical report. Paris, France: OECD publishing.

45.

Parks–Leduc

, Feldman

, & Bardi

(2015). Personality traits and personal values: A meta–analysis. Personality and Social Psychology Review, 19, 3–29 https://doi.org/10.1177/1088868314538548.

46.

Paulhus

D. L.

(1991). Measurement and control of response biases. In Robinson

, Shaver

& Wrightsman

(Eds.), Measures of personality and social psychological attitudes (Vol. 1, pp. 17–59). San Diego, CA: Academic Press, DOI: https://doi.org/10.1016/B978–0–12–590241–0.50006–X.

47.

Primi

, Zanon

, Santos

, De Fruyt

, & John

O. P.

(2016). Anchoring vignettes: Can they make adolescent self–reports of social–emotional skills more reliable, discriminant, and criterion–valid? European Journal of Psychological Assessment, 32, 39–51 https://doi.org/10.1027/1015–5759/a000336.

48.

Rammstedt

, Kemper

C. J.

, & Borg

(2013). Correcting big five personality measurements for acquiescence: An 18–country cross–cultural study. European Journal of Personality, 27, 71–81 https://doi.org/10.1002/per.1894.

49.

Revilla

M. A.

, Saris

W. E.

, & Krosnick

J. A.

(2014). Choosing the number of categories in agree–disagree scales. Sociological Methods & Research, 43, 73–97 https://doi.org/10.1177/0049124113509605.

50.

Rohling

M. L.

, Larrabee

G. J.

, Greiffenstein

M. F.

, Ben–Porath

Y. S.

, Lees–Haley

, & Green

(2010). A misleading review of response bias: Comment on McGrath, Mitchell, Kim, and Hough (2010). Psychological Bulletin, 137, 708–712 https://doi.org/10.1037/a0023327.

51.

Rossier

, Aluja

, Blanch

, Barry

, Hansenne

, Carvalho

A. F.

, … Karagonlar

(2016). Cross–cultural generalizability of the alternative five–factor model using the Zuckerman–Kuhlman–Aluja personality questionnaire. European Journal of Personality, 30, 139–157 https://doi.org/10.1002/per.2045.

52.

Rutkowski

, & Svetina

(2014). Assessing the hypothesis of measurement invariance in the context of large–scale international surveys. Educational and Psychological Measurement, 74, 31–57 https://doi.org/10.1177/0013164413498257.

53.

Rutkowski

, & Svetina

(2016). Measurement invariance in international surveys: Categorical indicators and fit measure performance. Applied Measurement in Education., https://doi.org/10.1080/08957347.2016.1243540.

54.

Saucier

(1994). Mini–markers: A brief version of Goldberg's unipolar Big–Five markers. Journal of Personality Assessment, 63, 506–516 https://doi.org/10.1207/s15327752jpa6303_8.

55.

Schafer

J. L.

, & Graham

J. W.

(2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147–177 https://doi.org/10.1037/1082–989X.7.2.147.

56.

Schmitt

D. P.

, Allik

, McCrae

R. R.

, & Benet–Martínez

(2007). The geographic distribution of big five personality traits. Journal of Cross–Cultural Psychology, 38, 173–212 https://doi.org/10.1177/0022022106297299.

57.

van de Schoot

, Kluytmans

, Tummers

, Lugtig

, Hox

, & Muthén

(2013). Facing off with Scylla and Charybdis: A comparison of scalar, partial, and the novel possibility of approximate measurement invariance. Frontiers in Psychology, 4, 770 https://doi.org/10.3389/fpsyg.2013.00770.

58.

Schwartz

S. H.

(1992). Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. Advances in Experimental Social Psychology, 25, 1–65 https://doi.org/10.1016/S0065–2601(08)60281–6.

59.

Schwartz

S. H.

, & Boehnke

(2004). Evaluating the structure of human values with confirmatory factor analysis. Journal of Research in Personality, 38, 230–255 https://doi.org/10.1016/S0092–6566(03)00069–2.

60.

Schwartz

S. H.

, Melech

, Lehmann

, Burgess

, Harris

, & Owens

(2001). Extending the cross–cultural validity of the theory of basic human values with a different method of measurement. Journal of Cross–Cultural Psychology, 32, 519–542 https://doi.org/10.1177/0022022101032005001.

61.

Schwartz

S. H.

, & Sagiv

(1995). Identifying culture–specifics in the content and structure of values. Journal of Cross–Cultural Psychology, 26, 92–116 https://doi.org/10.1177/0022022195261007.

62.

Smith

P. B.

(2004). Acquiescent response bias as an aspect of cultural communication style. Journal of Cross–Cultural Psychology, 35, 50–61 https://doi.org/10.1177/0022022103260380.

63.

Smith

P. B.

(2011). Communication styles as dimensions of national culture. Journal of Cross–Cultural Psychology, 42, 216–233 https://doi.org/10.1177/0022022110396866.

64.

Thalmayer

A. G.

, & Saucier

(2014). The questionnaire big six in 26 nations: Developing cross–culturally applicable big six, big five and big two inventories. European Journal of Personality, 28, 482–496 https://doi.org/10.1002/per.1969.

65.

van de Vijver

F. J. R.

, & Leung

(1997). Methods and data analysis of comparative research. Thousand Oaks, CA: Sage.

66.

van de Vijver

F. J. R.

, & Leung

(2001). Personality in cultural context: Methodological issues. Journal of Personality, 69, 1007–1031 https://doi.org/10.1111/1467–6494.696173.

67.

Wand

, & King

(2007). Anchoring vignetttes in R: A (different kind of) vignette. Retrieved from http://wand.stanford.edu/anchors/doc/anchors.pdf