Abstract
This study provides a critical evaluation of the gender role attitude scale included in the fourth wave of the European Values Study. The goodness of this scale is tested considering first of all its reliability and the stability of the factorial structure. The results suggest caution in the use of this instrument: the scale presents a deep variation in reliability across countries and its configuration is not stable, displaying several different factor structures from one country to another. In considering the source of this instability, this study addresses a priming effect due to questions introduced in 2008 immediately prior to the use of the gender role attitudes scale that modified the context of response.
Introduction
The European Values Study (EVS) includes a popular scale that aims to measure attitudes towards gender roles. A couple of recent studies have raised the issue of the goodness of this scale. Voicu and Tufiş (2012), analysing the change in gender role attitudes in Romania, reported that the multiple indicator structure of the EVS scale was not tenable across waves and gender groups. Similarly, Lomazzi (2016) performed preliminary and confirmatory factor analyses in a comparative study of gender role attitudes in Europe and referred to similar instability. Both studies solved the issue by selecting the most stable items and proceeding with single indicators; however, this can be both a good solution and a limitation. Therefore, a deeper investigation of this scale is needed. This study provides an evaluation of the instrument employed in EVS 2008, aiming to provide options for improving the scale that would allow for confidence in its full use.
As the scale is often used to compare countries, the aim of the tests is not to generally assess its goodness in terms of the overall sample, but to evaluate the stability and reliability of the scale country by country. The tests show controversial results that invite caution in using this scale. The scale presents a deep variation in reliability across countries and its configuration is not stable, displaying several different factor structures. In considering the source of this instability, this study addresses a priming effect due to questions introduced in 2008 immediately prior to the use of the gender role attitudes scale.
Gender Role Attitudes Scale
The EVS Gender Role Attitudes Scale
The EVS 1 is a trustworthy cross-national and repeated survey program that investigates Europeans’ values in an increasing number of countries. Since the first wave in 1981, a specific scale aiming to measure attitudes towards gender roles has been included. The scale has been altered slightly over time, but it has retained its core items, allowing comparisons over time and with other international survey programs that adopt similar and partly overlapping measures, such as World Values Survey, International Social Survey Programme, Generation and Gender Programme, European Social Survey, and Eurobarometer.
In 2008, the EVS scale included the following eight items: A working mother can establish just as warm and secure a relationship with her children as a mother who does not work A pre-school child is likely to suffer if his or her mother works A job is alright but what most women really want is a home and children Being a housewife is just as fulfilling as working for pay Having a job is the best way for a woman to be an independent person Both the husband and wife should contribute to household income In general, fathers are as well suited to looking after their children as mothers Men should take as much responsibility as women for the home and children
Critical Aspects in the Gender Role Attitude Scales
This paper focuses on the instability of the EVS scale, but it is important to consider that, generally speaking, the gender role scale can present other critical aspects that could also affect its stability across different social groups. The literature often refers to issues related to the item wording, cultural interpretations and the limited coverage of the concept being investigated.
Most items are formulated by expressing traditional views of gender roles and are generally focused on the role of the woman. Braun (1998, 2008) argues that this reflects the historical roots of these survey programs, which developed in the late 1970s and today still retain this influence.
Nevertheless, in trying to follow social change, new ‘egalitarian items’ have been introduced, with the intent to grasp an understanding of the emerging new lifestyles. If the formulation of the traditional items could be considered obsolescent, the egalitarian items could also present problems. First, they risk not being able to capture the social reality, and second, they adopt a reversal gender schema that egalitarian respondents could also disagree with because, in any case, there is a gender expectation (Braun, 2008). Nowadays, respondents can react differently to the strong traditional (or egalitarian) formulation of the questions. For example, in countries where a good level of gender equality has already been achieved, traditional statements can appear distant from people’s daily experience; conversely, this dissonance may not be perceived in less egalitarian countries.
Different from other surveys, the EVS scale does consider male-role items, but in this it focuses on the domestic sphere. It hints at the workplace experience, but the concept of gender role attitudes should imply multidimensional aspects of both the private and the public sphere that go far beyond that (Constantin and Voicu, 2014; Lomazzi, 2016). In other words, the formulation of these items is affected by the theoretical and ideological reflection of the time in which they were developed. As pointed out by Larsen and Long (1988), these instruments risk becoming less stable and less reliable over time.
The interpretation of the items may also vary across and within countries. For example, items such as ‘A working mother can establish just as warm and secure a relationship with her children as a mother who does not work’ and ‘A pre-school child is likely to suffer if his or her mother works’, used also by ISSP, include in their formulation two dimensions, and respondents’ reaction to different objects can vary from one cultural group to another. As Braun (1998) and Philipov (2008) claim, people can react to the maternal skills of being a ‘good mother’ even if she works, or to the emotional needs of children. In addition, items like ‘Both the husband and wife should contribute to household income’ should be handled with caution, because in weak economic living conditions, agreement with this statement could simply show as economic necessity rather than support for gender equality (Braun, 2009).
Method
Data Used
The instrument evaluated is the gender role attitudes scale included in EVS 2008. 2 For some tests, previous waves have been used as benchmarks, while the tests considered the 26 countries to have participated since 1990.
Results - Critical Evaluation of the Instrument
Reliability
The internal consistency of the scale and its reliability were tested employing the Cronbach’s alpha index. Scoring of this well-known index ranges from 0 to 1, and a score of 0.7 is generally used as a rule of thumb for scale acceptability. Similar values of Cronbach’s alpha across countries would mean that the reliability of the scale is also acceptable.
The results reported in Table 1 show an overall lack of sufficient reliability (.626) and a dramatic variation across countries. In some cases, the scale achieves a score above the threshold, such as in Germany (.786) and Spain (.725). In others, the score is quite low, as in Latvia (.412), Malta (.451) and Romania (.293). In other words, the scale is not very consistent and it is not reliable to the same extent in all the countries.
Reliability of the scale across countries
Stability of the Factorial Structure
The low Cronbach’s alpha scores also indicate weak internal consistency. This may be the case when the scale is not measuring a unique concept and its items refer to different concepts, even if these are related. One way to test the unidimensionality of the scale is to perform exploratory factory analyses. Running analysis country by country will also assess whether the factorial structure of the scale is stable across countries, meaning that the configuration assumed by the latent concept behind this scale is the same in all the countries.
The analysis of the whole sample highlighted three factors (Table 2). The first factor is ‘caring responsibilities’, and explains almost one third of the variance (28.4%); the second is ‘gender ideology’ (19.3%) and the third ‘economic role of women’ (14.2%). Some items load on two factors, such as ‘Working mother can establish just as warm and secure a relationship with her children as a mother who does not work’; this could also be due to the ambiguity of the wording, as anticipated earlier.
Exploratory factor analysis and factor loadings
The exploratory factor analyses run country by country 3 provide a controversial picture. The critical aspects mainly concern two issues: the ordering of the explaining factors and the loading of the item ‘Working mother can establish just as warm and secure a relationship with her children as a mother who does not work’, which is much more confusing than the other items as it loads on different factors in each country.
According to the first factor put forward to explain the variance, three main groups are identifiable. As shown in Table 3, the first group includes Belgium, Germany, France, Slovakia, Italy and the Netherlands. Here, the factor ‘gender ideology’ has the biggest eigenvalue. In Germany, this factor explains about 40 percent of the total variance. The second group is the largest and includes Spain, Sweden, Finland, Ireland, Poland, Northern Ireland, Great Britain, Portugal, Denmark, Latvia, Slovenia, Iceland, Czech Republic and Austria. In all these countries, the variance in the gender role scale is first explained by the factor ‘caring responsibilities’. The last group includes Romania, Bulgaria and Lithuania. Here, the first factor to explain the variance is the ‘economic role of women’. The configuration of the underlying concept seems inconsistent across the entire sample.
Exploratory factor analysis and the first factor explaining variance by country
The analyses carried out confirm that the item ‘Working mother can establish just as warm and secure a relationship with her children as a mother who does not work’ hardly loads on only one factor (Table 4), as other authors have argued (Braun, 1998; Philipov, 2008). Even if maintaining the three-factor structure, in seven cases the configuration changes, and this item correlates with another item (*with ‘A pre-school child is likely to suffer if his or her mother works’; **with ‘In general, fathers are as well suited to looking after their children as mothers’) to create a dimension named ‘relational skills’. Unfortunately, excluding this item from the scale does not improve the situation 4 .
Exploratory factor analysis; item one-factor loadings by country
The exploratory factor analyses demonstrated that the scale is not unidimensional, but rather that the underlying concept includes three dimensions. However, in some countries the variables correlate differently, revealing different latent variables that do not emerge in other countries. The factorial structure is not stable enough in terms of the latent dimensions, the factor loadings and the explained variance. This is evidence of the problems raised by previous studies (Lomazzi, 2016; Voicu and Tufiş, 2012); however, what is the source of this instability?
A Priming Effect as a Source of Instability
A comparison of the questionnaires employed in the past two waves reveals a slight change in the question ordering. To the two items preceding the gender role scale in 1999, a further five items have been added: A man has to have children in order to be fulfilled (since 1999) A marriage or a long-term stable relationship is necessary to be happy (since 1999) Homosexual couples should be able to adopt children It is alright for two people to live together without getting married It is a duty towards society to have children People should decide for themselves whether to have children or not When a parent is seriously ill or fragile, it is mainly the adult child’s duty to take care of him/her
These items concern issues regarding traditional/non-traditional family relations. If attitudes are generally expressed on the basis of the information immediately available to the respondent at that precise moment (Wilson and Hodges, 1992), it could be the case that the recently added items create a different framework, thus affecting the answers in the immediately next scale that again asks about traditional/non-traditional aspects of gender roles.
Even if related to the respondents’ basic values, the expression of attitudes takes place in a situated context. According to the construal model of attitudes (Schwarz, 1999; Wilson and Hodges, 1992), the respondents make use of the most recent available information to interpret the question and express their judgment. From this perspective, the adjacent questions constitute the context for interpreting the scale on gender roles, and could therefore influence the answers.
To assess whether these new items actually changed the context of responses, I tested their effects on the gender role items while controlling for known predictors of egalitarian gender role attitudes (André et al., 2013; Bolzendahl and Myers, 2004; Mason and Lu, 1988; Philipov, 2008).
First, I gave the variables the same polarisation (pro-egalitarian gender role attitudes; pro-non-traditional family relations). Second, I computed two synthetic variables: nt99, which refers to the mean of the two items already available in 1999; and nt08, which computes the mean of the five variables added in 2008. The control variables were dummies, as follows: Gender: Female (reference=male) Educational level: High, middle and low (reference=low level) Employment status: Worker (reference=not worker) Church attendance: Weekly, monthly (reference=less than once a month) Importance given to God: Very important (reference=not very important)
I adopted a stepwise model to better observe the effect of each variable. In the first step, I tested the effect of gender, educational level and employment status. I separated the effect of the two measures of religiosity by introducing church attendance in the second step, and the importance given to God in the third. In the last step, I included two new variables. I ran the analyses for each item in every country, 5 both in 1999 and 2008, to compare the effect of the variables already available in the previous wave. Table 5 provides a summary of the effects of the two variables nt99 and nt08 for each case.
Stepwise regression model; significant effects of support for the non-traditional family (2008 – nt08 and 1999 – nt99) on egalitarian gender role attitudes by country and item
Different from the items present since 1999, those added in 2008 have a significant and positive effect in almost all the countries for most of the gender role items. Among the seven items, ‘A job is alright but what most women really want is a home and children’, ‘A working mother can establish just as warm and secure a relationship with her children as a mother who does not work’, ‘A pre-school child is likely to suffer if his or her mother works’ and ‘In general, fathers are as well suited to looking after their children as mothers’ were most affected by the adjacent questions introduced in 2008.
Generally, this means that when the respondents support less traditional family relations, they are more likely to show egalitarian attitudes towards gender roles immediately afterwards. Similarly, when the respondents express traditional attitudes regarding family relations, then they will also provide a traditional view on gender roles. In this case, the traditional family relation items could prompt the respondents to answer while avoiding incoherencies (Tourangeau and Rasinski, 1988).
Furthermore, the respondent’s reaction to the object in the gender role items becomes more complicated because of the combination of the priming effect and the critical aspects of the wording of the items.
Concluding Remarks
The study presented here confirms that the EVS gender roles scale was not tenable enough in 2008, and provides preliminary results that address the source of this instability in the priming effect caused by the items on traditional family relations introduced in 2008 immediately preceding the gender roles scale. Considering that this problem may not affect to the same extent all the items, it could be worthy running measurement invariance tests to assess whether a reduced scale could be meaningfully used in cross-cultural research. In addition, further development of this study should deal with at least two issues, on the one hand confirming the priming effect, and on the other looking for possible ways to control this effect and allow for the possible use of the full scale in 2008.
As an alternative to an experimental setting currently used, a structural equation model could be employed to confirm the priming effect, as B. Voicu (2015) used to detect it in the measurement of life satisfaction. To deal with the second issue, solutions can be explored following the example illustrated by Schwartz and colleagues (2012), who included in a multilevel SEM a common factor to correct a method bias.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.