Abstract
This meta-analysis evaluated theoretical predictions from balanced identity theory (BIT) and evaluated the validity of zero points of Implicit Association Test (IAT) and self-report measures used to test these predictions. Twenty-one researchers contributed individual subject data from 36 experiments (total N = 12,773) that used both explicit and implicit measures of the social–cognitive constructs. The meta-analysis confirmed predictions of BIT’s balance–congruity principle and simultaneously validated interpretation of the IAT’s zero point as indicating absence of preference between two attitude objects. Statistical power afforded by the sample size enabled the first confirmations of balance–congruity predictions with self-report measures. Beyond these empirical results, the meta-analysis introduced a within-study statistical test of the balance–congruity principle, finding that it had greater efficiency than the previous best method. The meta-analysis’s full data set has been publicly archived to enable further studies of interrelations among attitudes, stereotypes, and identities.
Balanced identity theory (BIT) was developed as an account of relationships among several of social psychology’s most prominent theoretical constructs—identities, attitudes, stereotypes, and self-esteem. The theory (Greenwald et al., 2002) drew on principles originating in social psychology’s consistency theories of the late 1950s, especially Heider’s (1958) balance theory, Osgood and Tannenbaum’s (1955) congruity theory, Newcomb’s (1953) symmetry theory, and Festinger’s (1957) cognitive dissonance theory.
BIT’s main theoretical devices are (a) its definitions of identities, attitudes, stereotypes, and self-esteem as associations 1 involving self, groups, stereotypic attributes, and valence and (b) the balance–congruity principle—the proposition that an association between concepts A and B should strengthen to the extent that each of A and B is associated with the same third concept, C. The balance–congruity principle is a close relative of the concept of mediated generalization, first described by Cofer and Foley (1942). Its name acknowledges its additional roots in the affective–cognitive consistency theories of Heider (1958; balance theory) and Osgood and Tannenbaum (1955; congruity theory).
Historical Background
Balance theory and congruity theory sought to explain how naturally arising affective–cognitive configurations induce further affective–cognitive changes. For example, a positive attitude toward a group will not lead to identification if self-esteem is negative, and will occur in proportion to the positivity of self-esteem (i.e., more when self-esteem is strongly positive than when it is moderately or weakly positive). The balance–congruity principle extended those prior theories to explain the collection of implicit self-esteem effects that were identified by Greenwald and Banaji (1995), including minimal group effects (Tajfel et al., 1971), endowment effects (Kahneman et al., 1990), mere ownership effects (Beggan, 1992), self-anchoring effects (Cadinu & Rothbart, 1996; Gawronski et al., 2007; Roth & Steffens, 2014), implicit self-referencing (Perkins & Forehand, 2006, 2012), and implicit self–object linking (Ye & Gawronski, 2016). These are phenomena in which newly created associations between the self and social or nonsocial objects produce an associative transfer of self-evaluation (i.e., self-esteem) to those self-linked objects.
BIT Compared to Heider’s Balance Theory
Both BIT and Heider’s (1958) balance theory predict social knowledge to be organized in ways that maintain affective–cognitive consistency: In balance theory, consistency was conceived in terms of cognitive structures that link a focal person (“p”) to other persons (“o”) or external objects (“x”) via either sentiment (attitude) or unit relationships (Heider, 1958). By replacing Heider’s distinct sentiment and unit connections with the more general conception of association as the cognitive link between persons and other entities, BIT was able to expand theoretical scope beyond the attitudes with which balance theory was concerned, including additionally stereotypes, identities, and self-esteem.
BIT Compared to Social Identity and Self-Categorization Theories
BIT and two other well-established theories on social identity—Turner et al.’s (1987) self-categorization theory (SCT) and Tajfel’s (1982) social identity theory (SIT)—allow consideration of identities in relation to self-esteem. Relationships among self-esteem, group membership, and in-group attitude are considered by all three theories (BIT, SCT, SIT). The three theories agree in expecting that persons with a strong in-group identity should have a stronger positive attitude toward their group (i.e., in-group attitude) than those with weak in-group identity.
A substantial difference between BIT and SIT arises from the difference in the way self-esteem is conceived in the two theories. SIT conceives self-esteem as a fundamental human need (Tajfel & Turner, 1979), whereas BIT conceives self-esteem non-motivationally as an association of the concept of one’s self with positive valence. This leads to substantial differences in how the two theories see the relation of self-esteem to formation of a novel identity. This was investigated in numerous experiments in which subjects were assigned to one of two previously unknown groups that differed in meaningless or arbitrary aspects. Tajfel et al. (1971) interpreted the repeated finding of subjects evaluating their own group more positively than the other group as a cognitive strategy occurring because subjects could achieve a boost in self-esteem by conceiving their group as the superior one (see also Abrams & Hogg, 1988; Hogg, 2000). The theoretical expectation was that this minimal group effect should occur most noticeably in subjects who had relatively low self-esteem, who would have stronger need for the self-esteem boost. In BIT, the minimal group phenomenon occurs as a consequence of the association of self with both the novel group and positive valence. The effect should be greater for those for whom self-esteem is already high, rather than for those with low self-esteem. The meta-analysis of 34 studies by Aberson et al. (2000), examining the relation between self-esteem and in-group bias, favored the relationship predicted from BIT, rather than the one predicted from SIT (see also Hewstone et al., 2002).
Methods for Evaluating the Balance–Congruity Principle
Implicit and Explicit Measures
This article examines evidence available to assess validity of BIT’s balance–congruity principle in studies using either explicit (direct) or implicit (indirect) measures (cf. Fazio & Olson, 2003). Explicit measures generally use self-report, allowing research subjects to be aware of what is being investigated. In contrast, implicit measures do not use self-report and do not require the subject to know the nature of the construct being assessed, which might be an attitude, a stereotype, an identity, or self-esteem. A recent treatment comparing the two types of measures is available in Greenwald and Lai (2020).
Implicit Association Test (IAT)
The focus of this meta-analysis is on the IAT (Greenwald et al., 1998). In the 20 years since its initial publication, the IAT has been applied in a diverse array of disciplines including social and cognitive psychology (Axt & Lai, 2019; Critcher & Ferguson, 2016), neuroscience (Mitchell et al., 2009; Schindler et al., 2015), education (Cvencek et al., 2015; Devos & Cruz Torres, 2007; Nosek et al., 2009), developmental science (Baron & Banaji, 2006; Cvencek et al., 2011), clinical psychology (Creemers et al., 2013; Leeuwis et al., 2015), health psychology (Cooper et al., 2012; Sabin & Greenwald, 2012), and marketing research (Horcajo et al., 2010; Trendel et al., 2018). The wide range of applications provides sufficient research literature from which to review the strength of the evidence for the support of the balance‒congruity principle.
The IAT is a method for indirectly measuring the strengths of associations among concepts. In its data-providing combined-task blocks, the IAT requires sorting of stimulus exemplars belonging to four different categories using just two response keys, each of which is assigned to two of the four categories. The underlying principle of the IAT is that it is easier to give the same response to items representing categories that are associated in memory than to ones representing categories that are not associated. For example, the assignment of items representing summer and warm to the same key should be easier for subjects to deal with than the assignment of items representing summer and cold to the same key.
The two combined tasks of a standard IAT most typically include two concept categories (e.g., summer and winter) and two attribute categories (e.g., warm and cold). In each combined-task block, there is a strict alternation between concept category exemplars (typically on odd-numbered trials) and attribute category exemplars (on even-numbered trials). In one of the two combined tasks, exemplars of summer and warm (e.g., images of beaches and words such as “hot” and “sunny”) require pressing one of two response keys (generally positioned left and right on a computer keyboard) for a correct response, and exemplars of winter and cold (e.g., images of snow and words such as “freeze” and “icy”) require response with the other key. In the second critical block, exemplars of summer and cold are sorted with one response key, and exemplars of winter and warm are sorted with the alternative response key. The faster the responses, the stronger the presumed underlying association between the two categories sharing the same key. For participants who possess the expected stronger associations of concept summer with attribute warm and winter with cold, the first sorting task will likely be much easier than the second. Ease of sorting is indexed by the speed of producing correct responses. Most IAT procedures oblige occurrence of a correct response to end every trial; when trials are permitted to end on occurrence of an incorrect response, a time penalty for trials on which errors occurred is applied in the scoring procedure (described by Greenwald et al., 2003).
Aims of the Research
Despite the centrality of identities, self-esteem, and in-group attitudes in social psychology, there were no studies of affective–cognitive consistency among these constructs prior to formulation and the first tests of balanced identity. This study examines the evidence for affective–cognitive consistency in naturally arising configurations of these constructs, comparing the strength of that evidence for implicit and explicit measures. The available quantitative evidence was first examined via meta-analytic hypothesis tests using the established standard approach for testing BIT’s balance–congruity principle. The meta-analysis additionally allowed testing a novel, within-study meta-analytic method that was found to be more efficient than the already established method. Both approaches included (a) analyses of studies using self-esteem measures alongside studies using other self-concept measures and (b) subject-level data from each study in the meta-analysis to assure use of the same analysis methods for all studies.
Meta-Analytic Evaluation of the Balance–Congruity Principle
Cvencek et al. (2012) found support for balance–congruity principle expectations in a review of studies including about 1,900 subjects. The present meta-analysis increases the number of subjects available for study by a factor of 6. In addition, by obtaining individual subject IAT and self-report measures (where available) from authors for all 36 samples reviewed in this article, it was possible to use the same analysis method for all studies. A further contribution of this article is that the full data set, consisting of 12,773 participants (ranging from young children to adults) across 36 studies is being made publicly available in a widely accessible archive. 2
Search Method for Locating Balanced Identity Data Sets
PsycINFO, PubMed, and Google Scholar were searched, using the following as individual keywords/phrases: cognitive balance, cognitive consistency, balanced identity, IAT, Implicit Association Test, implicit attitude, implicit identity, implicit self-esteem, implicit stereotype, implicit self-concept, 3 IATs, and 3 Implicit Association Tests. The cut-off date for the search was May 31, 2013. PsycINFO was also used to find studies that referenced Greenwald et al. (2002). These searches identified 19 reports. The IAT scores used in this meta-analysis are the exact ones that appear in each of the published reports. 3 Authors of these 19 reports were then contacted in search of additional studies, yielding eight more reports. The resulting 27 reports included 36 independent samples, with a total of 9,808 subjects providing data for IAT measures and 12,773 providing data for explicit measures. Table 1 describes the 36 samples. These samples include both male and female participants, multiple ethnic groups (e.g., Asians, Blacks, Latinos, Whites), and multiple age groups (e.g., pre- and elementary-school children, undergraduates, adults, elderly). In addition (see Table 1), the studies in this meta-analysis included measures of attitudes toward both social (e.g., gender, race, age) and nonsocial categories (e.g., weight, food), along with multiple stereotypes (e.g., math, parenthood, achievement).
Characteristics of 36 Independent Samples for Meta-Analysis of Studies Using Balanced Identity Designs.
Note. Participants = participant groups comprising independent samples in each study. Participant groups are identified in terms of the two contrasted identity groups used in the IAT measures; n = number of participants in each independent sample. All studies included measures of associations of (a) self (contrasted with other) with a contrasted pair of group concepts, (b) self with a contrasted pair of attribute concepts, and (c) the contrasted pair of group concepts with the contrasted pair of attribute concepts. Capitalized group and attribute concepts were the ones scored as positive when associated with self. The IAT measure of group–attribute association was scored so that the association between the capitalized group and the capitalized attribute was positive. All studies reported implicit effects as IAT D measures (Greenwald et al., 2003, p. 214), except for the two Lane et al. (2005) studies and Steffens et al. (2010), as described in their respective, original publications. Tests passed refer to the 12 possible tests of the 4-test method. BMI = body mass index; IAT = Implicit Association Test.
The research reviewed in this article was done at 16 institutions and was published in 17 peer-reviewed journals. 4 Part of the explanation for there not being more such studies is that these studies are effort-demanding. Seventeen of the 36 studies required locating and recruiting nonstandard subject populations (see the “Participants” column in Table 1). Authors were also obliged to create novel IATs for 17 of the 36 studies. Five of the 36 studies included in the meta-analysis had unpublished data sets.
Use and Evaluation of BIT
Each association in a balanced identity study is embedded in an associative network that includes many other associations, as indicated in following Equations 1 to 3. The three associations of a balanced identity research design are represented in these equations by SG (self–group association), SA (self–attribute association), and GA (group–attribute association). Each association in the design is embedded in multiple trios of associations (see Figure 1). The specific concepts included in a study’s “focal” trio of measured associations are indicated in the equations with filled-triangle subscripts. Additional concepts that enter the prediction of the (criterion) measure on the left side of each equation have numerical subscripts. The b coefficients in the three equations represent weights that, in principle, could be empirically assessed with regression analyses—if measures were available for all associations in the equation. In practice, however, measures are not available for associations linking self and group to other attributes (the other As in Equation 1), self and attribute to other groups (the other Gs in Equation 2), or group and attribute to persons other than self (the Ps in Equation 3). The ellipsis (“. . .”) that ends each equation acknowledges the indefinite multiplicity of such additional groups, attributes, and persons that may contribute to the strength of the criterion association. Nevertheless, the first predictor in each equation (e.g., S▲A▲∙G▲A▲ in Equation 1) should be the strongest predictor and, consistent with the balance–congruity principle, should be correlated with other predictors in the equation.
Schematic fragment of a social knowledge structure.
The 4-Test Method
Testing the balance–congruity principle requires a statistical method to evaluate the predictions involving just the three focal variables in Equations 1 to 3 (e.g., S▲G▲ = b▲∙[S▲A▲]∙[G▲A▲] in Equation 1). Greenwald et al.’s (2002) 4-test method depended on an assumption that these three associations were measured (at least to a good approximation) on interval scales that had rational zero points. In the first decade after the 2002 publication, the two measurement assumptions (interval scales and rational zero point) were plausible and were consistent with observed data, but they were not yet empirically testable with any precision. Subsequent accumulation of the data reported in this meta-analysis made possible the development of more precise empirical tests that are reported in this article. Detailed overview of the 4-test method, as well as the results of analyses applying the 4-test method to the current data set, can be found in the Supplemental Materials.
An Alternative to the 4-Test Method: Within-Study Meta-Analysis
One limitation of the 4-test method is its cumbersomeness: It requires a computation of 12 statistical tests—four in each of the two-step regressions for each of three measures predicted by the product of the other two. A second limitation is that the 4-test method provides no quantitative indicator of magnitude of confirmation of the balance–congruity principle. A third limitation is an increase in possibly spurious confirmations when the two product-component predictors (a) are additively (not multiplicatively) correlated with the criterion and (b) both have means deviating from zero by more than 1 SD in the positive direction (see Greenwald et al., 2006, Figure 1E). This last difficulty is due to collinearity of two individual predictors that are positively correlated with both (a) the regression’s criterion measure and (b) their own product. Despite these three limitations, the 4-test method is superior to the traditional simultaneous multiple regression method in detecting the presence of a pure multiplicative relationship (see Greenwald et al., 2006, Figure 2).
Effects of displacements of zero points of (A) IAT and (B) self-report measures on magnitude of confirmations of the balance–congruity principle in Test 1 of the 4-test method.
Seeking a possibly superior alternative to the 4-test method, this meta-analysis’s data were used to evaluate a new method that combined the three correlation effect sizes produced by Test 1 in each sample and (separately) those produced by Test 2 (see Supplemental Material for details). For each of the three types of criterion measure (SG, GA, and SA), separately for self-report and IAT measures, the three effect sizes were transformed to Fisher Z values and then aggregated in a random-effects, within-study meta-analysis.
The within-study meta-analytic summaries of Tests 1 and 2 were then themselves meta-analytically combined (separately for IAT and self-report and, within those, separately for Tests 1 and 2) across all studies in the meta-analysis. 5 The within-study meta-analyses used only Tests 1 and 2 because (a) the r associated with Test 1 can be interpreted as a basic test of fit of a multiplicative model and (b) the pr associated with Test 2 can be interpreted as an index of fit of a pure multiplicative model. The detailed results of the within-study meta-analysis method can be found in the Supplemental Materials. This novel within-study meta-analysis method provides what may be a more efficient indicator of conformity to BIT predictions than the 4-test method provides (see Supplemental Materials for details).
Evaluation of Within-Study Meta-Analysis Method in Comparison With 4-Test Method
The 4-test method provides the existing standard indicator of conformity of data from the three measures of a balanced identity design to the multiplicative model prediction of BIT’s balance–congruity principle. Predicting the number of tests passed for each study (maximum of 12), the Test 1 within-study aggregate was entered on Step 1 of a two-step regression, and the Test 2 within-study aggregate was entered on Step 2. This weighted two-step regression found that the within-study Test 1 aggregates were not significant predictors of total number of tests passed, whereas the Test 2 aggregates were significant predictors; partial r for Test 2 in the second step was pr = .384, t(33) = 2.39, p = .02. The interpretation of these findings for drawing conclusions about usefulness of the within-study meta-analysis method is considered in the “General Discussion” section.
Comparing Studies Using Self-Esteem Measures With Those Using Other Self-Concept Measures
Available evidence for validity of IAT measures of self-esteem is limited (Bosson et al., 2000; Greenwald & Farnham, 2000), with some of the strongest evidence coming from empirical tests of the balance–congruity principle. Within the BIT framework, self-esteem can be distinguished from another class of social–cognitive constructs involving the category “self”: self-concepts (or identities). According to the original formulation of BIT, “self-esteem is the association of the concept of self with a valence attribute,” whereas “self-concept [or identity] is the association of the concept of self with one or more (nonvalence) attribute concepts” (Greenwald et al., 2002, p. 5). The present meta-analysis affords an opportunity to compare evidence from studies involving valence (i.e., self-esteem measures as SA measures) with those involving other attributes (i.e., self-concept measures as SA measures). The available evidence was compared in two ways. First, the aggregated mean outcomes of Tests 1 and 2 of the 4-test method for these two groups of samples were compared meta-analytically. Second, the within-study meta-analysis method was applied to both self-esteem (k = 22) and self-concept (k = 14) measures (see Supplemental Materials).
The results showed that confirmations for self-esteem measures were mostly comparable to the self-concept measures: For implicit measures, the difference between the weighted aggregate effect sizes of self-esteem and self-concept measures was statistically significant for Test 2 (p = .005), but not for Test 1, p > .26; for self-report measures, this difference was not statistically significant for either Test 1 or Test 2 (ps > .10). In addition, the level of support for balance–congruity from studies involving valence (i.e., self-esteem rather than self-concept) was substantial (see Supplemental Materials for details). Implications of these findings for understanding validity of IAT and self-report measures of self-esteem are considered further in Supplemental Materials.
Interpretation of Zero Points of Attitude and Stereotype Measures
The IAT’s Zero-Point Assumption
The first use of the balance–congruity principle was to test the prediction that a woman possessing both an association of self with female (in-group identity) and an association of self with positive valence (positive self-esteem) should be expected also to have a positive in-group attitude (favorable toward female; Greenwald et al., 2002). In this self–gender–valence balanced identity design, the balance–congruity principle predicts that the association of female with positive valence should be strengthened if both female and positive valence are associated with self (BIT’s “shared first-order link” configuration; Greenwald et al., 2002). If either the self–female or the self–positive association is zero, no strengthening is predicted. Furthermore, strengthening should be greater when both associations in the shared first-order link are strong. Tests of this multiplicative prediction require that the associations used in the test are measured on scales that have valid zero values (Greenwald et al., 2002). The dependence of this prediction on a valid interpretation of the zero points of measures used to test the prediction is a main topic of this article.
Two Types of Zero Points
For attitude measures, two types of zero points can be useful. The more intuitive zero point is one that indicates neutrality (absence of attitudinal valence). An attitude can be said to be absent or neutral when a person’s evaluation of the attitude’s object has neither negative nor positive valence. A 7-point self-report item to assess this understanding of zero might range from a value of −3 (labeled “strong dislike,” indicating negative valence) to +3 (labeled “strong liking,” indicating positive valence), with a midpoint of 0 (labeled “neutral,” or “neither like nor dislike”). Evaluation items using the semantic differential method (Osgood et al., 1957) are of this type.
The second type of zero point indicates indifference (lack of preference) between two contrasted attitude objects. In a 7-point self-report item, the end points for an item that can assess the indifference meaning of zero might range from −3 (labeled “strongly prefer A relative to B”) to +3 (labeled “strongly prefer B relative to A”) with a midpoint of 0 (labeled “equal liking of A and B”). Items with the indifference-indicating zero point are useful in investigations of choice among available alternatives, such as pre-election polls. 6
IAT attitude measures allow only the indifference interpretation of zero, indicating no preference. More conceptually stated, the zero value of an attitude IAT (i.e., one in which the attribute category contrast is pleasant vs. unpleasant or good vs. bad) indicates lack of difference in strengths of associations of the contrasted concept categories (e.g., White vs. Black race) with positive or negative valence. This zero value is obtained when a research subject performs equally rapidly on the attitude IAT’s two combined tasks. In a stereotype IAT, the IAT’s zero value indicates lack of difference in strengths of associations of two contrasted attribute categories (e.g., career vs. family) with the two contrasted concept categories (e.g., female vs. male).
Studies that report data for both IAT and self-report measures of intergroup attitudes or stereotypes typically find greater proportions of respondents showing biases on the IAT measure than on its parallel (i.e., indifference-zero format) self-report measure. For example, in a large study that included measures of attitudes toward White and Black races, approximately 20% more people showed White-race preference on the IAT measure than on the parallel self-report measure (Nosek et al., 2007). Such findings call for an explanation for why IAT and self-report preference measures differ in this fashion. The most favorably regarded explanation for this difference is that IAT and self-report measures are based on different types of mental representation (perhaps associations vs. propositions, as suggested by Strack & Deutsch, 2004). A second favorably regarded explanation is that zero points of self-report measures may be distorted by respondents’ desires to appear unprejudiced (e.g., Greenwald et al., 2002). A third explanation is that zero points of IAT measures may be distorted due to characteristics of the IAT’s procedure (Blanton et al., 2015). 7
Methods to Evaluate Interpretations of the IAT’s Zero Point
All three of the preceding paragraph’s explanations could be correct. Available empirical evidence does not rule any of them out. This article evaluates specifically the third explanation—the one based on presumed invalidity of zero points of IAT measures, which could be tested using a new method applied to individual subject data. The method displaces zero points of predictors by adding or subtracting constants, followed by observing the extent to which tests of BIT’s balance–congruity principle are (or are not) impaired by the displacements. If the IAT or self-report measures used in these predictors have valid zero points, these zero-point displacements should impair the support for the balance–congruity provided by the undisplaced tests of those predictions (see Supplemental Materials for relevant tests). Relatedly, increasingly large zero-point displacements should produce increasingly large impairments of that support.
This article’s use of the zero-point displacement method may be appreciated by considering its relation to the entirely familiar use of multiplication for adjacent side lengths in computing the area of a rectangle. Like this article’s correlational Test 1 for balance–congruity predictions, valid use of Euclidean geometry to compute rectangle areas from rectangle side lengths requires that those length measures have valid zero values. Therefore, increasing magnitudes of displacements of zero values in side-length measures should increasingly impair the accuracy of area values computed using those zero-displaced measures.
Figure 2A presents results of applying the zero-point displacement strategy to the meta-analysis’s IAT data. Figure 2A reports the observed data, along with two simulations, one based on the unrealistic assumption of perfect reliability of measures used in the test and one based on the realistic assumption that IAT measures have average reliability of r = .50. 8 Figure 2B does likewise for the meta-analysis’s self-report data, using the conservative assumption of test–retest reliability of r = .80. These reliability simulations assume the (likely unrealistic) assumption that the only determinant of Test 1’s results is the multiplicative effect predicted by BIT’s balance–congruity principle.
The IAT results showed the expected decline in support as a function of magnitude of zero-point displacement with a maximum at the value of zero displacement. Remarkably, the curve for the obtained data was very close to the values expected from the assumption that test–retest reliability was the only source of noise in the findings. For self-report results, the decline as a function of displacement was noticeable, but the observed data were quite far from those expected if test–retest reliability was the only source of noise. Combining (a) the finding that support for the BIT prediction was generally weaker for self-report than for IAT with (b) the finding that this support with self-report was much further below expectation based on expected test–retest reliability, there is strong support for the conclusion that IAT measures come closer than do self-report measures to measuring the constructs described by BIT.
Additional Zero-Point Tests
An additional set of measures provided data for which the balance–congruity principle’s predictions depended on validity of zero points: a correlation scatterplot prediction for positive slope and zero intercept for the regression of the correlation between a subset of two of each study’s three measures (used as criterion) on the mean of the third measure. The correlation scatterplot prediction was described by Greenwald et al. (2002): When [the mean of] any variable in the balanced identity design is polarized toward its high end, the zero-order correlation between the other two variables should be positive; when any of the variables is polarized toward its low end, the zero-order correlation between the other two variables should be negative; and if a variable in the balanced identity design is not polarized, correlations between the other two variables should not differ from zero. (p. 11)
The trio of measures in each study can be sorted into one self–attribute association (SA), one self–group association (SG), and one group–attribute association (GA). Correlations between any two of these should be predicted by the mean of the third. For example, correlations between SG and SA should be predicted by GA. If GA has a negative mean, the SG–SA correlation should be negative; if GA has a positive mean, the SG–SA correlation should be positive; and if GA has a mean of zero, the SG–SA correlation should be zero. The test of the set of these predictions comes from examination of the scatterplot in which all such correlations for each type of measure (IAT or self-report) are plotted as a function of the varying means of the third measures of each trio. 9 This scatterplot is presented for the meta-analysis’s IAT measures in Figure 3A and for the meta-analysis’s self-report measures in Figure 3B.
Plots of Fisher Z-transformed correlations between pairs of association strength measures in balanced identity studies, plotted as a function of the mean of the third predictor.
Figure 3’s scatterplots combine, separately for IAT and self-report measures, the scatterplots for rSG–SA predicted by MGA, rSG–GA predicted by MSA, and rGA–SA predicted by MSG. These analyses improve in two ways on the only previous test (available in Figure 8.3 of Cvencek et al., 2012). First, the substantially larger numbers of studies and subjects in the present tests substantially increase power and precision. Second, the new analysis manages the treatment of scoring direction of IAT measures in a way that adds substantially to the statistical power and precision of regression intercept estimates. 10
For IAT measures, the 19 regression scatterplots, each combining all 108 (=3 × 36) correlation values (Figure 3A) all had strongly positive slopes, corresponding to correlations between .83 and .85. The 95% confidence intervals (CIs) for 17 of the 19 regression intercepts included the origin, and the widths of these CIs ranged from .049 to .053 on the Fisher Z scale used for the vertical axis. The scatterplot for the sample with the median of the 19 intercepts (−0.004) is displayed in Figure 3A. The data are remarkably consistent with the pattern expected from the balance–congruity principle’s prediction, which is not expected unless the measures being used have valid zero points.
The method of Figure 3A was applied also to the 48 correlations that were available for self-report measures in Figure 3B, producing 19 regressions and selecting the one with the median intercept for display. These 19 regressions also had positive slopes, corresponding to correlations ranging from .49 to .56. Fifteen of the 19 had 95% CIs that included the origin. The widths of those CIs ranged from .121 to .131 on the Fisher Z scale. Figure 3B’s data are therefore also consistent with validity of zero points for the collection of 48 self-report measures used in the present research, but suggest that, even with a measure that has relatively high test–retest reliability, there may be substantial individual subject variability in proximity of measures’ zero points to the desired indifference meaning of zero.
General Discussion
This review quantitatively assessed results obtained in studies of both IAT and self-report measures to evaluate both the balance–congruity principle of BIT (Greenwald et al., 2002) and the validity of zero points of IAT measures. In doing so, this review established four new findings.
First, the review found that predictions from BIT’s balance–congruity principle are confirmed not only for IAT measures, but also for self-report measures. In every way in which conformity to the balance–congruity principle could be compared for IAT and self-report measures, results revealed stronger confirmation of predictions with IAT than with self-report measures. In retrospect, the previously observed lack of confirmation for self-report measures (Cvencek et al., 2012) is most plausibly attributed to the lesser statistical power available in previously analyzed data sets.
Second, this review developed and reported a within-study meta-analytic test of the balance–congruity principle that is more efficient than the previously standard 4-test method. The within-study method not only reduced 12 statistical tests to two, but provided an index of fit to a pure multiplicative model.
Third, this review reported the first tests of the assumption that zero points of IAT and self-report attitude measures are validly interpretable as indicating absence of preference for one of (i.e., indifference between) two alternative concepts contrasted in the test. This valid zero value is required not only for tests of the balance–congruity principle, but also for meaningful interpretation of IAT measures. 11 Results of these tests (presented in Figures 2 and 3) consistently supported validity of the zero values, and did so more strongly for IAT than for self-report measures.
Fourth, this review confirmed the balance–congruity principle for the subset of studies that used self-esteem IAT measures, separately from confirming it for studies that used self-associations other than self-esteem. This finding for studies using IAT measures of self-esteem is useful, considering that the validity of measures of both explicit and implicit self-esteem has been questioned in the published literature (Baumeister et al., 2003; Bosson et al., 2000; Buhrmester et al., 2011).
Pure Multiplicative Model?
The introduction of this article explained that the three associations examined in any test of the balance–congruity principle are embedded in a larger associative network (see Figure 1) that includes other associations that can influence strengths of the three focal associations (see Equations 1–3). The involvement of each of the three focal associations in multiple triads of associations, only one of which is assessed in each study in the meta-analysis, necessarily diminishes the expectation that the data for any individual balanced identity study’s trio of associations should conform in pure fashion to the multiplicative form of the balance–congruity principle’s prediction. Nevertheless, the results in Figures 2A and 3A show that the data for IAT measures (much more than for self-report measures) were quite close to expectations based on the assumption of a pure multiplicative model. This suggests that the consistency processes theorized in BIT may be sustained or enhanced by the multiple triads in which any one measure participates.
Usefulness of the Within-Study Meta-Analytic Method for Testing BIT’s Balance–Congruity Principle
The within-study meta-analytic method introduced in this report provides an efficient alternative to the previously standard 4-test method for evaluating the balance–congruity principle. This method revealed (a) increased power of Test 2 (compared with the power of that test in the 4-test method), demonstrated by its averaged partial correlation coefficient being statistically significant for 25 of the 36 samples for which implicit measures were available, and (b) Test 2 successfully predicting fit of the pure multiplicative model as indexed by each sample’s total number (out of 12) of 4-test method tests passed. Although passage of all 12 tests of the 4-test method indicates purity of fit to the multiplicative prediction, it does not provide a quantitative index of strength of the multiplicative relationship—something that is provided by Test 2 of the within-study meta-analysis. In future research it will be reasonable to continue the use of the 4-test method, but it should be useful to report the within-study meta-analysis alongside. The two analyses complement one another. The two tests assess fit of data to balance–congruity predictions in complementary and mutually supportive ways.
Sources of Variance in IAT and Self-Report Measures
Interpretations of the present findings depend on understanding how IAT and self-report measures vary across testing occasions. Measures of internal consistency (such as split-half correlations or Cronbach’s alpha) estimate the proportion of variance on a single testing occasion that is consistently measured. For the IAT, internal consistency has been found to average r = .80 in the meta-analysis of 257 studies located by Greenwald and Lai (2020). The difference between the percentage of variance represented by this internal consistency (80%) and that represented by the same meta-analysis’s finding of test–retest reliability of r = .50 (50% of variance) indicates that 30% (=80% − 50%) of variance of IAT measures is attributable to variance across testing occasions.
Although meta-analytic estimates of internal consistency and test–retest reliability are not available specifically for the parallel self-report measures used in the meta-analyzed studies, these can be approximately estimated, respectively, as r ≈ .90 and r ≈ .80. Using those numbers, 10% (=90% − 80%) of variance of these self-report measures can be understood as variance across testing occasions. If both IAT and self-reports have valid zero points and both function exactly in the fashion predicted by the balance–congruity principle, the finding for the meta-analysis’s observed self-report data in Figure 3B should be close to the simulation for reliability r = .80, in the same way that the finding for the meta-analysis’s IAT data in Figure 3A is close to the simulation based on its expected reliability of r = .50. A plausible interpretation of the close similarity for IAT measures in Figure 3A is that the IAT measures behave closely in accordance with the balance–congruity predictions, with only small additional systematic sources of variance; additional variance across testing occasions is non-systematic, meaning that, on average, it does not create any directional distortion for IAT measures. For Figure 3B, the substantial gap between the predicted reliability of the r = .80 simulation and the observed self-report data indicates the presence of substantial systematic influences other than the balance–congruity principle contributing to the observed self-report data. The logical conclusion is that self-report measures have more systematic sources of artifact than do IAT measures.
Evidence for Validity of Zero Points of IAT Measures Used in Tests of BIT
Because BIT’s balance–congruity principle predictions are based on assumed validity of the zero interpretations of measures used in their tests, those predictions should be confirmed to the extent that measures used in their tests indeed possess valid zero points. Cvencek et al. (2012) meta-analyzed 18 studies that reported tests of BIT’s balance–congruity principle (54 correlational tests, three per study). Although Cvencek et al.’s results were consistent with the assumption that IAT measures had theoretically valid zero points, their analyses had insufficient power for precise tests for either IAT or self-report measures. The present meta-analysis had substantially greater power and precision.
Relative to non-displaced measures, zero-displaced IAT and self-report measures reduced correlations between products of two of the triad of measures in each study with the third measure. This was demonstrated in examination of the aggregated within-study results of Test 1 of the 4-test method (see Figure 2A and 2B). Still greater precision was available in the tests that examined BIT’s predictions concerning the correlation between the means of each of the three variables in the balanced identity design with the numerical value of the correlation between the other two measures. If zero points of the measures used in this test are valid, the regression of correlations of pairs of variables on means of the third variable should be positive in slope and should pass through the origin of the regression plot (i.e., the intercept of the regressions should be at or close to zero). The expected positive slopes were found for both IAT and self-report measures (Figure 3A and 3B), with the clarity of confirmation of this prediction again being considerably clearer for IAT measures (Figure 3A) than for self-report measures (Figure 3B).
This article’s zero-point validity analyses were conducted using sample-level correlations. That fact prompted a reviewer to question whether the article’s meta-analytic tests say more about sample-aggregate values of zero points than about individual-respondent zero points. This is not the case. The first BIT publication made clear that balance–congruity predictions depended on validity of the zero value at the level of individual subjects (Greenwald et al., 2002). Also relevant is that the present findings were based entirely on analyses that used individual subject data, never using multi-subject aggregations as variables in computed correlations.
Why Are Balanced Identity Patterns More Strongly Apparent With IAT Than With Self-Report Measures?
Greenwald et al. (2002) suggested two plausible causes for the relatively weak fit to expectations of the balance–congruity principle that they observed with self-report measures. First, subjects might lack introspective access to the strengths of the associations they are asked to report. Second, subjects might suppress accurate report on associations to which they have introspective access due to response factors such as demand characteristics (Orne, 1962), evaluation apprehension (Rosenberg, 1969), and subject role-playing (Weber & Cook, 1972). As an example, in a balanced identity design involving White–Black contrasts in racial identity (SG), self-esteem (SA), and racial attitudes (GA), White participants who want to appear unbiased might suppress report of an internally known racial bias, instead reporting either no bias or perhaps an out-group preference.
Previous discussions of differences between IAT and self-report measures have focused on results for single IAT or self-report measures. These explanations do not immediately apply to differences in affective–cognitive consistency findings involving relations among trios of measures, as in the balance–congruity tests summarized in this article. Nevertheless, the response factors explanation of IAT–self-report differences provides some basis for expecting greater consistency among self-report than IAT measures, to the extent that response factors include conscious reasoning that might increase evidence for consistency. The present findings of greater evidence for consistency with implicit measures, which presumably limit opportunity for conscious reasoning, oppose that suggestion. It is time to consider the possibility that consistency processes may operate outside of conscious awareness. Theory to explain such automatic operation of affective–consistency processes is not yet developed.
One of the noteworthy findings of this report is the confirmation of BIT predictions with self-report measures. This result is in line with general idea of differences between associative and propositional representations (Strack & Deutsch, 2004). According to Strack and Deutsch (2004), “the reflective system is driven by the principle of consistency as it strives to avoid or remedy inconsistencies between its elements” (p. 225). Following this first theoretical effort to account for implicit–explicit divergence in attitudes, a subsequent conceptual model was proposed: the associative–propositional evaluation (APE) model (Gawronski & Bodenhausen, 2006). The APE model holds that explicit evaluations are the behavioral outcome of propositional processes, which are defined as the validation of the information implied by activated (automatic) associations (Gawronski & Bodenhausen, 2014). According to the APE model, “cognitive consistency is exclusively a concern of propositional reasoning” (Gawronski & Bodenhausen, 2006, p. 695). However, the APE model was not stated in a way that would account for the strong findings with IAT measures, nor does it generate a prediction that associative measures should outperform propositional measures. Future research will profit from examining conditions under which consistency is more likely to arise from propositional versus associative processes.
Conclusion
BIT’s balance–congruity principle predicts that people with positive self-esteem should have positively valenced attitudes toward concepts or groups that are closely associated with the self. This review provides the strongest evidence yet available for this theoretical expectation of BIT. It also solidifies previous indications that BIT’s support is more evident in studies using IAT measures than in those with self-report measures. The within-study meta-analytic strategy introduced in this article to test balance–congruity predictions was found to offer an efficient alternative to the original 4-test method for testing those predictions. Tests that displaced the zero points of IAT measures (prior to using them in tests of predictions involving multiplicative products of measures) confirmed the assumption that (non-displaced) IAT measures have the valid zero points that are required in tests of BIT’s balance–congruity principle. As a group, the subset of studies in this meta-analysis that included IAT self-esteem measures confirmed BIT predictions, comparable to the subset of studies in which IATs measured the associations of self with attributes other than valence. BIT’s balance–congruity principle has now been confirmed when tested either with IAT or self-report measures.
Supplemental Material
PSPB_916631_Corrected_SUPPLEMENT – Supplemental material for Meta-Analytic Use of Balanced Identity Theory to Validate the Implicit Association Test
Supplemental material, PSPB_916631_Corrected_SUPPLEMENT for Meta-Analytic Use of Balanced Identity Theory to Validate the Implicit Association Test by Dario Cvencek, Andrew N. Meltzoff, Craig D. Maddox, Brian A. Nosek, Laurie A. Rudman, Thierry Devos, Yarrow Dunham, Andrew S. Baron, Melanie C. Steffens, Kristin Lane, Javier Horcajo, Leslie Ashburn-Nardo, Amanda Quinby, Sameer B. Srivastava, Kathleen Schmidt, Eugene Aidman, Emilie Tang, Shelly Farnham, Deborah S. Mellott, Mahzarin R. Banaji and Anthony G. Greenwald in Personality and Social Psychology Bulletin
Footnotes
Author Contributions
All listed authors contributed data; D.C., C.D.M., and A.G.G. analyzed data; D.C., A.G.G., A.N.M., and C.D.M. wrote the paper.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported in part by a Grants from the National Science Foundation (HRD-1661285 to A.N.M. and SBE-1640889 to D.C.) and by the University of Washington’s Implicit Cognition Research Fund.
ORCID iDs
Notes
Supplemental Material
Supplemental material is available online with this article.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
