The Language of Performance Evaluations

Method

Actual performance evaluations of all 268 junior attorneys working in a single Wall Street law firm were obtained by the second author (M.J.T.) who was hired by the firm as a consultant to evaluate its annual review procedures. The data were collected in 2006 as part of the regular performance review process. The firm can be characterized as a typical Wall Street firm and perhaps typical of mid-size to large New York law firms. As part of the consultancy contract, clients agree to allow for publication of findings as long as anonymity is maintained.

Each junior attorney had been evaluated by senior lawyers in the firm (referred to throughout as evaluators) with whom they had had some work contact. Each evaluator offered numerical judgments of and open-ended narrative comments about the target attorney. These individual judgments were not made available to the junior attorneys but did form the basis of a summary evaluation each attorney received at the end of the review period. The junior attorneys were predominantly White (75.4%) or Asian (13.4%), and our analyses focused on this subgroup of 234 (84 female and 150 male) attorneys. The vast majority of the evaluators were men, and indeed, we focus here only on evaluations offered by these senior supervising male attorneys (mean [M] number of evaluators = 3.59, standard deviation [SD] = 2.31, range = 0–22). ¹

Numerical judgments were made on 26 dimensions, using 1–5 response scales where 1 = seriously below expectations, 2 = below expectations, 3 = meets expectations, 4 = exceeds expectations, and 5 = outstanding. The 26 judgments were divided by the firm into 7 categories: technical excellence, effective lawyering, delivery of client services, teamwork and leadership, delegation and training of subordinates, attitude and personality, and dedication to the firm’s mission. The items were highly intercorrelated, rs from .76–.93, strongly suggestive of a halo effect rather than nuanced discrimination among performance factors. To reduce the data, we computed the average numerical rating across all 26 dimensions (α = .98) and refer to this index as the “numerical rating.” This is the most consequential of the evaluations attorneys receive as numerical ratings matter for outcomes such as raises and promotion: Common knowledge in the firm is that only attorneys who receive mostly “5s” are headed toward partnership.

In addition to the numerical ratings, evaluators offered open-ended narrative comments about the attorneys. These were analyzed in two ways. First, the narratives were submitted to a text analysis software system (Linguistic Inquiry Word Count; Pennebaker, Francis, & Booth, 2001), which performs counts of many types of word categories. Our interest was in the positivity of these comments, so we focused on the frequency of positive performance words (including excellent, good, awesome, terrific, stellar, wonderful, etc.).There was no difference in the total number of words written about female and male attorneys, t < 1 (M _male = 386, SD = 285; and M _female = 394, SD = 304).

Second, we trained independent coders to rate the open-ended descriptions, with gendered pronouns and any other references to gender removed. Two law students rated each narrative comment for technical competence and interpersonal warmth (using 1–5 [very negative–very positive] rating scales). Technical competence included references to analysis skills, drafting, legal judgment, efficiency, and productivity; interpersonal warmth included references to friendliness, warmth, attentiveness, responsiveness, and good communication/relationships with coworkers. Two undergraduate student coders also rated each block of text on overall warmth/friendliness and overall technical/drafting skills using the same scales. They also indicated whether the evaluator made any mention of partnership, using a 3-category system (−1 = negative mention, e.g., “not partner material;” 0 = no mention, 1 = positive mention, e.g., “on the right track to partnership”).

We averaged across the four coders’ judgments of technical competence/drafting skill to compute an overall “narrative rating of technical competence” index (Krippendorff’s α = .63) and across the four coders’ warmth/friendliness ratings to produce a “narrative rating of interpersonal warmth” index (Krippendorrf’s α = .61). The two undergraduate coders’ partnership likelihood judgments were combined to produce a “partner likelihood” index (Krippendorrf’s α = .81). ²

Results

Correlations Among Judgments

We next computed correlations among the numerical ratings and narrative-coded attributes, separately for female and male attorneys. These appear in Table 1 ; entries above the diagonal refer to female attorneys and those below to male attorneys.

OPEN IN VIEWER

Table 1.

Correlations Among Evaluation Components, by Attorney Sex

	Numerical Rating	Narrative Technical	Narrative Warmth	Positive Performance	Partner
Numerical rating	–	.63 *	.65 *	−.06	.21 *
Narrative technical competence	.72 *	–	.55 *	.13	.16
Narrative warmth	.63 *	.70 *	–	.02	.13
Positive performance words	.17 *	.30 *	.22 *	–	.14
Partner likelihood	.51 *	.33 *	.39v	.04	–

Correlations above diagonal and boldfaced values = female attorneys (Ns = 83–86); below diagonal = male attorneys (Ns = 147–152).

^* p < .05.

In the case of male attorneys, numerical ratings were correlated with all of the other coded variables, which themselves were intercorrelated, suggesting a great deal of consistency in supervisors’ judgments of men. For female attorneys, there was less consistency overall among these components, although narrative ratings of technical competence, interpersonal warmth, and partner likelihood did predict numerical ratings. Partner likelihood predicted positive narrative mentions of technical competence and interpersonal warmth for men but not for women. We examined apparent sex differences more explicitly by computing multiple regressions that included interactions between predictors and attorney sex.

Effects of narrative ratings of technical competence and interpersonal warmth on numerical ratings, by attorney sex

We regressed numerical ratings on attorney sex (centered), technical competence (centered), interpersonal warmth, and interactions between these predictors and attorney sex. Both interactions were significant: Attorney Sex × Technical Competence, B = .15, SE = .07, t(228) = 2.20, p < .05, and Attorney Sex × Interpersonal Warmth, B = −.13, SE = .07, t(228) = −2.01, p < .05.

We decomposed these interactions using utilities developed by Preacher, Curran, and Bauer (2006). These are depicted in Figure 1A (technical competence) and 1B (interpersonal warmth), using levels of the relevant predictor variable set at 1 SD above and below the mean. As can be seen in Figure 1A, narrative ratings of technical competence predicted numerical ratings for both female and male attorneys, ps < .001, but the effect of technical competence was stronger for men than women. Additionally, the effect of attorney sex was not significant at low levels of perceived technical competence, p > .30 but was significant at high levels of perceived technical competence, B = .10, SE = .03, t(228) = 2.83, p < .01. That is, numerical ratings of male and female attorneys did not differ when attorneys were rated low on the narrative measure of technical competence, but numerical ratings were higher for men than women when narrative technical competence was high.

OPEN IN VIEWER

Figure 1.

Interactions between attorney sex and narrative predictors on numerical ratings: Technical competence (A), interpersonal warmth (B), positive performance words (C), and partner likelihood (D)

Figure 1B indicates that narrative ratings of interpersonal warmth predicted numerical ratings for both female and male attorneys, ps < .001, but warmth mattered significantly more for women. Furthermore, the effect of attorney sex was significant at low levels of warmth, B = .09, SE = .01, t(228) = 6.96, p < .001, but not at high levels of warmth, p > .25: Among attorneys described as low in interpersonal warmth, women received lower numerical ratings than men, but no sex difference emerged among attorneys described as high in interpersonal warmth.

Positive performance word effects on numerical ratings by attorney sex

The interaction between attorney sex and positive emotion words was also significant, B = .04, SE = .02, t(229) = 2.10, p < .05 and is plotted in Figure 1C. For men, positive performance words signaled positive numerical ratings, B = .05, SE = .02, t(229) = 2.38, p < .02, but for women they did not, p > .20. Additionally, the sex difference in numerical ratings was significant when attorneys were described with high numbers of positive performance words, B = .22, SE = .08, t(229) = 2.84, p < .01, but not when they were described with few positive performance words, p > .35.

Partner likelihood effects on numerical ratings and attorney sex

Regression analysis also confirmed a significant Attorney Sex × Partner Likelihood interaction on numerical ratings, B = .45, SE = .21, t(228) = 2.17, p < .01 (see Figure 1D). Partner likelihood predicted ratings for both sexes, ps < .01, but the effect was stronger for male attorneys. At low levels of partner likelihood, the effect of attorney sex was not significant, p > .70, but at high levels of partner likelihood, men received better numerical ratings than women, B = .13, SE = .04, t(228) = 3.61, p < .001. We also examined this pattern by comparing numerical ratings received by attorneys for whom at least one evaluator mentioned that partnership was likely. The 5 women who received any mention of partnership potential achieved numerical ratings of M = 4.27, SD = .27, whereas the mean numerical rating for the 22 partner-identified men was M = 4.46, SD = .32, d = .43.

Other variables

Other apparent differences in correlational patterns for female and male attorneys (Table 1) were not supported in multiple regressions testing interactions. For example, narrative technical competence and interpersonal warmth did not differentially predict partner likelihood mentions for women and men.

Discussion

Given the masculine context of finance law, we predicted that male attorneys would receive more favorable numerical ratings than female attorneys in this Wall Street firm. This was indeed the case, and based on the requirement of high ratings for partnership, more men than women in this firm were headed toward partnership. Of course, this finding alone tells us little about the presence of gender bias, for it is possible that men were objectively “better” performers than women. But arguing against this possibility is the fact that the pro-male bias was not apparent in narrative comments, which revealed greater reference to positive performance words for women than men, and no sex differences in narrative ratings of technical competence and interpersonal warmth. If men were objectively better, one might expect to see evidence of this in all forms of evaluation. Furthermore, when even the best men and women were compared—those judged equally highly favorably in narrative mentions of partner likelihood and technical competence—men still fared better in terms of the numerical ratings they received (more on this below).

Other findings in the performance evaluation literature have pointed to relatively low correlations between types of workplace evaluation and to different patterns of group differences depending on the type of evaluation (Roth, Huffcutt, and Bobko, 2003; Wilson, 2010). The precise cause of the different patterns of attorney sex effects cannot be specified with these data, but we suggest that gender stereotypes led to numerical judgments that favored men and to the use of shifting standards in narrative language (Biernat, 2003). To the extent that men’s interpersonal warmth and women’s technical competence were considered relative to lower within-sex standards, stereotypical patterns of judgment may have been masked in open-ended language.

But it is also possible that heightened positivity toward women in narrative comments could reflect a kind of motivated “softening” of the message delivered in numerical ratings. Evaluators may have wanted to “throw a bone” to women, making their relatively negative numerical ratings appear less harsh (whether this harshness was driven by stereotyping or objective performance). They may also have been concerned about appearing prejudiced and sought to amplify their praise of women accordingly, describing them with more positive performance words and with assessments of technical skills and interpersonal warmth equaling that of men (see Harber, 1998, for a race-based example). But because junior attorneys did not have direct access to the narrative comments, we think it unlikely that motivated “softening” occurred, unless this pose was for the benefit of the other supervisors.

Instead, we suggest that gender stereotypes led to pro-male bias on the evaluative judgments that mattered, but that narrative references to technical competence and performance reflected the use of lower standards for women, which therefore muted or reversed the pro-male bias in numerical ratings.

Evidence of gender stereotyping is further suggested by the differential patterns of correlation between numerical ratings and other judgments for female and male attorneys. In general, there was more consistency between numerical ratings and attributes coded from the open-ended narratives for male than female attorneys. And though narrative interpersonal warmth and technical competence were correlated with numerical ratings for both female and male attorneys, multiple regression analysis indicated a sex-typed pattern: Technical competence mattered more for numerical ratings of men than women, and interpersonal warmth mattered more for numerical ratings of women than men.

Other research points to the importance of likeability in judgments of women in the workplace (Sidanius & Crane, 1989). For example, women who are described as “successful managers” nonetheless suffer evaluation decrements relative to men because they are perceived as lacking warmth (Heilman, Block, & Martell, 1995; Heilman, Wallen, Fuchs, & Tamkins, 2004). Our data also indicate that women’s numerical ratings suffered, relative to men’s, when they received low assessments of warmth. However, women did not particularly benefit relative to men when they were judged as high in warmth: Numerical ratings were roughly equal for “high-warmth” women and men. Thus, lacking warmth harmed women but having warmth did not offer them any greater benefit than it did men.

Also consistent with gender stereotyping patterns, narrative perceptions of technical competence mattered more for the numerical ratings of male than female attorneys. Partner likelihood was also higher for men than women, and mention of partnership better predicted positive numerical ratings of male attorneys. What distinguished the “male competence” connection from the “female warmth” connection was that for men, receiving a low narrative assessment of technical competence did not harm them relative to women, but receiving narrative comments indicating high technical competence benefited them relative to women. That is, while both men and women may have been held to stereotyped expectations, women particularly suffered when they did not meet expectations, and men gained when they did.

These data also suggest that for women who were described in narratives as having high levels of perceived technical competence (and as partner material), numerical evaluations still did not match those of comparable men. For example, women described in the narratives as equivalently high in technical competence to men (e.g., at 1 SD above the mean) received numerical ratings of about half a SD lower than men’s. This difference could easily reduce partnership likelihood for women.

We also found that greater use of positive performance words in narrative descriptions of women—the only mean-level outcome that favored women—had no impact on the numerical ratings women received. However, positive performance word use for men did predict positive numerical ratings. This suggests that the positive narrative characterizations of women were more apparent than real; narrative comments about women may have reflected inflated positivity relative to low group level expectations—which did not translate into the consequential numerical ratings that matter for promotion.

This pattern of findings is reminiscent of Castilla’s (2008) performance–reward bias effect, whereby performance evaluations matter less for salary increases for women and minorities than for White men. Castilla argues that “employers consciously or unconsciously discount the performance ratings of employees because of their gender, race, or nationality,” when it comes to making salary decisions (p. 1483). We are doubtful that explicit discounting played a role in our data, as narrative comments were generated at the same time as numerical ratings, by the same evaluators. In Castilla’s data, evaluations were generated in one stage and wage determinations made in a second stage, often by different evaluators. Still, stereotype-based expectations underlie both effects. Of course, we did not examine the effects of evaluations on salary increases. But in this law firm, close-ended ratings are the outcomes that matter; only attorneys who receive mainly 5s are headed for partnership. Men are clearly favored in these judgments, and these judgments are more directly tied to the positive narrative comments about performance that men than women received.

Overall, our data indicate that male attorneys received evaluative advantages at this firm. While the absolute sex difference in numerical ratings (Ms of 3.90 vs 3.76 on 5-point scale) may not seem very large, the firm’s reliance on this number for partnership consideration makes it nearly three times more likely that men than women will be promoted to partner. Again, it is possible that this difference reflects objective performance differences rather than gender bias, but we think this is unlikely for several reasons. First, narrative content did not support the numerical rating difference—if men were objectively “better,” this should have been evident in all forms of evaluation. Second, though men and women were perceived equivalently in technical competence and interpersonal warmth, these factors predicted consequential numerical ratings in sex-typed ways. Such patterns cannot be explained by objective performance differences. Third, even when one considers the most highly regarded women—those who were “matches” to men in terms of narrative description of high technical competence and partnership material—their numerical ratings did not match those of men.

These data provide a snapshot of the evaluation process of one law firm at one point in time, and our analyses suggest that gender affected the evaluations junior attorneys received. The real-world context did not offer us the tight control of the lab, in which employee sex could be isolated as a causal factor in evaluative bias and process could be assessed. But this setting provided a valuable venue for testing predictions about subtle and complex forms of gender-stereotyping and for considering consequences for real employees being judged by real evaluators in real time.