An Effort to Close Achievement Gaps at Scale Through Self-Affirmation

A Pattern of Disengagement

GPA, motivation, academic engagement, and achievement goals appear to decline during middle school for all students, and African American and Latino students appear to suffer steeper declines in school performance than their Asian and White peers (Anderman, 2003; Borman, Stringfield, & Rachuba, 2000; Cook, Purdie-Vaughns, Garcia, & Cohen, 2012; Sherman et al., 2013; Shim, Ryan, & Anderson, 2008). ¹ Student achievement appears to mirror this trend. Although there are empirical exceptions (see Reardon & Galindo, 2009), substantial evidence suggests that achievement gaps between White and non-White students persist and even grow as they progress through school (Benson & Borman, 2010; Downey, von Hippel, & Broh, 2004; Fryer & Levitt, 2004; Jencks & Phillips, 1998). The National Assessment of Educational Progress (NAEP) shows that achievement gaps persist between Black and White students (Vanneman, Hamilton, Baldwin Anderson, & Rahman, 2009) and between Hispanic and White students (Hemphill, Vanneman, & Rahman, 2011) and that differences in mathematics (but not reading) achievement appear to grow between age 9 and age 13. Recent work in social psychology suggests that stereotype threat may contribute to these patterns of disengagement and inequality.

Stereotype Threat

Claude Steele and Joshua Aronson coined the term “stereotype threat” in a seminal article published in 1995. As they originally described,

[W]henever African American students perform an explicitly scholastic or intellectual task, they face the threat of confirming or being judged by a negative societal stereotype—a suspicion—about their group’s intellectual ability and competence. . . . And the self-threat it causes—through a variety of mechanisms—may interfere with the intellectual functioning of these students, particularly during standardized tests. (p. 797)

Stereotype threat also has been shown to affect stereotyped individuals’ performance in a number of domains beyond academics, such as White men in sports (e.g., Stone, Lynch, Sjomeling, & Darley, 1999), homosexual men in providing child care (Bosson, Haymovitz, & Pinel, 2004), and women in negotiation (Kray, Galinsky, & Thompson, 2002), but within the academic realm, it is thought to depress the performance of women in mathematics and related fields and African Americans and other non-Asian racial/ethnic groups in all academic areas. Students who are aware that they belong to a group that is perceived to perform poorly academically often fear behaving in a way that fits the negative cultural image associated with a group stereotype, thereby marking them as inferior. This largely unconscious fear elicits anxiety and other counterproductive responses that can severely interfere with thinking and performance on standardized tests or other evaluative activities in the classroom.

Although evidence continues to accumulate regarding the specific mechanisms through which stereotype threat depresses academic performance, most researchers agree that physiological and psychological mechanisms such as stress responses, self-monitoring, and self-regulation are at play (Schmader, Johns, & Forbes, 2008), which are in turn manifest in test-taking behaviors (Scherbaum, Blanshetyn, Marshall-Wolp, McCue, & Strauss, 2011). In addition, Aronson (2002) noted that perceptions of negative stereotypes lead many individuals to engage in activities such as self-handicapping (Smith, 2004), challenge avoidance (Good et al., 2003), and self-suppression (Pronin, Steele, & Ross, 2004; C. M. Steele, 1997). Beyond these correlates with poor academic performance, stereotype threat has also been linked to high blood pressure among African Americans (Blascovich, Spencer, Quinn, & Steele, 2001) and social distancing, particularly from the stigmatized social group of which the participants are members (Pronin et al., 2004). Meta-analyses of dozens of laboratory experiments suggest that standard measures of ability underestimate the true abilities of Black and Latino students by about one quarter of a standard deviation (Nguyen & Ryan, 2008; Walton & Cohen, 2003). A meta-analysis of field experiments suggests that the psychological threats that African American and Hispanic students suffer while taking standardized tests cause them to underperform by one fifth of a standard deviation (Walton & Spencer, 2009).

Recent work has extended the initial concern with performance to address how stereotype threat might implicate ongoing learning. In other words, stereotype threat may be a form of “double jeopardy” that interferes with both short-term performance and long-term knowledge acquisition (Taylor & Walton, 2011). Boucher, Rydell, Van Loo, and Rydell (2012) found that the mathematics learning of women—but not men—was compromised by negative stereotypes about women’s math ability. Also, Taylor and Walton (2011) found that African American students—but not White students—who studied under threatening conditions fared worse on a follow-up assessment that was conducted under nonthreatening conditions. The implication of these findings is that removing threats during evaluative situations alone will not be sufficient if negative stereotypes are pervasive. Short-term underperformance due to stereotype threat may lead students to alter their career and/or professional aspirations and their sense of belonging in academic domains and contexts (J. Steele, James, & Barnett, 2002) and to “protectively disidentify” from academics (Aronson, 2002; Major, Spencer, Schmader, Wolfe, & Crocker, 1998; C. M. Steele, 1997), which in turn could lead to increasing learning deficits (Appel & Kronberger, 2012). In this way, disengagement with the task or the context in which the task is to be performed can lead to growing disadvantages among negatively stereotyped students and may play a prominent role in the patterns of disengagement and widening achievement gaps.

Members of nonstereotyped groups may also respond to stereotypes, although in their case their performance is boosted by “stereotype lift,” the converse of stereotype threat (Walton & Cohen, 2003). Introducing negative stereotypes about other groups—which happens “more or less automatically” during evaluative tests (Walton & Cohen, 2003, p. 456)—may improve performance unless the negative stereotype is removed. This phenomenon has been observed in laboratory settings but not in field experiments. Although it is possible that removing the effect of stereotype threat could negatively affect the performance of White and Asian students, field experiments have not revealed a trade-off between the performance of White students and African American or Latino students (Cohen et al., 2006; Cohen et al., 2009; Sherman et al., 2013). The possibility, however, suggests that we should evaluate whether an intervention to reduce stereotype threat influences all student subgroups.

Self-Affirmation

Self-affirmation (Liu & Steele, 1986; C. M. Steele, 1988; C. M. Steele & Liu, 1983) is one way in which students may be able to cope with stereotype threat in a productive way. Closely related to “compensatory self-inflation” (Greenberg & Pyszczynski, 1985), self-affirmation helps individuals bolster other dimensions of their self-worth to compensate for the harm done by stereotype threat (Sherman & Cohen, 2006; C. M. Steele, 1988). In the absence of a means to repair their self-image, students may respond to stereotype threat in defensive and unproductive ways, such as disidentification with academics (Appel & Kronberger, 2012; Sherman & Cohen, 2006).

Values affirmation—in which individuals focus on things that are important to them—is an especially promising form of self-affirmation that has been shown to have beneficial effects on numerous outcomes in laboratory experiments, mostly involving undergraduate students in the United States (McQueen & Klein, 2006). ² Because self-worth is multidimensional (Crocker & Wolfe, 2001), individuals can compensate in one area—that is, values—for a deficit in another—that is, group identity—and thereby maintain a positive overall sense of themselves (Sherman & Cohen, 2006). There are a number of ways to engage in values affirmation, but expressive writing is the most commonly used approach, especially among large groups of students as one would find in a school setting (Yeager & Walton, 2011).

One field-based trial of self-affirmation exercises, in particular, is of key policy and evaluative concern; because of its relatively brief and simple administration yet remarkably positive results, it has garnered a great deal of both positive attention and skepticism. In results published in Science, Geoffrey Cohen and his colleagues (Cohen et al., 2006; Cohen et al., 2009) reported that brief self-affirmation tasks aimed at affirming students’ personal values reduced the Black–White GPA gap by as much as 40%. Specifically, having middle school students write a series of brief essays affirming their personal values over the course of the school year improved the performance of African American students in four core academic subjects: science, social studies, math, and English. The three classrooms in a suburban northeastern middle school in which the research took place were composed of approximately half African American and half European American students. The seventh-grade students were placed at random into intervention and comparison groups near the start of the school year. Both groups were given structured 15-min writing assignments a total of 3 to 4 times during the academic year. These assignments primed the students to think about important qualities and values; students selected three values (e.g., sports talent, family, sense of humor) and explained in writing why they were personally important. Compared with control group students (who wrote about other non-self-affirming topics, such as why values they considered unimportant might be important to others), the African American students in the affirmation group had substantially higher GPAs, both during the school year in which the interventions were deployed (Cohen et al., 2006) and 2 years later (Cohen et al., 2009).

How might a relatively small and inexpensive intervention lead to such substantial changes in academic performance? As proximal, relatively “quick wins” accumulate, the researchers note that recursive processes acting like chain reactions then carry forward the initial effects of the intervention (Cohen et al., 2009). As Purdie-Vaughns and her colleagues (2009) argued, a small improvement early in the year due to the intervention might, for example, give children a little extra confidence, and this confidence might lead to further gains in performance in a repeating cycle. Teachers may also play a role. Even though they do not know the experimental condition to which students were randomized, small early improvements may cause teachers to see students as more able and worthy of attention and mentoring, thus amplifying the effects of the invention via teacher expectancy effects (Purdie-Vaughns et al., 2009).

Indeed, the greatest impacts of self-affirmation came in the final term of the school year, which Cohen and his colleagues offer as evidence of a beneficial recursive process for students exposed to the self-affirmation intervention: “Findings suggest that because initial psychological states and performance determine later outcomes by providing a baseline and initial trajectory for a recursive process, apparently small but early alterations in trajectory can have long-term effects” (Cohen et al., 2009, p. 400). By enhancing students’ feelings of personal worth, the authors surmised, the exercise changed their perception of bias at school and shifted how they interpreted their academic successes and failures. These steps particularly protected African American students who had been struggling in school. Instead of feeling discouraged and falling into a pattern of disengagement, it as though the psychologists gave Black students an inoculation against the threat presented by negative stereotypes.

This research—which was reviewed by the What Works Clearinghouse (WWC) and deemed consistent with WWC evidence standards (WWC, 2010)—demonstrated that African American students who completed the writing exercises about their values increased their average seventh- and eighth-grade GPA by a quarter of a letter grade (0.24 points), a change that was statistically significant and equivalent to an effect size of approximately d = 0.30. Among low-achieving African American students, the effect was somewhat larger, with an increase in average seventh- and eighth-grade GPA of 0.41 points. In addition, the intervention reduced the likelihood that low-achieving African American students would be assigned to a remedial program or retained in grade (5% vs. 18%). The intervention did not have a statistically significant effect on the academic outcomes of White students.

More recently, Sherman et al. (2013) reported on a similar long-term field experiment of self-affirmation conducted with Latino and White students. The authors reported findings from two studies from two schools in which they randomly assigned values-affirmation writing exercises to White and Latino students and treated GPA as the outcome of interest (by term and cumulatively). The authors observed no treatment effect for GPA among the White students, but affirmed Latino students had higher GPAs than nonaffirmed, control Latino students. In Study 1, the cumulative GPA of affirmed Latino middle school students (Grades 6–8) was 0.22 points higher than nonaffirmed Latino students at the end of the first year, representing a 22% reduction in the initial White/Latino GPA gap, a difference that persisted for the 3-year duration of the study (Sherman et al., 2013). At the end of the first year of Study 2, the cumulative GPA among affirmed Latino seventh-grade students was 0.38 points higher than that of nonaffirmed Latino control students, which represents approximately a 30% reduction in the White–Latino GPA gap (Sherman et al., 2013). The direction and magnitude of the impacts of self-affirmation found by Sherman and his colleagues (2013) among Latino students are similar to those found by Cohen and his colleagues (2006, 2009) among African American students. Stereotype threat appears to influence both African American and Latino students, and interventions to address stereotype threat appear to benefit both African American and Latino students similarly (Cohen et al., 2006; Cohen et al., 2009; Good et al., 2003; Sherman et al., 2013).

Scaling Up Self-Affirmation

Our districtwide sample includes both African American and Latino students, and we have little theoretical or empirical reason to believe that the two groups will respond to self-affirmation significantly differently from one another, on average. Consequently, in our analyses, we contrast two groups of students: those who are potentially subject to stereotype threat and those who are not potentially subject to stereotype threat. We deliberately use the word potentially because we recognize that these racial/ethnic classifications are merely proxies for students who are suffering from stereotype threat. Not all students who could experience stereotype threat in fact do experience it. Stereotype threat includes a number of boundary conditions that we cannot identify in the field, such as awareness of the stereotype (C. M. Steele & Aronson, 1995) or identification with the domain (Aronson et al., 1999). Laboratory experiments can screen participants on these conditions, but field experiments cannot do so as readily. The students we identify as being potentially subject to stereotype threat—and therefore potentially benefiting from self-affirmation—will vary substantially in their susceptibility to stereotype threat and their responses to the self-affirmation treatment. We can, however, evaluate the extent to which self-affirmation benefits, on average, a sample of students who are potentially suffering from stereotype threat and evaluate whether they benefit from being assigned self-affirmation writing exercises in school. As a result, our results offer a conservative estimate of the impact of self-affirmation on students who are subject to stereotype threat because, presumably, the larger effects of self-affirmation on threatened students may be diluted by the null effects on nonthreatened students.

This estimate, however, offers a realistic expectation of what a district with large achievement gaps might expect if it introduces these activities to its curriculum. Recent evidence from the NAEP suggests that the academic achievement gaps between Whites and Blacks (Vanneman et al., 2009) and Whites and Hispanics (Hemphill et al., 2011) in Wisconsin—where the experiment was conducted—are among the largest in the United States. If the results from this remarkably inexpensive and relatively simple intervention can be replicated and better understood, it could hold tremendous potential for closing persistent achievement gaps both in Wisconsin and in the United States.

Our overarching research question—Can self-affirmation mitigate the effects of stereotype threat on non-White students in a real-world setting?—is applied to several standard measures of students’ academic performance. Three measures of school success provide distinct evidence concerning how self-affirmation may affect both learning and test performance. As our literature review suggested, stereotype threat may operate through two distinct mechanisms: by increasing test anxiety and thus lowering the validity of test scores and by provoking academic disengagement and thus reducing actual learning. Our analyses of the cumulative effects of self-affirmation on year-end GPA assess the potential for self-affirmation to operate recursively—potentially affecting learning—as suggested by the work of Cohen et al. (2009), Sherman et al. (2013), and Taylor and Walton (2011). In separate analyses, we assess impacts on a fall standardized test given after a low dosage of treatment and a spring standardized test after a higher dosage. Specifically, we ask (a) to what extent two doses of the affirmation intervention affect reading and mathematics scores on a state standardized test; (b) the extent to which four doses affect reading, language use, and mathematics scores on a different standardized test; and (c) the extent to which four doses affect the cumulative GPA of seventh-grade students. We test two hypotheses for each of these three outcome measures.

Prior research generally finds no impact of self-affirmation on these students (primarily White; Cohen et al., 2006; Cohen et al., 2009; Sherman et al., 2013), but the possibility of “stereotype lift” suggests that the impact of self-affirmation could be negative for White and Asian students (Walton & Cohen, 2003). Our null hypothesis is that there is no difference associated with affirmation for students who are not potentially subject to stereotype threat. We evaluate this hypothesis with the “self-affirmation” estimates in the statistical models that interact an indicator of students’ status as a potentially threatened racial/ethnic minority with assignment to self-affirmation. A related question is whether self-affirmation affects all students on average regardless of their racial/ethnic background, which we evaluate in models that do not include interaction terms.

Prior research suggests that African American and Latino students respond differently to self-affirmation than White students (Cohen et al., 2006; Cohen et al., 2009; Sherman et al., 2013). Our null hypothesis is that all students respond equivalently to self-affirmation regardless of their racial/ethnic background. We evaluate this hypothesis with a statistical interaction term between treatment condition and an indicator of students’ status as a potentially threatened racial/ethnic minority. We also ask a closely related question: Is there a self-affirmation treatment effect for the subgroup of students who are potentially subject to stereotype threat? Prior research indicates self-affirmation positively affects the GPA of African American and Latino students (Cohen et al., 2006; Cohen et al., 2009; Sherman et al., 2013). Our null hypothesis is that there is no impact of self-affirmation on African American and Latino students. We evaluate this hypothesis by using a postestimation strategy to compute marginal effect estimates from our statistical interaction model, thereby providing estimates of the effects of self-affirmation for students who are potentially subject to stereotype threat.

Madison Writing and Achievement Project (MWAP)

To evaluate these hypotheses, we use data from the MWAP, a multiyear, districtwide randomized field trial of self-affirmation writing conducted in the Madison Metropolitan School District (MMSD). The study involved the entire seventh grade in 2011–2012, which consisted of 80 classrooms taught by more than 50 teachers in 11 middle schools. To our knowledge, MWAP is the first evaluation of self-affirmation writing to be conducted across an entire school district. The design of the study is shown in Figure 1.

OPEN IN VIEWER

Figure 1.

MWAP study design and data collection.

The intervention consisted of a sequence of four writing exercises students completed in school over the course of the school year. Following Cohen and his colleagues (2006, 2009; Sherman et al., 2013), one half of the students was randomly assigned to complete a self-affirmation writing exercise while the other half was assigned to a similar, but nonaffirming control writing exercise. We implemented a randomized block design, in which the 11 schools served as blocks, and randomization was conducted at the student level within each of the blocks. We randomized students after securing parental consent and student assent. The self-affirmation assignments varied slightly over the course of the year to maintain student interest. The first exercise took place as close to the beginning of the school year as was feasible (Cook et al., 2012; Critcher, Dunning, & Armor, 2010). The remaining exercises were intended to take place prior to assessments: the Wisconsin Knowledge and Concepts Exam (WKCE) in November and the Measures of Academic Progress (MAP) in February and May. Nonconsented students completed an expository writing exercise provided by the research team so that all students were writing at the same time.

The first two writing exercises were structured, the third was open-ended, and the fourth exercise was tailored to the student based on his or her choices in the first or second exercise (see Sherman et al., 2013). The structured exercises asked students to choose two or three items that are most important to them from an 11-item list, and the comparison students were offered an identical list and asked to identify the two or three least important items and write about how they might be important to someone else. ³ Both structured exercises included follow-up questions to reinforce their reflection. The open-ended exercise presented students with a paragraph summarizing the kinds of things that can be important to people, using examples from the structured exercise list. Students in the affirmation condition were then asked to describe something that is important to them. Students in the comparison condition were presented with a paragraph summarizing the kinds of things students do in the morning before they go to school and were asked to describe what they did that day before school. The tailored exercise presented an item that the student had chosen in the first two structured exercises and asked the student to reflect on it and describe how it is now important to them later on in the school year. The item we selected to present to each student in the tailored exercise was identified as the selection about which the student wrote most extensively in the second exercise. Students in the comparison condition wrote an expository essay describing what they do after school.

Our methods were intended to replicate, as closely as possible, the trainings and implementations conducted by Cohen and colleagues (Cohen et al., 2006, 2009). We worked with principals and teachers to develop final versions of the prompts, with the intention of offering intervention materials that resembled those that they would assign their students as normal instructional routine. The instructions and script provided to teachers were further intended to frame the writing activities as typical classroom activities, and our trainings and materials urged teachers to present the exercises as such and avoid any mention of the activity as an external research project. MMSD has adopted social-emotional standards, and schools and teachers may adopt strategies to raise attention to concerns similar to those addressed by self-affirmation, yet our anecdotal observations and continued interactions with the participating schools and staff revealed no other systematic efforts underway to address stereotype threat.

At the beginning of the school year, administrators at each school (principal and/or learning coordinator) decided which classrooms the intervention would be fielded in; seven schools elected for English/language arts classrooms and four chose homeroom classes. We provided brief training sessions (no more than 1 hour) with the corresponding teachers at each school. The training provided teachers with the instructions necessary to administer the exercises as intended and provided them with a script and responses to students’ frequently asked questions.

Teachers distributed the writing exercises to students during the school day as part of normal classroom activities, and each administration took 15 to 20 minutes. Students were provided with personalized copies of the exercises to maintain fidelity of the random assignment. Each exercise included an identical cover sheet to disguise the condition to which students were assigned, and all versions of the exercises were designed to have a similar appearance. Students completed the assignments quietly and independently; teachers collected completed exercises and eventually returned them to the research staff without reviewing student responses. Exchanges of materials were conducted via school administrators or, in the few cases in which research staff interacted directly with classroom teachers, when students were not present. Classroom teachers were not informed of the condition to which their students were assigned or of the study’s hypotheses (including any mention of stereotype threat). ⁴

Data

All student demographic and outcome data were provided to us by MMSD. We evaluate the impact of self-affirmation on three outcomes, each consistent with a different strain of stereotype threat theory. First, consistent with the original formulation of stereotype threat (C. M. Steele & Aronson, 1995), the evaluative, high-stakes nature of standardized tests may prove to be especially threatening to students (C. M. Steele & Aronson, 1995), which may curtail the performance of racial and ethnic minorities subject to stereotype threat. If self-affirmation buffers students from the impact of stereotypes and allows them to demonstrate their true abilities, this effect should be apparent on the WKCE. ⁵

Second, we can evaluate the potential “recursive” nature of self-affirmation (Cohen et al., 2009) by investigating impacts on students’ cumulative GPAs in seventh grade. GPA represents how classroom teachers feel students are meeting their expectations, and can reward not only knowledge and skill but also classroom participation, homework completion, attendance, and behavior (Coleman, 1997; Sizer, 1984). Grades, therefore, reveal how well students negotiate “both the social and pedagogical aspects” of a school (Schiller, 1999). MMSD provided student transcript data from sixth and seventh grade; the transcripts include each course the student was enrolled in and the letter grade the student earned in each of the four quarters of the school year. As shown in Figure 1, the report cards are filed at the end of each term, in November, February, April, and June. Following MMSD protocol, we compute a student’s GPA by translating the letter grade into a score (e.g., A = 4, B = 3, etc.) and then creating a weighted average based on the number of credits earned per course and quarter. ⁶

MMSD seventh graders also take the Northwest Evaluation Association MAP, a computer adaptive test administered at multiple time periods during the school year. ⁷ In practice, teachers may use the information from these formative assessments to help tailor instruction to individual students or classrooms. Students take three MAP tests (Reading, Language Usage, and Mathematics) in the fall and spring, and one MAP test (Reading) in the winter. Reading and Mathematics are self-explanatory; Language Usage covers writing and grammar. MAP scores are produced using a Rasch measurement model, which yields an equal-interval scale. The 2011–2012 school year was the first year of MAP adoption in MMSD, and some students in our analytical sample were randomized to self-affirmation after their fall MAP tests. For these two reasons, we use the spring 2012 MAP Reading, Language Usage, and Mathematics scores as outcomes. These scores inform both the original and recursive theories of stereotype threat; by the time of the spring assessment, the students had completed multiple self-affirmation exercises over the course of the year, including a final one shortly before the spring MAP testing. In addition, we believe that to the extent that self-affirmation buffers potentially stereotyped students and allows them to exhibit their true abilities on the earlier fall and winter formative assessments, that teachers using the data for formative purposes could develop more realistic insights into potentially stereotyped students’ academic capacities and, as a result, hold higher expectations for the students’ future performances in the classroom and on later standardized tests.

The most important variable for this analysis is an identifier of students who are potentially subject to stereotype threat, which we construct using administrative race/ethnicity data. We do not expect that all students will be affected by self-affirmation equally. In fact, prior research in middle schools has found no impact of self-affirmation on White students and substantial impacts on GPA for Black (Cohen et al., 2006; Cohen et al., 2009) and Latino students (Sherman et al., 2013). Our sample includes students identified as White, Asian, American Indian, African American, Hispanic, Pacific Islander, and students who identified with multiple groups. We expect that White and Asian American students are not threatened by academic stereotypes (and could benefit from stereotype lift) and that American Indian, African American, Hispanic, and Pacific Islander students may be subject to stereotype threat. ⁸

We divide the sample into two groups: students potentially subject to stereotype threat (American Indian, African American, Hispanic, and Pacific Islander) and students not subject to stereotype threat (White and Asian). Multiracial students are assigned to one of the two groups based on the racial/ethnic categories they identify. For example, multiracial students identified as African American and White or Asian and Hispanic are assigned to the group that is potentially vulnerable to stereotype threat, but multiracial students identified as White and Asian are not. ⁹ We also include indicators of student gender, free/reduced-price lunch status, limited English proficiency status, and disability status as statistical controls to improve the precision of our analytical models.

Sample

The official count on the third Friday of September 2011 tallied 1,706 seventh-grade students, all of whom were invited to participate in the study. Student enrollment began during student registration at each of the 11 middle schools in August 2011 prior to the start of the academic year. We obtained parental consent and student assent for 1,048 students (61%) by the end of October, prior to the administration of the second of the writing exercises. ¹⁰ These students constitute the baseline for our attrition analyses. The small number of students who enrolled in the study after the second writing exercise are not included in the analysis.

We conduct analyses on two samples, a GPA sample and a test score sample. The GPA sample includes 1,012 students with a GPA recorded in the fourth and final term of sixth grade (preintervention) and at least the first three terms of GPA data in seventh grade. ¹¹ These students constitute 97% of the consented sample of 1,048 and 59% of the entire MMSD seventh-grade population of 1,706. The test score sample consists of 926 students with complete seventh-grade test scores (WKCE Reading, WKCE Mathematics, and spring MAP scores in Reading, Language Usage, and Mathematics) as well as sixth-grade outcomes for WKCE Reading and Mathematics. All students in both the GPA sample and test score sample also had complete demographic information. The number of consented students omitted due to missing data is relatively small and well-defined, namely, it is restricted to current MMSD students who moved into the district after November of their sixth-grade year and a small number of students who did not take both tests in seventh grade. Overall, missing data claimed 3% of the consented students for the GPA analysis and 12% of the students for the test score analysis. The rates of attrition due to missing data are equivalent by experimental condition, as well as by the experimental condition and potentially threatened status, in both the GPA and test score samples. ¹²

Descriptive statistics for the GPA sample are reported in Table 1 (an equivalent report for the test score sample is shown in Table A1 in the online appendix, available at http://epa.sagepub.com/supplemental). The first column of Table 1 shows the overall sample means and, for the prior measures of achievement, standard deviations. Fifty percent of the sample is female, 16% has been identified as having limited proficiency in English, 14% has been identified as requiring services for some form of disability, and 43% of the sample is eligible for free/reduced-price lunch. ¹³ Thirty-seven percent of the GPA sample is composed of students from racial/ethnic groups that could be subject to stereotype threat (Black, Hispanic, American Indian, Pacific Islander, or a Multiracial student identified as belonging to one of these categories). The standard deviations of the sixth-grade achievement test scores for the 926 students with available measures are 54.45 in Reading and 56.49 in Mathematics. Finally, the mean sixth-grade GPA in the sample is 3.24, with a standard deviation of 0.65.

OPEN IN VIEWER

Table 1

Demographic and Prior Achievement Variables for Overall Sample and by Experimental Condition

	Overall mean	Self-affirmation mean	Comparison mean	p value (T − C)
Female	0.50	0.49	0.52	.23
Limited English proficiency	0.16	0.14	0.17	.19
Special education services	0.14	0.14	0.14	.85
Free/reduced lunch eligible	0.43	0.43	0.43	.80
Potentially threatened racial/ethnic group	0.37	0.37	0.37	.94
Hispanic	0.17	0.16	0.18	.43
American Indian	0.04	0.03	0.05	.18
Asian	0.11	0.12	0.10	.34
African American	0.20	0.21	0.18	.13
Pacific Islander	0.00	0.00	0.00	.55
White (non-Hispanic)	0.57	0.57	0.57	.88
Multiracial	0.08	0.08	0.07	.64
Grade 6 reading	511.75	514.00	509.45	.20
	(54.45)	(54.28)	(54.58)
Grade 6 math	526.21	528.66	523.71	.18
	(56.49)	(56.04)	(56.89)
Grade 6 GPA	3.24	3.23	3.25	.54
	(0.65)	(0.65)	(0.66)

Note. Standard deviations in parentheses. n_GPA = 1,012 (513 [51%] assigned to self-affirmation writing); n_READING/MATH = 926 (468 [51%] assigned to self-affirmation writing). Race/ethnicity indicators are not mutually exclusive and sum to greater than 1.00. Binary variables tested with two-sample proportion test and scale variables tested with two-sample t test (H₀: T − C = 0). GPA = grade point average.

As Table 1 also shows, the experiment has good internal validity: The two experimental groups are balanced on demographics and prior achievement. The second and third columns show the means for the experimental conditions (self-affirmation vs. comparison) and the p values testing the differences between the experimental conditions. This information provides an indication of the baseline equivalence of the two groups and, thus, the internal validity of the experiment. None of the differences were statistically significant (see Tables 1 and A1). The self-affirmation group scored higher on the sixth-grade standardized tests, but the differences are not statistically significant and we use sixth-grade achievement as a control variable in our statistical models to address any potential imbalance in prior achievement and to increase statistical precision. The two groups were comparable with respect to sixth-grade GPA.

We also examine how similar the MWAP sample is to the population of MMSD seventh-grade students to evaluate the external validity of the sample. Demographic characteristics available for Grades 6 to 8 in MMSD suggest that the MWAP sample has a higher proportion of White students (55% vs. 44%) and a smaller proportion of students identified as Black (18% vs. 23%) and eligible for free/reduced-price lunch (43% vs. 52%). With respect to prior achievement, 53% and 49% of the MWAP sample scored at an “advanced” level on the Reading and Mathematics WKCE, respectively, as compared with the districtwide means of 43% and 40%. A comparable proportion of students scored at the “proficient” level, with 34% of the MWAP sample and 36% of the MMSD population achieving this performance level. A substantial proportion of the MWAP sample scored at the “minimal” or “basic” level in Reading and Mathematics, with 5% minimal in Reading, 12% minimal in Mathematics, 8% basic in Reading, and 9% basic in Mathematics. These rates were higher in MMSD overall, with 8% minimal in Reading, 19% minimal in Mathematics, 13% basic in Reading, and 11% basic in Mathematics. On the whole, the MWAP sample is somewhat more advantaged than the districtwide MMSD middle school population, but it still includes substantial numbers of poor students (as indicated by free or reduced-price lunch status), students with disabilities, English-language learners, and students at the low end of the achievement distribution.

As is shown in Table 2, membership in a potentially threatened racial/ethnic group is strongly associated with prior achievement. In other words, there is strong evidence of an unconditional racial/ethnic achievement gap in the MWAP sample. Potentially threatened students lag behind Asian and White students on all measures of preintervention, sixth-grade academic performance, including Reading and Mathematics test scores, and GPA. Specifically, potentially threatened groups score lower than Asian and White students by 50 scale score points in Reading, 52 points in Mathematics, and 0.66 GPA points. The nonthreatened students have an average GPA of 3.5 (B+/A–) and the potentially threatened students have an average GPA of 2.8 (B–). These unconditional gaps on measures of prior academic performance are equivalent in size to nearly one standard deviation.

OPEN IN VIEWER

Table 2

Prior Achievement Gaps by Stereotype Threat Vulnerability

	Potentially threatened	Not potentially threatened	Difference	Pooled standard deviation	t
Sixth-grade GPA	2.82	3.48	−0.66	(0.65)	−17.82
n	374	638
Sixth-grade Reading	479.29	529.55	−50.27	(54.45)	−14.97
n	328	598
Sixth-grade Math	492.41	544.75	−52.34	(56.49)	−15.04
n	328	598

Note. Total n_GPA = 1,012; n_READING/MATH = 926; pooled standard deviation from Table 1. GPA = grade point average.

Descriptive statistics for the seventh-grade outcome variables are shown in Table 3. As shown in Figure 1, the WKCE data were collected in November of the seventh-grade year and the GPA measures were collected at the end of each term. Cumulative seventh-grade GPA is shown along the top row, the WKCE scale scores in Reading and Mathematics are shown next, followed by the spring MAP scores in Reading, Language Usage, and Mathematics. A few features of these variables are worth noting. First, the mean values of the scale scores in Reading and Mathematics increased from sixth to seventh grade, but the mean GPA in the sample decreased from 3.24 in sixth grade (see Table 1) to 3.13 in seventh grade. ¹⁴ Data from each of the four terms comprising cumulative GPA show a slight decrease over the course of the year, from 3.20 and 3.22 in the first and second terms to 3.16 in both the third and fourth terms. This decline in mean GPA during middle school and during the academic year is both concerning and consistent with prior research (e.g., Anderman, 2003; Cook et al., 2012; Sherman et al., 2013).

OPEN IN VIEWER

Table 3

Student Achievement Outcomes

	M	SD	Minimum	Maximum
Seventh-grade GPA-cumulative	3.13	0.75	0.46	4
Seventh-grade Reading	527.22	57.29	310	780
Seventh-grade Math	540.53	48.21	330	671
Spring MAP Reading	221.68	15.90	145	261
Spring MAP Language Usage	220.22	14.19	154	256
Spring MAP Math	230.62	19.25	153	308

Note. n_GPA = 1,012; n_READING/MATH = 926. GPA = grade point average; MAP = Measures of Academic Progress.

Analysis

To evaluate whether self-affirmation affected student achievement differentially by student racial/ethnic background, we estimated the following multilevel model in which assignment to self-affirmation interacts with student membership in a racial/ethnic group that is potentially subject to stereotype threat:

Y i j = α + β ( A f f i r m e d i j ) + γ ( T h r e a t e n e d i j ) + δ ( A f f i r m e d i j × T h r e a t e n e d i j ) + ∑ φ X i j + u j + ε i j .

In this model, Y i j represents the GPA or standardized test score of students within schools, α represents the model intercept (the grand mean for the reference group), β is the coefficient representing the impact of self-affirmation for the reference (not potentially threatened) subgroup, γ represents the average difference for the outcome associated with membership in a racial/ethnic group potentially subject to stereotype threat, δ represents the interaction between self-affirmation and racial/ethnic group membership, ∑ φ X i j includes the set of additional covariates in the model (i.e., prior achievement, gender, free/reduced-price lunch (FRL) status, limited English proficiency, and special education status), u j is the school-specific error, and ε i j is the student-specific error term.

The coefficient δ tells us whether potentially threatened students responded to self-affirmation differently than majority students responded. The coefficient γ for membership in a potentially threatened group tells us whether there is an achievement gap between majority and potentially threatened students net of the other covariates and absent the intervention. To determine whether self-affirmation has an impact on potentially threatened students, we estimate marginal effects of self-affirmation. We transform the parameter estimates from the impact model to estimate the average effect of self-affirmation for the students who are potentially subject to stereotype threat and the students who are not subject to stereotype threat, at the average values of each racial/ethnic group’s covariates. ¹⁵ We estimate these margins at the group means because the preexisting differences on some of the covariates—namely, prior achievement—are large. Given that potentially threatened students lag nonthreatened students by approximately one standard deviation, there are few examples of potentially threatened students with prior achievement at the sample mean. The full results from our statistical models are presented in the online appendix (Tables A2–A4 in the online appendix, available at http://epa.sagepub.com/supplemental); we present the marginal effect estimates and effect sizes in Table 4 and the GPA results in Figure 2. We calculated effect sizes using the overall standard deviations of the outcomes reported in Table 3.

OPEN IN VIEWER

Table 4

Estimated Interactions Between Assignment to Self-Affirmation and Student Membership in a Racial/Ethnic Group That Is Potentially Subject to Stereotype Threat

	Estimates	d
Cumulative seventh-grade GPA	0.082*	0.11
	[0.008, 0.155]
Fall Reading test	−0.046	0.00
	[−7.415, 7.323]
Fall Math test	4.224*	0.09
	[0.939, 7.510]
Spring Reading test	−0.140	0.01
	[−2.933, 2.654]
Spring Math test	1.512	0.08
	[−0.603, 3.627]
Spring Language Usage test	0.417	0.04
	[−1.546, 2.379]

Note. n_GPA = 1,012; n_TESTS = 926; multilevel models of students in 11 schools. 95% confidence intervals based on delta-method standard errors in parentheses. GPA = grade point average.

*

p < .05.

OPEN IN VIEWER

Figure 2.

GPA estimates by term, threat status, and experimental group.

Interpreting the Impacts

In addition to quantifying effects as standardized mean differences, or effect sizes, as reported in the previous section, one can interpret the impacts as a proportion of the preexisting unconditional achievement gap (see Cohen et al., 2006; Cohen et al., 2009; Sherman et al., 2013) or relative to the residual or “unexplained” gap in achievement by race/ethnicity. The unconditional or “raw” achievement gap represents the accumulation of disadvantages that accrued to students over the course of their lives, including early childhood influences, nutrition, poverty, and so on. The “unexplained” gap adjusts for the fact that students from different racial/ethnic backgrounds have different circumstances and experiences that produce different achievement levels. Observable student characteristics and their performance on preintervention measures of academic performance account for most—but not all—of the achievement differences we observe in seventh grade. These differences in achievement observed among the comparison students constitute the “unexplained” gap and may be due to social and psychological experiences that unfold during seventh grade.

To illustrate, the estimated interaction term (0.082) and marginal effect (0.065) for the cumulative GPA outcome correspond to effect sizes of 0.11 and 0.09, respectively. As sixth graders, students who were potentially subject to stereotype threat had GPAs that were 0.66 lower than those of their peers (see Table 2); the marginal effect estimate of 0.065 is equivalent to 10% of this gap. Finally, the residual achievement gap in seventh-grade cumulative GPA that is not explained by demographic characteristics or a student’s sixth-grade WKCE Mathematics test score is 0.101 (see Table A2 in the online appendix, available at http://epa.sagepub.com/supplemental). This estimate demonstrates that stereotype-threat-vulnerable students who had been equal to their White and Asian peers perform worse a year later. By this standard, self-affirmation reduced the “unexplained” achievement gap by 64% (0.065 / 0.101 = 0.64). Relative to the overall raw gap in achievement, the impact of self-affirmation is modest, but it accounts for a substantial portion of the achievement gap that cannot be accounted for by demographic characteristics other than race/ethnicity and prior sixth-grade achievement.

Our estimate of the net impact of self-affirmation on cumulative GPA, which is equivalent to a 0.065 advantage on seventh-grade GPA, is smaller than the previously reported impacts, which range from 0.22 (Sherman et al., 2013, Study 1) to 0.38 (Sherman et al., 2013, Study 2). ¹⁶ The authors of the prior research found larger unadjusted gaps in sixth-grade GPA (e.g., Sherman et al., 2013, observed a GPA gap of 1.02), and reported that the impact of self-affirmation corresponds to 22% to 38% of the raw gap, which is substantially larger than the 10% we found. It is common among social-psychological interventions to find evidence of larger treatment impacts on GPA as time goes on (e.g., see Aronson, Fried, & Good, 2002; Wilson & Linville, 1982, 1985). Evidence from our study suggests a similar result but, again, our end-of-year impact estimate (d = 0.13) is smaller than similar estimates from prior research, such as an effect size of d = 0.57 reported by Sherman et al. (2013).

The suggestive marginal effect estimates on the Mathematics tests (2.948 and 1.578 in the fall and spring, respectively) represent standardized effect sizes of 0.06 in the fall and 0.10 in the spring. The fall math impact estimate represents 6% of the unconditional achievement gap on the sixth-grade test (2.948 / 52.34 = 0.06) and 56% of the “unexplained” achievement gap (2.948 / 5.311 = 0.56). Spring MAP scores are not available from the prior year, but the marginal effect estimate represents 54% of the “unexplained” achievement gap (1.578 / 2.927 = 0.54).