Taking Note of Our Biases: How Language Patterns Reveal Bias Underlying the Use of Office Discipline Referrals in Exclusionary Discipline

Abstract

The comments teachers write when sending students to the office have the potential to increase our understanding of how bias may contribute to longstanding racial disparities in school discipline. However, large-scale analysis of open text has traditionally had a prohibitive cost. Through natural language processing techniques, we examined over 3.5 million office discipline records from national samples of more than 4,000 schools for whether teachers’ linguistic patterns differed when describing incidents depending on the race/ethnicity and gender of the students. Results of such analyses consistently showed that teachers wrote longer descriptions and included more negative emotion when disciplining Black compared to White students, especially for Black girls. In conjunction with psychology of language theory, the patterns suggest that teachers may perceive and process student behavior differently depending on student identities. Implications of the findings and potential for research on naturally occurring language data in education are discussed.

Keywords

diversity equity language processes textual analysis

A substantial body of research shows that, in the United States (US), students of color, particularly Black students, are far more likely than White students to experience exclusionary school discipline (Skiba et al., 2011; Yeager et al., 2017). Beyond documenting the differences, results of recent research suggest that the disproportionate discipline likely stems, at least in part, from biases in the ways that teachers and administrators perceive, process, and respond to behaviors of students from different racial/ethnic backgrounds (Chin et al., 2020; Gregory et al., 2010; Ispa-Landa, 2018; Skiba et al., 2002). However, much of the large-scale field research on the topic focuses on quantitative analysis of disparities in discipline outcomes, along with fixed, predefined indicators of the structural, educational, psychological, and demographic characteristics associated with discipline incidents. Thus, there is a need to examine how educators describe discipline incidents involving students from different racial/ethnic backgrounds for potential disparities in how educators understand or find most important to convey about them.

The goal of the present study was to expand on the literature by exploring whether teachers who are disciplining students tend to use systematically different linguistic patterns when describing the underlying incident if it involves students of color as compared to White students. To do so, we use psychological theory and natural language processing to examine the narratives written by teachers or other school personnel in over 1.5 million discipline incidents from the 2015–2016 school year and replicating the analysis in a separate sample of 1.9 million discipline incidents that occurred in the 2018–2019 school year. By examining natural language patterns, we have a uniquely direct lens into potential differences in how teachers themselves construe discipline incidents and the information they think is most important to convey about the incidents, depending on the race/ethnicity of the student involved. Relatedly, it is rare to observe student discipline incidents on a systematic level, and therefore, using narrative descriptions of student discipline incidents provides an in-depth examination into these classroom-level interactions between students and teachers.

In parallel, we illustrate the potential for computational social science techniques to examine large bodies of information from educational records, thus helping to increase our understanding of the factors that may contribute to one of society’s greatest challenges, including educational disparities and racial justice (Ben Shahar, 2017). Improving our comprehension of how teachers understand and communicate about discipline incidents at scale will help us to develop more nuanced understandings of racial disproportionality in school discipline and more effective ways to address it.

Racial Disparities in Exclusionary Discipline

The purpose of school is to provide access to safe and supportive learning environments for all students to be successful (Girvan, 2019). When students are perceived to engage in serious behavioral infractions, educators commonly respond with exclusionary discipline, using, for example, office discipline referrals (ODRs) and ultimately suspensions or expulsions. Such exclusionary disciplinary actions vary in intensity and result in the loss of instructional time. Significant, widespread racial inequities in exclusionary school discipline pose a threat to the goal of providing safe and supportive educational environments (Bradshaw et al., 2010; Losen et al., 2021; Smolkowski et al., 2016). During the 2017–2018 school year, for example, Black students constituted 15% of public school enrollment but 30% of students suspended in school, 38% suspended out of school, 29% of students referred to law enforcement, 32% of students arrested for school-related incidents, and 43% of students transferred to alternative settings (U.S. Department of Education Office for Civil Rights, 2021). There are also pronounced gender-based differences, with boys receiving more exclusionary discipline than girls, although gender effects are often not as strong as race effects (Fabelo et al., 2011).

Evidence of Racial Bias in Discipline

Possible explanations for the causes of racial disparities in exclusionary discipline practices are often grouped into the effects of contextual or structural processes on student behavior and cultural differences in perceived norms and values (Girvan, 2019; Girvan et al., 2017; Huang, 2020; Skiba et al., 2011). Contextual or structural processes (e.g., poverty, schools, neighborhoods with limited resources) are associated with childhood stressors and poor student outcomes (e.g., lack of academic engagement, suspensions) (Sullivan et al., 2013). However, even after controlling for these variables, significant racial disparities in school discipline remain (Morgan et al., 2019; Sullivan et al., 2013).

Cultural differences in perceived social norms and values can result in discipline when there is a disconnect between behavioral expectations set in schools and homes for students of diverse cultural backgrounds (McIntosh et al., 2018; Skiba et al., 2011). For example, based on cultural expectations, school personnel may perceive Black students to be engaging in more aggressive behaviors when participating in recreational sports with peers (McIntosh et al., 2018). These cultural differences can cause school personnel to enforce harsher disciplinary actions for behaviors perceived as disrespectful and defiant (Girvan, 2019; Girvan et al., 2021; Okonofua et al., 2016; Skiba et al., 2011).

A growing body of research also suggests racial disparities in exclusionary discipline may relate to racial/ethnic biases (e.g., negative or positive attitudes or beliefs concerning members of a racial group; Allport, 1954) that impact how educators perceive and understand student behaviors (Girvan et al., 2017, 2021; Liu et al., 2022b; McIntosh et al., 2018; Riddle & Sinclair, 2019; Smolkowski et al., 2016). Because of strong norms against biases like racism, people tend to be motivated to avoid applying, or appearing to apply, racial/ethnic stereotypes or attitudes (Butz & Plant, 2009; Dovidio et al., 2017; Fazio & Olson, 2014). However, in practice, it is difficult for people to understand and respond to intergroup situations without relying to some extent on attitudes and stereotypes (Girvan, 2016; Okonofua & Eberhardt, 2015; Yzerbyt et al., 1994).

Limitations of Past Research

Previous research linking racial/ethnic biases to discipline decisions in large, naturalistic samples generally relies on quantitative analysis of fixed indicators of the conditions associated with the discipline incident (e.g., selections of behavior type or perceived motivation from broad lists) and outcomes (Girvan et al., 2021; Smolkowski et al., 2016). The results of this work have shown, for example, that racial disparities are more likely for subjectively defined behaviors (Girvan et al., 2017), in particular settings and at particular times of day (Smolkowski et al., 2016), at particular times of year (Darling-Hammond et al., 2023), and related to community levels of explicit (conscious) and implicit (automatic) racial biases (Girvan et al., 2021; Riddle & Sinclair, 2019). None of these studies directly examine teachers’ or administrators’ understandings or interpretations of the events that resulted in the discipline in their own terms. Rather, the results depend on inferences about how educators themselves understand the reason for using exclusionary discipline. Methodological advances offer the potential to address the shortcoming through examination of educators’ descriptions of incidental discipline directly.

In particular, aside from fixed response fields, ODRs typically include a single, open-ended text field (e.g., “Comments”) for school personnel to describe the incident that resulted in a referral to the office in their own words. These fields contain natural language data that can be used to explore more directly the ways in which educators’ understandings of what happened and what they thought was the most important information to convey may differ depending on the race/ethnicity of the students involved. A major benefit of using ODRs and evaluating the language within them is that they “provide increased consistency and efficiency for summaries and interpretation” of problematic events (McIntosh et al., 2010, p. 381). Understandably, given the sheer volume of the incidents, we are not aware of any previous research that attempted to examine the structure, content, and style of these narrative fields, particularly at scale. New, computational approaches to natural language processing, however, make this possible.

ODRs, Language Patterns, and Indicators of Bias

Recent advances in natural language processing have facilitated the examination of often elusive or difficult-to-study phenomena in the social sciences like bias. For example, a recent paper used language patterns to examine how physicians attended to critical care patients differently based on demographic characteristics. There, Markowitz (2022) analyzed 1.8 million critical care notes and observed physicians tended to focus more on the emotions of women than men. Physicians also used more impersonal pronouns (e.g., it, who, this) for women than men, suggesting more depersonalized care. Finally, physicians focused less on the emotions of Black and Asian patients versus White patients. These results provided some of the first evidence, at scale, that physician biases may alter perceptions, understandings, and accounting of patients using their own descriptions.

In research that follows the psychology of language tradition, content words (e.g., what people are communicating about) and style words (e.g., how people are communicating) serve as attention markers to reveal what people are thinking, feeling, and experiencing psychologically (Boyd & Schwartz, 2021; Pennebaker, 2011). We used this foundation to conduct an exploratory examination of how the language used when describing discipline incidents in ODRs differ based on the race/ethnicity of the student who was disciplined. In addition, mindful that racial/ethnic biases often interact with gender biases (Smolkowski et al., 2016; Wun, 2016), we also examined how linguistic differences may themselves differ across the intersections of student race/ethnicity and gender. Although the work is exploratory and we do not have formal hypotheses, based on prior work in other contexts (Markowitz, 2022), psychological theory regarding the operation of biases, and research on school discipline decision outcomes, we limited our examination to concepts and linguistic dimensions that may reveal how educators understand and describe discipline events involving students of different identities.

First, we investigated how much teachers wrote about their students to identify disparities in verbosity. The number of words in a disclosure provides important information about its structure and the amount of elaboration provided by the communicator (Larrimore et al., 2011; Markowitz, 2023). On the one hand, consistent with psychological theory (Yzerbyt et al., 1994), teachers may feel the need to overexplain, or elaborate in greater detail, why they are disciplining Black students compared to White students in order to satisfy the need to justify their decision and not appear discriminatory (see Hodson et al., 2002). On the other hand, teachers may use fewer words, as a reflection of effort, when describing why they disciplined Black versus White students if they felt apathetic or negatively toward the Black students (Bishop, 1989; Lafond-Brina et al., 2023). We therefore explored how the number of words used to describe a discipline referral differed based on the demographic characteristics of the student.

Second, we examined two content dimensions: negative emotion and verb use. Emotion is often a key indicator of racial/ethnic and gender bias in medical evaluations (Hagiwara et al., 2016; Markowitz, 2022). In other settings like police-civilian interactions, researchers have found that officers were less respectful and less polite to Black community members compared to White community members during traffic stops (Voigt et al., 2017). Consistent with such evidence, we expected educators to describe incidents involving Black students with more negative emotion, which could reflect perceptions of Black youth as more threatening (Goff et al., 2014), differences in affective responses such as interpreting Black students’ behavior more personally, or a form of intergroup anxiety (Stephan & Stephan, 1985). Similarly, we also examined the degree to which teachers use verbs (annoy, hope, or is mean) in discipline reporting. Verbs describe actions, and we therefore explored how teachers focused on the actions of certain students more than others depending on student race/ethnicity and gender (Skiba et al., 2002). Other evidence has observed disparities in action-oriented words by race/ethnicity (e.g., police officers were more likely to tell Black community members to put their hands on the wheel in traffic stops than White community members; Voigt et al., 2017), suggesting verbs may play an important role in revealing bias in educational ODRs as well.

Finally, with respect to linguistic style, we examined the use of impersonal pronouns. Impersonal pronouns (e.g., it, that) reflect one’s psychological or social distance from others (Markowitz & Slovic, 2020a). For example, Markowitz and Slovic (2020b) found that participants who favored longer jail sentences for immigrants labeled as illegal also depersonalized them by writing with more impersonal pronouns. Prior research has also found that impersonal pronouns are an important indicator of medical bias and depersonalization (Markowitz, 2022). We therefore examined how teachers’ use of impersonal pronouns may differ based on the race/ethnicity of students being disciplined.

Method

Database Description

Extant data in this study consisted of over 3 million ODRs issued to K–12 students by school personnel during the 2015–2016 and 2018–2019 school years. From the full set of ODRs within each wave, we created a refined dataset that excluded incident reports without text and those outside of the K–12 range.¹ We also included incident reports only for students whose gender was identified as boy or girl.² Incident reports were included only for students identified as Asian, Black, Latino/a/e, or White. Incident reports about students without a recorded race/ethnicity (n = 232,277; 12.01%), or that were infrequent (e.g., comprising less than 3% of the sample), were excluded (after all exclusion criteria, 17.44% of referrals were removed; 337,196/1,933,282). Data from the second wave (2018–2019) initially contained over 2.4 million incident reports from teachers, and the same inclusion and exclusion criteria were applied to this dataset as well (16.91% of referrals excluded; 404,966/2,394,539). Therefore, the final number of ODRs administered during the 2015–2016 school year included 1,596,086 regarding 407,801 students from 4,117 schools in 44 US states plus the District of Columbia. The number of ODRs in 2018–2019 was 1,989,573 regarding 462,056 students from 4,586 schools in 44 US states plus the District of Columbia. Table 1 includes a detailed summary of ODR-level data disaggregated by school year, grade level, gender, and race/ethnicity.

Table 1

Descriptive Details Across Genders and Racial Groups

Incident Report Descriptive Information
		2015–2016		2018–2019
		Count	Percentage	Count	Percentage
Gender	Boys	1,210,054	75.81%	1,510,579	75.92%
Gender	Girls	386,032	24.19%	478,994	24.08%
Race	Asian	22,364	1.40%	30,212	1.52%
	Boys	17,800	79.59%	23,973	79.35%
	Girls	4,564	20.41%	6,239	20.65%
	Black	338,191	21.19%	379,866	19.09%
	Boys	240,260	71.04%	270,194	71.13%
	Girls	97,931	28.96%	109,672	28.87%
	Latino/a/e	286,339	17.94%	408,843	20.55%
	Boys	219,841	76.78%	308,506	75.46%
	Girls	66,498	23.22%	100,337	24.54%
	White	949,192	59.47%	1,170,652	58.84%
	Boys	732,153	77.13%	907,906	77.56%
	Girls	217,039	22.87%	262,746	22.44%
Student and School Descriptive Information
		2015–2016		2018–2019
		Count	Percentage	Count	Percentage
Number of unique students		407,801	–	462,056	–
Number of unique teachers		130,320	–	142,157	–
Number of unique schools		4,117	–	4,586	–
	Kindergarten	137,331	8.60%	181,526	9.12%
	Grade 1	158,215	9.91%	196,962	9.90%
	Grade 2	161,131	10.10%	189,161	9.51%
	Grade 3	151,286	9.48%	182,585	9.18%
	Grade 4	146,710	9.19%	187,486	9.42%
	Grade 5	138,533	8.68%	185,759	9.34%
	Grade 6	146,700	9.19%	195,324	9.82%
	Grade 7	170,239	10.67%	217,272	10.92%
	Grade 8	161,956	10.15%	201,094	10.11%
	Grade 9	78,026	4.89%	83,523	4.20%
	Grade 10	63,213	3.96%	73,568	3.70%
	Grade 11	46,811	2.93%	54,046	2.72%
	Grade 12	35,916	2.25%	40,692	2.05%
	Post 12	19	0.00%	575	0.03%

Percentages in bold were calculated by dividing the raw count by the total number of incident reports in the 2015–2016 wave and 2018–2019 wave, respectively. In the top panel of this table, unbolded percentages are calculated within each ethnicity group.

Data were obtained from student ODR records entered into the School-Wide Information System (SWIS) web application (Educational & Community Supports, 2021). SWIS is used by schools across the United States to document information on student discipline incidents (e.g., type of behavior, time of day, individuals involved) and use the data for school improvement. Extant data from SWIS are housed and maintained by Educational and Community Supports (ECS), a research unit at the University of Oregon. When purchasing an SWIS license, users have the option of signing a data-sharing agreement that provides permission for researchers to use their de-identified data collected through SWIS to be used for research purposes.

Automated Text Analysis

Language patterns from written teacher incident reports were quantified using the automated text-analysis program, Linguistic Inquiry and Word Count (LIWC) (Pennebaker et al., 2022). LIWC uses a simple word-counting procedure to identify if words from an input text appear in one of its many internal dictionaries, which include words related to social (e.g., words related to friends or family), psychological (e.g., words related to emotion, cognition), and part-of-speech categories (e.g., articles, pronouns). Most LIWC dimensions are counted as a percentage of the total word count per ODR note. For example, the phrase “This student behaved poorly” contains four words and identifies several LIWC categories, including but not limited to impersonal pronouns (e.g., this; 25% of the word count) and social words (e.g., student; 25%).

Measures

We took an exploratory approach to identify linguistic differences in educator descriptions of discipline incidents across student race/ethnicity and genders. Several language dimensions were selected that—based on prior work—we argue may be theoretically relevant to potential differences in how teachers perceived, processed, and explained discipline incidents depending on the race/ethnicity or gender of the student: word count (e.g., the number of words in each incident report), negative emotion (e.g., the number of words tapping negative emotions such as aggressive, terror, or anger), verbs (e.g., the number of action words such as annoy, help, or wouldn’t), and impersonal pronouns (e.g., words such as it or this). To correct for skew in the data, all dependent variables were transformed using the formula log10(X + 1). Raw descriptive statistics for each variable are in the online supplement. Please see our full analytic plan placed in the online supplement due to space considerations.

Results

Estimated marginal means across race/ethnicity, gender, and their interaction are in Table 2 for 2015–2016. After the statistical findings, we offer excerpts to demonstrate how each linguistic dimension was observed in actual ODR notes across student race/ethnicity and gender. Nearly all results were identical across years unless noted (see supplement for 2018–2019 data).

Table 2

Estimated Marginal Means Across Factors and Models (2015–2016)

	Word Count		Negative Emotion		Verbs		Impersonal Pronouns
Race/Ethnicity	M	SE	M	SE	M	SE	M	SE
White	1.100	0.005	0.079	0.001	0.955	0.005	0.310	0.003
Asian	1.070	0.006	0.072	0.002	0.940	0.006	0.320	0.004
Latino/a/e	1.090	0.005	0.072	0.001	0.943	0.005	0.308	0.003
Black	1.110	0.005	0.084	0.001	0.960	0.005	0.308	0.003
	Word Count		Negative Emotion		Verbs		Impersonal Pronouns
Gender	M	SE	M	SE	M	SE	M	SE
Girls	1.080	0.005	0.076	0.001	0.947	0.005	0.309	0.003
Boys	1.100	0.005	0.078	0.001	0.952	0.005	0.313	0.003
	Word Count		Negative Emotion		Verbs		Impersonal Pronouns
Race/Ethnicity × Gender	M	SE	M	SE	M	SE	M	SE
Girls—White	1.090	0.005	0.077	0.001	0.953	0.005	0.306	0.003
Boys—White	1.100	0.005	0.081	0.001	0.957	0.005	0.312	0.003
Girls—Asian	1.060	0.007	0.064	0.004	0.935	0.009	0.315	0.007
Boys—Asian	1.080	0.006	0.076	0.002	0.943	0.006	0.323	0.004
Girls—Latino	1.080	0.005	0.071	0.001	0.936	0.005	0.305	0.003
Boys—Latino	1.090	0.005	0.073	0.001	0.947	0.005	0.310	0.003
Girls—Black	1.110	0.005	0.086	0.001	0.960	0.005	0.308	0.003
Boys—Black	1.110	0.005	0.084	0.001	0.962	0.005	0.309	0.003

All measures are percentages of the total word count except for raw word count. All variables are on the log10 scale and were transformed using the formula log10(X + 1) when true zero values existed in the data.

Differences in Educator Descriptions Based on Student Race/Ethnicity

Narrative Length

In the 2015–2016 school year, teachers wrote longer descriptions of incidents involving Black students than those involving White students (t = 11.31, p < .001). Teachers also wrote less about Asian (t = −11.41, p < .001) and Latino/a/e students (t = −14.37, p < .001) than White students.

Linguistic Content

Negative emotion

In the 2015–2016 school year, teachers’ descriptions included more negative emotion when writing about incidents involving Black students than White students (t = 7.15, p < .001). For example, a teacher for a Black boy wrote, “[Name] turned over the chess board during inside recess, showed rude and disrespectful behavior to his peers and called them losers,” which contains one negative emotion term (e.g., rude). The same teacher for a White boy wrote, “[Name] threatened to stab another student with a pencil. He was not able to be honest although there were several witnesses,” which is descriptive and does not contain negative emotion terms. Teachers also used less negative emotion for Asian (t = −3.64, p < .001) and Latino/a/e students (t = −10.41, p < .001) than White students.³

Verbs

In 2015–2016, teachers used more verbs when describing incidents involving Black students versus White students (t = 3.79, p < .001). A teacher for a Black boy wrote, “Was asked to sit safely w/ voice off; con’t to bother peers & talking: left room blurting math answers; displayed disrespectful attitude & body language to tchr; refused to go to CO,” which contained five verbs (e.g., was, asked, sit, talking, go). The same teacher for a White boy wrote, “Said ‘shut up’ to a peer after warnings & class rule to not say such things,” which contained two verbs (e.g., said, say). Teachers used fewer verbs when describing discipline referrals for Asian (t = −4.20, p < .001) and Latino/a/e students (t = −9.09, p < .001) than White students. That is, a teacher for a Latino/a/e boy wrote, “gum or candy after the teacher asked him to remove it,” which contains one verb (e.g., asked). The same teacher for a White boy wrote, “not having science materials to do the lab and writing,” with three verbs (e.g., having, do, writing).

Linguistic Style

Impersonal pronouns

In 2015–2016, there were no significant differences in the rate at which teachers used impersonal pronouns to describe discipline incidents of Black and White students (p = .163) nor Latino/a/e and White students (p = .101) in 2015. However, in 2018–2019, teachers used fewer impersonal pronouns when writing up Black and Latino/a/e students versus White students (see online supplement).

By comparison, in 2015–2016, teachers used more impersonal pronouns when writing up Asian students versus White students (t = 3.27, p = .001). A teacher for an Asian girl wrote, “[Student name] was escorted late to class by [Staff name]. When I was talking to another student she left. Result of conference with counselor was that she was removed from the class roster,” which contains two impersonal pronouns (e.g., another, that). The same teacher for a White girl wrote, “Student was given a hall pass and never returned to class,” which did not contain an impersonal pronoun.

Differences in Educator Notes Based on Student Gender

Narrative Length

In 2015–2016, teachers wrote more about students identified as boys than girls (t = 19.67, p < .001, R²c = 0.65).⁴

Linguistic Content

Negative emotion

In 2015–2016, teachers focused on more negative emotion when writing discipline reports about boys versus girls (t = 4.36, p < .001, R²c = 0.12). For example, a teacher for a boy wrote, “Physically went after another student. He hit him and was holding him trying to hurt him. I pulled him away he would not stop,” which contains one negative emotion term (e.g., hurt). The same teacher for a girl wrote, “[Name] left class, returned and continued to disrupt class. Did not follow directions and called others names,” which does not contain a negative emotion term.

Verbs

The 2015–2016 data revealed that teachers used more verbs when writing reports about students who are boys versus girls (t = 4.87, p < .001, R²c = 0.42). For example, a teacher for a boy wrote, “Had to be addressed 4 times in 20 minutes about listening to directions and would not follow them when asked to return to his seat,” which contains seven verbs (e.g., had, to, be, listening, would, follow, asked). Comparatively, the same teacher for a girl wrote, “Stole a snack from another student’s desk. Finally admitted to doing it. Responsibility Room,” which contains two verbs (e.g., stole, doing).

Linguistic Style

Impersonal pronouns

Finally, in 2015–2016, teachers used more impersonal pronouns when writing about boys versus girls (t = 5.45, p < .001, R²c = 0.22). For example, a teacher for a boy wrote, “[Student name] spanked another student while walking down the hall,” which contains one impersonal pronoun (e.g., another). The same teacher for a girl wrote, “[Student name] began the day agitated. She constantly yelled, argued, burped and refused directions,” with no impersonal pronouns.

Differences in Educator Notes Based on the Interaction of Race/Ethnicity and Gender

Using the 2015–2016 data, interaction effects for gender × ethnicity were statistically significant (Fs > 3.67, ps < .012) for all dependent variables except for use of impersonal pronouns (F = 2.48, p = .059). Figure 1 displays trends in these relationships, with specific pairwise comparisons in the online supplement. In general, Black boys and girls were described with more words and more negatively than their counterparts, with Asian girls being described with the fewest number of words and the least amount of negative affect overall. Such findings are consistent with other work that highlights the importance of intersectionality in evaluating bias-based disparities (Annamma et al., 2019; Welsh, 2022; Wun, 2016).

Figure 1.

Interaction effect descriptions for word count, negative emotion, and verbs in the 2015–2016 data.

Using the 2018–2019 data, interaction effects for gender × ethnicity were statistically significant (Fs > 4.31, ps < .005) for all dependent variables except for verb use (F = 1.69, p = .167). Figure 2 displays trends in these relationships, with specific pairwise comparisons in the online supplement. In general, consistent with the 2015–2016 data, Black boys and girls were described with more words and more negatively than other students. Asian girls were described with the fewest number of words and the least amount of negative affect.

Figure 2.

Interaction effect descriptions for word count, negative emotion, and impersonal pronouns in the 2018–2019 data.

Discussion

We present one of the first large-scale, natural language processing evaluations to reveal that students of color are attended to differently than White students. Black students were attended to in more negative terms than White students, teachers focused more on the actions of Black students compared to White students, and teachers used more impersonal pronouns when attending to Asian students compared to White students as well. Analyses also suggested teachers attending to boys wrote about them differently compared to those attending to girls. Teachers used more negative affect and impersonal pronouns when attending to boys versus girls. Disparities were greatest for Black boys and girls, as teachers referred to them more negatively than other groups and elaborated on their discipline referrals the most. In fact, Black girls were described with the most negative emotion, even more negatively than Black boys.

Findings from this study are consistent with and extend previous literature examining gender and racial disparities in exclusionary discipline in schools (Girvan et al., 2017, 2021; Losen et al. (2021); U.S. Department of Education Office for Civil Rights, 2021) and racial and gender disparities in medicine (Markowitz, 2022). For example, using 1,354,010 extant ODR records from students enrolled in 2,100 US schools, Girvan and colleagues (2021) found both explicit and implicit biases were related to prevalence of ODRs administered to Black students compared to White students. In another extant study using 483,686 student ODR records from 1,666 elementary schools in the United States, Smolkowski and colleagues (2016) examined the types of student behaviors, the time of the day, and specific settings in the school where racial disparities in exclusionary discipline practices are more likely to occur. Consistent with the current study, Smolkowski and colleagues (2016) found that Black students were more likely to receive ODRs for subjective behaviors (e.g., defiance, disrespect) and male students were much more likely to receive ODRs. While prior work suggests Black and Latino/a/e students tend to receive harsher disciplinary sentences than White students (Liu et al., 2022a; Welsh & Rodriguez, 2023), our work finds different disparities in how teachers talk about students of color. The fact that teachers writing about Latino/a/e students used fewer negative emotions in ODRs relative to White students may reflect a psychological difference in how teachers attend to students of different ethnicities. Lower rates of negative emotion (for Latino/a/e and Asian students compared to White students) reflect a disparity just as critical to understand and mitigate as do higher rates of negative emotion (for Black students compared to White students). Perhaps teachers report less negative emotion for Latino/a/e students compared to White students because of differences in infraction severity or perceived probability of a repeat offense (Shi & Zhu, 2022). These contentions deserve greater treatment in future work.

It is also important to position this research within other work that has investigated bias at the language level. At the onset, such patterns in Markowitz (2022) appear inconsistent with the evidence revealed in the current study. It is critical to remember that the populations under investigation (e.g., adults versus students), the institutions (e.g., medicine versus education), and the writers (e.g., physicians versus teachers) are substantially different across investigations. A one-size-fits-all model of bias in language cannot conceptualize how people of color are written about compared to White people. Bias is therefore context dependent (Barden et al., 2004; Warikoo et al., 2016), and even for the same linguistic dimension (e.g., negative emotion), signals of bias may be inconsistent across studies yet still reflect disparities.

Theoretical and Practical Implications

We provide one of the first investigations to identify differences in language use among a large selection of ODRs across two waves of data. This work underscores the utility and importance of evaluating educators’ own words in the process of understanding ODRs and disparities that exist in how educators construe and respond to the events that lead to them (e.g., what writers attend to psychologically). Future research might use this evidence to create applications that can track how teachers communicate about students of different demographic backgrounds and flag those who need additional training. More work would be required to identify the utility of such linguistic signals, but these findings provide an entry point for systems to be developed and to identify and hopefully attenuate bias at scale.

Limitations and Future Directions

As is generally the case with use of naturalistic data, the advantages of external validity in illuminating patterns in real-world behavior are inevitably offset somewhat by lack of experimental control and limitations in internal validity. We therefore recommend that future work experimentally examine the relationships observed in this paper. The schools under investigation also accounted for only approximately five percent of traditional US public schools. They may not be representative of the overall sample of US schools, and therefore we recommend caution when generalizing the study findings. For example, schools are likely to vary in their process and protocols for what types of student behaviors may occasion an ODR (e.g., objective vs. subjective behavior incidents, minor vs. major behavior incidents), how open text fields are utilized to describe student behaviors (e.g., details and depth of information provided), and who is responsible for administering and writing ODRs (e.g., teachers with or without school administrator authorization).

Future research could disaggregate ODR open-ended text fields further by including additional student, classroom, and teacher demographic data and conducting additional analyses to examine differences between ODRs for more subjectively vs. objectively defined behavior incidents (Girvan et al., 2017) or other vulnerable decision points (Ash & Maguire, 2023; Darling-Hammond et al., 2023; McIntosh et al., 2021). Similarly, research should examine and account for the effects of student special education status, race/ethnicity classroom composition and school enrollment, teacher race/ethnicity and experience, administrators’ discipline orientation (Skiba et al., 2014), and the extent to which the school is implementing particular discipline interventions (Ash & Maguire, 2023). Such findings could have meaningful implications for identifying where and with whom biases in student behavior incidents are most likely to occur. Given that the effect sizes in the paper are small, examination of contextual and structural moderators may inform the conditions where differences are acute or minimized.

Supplemental Material

sj-pdf-1-edr-10.3102_0013189X231189444 – Supplemental material for Taking Note of Our Biases: How Language Patterns Reveal Bias Underlying the Use of Office Discipline Referrals in Exclusionary Discipline

Supplemental material, sj-pdf-1-edr-10.3102_0013189X231189444 for Taking Note of Our Biases: How Language Patterns Reveal Bias Underlying the Use of Office Discipline Referrals in Exclusionary Discipline by David M. Markowitz, Angus Kittelman, Erik J. Girvan, María Reina Santiago-Rosario and Kent McIntosh in Educational Researcher

Footnotes

ORCID iDs

David M. Markowitz

Angus Kittelman

María Reina Santiago-Rosario

Kent McIntosh

Notes

Authors

DAVID M. MARKOWITZ, PhD, is an associate professor in the Department of Communication at Michigan State University, 404 Wilson Road East Lansing, MI 48824; dmm@msu.edu . His research focuses on evaluating social and psychological dynamics through natural language processing.

ANGUS KITTELMAN, PhD, is an assistant professor in the Department of Special Education at the University of Missouri, 118 Hill Hall, Columbia MO 65211; anguskittelman@missouri.edu . His research focuses on multitiered systems of support, implementation science, and school-based prevention of emotional and behavioral problems.

ERIK J. GIRVAN, J.D., Ph.D., is a professor of law at the University of Oregon, 1221 University of Oregon, Eugene, OR 97403-1221; girvan@uoregon.edu . His research focuses on developing more effective approaches for identifying and reducing racial disparities in legal settings and related contexts, such as school discipline.

MARÍA REINA SANTIAGO-ROSARIO, PhD, is a research associate at Educational and Community Supports (ECS), 1235 University of Oregon Eugene, OR 97403-1235; maresaro@uoregon.edu . Her research focuses on equity in school discipline practices, teacher expectations, culturally responsive classroom management, and middle- to high-school transition supports.

KENT MCINTOSH, PhD, is the Philip H. Knight Chair of Special Education at the University of Oregon and Director of Educational and Community Supports, a research unit in the College of Education, 1215 University of Oregon Eugene, OR 97403; kentm@uoregon.edu .

References

Allport

G. W.

(1954). The nature of prejudice. Addison-Wesley.

Annamma

S. A.

Anyon

Joseph

N. M.

Farrar

Greer

Downing

Simmons

(2019). Black girls and school discipline: The complexities of being overrepresented and understudied. Urban Education, 54(2), 211–242. https://doi.org/10.1177/0042085916646610

Ash

T. L.

Maguire

S. C.

(2023). A scoping review of diversity training for teachers: The potential for school psychology. School Psychology Review. https://doi.org/10.1080/2372966X.2023.2175585

Barden

Maddux

W. W.

Petty

R. E.

Brewer

M. B.

(2004). Contextual moderation of racial bias: The impact of social roles on controlled and automatically activated attitudes. Journal of Personality and Social Psychology, 87, 5–22. https://doi.org/10.1037/0022–3514.87.1.5

Bartoń

(2020). MuMIn. https://cran.r-project.org/web/packages/MuMIn/index.html

Ben Shahar

T. H

. (2017). Educational justice and big data. Theory and Research in Education, 15(3), 306–320. https://doi.org/10.1177/1477878517737155

Bishop

J. H.

(1989). Why the apathy in American high schools? Educational Researcher, 18(1), 6–42. https://doi.org/10.3102/0013189X018001006

Boyd

R. L.

Schwartz

H. A.

(2021). Natural language analysis and the psychology of verbal behavior: The past, present, and future states of the field. Journal of Language and Social Psychology, 40(1), 21–41. https://doi.org/10.1177/0261927X20967028

Bradshaw

C. P.

Mitchell

M. M.

O’Brennan

L. M.

Leaf

P. J.

(2010). Multilevel exploration of factors contributing to the overrepresentation of black students in office disciplinary referrals. Journal of Educational Psychology, 102, 508–520. https://doi.org/10.1037/a0018450

10.

Butz

D. A.

Plant

E. A.

(2009). Prejudice control and interracial relations: The role of motivation to respond without prejudice. Journal of Personality, 77(5), 1311–1342. https://doi.org/10.1111/j.1467–6494.2009.00583.x

11.

Chin

M. J.

Quinn

D. M.

Dhaliwal

T. K.

Lovison

V. S.

(2020). Bias in the air: A nationwide exploration of teachers’ implicit racial attitudes, aggregate bias, and student outcomes. Educational Researcher, 49(8), 566–578. https://doi.org/10.3102/0013189X20937240

12.

Darling-Hammond

Ruiz

Eberhardt

J. L.

Okonofua

J. A.

(2023). The dynamic nature of student discipline and discipline disparities. Proceedings of the National Academy of Sciences, 120(17), e2120417120. https://doi.org/10.1073/pnas.2120417120

13.

Dovidio

J. F.

Gaertner

S. L.

Pearson

A. R.

(2017). Aversive racism and contemporary bias. In Sibley

C. G.

Barlow

F. K.

(Eds.), The Cambridge handbook of the psychology of prejudice (pp. 267–294). Cambridge University Press. https://doi.org/10.1017/9781316161579.012

14.

Educational & Community Supports. (2021). School-wide Information System (SWIS) Suite. www.pbisapps.org

15.

Fabelo

Thompson

M. D.

Plotkin

Carmichael

Marchbanks

M. P. I.

Booth

E. A.

(2011). Breaking schools’ rules: A statewide study of how school discipline relates to students’ success and juvenile justice involvement. Council of State Governments Justice Center.

16.

Fazio

R. H.

Olson

M. A.

(2014). The MODE model: Attitude-behavior processes as a function of motivation and opportunity. In Sherman

J. W.

Gawronski

Trope

(Eds.), Dual-process theories of the social mind (pp. 155–171). The Guilford Press.

17.

Girvan

E. J.

(2016). Wise restraints?: Learning legal rules, not standards, reduces the effects of stereotypes in legal decision-making. Psychology, Public Policy, and Law, 22, 31–45. https://doi.org/10.1037/law0000068

18.

Girvan

E. J.

(2019). The law and social psychology of racial disparities in school discipline. In Bornstein

B. H.

Miller

M. K.

(Eds.), Advances in Psychology and Law: Volume 4 (pp. 235–276). Springer International Publishing. https://doi.org/10.1007/978–3–030–11042–0_8

19.

Girvan

E. J.

Gion

McIntosh

Smolkowski

(2017). The relative contribution of subjective office referrals to racial disproportionality in school discipline. School Psychology Quarterly, 32, 392–404. https://doi.org/10.1037/spq0000178

20.

Girvan

E. J.

McIntosh

Santiago-Rosario

M. R.

(2021). Associations between community-level racial biases, office discipline referrals, and out-of-school suspensions. School Psychology Review, 50(2–3), 288–302. https://doi.org/10.1080/2372966X.2020.1838232

21.

Goff

P. A.

Jackson

M. C.

Di Leone

B. A. L.

Culotta

C. M.

DiTomasso

N. A.

(2014). The essence of innocence: Consequences of dehumanizing Black children. Journal of Personality and Social Psychology, 106, 526–545. https://doi.org/10.1037/a0035663

22.

Gregory

Skiba

R. J.

Noguera

P. A.

(2010). The achievement gap and the discipline gap: Two sides of the same coin? Educational Researcher, 39(1), 59–68. https://doi.org/10.3102/0013189X09357621

23.

Hagiwara

Slatcher

R. B.

Eggly

Penner

L. A.

(2016). Physician racial bias and word use during racially discordant medical interactions. Health Communication, 32(4), 401–408. https://doi.org/10.1080/10410236.2016.1138389

24.

Hodson

Dovidio

J. F.

Gaertner

S. L.

(2002). Processes in racial discrimination: Differential weighting of conflicting information. Personality and Social Psychology Bulletin, 28(4), 460–471. https://doi.org/10.1177/0146167202287004

25.

Huang

F. L.

(2020). Prior problem behaviors do not account for the racial suspension gap. Educational Researcher, 49(7), 493–502. https://doi.org/10.3102/0013189X20932474

26.

Ispa-Landa

(2018). Persistently harsh punishments amid efforts to reform: Using tools from social psychology to counteract racial bias in school disciplinary decisions. Educational Researcher, 47(6), 384–390. https://doi.org/10.3102/0013189X18779578

27.

Lafond-Brina

Pham

B.-T.

Bonnefond

(2023). Initiative apathy trait underlies individual differences in the ability to anticipate and expend cognitive effort in cost-benefit decision-making tasks. Cerebral Cortex, 33(12), 7714–7726. https://doi.org/10.1093/cercor/bhad074

28.

Larrimore

Jiang

Larrimore

Markowitz

D. M.

Gorski

(2011). Peer to peer lending: The relationship between language features, trustworthiness, and persuasion success. Journal of Applied Communication Research, 39(1), 19–37. https://doi.org/10.1080/00909882.2010.536844

29.

Liu

Hayes

M. S.

Gershenson

(2022a). JUE Insight: From referrals to suspensions: New evidence on racial disparities in exclusionary discipline. Journal of Urban Economics. https://doi.org/10.1016/j.jue.2022.103453

30.

Liu

Penner

Gao

(2022b). Troublemakers? The role of frequent teacher referrers in expanding racial disciplinary disproportionalities. In EdWorkingPapers.com. Annenberg Institute at Brown University. https://www.edworkingpapers.com/ai22–616

31.

Losen

D. J.

Martinez

Shin

G. H. R.

(2021). Disabling inequity: The urgent need for race-conscious resource remedies. The Center for Civil Rights Remedies at the Civil Rights Project.

32.

Markowitz

D. M.

(2022). Gender and ethnicity bias in medicine: A text analysis of 1.8 million critical care records. PNAS Nexus, 1(4), pgac157. https://doi.org/10.1093/pnasnexus/pgac157

33.

Markowitz

D. M.

(2023). Instrumental goal activation increases online petition support across languages. Journal of Personality and Social Psychology, 124(6), 1133–1145. https://doi.org/10.1037/pspa0000333

34.

Markowitz

D. M.

Slovic

(2020a). Communicating imperatives requires psychological closeness but creates psychological distance. Journal of Language and Social Psychology, 39(5–6), 598–625. https://doi.org/10.1177/0261927X20902816

35.

Markowitz

D. M.

Slovic

(2020b). Social, psychological, and demographic characteristics of dehumanization toward immigrants. Proceedings of the National Academy of Sciences, 117(17), 9260–9269. https://doi.org/10.1073/pnas.1921790117

36.

McIntosh

Ellwood

McCall

Girvan

E. J.

(2018). Using discipline data to enhance equity in school discipline. Intervention in School and Clinic, 53(3), 146–152. https://doi.org/10.1177/1053451217702130

37.

McIntosh

Frank

J. L.

Spaulding

S. A.

(2010). Establishing research-based trajectories of office discipline referrals for individual students. School Psychology Review, 39(3), 380–394. https://doi.org/10.1080/02796015.2010.12087759

38.

McIntosh

Girvan

E. J.

McDaniel

S. C.

Santiago-Rosario

M. R.

St. Joseph

Fairbanks Falcon

Izzard

Bastable

(2021). Effects of an equity-focused PBIS approach to school improvement on exclusionary discipline and school climate. Preventing School Failure: Alternative Education for Children and Youth, 65(4), 354–361. https://doi.org/10.1080/1045988X.2021.1937027

39.

Morgan

P. L.

Farkas

Hillemeier

M. M.

Wang

Mandel

DeJarnett

Maczuga

(2019). Are students with disabilities suspended more frequently than otherwise similar students without disabilities? Journal of School Psychology, 72, 1–13. https://doi.org/10.1016/j.jsp.2018.11.001

40.

Okonofua

J. A.

Eberhardt

J. L.

(2015). Two strikes: Race and the disciplining of young students. Psychological Science, 26(5), 617–624. https://doi.org/10.1177/0956797615570365

41.

Okonofua

J. A.

Walton

G. M.

Eberhardt

J. L.

(2016). A vicious cycle: A social–psychological account of extreme racial disparities in school discipline. Perspectives on Psychological Science, 11(3), 381–398. https://doi.org/10.1177/1745691616635592

42.

Pennebaker

J. W.

(2011). The secret life of pronouns: What our words say about us. Bloomsbury Press.

43.

Pennebaker

J. W.

Boyd

R. L.

Booth

R. J.

Ashokkumar

Francis

M. E.

(2022). Linguistic inquiry and word count: LIWC-22. Pennebaker Conglomerates. https://www.liwc.app

44.

Riddle

Sinclair

(2019). Racial disparities in school-based disciplinary actions are associated with county-level rates of racial bias. Proceedings of the National Academy of Sciences, 116(17), 8255–8260. https://doi.org/10.1073/pnas.1808307116

45.

Shi

Zhu

(2022). Equal time for equal crime? Racial bias in school discipline. Economics of Education Review, 88, 102256. https://doi.org/10.1016/j.econedurev.2022.102256

46.

Skiba

R. J.

Chung

C.-G.

Trachok

Baker

T. L.

Sheya

Hughes

R. L.

(2014). Parsing disciplinary disproportionality: Contributions of infraction, student, and school characteristics to out-of-school suspension and expulsion. American Educational Research Journal, 51(4), 640–670. https://doi.org/10.3102/0002831214541670

47.

Skiba

R. J.

Horner

R. H.

Chung

C.-G.

Rausch

M. K.

May

S. L.

Tobin

(2011). Race is not neutral: A national investigation of African American and Latino disproportionality in school discipline. School Psychology Review, 40(1), 85–107. https://doi.org/10.1080/02796015.2011.12087730

48.

Skiba

R. J.

Michael

R. S.

Nardo

A. C.

Peterson

R. L.

(2002). The color of discipline: Sources of racial and gender disproportionality in school punishment. The Urban Review, 34(4), 317–342. https://doi.org/10.1023/A:1021320817372

49.

Smolkowski

Girvan

E. J.

Mcintosh

Nese

R. N. T.

Horner

R. H.

(2016). Vulnerable decision points for disproportionate office discipline referrals: Comparisons of discipline for African American and White elementary school students. Behavioral Disorders, 41(4), 178–195. https://doi.org/10.17988/bedi-41–04–178–195.1

50.

Stephan

W. G.

Stephan

C. W.

(1985). Intergroup anxiety. Journal of Social Issues, 41(3), 157–175. https://doi.org/10.1111/j.1540–4560.1985.tb01134.x

51.

Sullivan

A. L.

Klingbeil

D. A.

Van Norman

E. R.

(2013). Beyond behavior: Multilevel analysis of the influence of sociodemographics and school characteristics on students’ risk of suspension. School Psychology Review, 42(1), 99–114. https://doi.org/10.1080/02796015.2013.12087493

52.

U.S. Department of Education Office for Civil Rights. (2021). Civil rights data collection. https://ocrdata.ed.gov/resources/downloaddatafile

53.

Voigt

Camp

N. P.

Prabhakaran

Hamilton

W. L.

Hetey

R. C.

Griffiths

C. M.

Jurgens

Jurafsky

Eberhardt

J. L.

(2017). Language from police body camera footage shows racial disparities in officer respect. Proceedings of the National Academy of Sciences, 114(25), 6521–6526. https://doi.org/10.1073/PNAS.1702413114

54.

Warikoo

Sinclair

Fei

Jacoby-Senghor

(2016). Examining racial bias in education: A new approach. Educational Researcher, 45(9), 508–514. https://doi.org/10.3102/0013189X16683408

55.

Welsh

R. O.

(2022). Schooling levels and school discipline: Examining the variation in disciplinary infractions and consequences across elementary, middle, and high schools. Journal of Education for Students Placed at Risk (JESPAR), 27(3), 270–295. https://doi.org/10.1080/10824669.2022.2041998

56.

Welsh

R. O.

Rodriguez

L. A.

(2023). The plight of persistently disciplined students: Examining frequent flyers and the conversion of office discipline referrals into suspensions. Educational Evaluation and Policy Analysis. https://doi.org/10.3102/01623737231155155

57.

Wun

(2016). Against captivity: Black girls and school discipline policies in the afterlife of slavery. Educational Policy, 30(1), 171–196. https://doi.org/10.1177/0895904815615439

58.

Yeager

D. S.

Purdie-Vaughns

Hooper

S. Y.

Cohen

G. L.

(2017). Loss of institutional trust among racial and ethnic minority adolescents: A consequence of procedural injustice and a cause of life-span outcomes. Child Development, 88(2), 658–676. https://doi.org/10.1111/cdev.12697

59.

Yzerbyt

Schadron

Leyens

J.-P.

Rocher

(1994). Social judgeability: The impact of meta-informational cues on the use of stereotypes. Journal of Personality and Social Psychology, 66(1), 48.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.36 MB