A Matter of Measurement: How Different Ways of Measuring Racial Gaps in School Discipline Can Yield Drastically Different Conclusions About Racial Disparities in Discipline

Abstract

Racial disparities in school discipline remain central to policy discussions around school discipline. Recent research points to the importance of how discipline disparities are measured for the conclusions that are drawn about the extent of the problem or whether such disparities are improving. This brief uses data from Maryland to demonstrate how the choice of metric of the Black-White discipline gap can drastically change conclusions about whether the gap is closing or widening as well as conclusions about whether particular districts or schools have high or low racial disparities in discipline. This brief shows how interpretations of the Black-White discipline gap can be drastically different when using different metrics. Implications for educational researchers and practitioners studying school discipline are given.

Keywords

descriptive analysis discipline discipline gap disparities educational policy equity measurement suspensions

Purpose of This Research Brief

School discipline has garnered increasing attention among policymakers and the public. Much of this has been driven by evidence of racial disparities in discipline (Gregory et al., 2010). Largely absent from such discussions, however, has been a consideration of how the metrics used to estimate racial disparities may matter for the conclusions drawn. Recent work by Girvan et al. (2019) found that metrics of discipline disparities differ in their properties in ways that may influence conclusions drawn from their use.

Although such measurement issues have been described in a select set of practitioner guides (Bollmer et al., 2014; Petrosino et al., 2017) and policy reports (Curran & Finch, 2018; Porowski et al., 2014), in public testimony (Losen, 2018), and in other commentary (Scanlan, 2015), most policy documents and peer-reviewed research studies of school discipline rely on a single metric of racial disparities, with little consideration of how doing so may affect the conclusions (Gregory et al., 2010; Scanlan, 2015). This contrasts with the study of racial disparities in other areas of education (i.e., Bollmer et al., 2007). For example, in special education much consideration has been given to how racial disparities in placement are measured, and federal guidance codifies approaches to measuring special education disparities (see Bollmer et al., 2014; Office of Special Education Programs, 2017).

This research brief compares the use of two commonly used approaches of measuring racial disparities in discipline, particularly risk ratios (RR) and risk differences (RD), along with the use of a third metric, the raw differential representation (RDR), which prior work has argued has desirable properties (Girvan et al., 2019). Using data from Maryland as an example, this brief demonstrates that the choice of metric can significantly alter conclusions about whether discipline gaps are closing over time and whether particular districts have relatively high or low racial disparities in discipline. In doing so, it highlights the practical implications of the measurement issues discussed by Girvan and colleagues (2019).

Methodology

This analysis drew on data from the Maryland Department of Education and from the federal Department of Education’s Office for Civil Rights. Conclusions about the Black-White discipline gap were compared using RRs, RDs, and RDR. The methodological appendix contains further details about the data and analysis.

RRs (see Equation 1) are one of the most common ways to measure racial disparities in discipline. They are also the metric codified by the Department of Education for measuring racial disparities in special education placement (Office of Special Education Programs, 2017).

\begin{array}{l} R i s k r a t i o = (\begin{array}{l} # o f B l a c k s t u d e n t s d i s c i p l i n e d / \\ # o f B l a c k s t u d e n t s e n r o l l e d \end{array}) \\ / (\begin{array}{l} # o f W h i t e s t u d e n t s d i s c i p l i n e d / \\ # o f W h i t e s t u d e n t s e n r o l l e d \end{array}) \end{array}

(1)

RRs yield the commonly cited statistic that Black students are three to four times as likely as their White peers to experience a suspension from school.¹

In contrast, RDs (see Equation 2) represent the difference in the proportion of Black students suspended and the proportion of White students suspended.

\begin{array}{l} R i s k d i f f e r e n c e = (\begin{array}{l} # o f B l a c k s t u d e n t s d i s c i p l i n e d / \\ # o f B l a c k s t u d e n t s e n r o l l e d \end{array}) \\ - (\begin{array}{l} # o f W h i t e s t u d e n t s d i s c i p l i n e d / \\ # o f W h i t e s t u d e n t s e n r o l l e d \end{array}) \end{array}

(2)

For example, if a given district disciplined 15% of its Black students and 5% of its White students, it would have an RR of 3 (15% divided by 5%) and an RD of 0.10 or, equivalently, 10 percentage points (15% minus 5%).

A third measure, RDR, captures information about the number of students affected by the disparity (a feature not captured by RR or RD).

R D R = # o f B l a c k s t u d e n t s e n r o l l e d \times (r i s k d i f f e r e n c e)

(3)

RDR represents the number of Black students who experience a disciplinary event above and beyond the number expected if rates of discipline for Black and White students were equal. The RDR metric has the benefit of being more stable over time and communicating the magnitude of students affected by the disparity (Girvan et al., 2019). It is, however, highly correlated with enrollment size of the group of interest and therefore has limited utility for comparisons across districts with varying enrollment of the target group.² Table 1 provides a concise summary of the relative strengths and weaknesses of each of these three metrics.

Table 1

Comparison of Discipline Disparity Metrics

Metric	Strengths	Weaknesses
Risk ratio	• Most commonly used • Intuitive to interpret • Comparable across contexts with different enrollment sizes • Communicates the relative magnitude of the discipline rates between subgroups	• Sensitive to small cell sizes or small changes • Relatively unreliable over time • Provides little information about the number of students affected
Risk difference	• Commonly used • Less sensitive to small cell sizes or small changes • Comparable across contexts with different enrollment sizes	• Provides little information about the number of students affected • Relatively unreliable over time • Does not communicate the relative magnitude of the discipline rates between subgroups
Raw differential representation	• Provides information about the number of students affected • Intuitive to interpret • More reliable over time • Lower correlation with risk ratio and risk difference, thereby providing more unique information about the disparity	• Harder to compare across contexts with different enrollment sizes because not scaled to enrollment of target group • Does not communicate the relative magnitude of the discipline rates between subgroups • Less frequently used

Note. Strengths and weaknesses summarized here are drawn in part from findings in Girvan et al. (2019), where readers can find greater detail about the technical properties of each metric.

Key Findings

Is the Black-White Discipline Gap in Maryland Closing or Widening? Depends on the Metric

Conclusions about changes in discipline gaps over time can be sensitive to the metric used. Panel 1 of Figure 1 shows the percentage of students who experienced a suspension or expulsion in Maryland from 2010 to 2015. Overall rates of suspension or expulsion for both Black and White students decreased during this time period; however, whether the Black-White discipline gap decreased or increased depends on how it is measured. As shown in Panel 2 of Figure 1, the Black-White discipline gap as measured by an RR increased during this period, whereas RD decreased. This is consistent with findings that RRs tend to be negatively correlated with underlying rates of discipline whereas RDs are positively correlated with underlying rates of discipline (Girvan et al., 2019; Scanlan, 2015). As shown in Panel 3, RDR also decreased from approximately 20,000 students to 15,000, more closely aligning with the trend of RD than RR. To the extent that stakeholders have evidence that overall rates of discipline are decreasing, RD or RDR may be a more desirable measure of disproportionality as these measures are not as susceptible to showing increasing disparities as a function of decreases in underlying discipline rates.

Figure 1.

Black and White suspension or expulsion rates over time with corresponding Black-White discipline gap measured as risk ratios, risk differences, and raw differential representation.

The Curious Case of Montgomery County Public Schools

The choice of metric also can influence conclusions about whether a particular district has a relatively large or small racial discipline gap (Curran & Finch, 2018). Table 2 shows the Black-White discipline gap measured at the school district level for each district in Maryland for the 2015–2016 school year. The left-hand panel shows district-level enrollment and number of suspensions by race for districts ordered alphabetically. The right-hand panel shows rates of discipline by race along with RR, RD, and RDR, each ordered by magnitude of the rate or metric of disparity.

Table 2

Maryland School District Enrollment and Discipline Data Arranged by Black-White Discipline Gap by Metric Type

	Ordered Alphabetically by District				Ordered by Discipline Measure From Largest to Smallest
	White Enrollment	Black Enrollment	# White OSS	# Black OSS	White OSS Rate (%)		Black OSS Rate (%)		Risk Ratio		Risk Difference (p.p.)		Raw Differential Representation
Allegany (ALGY)	7,846	266	420	54	SMST	6.4	DCHR	20.9	MNTG	5.2	ALGY	14.9	PG	3,390.7
Anne Arundel (AA)	45,074	16,455	1,379	1,748	DCHR	6.0	ALGY	20.3	STM	5.1	DCHR	14.9	B. City	2,872.9
Baltimore City (B. City)	6,665	68,466	217	5,102	CCL	5.6	SMST	20.1	TLBT	4.7	SMST	13.7	B. Cnty	1,715.8
Baltimore County (B. Cnty)	45,579	42,829	892	2,554	ALGY	5.4	WICM	18.2	HWRD	4.3	WICM	13.4	AA	1,244.6
Calvert (CLVT)	11,511	2,149	248	165	WICM	4.8	CCL	15.4	HRFD	4.0	HRFD	10.1	MNTG	918.8
Caroline (CRLN)	3,612	828	93	75	KNT	3.7	HRFD	13.5	WASH	3.8	CCL	9.8	CLS	824.2
Carroll (CRL)	21,838	980	591	88	HRFD	3.4	AA	10.6	FRDK	3.8	STM	8.3	WICM	709.6
Cecil (CCL)	12,303	1,421	694	219	CLS	3.3	KNT	10.4	WICM	3.8	TLBT	7.8	HRFD	696.2
Charles (CLS)	7,419	14,288	245	1,296	B. City	3.3	STM	10.3	ALGY	3.8	AA	7.6	HWRD	618.2
Dorchester (DCHR)	2,144	1,868	129	390	AA	3.1	TLBT	9.9	QA	3.7	KNT	6.7	FRDK	290.9
Frederick (FRDK)	25,821	4,629	575	394	PG	2.7	CLS	9.1	CLVT	3.6	CRLN	6.5	DCHR	277.6
Harford (HRFD)	24,516	6,897	838	932	CRL	2.7	CRLN	9.1	CRLN	3.5	FRDK	6.3	STM	269.9
Howard (HWRD)	22,389	12,324	343	807	WSTR	2.6	CRL	9.0	AA	3.5	CRL	6.3	SMST	175.0
Kent (KNT)	1,270	472	47	49	CRLN	2.6	WSTR	8.7	DCHR	3.5	WSTR	6.2	CCL	138.8
Montgomery (MNTG)	47,237	33,569	307	1,137	FRDK	2.2	FRDK	8.5	WSTR	3.4	CLS	5.8	WASH	134.6
Prince George’s (PG)	5,801	86,953	157	5,744	CLVT	2.2	CLVT	7.7	CRL	3.3	CLVT	5.5	CLVT	118.7
Queen Anne’s (QA)	6,221	548	102	33	TLBT	2.1	B. City	7.5	SMST	3.1	HWRD	5.0	WSTR	80.9
Somerset (SMST)	1,201	1,279	77	257	STM	2.0	PG	6.6	B. Cnty	3.0	WASH	4.7	CRL	61.5
St. Mary’s (STM)	11,767	3,247	236	335	B. Cnty	2.0	HWRD	6.5	KNT	2.8	QA	4.4	TLBT	58.3
Talbot (TLBT)	2,882	744	61	74	WASH	1.7	WASH	6.3	CLS	2.7	B. City	4.2	CRLN	53.7
Washington (WASH)	15,574	2,870	257	182	QA	1.6	QA	6.0	CCL	2.7	B. Cnty	4.0	ALGY	39.8
Wicomico (WICM)	6,673	5,293	317	961	HWRD	1.5	B. Cnty	6.0	PG	2.4	PG	3.9	KNT	31.5
Worcester (WSTR)	4,402	1,315	114	115	MNTG	0.6	MNTG	3.4	B. City	2.3	MNTG	2.7	QA	24.0

Note. Disparities in OSS rate calculated from 2015–2016 Civil Rights Data Collection data. OSS rates represent unduplicated counts of the number of students of a subgroup experiencing OSS relative to the number of students of each subgroup in the district as reported by the district to the federal government. Districts with fewer than 20 White or Black students were omitted due to low sample size. OSS = out-of-school suspension; p.p. = percentage points.

Ironically, Montgomery County Public Schools (in bold in Table 2) appears at the top of the RR list and the bottom of the RD list. In other words, Montgomery has either the largest Black-White discipline gap in the state (as measured by RR) or the smallest in the state (as measured by RD). Interesting to note, in terms of raw rates of discipline, Montgomery has the lowest White and Black suspension rates of any district in the state. This may explain, in part, the stark differences seen in Montgomery’s ranking across RR and RD as RRs tend to be higher and RDs lower when underlying rates of discipline are lower. Consequently, examining RRs and RDs together with the underlying rates of discipline is important for contextualization.

Although Montgomery ranks higher on RDR (fifth among all districts), this is largely driven by the size of Black student enrollment in Montgomery (fourth in the state). Although this points to a greater absolute number of Black students affected by the racial disparity in Montgomery than in many other districts, the RDR metric is not as useful for determining whether districts with differing enrollment of Black students are excelling or struggling at promoting equitable discipline by race. Coupling RDR with metrics that are scaled to the target group’s enrollment is useful when attempting to gauge relative performance.

Underlying Rates Matter Too

Finally, it is worth nothing that RRs, RDs, and RDR can miss important differences in schools’ use of discipline. For example, in 2015–2016, Benjamin Stoddert Middle School (Charles County, Maryland) had an RR of 1.08, an RD of 0.97, and an RDR of less than five, indicating virtually no Black-White discipline gap. Similarly, Violetville Elementary/Middle (Baltimore City, Maryland) had an RR of 1.13, an RD of 0.82, and an RDR of about two, all quite similar to Stoddert’s. Despite similar values on each metric, Stoddert suspended nearly twice as many students of each subgroup (12.7% of White students and 13.6% of Black students) as Violetville (6.3% of White students and 7.1% of Black students). In short, attention to the overall levels of suspensions for subgroups is also important.

Implications

This brief highlights the drastic differences in our understanding of discipline disparities that can occur as a result of the choice of metric. The choice of metric can alter whether we conclude that discipline gaps are increasing or decreasing over time and can drastically alter whom we identify as high or low performers in discipline equity. In doing so, this brief highlights the potentially large practical implications of measurement issues raised elsewhere (Girvan et al., 2019; Gregory et al., 2010; Losen, 2018; Petrosino et al., 2017; Scanlan, 2015).

It is important to note that no single metric is inherently correct or wrong. They merely represent different ways of measuring discipline disparities, and each has its own strengths and weaknesses (see Table 1). To the extent that policymakers and practitioners value reducing suspension rates for all subgroups in addition to improving racial equity in discipline, using alternative metrics in addition to RR may be desirable as RR is prone to increase as underlying rates of discipline decrease. The use of RDs would capture decreases in the percentage point difference of students disciplined, and RDR would capture changes in the magnitude of minority students disciplined above and beyond what would be expected if rates were equal across subgroups. Coupling these with presentation of the rates of discipline for each subgroup helps contextualize a district or school’s performance. In sum, a sole reliance on a single metric may yield narrow conclusions. A consideration of multiple metrics then can help guide efforts to identify and combat differential identification of misconduct, differential administrative responses to misconduct, and differential effectiveness of reforms to discipline. Researchers, policymakers, and educators should be aware of how measurement choices can change conclusions drawn from the data.

Recommendations

Policies pertaining to and research examining discipline disparities should employ multiple metrics of disparities.

Raw levels of discipline for subgroups should also be reported and examined.

Researchers should further probe the sensitivity of discipline disparities to the metric used to better understand under what circumstances differing conclusions may be drawn based on the metric used (see Girvan et al., 2019, for a detailed examination of the properties of various metrics).

Addressing racial disparities in discipline remains a crucial priority for policymakers and educators. Doing so with appropriate attention to the metrics used can help ensure such efforts are as productive as possible at improving outcomes and equity for all students.

Footnotes

ORCID iD

F. Chris Curran

Notes

Author

F. CHRIS CURRAN, PhD, is an associate professor of educational leadership and policy and a codirector of the Education Policy Research Center at the University of Florida’s College of Education, Norman Hall, PO Box 117049, Gainesville, FL 32611; chriscurran@coe.ufl.edu . His research focuses on improving equity in educational outcomes through the study of school discipline and safety as well as early childhood education.

Methodological Appendix

This research brief drew on data from two sources. The first was the Maryland State Department of Education’s ([MSDE’s] 2010a, 2010b, 2011a, 2011b, 2012a, 2012b, 2013a, 2013b, 2014a, ) annual reports on school enrollment and discipline use. The second was data collected by the federal Department of Education’s Office for Civil Rights as part of the Civil Rights Data Collection (CRDC).

This methodological appendix organizes the discussion of the methodology by subsection of the research brief. First, I present the analytic approach and data sources that support the findings of the section “Is the Black-White Discipline Gap in Maryland Closing or Widening? Depends on the Metric.” Then, I present the analytic approach and data sources that support the findings of the sections “The Curious Case of Montgomery County Public Schools” and “Underlying Rates Matter Too.”

References

Bollmer

Bethel

Garrison-Mogren

Brauen

(2007). Using the risk ratio to assess racial/ethnic disproportionality in special education at the school-district level. Journal of Special Education, 41(3), 186–198.

Bollmer

Bethel

Munk

Bitterman

(2014). Methods for assessing racial/ethnic disproportionality in special education: A technical assistance guide (revised). Westat. https://ideadata.org/sites/default/files/media/documents/2017-09/idc_ta_guide_for_508-010716.pdf

Curran

F. C.

Finch

(2018). Maryland schools’ codes of conduct: Comparing discipline policy across districts. http://www.marylandpublicschools.org/stateboard/Documents/01242019/RR/MarylandSchoolsCodesConductComparingDisciplinePolicyAcrossDistricts.pdf

Girvan

E. J.

McIntosh

Smolkowski

(2019). Tail, tusk, and trunk: What different metrics reveal about racial disproportionality in school discipline. Educational Psychologist, 54(1), 40–59.

Gregory

Skiba

R. J.

Noguera

P. A.

(2010). The achievement gap and the discipline gap: Two sides of the same coin? Educational Researcher, 39, 59–68.

Losen

(2018). Response to December briefing before the U.S. Commission on Civil Rights: The school-to-prison pipeline: The intersections of students of color with disabilities. Center for Civil Rights Remedies, UCLA. https://www.academia.edu/35905626/Response_0116_415pm_final_Testimony

Office of Special Education Programs. (2017). Significant disproportionality (equity in IDEA). U.S. Department of Education. https://sites.ed.gov/idea/files/significant-disproportionality-qa-03-08-17.pdf

Petrosino

Fronius

Goold

C. C.

Losen

D. J.

Turner

H. M.

(2017). Analyzing student-level disciplinary data: A guide for districts (REL 2017-263). Regional Educational Laboratory Northeast & Islands. https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/REL_2017263.pdf

Porowski

O’Conner

Passa

(2014). Disproportionality in school discipline: An assessment of trends in Maryland, 2009–12 (REL 2014-017). Regional Educational Laboratory Mid-Atlantic.

10.

Scanlan

J. P.

(2015). Relative versus absolute differences in school discipline. http://jpscanlan.com/disciplinedisparities/relativeabsolutediff.html

11.

Henry

(2015). Out-of-school suspension in Maryland public schools, 2008–2014. Maryland Equity Project, University of Maryland, College Park. https://education.umd.edu/file/3608/download?token=KbYzSpSM

12.

Maryland State Department of Education. (2010a). Maryland public school enrollment by race/ethnicity and gender and number of schools. http://www.marylandpublicschools.org/about/Documents/DCAA/SSP/20102011Student/2011EnrollbyRace.pdf

13.

Maryland State Department of Education. (2010b). Suspensions, expulsions, and health related exclusions. http://www.marylandpublicschools.org/about/Documents/DCAA/SSP/20102011Student/2011SuspExpulHRExc.pdf

14.

Maryland State Department of Education. (2011a). Maryland public school enrollment by race/ethnicity and gender and number of schools. http://www.marylandpublicschools.org/about/Documents/DCAA/SSP/20112012Student/2012EnrollbyRace.pdf

15.

Maryland State Department of Education. (2011b). Suspensions, expulsions, and health related exclusions. http://www.marylandpublicschools.org/about/Documents/DCAA/SSP/20112012Student/2012SuspExpulHRExc.pdf

16.

Maryland State Department of Education. (2012a). Maryland public school enrollment by race/ethnicity and gender and number of schools. http://www.marylandpublicschools.org/about/Documents/DCAA/SSP/20122013Student/2013EnrollbyRace.pdf

17.

Maryland State Department of Education. (2012b). Suspensions, expulsions, and health related exclusions. http://www.marylandpublicschools.org/about/Documents/DCAA/SSP/20122013Student/2013SuspExpulHRExc.pdf

18.

Maryland State Department of Education. (2013a). Maryland public school enrollment by race/ethnicity and gender and number of schools. http://www.marylandpublicschools.org/about/Documents/DCAA/SSP/20132014Student/2013_2014_Enrollment.pdf

19.

Maryland State Department of Education. (2013b). Suspensions, expulsions, and health related exclusions. http://www.marylandpublicschools.org/about/Documents/DCAA/SSP/20132014Student/2014_Susp_Expul_HRExc.pdf

20.

Maryland State Department of Education. (2014a). Maryland public school enrollment by race/ethnicity and gender and number of schools. http://www.marylandpublicschools.org/about/Documents/DCAA/SSP/20142015Student/2015_Enroll_Release.pdf

21.

Maryland State Department of Education. (2014b). Suspensions, expulsions, and health related exclusions. http://www.marylandpublicschools.org/about/Documents/DCAA/SSP/20142015Student/2014-2015_Suspensions.pdf

22.

Porowski

O’Conner

Passa

(2014). Disproportionality in school discipline: An assessment of trends in Maryland, 2009–12 (REL 2014-017). Regional Educational Laboratory Mid-Atlantic.

23.

U.S. Department of Education. (2018). Civil rights data collection. https://www2.ed.gov/about/offices/list/ocr/data.html