School District and Community Factors Associated with Test Score Declines During the COVID-19 Pandemic

Abstract

Using data from approximately 7,800 school districts, we show that test score declines during the pandemic were large and highly variable. We find that test score declines were larger in lower-income and minority districts, but that within districts, White students and non–economically disadvantaged students lost about the same amount of ground as Black, Hispanic, and economically disadvantaged students. We find that the test score declines were larger in districts where more of the 2020–2021 school year was spent in remote or hybrid instructional modalities, where students had less access to broadband access at home, where the pandemic led to larger disruptions to local social and economic activity, and in communities where trust in government institutions was low.

Keywords

COVID-19 test score declines academic recovery achievement community context

Get full access to this article

View all access options for this article.

References

Abadie

Athey

Imbens

G. W.

Wooldridge

J. M.

(2022). When should you adjust standard errors for clustering? The Quarterly Journal of Economics, 138(1), 1–35. https://doi.org/10.1093/qje/qjac038

American Enterprise Institute. (2022). Return to learn tracker data. Version 45f. Author.

Amplify Education. (2021). COVID-19 means more students not learning to read [Research brief]. Author

Ananat

E. O.

Gassman-Pines

Francis

Gibson-Davis

(2011). Children left behind: The effects of statewide job loss on student achievement (Working Paper No. 17104). National Bureau of Economic Research. https://doi.org/10.3386/w17104

Betthäuser

B. A.

Bach-Mortensen

A. M.

Engzell

(2023). A systematic review and meta-analysis of the evidence on learning during the COVID-19 pandemic. Nature Human Behaviour, 7(3), 375–385. https://doi.org/10.1038/s41562-022-01506-4

Bielinski

Brown

Wagner

(2021). No longer a prediction: What new data tell us about the effects of 2020 learning disruptions [White paper]. Illuminate Education. https://www.illuminateed.com/wp-content/uploads/2021/06/LearningLoss_2021_Update_FB_031221.pdf

Bronfenbrenner

(1979). The ecology of human development. Harvard University Press.

Camp

A. M.

Zamarro

(2022). Determinants of ethnic differences in school modality choices during the COVID-19 crisis. Educational Researcher, 51(1), 6–16. https://doi.org/10.3102/0013189X211057562

COVID-19 School Data Hub. (2023a). All district learning model data (Version 2/28/23). https://www.covidschooldatahub.com/data-resources

10.

COVID-19 School Data Hub. (2023b). All school learning model data (Version 3/8/23). https://www.covidschooldatahub.com/data-resources.

11.

Darling

(2007). Ecological systems theory: The person in the center of the circles. Research in Human Development, 4(3–4), 203–217. https://doi.org/10.1080/15427600701663023

12.

Dee

T. S.

(2023). Where the kids went: Nonpublic schooling and demographic change during the pandemic exodus from public schools. Teachers College Record, 125(6), 119–129. https://doi.org/10.1177/01614681231190201

13.

Delphi Group at Carnegie Mellon University & University of Maryland Social Data Science Center, in partnership with Meta CTIS. (2022). Methodology report for the COVID-19 Trends and Impact Survey (Version 1). https://dataforgood.facebook.com/dfg/resources/CTIS-methodology-report

14.

Dewey

Fahle

Kane

T. J.

Reardon

S. F.

Staiger

D. O.

(2024). Federal pandemic relief and academic recovery (NBER Working Paper 32897). https://www.nber.org/papers/w32897

15.

Dewey

D. C.

Fahle

E. M.

Kane

T. J.

Reardon

S. F.

Staiger

D. O.

(2025) Pivoting from pandemic recovery to long-term reform: A district-level analysis. https://educationrecoveryscorecard.org/wp-content/uploads/2025/02/Pivoting-from-Pandemic-Recovery-to-Long-Term-Reform-A-District-Level-Analysis.pdf

16.

Domina

Hashim

Kearney

Pham

Smith

(2022). COVID-19 and the system resilience of public education: A view from North Carolina [Brief]. Urban Institute. https://www.urban.org/research/publication/covid-19-and-system-resilience-public-education-view-north-carolina

17.

Domingue

B. W.

Dell

Lang

Silverman

Yeatman

Hough

(2022). The effect of COVID on oral reading fluency during the 2020–2021 academic year. AERA Open, 8. https://doi.org/10.1177/23328584221120254

18.

Ettman

C. K.

Fan

A. Y.

Subramanian

Adam

G. P.

Badillo Goicoechea

Abdalla

S. M.

Stuart

E. A.

Galea

(2023). Prevalence of depressive symptoms in U.S. adults during the COVID-19 pandemic: A systematic review. SSM—Population Health, 21, 101348. https://doi.org/10.1016/j.ssmph.2023.101348

19.

Fahle

Reardon

S. F.

Shear

Saliba

Kalogrides

(2023). Stanford education data archive technical documentation: SEDA2022 2.0. http://purl.stanford.edu/db586ns4974

20.

Gassman-Pines

Ananat

E. O.

Fitz-Henley

II . (2020). COVID-19 and parent-child psychological well-being. Pediatrics, 146(4), e2020007294. https://doi.org/10.1542/peds.2020-007294

21.

Goldhaber

Kane

T. J.

McEachin

Morton

(2022a). A comprehensive picture of achievement across the COVID-19 pandemic years: Examining variation in test levels and growth across districts, schools, grades, and students (CALDER Working Paper No. 266–0522). Center for Analysis of Longitudinal Data in Education Research. https://caldercenter.org/publications/comprehensive-picture-achievement-across-covid-19-pandemic-years-examining-variation

22.

Goldhaber

Kane

T. J.

McEachin

Morton

Patterson

Staiger

D. O.

(2022b). The consequences of remote and hybrid instruction during the pandemic (Working Paper No. 30010). National Bureau of Economic Research. https://doi.org/10.3386/w30010

23.

Grossmann

Reckhow

Strunk

K. O.

Turner

(2021). All states close but red districts reopen: The politics of in-person schooling during the COVID-19 pandemic. Educational Researcher, 50(9), 637–648. https://doi.org/10.3102/0013189X211048840

24.

A. D.

(2008). The problem with “proficiency”: Limitations of statistics and policy under No Child Left Behind. Educational Researcher, 37(6), 351–360. https://doi.org/10.3102/0013189X08323842

25.

Jack

Halloran

Okun

Oster

(2022). Pandemic schooling mode and student test scores: Evidence from U.S. school districts. https://emilyoster.net/wp-content/uploads/MS_Updated_Revised.pdf

26.

Klinenberg

(2002). Heat wave: A social autopsy of disaster in Chicago. University of Chicago Press.

27.

Kogan

Lavertu

(2021). How the COVID-19 pandemic affected student learning in Ohio: Analysis of spring 2021 Ohio state tests. The Ohio State University, John Glenn College of Public Affairs. https://glenn.osu.edu/sites/default/files/2021-10/210828_KL_OST_Final_0.pdf

28.

Kozakowski

Gill

Lavallee

Burnett

Ladinsky

(2021, December). Changes in academic achievement in Pittsburgh Public Schools during remote instruction in the COVID-19 pandemic [Slide deck]. Regional Educational Laboratory Program (REL). https://ies.ed.gov/ncee/rel/Products/Region/midatlantic/Resource/100413

29.

Kuhfeld

Soland

Lewis

(2022). Test score patterns across three COVID-19-impacted school years. Educational Researcher, 51(7), 500–506. https://doi.org/10.3102/0013189X221109178

30.

Kush

J. M.

Badillo-Goicoechea

Musci

R. J.

Stuart

E. A.

(2022). Teachers’ mental health during the COVID-19 pandemic. Educational Researcher, 51(9), 593–597. https://doi.org/10.3102/0013189X221134281

31.

Lewis

Kuhfeld

(2022). Progress towards pandemic recovery: Continued signs of rebounding achievement at the start of the 2022–23 school year [Research brief]. NWEA. https://www.nwea.org/research/publication/progress-towards-pandemic-recovery-continued-signs-of-rebounding-achievement-at-the-start-of-the-2022-23-school-year/

32.

Lewis

Kuhfeld

Ruzek

McEachin

(2021). Learning during COVID-19: Reading and math achievement in the 2020–21 school year [Research brief]. NWEA. https://www.nwea.org/uploads/2021/07/Learning-during-COVID-19-Reading-and-math-achievement-in-the-2020-2021-school-year.research-brief-1.pdf

33.

Spoer

B. R.

Lampe

T. M.

Hsieh

P. Y.

Nelson

I. S.

Vierse

Thorpe

L. E.

Gourevitch

M. N.

(2023). Racial/ethnic and income disparities in neighborhood-level broadband access in 905 US cities, 2017–2021. Public Health, 217, 205–211. https://doi.org/10.1016/j.puhe.2023.02.001

34.

Locke

V. N.

Patarapichayatham

Lewis

(2021). Learning loss in reading and math in U.S. schools due to the COVID-19 pandemic. Istation. https://www.istation.com/hubfs/Content/downloads/studies/COVID-19_Learning_Loss_USA.pdf

35.

National Center for Health Statistics. (2023). Provisional COVID-19 deaths by HHS region, race, and age. https://data.cdc.gov/d/tpcp-uiv5

36.

Opportunity Insights. (2022). Opportunity Insights economic tracker. https://www.tracktherecovery.org

37.

Oster

Jack

Halloran

Schoof

McLeod

Yang

Roche

(2021). Disparities in learning mode access among K–12 students during the COVID-19 pandemic, by race/ethnicity, geography, and grade level—United States, September 2020–April 2021. Morbidity and Mortality Weekly Report, 70(26), 953–958. https://doi.org/10.15585/mmwr.mm7026e2

38.

Parolin

Lee

E. K.

(2021). Large socio-economic, geographic and demographic disparities exist in exposure to school closures. Nature Human Behaviour, 5(4), 522–528. https://doi.org/10.1038/s41562-021-01087-8

39.

Partelow

(2021). Social capital and community disaster resilience: Post-earthquake tourism recovery on Gili Trawangan, Indonesia. Sustainability Science, 16(1), 203–220. https://doi.org/10.1007/s11625-020-00854-2

40.

Pier

Christian

Tymeson

Meyer

R. H.

(2021). COVID-19 impacts on student learning: Evidence from interim assessments in California. https://edpolicyinca.org/sites/default/files/2021-06/r_pier_jun2021.pdf

41.

Reardon

S. F.

Fahle

A. D.

Shear

Kalogrides

Saliba

Kane

T. J.

(2023). Stanford education data archive (Version SEDA 2022 2.0). Stanford University. https://purl.stanford.edu/dt080zr0625

42.

Reardon

S. F.

Kalogrides

A. D.

(2021). Validation methods for aggregate-level test scale linking: A case study mapping school district test score distributions to a common scale. Journal of Educational and Behavioral Statistics, 46(2), 138–167. https://doi.org/10.3102/1076998619874089

43.

Reardon

S F.

Owens

Kalogrides

Jang

Tom

(2024). The Segregation Explorer (Version 1.0). http://edopportunity.org/segregation.

44.

Reardon

S. F.

Shear

B. R.

Castellano

K. E.

A. D.

(2017). Using heteroskedastic ordered probit models to recover moments of continuous test score distributions from coarsened data. Journal of Educational and Behavioral Statistics, 42(1), 3–45. https://doi.org/10.3102/1076998616666279

45.

Rege

Telle

Votruba

(2011). Parental job loss and children’s school performance. The Review of Economic Studies, 78(4), 1462–1489. https://doi.org/10.1093/restud/rdr002

46.

SafeGraph. (2022). Weekly patterns data set. https://docs.safegraph.com/docs/weekly-patterns

47.

Salomon

J. A.

Reinhart

Bilinski

Chua

E. J.

La Motte-Kerr

Rönn

M. M.

Reitsma

M. B.

Morris

K. A.

LaRocca

Farag

T. H.

Kreuter

Rosenfeld

Tibshirani

R. J.

(2021). The US COVID-19 Trends and Impact Survey: Continuous real-time measurement of COVID-19 symptoms, risks, protective behaviors, testing, and vaccination. Proceedings of the National Academy of Sciences, 118(51), e2111454118. https://doi.org/10.1073/pnas.2111454118

48.

Stevens

A. H.

Schaller

(2011). Short-run effects of parental job loss on children’s academic achievement. Economics of Education Review, 30(2), 289–299. https://doi.org/10.1016/j.econedurev.2010.10.002

49.

Thomeer

M. B.

Moody

M. D.

Yahirun

(2023). Racial and ethnic disparities in mental health and mental health care during the COVID-19 pandemic. Journal of Racial and Ethnic Health Disparities, 10(2), 961–976. https://doi.org/10.1007/s40615-022-01284-9

50.

U.S. Bureau of Labor Statistics. (2023). Labor force characteristics by race and ethnicity, 2021. https://www.bls.gov/opub/reports/race-and-ethnicity/2021/home.htm

51.

U.S. Census Bureau. (2019). 2015–19 American Community Survey 5-year estimates. https://www.census.gov/programs-surveys/acs/data.html

52.

U.S. Congress, Joint Economic Committee, Social Capital Project. (2018). The geography of social capital in America. Report prepared by the Vice Chairman’s staff, 115th Cong., 2nd Sess. https://www.jec.senate.gov/public/index.cfm/republicans/2018/4/the-geography-of-social-capital-in-america

53.

U.S. Department of Education. (n.d.). The nation’s report card. https://www.nationsreportcard.gov

54.

U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. (2019). Education Demographic and Geographic Estimates Program (EDGE): Composite school district boundaries. https://nces.ed.gov/programs/edge/Geographic/DistrictBoundaries

55.

U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. (2022a). National Assessment of Educational Progress (NAEP) pre-assessment activity data collection. https://ies.ed.gov/schoolsurvey/2022NAEPEnrollment_Policies/

56.

U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, Common Core of Data (CCD). (2022b). Public elementary/secondary school universe survey. 2017–18 v.1a, 2018–19 v.1a, 2021–22 v.1a.

57.

USAFacts. (2022). Understanding the COVID-19 pandemic. https://usafacts.org/issues/coronavirus/

58.

Voting and Election Science Team. (2020). 2020 precinct-level election results. Harvard Dataverse, V40. https://doi.org/10.7910/DVN/K7760H

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.50 MB