Early Warning Indicators in Education: Innovations,Uses,and Optimal Conditions for Effectiveness

Abstract

Background/Context

The passage of the No Child Left Behind Act in 2002 and the Every Student Succeeds Act in 2015 spurred changes in the way educators use data. On the one hand, the policies inspired educators’ awareness of large gaps in achievement between subgroups based on gender, race, and socioeconomic status. On the other hand, the policies inspired the use of data-based indicators in day-to-day routines in schools. In some cases, practitioners started working with researchers to analyze data using advanced measurement techniques to develop “early warning indicators.” These indicators identify when students are at risk of not achieving key milestones, such as graduating from high school, in hopes of providing practitioners with time to intervene early enough to change the students’ trajectories. Educational leaders have created “early warning systems” that produce reports summarizing these indicators, provide a process for the leaders to organize interventions based on the indicator reports, and a further process to examine whether students are back on track toward milestones such as high school graduation as a result of the intervention. Given the pace at which the science and use of these indicators is advancing, the topic is in need of a new resource explaining the most up-to-date elements involved with early warning indicators.

Purpose/Focus of the Study

This article will describe the evolution of the research, use, and conditions that influence the use of early warning indicators. The review of the literature includes a description of each of the yearbook articles and how they summarize some of the most recent advancements in early warning indicators in education.

Research Design

This is a review of literature.

Findings

As indicators in education have improved, these indicators show practitioners when students are not on track to reach certain milestones and provide practitioners with time to intervene early enough to change the child's trajectory. Allensworth and other authors (e.g., Balfanz et al.; Faria et al.; and Soland) provide evidence that these early warning indicators are impactful when used by practitioners in certain ways. Earlier efforts around early warning indicators focused on identifying measures that were predictive of key outcomes, such as high school graduation. As efforts to use indicators in practice became more mature, the focus expanded from providing information through indicators and information systems to ensuring that practitioners had the capacity and opportunity to engage in indicator use to guide their daily practice, and that the conditions in schools, school districts, and states supported indicator use. More recently, methodological advances around data analytics and a growing interest in social–emotional learning have expanded the approach and scope of indicators and their use. That said, the research on the nature and use of these indicators needs further documentation to keep up with the rapid pace at which the indicators are advancing and becoming more precise. The field still lacks a robust research base on how to use the indicators to improve practices and policies, and what conditions support the development and use of the indicators in education.

Conclusions/Recommendations

We will explain how the studies in this yearbook provide more up-to-date information on measurement related to the indicators, on the influence of certain approaches to using the indicators, as well as research on how conditions interact with the development and use of the indicators. The article demonstrates an important slice of what the field knows, as well as challenges and opportunities for further work to improve the development and use of early warning indicators.

Get full access to this article

View all access options for this article.

References

Allensworth

E. M.

(2013). The use of ninth-grade early warning indicators to improve Chicago schools. Journal of Education for Students Placed at Risk (JESPAR), 18(1), 68–83.

Allensworth

E. M.

, & Easton

J. Q.

(2005). The on-track indicator as a predictor of high school graduation. UChicago Consortium on Chicago School Research.

Allensworth

E. M.

, & Easton

J. Q.

(2007). What matters for staying on-track and graduating in Chicago public high schools: A close look at course grades, failures, and attendance in the freshman year. UChicago Consortium on Chicago School Research.

Allensworth

E. M.

, Gwynne

J. A.

, Moore

, & de la Torre

(2014, November). Looking forward to high school and college: Middle grades indicators of readiness in Chicago public schools. UChicago Consortium on Chicago School Research.

Allensworth

E. M.

, Nagaoka

, & Johnson

D. W.

(2018). High school graduation and college readiness indicator systems: What we know, what we need to know. UChicago Consortium on Chicago School Research.

Balfanz

(2012). A path forward: Evidence-based approaches to educational policy and practice. Journal of Applied Research on Children: Informing Policy for Children at Risk, 3(2), Article 1.

Balfanz

, Herzog

, & Mac Iver

D. J.

(2007). Preventing student disengagement and keeping students on the graduation path in urban middle-grades schools: Early identification and effective interventions. Educational Psychologist, 42(4), 223–235.

Baltimore Education Research Consortium. (2011, February). Destination graduation: Sixth grade early warning indicators for Baltimore city schools: Their prevalence and impact. http://baltimore-berc.org/pdfs/SixthGradeEWIFullReport.pdf

Barnett

W. S.

, Jung

, Friedman-Kraus

, Frede

E. C.

, Milagros

, Hustedt

J. T.

, Howes

, & Daniel-Echols

(2018). State prekindergarten effects on early learning at kindergarten entry: An analysis of eight state programs. AERA Open, 4(2).

10.

Bassok

, & Latham

(2017). Kids today: The rise of children's academic skills at early kindergarten. Educational Researcher, 46(1), 7–20.

11.

Bertrand

, & Marsh

J. A.

(2015). Teachers’ sensemaking of data and implications for equity. American Educational Research Journal, 52(5), 861–893.

12.

Biag

, Gerstein

, Fehrer

K. C.

, Sanchez

, & Sipes

(2016). Data use and inquiry in research–practice partnerships: Four case examples. John W. Gardner Center for Youth and Their Communities.

13.

Bowers

A. J.

, & Zhou

(2019). Receiver operating characteristic (ROC) area under the curve (AUC): A diagnostic measure for evaluating the accuracy of predictors of education outcomes. Journal of Education for Students Placed at Risk, 24(1), 20–46.

14.

Bruce

, Bridgeland

J. M.

, Fox

J. H.

, & Balfanz

(2011). On track for success: The use of early warning indicator and intervention systems to build a grad nation. Civic Enterprises.

15.

Burke

(2015). Early identification of high school graduation outcomes in Oregon Leadership Network schools (REL 2015–079). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Northwest. http://ies.ed.gov/ncee/edlabs

16.

Carl

, Richardson

J. T.

, Cheng

, Kim

, & Meyer

R. H.

(2013, February). Theory and application of early warning systems for high school and beyond. Journal of Education for Students Placed at Risk (JESPAR), 18(1), 29–49.

17.

Claessens

, Engel

, & Curran

F. C.

(2014). Academic content, student learning, and the persistence of preschool effects. American Education Research Journal, 51(2), 403–434.

18.

Coburn

C. E.

, & Penuel

W. R.

(2016). Research–practice partnerships in education: Outcomes, dynamics, and open questions. Educational Researcher, 45(1), 48–54.

19.

Coburn

C. E.

, Penuel

W. R.

, & Geil

(2013). Research–practice partnerships at the district level: A new strategy for leveraging research for educational improvement. University of California and University of Colorado.

20.

Connolly

, & Olson

L. S.

(2012). Early elementary performance and attendance in Baltimore city schools’ pre-kindergarten and kindergarten. Baltimore Education Research Consortium. Johns Hopkins University. http://baltimore-berc.org/pdfs/PreKKAttendanceFullReport.pdf

21.

Davis

, Herzog

, & Legters

(2013). Organizing schools to address early warning indicators (EWIs): Common practices and challenges. Journal of Education for Students Placed at Risk (JESPAR), 18(1), 84–100.

22.

Davis

, Mac Iver

M. A.

, Balfanz

R. W.

, Stein

M. L.

, & Fox

J. H.

(2019). Implementation of an early warning indicator and intervention system. Preventing School Failure: Alternative Education for Children and Youth, 63(1), 77–88.

23.

Deussen

, Hanson

, & Bisht

(2017). Are two commonly used early warning indicators accurate predictors of dropout for English learner students? Evidence from six districts in Washington state (REL 2017–261). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Northwest. http://ies.ed.gov/ncee/edlabs

24.

Duckworth

A. L.

, & Yeager

(2015). Measurement matters: Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44(4), 237–251.

25.

Dynarski

, Clarke

, Cobb

, Finn

, Rumberger

, & Smink

(2008). Dropout prevention: A practice guide (NCEE 2008–4025). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. http://ies.ed.gov/ncee/wwc

26.

Faria

A.-M.

, Sorensen

, Heppen

, Bowdon

, Taylor

, Eisner

, & Foster

(2017). Getting students on track for graduation: Impacts of the early warning intervention and monitoring system after one year (REL 2017–272). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Midwest. http://ies.ed.gov/ncee/edlabs

27.

Farrell

C. C.

, Coburn

C. E.

, & Chong

(2018). Under what conditions do school districts learn from external partners? The role of absorptive capacity. American Educational Research Journal, 56(3), 955–994.

28.

Finnigan

K. S.

, & Daly

A. J.

(Ed.). (2014). Using research evidence in education: From the schoolhouse door to Capitol Hill (Policy Implications for Research in Education, Vol. 2). Springer.

29.

Frazelle

, & Nagel

(2015). A practitioner's guide to implementing early warning systems (REL 2015–056). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Northwest. http://ies.ed.gov/ncee/edlabs

30.

Gleason

, & Dynarski

(2002). Do we know whom to serve? Issues in using risk factors to identify dropouts. Journal of Education for Students Placed at Risk, 7(1), 25–41.

31.

Gottfried

(2017). ELL School Readiness and Pre-Kindergarten Care. Educational Policy, 31(1), 39–72.

32.

Grodsky

, Pyne

, Vaade

, Camburn

, & Bradley

(2017). Preparing the littlest learners: Four-year-old kindergarten enrollment and kindergarten readiness in MMSD. Madison Education Partnerships, 1–22.

33.

Gurantz

, & Borsato

G. N.

(2012). Building and implementing a college readiness indicator system: Lessons from the first two years of the CRIS initiative. Voices in Urban Education, 35, 5–15.

34.

Gwynne

, Lesnick

, Hart

H. M.

, & Allensworth

E. M.

(2009, December). What matters for staying on-track and graduating in Chicago public schools: A focus on students with disabilities. Paper presented at the UChicago Consortium on Chicago School Research, Chicago.

35.

Gwynne

, Pareja

A. S.

, Ehrlich

S. B.

, & Allensworth

(2012). What matters for staying on-track and graduating in Chicago Public Schools: A focus on English language learners. UChicago Consortium on Chicago School Research.

36.

Hawkins

(2019, June 26). Pulling all the levers —Chicago Public Schools CEO Janice Jackson on getting more students to and through college. The 74. https://www.the74million.org/article/74-interview-pulling-all-the-levers-chicago-public-schools-ceo-janice-jackson-on-getting-more-students-to-and-through-college/

37.

Heifetz

R. A.

, Grashow

, & Linsky

(2009). The practice of adaptive leadership: Tools and tactics for changing your organization and the world. Harvard Business School Press.

38.

Henrick

E. C.

, Klafehn

A. B.

, & Cobb

(2018). Assessing the impact of partnership recommendations on district instructional improvement strategies. In Cobb

, Jackson

, Smith

T.M.

, & The MIST Team (Eds.), Systems for instructional improvement: Creating coherence from the classroom to the district office (pp. 209–220). Harvard Education Press.

39.

Heppen

J. B.

, & Therriault

S. B.

(2008, July). Developing early warning systems to identify potential high school dropouts. National High School Center. American Institutes for Research.

40.

Hough

, Kalogrides

, & Loeb

(2017). Using surveys of students’ social–emotional skills and school climate for accountability and continuous improvement. Policy Analysis for California Education. Stanford Graduate School of Education. https://edpolicyinca.org/sites/default/files/SEL-CC_report.pdf

41.

Hough

, Willis

, Grunow

, Krausen

, Kwon

, Mulfinger

L. S.

, & Park

(2017, November). Continuous improvement in practice. Policy Analysis for California in Education. Stanford Graduate School of Education. https://edpolicyinca.org/sites/default/files/CI%20in%20Pratice.pdf

42.

Irvin

L. K.

, Horner

R. H.

, Ingram

, Todd

A. W.

, Sugai

, Sampson

N. K.

, & Boland

J. B.

(2006). Using office discipline referral data for decision making about student behavior in elementary and middle schools: An empirical evaluation of validity. Journal of Positive Behavior Interventions., 8(1), 10–23.

43.

John W. Gardner Center for Youth and Their Communities. (2011, June). Youth data archive issue brief: Developing early warning indicators for the San Francisco unified school district. https://gardnercenter.stanford.edu/sites/g/files/sbiybj11216/f/Developing%20EWI%20for%20SFUSD%20Issue%20Brief.pdf

44.

Kelleher

(2018, January). Progress and promise: Chicago's nation-leading education gains. The Joyce Foundation. ERIC. https://files.eric.ed.gov/fulltext/ED588379.pdf

45.

Kemple

J. J.

, Segeritz

M. D.

, & Stephenson

(2013). Building on-track indicators for high school graduation and college readiness: Evidence from New York City. Journal of Education for Students Placed at Risk (JESPAR), 18(1), 7–28.

46.

Koon

, & Petscher

(2015). Comparing methodologies for developing an early warning system: Classification and regression tree model versus logistic regression (REL 2015–077). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southeast. http://ies.ed.gov/ncee/edlabs

47.

Kurlaender

, Reardon

, & Jackson

(2008). Middle school predictors of high school achievement in three California school districts. California Dropout Research Project. http://cdrp.ucsb.edu/dropouts/pubs_reports.htm

48.

Mac Iver

M. A.

(2013). Early warning indicators of high school outcomes. Journal of Education for Students Placed at Risk (JESPAR), 18(1), 1–6.

49.

Mac Iver

M. A.

, & Messel

(2013). The ABCs of keeping on track to graduation: Research findings from Baltimore. Journal of Education for Students Placed at Risk (JESPAR), 18(1), 50–67.

50.

Mac Iver

M. A.

, Plank

S. B.

, Durham

, Farley-Ripple

, & Balfanz

(2008). The challenge of on-time arrival: The seven-year fight paths of Baltimore's sixth graders of 1999-2000 (Research report). Johns Hopkins University. http://www.baltimore-berc.org/pdfs/SIXTH%20pathways5-13-08.pdf

51.

Mac Iver

M. A.

, Sheldon

, Naeger

, & Clark

(2016). Mentoring students back on track to graduation: Program results from five communities. Education and Urban Society, 49(7), 643–675.

52.

Mac Iver

M. A.

, Stein

M. L.

, Davis

M. H.

, Balfanz

R. W.

, & Fox

J. H.

(2019). An efficacy study of a ninth-grade early warning indicator intervention. Journal on Research on Educational Effectiveness, 12(3), 363–390.

53.

Marsh

J. A.

, Pane

J. F.

, & Hamilton

L. S.

(2006). Making sense of data driven decision-making in education: Evidence from recent RAND research (Occasional paper). RAND.

54.

Mazzeo

, Fleishman

, Heppen

, & Jahangir

(2016). Improving high school success: Searching for evidence of promise. Teachers College Record, 118(3), 1–32.

55.

McKown

, Allen

A. M.

, Russo-Ponsaran

N. M.

, & Johnson

J. K.

(2013). Direct assessment of children's social–emotional comprehension. Psychological Assessment, 25(4), 1154–1166.

56.

Miller

S. R.

, Allensworth

E. M.

, & Kochanek

J. R.

(2002, May). The state of Chicago public high schools 1993-2000. Student performance: Course taking, test scores, and outcomes. UChicago Consortium on Chicago School Research. https://consortium.uchicago.edu/sites/default/files/2018-10/p52.pdf

57.

Miller

S. R.

, Easton

J. Q.

, Gladden

R. M.

, & Luppescu

(1999, August). How do Barton graduates perform in CPS high schools? UChicago Consortium on Chicago School Research. https://consortium.uchicago.edu/sites/default/files/2018-10/report2170.pdf

58.

Moeller

, Seeskin

, & Nagaoka

(2018). Practice-driven data: Lessons from Chicago's approach to research, data, and practice in education. UChicago Consortium on Chicago School Research.

59.

Nagaoka

, Farrington

, Roderick

, Allensworth

, Keyes

T. S.

, Johnson

D. W.

, & Beechum

N. O.

(2013, Fall). Readiness for college: The role of non-cognitive factors and context. Voices in Urban Education. Annenberg Institute for School Reform.

60.

Neild

R. C.

(2009, Spring). Falling off track during the transition to high school: What we know and what can be done. The Future of Children, 19(1), 53–76.

61.

Neild

R. C.

, & Balfanz

(2006). Unfulfilled promise: The dimensions and characteristics of Philadelphia's dropout crisis, 2000–2005. Philadelphia Youth Network. The Johns Hopkins University and University of Pennsylvania. ERIC. https://files.eric.ed.gov/fulltext/ED538341.pdf

62.

Neild

R. C.

, Balfanz

, & Herzog

(2007). An early warning system. Educational Leadership, 65(2), 28–33.

63.

Nutley

S. M.

, Walter

, & Davies

H. T. O.

(2007). Using evidence: How research can inform public services. University of Chicago Press.

64.

Park

, Hironaka

, Carver

, & Nordstrum

(2013). Continuous improvement in education. Carnegie Foundation for the Advancement of Teaching. https://www.carnegiefoundation.org/wp-content/uploads/2014/09/carnegie-foundation_continuous-improvement_2013.05.pdf

65.

Penuel

W. R.

, Briggs

D. C.

, Davidson

K. L.

, Herlihy

, Hill

H. C.

, Farrell

C. C.

, & Allen

A-R.

(2017). How school and district leaders access, perceive, and use research. AERA Open, 3(2), 1–17.

66.

Penuel

W. R.

, & Gallagher

D. J.

(2017). Creating research practice partnerships in education. Harvard Education Press.

67.

Pitcher

M. A.

, Duncan

S. J.

, Nagaoka

, Moeller

, Dickerson

, & Beechum

N. O.

(2016). The network for college success: A capacity building model for school improvement. The Network for College Success. The University of Chicago School of Social Service Administration.

68.

Planty

, & Carlson

(2010). Understanding education indicators: A practical primer for research and policy. Teachers College Press.

69.

Reardon

S. F.

, & Hinze-Pifer

(2017). Test score growth among public school students in Chicago, 2009-2014. Stanford Center for Education Policy Analysis. https://cepa.stanford.edu/content/test-score-growth-among-chicago-public-school-students-2009-2014

70.

Reardon

S. F.

, & Portilla

X. A.

(2016, August). Recent trends in income, racial, and ethnic school readiness at kindergarten entry. AERA Open, 2(3).

71.

Roderick

, Easton

, & Sebring

(2009). The Consortium on Chicago School Research: A new model for the role of research in supporting urban school reform. UChicago Consortium on Chicago School Research. https://ccsr.uchicago.edu/sites/default/files/publications/CCSR%20Model%20Report-final.pdf

72.

Rumberger

, Addis

, Allensworth

, Balfanz

, Bruch

, Dillon

, Duardo

, Dynarski

, Furgeson

, Jayanthi

, Newman-Gonchar

, Place

, & Tuttle

(2017). Preventing drop-out in secondary schools (NCEE 2017-4028). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. https://whatworks.ed.gov

73.

Soland

(2013). Predicting high school graduation and college enrollment: Comparing early warning indicator data to teacher intuition. Journal of Education for Students Placed at Risk (JESPAR), 18(3/4), 233–262.

74.

Thompson

(2015). English learners’ time to reclassification: An analysis. Educational Policy, 31(3), 330–363.

75.

Umansky

(2016, December). To be or not to be EL: An examination of the impact of classifying students as English learners. Educational Evaluation and Policy Analysis, 38(4), 714–737.

76.

U.S. Department of Education. (2016a, September 16). Non-regulatory guidance: Using evidence to strengthen education investments. https://www2.ed.gov/policy/elsec/leg/essa/guidanceuseseinvestment.pdf

77.

U.S. Department of Education. Office of Planning, Evaluation and Policy Development Policy and Program Studies Service. (2016b, September). Issue brief: Early warning systems. https://www2.ed.gov/rschstat/eval/high-school/early-warning-systems-brief.pdf

78.

U.S. Department of Education. (2020). Early warning systems. Institute of Education Sciences Regional Educational Laboratory Program. https://ies.ed.gov/ncee/edlabs/projects/ews.asp

79.

West

, Buckley

, Krachman

S. B.

, & Bookman

(2018, March–April). Development and implementation of student social emotional surveys in the CORE Districts. Journal of Applied Developmental Psychology, 55, 119–129.

80.

Whitney

, & Liu

(2017). What we're missing: A descriptive analysis of part-day absenteeism in secondary school. AERA Open, 3(2).