Abstract
The continuous improvement (CI) approach to systems change has rapidly spread across education policy circles in recent years and has been hailed as a promising means to achieve educational equity and social justice. CI’s highly routinized, scientific process for improving efficiency and productivity is a somewhat unexpected means to pursue equity. To understand this puzzle, I examine the use of CI to promote equity through two qualitative, multilevel case studies. I draw on institutional theory to understand how CI has integrated logics of racial equity and performance, and how local actors have improvised novel approaches. This analysis illuminates the complex institutional dynamics at play with CI implementation and identifies the challenges and promise of using CI to promote educational equity.
The continuous improvement (CI) approach to systems change has rapidly spread across education policy circles in recent years and has been hailed as a promising means to achieve educational equity and social justice. The CI approach, which has been widely adopted by state and local educational agencies as well as schools, engages practitioners in enacting systemic change through a rigorous, data- and evidence-infused approach to decision making, often called Improvement Science, cycles of improvement, and quality improvement. Derived from the scientific management and total quality control movements in the private sector and widely used in health care, CI involves iterative, structured cycles of data analysis to solve problems and test new solutions often paired with purposeful facilitated interorganizational networks intended to promote knowledge sharing, transparency, and accountability. After nearly two decades of one-size-fits-all school reforms focused on quick turnaround, 1 the current reform movement has centered on, or rather returned to, incremental, locally designed reforms aimed at longer-term gains.
This movement has been spurred on by legislative and investment incentives, primarily aimed at promoting CI as a means to promote equity and racial justice (in terms of closing opportunity gaps and improving the outcomes of BIPOC 2 students). By 2018, 16 states had already indicated that CI would be a key component of their theory of action in state accountability plans under the Every Student Succeeds Act, aimed at closing achievement gaps (Results for America, 2018). With CI collaboratives and networks springing up at all levels of school systems, the Bill and Melinda Gates Foundation provided generous funding to over 30 Networks for School Improvement specifically to support their use of CI to improve the outcomes of Black, Latinx, and/or students experiencing poverty. While the goal of educational equity has generally seen strong support and legitimacy in the field, decades of research have documented the intransigence of the opportunity gap and the essential challenges in disrupting existing power dynamics entrenched in the regulations, norms, habits, and practices of an institution (Castro, 2015). Given the urgency and necessity of closing the opportunity gap, is the CI approach well-suited to address this critical problem? Can this new movement of CI gain traction in achieving these key goals of education—promoting educational equity and closing the opportunity gap?
Indeed, CI is a rather unexpected method to promote equity. With its roots in Taylorism and the Total Quality Control movement in industry, CI is typically a highly routinized, scientific process to promote incremental changes that improve efficiency, productivity, and profitability. While the goals of efficiency and productivity are not necessarily at odds with promoting equity, the underlying assumptions and means typically associated with these goals can diverge substantially. For these reasons, some scholars have argued that CI is incompatible with the culture of schooling (e.g., Ingram et al., 2004; Kohn, 1993; Prawat, 1996; Sarason, 1996). That is, CI’s original focus on attaining higher profitability by efficiently producing better products or services for customers and streamlining systems and processes conflicts with the liberal values underlying modern education (Ingram et al., 2004).
To understand the complexity of using CI to promote equity, I analyze data from two multilevel cases of intermediary organizations and school districts using CI. Drawing on concepts from institutional theory, I explore how these cases draw on and operationalize the guiding principles, material practices, and symbolic structures embedded in two orders of education policy—the performance-based policy order and the racial equity policy order—through specific modes of reproduction during early institutionalization. Specifically, I answer the following research questions:
I find that these case organizations sought to close the opportunity gap and promote racial justice through typical CI approaches with varying levels of formality and discipline. I find that cases used three primary mechanisms— performance metrics, routines and tools, and professional norms—to combine these logics in implementing CI and I identify promising practices that functioned to connect the guiding principles of performance and equity. Although individual respondents varied in the extent to which they endorsed the guiding principles of performance versus equity, almost all respondents invoked aspects of both in their description of CI work. On the whole, these findings indicate that CI was neither the silver bullet to achieve equity, nor an unsuitable technology for promoting racial justice. CI shows potential to allow local actors to mobilize accepted practices of quantification and accountability to challenge entrenched power dynamics and modify systems to provide more equitable educational opportunity.
Understanding Continuous Improvement
CI is an umbrella term used to describe an approach to practice intended to “develop the necessary know-how for a reform idea ultimately to spread faster and more effectively” through structured, collaborative, iterative cycles of data analysis, and intervention testing (Bryk et al., 2015, p. 8 [emphasis in original]). Derived from a long history of efforts to improve productivity, efficiency, and profitability in industry and a shorter history of applications in health care and education (see Bhuiyan & Baghel, 2005, for a full history), the current CI movement in education is rapidly gaining popularity. CI is not new to education and has been used in various forms in education over the past few decades, including professional learning communities (PLCs), results-oriented cycle of inquiry (ROCI), action research, and quality management, with varied success. In contrast to past approaches, the current movement of CI in education (particularly the Improvement Science methodology) makes great strides in emphasizing the importance of context and attention to variability, which may present opportunities for equity-focused CI.
Most current CI methodologies used in education include a common set of inquiry tools and procedures and emphasize transparency and documentation (LeMahieu et al., 2017) to aid in the development and testing of hypotheses against (local) data and evidence through experimentation. CI typically begins with an identified problem of practice and causal systems mapping, which involves identifying and visually mapping the various systems and subsystems relevant to the problem. This process can help identify potential implementation challenges and select appropriate levers of change. Fishbone diagrams are generally used during the process of brainstorming potential root causes for problem of practice. These diagrams situate various contributing causes for the ultimate effect (problem), grouped by major factors (e.g., job design, culture, norms, compensation, leadership, materials, and motivation). To better focus the root cause analysis, educational organizations may use the Five Whys protocol, in which the team seeks to question each link in the causal system.
These common tools are utilized in concert with a standard process of experimentation referred to as a plan–do–study–act (PDSA) cycle. Essentially, after identifying a root cause and potential change idea to address a problem of practice, the CI team would then plan the implementation of a change idea (most often within a small-scale pilot), implement the change, measure the impact of the change (using various leading indicators, predictive measurements, or intermediary outcomes), and then choose to adopt, adapt, or abandon the change idea. These PDSA cycles are intended to be relatively short (often 90 days) and iterative.
Park et al. (2013) point out that the word continuous in CI refers both to the iterative nature and part of the daily practice but is also integrated into organizational structures and mindsets. This approach is necessarily centered on “focused and continuous incremental innovation” (Bessant et al., 1994, p. 18) rather than dramatic innovation. The focus of change is also deeply contextually dependent, “where the varied demands and details of local contexts are a direct object of study and design” (Bryk et al., 2010, p. 10). It is through this scientific and pragmatic approach that CI seeks to understand what works in addressing a particular problem, for whom, and under what set of specific conditions (Berwick, 2008). While intuitive in concept, the implementation of CI to pursue equity presents substantial complexities as it implemented in educational systems that have seen repeated substantive shifts in principles, structures, and practice.
Institutional Theory: Orders and Institutionalization
Institutional theory is a powerful lens to examine the implementation of education policy, as it can help to answer fundamental questions about “persistence and change,” what “spreads,” and what “sticks” (Colyvas & Jonsson, 2011). By drawing attention to the duality of structure and agency and the complexity of interrelated micro-, meso-, and macro-level dynamics, institutional theory illuminated the intricacy and nuance of how policy becomes practice within school systems.
Essentially, this body of theory is built on the core understanding of institutions as resilient, stable social structures, “composed of cultural cognitive, normative, and regulative elements that, together with associated activities and resources, provide stability and meaning to social life” (W. R. Scott, 1995, p. 33). Institutions (which may exist apart from or embedded within organizations or fields) are shaped by institutional logics: the “the socially constructed, historical patterns of material practices, assumptions, values, beliefs, and rules by which individuals produce and reproduce their material subsistence, organize time and space, and provide meaning to their social reality” (Thornton & Ocasio, 1999, p. 804). Logics, in this context, are not merely beliefs, but also include rules and practices shaping formal structure, work, and social relations.
Furthermore, logics are “fundamentally argued to exist in relation to each other, with the friction between them being the trigger for institutional change and agency” (Johansen & Waldorff, 2017, p. 58). The institutional logics approach focuses on the processual aspects of institutions, including how micro-level aspects of institutions (including individuals’ focus of attention, schema, and social interactions) influence the macro-level (Thornton et al., 2012). In this way, institutional logics play a crucial role in “shaping what problems and issues get attended to and what solutions are likely to be considered in decision-making” (Thornton et al., 2012, p. 90). Essentially, “Institutional Logics shape power in organizations” (Thornton & Ocasio, 1999, p. 803). This attention to problems and solutions, decision making, and power dynamics make this framework particularly apt for understanding efforts to pursue equity through CI.
Institutional scholars observed that the guiding principles, material practices, and symbolic structures of institutional logics can be recognized and categorized into (societal-level) institutional orders, including markets, states, corporations, professions, families, religions, and communities (Friedland & Alford, 1991; Johansen & Waldorff, 2017; Thornton, 2004; Thornton et al., 2012). These orders, guided by their institutional logics, often dominate particular sectors, but, just as institutional logics are inherently in conflict, fields, and organizations are subject to, and often draw on, multiple, competing logics, leading to institutional complexity (Greenwood et al., 2011; Rao & Giorgi, 2006; Seo & Creed, 2002). Moreover, institutional logics are not adopted wholesale but rather drawn on to bring order to work (Spillane, 2012). Logics present the repertoire of guiding principles, material practices, and symbolic structures available to individuals and organizations; they serve as frames of reference, helping actors make sense of their world and construct action within and across orders and providing opportunity for elaboration or improvisation (Friedland & Alford, 1991; Johansen & Waldorff, 2017; Thornton et al., 2012).
The process through which Logics inform and change institutions can be understood as institutionalization: “the activities and mechanisms by which structures, models, rules, and problem-solving routines become established as a taken-for-granted part of everyday social reality” (Schneiberg & Soule, 2005, p. 122). Focusing on “what sticks” (Colyvas & Jonsson, 2011, p. 38), institutionalization is both a process and outcome of the “formal instantiation of organizational structures . . . as well as mechanisms at each stage that sustain and reinforce the process” (p. 346). A new stream of analysis examines the Modes of Reproduction, or mechanisms, through which institutionalization occurs. Anderson and Colyvas (2021) identify seven modes of reproduction relevant to educational organizations: organizational routines, performance metrics, professional norms, students’ identity categories, formal policies, resonant frames, and stable stakeholder interests. These mechanisms function by defining and directing attention and resources toward certain practices, behaviors, and beliefs—resulting in stasis or change to institutions.
An Analogy for Understanding Policy and Practice
This foundation in institutional theory presents a framework for understanding how related guiding principles, material practices, and symbolic structures inform somewhat stable, if conflicting arrangements of social structure. It also describes how shifts occur in institutions when conflicting logics (and their instantiation in orders) are institutionalized at the macro-, meso-, and micro-levels. I utilize this broad framework as an analogy to understand the way in which the guiding principles of contradictory streams of education policy have informed the use of CI for equity.
This is not a unique approach, as education policy researchers utilized an organizational and field-level approach to institutional theory, identifying challenges presented by competing logics to advancing policy (Cuban, 1990; Ravitch, 2000) and achieving organizational coherence (Bacharach et al., 1996; Hallett, 2010; Piderit, 2000). For example, research in education has explored competing field-level logics of “academics” versus “development” in rationalizing kindergarten instructional practice (Russell, 2011), and examined how school context influences the relative weight of logics, such as “market accountability” or “community sentiment” (Bridwell-Mitchell & Sherer, 2017).
Figure 1 demonstrates how I conceptualize these abstract institutional concepts playing out in the empirical case in question. As this figure shows, I identify key societal-level institutional orders and their logics that informed the backbone of two policy movements (orders) in education. These orders instantiate conflicting sets of guiding principles (analogously, logics) that have been mobilized in the CI approach. 3 Modes of reproduction illustrate how these logics come together in the implementation of CI for equity at the meso- and micro-levels.
Theoretical framework illustrating institutional influences in policy implementation.
This framework presents a lens to illuminate if and how the logics of performance based and racial equity policy orders conflict or complement each other through CI implementation in ways that influence the social arrangements, structures, and practices of school systems.
Literature Documenting Extant Education Policy Logics and Orders
Major policy shifts over the past 50 years reveal two significant sets of logics evolving and taking precedence in the field of education, evidenced in a performance-based policy order and a racial equity order. To set the stage for understanding how CI for equity came about and is implemented, I discuss the empirical literature documenting substantial tensions and friction between these two policy orders and identify guiding principles of logic.
Performance-Based Education Policy Order
Since the 1980s, a dominant order of education policy has been informed by societal logics associated with the state and market orders and guided by the principles of standardization, accountability (enforced through sanctions and incentives), and quantification. These guiding principles are associated with both structures, such as a uniform set of standards for knowledge and skills to be attained by all students, as well as practices, such as the use of standardized assessments and ubiquitous use of data to inform instruction.
The 1983 National Commission on Excellence in Education report, A Nation at Risk, famously warned of “a rising tide of mediocrity” in schooling that tied performance to economic competitiveness, invoking the logics of dominant societal orders of the market and the state. This report, and its underlying beliefs and assumptions, sparked reforms focusing on holding schools accountable for student performance on standardized tests (Mehta, 2013). School accountability, under No Child Left Behind/Elementary and Secondary Education Act (NCLB/ESEA), was enforced through a bureaucratic system of federal and state agencies. These policies set standard performance targets for schools, employing standardized test data to measure performance and progress. Schools that failed to meet their targets were met with sanctions and standardized solutions.
While the aim of NCLB/ESEA included improving the academic performance of BIPOC students, the methods of improving schools and accountability arrangements under these policies led to distortive responses that endangered racial justice. For example, standardization of the content and assessment across schools was intended to ensure that all children were held to equally high standards. Sanctions associated with poor performance, however, motivated school staff to “hide” low-performing students in nontested categorizations or through absence (Booher-Jennings, 2005; Chakrabarti & Schwartz, 2013; Figlio, 2006; Price, 2010), to focus primarily on “bubble” students likely to meet the standards with minimal additional support (Booher-Jennings, 2005; Jennings & Sohn, 2014), and to modify instruction to a narrow set of tested standards (Dee et al., 2013; Jennings & Bearak, 2014; Jennings & Sohn, 2014; Judson, 2013). At the extreme, several districts and schools were found guilty of facilitating cheating on assessments (Jacob & Levitt, 2003; Koretz, 1996). In part, these behaviors may have been motivated by documented resistance to the policies by educators who felt that the policy placed blame while they believed that nonschool factors influenced their students’ performance (Smith, 1991) and that tests only captured a snapshot of student learning (Jones & Egley, 2004).
In practice, standardization, quantification, and accountability regimes often function to uphold traditional power arrangements that privilege White students (see Dixon-Roman, 2017). As this empirical literature demonstrates, many of the performance-focused policies purportedly aimed at addressing the achievement gap, like NCLB, have often exacerbated challenges faced by BIPOC students in accessing equitable educational opportunities (Darling-Hammond, 2007; Gay, 2007; Horn, 2018). For this reason, it is essential to examine the complexity of utilizing CI, particularly during early stages of implementation, to avoid responses that may endanger racial justice.
Racial Equity Policy Order
A perennial order of education policies, at least since the 1950s, the racial equity policy is tied to societal logics (and associated orders) of community and profession and guided by the principles of equality (e.g., providing same inputs), equity (e.g., providing different inputs to reach the same outcome), and justice (e.g., righting systems to result in equitable opportunities and outcomes). The vision of such policies involves ensuring “equality of access to and opportunities for meaningful, authentic learning experience, which are inclusive of learners from diverse backgrounds, which are relevant to learners” (Grant & Ladson-Billings, 1997, p. 103). Furthermore, deeply symbolic work has drawn attention to the instantiation of White supremacy and racism in U.S. culture (e.g., Gotanda, 2004; Ladson-Billings, 2004) and the essential links between educational opportunities, economic disparity, power asymmetry, and historical and present institutionalized racism (e.g., Bobo et al., 2000; Gutiérrez & Jaramillo, 2006). The guiding principles are evidenced in changes in rules that present more equal access to schools (e.g., desegregation orders), changes in funding structures to provide more equitable distribution, and challenges to instructional and disciplinary practices designed to solely serve the interests of White students.
Key policies that have manifest the racial equity logic have touched all areas of schooling, from enrollment and access, to content and instruction, to resource allocation. Legal remedies have often played a substantial role in challenging entrenched racism and changing practice, across core structures for desegregation (e.g., Brown v. Board of Education of Topeka, 347 U.S. 483 [1954]; Green v. County School Board of New Kent County, Virginia, 391 U.S. 430 [1968]), school funding (e.g., the Serrano v. Priest cases decided by the California Supreme Court; the Abbott v. Burke cases decided by the New Jersey Supreme Court), and language equity (e.g., Lau v. Nichols, 414 U.S. 563 [1974]; Plyler v. Doe, 457 U.S. 202 [1982]). Desegregation, which presents one of the longest standing movements instantiating this logic, demonstrates legal remedies are crucially important in addressing deeply institutionalized racism.
In recent years, increasing attention has been paid to closing opportunity gaps. The persistent achievement gap between BIPOC and White students is well documented (e.g., Reardon, 2008, Reardon et al., 2009; Strutchens et al., 2004; Wilkins et al., 2006) and researchers have sought to understand the underlying opportunity gaps that have led to differences in achievement (e.g., Carter & Welner, 2013; Darling-Hammond et al., 2014; Flores, 2007; Ladson-Billings, 2013; J. Scott & Wells, 2013). Researchers have demonstrated that many factors impede students’ opportunity to learn, such as inequitable placement of experienced teachers (Cowen et al., 2017; Mayer et al., 2000; Wilkins et al., 2006), teachers’ low expectations for students or misunderstanding of student needs (Boser et al., 2014; Irvine & York, 1993; Madaus et al., 1992; Walkey et al., 2013), inappropriate remedial course placement (Dauber et al., 1996; Oakes, 1995), and inequitable school funding (e.g., Baker & Corcoran, 2012; Knight, 2017; Kozol, 2005). In response, state and district policies have sought to provide access and opportunity to a meaningful education through changes to accountability, restorative justice (and reforms to disciplinary policy), access to higher education, and use of culturally responsive pedagogy (e.g., Anfara et al., 2013; Darling-Hammond & Friedlaender, 2008; Hashim et al., 2018; Ladd, 2008).
As these two brief histories indicate, there are deeply entrenched differences between the guiding principles of standardization, quantification, and accountability versus equality, equity, and justice. This presents challenges as the CI approach draws on elements of performance logic to pursue equity aims. Specifically, CI is informed by the guiding principles of scientific inquiry, pragmatism, collaboration, and systems adaptation. As such, as CI espouses certain principles similar to performance-based reforms (such as quantification and scientific inquiry), it also holds promise to promote equity, as it may involve empowering local actors to adjust systems in ways that challenge power arrangements that privilege White students and identify contextually appropriate practice.
Data and Method
I utilized a multiple, exploratory, embedded case-study approach drawing on several levels of analysis (Yin, 2013). Case study was particularly apt, as this study examined “process rather than outcomes” (Merriam, 1988, p. xii). I selected complementary cases of multilevel efforts to promote and implement CI in educational organizations: two intermediary organizations seeking to build the capacity of counties, districts, and schools to engage in CI. In addition, I completed case studies of one focal district participating in each network, which I refer to by the pseudonyms Manzanita and Sage, to illustrate how CI was implemented at the district and school levels. These two districts were both identified by their intermediaries as fully participating districts with more developed CI processes than their peers. That is, among the network of districts participating in capacity building activities, these districts had (1) participated fully in network activities, (2) been part of the networks since early on in the network’s creation, and (3) were beginning to implement CI processes, where other network districts were still in the planning stages. As such, these district cases provide useful information on how CI efforts were initiated and executed at the early stages of these reform efforts.
Data
To understand the phenomenon in question, I utilized observations, physical artifacts, documentation, and interviews, including think-aloud protocols. Data collection for this study spanned from 2015 to 2018. I began following the California Office to Reform Education (CORE) Collaborative and California Collaborative for Educational Excellence (CCEE) in 2015, by collecting documentation (e.g., board meeting video, CORE’s NCLB waiver application, LCFF policy documents about CCEE, and training guidelines) and conducting informative interviews with leaders. This study focuses on the period from 2016 to 2018, during which I conducted 153 hours of observation and 25 interviews (including think-aloud interview data from the subset of individuals involved in district-level CI efforts). I collected hundreds of pages of documentation and approximately 50 photographs of artifacts.
Interviews
I conducted formal, semistructured interviews with key informants, including organizational leaders in CORE and CCEE (n = 2, for each organization), district administrators in Manzanita (n = 6) and Sage (n = 6), and school administrators in Manzanita (n = 4) and Sage (n = 6). I identified district participants as those most directly involved with designing and implementing the district’s CI process (the superintendent, cabinet, and principals in one district; senior district leaders, administrators, and school leaders in the other district), and 100% of identified respondents agreed to participate. These interviews asked about participants’ beliefs, understandings, and perspectives of CI activities. They were also asked about the capacity building support they received, their engagement in the network, and how their CI process was created and implemented.
In addition, I utilized a think-aloud protocol to enhance my interview data (Chipman et al., 2000). For this portion of the interview, I provided all district staff directly involved in designing and implementing the CI (n = 6 in Manzanita and n = 8 in Sage) a prompt containing a plausible performance problem. The prompt was similar across the two cases, differing only by the level of the student population in question (because the district teams differed in their focus). For example, for Sage, I provided the following prompt: Let’s imagine that you learn from a colleague that reclassified English Language Learners in your district exhibit disproportionately lower GPA, compared to other students, despite similar test scores.
I then asked the respondent to describe how they would proceed, probing with questions regarding how they would respond to this information, what kinds of information they would seek out, what tools they might use, and where they might go for advice, support, or information. The responses help to demonstrate (1) to what extent respondents call upon CI processes versus traditional routines or innovative means when asked to consider a novel problem of practice, (2) who respondents include in the conversation and where they go for advice or solutions, and (3) what kinds of evidence and information they draw on. This hypothetical problem of practice also alluded to a potential equity concern, inviting opportunities to discuss their perceptions regarding equity and strategies to address inequity. I asked about the theoretical problem of practice at the end of a comprehensive interview discussing the district’s adoption and use of CI and their most recent CI cycle. With this preparation, I expected that respondents might be more likely to align their responses to the district’s CI approach; in practice, there was broad variation in responses despite the potential for priming.
Observations
Overt, nonparticipant observations during meetings and training sessions helped illuminate the implementation of both CI processes and accompanying capacity-building efforts. I observed and collected detailed field notes (i.e., near-verbatim notes, along with a description of the setting as well as nonverbal gestures and expressions) of all CORE network meetings and training (n = 49 hours); a selection of board meetings that specifically covered CI adoption and use (n = 16 hours); all CCEE network meetings and training (n = 51 hours); a selection of Manzanita CI activities, including planning meetings and a PDSA cycle working session (n = 17 hours); and a selection of Sage CI activities, including board meetings regarding CI adoption and a needs-analysis visit (n = 20 hours). These observations provided valuable information regarding CI practice and implementation.
Documents and Artifacts
During observations, I collected hundreds of pages of documentation and about 50 photographs of physical artifacts, such as fishbone diagrams, driver diagrams, and photographs of group notes (i.e., charting) completed during network meetings. These documents and artifacts helped to capture the language of CI and provided additional depth and specificity to observation notes about the CI process.
Analysis
I cyclically analyzed case data through coding, within-case narratives, and cross-case analysis using case-ordered meta-matrices and theoretically informed visual data displays (Bush-Mecenas & Marsh, 2018; Miles et al., 2013). My first phase of analysis was iterative, utilizing both deductive and inductive techniques. I began by organizing all transcribed interviews and observation field notes in NVivo qualitative research and coding software. All coding was completed within NVivo to ensure data remain organized and to facilitate coding comparison. Initially, I coded all interviews, observation field notes, and relevant documents according to sets of descriptive codes (case characteristics, intervention design), thematic codes (implementation, organizational characteristics, elements of the racial equity logic, and the performance logic), and analytic codes (institutional logics and modes of reproduction; see Supplemental Appendix A, available in the online version of the journal). To ensure reliability, I compared my coding biweekly by recoding 10% of interview transcripts and observation notes, comparing reliability over time and consistently met a minimum threshold of 90% agreement.
I conducted three analyses to answer my research questions. To answer the first research question (i.e., how do educational organizations structure their CI approach to promote racial justice?), I drew on interview (excluding think-aloud prompts), observation, and document data. I analyzed the descriptive coded data and created case reports using a standard outline. These case reports allowed me to catalog and compare elements of the CI processes at the intermediary and district level in each case. I utilized triangulation of the different data types and sought data saturation as a heuristic for creating a coherent narrative of CI design and structure. Findings are presented in cohesive narratives for each case.
To answer the second research question (i.e., what mechanisms allow districts and intermediary organizations to combine the guiding principles of performance and racial justice in using CI?), I drew on all interview, observation, and document data, analytically coded across the seven modes of reproduction (identified by Anderson & Colyvas, 2021) involved in the institutionalization. I looked for references coded at these modes of reproduction that was also coded at both the performance and racial equity logics. I utilized case-ordered metamatrix analysis to analyze my coded data (Miles et al., 2013; see Supplemental Appendix B, available in the online version of the journal). The matrix allowed me to unearth patterns across individual responses and cases and to triangulate across data sources and informants to enhance validity. The findings are presented in narrative form.
To answer the third research question (i.e., to what extent do actors invoke the guiding principles of performance versus racial equity in their discussion of CI efforts?), I focused exclusively on the interview responses to think-aloud prompts to provide bounded, consistent references for analysis. I coded think-aloud prompt responses at the sentence level according to the performance and racial equity logics defined by a set of dichotomous characteristics representing the guiding principles of each education policy order (i.e., quantification, standardization, and accountability; equality, equity, and racial justice). Specifically, I operationalized performance logic as valuing standardization, relying on data, assessment, and measurement, and utilizing incentives and sanctions as accountability tools. To operationalize the racial equity logic in education, I utilized a practitioner-facing conceptualization of the beliefs and practices that engender equity developed by Gorski and Swalwell (2015; Gorski, 2017), including rejecting deficit-orientations (equality), prioritizing the interests of marginalized communities and engaging in redistributive practices (equity), and engaging in confrontation with inequity and questioning bias embedded in systemic and social systems (justice).
I examined the prevalence of words coded in each dichotomous code (see Supplemental Appendix C, available in the online version of the journal) as well as the qualities of what was said (see Supplemental Appendix B, available in the online version of the journal). For each of these dichotomous codes, the prevalence of coding led to a score of 0 (i.e., no coded material on either side of the dichotomous code or equal quantities of coded material on each side), −1 (more material coded under the “low” side of the construct; e.g., “deficit oriented”), or +1 (more material coded at the “high” side of the construct; e.g., “strengths oriented”). These code scores were summed for each of the logics, leading to a racial equity logic score ranging from −3 to +3 and a performance logic score ranging from −3 to +3. These scores were then transposed onto a grid, seen in Figure 2. In addition to considering the prevalence of coded material, the discussion draws on thematic and narrative analysis of the content and quality of coded material (as seen in Supplemental Appendix B, available in the online version of the journal).
Visual display of the prevalence of think-aloud responses. 5
Limitations
Case study is a particularly useful tool for developing theoretical propositions and local causality because of the emphasis on analytical generalizability. I sought to strengthen the validity of findings in several ways. First, I triangulated data, wherever possible, drawing on multiple measures of the same phenomenon, and ensured a clear “chain of evidence” from data to findings (Yin, 2013, p. 186). Throughout my analysis, I engaged in memoing aimed at exploring potential interpretations of my data, such as alternative theoretical explanations, implementation issues, and trustworthiness, to determine if findings are adequately robust. Because my data were coded in NVivo software and arranged in several matrices, I was able to further analyze patterns in my data that might support or contradict the key findings.
Despite these efforts, there are several limitations to this study. I was limited to a small sample of two districts and the early period of policy design and initial implementation (2016–2018). My findings are constrained to early, volunteer efforts to implement CI. Because I focus on the nascence of these CI efforts, findings would likely vary for organizations with more established CI practices. As such, my conclusions must be understood as exploratory concerning the processes of adoption, design, and early implementation of CI. My case districts also volunteered to participate in capacity building and CI efforts with CORE and CCEE; therefore, they should not be compared with cases of mandatory CI implementation, as such efforts proliferate and become codified in policy.
Findings
I present the findings organized by the research questions and related elements of the theoretical framework. I begin by describing how these nested cases structured their CI approaches to promote equity, referencing the design and implementation of CI. I find that the case organizations varied substantially in their levels of formality and discipline, as well as their local contexts. Next, I identify the key modes of reproduction—performance metrics, routines and tools, and professional norms—that appeared to function in combining performance and racial equity logics. Third, I discuss the extent to which CI team members in districts invoked the equity and performance logics in their response to structured inquiry about CI, to understand early institutionalization at the micro-level. Although individual respondents varied in the extent to which they endorsed the guiding principles of performance versus equity, almost all respondents invoked aspects of both in their description of CI work. On the whole, these findings illustrate the potential for using CI to promote equity and the structural and individual challenges in doing so.
How do Educational Organizations Structure Their CI Approach to Promote Racial Justice?
This multileveled case study analysis examined two intermediary organizations promoting the use of CI and two nested districts using locally designed CI processes. For all four organizations, the primary goal of equity (i.e., closing opportunity gaps) was combined with the key technology of CI (i.e., root cause/causal system analysis and PDSA cycles). These organizations varied, however, in the organization and structure of their CI processes. I first sought to understand how CI was used to promote equity in these organizations, in practice, including its structures, methods, and approach. Next, I briefly describe these organizations’ CI processes and intents.
Intermediary Organization Cases
The two case intermediary organizations, CORE and CCEE, were independent from one another, even as they fulfilled the same mission: to build the capacity of districts to utilize CI to promote equity, efficiency, and effectiveness. These two intermediary organizations developed very similar organizational structures and aims, along an almost identical timeline. 4 In this way, these two intermediary organizations presented the opportunity for a complementary analysis of two different approaches to utilizing CI. While these intermediaries looked very similar as organizations and support providers, CORE espoused a highly structured Improvement Science approach, while CCEE encouraged local design of CI.
CORE
In 2011, the Obama administration offered waivers from the stringent performance-based accountability system of NCLB for states that promoted particular educational reforms (ESEA Section 9401). In response, six California districts formed a consortium with the nonprofit CORE to meet the federal waiver requirements. Together, the six CORE districts served more than one million students. Essentially, these superintendents came together to form a board, which hired and oversaw a small group of experienced educational administrators (mostly from districts) with a strong vision, commitment to equity, and proficiency in working with data. CORE initially focused on the creation of a comprehensive accountability system, including many unique metrics for student learning and school quality (e.g., surveys measuring socioemotional learning, classroom observation), along with enhanced instructional support around the Common Core curriculum, use of multiple-measure teacher evaluation reforms, and the use of peer-to-peer networks (school-to-school, role-alike) as a vehicle for capacity-building through the sharing of ideas and insights.
With the passage of ESSA, the CORE collaborative was presented with the opportunity to reflect on their work and decide whether to continue collaborating and how to reenvision their agenda. In the words of one CORE staff members, Improvement Science was adopted to help districts, “work to move the needle on a big goal [closing the achievement gap], but what it boils down to is what types of things could we do on a small cycle in a PDSA.” In this iteration, CORE brought together district administrators (of now 10 districts) to promote CI through facilitating an interorganizational network, or Networked Improvement Community (NIC), and training district teams in Improvement Science (the CI process described in Bryk et al., 2015, and promoted by the Carnegie Foundation), facilitating localized interventions around shared problems of practice.
Improvement Science is defined by six principles: (1) identify specific problems and focus on key participants; (2) attend to variation in performance (what works, for whom, under what set of conditions); (3) reflect on the existing system that is designed to produce current outcomes; (4) measure processes and outcomes to assess the efficacy of strategies; (5) utilize rapid PDSA cycles to promote quick improvement; and (6) draw on collaboration through networked communities to accelerate growth (Bryk et al., 2015). NICs, or facilitated interorganizational networks designed to provide accountability and knowledge-sharing opportunities to participating organizations, are the vehicle for Improvement Science, along with a set of tools and processes (e.g., PDSA cycles, fishbone, or Ishikawa diagrams). CORE trained their board and the network of member district administration in the key principles of improvement and Improvement Science methodology. As their partnership increased their focus on Improvement Science, CORE also utilized capacity-building support from the Carnegie Foundation. One CORE leader described the connection between Improvement Science and the opportunity gap in the following way: Those [equity] issues are system issues, like human capital. So how do we engage in that conversation? That’s what is so attractive about engaging in the next generation of PDSA cycles and Improvement Science. We can use this . . . clearly, this [the opportunity gap] isn’t just a socioeconomic issue, but it’s other issues that come with race.
As CORE utilized Improvement Science in board meetings, conversations centered on examining data to surface equity challenges. Through this process, leaders communicated clear stances regarding equity, including “It’s really important that we consider that the opportunity gap is really about race” and “indicative of systemic gaps.”
CCEE
In 2013, the California Legislature adopted the Local Control Funding Formula (LCFF), which dramatically changed school funding and accountability in the state. This new law accomplished the following: (1) allocated funding along a weighted-student formula, providing additional funding to students with greater needs; (2) required schools and districts to convene stakeholder committees to determine funding priorities and provided these committees considerable autonomy in funding allocations; and (3) promoted accountability for expenditures through transparency by requiring that district complete a Local Control Accountability Plan listing their expenditures alongside expected outcomes. The CCEE was created to “provide advice and assistance to LEAs (charter schools, school districts, and county offices of education) in achieving the goals outlined in the Local Control and Accountability Plans (LCAPs)” (Education Code Section 52074). CCEE leadership included a state-appointed board (membership was stipulated in the law), governing a small, flat organization responsible for supporting counties and districts. Early on, the CCEE decided to recruit a set of volunteer districts and counties to engage in a pilot program to help develop their system of support. These pilot districts received tailored support in CI as well as funding to support their improvement efforts from a set of coaches, including former district superintendents and educational consultants.
CCEE’s adoption of CI differed substantially from that of CORE. CI was introduced gradually, and CCEE did not advocate a single CI methodology. At one convening focused on introducing participants to CI, a staff member opened the session by saying: “This [Improvement Science] will be work done with you, not to you.” CCEE staff prided themselves on taking a locally oriented approach to reform, engaging as equal partners with the districts they supported and promoting local ownership of reform efforts. In one initial CI meeting, participants noted that the business-oriented language in their selected model was “rubbing us all the wrong way, [with words] like ‘problem’, ‘defect’, ‘solution.’” Indeed, a CCEE facilitator initially observed that CI was “a huge industry,” acknowledging the potential concerns among participants regarding the fit of CI to the educational context.
While they expressed less concern about the particular model of CI in use, CCEE sought to build district capacity to use CI to make decisions that would be more equity-focused, evidence-based, and with greater stakeholder engagement (referred to as decision making “the California way”) to improve student outcomes. Pilot districts were asked to design local CI processes that typically included identifying problems, reviewing data, determining root causes (often with fishbone diagrams though not required), and completing PDSA cycles to test solutions.
District Cases
The district cases offered the opportunity for a second complementary analysis of differing CI approaches. Manzanita, with over 50,000 students, is a typical large district in California with a majority of Latinx students and over three quarters eligible for Free or Reduced-Priced Lunch. Sage Unified has very similar demographics, with around 3,000 students—reflecting the average sized district in the state. The two districts developed differing, locally informed approaches to CI. While Manzanita utilized a disciplined Improvement Science approach, Sage engaged local administrators in developing a unique data-informed CI process. The differences in district size and approach provides useful insight into how CI approaches might play out across varied contexts.
Manzanita Unified
Manzanita Unified School District is a large urban school district with a history of stable leadership and staffing and a highly developed central office. This district has positioned itself at the forefront of data use, developing sophisticated data and accountability systems. Supplementing their CORE involvement, multiple streams of Improvement Science work exist within the district. Over a few years, several Manzanita staff had received training through the Carnegie Foundation’s annual Summit on Improvement in Education, learning the Improvement Science approach. I examined Manzanita’s most advanced stream of work, which resided within a department that manages data and accountability as well as equity-focused work around student pathways (i.e., course-taking, performance trajectories, postsecondary access, and attainment). Within this department (and alongside a set of school-level administrators directly involved with change idea implementation), administrators utilized a highly structured CI process following the principles of improvement science. In practice, this included a documented set of routines, including identifying an equity-focused problem of practice, examining root causes using a fishbone diagram, and selecting and testing a change idea through PDSA cycles. These quarterly cycles occurred on a set of related problems consecutively, returning to the same problem of practice and change ideas (or modifications) annually.
Manzanita CI team members described how they came to their particular perspective on equity, one informed heavily by the concepts of racial justice (e.g., adjusting systems to promote equitable opportunities and outcomes). One team member described his leader sharing the following response to setting data-based goals, We were building logic models, and setting outcomes and expectations, and for FAFSA he asked us “What is a reasonable expectation? What percentage?” We all said, “about 90%,” and he said, “What about the other 10% of the kids? Are you going to tell them that they don’t deserve to have a financial aid packet?” It totally changed my view of our work.
In this example, the leader set expectations for using data and accountability in ways that support equity. A CI team leader further described the value of lending the power of quantification and accountability to efforts to promote equity. In his words, there was no accountability towards that [equity] principle and it’s just something that people were starting to believe in and talk about, but there was no element of accountability associated to it. When CORE comes around, then I was finally able to tie our work to a framework of accountability.
He described in specific terms how the CI approach, with its reliance on the practices of quantification and accountability, helped his team to pursue equity: We have a lot of stakeholders that . . . blame external forces for why we can’t impact certain datasets [showing the achievement gap] and stuff like that. They’ll negotiate their measure of success. They’ll say, “Yeah, but you know one of the foster kids that I worked with passed the AP because I went to his home and I visited him and I came on Saturdays.” That’s the behavior we would see, people would just come up with anecdotal stories and that’s how in their mind they say, “But I am successful,” you know? Obviously, for us it was: you’ve got to come closer to your impacts, it’s not just one or two [students], it’s everybody and you’ve got to change conditions in the present and we’ll articulate measures of success with you. If 70% of our students are passing the class but not passing the test, then that’s a measure of success that we’re talking about—not your success with one or two students.
As this example demonstrates, the use of CI helped to bolster existing district goals of closing opportunity gaps and disrupting unjust systems, specifically by utilizing data, metrics, and accountability to question assumptions and entrenched responses to problems of inequity.
Sage Unified
Sage Unified School District is a smaller, rural district, with robust and long-standing community partnership but with challenges in limited human capital supply and less developed data systems. As part of their CCEE activities, the superintendent of Sage sought to infuse the district with data-centered decision-making processes and a focus on equity. Both goals represented a substantive shift for district administrators and school staff. In developing this local equity-oriented, data-driven CI process, the superintendent led all five Sage school principals in reading several texts outlining procedures for using data, through PDSA cycles, to solve problems of practice. The principals then initiated the creation of a set of documents and tools to guide district- and school-level staff in identifying problems of practice and root causes, testing assumptions and predictions utilizing various data sources, and completing PDSA cycles. These processes were informally documented, with little to no oversight over school-level CI processes.
In this initial phase, the CI process was focused on addressing concerns around attendance and suspension rates, with a focus on ensuring equitable access to instruction for all students. A leader at Sage sought to utilize CI processes to expose and address inequity in the district as well as to start conversations about racial injustice among staff and stakeholders. Reflecting on these goals, this leader shared, Why we’re here is to build and provide the best educational program for all kids in [this region]. . . . If you asked them, “Is this [policy or practice benefitting] for some kids or all kids?” They would say, “Of course, it’s for all kids,” but their actions and their words don’t align. Their actions really talk about elitism and special interests and the people that have been here the longest.
Observations of board meetings during early discussions about adopting CI and participating with CCEE evidenced some of these challenges. As requests were brought to the board regarding modifying policies requiring participants in some sports or clubs to pay fees or providing shade structures to some school campuses, conversations about the implications for equity ensued with the leader pushing the underlying assumptions of these policies. For this district, CI presented a local-driven improvement process that could shift beliefs, policy, and practice to promote equity.
What Mechanisms Allow Districts and Intermediary Organizations to Combine the Guiding Principles of Performance and Racial Justice in Using CI?
The descriptive analysis of these CI efforts indicates that the organizations intended to promote equity and utilized varied structures to achieve this. Next, I sought to understand how, within this design, performance and racial equity logics came together. Three modes of reproduction appeared to function most prominently in my cases to combine and institutionalize the equity and performance logics (with aspects of both logics appearing in all coded references): performance metrics, organizational routines and tools, and professional norms. Of note, the examples provided often combine aspects of these mechanisms (e.g., using a new organizational routine that involves different kinds of performance metrics, with new norms around interpretation).
Performance Metrics
This mode of reproduction describes the metrics designed to assess progress and outcomes, which function as a mode of reproduction in how they “tend to focus attention and resources narrowly on what is measured and reify the values embedded in those measurements” (Anderson & Colyvas, 2021, p. 12). While the focus on utilizing metrics and data in CI aligns with the elemental properties of the performance logic, my case data indicate that the kinds of metrics that were selected and the way they were used functioned to bolster an equity focus (22 excerpts coded for metrics, equity, and performance).
The recent CI movement in education typically encourages educational organizations to draw on a range of data, including both qualitative and quantitative sources. CI typically engages educational organizations in identifying both leading (input and process-oriented) and lagging (outcome-oriented) indicators. This process represents a stark departure from accountability policies relying solely on standardized tests and measures of student outcomes. For example, early on, the CORE collaborative created a dashboard accountability system that sought to provide a range of school quality indicators provided as a suite rather than a summative measure. The idea was that this dashboard would provide stakeholders a more holistic picture of school performance. Furthermore, the CORE districts designed nontraditional standards of school quality, such as using student and teacher-reported survey data to measure socioemotional learning and school culture and climate, and creating a high school readiness measure (see Marsh et al., 2017). Of particular value, the CORE districts also decreased the threshold to define a subgroup for disaggregating data. In doing so, CORE directed attention to the experiences, opportunities, and achievement of BIPOC students, even if their populations were relatively small in the district or school. In the words of one CORE participant, Our teachers and principals sit in the intersection of race and class every day. . . . I do feel that we have an important role to move it forward. When we decided to bring n size [threshold number of students to constitute a subgroup for analysis] down to 20 and looking at disproportionality, we can be proud of that. We can now systemically measure the fact that our Black boys in particular are being treated differently, systemically. . . . So it’s about telling everyone across the line that this is the intersection of race and poverty, and we need to be the one to actually say, “Not in my school.” No one else wants to do this.
Changing metrics and criteria in thoughtful ways functioned to change where attention was directed, presenting potential for more informed resource distribution.
The act of engaging a range of actors from across the educational organization in defining and analyzing metrics provided opportunities to shift attention, resources, and shared values. For example, a Sage Unified principal shared her experience of engaging teachers in identifying and analyzing metrics to test their predictions and assumptions regarding student attendance problems (documentation of this process was also reviewed). Sage’s principal shared that, when she asked teachers to predict why attendance rates were low, they typically identified value-laden, deficit-oriented perspectives on parent and student behavior, such as parents of special education students think that their children do not need an education, parents are taking students on long weekend trips to Disneyland because they don’t appreciate the importance of school, or students who live close to campus are sleeping in because they don’t value education. The principal then led teachers in disaggregating the data by day of the week, proximity to school, and special education status and, subsequently, interviewing frequently absent students to gain more information about why they were not attending.
Teachers were intrigued, she said, to discover that Wednesdays had the lowest attendance because these were shortened days (to allow for afternoon teacher professional development), and that students who lived far from campus were more likely to miss school due to transportation issues (e.g., commuting 2 hours to attend just 3 hours of school presented challenges to families and/or mid-day pickup was incompatible with parents’ work schedules). By promoting the identification and analysis of alternative metrics, teachers’ beliefs about students’ and their families’ values and behavior were shaken, and an opening for meaningful dialogue about equity was created.
In practice, this school CI team responded to the urgency of addressing attendance, but rushed to adopt more traditional incentives for attendance rather than adjust systems to meet students’ transportation needs. While they departed from the CI process at this point, respondents believed that the root cause analysis had helped to challenge teachers’ assumptions and might have longer term influence on teaching practice.
Routines and Tools
Routines are defined as recurrent, semistable, collective patterns of behavior that function to codify and store knowledge, coordinate activities, reduce uncertainty, and promote stability (Becker, 2004; Spillane et al., 2011). I also include tools as a component of this mode of reproduction, as CI typically relies on scripted routines with accompanying tools (intended for documentation and to ensure discipline or fidelity). As noted earlier, CORE, CCEE, Manzanita, and Sage embraced key CI routines, including the process of root cause analysis using fishbone diagrams, causal analysis using driver diagrams, and change idea development and testing through PDSA cycles (12 excerpts coded for routines, equity, and performance). Predictably, given the variation in the CORE and CCEE approaches to discipline in CI, the Manzanita Unified routines were highly specified, documented for evaluation, and disciplined. In contrast, the Sage Unified routines were more flexible and documented only to focus on progress.
Interestingly, both Manzanita and Sage customized their utilization of the CI root cause analysis process in ways intended to promote equity. Manzanita and Sage did so through the use of additional formal and informal routines, respectively. Specifically, the Manzanita team designed a set of “filters” to apply to their root cause analysis and fishbone diagram (supporting data include interviews, observations of root cause analysis, and fishbone diagram documentation). They identified two potential mindsets around problem solving: the mindset of influence (i.e., intractable external forces are responsible for current performance, over which actors can have minimal/moderate influence) versus the mindset of impact (i.e., actors can change conditions in the present to impact performance through targeted intervention).
After identifying a problem of practice (e.g., students not applying to a broad range of colleges), the Manzanita CI team, including school-level administrators, brainstormed all possible root causes. Any suggested root causes were written onto the fishbone diagram and documented. Then, the team applied the impact and influence filter, essentially crossing out any root causes over which the educational system would not have control. In practice, this often meant crossing out a deficit-oriented perspective. One Manzanita administrator described how this process played out: So, lack of motivation to apply was one of the reasons why that our counselors identified [for students not applying to a broad range of colleges]. But then . . . we kind of said, “Now go through impact versus influence, and take out those root causes that are more about, that are student centric or that are about external forces.” So, lack of motivation to apply is a legitimate root cause, but we’ve operated on that front forever and ever. That’s why we have this problem . . . If you want to lead “lack of motivation to apply,” you’re probably not going to succeed. I mean, you won’t have the resources to scale whatever change idea you try to implement, to try to change how students think or whether they are motivated. Or you’ll say, “Why don’t we start with our TK [transitional kindergarten]?” Right? And we won’t know whether it worked until 13 years from now or something.
While accepting this root cause as potentially reasonable or “legitimate,” they also noted that it was intractable or required long-term or large-scale solutions.
As a result, the Manzanita CI team identified different root causes: the lack of familiarity among counselors, alternatives to the local university, and lack of available information regarding student college eligibility. One administrator noted that the college application counseling process, relied on counselors a lot of times just giving their personal experience. A lot of them are local, and a lot of them went to [the local university]. Their personal experience would lend them to steer kids towards [the local university], which again isn’t a bad thing, but these kids have stronger academic profiles.
Sage Unified took a different approach to promote equity in the root cause analysis routine. Here, the superintendent informally created a filter routine by telling participants that the only acceptable root cause ideas concerned the system, rather than the students, their families, or their communities. In his words, “we control the conditions for success—period!” As principals utilized the CI process with their teachers, they saw the influence on the equity-orientations of the school staff. One principal described how the process of identifying root causes for high suspension rates at school “got heated” and resulted in new, uncomfortable, but productive conversations regarding student success. As she described, It becomes a pretty tense conversation when they [teachers] have to realize it’s their responsibility [to solve school problems of practice, including behavior issues and high suspension rates]. And that’s one thing I like about this [CI] process that we never had before. Because it was always the blame game, previously, it seemed like nobody ever took responsibility for their teaching, their instruction . . . I taught for 14 years, and I remember sitting in these groups, and it was always something else: blame it on their family, their home life . . . never what’s going on right here, right now in the classroom. So, I really like that this is breaking it all the way down.
These examples demonstrate how the CI approach, and in particular the modifications designed ad hoc by organization leaders, involved using routines that exemplified the data-centered, scientifically informed approach of the performance logic as a tool to promote the racial equity logic. In the words of one leader, these routines led to the identification of “honest whys” regarding equity challenges.
Of note, the routines and tools adopted appeared to be successful, in part, because they were well-suited to local context. For example, Manzanita created a set of online forms to structure their CI process because the district was already very sophisticated in its use of data and technology. In contrast, the CI process in Sage was designed locally to involve a printed folder filled with handouts. This format was more accessible to school staff and avoided challenges associated with adopting new technology.
Professional Norms
Professional norms can be understood as unspoken rules governing behavior, typically communicated either as common behavior (descriptive) or behavior viewed as appropriate (prescriptive; Dannals & Miller, 2017). As a mode of reproduction, norms can be enacted and reproduced across organizations, professional training, and professional communities (W. R. Scott, 2013) and can function to bolster the legitimacy and taken-for-grantedness of values, beliefs, and practices. While observed throughout the cases (10 excerpts coded for norms, equity, and performance), a particularly interesting example can be drawn from observations of CORE board meetings. I began these observations prior to the use of and training in CI and noted a fascinating shift in norms over time.
In a 2015 CORE board meeting, the CORE district superintendents were presented with a scatterplot showing the performance of schools on measures of academic performance against socio-emotional learning metrics. This exercise led to an in-depth conversation among board members about how they made sense of the data. When the CORE staff member pointed out that two schools in the same district were doing similarly academically but had very different socioemotional index scores, that district’s superintendent immediately tried to identify the names of the schools and share the contextual factors that might explain this relationship. Board members then discussed their concerns over the validity of socioemotional learning metrics (particularly suspension rates, for which reporting varied widely). One board member responded to concerns over the validity of suspension data, by pointing out that family characteristics might explain differences in achievement among schools with similar student populations, saying, That comparison [between schools with similar populations] in itself isn’t true. How did the kids get into the school? . . . Which poor kids did you get? That matters. That’s contextual. How so? If a poor kid was in a family like mine, being raised like my mother, they’re communicating that education is important, and I need to find a school with opportunities for my kid. If the poor kid is raised by a parent like my aunt, there’s no conversation about the school. You’re going to go to the school down the street because that’s closest.
This type of conversation reflected a relatively superficial response to the equity concerns raised by these data—rationalizing by identifying hypothetical contextual explanations and questioning the validity of metrics. This exercise, however, was intended to introduce the board to the act of collaboratively examining equity concerns through CI.
In a board meeting 1 year later (observation field notes from 2016), it was evident that the superintendents had enacted and were reproducing new norms in examining equity concerns raised in response to data. As the board began to again review data visualizations demonstrating achievement gaps in mathematics, a CORE staff member summarized the data by saying, “We find poor Whites outperform non-poor African-Americans. I point this out because we’ve received pushback, why not all kids? Or this is an issue of poverty. Well, the data shows . . . that’s not true.” Board members responded by acknowledging their surprise and the power of identifying these achievement gaps within their district data. As one board member stated, It’s staggering to see it. When this is representative of our district right here and not just a research piece, to see it [the achievement gap] demonstrated that poverty, we use that as the reason, but that’s not the reason. We need to think about the other factors that contribute to the achievement gap.
Another board member agreed enthusiastically, “There’s something else going on, right? Clearly!” The board member endorsed CI as a useful methodology for exploring these equity issues, saying, You cannot leave it up to the school—those issues are system issues . . . So clearly, this isn’t just a socioeconomic issue, but it’s other issues that come with race.
After just 1 year of using CI, these board members demonstrated a marked difference in the enacted norms in discussing difficult issues of equity. One board leader shared that these norms were also becoming apparent within their organizations. As he shared, Our teachers and principals sit at the intersection of race and class every day. . . . We can now systemically measure the fact that our Black boys in particular are being treated differently, systemically. . . . So, it’s about telling everyone across the line that this is the intersection of race and poverty, and we need to be the one to actually say, “Not in my school.”
The differences in the discussions and responses to data in this example illustrated the ways in which changes in norms showed up. Prior norms of acceptable behavior allowed leaders to attribute achievement gaps to poverty or deficit-oriented perspectives on specific communities and students and to potentially rationalize patterns exhibited in data. Even as the data remained unchanged, new norms encouraged leaders to accept evidence of a racial opportunity gap and seek systemic solutions. In essence, the board’s year of training in CI appeared to promote a more open and thoughtful approach to understanding perennial gaps in student performance evident in their data. Indeed, these shifts in discussion evidenced key principles of Improvement Science, such as attending to variation in performance and questioning system-level influences on performance.
In another example (briefly discussed in reference to new routines and tools above), explicit shifts in norms in the context of CI had the potential to influence which choices and solutions were available to CI team members. One Manzanita administrator described the change in mindset they observed as a result of CI: When you get to that part of the fishbone [diagram], I think that naturally some of us are inclined to say, “Oh, no. Definitely. That’s [problem of practice is] not my fault. I definitely can’t [change it].” But it’s this shift in mindset that you have to have when you think about our theory of change, our guiding principle. What we believe is equity and access. What we stand for. There aren’t any excuses allowed. We understand that we all signed up for this, to support students, to improve student outcomes.
Once again, shifting explicit professional norms in CI, as a mode of reproduction, changed how actors interpreted data, brainstormed root causes, and the kinds of change ideas, solutions, and practices available to them.
To What Extent do Actors Invoke the Guiding Principles of Performance Versus Racial Justice in the Structured, Hypothetical Discussion of CI Efforts?
Analysis of how the case organizations structured and operationalized CI efforts demonstrated intent and potential to meaningfully address inequity. Institutionalization, however, relies on both structure and agency, at the meso- and micro-levels. This analysis of think-aloud responses showed that most respondents identified CI processes as their chosen method to address the hypothetical problem of practice, and most invoked elements of both performance and racial equity (Figure 2). There was substantially variation, however, in the extent to which they invoked the guiding principles of racial equity versus performance, outside of a structured, facilitated CI process.
On the whole, seven of 14 participants responded to the hypothetical problem of practice by describing specific CI methods by name. For example, one Manzanita respondent stated, I think we’d want to do the deeper dive—the causal system analysis, the fish bone diagram—just for our own practice because, again, we may know of it [ELD] but we’re not the experts but I definitely don’t want to show up empty handed when I reach out to . . . [the ELD department leader]. I’d want her to feel that we are adequately attuned with a specific designated population . . . and then do a full-on fish bone diagram, deeper dive with her and her team because she can . . . probably attest more to the true root causes the are more applicable in her day-to-day work.
Another three described CI-like processes without the associated terminology. These respondents noted that they would respond to the hypothetical problem of practice by examining data, most often traditional measures, such as test scores and attendance ranges, but also sometimes additional sources of data, such as the distribution of students across classes and schools or even interviews with teachers or students.
Across the 14 participants, most invoked aspects of both the racial equity logic and the performance logic, with just one respondent showing a performance-only orientation. Illustrating the combination of logics, one Sage administrator (represented in the top right quadrant in Figure 2) responded to the prompt: Well, the first thing that I would do is . . . to verify the assumption with some data. Then really, the next part would be beginning the analysis of why. [In a similar situation in my last school district], what we found out was absolutely, positively without a doubt, the issues were about cultural biases that the teachers held. Well, that the staff as a whole held, not just teachers. That were almost unconscious.
While discussing the performance-oriented structures of CI (utilizing quantitative student performance data), this respondent immediately identified a potential root cause grounded in implicit bias. This respondent further suggested that teachers’ implicit bias against Latinx students was influencing their grading, resulting in inconsistency between students’ test performance and GPA.
Despite consistent discussion of CI methods and invocation of performance and racial equity, there was substantial variation in individual embrace of each logic. As Figure 2 demonstrates, there was moderate engagement of the performance logic on the whole with almost all cases falling within one step from the central axis. Most of these responses invoked performance in terms of quantification, drawing on already institutionalized practices such as data analysis procedures and practices. A few respondents described using performance data to confirm the problem of practice (e.g., “I definitely would work with our EL [English Learner] department, run the data to verify that it [the hypothetical problem of practice] is in fact true, which I don’t think it’s a hypothetical. I think that is a fact.”)
About half of the respondents also mentioned using qualitative forms of data (which are not coded for quantification), including empathy interviews with teachers, students, and/or families regarding potential reasons for student performance problems in class and particulars of instruction including grading practices. As we might expect with implementation of CI, almost all respondents discussed utilizing quantitative data and about half discussed using qualitative data to understand the problem and its root causes.
A little over half of the respondents discussed efforts to personalize supports for students, though just two respondents (both in Sage) invoked elements of standardization in their proposed solutions: namely, ensuring faithful implementation of existing district-wide approaches to ELL instruction and ensuring students were involved in existing EL intervention programs. Just one respondent described accountability based on sanctions and incentives, but some Manzanita staff did call upon professional forms of accountability enacted through their NICs. These findings are expected as they evidence how CI draws heavily about quantification (from the performance logic) paired with the use of qualitative data, while encouraging participants to draw on moral and professional accountability (over incentives and sanctions) and vary solutions to local needs (over standardizing).
Integration of the racial equity logic appeared more varied, with scores ranging to three steps from the central axis in either direction and a great deal of variation. A little over half of the participants described how data might surface evidence of teacher bias, influencing grading practices in ways that led to the hypothetical problem (i.e., reclassified EL students earning a lower grade point average while their standardized test scores were commensurate to peers). A few respondents with more equity-oriented perspectives suggested looking to the data to identify any variation in instructional quality across the district and to identify any possible barriers to achievement that can be remedied on a systemic level. Their approach to the equity concerns embedded in the problem was one of acceptance and responsibility (i.e., acknowledging the equity concern and need to fix it at a system level).
A little less than half of the participants, however, described deficit-oriented potential causes for the problem of practice. For example, one respondent suggested that the hypothetical students lost their language skills during extended trips back to Mexico, while another asserted that “our ELL kids don’t have a desire to go to a university, they’re about the workforce.” One of these respondents also suggested that students might be struggling because, We have some teachers where English is not their first language, so they have a heavy accent and sometimes my English-only kids might struggle with those teachers. But if English is your second language, you might really struggle with those teachers.
A few respondents stated that hypothetical students might not be doing their homework and suggested that these students attend an extra before-school study-hour.
While most respondents consistently evidenced a moderate level of data use (and to some extent standardization) central to the performance logic, the conclusions that they drew from data exhibited differences in the level to which they had adopted the racial equity logic individually and as an organization, despite both districts asserting their goal as improving equity. On the whole, the qualitative data show how individuals using CI procedures may have integrated some elements of performance and racial equity logics into their practice, as they evidenced moderate levels of each logic. Very few respondents evidenced stark divisions between performance and racial equity (far top left or far bottom right). Nonetheless, these think-aloud responses also show that few respondents strongly endorsed both principles of racial equity and performance concurrently (top right quadrant). Interestingly, some respondents appeared to strongly invoke racial equity logics while less strongly bringing in performance principles of standardization, accountability, and data use (seen in bottom right quadrant). This may be a function of particular elements of performance-based logic, such as data-use, being evoked more strongly in the context of CI than other elements like standardization. Furthermore, approaches to equity varied widely and evidenced the challenges of changing the essential beliefs, biases, and assumptions versus structures.
Discussion and Implications for Policy and Practice
The most recent stream of CI reforms has purposefully sought to use a scientifically informed, data-driven approach in ways that also engage the values of democratic educational organizations and promote equity. By questioning the premise that CI would be a useful approach to promoting equity in districts and schools, this article offers an opportunity to understand the inherent challenges in doing so and how policymakers and practitioners may act in ways that can combine this approach with their goals.
This analysis demonstrates that, in these multilevel case studies of CI in educational organizations, the competing logics of performance and racial equity were both draw on in CI design and implementation through three main modes of reproduction: performance metrics, routines and tools, and professional norms. Building on decades of performance- and racial equity-focused reforms, CI presents an exciting proving ground for leveraging entrenched performance-based principles to promote equity. Nonetheless, these cases demonstrate not only the potential associated with CI but also the challenges that emerge in implementation. Even as CI was intended to achieve the goal of a more equitable provision of educational opportunity, there was variation in the extent to which individuals exhibited racial equity logics in their CI practice and perspectives. In response to these challenges, both nested cases improvised unique additions to their CI processes, such as modified metrics, root cause “filters,” and explicit shifts in norms around data interpretation and root cause analysis.
One possible explanation for the combining of these logics in CI is that these two logics possessed complementary elements. That is, the racial equity logic may have been widely viewed as legitimate in concept, but may not have been a taken-for-granted element of structure, policy, and practice in educational organizations. In contrast, there were broad questions regarding the legitimacy of the performance logic (as documented in research on the implementation of standards-based reforms). Still the practices and structures associated with the performance logic were embedded within the day-to-day practice of educational organizations and broadly taken-for-granted. The structures and practices associated with performance logic may have served as a vehicle for the guiding principles of racial equity. In fact, specific principles, such as data use rather than standardization, may have been better suited to support the racial equity principles. As such, combining competing, but complementary, logics may provide unique opportunities for the institutionalization of new arrangements of principles, structures, and practices.
While there are many advocates of and a growing number of toolkits and methods for using CI for equity, few empirical studies have addressed this question as yet (for a key exception examining equity in data interpretation, see Garner et al., 2017). Promoting equity through CI held some challenges in these cases and also resulted in interesting improvised routines, tools, and norms in the case districts. While there was some evidence that the use of CI was linked to racial equity logic in addition to performance logic, the prevalence of these logics, however, was mixed and varied. While this study cannot assess the effectiveness of CI for achieving equity, my findings suggest that the implementation of this intuitive approach has potential, but is not without challenges.
Given this evidence and the prevalence of the CI approach, state departments of education and statewide support organizations may consider providing additional guidance on how to define equity. The intermediary organizations and districts in this study that defined equity clearly and unequivocally demonstrated more consistent views on equity and more frequent attention to equity throughout the CI process. In addition, my cases indicate that districts used a range of improvised modifications and supplements to the standard CI processes to address equity challenges more consistently. State departments of education and statewide support organizations may consider facilitating and coordinating learning across intermediary organizations and networks of districts. In particular, some structured sharing of metrics, tools, and routines may be particularly useful as more districts adopt this approach to decision making.
My findings indicate that these case districts and intermediary organizations were able to combine the racial equity and performance logics in implementing CI through their use of new metrics, routines, and norms. Intermediary organizations, networks, and districts may similarly find these approaches helpful as they adapt CI to their local context. First, those implementing CI should carefully consider the value of nontraditional measures of success. For example, rather than examining the number of suspensions and expulsions in a given year, a school might seek to document and measure any behavioral referral with both the teacher’s and student’s reason for referral and disaggregating these measures by student race and ethnicity and student interviews to understand the nature of such conflicts. Examining the problem of the practice of “disciplinary infractions” in this light directs inquiry toward potential root causes, such as teachers’ implicit bias, miscommunication between teachers and students, unmet student needs (e.g., instructional engagement, inadequate use of culturally responsive teaching), lack of restorative justice practices, and disciplinary policies (such as infraction definitions) that are racist. These kinds of measures also allow for a more nuanced understanding of success, as short-term gains may be evident in the types of referrals or who are referred and referring but not yet visible in long term suspension and expulsion rates. The use of alternative performance metrics, including qualitative and quantitative measures of various aspects of implementation and a wide range of outcomes, can help to undercut the decidedly normative character of traditional standards-based accountability systems that otherwise privilege dominant or status-quo conceptualizations of student success.
Second, the district cases created equity-focused modifications to CI routines and tools where they found shortcomings in the existing protocol and procedures. These gaps were often noticed during the process of using CI, as in the example of Manzanita’s filtering of root causes. As the data show, this filtering procedure effectively encouraged transparency, while also making clear that root causes rooted in deficit-oriented perspectives would not be validated. Because these improvisations were discovered during implementation, it may be necessary for districts and their support providers to understand the necessity of adapting CI methodologies to challenge White supremacy and entrenched racism. While additional research would be useful to gauge the efficacy of these practices across multiple contexts, these improvised modifications to CI processes may help districts and schools to engage in courageous conversations about race, equity, and the opportunity gap and to enact change.
Third, these districts and intermediary organizations relied on shifts in norms that prioritize equity and challenge racism. As these organizations explicitly employed CI with the aim of promoting equity, they evidenced some shifts in the spoken and unspoken rules or ideas that guide practice. In doing so, it was particularly important to define and communicate a concrete, unambiguous definition of equity. Leaders, administrators, and educators may assume that concepts like equity, racism, and implicit bias are defined consistently and widely accepted. The cases demonstrate that, particularly in the early stages of CI use, varied definitions of equity arose, including some that were built on notions of colorblindness and “pobrecito syndrome” (Garcia, 2001; Noguera, 2009). Defining equity may be an essential part of designing and adapting the CI process, especially as shared understandings of equity inform and are informed by improvised modifications to CI routines and tools. In practice, these shifting norms were particularly evident in how actors responded to data (i.e., with curiosity and openness rather than defensiveness) and how they attributed root causes to patterns identified in the data.
Of note, this study compared two nested cases that relied on very different approaches to CI and varied levels of structure and discipline in methodologies. The findings might appear to draw a connection between a highly structured CI approach and stronger evidence of the guiding principles of racial equity. This comparison fails to take into account the contextual differences between the cases, particularly substantial differences in sophistication of data access and analysis, and existing commitment to equity and racial justice. This study sought to describe and understand different approaches and, therefore, did not include a baseline analysis of guiding principles, structures, and practices. A generative area of future research would include identifying how differences in CI approach lead to institutional change across varied contexts in terms of performance and racial equity capacities.
Supplemental Material
sj-pdf-1-aer-10.3102_00028312221074404 – Supplemental material for “The Business of Teaching and Learning”: Institutionalizing Equity in Educational Organizations Through Continuous Improvement
Supplemental material, sj-pdf-1-aer-10.3102_00028312221074404 for “The Business of Teaching and Learning”: Institutionalizing Equity in Educational Organizations Through Continuous Improvement by Susan Bush-Mecenas in American Educational Research Journal
Footnotes
Supplemental material for this article is available online.
Notes
S
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
