Abstract
In late 2015, the passage of the Every Student Succeeds Act strengthened federal commitment to the use of evidence in educational innovation, focusing squarely on increasing the impact of innovation on educational outcomes. Concurrently, articles published in Educational Researcher examined new potential to increase the success of educational innovation through researcher-practitioner collaboration, focusing squarely on continuous improvement grounded in problems of practice. This essay examines how impact and improvement play out in the leadership of network-based improvement initiatives. Integrating critical scholarship with analysis of 20 i3-funded networks, the argument is that the practical challenges of network leaders vary inversely with system-level innovation infrastructure, which currently provides strong support for evaluating impact but weak support for continuous improvement. Thus, additional potential to increase the success of educational innovation is argued to lie in strengthening system-level innovation infrastructure to support the simultaneous pursuit of impact and improvement.
Keywords
The transition from 2015 to 2016 marked a moment when essential parameters of the U.S. educational reform agenda were under active negotiation, in plain view and for all to see.
In December of 2015, the passage of the Every Student Succeeds Act (ESSA) increased states’ discretion over standards, accountability, and school turnaround. ESSA also introduced new grant programs—including the Education Research and Innovation program—that strengthened federal commitment to the use of evidence in educational innovation. The policy focus is squarely on increasing the impact of innovation on educational outcomes.
Concurrently, papers published in Educational Researcher discussed ways that the research community could contribute to efforts to increase the effectiveness, pace, and reach of educational innovation (Bryk, 2015; Coburn & Penuel, 2016; Snow, 2015). Common among them—and a chief contribution to the national reform agenda—is a keen focus on improvement: on organizing practitioners, researchers, and others to support continuous learning and improvement grounded firmly in problems of educational practice.
This essay examines ways that impact and improvement play out in the leadership of network-based improvement initiatives. The first half examines networks and their leaders, parameters of their practice, and the environments in which they work. The second half examines challenges encountered by teams of leaders from 20 initiatives funded under the federal Investing in Innovation program as they worked at the nexus of impact and improvement.
One aim is to draw attention to a novel, essential, but under-the-radar category of educational practice: network leadership. Another is to use deeper understandings of the practice and environments of network leadership to extend conversation about possibilities to support educational innovators in simultaneously pursuing impact and improvement: specifically, by strengthening system-level innovation infrastructure.
Networks and Their Leaders
Over the past 25 years, network-based educational improvement has gained currency as a strategy for improving education for large numbers of students, especially those historically underserved by U.S. public schools.
Networks typically feature a central, hub organization that collaborates with members to develop, use, and refine a common approach to improvement (Bryk, Gomez, Grunow, & LeMahieu, 2015; Peurach & Glazer, 2012). In contrast to institutionalized, hierarchical administrative structures supported by stable funding sources, networks feature dynamic, reciprocal relationships among hubs and members, often dependent on grants and other soft funding. Leading examples include Reading Recovery, Success for All, the National Writing Project, BTEN, and the Knowledge Is Power Program.
Enthusiasm for network-based educational improvement has emerged in interaction with policy, philanthropy, and speculative investment that, since the advent of the standards-based reform movement, has established essential priorities for educational reform. 1 Key themes include:
a steady focus on schools as the essential unit of improvement and accountability, with a particular focus on improving or replacing chronically underperforming schools;
an emphasis on scalable, systemic (vs. boutique, targeted) interventions that address interdependent needs and problems in schools that routinely undermine the performance of students, teachers, and leaders;
movement away from the diffusion of interventions and toward sustained efforts to coordinate improvements in capabilities, structure, and culture in schools, with particular attention to improving capabilities for instructional practice as a primary resource for improving achievement.
Beyond creating the niche in which networks operate, the preceding reform activity has also framed their charge: not in terms of incremental change but, instead, in terms of transformative innovation. This charge is evident in metaphors widely used to convey a need for novelty and inventiveness characteristic of innovation: for example, “break the mold” designs as an antidote to a stagnating educational system and “turnaround” as the radical improvement required of chronically underperforming schools. 2
This charge is also evident in federal policy, from the founding of the Office of Innovation and Improvement in 2002 to policy documents defining the very meaning of innovation (Institute of Education Sciences, 2004). Central to the federal conceptualization is to dissociate innovation from fad and success from number of adoptions. Instead, the federal conceptualization associates innovation with evidence-based solutions to pressing needs and success with evidence of positive impact using rigorous evaluations.
Under the Obama administration, one chief source of funding structured around the preceding parameters is the Investing in Innovation (i3) program: a policy initiative that, since its launch in 2010, has provided over $1.2 billion in funding to over 140 improvement initiatives, selected from among thousands of proposals. 3 A scan of networks funded under the i3 program suggests key dimensions along which they vary, including:
types of hub organizations (e.g., university-based projects, nonprofit organizations, and newly constituted teams within state and local education agencies),
primary membership (e.g., teachers, leaders, and entire schools),
resources for improvement (e.g., tools, routines, and professional development),
geographic distribution (e.g., nation-wide, regional, or within-state),
their location on the “innovation development curve” (Supovitz, 2013) as they evolve from concept to large-scale use.
One common feature of these networks is that responsibility for their emergence, growth, sustainability, and effectiveness rests with cadres of leaders in (often newly created) hub organizations and on their efforts to establish novel enterprises that can improve educational outcomes at a pace and scale difficult to achieve within institutionalized organizational arrangements.
Some leaders are seasoned reformers with extensive experience managing large-scale enterprises. Others are impassioned educators, researchers, advocates, and entrepreneurs with little experience managing the types of networks that they seek to create.
Parameters of Practice
Following the reform priorities sketched previously, research on network-based educational improvement has focused largely on program impact, with success the exception (not the rule). A simple scan of the What Works Clearinghouse carries the point.
At the same time, a small (and growing) body of research examines the practice—the day-to-day work—of the hub organizations that manage these networks. 4 Themes running through this research provide perspective on the ratio of success to struggle.
One theme is that the work of hub organizations is very complex and requires that network leaders manage such functionally diverse activities as program development, recruiting, implementation support, evaluation, business systems development, logistics (conference planning, travel, publication), fund raising, and strategic management (Glennan, Bodilly, Galegher, & Kerr, 2004; Peurach, 2011; Peurach & Gumus, 2011).
A second theme is that organizing and managing this work in ways that quickly yield positive results is fraught with uncertain, interdependent problems and puzzles (e.g., Cohen, Peurach, Glazer, Gates, & Goldin, 2014; McDonald, Klein, & Riordan, 2009; Peurach, 2011). The roots of these problems and puzzles lie within networks, among newly recruited (and often initially weak) members; systemic, multicomponent improvement programs; and emerging and still-developing hub organizations.
These roots stretch further, into fragmented, turbulent environments notorious for uncoordinated national, state, and local policy initiatives that prioritize improvement needs, timelines, practical guidance, and metrics of success all their own. Improvement activity in these environments can conflict with that of networks, require rapid and radical program redesign, and increase risks of sudden discontinuation among members.
A third theme is that navigating this complexity and uncertainty requires continuous, collaborative learning and improvement among hubs and members to establish effective initiatives, coordinate with members and environments, and maintain viability (Cohen et al., 2014; Datnow & Park, 2009; Glazer & Peurach, 2015; Peurach, 2011; Peurach & Glazer, 2016). 5
Two learning processes appear to be key: exploration and exploitation (Hatch, 2000; March, 1996; Peurach, Glazer, & Lenhoff, 2016). Often associated with “local adaptation,” exploration is a type of divergent learning that involves reconsidering premises, addressing local needs, and building agency through search, discovery, and invention. Often associated with “fidelity of implementation,” exploitation is a type of convergent learning that involves using established knowledge, selecting from among tested alternatives, and refining through repetition.
While adaptation and fidelity are often viewed as in tension, case studies of successful large-scale networks suggest that exploration and exploitation can actually function as synergistic learning strategies (McDonald et al., 2009; Peurach et al., 2016). The process is one of iteratively recreating tested, base-level operations in schools (exploitation); refining and extending capabilities at the school level in response to local needs and environments (exploration); and selecting and exploiting favorable program improvements network-wide.
Comparative, cross-sector research on the innovation process suggests that capabilities to manage this sort of distributed, interorganizational learning are generally scarce, and often tacit and subconscious (Van de Ven, Polley, Garud, & Venkataraman, 1999). Such leadership requires a sort of ambidexterity: the capability to balance seemingly contradictory strategies that encourage divergence, exploration, and problem solving while simultaneously driving convergence around a common approach that can be exploited widely and effectively.
Environments of Practice
Thus, consistent themes in national educational reform have created a dependence on network leaders able to bring into existence novel enterprises able to improve educational outcomes at an uncommon pace and scale. Difficulty doing so arises from complexity and uncertainty endemic to their work, uncommon capabilities needed to manage network-wide learning and improvement, and, in some cases, inexperienced network leaders.
Yet additional difficulty arises from the environments in which network leaders practice. Beyond fragmentation and turbulence, this additional difficulty arises from the variable development of system-level innovation infrastructure: interdependent political, policy, philanthropic, private, and professional activity that motivates, enables, and constrains the practice of educational innovation. 6
Robust Impact Infrastructure
Over the past 25 years, educational environments have evolved to include robust impact infrastructure advancing research, evidence, and evaluation as central to the practice of developing effective, scalable educational innovations. Key components include:
the creation of the Office of Innovation and Improvement and Institute of Education Sciences to advance federal activity supporting evidence-driven educational innovation;
public and private grant programs that require, as a condition for funding, both rigorous evidence of potential program effectiveness and rigorous evaluations capable of producing scientific evidence of program impact;
the establishment of the What Works Clearinghouse and its evidence standards;
federal investment in the advancement of statistical methods and in early-career training programs aimed at putting these methods into practice;
the proliferation of private research firms with highly developed capabilities for rigorous program evaluation;
the emergence of the Society for Research on Educational Effectiveness (SREE), a professional association with the express purpose of “provid[ing] an organizational infrastructure that supports and promotes research focused on cause-and-effect relations important for education” (SREE, 2016).
This impact infrastructure is further anchored in a political context characterized by what Mehta (2013) describes as the “allure of order”: longstanding faith among policymakers, reformers, and thought leaders in the potential to use principles and processes of rational management to discipline otherwise “soft” and undisciplined educational practices. From that follows strong support for introducing rationality, structure, discipline, and urgency presumed missing from the practice of educational innovation.
This impact infrastructure is buttressed by the rapid institutionalization of standards and accountability among states, districts, and schools as well as by established divisions, SIGs, journals, and research standards within the American Educational Research Association that have long sought to improve the rigor and quality of educational research. 7
Under the Obama administration, the most vivid example of the strengthening of this impact infrastructure lies in the introduction of a “tiered evidence sequence” to structure competitive grant programs that guide the process of developing, validating, and scaling up educational innovations. At all three levels, eligibility for funding is tied to (a) using rigorous evidence to demonstrate the potential for positive impact on student outcomes and (b) incorporating rigorous evaluations that, ideally, meet the evidence standards of the What Works Clearinghouse. Evidence of positive impact is rewarded with increased legitimacy and with eligibility for higher levels of funding.
Initially used to structure the i3 program, the logic of the tiered evidence sequence has since been carried into formula and competitive grant programs within ESSA, including the Educational Innovation and Research program. Indeed, this tiered evidence sequence is one dimension of what Haskins and Baron (2011) describe as historically unprecedented efforts by the Obama administration to advance the use of evidence in social policy and innovation.
Weak Improvement Infrastructure
However, over this same period, educational environments have not evolved to include robust improvement infrastructure that would support networks and their leaders in developing capabilities for the distributed, continuous learning and improvement on which impact depends. 8
Paralleling impact infrastructure as described previously, key components of improvement infrastructure would include a political context that legitimizes complexity and uncertainty in the work of networks and that accords them time and patience, policy and philanthropic initiatives that include requirements and funding for continuous learning and improvement, and private and professional organizations with capabilities to support networks and their leaders in developing capabilities for continuous learning and improvement.
It is not that attention to improvement is wholly lacking. Components of such an improvement infrastructure are emerging, including:
two recent federal grants programs to support program improvement, one focused on continuous improvement in education and another on research-practice partnerships;
the development of the learning sciences as an interdisciplinary field advancing the development and application of knowledge of individual, group, and organizational learning, complemented by the development of formal methods of practical problem solving among researchers and practitioners;
university-based centers and private organizations with methods and capabilities to support network improvement, including the LearnDBIR initiative, the National Center on Research-Practice Partnerships, the National Center on Scaling Up Effective Schools, the Carnegie Foundation for the Advancement of Teaching, and the SERP Institute;
philanthropists that have supported the practice and development of network leaders, including William T. Grant Foundation, the Spencer Foundation, the William and Flora Hewlett Foundation, and the Broad Foundation.
Akin to impact infrastructure, these components of improvement infrastructure are buttressed by efforts in states and districts to structure and support continuous school improvement (as via annual school improvement reporting-and-planning processes) as well as by pockets of attention to systems thinking, design thinking, and continuous improvement within AERA. Indeed, Educational Researcher has long served as an important forum for those engaging these issues. 9
However, these components of improvement infrastructure are nascent, limited in their reach, and weakly coordinated. Moreover, absent the political and policy support undergirding impact infrastructure, they are also fragile.
Challenges in Practice
The i3 Learning Community (i3LC) was established to support network leaders in managing practical challenges that they experienced working under these conditions: in environments that have evolved in ways intended to discipline their work and evaluate their impact but that are only beginning to evolve to support continuous learning and improvement.
The i3LC included teams of leaders from 20 enterprises funded in the first two years of the i3 program. All were pursuing some form of network-based educational improvement, with hub organizations and members working beyond institutionalized organizational and funding arrangements to develop, use, and refine common approaches to improvement (see Table 1). Participants were selected, in part, based on the potential of their evaluation designs to satisfy the evidence standards of the What Works Clearinghouse.
The i3 Learning Community: Participating Projects
The i3LC was constituted by the William T. Grant Foundation and the Spencer Foundation and managed in collaboration with the Forum for Youth Investment. Between September 2011 and September 2014, the i3LC met for six two-day meetings in Washington, DC, each consisting of multiple whole-group and small-group sessions from 30 to 90 minutes in length. These sessions provided opportunities for structured and unstructured interactions among participants as well as with representatives of research, policy, and practice communities.
Approach to Learning
I joined the i3LC beginning with the second meeting: not as part of a formal research-practice partnership but instead as an advisor and collaborator in its design and implementation. My contributions included presenting my research, facilitating sessions, and collaborating with organizers and participants in dynamically designing (and reflecting critically on) meetings.
This last point—the dynamic design, enactment, and analysis of meetings—is an important one. The i3LC did not feature a predetermined curriculum addressing challenges known to complicate the practice of network leaders. Rather, the i3LC was as much a learning experience for its organizers as for its participants as they worked meeting by meeting to identify challenges and structure support.
Organizers of the i3LC did not do so using formal methods of continuous improvement. Rather, their method was more the type of “fast learning” characteristic of commercial entrepreneurial activity (Supovitz, 2013). Organizers gathered subsets of participants, advisors, and guests during meetings and in conference calls. They then enlisted the assistance of these collaborators in analyzing feedback from prior meetings, reflecting on personal experiences and notes from those meetings, and drawing on the diverse perspectives that each brought to the i3LC. This combination of evidence, analysis, and reflection drove the design of subsequent meetings and sessions.
Lessons Learned
One artifact of this collaborative learning process is the set of agendas that structured the six meetings. These agendas map a total of 97 sessions focused on challenges for which participants sought support: roughly 16 sessions per two-day meeting. These challenge-focused sessions were complemented by additional sessions focused on building relationships, sharing information about project designs, and creating opportunities for ad hoc consultations.
Using an iterative coding process, I grouped the 97 challenge-focused sessions into six primary categories. 10 Listed in descending order of frequency, the first five categories focused on challenges associated with managing continuous learning and improvement; the sixth category focused on challenges associated with managing evaluations. They are as follows:
35% of the i3LC sessions focused on managing and improving networks, including their membership (“Recruiting Teachers, Schools, and Districts”), their hub organizations (“Creating a Business Model/Marketing Your Intervention”), and their relationships with their environments (“Communicating Effectively About This Work With Different Audiences”).
22% focused on improving programs, including their core designs (“Integrating Technology into the Curriculum”) and their supports for implementation (“Using Video for Coaching and Professional Development”).
12% focused on interpreting the policy environment, both as directly related to the i3 program (“What Has the Federal Government Learned From i3, and Implications for the Future?”) and more broadly (“Common Core—What’s Happening, and How Does It Affect Us?”).
11% focused on using performance evidence as a resource for network management and improvement, including evidence generated in the course of routine network operations (“Collecting, Reporting on, and Using Data: Meeting the Needs of Schools and Projects”) and evidence generated through the i3 evaluations (“What Are Our Evaluations Telling Us? Using Interim Results”).
11% focused on issues and tensions that arise in managing continuous learning and improvement, including coordinating exploitation and exploration (“Managing Fidelity, Flexibility, and Fluidity”) and coordinating with evaluations (“Continuous Improvement While in Evaluation”).
9% focused on managing the i3 evaluations, including establishing working relationships (“Building a Productive Relationship With Your Evaluator”), developing resources to structure leader-evaluator collaboration (“Logic Models and Alignment and Integration of Measures of Fidelity”), and measurement (“Measuring Teacher Effectiveness”).
Challenges, in Context
Juxtaposed against the preceding critical analysis of network-based educational improvement, what stands out about the challenges experienced by i3LC participants are not the disconnects but instead the overlaps. For example, the challenges experienced by i3LC participants present an image of practice consistent with that identified in other research on network-based improvement, with network leaders managing a functionally diverse array of responsibilities demanding broad knowledge and expertise.
Further, the challenges experienced by i3LC participants suggests that organizing, coordinating, and managing this work in ways that quickly yield positive changes is fraught with uncertainty and complexity and thus demanding of continuous learning and improvement. Indeed, 91% of the i3LC sessions focused on issues related to interpreting information from within and beyond the network, exploring new alternatives, refining established approaches, and coordinating with ongoing evaluations.
The third is that the categories of support provided by the i3LC varied inversely with the strength of system-level innovation infrastructure supporting the practice of network leaders.
Where system-level infrastructure was strong (as with impact), network leaders drew far less on the i3LC for support. Despite rigorous evaluation as a novel dimension of practice, only 9% of the i3LC sessions focused on challenges associated with managing evaluations. This can be interpreted as an artifact of strong evaluation designs being a requirement for participation in the i3LC. Yet those strong evaluation designs in turn can be interpreted as positive artifacts of 25+ years of system-level activity aimed at supporting innovators in quickly incorporating expert knowledge of evaluation into their practice with a minumum of challenge.
By contrast, where system-level infrastructure was weak (as with improvement), i3LC participants turned to the community for support. After all, the i3 program did not include requirements or funding to partner with experts in continuous learning and improvement. And even if it did, i3LC participants would have turned to environments with scarce capabilities to support such work and with little political patience for it. Indeed, i3LC participants were working in a political context demanding quick returns on a billion-dollar investment, poised to ask, simply, “What works?”
Reflection: Supporting Practicing Innovators
Thus, network leaders are key agents of a 25+-year reform agenda aimed at developing large-scale, evidence-based, and evidence-proven educational innovations, with a particular focus on improving education for historically underserved students. To date, system-level innovation infrastructure has evolved unevenly: strong in support for evaluating program impact but weak in support for continuous improvement. Working under these conditions, practical challenges for which one group of network leaders sought collegial support varied inversely with the strengths of system-level innovation infrastructure.
To the extent that the preceding analysis is on the mark and to the extent that the experiences of i3LC participants bear some resemblance to those of other educational innovators, then it provides a framework to extend recent conversation in Educational Researcher about possibilities to increase the effectiveness, pace, and reach of educational innovation by supporting practicing innovators.
One form of support is to strengthen the type of instrumental assistance described in recent articles in Educational Researcher: researchers working shoulder to shoulder with network leaders and other innovators to identify, understand, and address their problems of practice. Strengthening this type of instrumental, practice-based support will require continuing to develop and coordinate among:
basic research that takes practicing innovators as primary units of analysis, in order to increase understandings of their work; the practical problems that they routinely encounter; and the knowledge, capabilities, and dispositions needed to manage for impact and improvement;
a population of researchers on which innovators can quickly draw to support the work of continuous improvement, along with formal methods and metrics of continuous improvement that can be used to gauge progress;
a population of evaluators on which innovators can quickly draw to support them in evaluating their enterprises in ways sensitive both to the complexity of their programs and the demands of continuous improvement.
A second form of support is to understand and pursue the preceding as integral to broader efforts to build, refine, and sustain system-level innovation infrastructure.
One pressing need is to continue to refine and sustain impact infrastructure. To be clear, determining what works, for whom, and under what conditions is absolutely essential—if not for political, financial, and practical reasons then for moral reasons. After all, the work of network leaders and other educational innovators has them intervening in the lives and development of children, a great many of whom are from poor and historically underrepresented groups and communities.
Yet, while under development for 25+ years, system-level impact infrastructure is hardly institutionalized and under constant threat. Indeed, that impact infrastructure has been strengthened under ESSA owes much to the efforts of researchers, philanthropists, and others in championing the role of evidence in educational innovation.
Another pressing need—and a heavier lift—is for researchers to contribute to efforts to strengthen system-level improvement infrastructure. While components are emerging, the development of improvement infrastructure could be further catalyzed and strengthened with:
political support anchored in understandings of (and respect for) uncertainty in educational innovation;
the establishment of capabilities within the U.S. Department of Education to coordinate the development of improvement infrastructure;
federal investment both in (a) methods and metrics of continuous improvement and (b) the professional development of researchers able to support continuous improvement;
federal competitive grant programs that further draw philanthropists, private firms, and professional associations into the “improvement” niche: for example, by requiring that recipients secure both private matching funds and external support for continuous improvement;
continued efforts among states, districts, and the AERA community to buttress the preceding.
The foundation has been laid. Again, an emerging community of researchers, practitioners, philanthropists, and policymakers are deeply engaged in developing, using, and refining resources and incentives supporting continuous improvement in networks and other large-scale educational innovation initiatives. In doing so, they are building on the work of others too numerous to list who, for the past 50 years, have sought to understand the complexity of life in classrooms, schools, and systems as well as the work of supporting students, teachers, and leaders in collaborating in more positive ways and toward more meaningful ends.
The moment is ripe. Beyond recent articles in Educational Researcher, others are drawing attention to the improvement imperative: for example, by recognizing the possibility of making formative use of weak and equivocal impact evaluations (Jacob, 2015), advancing a “two stage model” that coordinates impact and improvement (Dynarski, 2015), and advancing strategies for coordinating impact and improvement among policies (federal, state, and local), professional networks, and local constituencies (O’Day & Smith, 2016).
However, in environments long drawn to rationality and impatient with uncertainty, coordinated political and policy supports are unlikely to develop without a bottom-up press from researchers, philanthropists, and others who recognize improvement infrastructure as an essential component of system-level innovation infrastructure.
This bottom-up press in turn will require that diverse organizations and interests recognize common cause, identity, and purpose and that they coalesce to assert a level of influence on politics and policies that none would be able to assert independently. Mustering and sustaining such coordination and influence will require formidable leadership in a vast, plural reform context prone to factionalism. However, the locus of such leadership is not yet clear.
Success developing coherent impact infrastructure is evidence of possibility. That doing so took 25 years (and counting) suggests that developing coherent improvement infrastructure will be an intergenerational effort better measured in decades rather than in years.
Looking Forward
With that, we have arrived exactly where we began: at a moment when essential parameters of the U.S. educational reform agenda are under active negotiation, in plain view and for all to see. But what is it that we see?
One risk is that some might see a new paradigm war taking shape, the broad outlines framed by impact and improvement, with sides chosen based on epistemologies, ideologies, and interests. But wars are destructive and often unnecessary and avoidable. More hopeful (and more pragmatic) is to understand impact and improvement as complementary dimensions of educational innovation, with success dependent on developing and coordinating comprehensive, system-level innovation infrastructure to support educational innovators in managing for both.
Metaphorically, developing and coordinating infrastructure does not conjure up images of quick fixes but instead of nation-building: of the extensive investment, effort, and time needed to establish the foundational systems of a functional, productive, and just society. In this case, the nation is our own, and the foundational systems are those through which our society advances, coordinates, and uses the many forms of knowledge needed to ensure that all of our children become contributing, successful members.
Footnotes
Notes
Author
DONALD J. PEURACH, PhD, is an associate professor of educational policy, leadership, and innovation at the University of Michigan, School of Education, 610 E. University, Ann Arbor, MI, 48109;