Abstract
Transcending the low status of educational research will require demonstrating its relevance to improvements in practice. Educational progress is most likely to emerge from approaches to research that create an equal footing for practitioners and researchers, recognizing that though these groups accumulate and curate knowledge in different ways, they both have a role in creating tools (curricula, practices, professional development approaches) that can be used to forge lasting improvements. A brief history of the ongoing shift toward practice-embedded educational research (PEER) demonstrates its increasing acceptance and popularity and suggests modifications to the future selection of research topics, funding mechanisms, and professional preparation of both practitioners and researchers.
Keywords
©2007 Tom Kates Photography
Over the past dozen years, a remarkable shift has taken place in the rhetoric relating educational research to practice. We have moved beyond the mottos of the 20th century—from research to practice, doing translation science—to a new model that emphasizes the interconnections of research and practice rather than the gap between them. It is my goal in this article to characterize this shift and the principles that underlie it, briefly trace its history and document its current success, and give a few examples of the challenges that surface in trying to work according to its principles.
The Shift to Practice-Embedded Research
Beyond “From Research to Practice” and “Translational Research”
The default 20th-century approach to research started from the high status awarded to basic research—the research carried out by discipline-based researchers in fields like child development, cognition, linguistics, mathematics, and anthropology. Those disciplinary fields, it was assumed, yielded insights and/or methods that could be applied to education through a process of translation. Ultimately, it was recognized that the translation/application process itself was a worthy target of study, generating the robust, though less prestigious, field of “applied science.” The basic/applied distinction remains most vigorous in the domain of language; linguists work to describe general principles of language structure and their specific instantiations in different languages, while applied linguists worry about the implications for teaching foreign/second languages. Following a similar model, mathematicians described abstract entities and relations, and educators figured out how curricula might make those accessible to students; cognitive scientists described concept development and problem solving, and educators extracted principles to guide science and history instruction; developmentalists described young children’s autonomous discovery of perspective, the animate/inanimate distinction, and conservation of quantity, and early childhood programs were designed in which discovery and free play were prioritized.
One drawback of this model, though, was the inevitable recognition that many challenging targets for educational research had no analogue in the basic sciences. Some analytic challenges arise from the contexts of schooling and would never be primary targets of investigation for basic scientists: Reading comprehension is a primary example, but others include peer relations, historical empathy, and classroom management. Investigation of topics like these always occurred in a no-man’s land between basic and applied—in the territory we are now reclaiming as “practice-embedded research.”
A second and even more disastrous drawback of the traditional basic/applied distinction was the unquestioned assumption that if the basic science was sound, the application process was simple, requiring only interpretation or translation. Thus, enormous energy was invested in applications (e.g., teaching foreign languages using contrastive analysis methods, withholding literacy and numeracy activities from early childhood classrooms) that emerged from basic science but that were themselves at worst severely flawed and at best in need of careful study and systematic evaluation.
Beyond the “Awful Reputation” and the “Elusive Science”
Though history of science is not a domain in which causal inferences are easy to defend, I would hypothesize that the low status of educational research, referred to by Kaestle (1993) as its “awful reputation” and alluded to by Lagemann (2000) with the term elusive science, reflected the basic/applied distinction and its shortcomings as an approach to actually solving educational problems. Critiques of educational research as an enterprise have been launched on the grounds of methodological eclecticism (Ravitch, 1985), absence of rigor (e.g., Lyon & Chabra, 2004), and paucity of theory (Lagemann, 2000). These critiques would, I contend, all have been trumped by research-informed success in improving educational outcomes. Ultimately, the awful reputation of the elusive science can be traced, at least in part, to its alarming fecklessness.
It was this ineffectiveness that led Bruce Alberts, during his term as president of the National Academies from 1993 to 2005, to propose and support an effort to design an educational research enterprise that would replicate the close connection between research and practice found in medicine, agriculture, and highway safety. Thus was a series of committees established, each charged with writing a report (Donovan & Pellegrino, 2003; Donovan, Wigdor & Snow, 2003; National Research Council [NRC], 1999)—report writing being how work gets done at the National Research Council, the operating arm of the National Academies. Out of that series of projects was borne the Strategic Education Research Partnership (SERP), an effort to do educational research in a radically new way and, as it turned out, the harbinger of what has become a broad and productive movement.
A New Set of Principles for Educational Research
The SERP reports proposed a set of principles to guide the new approach to educational research and to distinguish it from prior approaches. I will use the term practice-embedded educational research (PEER) to refer to this new approach; it is important to distinguish PEER from action research or from any particular research method by identifying the principles to which it adheres. Those principles were radical when they were formulated in 2003, though they have now become much more commonplace.
Partnership
A key proposal of the SERP report and key principle of PEER was the establishment of structured, supported, and sustained research-practice partnerships (Donovan, Snow, & Daro, 2013). The traditional relationship between researcher, the producer of knowledge, and practitioner, the user of knowledge, was replaced by a commitment to the notion of two sources of knowledge (research and practice). Though the two sources might generate somewhat different types of knowledge, both types are judged to be of equal value and importance to improving educational outcomes.
A corollary of the partnership model is that researchers need to acknowledge the realities of practice and practitioners need to acknowledge the commitment to rigor in research. Only through a true partnership, it was argued, would the inevitable challenges of, for example, fielding randomized trials in complex district settings or ensuring that research-related assessments not interfere either with instructional time or with district/state-mandated assessments, be taken on as shared problems rather than loci of competing interests. Partnerships offer the opportunity for both sets of participants to learn about the other’s commitments and constraints and thus to develop a collaborative rather than an antagonistic relationship.
Starting With Urgent Problems of Practice
A key element of PEER is abandoning, or at least modifying, the traditional academic procedures for deciding on the focal problem. Academics traditionally decide what to do next by identifying a problem that arises from theory or from a tractable gap in the literature. The 2003 SERP report proposed starting, instead, with the pressing concerns of practitioners (Donovan et al., 2003). This shift to a focus on urgent problems of practice constitutes a revolution within the academy, where a research contribution is evaluated as a function of its relation to prior research in the same field.
Attention Both to Innovations and to Their Implementation
Improving practice requires introducing innovations and evaluating their effectiveness. In “normal educational science,” rather little attention is paid to examining the nature of the innovation or analyzing its capacity to solve a problem educators actually care about. The PEER approach embraced by SERP, on the other hand, puts special focus on the careful study of how innovations are implemented. Of course, any evaluation of a randomized trial incorporates attention to implementation—but usually in order to explain (away) variation in impact. The practice-inspired approach treats variation in implementation not as a mediating variable but as a crucial source of information. Knowing what aspects of a new program or practice are easy or hard to implement, which ones are adopted after minimal versus only after intensive professional development, which are embraced by teachers, and which rejected is crucial to designing new innovations that are likely to take. Furthermore, whereas in normal educational science deviations from prescribed models are considered problematic, practice-embedded research often incorporates such deviations in subsequent versions of the practice or program, seeing them as a source of evidence to inform ongoing work. PEER adherents are committed to watching how new tools get used and listening to what teachers say about them so that through a process of iterative design more easily usable versions of evidence-based programs will emerge.
Attention to Systemic Change
An additional principle that is central to PEER is the recognition that students and teachers operate within systems and that improvements inside classrooms require thinking about and often operating at school and district levels at the same time. It would be easy to present dozens of cases of highly effective curricula being undermined by lack of sufficient attention to professional development or being diminished or terminated by the introduction of new, competing initiatives. Student learning is dependent on teacher learning, but teachers’ opportunities to learn are in turn dependent on organizational structures that support learning. Schools and districts are too often impatient for short-term results and unwilling to cede innovative practices the time they need to be mastered, adapted, evaluated, and institutionalized.
A New Movement?
Considerable interest in and acclaim for this new model of doing educational research has developed over the past dozen years, since the 2003 SERP report (Donovan et al., 2003). For example, various funding mechanisms launched by the Department of Education in the past few years prescribe and are designed to support research-practice partnerships. Similarly, the Regional Education Laboratories have been encouraged to partner with districts and states in establishing their research agendas. Two notable research consortia have emerged, both focused on using district data to answer questions of interest to the district: the Baltimore Education Research Consortium (led by Faith Connolly) and the Research Alliance for the New York City Schools (led by James Kemple). The MIST project (Middle School Mathematics and the Institutional Setting of Teaching, led by Paul Cobb) has established partnerships with the Fort Worth and the Grand Rapids school districts, focused on analysis of math teachers’ knowledge and practices as a prerequisite to formulating and evaluating evidence-based but location-specific solutions. The Consortium for Chicago School Research (CCSR), established in 1990 and long a pioneer in providing high-level data analysis to the Chicago Schools, in 2009 offered an updated conceptualization of its role as one of partnership in building district capacity to solve problems (Roderick, Easton, & Sebring, 2009). Meanwhile, the Carnegie Foundation for the Advancement of Teaching redirected its energies, under the leadership of Tony Bryk, to applying continuous quality improvement methods borrowed from the medical profession to educational problems. The Stanford Youth Data Archive has extended the partnership model beyond education to promote data sharing and collaborative data use across agencies providing health, mental health, housing, welfare, recreation, enrichment, and child protection services as well as school districts (McLaughlin & London, 2013; Nelson, London, & Strobel, 2015). All of these efforts reflect a healthy prioritizing of the needs of practice in establishing educational and education-related research agendas.
These various undertakings all conform, to greater or lesser extent, to the model first articulated in the SERP committee reports (Donovan et al., 2003; NRC, 1999). The principles laid out in the 2003 report have been widely adopted, for example by leaders of the movement dubbed DBIR (Design-Based Implementation Research), suggesting their value and their durability (Fishman, Penuel, Allen, & Cheng, 2013). Their reiteration in the mission statement of the importance of such efforts also suggests the good timing of the SERP report—that it served as a catalyst to speed up a shift in thinking for which the field was ready.
SERP Vision and SERP Reality
The SERP vision, as articulated in that 2003 report (Donovan et al., 2003), was ambitious, optimistic, perhaps even fantastic: Envision a cadre of leading scientists and practitioners working together on a coherent, highly focused program of education research that is tightly coupled and interactive with practice. They are guided and supported by the kind of organizational infrastructure needed to plan, manage, and carry out a sustained program of research and development. They work in collaborative teams, and much of the research is carried out in school settings around the country. As the research teams learn over time how to cultivate the substance and processes of research-based practice, they come to embody a new model of practitioners and researchers familiar with and comfortable in both the world of research and of practice. (p. 1)
The details of the vision included launching two networks in the first seven years, one on learning and instruction and one on schools as organizations, with 10 district-based projects across them, at an estimated cost of $500 million. The reality is somewhat more modest. SERP has established “field site” relationships with four districts and with the Minority Student Achievement Network (a group of 29 affiliated districts) as well as close working relationships with a number of other districts or networks of schools that are using or extending work begun in the field site districts. It has been remarkably successful in recruiting researchers willing, even eager, to do their work according to this model. Dozens of doctoral students have been involved in SERP-related research, and many of them have in the process been recruited to operating in practice-embedded rather than traditional educational research.
At the same time, SERP has been attentive to the principle that it should be building knowledge, not just solving problems. Three major contributions to knowledge could be mentioned, though these do not exhaust the list of SERP research efforts.
A within-teacher randomized study incorporating worked examples into Algebra 1 assignments in five districts showed an educationally and statistically significant impact as measured by an end of year test composed of released items from the five states involved (Booth et al., 2015). This study contributes to the growing evidence about the value of explaining pre-worked examples in math and other domains as compared to investing the same amount of time in solving multiple problems of the same sort. In a second effort, SERP’s relationship with Baltimore and San Francisco opened up the possibility of fielding an evaluation of Word Generation (see www.wordgen.serpmedia.org) that required school-level randomization and that generated evidence about the efficacy of the program and the role of classroom discussion in mediating program impacts (Lawrence, Crosson, Paré-Blagoev, & Snow, 2015).
A third project launched by SERP is exploring a new theory of deep reading comprehension and the power of curriculum to support changes in teacher practice in fourth- through eighth-grade classrooms (ccdd.serpmedia.org). Numerous researchers from Harvard University, Stanford University, Wheelock College, and Boston University have participated in the project. Multiple research instruments have been generated and validated in order to test the theory, including an academic language assessment that contributes substantially to predicting variation in reading comprehension (Uccelli et al., 2014, in press). Tier 1 (Word Generation) and Tier 2 (STARI) curriculum materials have been designed and evaluated across multiple districts.
The infrastructure supported by SERP was crucial to the success of this effort; for example, an experienced design team produced curricular units that were attractive and easy for students and teachers to navigate. Digital resources and videos produced by the SERP Design Center were indispensable to professional development delivery. The scope of the effort required interdisciplinary commitments from researchers and the capacity to continuously improve with feedback from practice. The collaboration among researchers, designers, and practitioners has produced educational tools for use in Grades 4 through 8 that are now freely available (serpmedia.org). In other domains, SERP is prototyping tools designed to respond to teacher suggestions and/or research findings (e.g., Poster Problems, http://math.serpmedia.org/diagnostic_teaching/, and the 5 × 8 Card, math.serpmedia.org/5x8card) that can be subjected to future rigorous evaluation efforts (Donovan, in press).
The research program of evaluating the effectiveness of Word Generation and STARI and of understanding the conditions under which both programs are most effective is barely begun. We have in an initial analysis confirmed the credibility of the claim that perspective-taking, academic language, and complex reasoning help explain variance in performance on challenging reading comprehension tasks (La Russo et al., 2015), but most of the research contributions from this program are still in process. Meanwhile, Word Generation has robustly proven its value to practitioners. The Word Generation website, where the curricular materials can be downloaded, has (as of mid-August, 2015) more than 17,700 registered users, and in the year ending July 31, 2015, received an average of 7,595 visits per month. In response to user requests, adaptations of the program and supplementary materials have been developed for classrooms serving English language learners, both second language learners in the United States and students in English foreign language classrooms around the world. Translations and adaptations incorporating locally relevant issues have been or are being developed in Norway, the Netherlands, China, Spain, and Germany, and schools in the United States and abroad have shown strong interest in expanded versions of the curriculum.
Lessons Learned
Experience with the Word Generation program (and with the other SERP initiatives) has taught us a lot about how to do PEER. We have learned, for example, how much time and effort needs to be invested in nurturing relationships with districts and schools that are being buffeted by numerous demands and an accumulation of initiatives (Hess, 1999). We have learned how crucial design expertise is to the enterprise; director of the Bay Area SERP site Philip Daro refers to the need to design not just the working end of the tool (the head of the hammer or clamp on the spanner) but also the handle—the end the user has to grasp in order to make use of the tool.
We have also learned how ill-suited the current project-focused mechanisms for funding are to the enterprise of practice-embedded research. The Institute of Education Sciences (IES) partnership grants are an effort to support this approach, and an acknowledgment of its value, but they still require too much pre-specification of plans that ought to be developed collaboratively, after the funding is available and the partnership is truly launched. There is a huge amount of work involved in negotiating the initial questions and assembling a working team—work that is invisible and typically unfunded, as is the work involved in making the findings useful to educators.
A commitment to making tools and insights available to the world of practice also leaves SERP somewhat vulnerable to getting too little credit for its work or alternately taking the blame for partial or unskilled efforts to implement its programs. How much responsibility for the skillful use of new tools is incumbent on those offering such tools to the field?
Institutionalizing Practice-Embedded Research
Ultimate success for practice-embedded approaches to research will position it as the default—the normal way of doing things. That will in turn happen only if future practitioners and future researchers are socialized into the ideas that are central to the undertaking: their mutual dependence, the value of iterative approaches to evaluation and improvement, a commitment to the rigor of practice that is as strong as commitment to the rigor of design. Those changes will in turn require a cascade of shifts in the larger research enterprise: deans and tenure committees who truly value practice-embedded research from education scholars and who recognize that it may be less quickly completed and perhaps published in less prestigious journals than traditional research; journals that are willing to publish the research, including highly informative negative findings, and that appreciate the difference between variation in implementation and lack of fidelity; funders that are willing to support the infrastructure for partnerships as well as the specific projects that emerge. They will, in addition, require a radical shift in the preparation of practitioners, teachers as well as school and district leaders, to provide guidance about the challenges of collaboration with researchers and to articulate the value and the limitations of both research findings and practice-acquired wisdom.
Traditional academic research is famously conservative. Getting funding often requires having such solid pilot data that the studies proposed will add very little new information or asking such minor questions that the results will have little value for practice. Too much energy is expended on the wrong questions. For example:
We do not need any more studies estimating effect sizes for exposure to early childhood education. We need instead careful studies of what defines quality in early childhood programs and how teacher skills, curricular resources, and professional development contribute to quality.
We do not need another 10 years of .15 effect sizes from evaluations of interventions and programs. We need to interpret those effect sizes and variation in them across sites in light of what we know about teaching and learning.
We do not need more studies evaluating the impact with general education students of interventions developed and demonstrated to be effective with special populations (delayed readers, dyslexics, emotionally stressed students). We need theory and the wisdom of practice to guide the generalization of findings from limited samples to the general population.
We do not need more studies evaluating impacts of add-on programs to specific, targeted outcomes. We need studies of how added practices and approaches can be integrated into and thus enhance the impact of well-designed core instructional programs.
What Would Make a Difference?
The PEER model proposed here suggests a new and different research agenda for the next 20 years. That research agenda in turn would rely on some new structures and tools for researchers and practitioners to use in collaboration with one another.
First, it will be necessary to build the partnership model into the preparation of the next generation of educational researchers by connecting the clusters of practice-embedded projects currently underway more closely to universities. Unfortunately, much of the work done under this model is peripheral to the academy. The Research Alliance for New York Schools and the Baltimore Education Research Collaborative have only loose university connections. The CCSR has operated at the University of Chicago but with considerable autonomy and thus has rarely involved doctoral students in its work. The nature of the work is, admittedly, not a perfect fit for doctoral careers—somewhat unpredictable, sometimes undertaken with a lengthy trajectory toward completion, and subject to nonacademic influences. Nonetheless, it is possible to involve doctoral students in the work without threatening their progress; at Harvard, doctoral research practica have been taught annually since 2008, each of them linked to some stage in the development of SERP-related projects. The practica constitute opportunities both to teach about the model and to induct students into the model while giving them opportunities to complete short-term research projects that may or may not lead to dissertation studies.
If research-practice partnerships are to thrive, we need to seek accountability metrics that are seen by teachers as helpful rather than threatening. Accountability policies could be very different if we had theoretically defensible, reliable, and feasible measures of classroom processes—teaching practices and student engagement—that might eventually take the place of student outcome measures.
Taking the wisdom of practice seriously means developing a mechanism for systematizing and curating it—an epistemological structure equivalent to the (clunky but generally admired) set of procedures in place for reviewing research contributions. At the moment, we have no way of distinguishing insightful teacher tips and effective invented practices from their opposites. Thus, it is easy to dismiss the “wisdom of practice” as an accumulation of anecdotes. A quick read of teacher sharing sites and blogs makes clear that the quality of recommendations, lesson plans, and insights shared is heterogeneous. But mixed in with the many ideas are some really good ones—ones that other teachers would recognize, adopt, adapt, and endorse. Unfortunately, there is no way for this to happen—no forum where reactions can be collected and systematically documented. One might think that we could do for educational practice what Tripadvisor has done for hotels, or Yelp for hot dog stands, but as yet it has not happened.
The incoherence of the U.S. educational system is a feature widely commented upon; Cohen and Bhatt (2012) describe it as a system designed to be ungovernable. If this feature is indeed an obstacle to better academic outcomes for students, then we might consider creating a few small trial “coherence zones”—clusters of schools that draw teachers from aligned teacher education institutions, where professional development is planned to extend systematically the learning provided in the preservice program, where the preservice and in-service learning opportunities are linked to the curriculum to be taught. Such a system would require some time to show its value—always a challenge in the hurry-up-and-improve atmosphere of U.S. schools. But it could be tried, for example, in one of the many failed districts scattered around the country that are subject to state takeover as a last resort.
Conclusion
There are many sources of frustration about educational research. Most of the critiques focus on rigor or quality. I am more worried about relevance and utility—the gulf between the science we do and the science we need to improve educational outcomes. In the world of medicine, the equivalent would be moving from providing evidence for the germ theory of infection to ensuring that all medical practitioners wash their hands at the patient’s bedside—a notoriously difficult task (Gawande, 2010).
Frustration arises not just from the vast enterprise of second-rate research but also from the existence of multiple unexploited findings about things that work, solid knowledge that is not being used. For example, we know that:
High-quality early childhood programs staffed by professionals with rich language skills promote children’s development.
Test-based accountability systems create policy incentives to lower test difficulty, generate sometimes irresistible temptations to cheat, and almost universally encourage dedication of time to unproductive test preparation rather than real teaching.
Letting adolescents start school later in the day would increase their attendance and their receptiveness to learning. Thus, letting the bus schedule trump instructional concerns is not a good idea.
Ensuring teachers can plan and work together improves their practice as well as their professional commitment and satisfaction.
Classrooms in which authentic discussion occurs regularly, even if the discussion bouts are relatively brief, generate greater learning and higher student engagement
Typically designed and delivered school-level professional development is largely a waste of time and money. Professional development focused on specific learning/teaching challenges tied to specific curricular efforts, rather than on general pedagogical principles, is much more likely to be effective.
This is just a short list of research findings justifying changes in practice that could be authorized today and made tomorrow. It may seem like making any of those changes is a daunting challenge, but we should take heart from recalling a few of the ways in which education has improved over the past 150 years:
We no longer assume that beating children improves their learning.
We no longer assume that anyone who knows how to read can teach reading.
We no longer assign any adult who speaks English to teach it as a second language.
We are now convinced that punishing children for speaking their native language is both cruel and educationally counterproductive.
We recognize the value to students of learning in settings where they encounter a diversity of ethnicities, races, cultures, and religions, all treated with equal respect and honor.
Progress on these points suggests that perhaps we can move forward on making changes that may now seem challenging or even controversial but that in 20 years’ time may appear just as straightforward and obvious as these now do.
My goal in this article has been briefly to review the arguments for the value of an authentic SERP-like approach to educational research and improvement, consider what we have learned from some of the efforts undertaken so far in accordance with the principles of PEER first articulated by SERP, and propose some “unfinished business”—some initiatives that might change the way education and education research is done. It is important to emphasize that PEER is not an alternative to rigor, though it may impose an altered definition of rigor. Randomized controlled trials and sophisticated analyses of large data sets are entirely consistent with PEER principles, but attention to the gold standard of rigor should not require compromising other important principles, such as true partnership and learning from practice.
Footnotes
This article was adapted from the Wallace Foundation Distinguished Lecture given on April 4, 2014, at the American Educational Research Association annual meeting in Philadelphia, Pennsylvania.