Strengthening STEM Instruction in Schools: Learning From Research

Tweet

What works in STEM teacher professional development? New research points to curriculum integration, collaboration, and focus on how students learn.

Key Points

Introduction

U.S. students’ poor performance in science, technology, engineering and mathematics (STEM) is a widely documented problem. Sixty percent of children in the Unites States fail to meet standards for mathematical proficiency in fourth grade (National Assessment of Educational Progress [NAEP], 2017), while 62% fail to meet benchmarks in science (NAEP, 2015). These problems are particularly acute for students from less advantaged backgrounds, who disproportionately attend underresourced schools with less experienced teachers. Poor performance in mathematics and science limits children’s access to advanced STEM coursework in the later grades, and subsequently may block students’ access to future STEM careers.

Teacher professional development (PD) and curriculum change are two critical policy levers to improve children’s STEM learning. As such, teacher PD and new curriculum materials constitute major investments for districts. Districts spend between 1% and 6% of their budgets on teacher PD (see, for example, Corcoran, 1995; Miles, Odden, Fermanich, & Archibald, 2004; Miller, Lord, & Dorney, 1994), and most U.S. teachers participate in PD at least once annually (National Center for Education Statistics [NCES], 2001). Furthermore, the market for instructional materials is estimated to be in the neighborhood of US$12 billion per year (Cavanagh, 2015). The size of these investments dictates understanding when PD and curriculum interventions are most effective. From a policy standpoint, distilling the empirical research evidence is particularly important because the Every Student Succeeds Act requires that districts receiving Title I funds must adopt “evidence-based interventions,” including programs and strategies proven to be effective in raising student achievement.

The current article highlights the contemporary research base that bears on how to improve classroom instruction and subsequent student learning in STEM. Specifically, we survey the empirical evidence on two categories of STEM programs that intervene in the classroom: (a) teacher PD, which aims to improve teachers’ knowledge and skills, and (b) new curriculum, which aims to improve the richness of the content available to students. We do so based on the premise that without changes to classroom processes—what is happening in the classroom each day between teachers, students, and content—student learning is unlikely to improve.

This article is organized as follows. First, we review prior efforts to understand the impacts of teacher PD and curriculum improvement interventions on student learning in STEM. Second, we highlight recent work examining the characteristics of STEM PD and curriculum programs that are associated with stronger impacts on student achievement outcomes. In particular, our large-scale statistical meta-analysis of the recent experimental and quasi-experimental evaluations shows these programs influence student achievement. Third, we provide recommendations for policy, research, and practice stemming from the recent research on STEM instructional improvement initiatives, with the goal of strengthening teacher practice and bolstering students’ opportunities to advance in STEM.

How Teacher PD and Curriculum Programs Influence Student Learning: Theory of Action

What is the process by which teacher PD and curriculum programs influence student learning? Informative prior work (Scher & O’Reilly, 2009) posited a framework for understanding how elements of teacher PD influence teacher learning, teacher practices, and student outcomes. First, the resources available for the PD, such as duration, format (e.g., workshop, study group, workshop plus coaching), and structural characteristics, may contribute to PD outcomes. Second, the district, school, and political climate, or contexts of environmental support, may help or hinder the success of the PD. Third, PD activities may be more or less coherent in their alignment with school curricula, standards, and teachers’ beliefs and practices. Fourth, PD content may vary in terms of mathematics versus science focus, instructional strategies taught, a focus on how students learn, as well as fidelity of implementation.

So far, the Scher-O’Reilly model focused on PD alone. Because we seek to understand the role of curriculum materials, we amend their model to include characteristics of curriculum materials interventions that are hypothesized to contribute to student learning. A similar theoretical framework, developed by the National Research Council for curriculum materials, suggests that curriculum-materials evaluations should take into consideration how each component influences student outcomes (Confrey, 2006): program components (such as curriculum content and design elements), implementation components (such as teacher support and pedagogical elements), and secondary components (such as contextual and systemic factors). These curriculum elements may include the program’s focus on active student learning (such as in labs), alignment with National Council of Teachers of Mathematics (NCTM)/ National Science Teaching Association (NSTA) standards, and amount of teacher PD associated with the materials.

PD (and by extension, curriculum materials) is hypothesized to influence student outcomes via a three-phase process (Scher & O’Reilly, 2009). The first phase, immediate outcomes, includes changes in teacher content knowledge, pedagogical knowledge, and attitudes or beliefs. The second phase, intermediate outcomes, includes changes in teachers’ instructional practices. Finally, the third phase, long-term outcomes, includes changes in students’ attitudes and achievement. Here, we focus primarily on the research evidence on the third phase, student outcomes. We return to the issue of proximal outcomes on teachers below.

Impacts on Distal Outcomes: Student Achievement

In recent decades, increasingly sophisticated methods have honed researchers’ and policymakers’ understanding of the impacts of teacher PD and curriculum programs on student learning. Prior to 2002, studies often relied on expert views and teacher self-report data to distinguish best practices in teacher PD and curriculum development (e.g., Borasi & Fonzi, 2002; Elmore, 2002; Porter, Garet, Desimone, Yoon, & Birman, 2000). Although these studies illuminate teachers’ and experts’ opinions, they lack causal evidence about the impacts of the recommended features on student outcomes, limiting their conclusions.

Several reviews during this period ventured to synthesize the existing empirical evidence on teacher PD and curriculum improvement programs; however, these early efforts often found that the evidence base for making recommendations was sparse. Searching from 1990 through 2004 yielded only 18 studies for a meta-analysis of mathematics and science PD programs (Scher & O’Reilly, 2009), despite relaxing methodological criteria from an initial focus on randomized experiments to the eventual inclusion of any relevant study with a comparison group. The small sample size enabled examining only a limited number of variables as potential moderators of program impact. Multiyear PD interventions were more effective than those that transpired over a single year in mathematics, but not in science. Also, programs that focused on both content and pedagogy together were more effective on average than programs that focused on content or pedagogy alone. Another survey of the research encountered a similarly small pool of studies (Yoon, Duncan, Lee, Scarloss, & Shapley, 2007). In a search of the literature through 2003, only nine studies met the What Works Clearinghouse’s stringent criteria for methodological rigor, despite the inclusion of math, science, and English language arts interventions. Programs with teachers participating in 14 or more hours of PD yielded larger average impacts on student achievement than programs offering shorter-duration PD. Yet again, the small pool of studies allowed few conclusions about the characteristics associated with effective teacher PD. Synthesists from this period frequently advocated that more research reviews should be conducted at a future time when more rigorous studies had been conducted.

However, with changes to the Institute of Education Sciences (IES) funding guidelines in 2002, researchers seeking federal IES grants were newly required both to use research methods that support causal inference and to measure the impacts of such interventions on student outcomes. In the same period, the National Science Foundation’s (NSF) grantmaking also began to reflect similar interests, particularly around interventions focused on teacher learning and the use of novel curriculum materials in classrooms. Several influential reports in the early and mid-2000s also called for increased use of rigorous study designs in education research and specifically in research on teaching (Confrey & Stohl, 2004; Raudenbush, 2008; Shavelson & Towne, 2001).

Partially in response, many new studies of instructional programs are more methodologically rigorous, including many classroom- and school-level cluster randomized experiments. At the outset, our own work hypothesized that these new studies should provide a thicker evidence base than had been available in the past for understanding the effectiveness of teacher PD and curriculum programs in STEM. These studies also examined interventions using a range of new formats, such as online teacher PD and peer coaching.

Prompted by this larger and more rigorous pool of studies, a new meta-analytic review synthesized updated research evidence on instructional improvement programs in STEM (Lynch, Hill, Gonzalez, & Pollard, 2019). First, we sought to understand how effective STEM PD and curriculum improvement programs are on average at improving student outcomes. Second, we asked whether specific program or study characteristics were associated with differential effects of STEM PD and curriculum improvement programs on student outcomes. Third, given the persistent and severe challenges of improving STEM achievement for children from less socioeconomically advantaged backgrounds, for this review we also explored whether PD and curriculum interventions are more or less effective in high-poverty settings. We summarize the results of this new meta-analysis, as well as our analysis exploring program settings, below.

The goal of our meta-analysis was to identify all relevant studies published in 1989 or later and focused on classroom-level STEM instructional improvement through PD, curriculum materials, or both. Studies for review came from library electronic reference databases, research organizations’ websites, lists of IES and NSF STEM grant awardees, and prior reviews. Including only randomized or strong quasi-experimental design yielded 95 studies that met these criteria, notably more than reviewers found two decades ago. Newly developed codes captured features of the instructional improvement programs evaluated in the studies. Statistical meta-analytic techniques estimated the average impact of STEM instructional improvement programs on student achievement in math and science. Subsequent analyses then examined whether instructional improvement programs with specific features posted larger effects on student achievement. (See the full report for details; Lynch et al., 2019.)

Typical STEM PD and curriculum interventions improved student achievement by about 8 percentile points (+0.21 standard deviations). Considering only standardized assessments and excluding intervenor-developed outcomes, the interventions improved student achievement by about 3 percentile points (+0.07 standard deviations). Researchers have estimated that a typical teacher who increases student achievement on standardized assessments by +0.14 SD creates marginal gains of roughly US$7,000 per child in present value future earnings (Chetty, Friedman, & Rockoff, 2014). Extrapolating from this, the estimated average test score impact of STEM PD and curriculum interventions of +0.07 SD would be expected to yield approximately US$3,500 in present value future earnings per student.

The next analyses investigated relationships between PD characteristics and student impact estimates. Programs that included both teacher PD and curriculum materials together were more effective on average than programs that included only one. For programs that incorporated only PD or only new curriculum materials, a typical student in the treatment group could be expected to rank about 6 percentile points higher than a typical student in the control group. However, for programs that included both PD and curriculum materials together, a typical treatment group student could be expected to score about 10 percentile points higher than a typical control group student.

Several specific program foci and formats were associated with stronger than typical program impacts. Student outcomes were significantly larger among programs that focused on how to use curriculum materials, and among programs that focused on improving teachers’ content and pedagogical content knowledge and/or how students learned the content, relative to programs that did not have these focus areas. Regarding formats, on average, programs in which teachers participated in PD alongside other teachers in their school, same-school collaboration, as well as programs that included PD with implementation meetings, yielded outcomes that were larger than programs that lacked these components. Implementation meetings were normally convenings of teachers—after initially trying curriculum materials or ideas from PD—to troubleshoot and discuss ways to improve use of the intervention. Programs that included a summer workshop component were also more effective, on average, than programs that did not meet during the summer; perhaps the summer workshop format provided a prospective, concentrated dose of training at a time when demands on teachers’ time and attention were lower, thus bolstering teachers’ take-up of the program during the school year.

On the contrary, compared with programs where the PD was conducted entirely in person, programs where the PD included an online component had smaller (but still positive) impacts, on average.

No significant relationship emerged between the length of teacher PD (measured either as contact hours or as the time span over which the PD was spread) and impacts on student achievement outcomes. Likewise, the specific activities that teachers engaged in during the PD, (e.g., developing their own curriculum materials or lessons, observing teaching demonstrations) did not have any significant relationships with student impact estimates. Nor did significant associations emerge between specific features of the curriculum materials (e.g., whether they provided implementation guidance, included lab work and hands-on activities; or the intended length of time for students’ use of the materials) and the magnitude of impacts on student achievement outcomes.

Impacts on Proximal Outcomes: Teacher Practices

As the core purpose of schooling is student learning, analyses have so far focused on instructional improvement programs’ impact on student learning in STEM. However, for PD programs to change student learning, they must first alter teachers’ classroom practices (e.g., Hamre et al., 2012). Most curriculum programs also have the goal of altering teachers’ actions. A handful of meta-analyses pull together the evidence on whether PD and curriculum interventions result in these teacher-level changes. Synthesizing the research base on how teacher PD in mathematics and science affects teachers’ attitudes and practice, Scher and O’Reilly (2009) found average positive impacts, but only seven studies met the inclusion criteria. Examining the impact of teacher coaching interventions on instructional practices—operationalized using classroom observation scores from measures of teachers’ pedagogical practices, teacher–student interactions, student engagement, and/or classroom climate—a recent study found a moderate pooled effect size (Kraft, Blazar, & Hogan, 2018); however, the analysis could not evaluate the measurement properties of the teacher practice measures, and did not differentiate between outcomes on standardized versus researcher-developed teacher practice measures. Finally, a meta-analysis of PD interventions (Garrett, Citkowicz, & Williams, 2019) reviewed the literature on the impacts of programs aimed at improving classroom practice—defined broadly to include domains such as classroom management, classroom environment, and instructional practices. Although generally positive, impacts on classroom observation outcomes were variable.

Policy Insights: Strengthening STEM Instruction in Schools

Teacher PD and new curriculum can both change teachers’ practices and improve students’ STEM achievement. The estimates of overall impacts of STEM instructional improvement programs on student achievement are based on experimental and strong quasi-experimental evidence, lending confidence to the overall results. The analyses examining relationships between specific program features and the strength of program impacts are correlational, rather than experimental in nature; as a result, these conclusions about moderators are less definitive, but rather suggestive of promising practices. Nonetheless, the findings from dozens of recent evaluations of STEM instructional improvement programs align with prior work (e.g., Boyd, Grossman, Lankford, Loeb, & Wyckoff, 2009; Cohen & Hill, 2001) and are consistent with several recommendations for practice. Specifically, the findings point toward three recommendations for practices that should be implemented in schools and districts:

Although researchers have sometimes conceptualized teacher PD and new curriculum materials in isolation, we studied them together. This allowed us to observe that on average, instructional improvement programs that included both components were more effective than those that included PD or curriculum alone. When teachers attend PD that is not grounded in the specific content they teach, they may find it more difficult to map the PD’s techniques onto their own lessons. On the contrary, curriculum materials by themselves may not be enough to improve teachers’ instruction. Programs such as Building Blocks provide an example of how PD that is explicitly tied to the curriculum can be particularly effective. Building Blocks is an effective preschool mathematics intervention that involves both new curriculum materials and ongoing teacher PD focused on learning about the materials, targeted coaching, and practicing implementation (Clements et al., 2011).

Affording teachers opportunities to study the curriculum they will teach and to improve their own STEM content and pedagogical content knowledge is beneficial for student learning. What is more, these opportunities should be intellectually engaging to teachers and built on teachers’ motivation to learn, rather than being didactic or overly prescriptive. The type of content knowledge needed for teaching is specialized. Given this, programs that offer teachers opportunities to increase their content knowledge specifically in the service of broader pedagogical goals, such as exposing student thinking, may be more effective than programs that simply deliver content (Kennedy, 2016).

Carving out time for teachers to collaborate with their same-school colleagues and to meet regularly to discuss program implementation is linked to better program outcomes. Teachers learn from one another, and social motivation and the opportunity to share expertise may also encourage teachers to persist in implementing ambitious new practices. Yet, the content of these meetings matters: not all group learning experiences are created equal. For example, simply providing groups of teachers with data on their students’ test scores or classroom practices, without guidance on what the group should do with this information, does not appear promising. On the contrary, focusing group work specifically on curricular content, employing discussion leaders to steer conversations on track, and ensuring that group work is intellectually rich are promising avenues to support effective group participation (Kennedy, 2016).

Improving student achievement in STEM is a critical concern for policy. Students’ early success in mathematics and science coursework is clearly related to their long-term educational attainment and career prospects, and foundational math and science skills also enhance civic competence and general quality of life. Strategic investments in STEM instructional improvement programs work, and they can improve schools’ productivity and students’ long-run opportunities.