Sources and consequences of teacher attrition in large-scale intervention impact studies

Abstract

In this paper, we critically examine the way in which scholars have traditionally defined and problematized attrition. Through a series of examples of large-scale intervention impact studies, we share insights about the sources and consequences of attrition that expand our notion of how and why attrition occurs. We also discuss potential steps for anticipating, mitigating, and responding to attrition in the dynamic context of schooling. By expanding our understanding of attrition, we hope to engage the field in further dialogue that could lead to policies and practices that might lessen the potential impacts not only on our ability to conduct research, but also our ability to advance the learning of teachers and their students.

Keywords

Teacher attrition intervention impact studies teacher mobility threats to validity

Over the past several years, we (authors) have been engaged in large-scale and ambitious research studying the impacts of educational interventions on teacher and student learning in various contexts. Given the varied and constantly changing contexts of each of our studies, retaining teacher participants in our studies was an issue that needed to be addressed to complete the study successfully. Our goal in coming together was to explore the sources and consequences of teacher attrition from research studies and to develop strategies that may help plan for and mitigate such attrition in our research. We came to recognize, however, that some causes of attrition are unavoidable and unpredictable, despite the best-laid plans.

Based on our conversations, reflections, and synthesis, we present new insights on the sources and consequences of teacher attrition from research studies, illustrating teacher attrition through a series of examples grounded in our own research. We acknowledge that our examples likely do not exemplify all the possible sources and consequences of attrition. As such, we don’t view our perspectives as an exhaustive examination of the topic. Further, all of our examples are research studies in US-based science education. However, we contend that our syntheses and insights are informative for the larger education research field. In this paper, we critically examine the way in which scholars have traditionally defined and problematized attrition. We seek to promote a more proactive approach to minimizing the negative impacts of teacher attrition on research studies, where such attrition is viewed as a natural feature of school contexts that should be acknowledged in the design of interventions and in efforts to scale up and replicate prior research.

Descriptions of teacher attrition in the research literature

In 2001, Richard Ingersoll published an often cited and influential paper describing the rates in which teachers were either moving schools or leaving the profession, referring to either scenario as teacher turnover (Ingersoll 2001). Ingersoll’s idea of tracking teachers as “movers” and “leavers” was adopted by subsequent reports of the National Center for Education Statistics (NCES) in the US (Keigher 2010; Goldring et al. 2014), where researchers documented from nationally representative samples the percentage of teachers moving schools or leaving the profession. Alternatively, White and Arzi adopted the phrase “teacher attrition” and defined it as the loss of research participants in the context of longitudinal studies and noted its associated threats to validity (White and Arzi 2005). We note the meaning of “teacher attrition” has become entangled in the literature, and the phrase is currently used to refer to both teachers leaving the profession and the loss of teacher participants in research studies. We emphasize that while a teacher leaving the profession might coincide with their withdrawal or attrition from a research study in which they are participating, the two are separate phenomena. In the following section, we define attrition in a similar fashion to White and Arzi, extending the definition and discussion of attrition uniquely to the context of group design intervention impact studies (i.e., two groups: one experimental, one control/comparison).

Definition of attrition for intervention impact studies

In this paper, we use the term “teacher attrition” to reference when a teacher no longer participates in a research study or intervention activities, doesn’t contribute outcome data to the impact study, or both. We note here that external reviews of study evidence (e.g., for internal validity) can require a more limited definition than ours to calculate attrition rates. For example, review standards such as those used by the US-based What Works Clearinghouse (WWC; Institute of Education Sciences 2022a), define the overall attrition rate for randomized control trials as the proportion of randomly assigned individuals who do not contribute outcome data to the impact study (see What Works Clearinghouse Standards Handbook v.4.1; Institute of Education Sciences 2022b). This WWC attrition rate definition does not include teachers who provide outcome data but do not fully participate in study or intervention activities (e.g., do not provide all requested data, do not attend all or some program events).

While this conceptualization of attrition is useful and sufficient for WWC purposes, it does not address the complexity and nuance of why attrition may be occurring-- which can inform research design and attrition mitigation strategies. Further, the WWC definition does not seek to address how attrition is influenced by study design or characteristics of the contexts (e.g., schools, school systems) in which data are collected. This paper seeks to elaborate on these latter two aspects of how attrition is conceptualized.

Sources of teacher attrition

While it has become customary in the last decade to track teacher attrition rates for intervention impact studies where it is relevant, including recent studies (e.g., Cunningham et al., 2020; Yang et al., 2020), much less has been written on the sources of teacher attrition aside from its voluntary nature and origin in factors related to untenable requirements of participating in the intervention or study (e.g., data collection requirements). This limited view of attrition fails to account for the full range of possible attrition causes. Specifically, it overlooks the influence of external factors unrelated to an individual’s willingness or commitment to a study, such as a school’s or school district’s withdrawal from a study or change in the work context of the teacher.

This need for a broader view of teacher attrition and its sources was reinforced by our (authors) collective experiences. In re-conceptualizing attrition, we find it helpful, therefore, to place attrition sources into two categories: (1) those related to implementing the intervention and/or participating in the study of the intervention and (2) those unrelated to either the intervention or study but more reflective of the context in which the research was being conducted. These two categories require notably different strategies for mitigating the effects of attrition -- the former related to the implementation or study design and the latter to understanding the contexts (policy, workplace, and professional) in which the research takes place, including partnerships formed with schools and districts. We discuss these two categories and strategies for mitigating the impacts of attrition below.

Attrition related to implementation of an intervention or study design

There are several sources of attrition that can be mapped directly to the requirements of implementing an intervention or participating in an impact study of that intervention. Similar sources of attrition have also been noted in studies at the student level, where participating schools sometimes find research data collection to be too onerous, data collection methods to be problematic, and even being unaware of what they are committing to when agreeing to participate (Dawson et al. 2018). Specific to our work, teacher attrition can also result when a new intervention requires materials, supplies, or technology tools that exceed available resources. Attrition might occur due to study design because of random assignment methods to determine who receives the intervention. For example, teachers and schools often choose to participate in an intervention study because the intervention has shown promise in prior research. Despite prior knowledge of the 50% probability of not receiving the intervention, we have observed that participants can become demoralized by the assignment result and decide to leave the study. Another study design characteristic that may influence attrition rates is attempting to track the impacts of an intervention in a longitudinal fashion by collecting teacher outcome data several years after the conclusion of an intervention. The longer the longitudinal perspective of the study, the more likely it is that individual teachers or entire schools or districts may not be able or willing to participate in follow-up research. This problem is exacerbated if the longitudinal perspective straddles critical career junctures, such as the transition from student teaching to an early career teaching assignment where teachers may seek employment in different school districts or states. In addition, the nature of the outcome measures being collected might influence the likelihood that mid-study attrition will occur due to unanticipated decline in district, school, or student-level consent/assent to participate. Particularly, this can happen when the impact or implementation study requires collecting video footage of classroom activities, which can be invasive.

Attrition related to research context

In our own work, we also identified several sources of attrition that were outside of the immediate context in which the intervention or the research was conducted. These sources of attrition reflected unanticipated factors unrelated to the research design. By this, we do not mean contextual factors that might weaken an individual teacher’s commitment to participating in a study, such as personal challenges (see Kubitskey et al., 2012), but rather factors associated with the school or district, including teacher and/or administrator mobility, the cycle of policy change, and student mobility. For instance, there may be mid-study changes to academic standards and/or state testing content priorities that make an intervention being studied no longer aligned with school district goals and teacher attrition can result from this misalignment.

Teacher turnover has been discussed extensively as a source of attrition in research studies (Ingersoll 2001). The movement of teachers across assignments, grades, schools, and districts is collectively referred to as “ambient positional instability” (API) (Bowdon and Boruch 2014). Discussions of teacher turnover have most often focused on teachers who leave the profession (leavers) or those who move to a different school or district (movers); however, Blazar (2015) points out that an understudied aspect of teacher mobility is the movement of teachers within school from one grade level to another. As reported by Jacob and Rockoff (2011), over 20% of teachers across the US switch grade levels from 1 year to the next. Interventions that are multi-year and/or grade-level specific are particularly susceptible to this type of teacher mobility as a source of attrition.

Administrator turnover has also garnered attention, though primarily in relation to its influence on teacher turnover and/or teacher shortages (e.g., Boe et al. 2008). While administrative turnover may lead to teacher turnover, administrative turnover can also affect agreements between schools/districts and researchers regarding implementation and study of interventions. In this manner, administrative turnover can directly affect attrition. Administrative turnover may also be related to another source of attrition--the cycle of policy change. While some studies may focus on studying the implementation and impacts of a particular policy change, other studies may experience attrition when policy change occurs mid-study. Policy changes, like changes in administration, may mark a shift in priorities that lead to withdrawal from a study, or changes in the implementation of an intervention that affect the data collection activities.

Finally, student mobility is another factor that varies from context to context. While student mobility can be the focus of research in and of itself (e.g., impact on school dropout rates, student achievement, etc.), student mobility can also affect teacher attrition, particularly in longitudinal studies where student outcomes are a variable of interest. Student mobility can reduce the exposure a student has to an intervention, reduce the number of students in the sample, and limit the ability to link teacher data to student data. Thus, while not related to the teacher’s own willingness to participate in the study, this may result in the need to exclude some teachers from analysis due to incomplete or missing data. To some extent, these effects can be mitigated through choice of tracking strategy (see Ellickson et al. 1988); however, studies that seek to work with populations that are associated with higher rates of student mobility are inherently susceptible to teacher attrition as well.

Potential consequences of teacher attrition

There are potential consequences of attrition that can be particularly problematic for researchers engaged in either small- or large-scale intervention impact studies. Below, three major consequences are described.

Loss of intervention reach

Although the primary goal of large-scale impact studies in education is often to examine the efficacy or effectiveness of a well-defined intervention, it is also quite common that the premise of the study is that the intervention has already shown promise of efficacy in studies of prior implementations. Therefore, to the extent that the intervention demonstrates efficacy or effectiveness again in the subsequent impact study, one can think of teacher attrition as loss of intervention reach to teachers, and to students as well.

Loss of statistical power

Teacher attrition compromises the power to detect statistically significant intervention effects on teacher outcomes, and in some cases student outcomes, as teachers who leave a study usually discontinue engagement or data collection from their students. While statistical significance is only one consideration when interpreting an intervention impact, it is essential to understanding how likely it is that a similar intervention effect would be observed across future study replications. Similarly, attrition could result in disqualifying a study from inclusion in a meta-analyses. For instance, in determining inclusion criteria for a meta-analysis of the impact of teacher professional development (PD) studies on student achievement, Yoon and colleagues (2007) reported high attrition as an important disqualification factor.

Bias in impact estimates via differential attrition

In a randomized control trial, as the difference in attrition rates across study conditions (e.g., treatment vs. comparison) increases, the likelihood that this differential attrition will bias the impact estimate also increases (see Dawson et al. 2018). For example, although in theory random assignment would distribute baseline (e.g., pre-test) achievement levels equally across treatment conditions, if proportionally more lower achieving teachers or students leave the study in the comparison condition than in the treatment condition, the mean post-test outcome in the comparison condition will be artificially inflated and result in an attenuated impact estimate.

US national data on teacher mobility

To set the context for attrition rates in the study examples that follow in the next section, we draw on the work of Goldring et al. (2014), who conducted work with US nationally representative samples of teachers. Specifically, Goldring and colleagues examined teacher mobility using the 2011-12 administration of the National Center for Education Statistics (NCES) Schools and Staffing Survey (SASS) and 2012-13 administration of the NCES Teacher Follow-Up Survey (TFS). Based on survey sample records, they found that approximately 16% of teachers had left the profession or responded to the survey while working in a different school. We highlight this overall percentage as it could serve as a baseline expectation for teacher attrition in an intervention impact study, as either moving schools or leaving the profession would result in losing a teacher from a study. It should be noted that this is a conservative baseline expectation for attrition, especially in light of the current teacher shortages and social and economic conditions. In fact, emerging research suggests that the COVID-19 pandemic does have a marked effect on teachers leaving the profession (Dugger 2021). Further, the authors found teachers moving schools or leaving the profession at higher rates from urban and economically disadvantaged contexts (see Tables 1 and 2). The reasons for these attrition rates might overlap but not entirely with the reasons cited in the examples that follow.

Table 1.

Number and percentage of public school teacher stayers, movers, and leavers (2012–13) – adapted from Goldring et al. (2014).

Number					Percent
Year	Total teachers	Stayers	Movers	Leavers	Stayers	Movers	Leavers
2012–13	3,377,900	2,846,500	271,900	259,400	84.3	8.1	7.7

Table 2.

Percentage of public school teacher stayers, movers, and leavers by selected school characteristics (2012–13) – adapted from Goldring et al. (2014).

		Percent
School characteristic	Stayers	Movers	Leavers
Community type
City	82.3	9.7	7.9
Suburban	84.9	7.8	7.3
Town	86.3	7.3	6.4
Rural	84.6	7.0	8.4
School level
Primary	83.9	8.7	7.4
Secondary	85.5	6.5	7.8
Combined	78.1	10.3!	11.6!
Percent of students approved for free or reduced-price lunches
0–34	87.2	5.9	6.9
35–49	85.5	7.0	7.5
50–74	84.3	8.6	7.1
75 or more	78.0	12.2	9.8

! = Interpret data with caution. The standard error is greater than 30% of the estimate.

Examples of teacher attrition in research settings

In the following set of examples, we document a variety of sources and consequences of teacher attrition. Each subsequent example presents sources of teacher attrition, as well as both common and unique consequences of teacher attrition. Table 3 provides a summary of the four examples.

Table 3.

Summary of goals and designs for the four examples.

Project	Goals	Research design
Videocases for science teacher analysis plus (ViSTA plus)	Investigate the impact of a videocase-based, analysis-of-practice science methods course on both new teachers and their students Investigate the impact of ViSTA plus on students in the classrooms of participating student teachers and during participants’ first year teaching classrooms	Quasi-experiment with treatment (ViSTA plus methods course) and comparison groups of preservice teachers Participants were tracked starting at the beginning of the methods course, into their student teaching assignment, and through their first year of teaching
Quality elementary science teaching (QuEST)	Investigate the impacts of a “practicum-based” professional development program for elementary teachers that included experiences working with students in a summer program	Quasi-experiment that followed three grade-level cohorts of teachers from within the same schools who participated in practicum-based PD and comparison cohorts of teachers from other schools within the same districts that participated in a non-practicum PD program already identified as effective
Promoting science among English language learners (P-SELL)	Investigate the impacts of a fifth grade curricular and professional development (PD) intervention aimed at improving the science achievement of all students, with a focus on English language learners (ELLs)	A cluster randomized control trial was conducted. Within each of the three school districts, 22 elementary schools were randomly selected to participate in the study, yielding a total of 66 schools. Within each district, half of the selected schools were randomly assigned to the treatment group and half to the control group, yielding a total of 33 treatment schools and 33 control schools across the three districts. All fifth-grade science teachers in the 66 schools participated in the study.
SCALE-uP (scaling-up curriculum for achievement, learning, and equity project)	Document the implementation, scale-up, and sustainability of three “highly rated” middle school science curriculum units across a large and diverse school system over 6 years Identify the conditions under which the curricular interventions might prove to be effective for student learning of targeted science ideas	A quasi-experimental research design was used to test each unit in 15 classrooms and compare outcomes from those 15 treatment classrooms to that of 15 business-as-usual classrooms

Example 1: Videocases for science teacher analysis plus (ViSTA Plus)

The ViSTA Plus (Videocases for Science Teaching Analysis Plus) program, which is based on the successful Science Teachers Learning from Lesson Analysis (STeLLA) in-service professional development program (Roth et al., 2011, 2019; Taylor et al., 2017) aimed to follow pre-service teachers from their science teaching methods course into student teaching, and through their first year of teaching. This is a volatile time in the life of new teachers and represents a critical juncture in their development. Navigating new roles and responsibilities all while adhering to a research program proved quite challenging for many participants.

The original study design, implemented between 2014 and 2016, included a power analysis (Optimal Design Version 3.01; Spybrook et al., 2011), that estimated that we would need 70 teachers in the study (across both treatments) to detect an effect size for student achievement outcomes as small as 0.17 standard deviations. We increased this sample size to 84 (20%), to account for attrition. In our proposal, we acknowledged that this study was likely to experience high levels of attrition following the student teaching year, as teachers transitioned into their professional life. In the first year of study, we successfully recruited 120 participants (53 ViSTA Plus and 67 Business-as-Usual) from two participating universities. This represented 30% more teachers than the power analysis indicated would be needed to account for teacher attrition. Stipends were awarded to compensate teachers for their time.

At the end of the first year of the study, we were successful in keeping all 120 participants in the study, retaining adequate power to investigate the impact of the ViSTA Plus approach on teacher outcomes. As preservice teachers moved into the student teaching year, the number of participants decreased to 69 (42% attrition) active participants before student teaching began. A total of 51 teachers (43%) completed the student teaching year (27 treatment and 24 comparison). By the beginning of year 3 of the study, 47 participants continued to communicate with us about their continued participation in the study. However, the number of participants that continued to keep us informed on their future employment prospects dwindled. At the end of year 3 of the study we retained only 22 participants (82% attrition) and 19 of 22 were in the ViSTA Plus treatment group. While there was attrition in the final year from both groups, the differential attrition between treatment and comparison conditions in the final year of the study (first year of teaching) appeared to be related to assignment to the comparison group, rather than to other contextual factors.

As we tracked participants who left the study, we identified a few themes around participant attrition that fell into three categories: 1) unknown, 2) personal, and 3) external. Fifty-five percent of the attrition in the study had unknown sources. Among this 55%, 29% provided a formal withdrawal, usually by email, but did not state a reason for leaving the study and the other 26% of participants who left for unknown reasons simply stopped responding to our attempts to make contact. Approximately 19% of participants who left the study and offered a formal reason cited personal reasons for leaving (medical concerns, overwhelm, child care concerns, moving out of the area, or leaving the preservice program/not planning to teach). Another 10% that we included in the personal category are participants who stayed in contact with the project but delayed their student teaching and ultimately withdrew from the study. Finally, 16% of attrition occurred for reasons outside the teachers’ control. These included difficulties with district administration approving the research (7%), changes in teaching assignment (8%), and uncooperative-cooperating teachers in the student teaching year (1%).

The most important effect of attrition in the ViSTA Plus study was the lack of data available in year 3 of the study. While we were able to investigate change over time within the ViSTA Plus group in year 3, so few teachers were left in the comparison group that we could not compare teacher or student outcomes across conditions in the first year of teaching. While we expected significant attrition over the course of the project, this level differential attrition from the business-as-usual comparison was not anticipated. This reduced our 3-year quasi-experiment to just 2 years, decreasing the robustness of our findings.

Example 2: Quality Elementary Science Teaching (QuEST)

The QuEST (Quality Elementary Science Teaching) program sought to understand the impact of a practicum-based professional development model on teacher and student learning. Based on our past experience offering the PD as part of a statewide Improving Teacher Quality Grant, we anticipated little attrition (i.e., teachers withdrawing from either the PD program or the research). In the seven prior years that we offered the program, no teachers withdrew from our research, and only one could not complete the program due to health reasons. We made what we believed to be conservative estimates for participation, based on past recruitment success and initial conversations with district partners, to include 20 teachers in each cohort (both practicum and non-practicum) annually for an anticipated sample size of approximately 120 teachers over 3 years (see Table 4). At the onset, we had commitments from four school districts and anticipated meeting the target numbers of participants.

Table 4.

Example 2 (QuEST): Anticipated number of participants annually.

District	Number of schools	Number of teachers
A	10	20
B	2	4
C	6	12
D	4	8
Total	22	44

Our forecasted attrition seemed to be accurate. While, as before, we had no participants withdraw from the research, we did have several teachers who failed to complete the PD program for various reasons that were unrelated to the project, but that prohibited them from continued participation in the research. This included one teacher in Year 1 who became pregnant and took maternity leave, two teachers in Year 2 who took early retirement, and 1 teacher in Year 3 who did not return to the workshop in week 2 for personal reasons. This small level of attrition, by itself, was insufficient to negatively affect the power of our analyses.

However, we encountered several additional unanticipated sources of attrition that were unrelated to teachers’ own decisions and willingness to participate in the program and research. Rather, these were related to the dynamic work environment of K-12 education, and specifically changes to district administration, principal, and classroom teaching assignments. Just after funding was awarded and 1 month prior to the first scheduled summer institute, our largest partner district underwent a change in leadership. The previous district curriculum coordinator had secured commitments from principals to participate in the program; however, the new coordinator prioritized another initiative and asked principals to commit to that. As a result, this drastically reduced the number of schools from that district that would fulfill their commitment to participate in the program from 10 to 3 (equivalent to 14 teachers or 35% of our annual sample). The unanticipated attrition at this late juncture meant we fell short of our targeted sample size in year one, despite adding three additional districts (see Table 5). Similarly, changes to principal leadership led to inconsistent school-level commitment and participation from year to year within three district partners (A, C, F in Table 6). Finally, the number of teachers participating from each school varied from year to year, due to fluctuations in FTE and the number of classrooms at each grade level within a school (Districts D & E), open positions that were not yet filled with new hires at the time of our summer institute (District F), and the decision to departmentalize science within grades 4 and 5 (District A). Though we fell short of our cohort numbers overall, recruiting only 87% of our targeted participation, we did have comparable numbers of teachers within each of the treatment conditions from year to year (see Table 7).

Table 5.

Example 2 (QuEST): Planned versus actual number of participants in Year One.

District	Planned no. of schools	Planned no. of teachers	Actual no. of schools	Actual no. of teachers
A	10	20	3	6
B	2	4	2	4
C	6	12	2	6
D	4	8	4	8
E	0	0	2	4
F	0	0	2	2
G	0	0	2	2
Total	22	44	17	32

Table 6.

Example 2 (QuEST): Total Number of participants across all 3 years.

	Year 1		Year 2		Year 3
District	Schools	Teachers	Schools	Teachers	Schools	Teachers
A	3	6	3	7	2	3
B	2	4	2	4	2	6
C	2	6	4	9	3	6
D	4	8	4	9	4	11
E	2	4	2	1	3	5
F	2	2	1	1	2	3
G	2	2	2	3	2	4
Total	17	32	18	34	18	38

Table 7.

Example 2 (QuEST): Distribution of participants across cohorts and years.

Cohort	Year 1	Year 2	Year 3
Practicum	16	17	19
Non-practicum	16	17	19
Total	32	34	38

We encountered a second type of attrition during the period between participants’ attendance of the summer PD institute and their classroom implementation of what they learned. While all the teachers accounted for in Table 6 completed the summer institute, not all of them continued in their anticipated teaching assignments the following academic year. As shown in Figure 1, post-program mobility for the academic year following participation in the PD institute was 30% for 3rd grade and approximately 10% for grades 4 and 5-- this downward trend continued in subsequent years, with the result that approximately only 20% of our original cohort of 3rd grade teachers was still teaching 3rd grade. This type of attrition was not experienced in our prior efforts, because the PD was open to teachers of any grades K6, allowing them to continue their involvement from year to year regardless of any grade-level reassignment or within-school mobility they experienced. This program design was a better fit for the dynamic environment of schools, but a grade-level specific design was a better fit for conducting a more rigorous investigation of impacts on student learning. Thus, our research design inadvertently led to the attrition we experienced. Even if teachers had been willing to continue their involvement in the PD, they were no longer eligible to participate in the research due to their grade level reassignment.

Figure 1.

Example 2 (QuEST): Percentage of teachers remaining at grade level following PD participation.

The most immediate consequence of attrition in the QuEST research program was the lack of data regarding teachers’ implementation and related impacts on student learning. The variation we experienced in participation rates affected statistical power for our 3-year longitudinal analyses. While we were able to investigate changes in teacher knowledge resulting from participating in the summer institute, teacher reassignments meant that we could not investigate the differential impacts of the practicum-based PD on teacher practice and student learning.

Example 3: Promoting science among english language learners (P-SELL)

Promoting Science among English Language Learners (P-SELL) was a fifth grade curricular and professional development (PD) intervention aimed at improving the science achievement of all students, with a focus on English language learners (ELLs). P-SELL consisted of a comprehensive, stand-alone, year-long curriculum for fifth grade students and teachers, as well as PD workshops for teachers focusing on curriculum implementation. The effectiveness study of P-SELL was examined in three school districts. According to the National Center for Education Statistics (2006), District A was designated as urban, District B as urban/suburban, and District C as urban/suburban. The three districts encompassed a wide range of racial, ethnic, socioeconomic, and linguistic diversity.

During the 2012-2013 school year, District A had 103 elementary schools, District B had 44 elementary schools, and District C had 125 elementary schools. A cluster randomized control trial was conducted. Within each of the three school districts, 22 elementary schools were randomly selected to participate in the study, yielding a total of 66 schools. Within each district, half of the selected schools were randomly assigned to the treatment group and half to the control group, yielding a total of 33 treatment schools and 33 control schools across the three districts. All fifth-grade science teachers in the 66 schools participated in the study.

During the 3-year implementation of the intervention, we encountered both leadership changes in the school districts (e.g., superintendent turnover in the three school districts) and the schools (e.g., principal turnover). Despite these changes, all three school districts and all 66 schools were retained in the study (i.e., no district- or school-level attrition). Further, during a given year, most of the 5^th grade teachers in the treatment group participated in teacher PDs and completed teaching the P-SELL curriculum.

Yet, teacher attrition was high across the 3-year period (see Tables 8–10), resulting in lower fidelity of implementation via lower teacher exposure to the program, and the potential for overly pessimistic impact results. During the first year (2012–2013), the project involved 123 teachers in the treatment group and 135 teachers in the control group. During the second (2013–2014), almost 40% of the teachers were new to the study. During the third year (2014–2015), again almost 40% of the teachers were new to the study. Our study indicates that even when the implementation and study design are carried out fully, teacher attrition occurs for various reasons over multiple years of an intervention (Bowdon and Boruch 2014; Ingersoll 2001; Jacob and Rockoff 2011). Teacher attrition can be quite frequent in urban schools with economically disadvantaged students, racial/ethnic minority students, and ELLs as school-level leadership decides teacher assignments in response to perceived pressing needs. In the case of P-SELL, urban schools with low school grades responded to sanctions from the state and the district according to the state accountability system (e.g., state takeover of “failing” schools) and decided teacher assignments based on “school grades” in any particular year. Under such pressure, some teachers left to other schools within the same school district.

Table 8.

Example 3 (P-SELL): Number of teachers in first year of implementation (2012–2013).

	Treatment	Control	Total
Teachers	123	135	258

Table 9.

Example 3 (P-SELL): Returning and new teachers in second year of implementation (2013–2014).

Teachers	Treatment (n = 137), %	Control (n = 141), %	Total (n = 278), %
Returning for a 2^nd year	59	63	61
New to P-SELL study	41	37	39

Table 10.

Example 3 (P-SELL): Returning and new teachers in third year of implementation (2014–2015).

Teachers	Treatment (n = 122), %	Control (n = 128), %	Total (n = 250), %
Returning for a 3^rd year	42	39	40
Returning for a 2^nd year	19	24	22
New to P-SELL study	39	37	38

Example 4: SCALE-uP (Scaling-up curriculum for achievement, learning, and equity project)

SCALE-uP was a research project funded by the Interagency Education Research Initiative (administered by the National Science Foundation) to study the scale-up of evidence-based interventions—in this case, middle school science curriculum materials. SCALE-uP’s aim was to document the implementation, scale-up, and sustainability of three “highly rated” middle school science curriculum units across a large and diverse school system over 6 years (Lynch et al., 2012). This research began not only as a study of the effectiveness of three highly-rated, guided-inquiry, middle science curriculum units, but also was an exploration of large, diverse school district’s capacity to scale-up and sustain change. This required teacher professional development and supplying print and laboratory materials for each unit. A quasi-experimental research design was used to test each unit in 15 classrooms and compare outcomes to that of 15 business-as-usual classrooms. For each unit, approximately 10 teachers in five different middle schools implemented each unit, with similar numbers of teachers in 15 comparison classrooms in five different middle schools matched for demographic similarity. Teachers implementing the new units participated in summer professional development workshops focused on how to teach the guided-inquiry curriculum units, and were supported by science curriculum coaches (master teachers from the district) as they implemented the units during the school year. Teachers in comparison classrooms had professional development on the same targeted concepts and worked in generally well-equipped classrooms. The study design anticipated complex interactions between the intervention materials and the school and district contexts (Cohen and Ball 2007).

While the 8^th grade chemistry-focused unit showed positive impacts on student learning, the state’s science standards underwent two radical reorganizations and a decision to drop the state-level science assessments. In response to this shifting policy environment, the school district decided to completely revise the middle school science scope and sequence across 3 years of middle school. “Chemistry” was shifted from 8^th to 6^th grade. Because no 6^th grade teachers had participated in professional development for SCALE-uP, in a practical sense, there was virtually 100% teacher attrition by the end of the study for this unit, despite the fact that the findings showed that the unit was effective and equitable for 8^th grade students.

The 7^th grade unit, which focused on understanding reasons for Earth’s seasons, was unexpectedly problematic. When the curriculum materials were evaluated at small scale, the student assessment data indicated that the unit was not as effective as the business-as-usual comparison condition. Consequently, the intervention unit could not be scaled-up and this portion of the study was curtailed. For study purposes, teacher attrition was again 100%, but for very different reasons than for the 8^th grade chemistry unit.

Our initial studies of the 6^th grade unit on motion and force produced results that were unexpected, but rich with insights. Classroom observations by researchers and district science coaches indicated that teachers were not implementing the unit as intended. Consequently, a replication was done with yet another set of teachers with an emphasis on high fidelity of implementation to the unit’s instructional characteristics. Interestingly, with high fidelity, the 6^th grade unit proved to be effective and equitable; there was a demonstrable relationship between implementation fidelity and student outcomes when fidelity was monitored using tools developed for the study. Although the results were positive at small scale in this replication, the unit’s sustainability and scale-up in the district was nonetheless clearly in peril. The unit was viewed by teachers as time-consuming, cumbersome, and difficult to manage (Lynch et al., 2012).

Although the findings for SCALE-uP were complicated as a set of implementation studies, there were even more important contextual forces at work outside of the scope of the study that affected the scale-up and sustainability of the interventions. These contextual factors overshadowed the relative successes or failures of the three curriculum units and impeded the school district’s ability to use the findings to improve teaching and learning in science.

First, the school district’s K-12 science coordinator who was also the study’s co-Principal Investigator left the school district. He was extremely knowledgeable and was well-regarded by science teachers. He was replaced by a co-PI who was a master science teacher and an advocate for the study, but who had limited authority over curricular matters in the school system. Then, she was replaced by a K-12 science coordinator who was new to the district and unfamiliar with the study’s history. Because SCALE-uP’s findings were nuanced, this lack of continuity in leadership diluted the capacity of the district science program to learn from the study.

Second, the state completely reorganized its science standards two times over the course of the 6-year SCALE-uP study, as well as dropping the required state-level assessments that had relied on guided inquiry performance assessments. This left the school district with no state-level incentives or pressures for student mastery of the skills or content in the prior standards (although arguably, conservation of matter, motion and force, and the earth’s seasons would be important to know). The school district responded to perceived changes at the state level by developing a new middle school curriculum that would serve students better than the old scope and sequence. That was also eventually abandoned. That period of time that the study was characterized by nearly chaotic instability of science learning goals at the state and district levels, an instability unrelated to the research itself.

Third, at the teacher level, science teachers who participated in SCALE-uP had to simultaneously contend with shifting state standards and tests, revisions to the district science scope and sequence, and rapid turnover of K-12 science coordinators. By the end of the SCALE-uP study, there was a high level of middle science teacher attrition within the district. The redesign of the district’s middle school science scope and sequence had moved science topics to different grades levels; teachers would need to learn new material, or learn to teach students in different grades, or both. The SCALE-uP study scrambled to keep up with professional development challenges for teachers new to the study in this period of flux. Another factor was that the master teacher coaches trained for the study had the highest turnover rate. These experienced teachers were leaving their schools, the district, and sometimes, the state.

Because of the scope of this study and the capacity of the school district, teacher attrition from the study was not the major problem for implementing the units. As teachers left, they were replaced by teachers who were oriented through summer professional development, science teacher coaches, and the participation of the research team’s focus on implementation fidelity. The units were implemented at small and large scale. SCALE-uP’s success as a research project was well known; it has been cited as a rigorous study by the What Works Clearinghouse and other meta-analyses.

But if the question is not only about implementation, but also about scale-up and sustainability of the intervention units at the school district level, then ultimately, none of the units were sustained. In this sense teacher attrition was 100%. First, all three units required a commitment by teachers and support by the school district in myriad ways. The interventions required a high capacity system. Second, the context of the school district was a major factor in sustainability of the units and in teacher turnover. While the district’s capacity was high—there were strong science teachers and systems in place to provide them with what they needed to teach and innovate -- the science curricular leadership was too much in flux to formulate a cohesive plan and follow it through. This was due in part to the third factor, contextual forces at the state level that were unstable and unclear. The changing standards over a 6-year study were an enormous challenge, and the lack of state science assessments rendered the entire system nearly directionless. One important side effect of the instability in leadership, vision, and documented learning goals for middle school science was the high rate of teacher attrition.

For SCALE-uP, the problem was less about teacher attrition than about attrition of leadership and direction at the district and state levels that left the middle school science curriculum temporarily rudderless. The study goals were frustrated by the inability to communicate what was learned from the study to teachers and administrators who might use the ideas to improve the system. After all, two of the three units had proven effective, especially for students in groups that were often underserved in science in the district. SCALE-uP was able to suggest why these units were effective and had the potential to close achievement gaps. The important lesson from SCALE-uP was about understanding the interactions between curricular interventions and the environment or context of a school district, and about how systemic instability led to a strong research study, but little uptake at the district organizational level.

Discussion

This paper presented four examples of intervention impact research that experienced attrition to varying degrees and with sources and consequences that expand the current discussion of attrition in the research literature. These are summarized in Figure 2.

Figure 2.

Sources and consequences of teacher attrition.

While attrition varied across the studies, the rates were generally larger (sometimes much larger), than the 16% teacher mobility estimates from Goldring et al. (2014). This suggests that, over and above the attrition rate a study designer might expect solely from teachers leaving the profession, there are clearly other factors specific to research study participation that become influential. We argue that the importance of this topic is due to the need for identifying and mitigating these research-specific factors. Having noted this, we also acknowledge that some of the teachers who left the four studies described in this paper, left for unknown reasons. As such, we cannot be sure that their rationales for leaving their studies were solely research-specific. For these teachers, the reasons could have been related to the general demands of teaching or some hybrid set of demands. For example, general working conditions at the school often provide little time for teachers to do anything (including study participation) beyond their contracted duties.

Each of the examples generated insights about how future research designs might address the challenges of inevitable teacher attrition. Some of the insights were unique to select studies, while others applied across studies. From these examples, we converge on themes that blend implications at multiple levels. In this section, we discuss strategies within four categories: 1) study and intervention design, 2) participation incentives, 3) data systems, and 4) policy alignment.

Study and intervention design

Intervention research should be designed around the realities of public schools with teacher attrition from the study, and possibly the profession, as a given. Although attrition can be minimized, the attrition rates from our studies suggest that moderate rates should be expected and preparation for the effects of attrition is prudent. As seen across all four examples, a given study may suffer from multiple attrition sources, including teachers leaving the profession, changing schools, and switching grade-levels within a school. System level sources that eventually trickle down to loss of teacher participants include principal or superintendent mobility, and changes in standards or accountability testing programs.

Attrition can affect the reach of promising interventions, as well as compromise the statistical conclusions and/or potential generalizability of an intervention impact study (see QuEST discussion). In addition, high attrition rates can depress the impact of an intervention, by compromising its’ fidelity of implementation (see P-SELL Discussion). Exposure to an intervention is one of several criteria for measuring fidelity of implementation (see the review of literature in O’Donnell 2008). Researchers contending with high attrition rates should determine how to address the complication that the reported impact of any intervention may depend on exposure.

Attrition can be challenging for evaluating the efficacy of an intervention, or how the intervention works under very favorable conditions. High attrition rates in urban schools raise the question about the very notion of efficacy under suboptimal conditions. What does it mean to conduct an efficacy study when “very favorable conditions” for rigorous studies (e.g., low rates of teacher mobility or attrition) are not feasible? Moreover, the desire to conduct more rigorous research can privilege education contexts less prone to attrition--which are often those of higher socio-economic status. By not anticipating attrition as a natural feature of the school context, we may not be designing interventions that match the realities of dynamic school environments. This suggests that the very nature of our interventions and research should be reconsidered in terms of how they are designed to meet the needs and realities of schools. The intervention research community needs to devise new approaches to addressing teacher attrition, especially in the context of increasing student diversity and academically rigorous content standards. We caution that this is not a matter of simply adding more participants to account for attrition, as ‘good data’ may not reflect the realities of the diversity of school context. Rather, this involves both the design of interventions that will be implemented in changing conditions for teacher participants in schools, and the design of research that takes into account the consequences of attrition. If PD is the focus of an intervention, then that PD might be designed to account for attrition by focusing on what is portable and replaceable, enabling teachers who flow in and out of the program to continue or to be oriented more easily (see SCALE-uP discussion). For research design, attrition might be addressed by considering participant exposure to intervention or dosage effects (see PSELL discussion, above). New approaches are needed especially for longitudinal designs that seek to study teachers at and across critical junctures and transitions over time, as illustrated in the cases of VISTA-Plus and QuEST and in the work of White and Arzi (2005). As teachers find themselves in new contexts and circumstances within the duration of a study, there may be little infrastructure and support for continued participation. As a result, tracking participants in these transitions remains a challenge.

Incentives

There is evidence that monetary incentives can be effective in retaining teachers in the profession (e.g., Feng and Sass 2017), and monetary compensation was provided to teachers for their time toward participation in the professional development programs described in this paper. However, such compensation is not necessarily the only attractive incentive to participate in the associated research activities. The nature of the intervention and its ability to be implemented with relative high fidelity can influence teachers’ motivation to participate in the study, as well as make it easier for teachers entering the study context to adopt the intervention. It is critical to design interventions that are feasible to implement, are adaptable to student populations, and include the supports necessary for teachers to successfully implement without extraordinary resources. By the same token, we recognize that some promising new interventions may not be easily implemented because they incorporate innovative teaching techniques that lie beyond teachers’ current practices. These interventions warrant rigorous study as well and clearly there is a balance to be obtained. It is also important to consider the burden of participation in research studies, for both teachers and students. Intervention researchers can also play a role in reducing the burden of participation by administering only the most critical study measures or incremental incentives that are connected to successful data collection.

Incentives extend beyond individual teachers and should be considered at the school and district level as well. We assert that intervention studies will likely experience less attrition if interventions are research-based and have prior strong evidence of effectiveness; are aligned with content that matters (e.g., state/national standards or accountability test foci); and provide the necessary resources for sustaining implementation beyond the scope of the study. Yet, alignment with local district goals and priorities is also essential. As seen in the SCALE-uP example, when goals and priorities shift, research can be difficult to continue without ad hoc changes to the research plan. At the partnership and planning stages, stakeholders should explicitly consider potential consequences should they decide not to continue participating in either the research or the intervention, as well as subsequent scale-up and how decision-making at the school or district level (e.g., reassignment of teachers) might undermine the investment made in providing professional development to teachers (see QuEST discussion).

Data systems

Contextual stability is highly important but often difficult to predict. To support the reach and sustained implementation of effective programs, we propose new administrative data systems that allow tracking of teacher mobility, retention, and orientation to new interventions. Some states in the US are developing such systems. For example, in Colorado’s new Educator Identification System (EDIS), teachers are assigned a unique identification number that follows them throughout their service in Colorado public schools (e.g., Colorado Department of Education, 2020). Such data systems are necessary to allow researchers to track and predict teacher mobility for attrition planning, and can help determine “dosage” effects for teachers joining or leaving school-level interventions.

Data systems can also allow for the design of interventions that better align to the dynamic nature of the school context. For example, within-school teacher mobility, referred to as Ambient Positional Instability (API) (Bowdon and Boruch 2014) is a potential source of attrition that could be accounted for if data were available. While API rates have been documented at the secondary levels in STEM fields, API rates at the elementary level are not as widely available, though studies suggest shifts in grade levels for elementary teachers are common (cf. Rayes et al., 2016; Frisone et al., 2016; Chao et al., 2016). As seen in the QuEST study, API rates in partner districts could have informed the design of the PD to enable teachers who move from grade to grade or school to school within district to continue as part of the intervention and study. Furthermore, access to these types of data can allow for stronger predictions about the applicability or potential replicability and scale up of interventions across sites with differential rates of API. Schools are not controlled laboratories, and there is a need to test the robustness of interventions that succeed in one context against others.

Alignment with state standards and/or district priorities

While there is a need for evidence of the effectiveness of programs to assist with funding or other policy decisions, there are also policy-related barriers to building the evidence base. For example, as seen in the SCALE-uP study, flux in district leadership and loss of vision for a study’s goal can bring the use of a promising program into question. Moreover, changes in state standards and assessments might mean that a once promising intervention is no longer a good match for students and teachers. Perhaps more stability in standards would help curriculum developers, as well as teachers and school districts, to better design research around promising interventions. More coherence between common standards or large-scale testing would decrease the likelihood of an intervention becoming incompatible with a school or district’s instructional needs in the middle of an impact study, resulting in the loss of study participants and weakening the impact of important findings.

Policymakers and researchers need to engage in dialogue surrounding policies and practices that might lessen the potential effects of attrition on teacher and student learning. Each of our studies experienced a reduction in intervention reach, statistical power, and incurred possibly biased impact estimates from different rates of attrition across treatment conditions. Likewise, the same factors that make it difficult to conduct effectiveness studies also pose difficulties for full, faithful, and sustained implementation of potentially beneficial interventions. Our studies illustrate the difficulty of conducting intervention impact research in schools and call attention to the potential challenges for teachers and students, such as the “re-novicing” of grade-level reassigned teachers (QuEST program), the shifting demands on teachers (SCALE-Up, VISTA-Plus), and difficulty in long-term engagement in professional development (P-SELL).

Conclusion

In closing, we argue that the way in which scholars have traditionally conceptualized attrition is limited and needs to be reconsidered in light of the education contexts in which science education research is conducted. Clarifying ambiguity between attrition that refers to teachers leaving the profession versus leaving research studies is a necessary step towards addressing the current gap in the literature regarding teacher attrition from large-scale intervention studies. That is, while there is a body of literature on student attrition from research studies (Rickles et al., 2018; Seftor, 2016), there is little or no parallel work on teacher attrition from research studies, neither that documents typical teacher attrition rates, nor that discusses the corresponding attrition sources and consequences. Beyond our efforts in this paper to document such sources and consequences through authentic examples, more work is greatly needed.

The insights gained from our research studies described here highlight the importance of not only expanding our consideration of the sources and consequences of attrition, but also critically examining how we anticipate, respond to, and account for attrition in future work. These insights are particularly useful in contemporary research contexts, where teacher shortages are a timely challenge faced by our K-12 research partners. Educational interventions that include teacher professional development could likely have limited effects in a school, district, or state education context if they are designed as multi-year longitudinal doses of treatment to teachers who will not be static. Rather, studies should be designed to assume high levels of teacher movement and necessitate professional development that is flexible and repeatable for teachers new to the project. Further, when a research study is proposed to a school district, school district staff may see the study primarily as a vehicle for professional development, rather than as an opportunity to learn from the research to improve education within the system more generally.

We encourage further dialogue that could lead to policies and practices that might lessen the potential effects of teacher attrition not only on our ability to conduct rigorous research, but also our ability to advance the learning of teachers and their students, as well as the school, district and beyond. We acknowledge that the most important effect of teacher attrition may not be its negative effect on research, but rather the unrealized benefits to system learning. Essentially, investments in the professional development of teachers are squandered if circumstances prevent teachers from fully implementing what they learn, or the system from using research findings.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Joseph A Taylor

References

Blazar

(2015) Effective teaching in elementary mathematics: Identifying classroom practices that support student achievement. Economics of Education Review 48: 16–29.

BoeCook

Cook

Sunderland

(2008) Teacher turnover: Examining exit attrition, teaching area transfer, and school migration. Exceptional Children 75(1): 7–31.

Bowdon

Boruch

(2014) Teacher churn in Missouri’s five biggest cities, 2005-2014: A Briefing. CRESP Working Paper and Briefing Series. http://repository.upenn.edu/gse_pubs/264

Chao

Park

Boruch

(2016) Ambient positional instability among Illinois teachers, AY 2007-2012: A briefing. CRESP Working Paper/Briefin g . http://repository.upenn.edu/gse_pubs/395

Cohen

Ball

(2007) Educational innovation and the problem of scale. In: Schneider

McDonald

(eds) Scale-up in Education, Volume II: Ideas in Principle. Lanham, MD: Rowman and Littleeld Publishers.

Colorado Department of Education (2020). The educator identification system https://www.cde.state.co.us/datapipeline/yr_edis

Cunningham

Lachapelle

Brennan

, et al. (2020) The impact of engineering curriculum design principles on elementary students' engineering and science learning. Journal of Research in Science Teaching 57(3): 423–453.

Dawson

Yeomans

Brown

(2018) Methodological challenges in education RCTs: reflections from England’s Education Endowment Foundation. Educational Research 60(3): 292–310.

Dugger

(2021). In: McConnell

J. R.

(ed) COVID-19 and the Risk of Teacher Attrition in the United States”. [Austin Peay State University. https://search.proquest.com/dissertations-theses/covid-19-risk-teacher-attrition-united-states/docview/2546057568/se-2

10.

Ellickson

Bianca

Schoeff

(1988) Containing Attrition in School-Based Research. Evaluation Review 12(4): 331–351.

11.

Feng

Sass

(2017) The impact of incentives to recruit and retain teachers in “Hard-to-Staff” Subjects. Journal of Policy Analysis and Management 37(1): 112–135.

12.

Frisone

Hooks

, et al. (2016) Ambient positional instability among core subject Arkansas public school teachers: Interim report. http://repository.upenn.edu/gse_pubs/394

13.

Goldring

Taie

Riddles

(2014) Teacher attrition and mobility: Results from the 2012-13 teacher follow-up survey. First look. NCES 2014-077. National Center for Education Statistics.

14.

Ingersoll

(2001) Teacher turnover and teacher shortages: An organizational analysis. American Educational Research Journal 38(3): 499–534. DOI: 10.3102/00028312038003499.

15.

Institute of Education Sciences (2022a) What works clearinghouse. https://ies.ed.gov/ncee/wwc/

16.

Institute of Education Sciences (2022b) What works clearinghouse standards handbook, v.4.1. https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Standards-Handbook-v4-1-508.pdf

17.

Jacob

BA.

Rockoff

(2011) Organizing schools to improve student achievement: Start times, grade configurations, and teacher assignments. http://www.brookings.edu/∼/media/research/files/papers/2011/9/organization jacob rockoff/092011_organize_jacob_rockoff_brief

18.

Keigher

(2010) Teacher Attrition and Mobility: Results from the 2008–09 Teacher Follow-Up Survey” (NCES 2010-353). Washington, DC: U.S. Department of EducationNational Center for Education Statistics. http://nces.ed.gov/pubsearch

19.

Kubitskey

Vath

Johnson

, et al. (2012) Examining study attrition: Implications for experimental research on professional development. Teaching and Teacher Education 28(3): 418–427. DOI: 10.1016/j.tate.2011.11.008

(2012) A retrospective view of a study of middle school science curriculum materials: Implementation, scale‐up, and sustainability in a changing policy environment. Journal of Research in Science Teaching 49(3): 305–332.

21.

O’Donnell

(2008) Fidelity of implementation to instructional strategies as a moderator of curriculum unit effectiveness in a large-scale middle school science quasi-experiment. Dissertation Abstracts International 68(08). UMI No. AAT 3276564). Retrieved February 2, 2008, from ProQuest Dissertations and Theses Database.

22.

Rayes

Lee

, et al. (2016) Ambient positional instability among teachers in Minnesota public schools: 2010-2011 to 2014-2015. http://repository.upenn.edu/gse_pubs/400

23.

Rickles

Zeiser

West

(2018) Accounting for student attrition in power calculations: benchmarks and guidance. Journal of Research on Educational Effectiveness 11(4): 622–644. DOI: 10.1080/19345747.2018.1502384.

, et al. (2011) Videobased lesson analysis: Effective science PD for teacher and student learning. Journal of Research in Science Teaching 48(2): 117–148.

, et al. (2019) Comparing the effects of analysis-of-practice and content-based professional development on teacher and student outcomes in science. American Educational Research Journal 56(4): 1217–1253. DOI: 10.3102/0002831218814759.

26.

Seftor

(2017) Raising the bar. Evaluation Review 41(3): 212–239. DOI: 10.1177/0193841X16665023 .

, et al. (2011) Optimal design plus empirical evidence: documentation for the “Optimal Design” software. http://wtgrantfoundation.org/resource/optimal-design-with-empirical-information-od

28.

Taylor

Roth

Wilson

, et al. (2012) The effect of an analysis-of-practice, videocase-based, teacher professional development program on elementary students' science achievement. Journal of Research on Educational Effectiveness 10(2): 241–271. DOI: 10.1080/19345747.2016.1147628.

29.

WhiteArzi

Arzi

(2005) Longitudinal studies: Designs, validity, practicality, and value. Research in Science Education 35(1): 137–149.

30.

Yang

Porter

Massey

, et al. (2020) Curriculum-based teacher professional development in middle school science: A comparison of training focused on cognitive science principles versus content knowledge. Journal of Research in Science Teaching 57(4): 536–566.

31.

Yoon

Duncan

Lee

SWY

, et al. (2007) Reviewing the evidence on how teacher professional development affects student achievement. Issues and answers.” REL 2007-No. 033. Regional Educational Laboratory Southwest (NJ1).