Does Teacher Learning Last? Understanding How Much Teachers Retain Their Knowledge After Professional Development

Abstract

Teacher professional development (PD) is seen as a promising intervention to improve teacher knowledge, instructional practice, and ultimately student learning. While research finds instances of significant program effects on teacher knowledge, little is known about how long these effects last. If teachers forget what is learned, the contribution of the intervention will be diminished. Using a large-scale data set, this study examines the sustainability of gains in teachers’ content knowledge for teaching mathematics (CKT-M). Results show that there is a negative rate of change in CKT after teachers complete the training, suggesting that the average score gain from the program is lost in just 37 days. There is, however, variation in how quickly knowledge is lost, with teachers participating in summer programs losing more rapidly than those who attend programs that occur during school years. The implications of these findings on designing and evaluating PD programs are discussed.

Keywords

professional development teacher knowledge evaluation

Introduction

Classroom teachers are one of the most important in-school factors contributing to student learning (Aaronson, Barrow, & Sander, 2007; Chetty, Friedman, & Rockoff, 2013; Rivkin, Hanushek, & Kain, 2005). To improve teacher quality, federal and local education agencies have devoted substantial resources to teacher training and development (Correnti, 2007; Darling-Hammond & Sykes, 1999). Many of these professional development (PD) programs are designed based on different theories of how teachers learn and how teachers use the acquired knowledge in practice (Kennedy, 2016; Richardson & Placier, 2001). While these theories have different conceptions of teacher learning and teaching, they all perceive teacher knowledge to be an important outcome of PD and necessary condition for improving teaching practices and student outcomes (Cochran-Smith & Lytle, 2001; Guskey, 2000; Putnam & Borko, 2000; Vermunt, 2014).

As a result of the teacher learning research and the recent development in teacher assessments, teacher knowledge measures have been increasingly used as an outcome in empirical studies that examined the effectiveness of teacher PD programs (Carney, Brendefur, Thiede, Hughes, & Sutton, 2014; Hill & Ball, 2004; Koellner & Jacobs, 2015; Phelps, Kelcey, Jones, & Liu, 2016; Polly, Neale, & Pugalee, 2014). The logic for using measures of teacher knowledge as the outcome is straightforward and based on a basic set of causal assumptions: effective PD should lead to improvements in teacher knowledge, this knowledge should then have a positive impact on instruction, and in turn, effective instruction should have an impact on student achievement (Desimone, 2009; McCutchen et al., 2002; Scher & O’Reilly, 2009; Yoon, Duncan, Lee, Scarloss, & Shapley, 2007).

However, there is one critical issue with this set of assumptions that has received limited attention in research on teacher PD programs. For gains in knowledge to have an effect on teaching, these knowledge gains must persist. This may not hold because cognitive research on knowledge acquisition shows that newly learned knowledge is malleable and often impermanent rather than stable and unchanging (Grider, 1993; Vosniadou, 2007). One important finding is that for knowledge to be maintained, it needs to be applied or used in a relevant setting. Without use, knowledge tends to move from being active and available to inactive and difficult to retrieve (Arthur, Bennett, Stanush, & McNelly, 1998; Stothard & Nicholson, 2001). Another finding is that the application of new knowledge is inconsistent and this knowledge is often modified (Farr, 2012; Spiro, Collins, Thota, & Feltovich, 2003). These two processes, which can be labeled as forgetting and adaptation, can lead to treatments on knowledge and skills having immediate desired outcomes but undesirable long-term outcomes.

These ideas from cognitive research could also be applied to the field of teacher learning and teacher education. Just as teachers could develop knowledge, they can also forget what they have learned from the training programs. This is especially the case if the programs do not have sustained support or if knowledge is not applied or used in daily classroom instruction (Abdal-Haqq, 1995; Birman, Desimone, Porter, & Garet, 2000; Guskey & Yoon, 2009). If teachers lose the knowledge needed for effective teaching, then the PD might have little to no effect on student learning, even when there are substantial initial program effects on teacher knowledge outcomes.

The study of teacher knowledge loss, or what is often called knowledge decay, is important for both designing and evaluating teacher PD programs. On one hand, studying teacher knowledge decay is essential to properly understand the effectiveness of teacher PD programs. So far, prior research evaluating teacher PD has found mixed evidence on the effect of these programs. While many studies have found significant impacts, there is still plenty of evidence showing that PD programs have null effects on teacher and student outcomes (Battey et al., 2013; Bell, Wilson, Higgins, & McCoach, 2010; Blank, De las Alas, & Smith, 2008; Garet, Heppen, Walters, Smith, & Yang, 2016; Gersten, Taylor, Keys, Rolfhus, & Newman-Gonchar, 2014; Heller, Daehler, Wong, Shinohara, & Miratrix, 2012; Kraft, Blazar, & Hogan, 2018; Santagata, Kersting, Givvin, & Stigler, 2010). Although different possibilities have been hypothesized, knowledge decay may be one important reason for the null results. Specifically, if decay occurs after teachers complete the training, the estimated effects would be sensitive to how soon after a program the teacher knowledge outcome measures are administered. On the other hand, the estimated knowledge decay could also inform the design of the teacher PD programs. For example, in organizational contexts, estimated decay rate is used to optimize the scheduling of refresher trainings (Healy et al., 1998; Rail Safety and Standards Board [RSSB], 2011; Wisher, Sabol, Ellis, & Ellis, 1999). Similarly, in the context of teacher PD, the decay rate could be useful for deciding the number of training sessions and appropriate training intervals for maintaining the knowledge and skill level required for effective teaching.

Recognizing the importance of studying teacher knowledge decay, this study represents one of the initial efforts to understand knowledge decay by estimating the decay rate of content knowledge for teaching in mathematics (CKT-M) after completing PD. There are two reasons why the CKT measures are preferred to other teacher knowledge measures in estimating teacher learning and knowledge decay. First, these assessments are designed to measure a specialized type of content expertise used in teaching, including analyzing students’ work, determining the validity of a mathematical argument, or using proper representations for teaching particular math ideas (Ball, Thames, & Phelps, 2008; Baumert et al., 2010). As a result, they are more likely to capture teacher learning and knowledge decay associated with teacher PD programs that aim to improve teaching. Second, there is mounting empirical evidence showing that assessments of CKT are associated with teaching quality as well as student learning (Baumert et al., 2010; Hill, Rowan, & Ball, 2005; Hill, Umland, Litke, & Kapitula, 2012). These findings suggest that CKT is a key mediator that is closely connected with both program treatment and student learning and thus should be included for examining the effectiveness and the underlying mechanism of the PD programs. Based on a set of PD programs for mathematics teachers that took place during 2008 to 2013, this study investigates the following two research questions:

Research Question 1: What is the decay rate of CKT-M after teachers complete PD programs?

Research Question 2: If teachers forget what they learn, does the decay rate vary by program features and participant characteristics?

The rest of the article is organized as follows. In the next section, we review prior studies that examine the sustainability of teacher learning from PD programs and the extent to which the sustainability varies across different conditions. Next, we describe the analytic sample and empirical strategy used to estimate teacher knowledge decay. We then present results which suggest that there is significant knowledge decay after program treatment ends. We conclude with a discussion of these results and their implications for using teacher knowledge as an outcome to evaluate and study the improvement of teacher PD programs.

Review of Prior Research on the Sustainability of PD Outcomes

Substantial research has been conducted to examine the effectiveness of mathematics PD programs (Bell et al., 2010; Hill & Ball, 2004; Koellner & Jacobs, 2015; Polly et al., 2014). However, an equally important question is to what extent the effect persists after the completion of the programs. So far, studies that examined the sustainability of PD outcomes were based primarily on small samples of teachers, and some of them used self-report outcome measures. These studies provided preliminary evidence on how much teachers sustained their knowledge or teaching practices, and discussed possible factors that were associated with the sustainability. In this section, we summarize evidence from these studies and provide a framework for understanding the conditions that may affect the sustainability of PD outcomes.

One of the early studies that examined the sustainability of the PD outcomes was conducted by Supovitz, Mayer, and Kahle (2000). The program promoted inquiry-based instruction for science and math teachers. Using self-report survey data, the study found that teachers’ attitude and use of inquiry-based teaching practices sustained 3 years after the PD. This pattern holds good for teachers of different genders, race groups, grade levels, and school types (public vs. private). In another study, Boston and Smith (2011) found that a PD program improved math teachers’ ability to select high-level instructional tasks, but not all teachers were able to sustain what they learned from the training. Those teachers who showed sustainable changes demonstrated higher inclinations to reflect on their practices and to apply strategies learned from the training to their own teaching practices. Finally, Sandholtz and Ringstaff (2016) investigated how contextual factors influenced the sustainability of learning from a PD that provided science assistance to K-2 teachers. Based on survey and interview data from 15 teachers, the study found that school-level factors, such as administrative and collegial support, were important to teachers in sustaining science instruction. High turnover of principals and other colleagues often made it challenging for teachers to maintain the instruction quality over time.

In addition to the sustainability of teaching practices, several studies provided evidence on the presence of knowledge retention, which is more close to what is examined in this study. Franke, Carpenter, Levi, and Fennema (2001) followed up with teachers 4 years after they completed a PD that focused on improving understandings on students’ mathematical thinking. In addition, Steinberg, Empson, and Carpenter (2004) conducted a case study of one teacher who participated in a similar PD. Overall, findings from both studies suggested that after the completion of the training, engagement with student thinking continued to improve teacher knowledge of students’ mathematical thinking and the quality of math instruction. Evidence on knowledge retention or decay was also available in other subject areas. Goldschmidt and Phelps (2010) estimated the impact of teacher PD and its sustainability using measures of content knowledge for teaching in the area of elementary reading instruction and found that learning gains in comprehension and word analysis were not sustained 6 months after the training was completed. Roth et al. (2011) examined the impacts of two types of science PD programs and their sustainability on teachers’ content knowledge. Their results showed declining trends in knowledge for teachers in the program that focused only on deepening science knowledge (content only program), but teachers who participated in the program with analysis of teaching practice sustained their knowledge gain afterwards.

To summarize, prior research found variation in how much teachers sustained their learning after completing PD programs. Factors related to the content and design of training, to individuals, and to the school context in which teachers worked have been shown to influence sustainability of teacher learning. In terms of training content, PD programs with sustainable impact were usually embedded in teacher work and promoted understanding of students’ thinking (Franke et al., 2001; Roth et al., 2011; Yoon et al., 2007; Zehetmeier, 2008). The sustainability may also be affected by teacher characteristics, their attitude toward the program, and their ability to internalize what they learn during the training (Boston & Smith, 2011; Steinberg et al., 2004). Finally, teacher learning was more likely to sustain for those working in a stable and well-supported school environment (Gaikhorst, Beishuizen, Zijlstra, & Volman, 2017). Principal support and teacher collaboration increased the sustainability of PD impact on teachers (Sandholtz & Ringstaff, 2016).

In this study, we expand on previous small-scale research by estimating the decay of content knowledge for teaching using a large sample of K-12 math teachers from multiple PD programs representing diverse contexts in the United States. In addition, the data set contains a set of variables on teacher characteristics and program features, which allows us to examine empirically how some of the factors identified or hypothesized in prior studies affect knowledge decay after teachers complete their PD programs. Overall, this study provides more reliable and generalizable empirical evidence on the decay of teacher knowledge after they complete PD and how much the decay rates vary by teacher characteristics and program features.

Data and Methods

Sample

The analytic sample consists of data collected in the Teacher Knowledge Assessment System (TKAS), which is a Web-based platform administering assessments of mathematical knowledge for teaching, developed through the Learning Mathematics for Teaching project at the University of Michigan (http://sitemaker.umich.edu/lmt/home). TKAS was launched in June 2008 and has been widely used by researchers, PD providers, teacher educators, and program evaluators to assess teachers’ mathematical knowledge and its development. While the system allowed for pre, pre-post, and pre-post-delayed post designs, most of the TKAS users (over 90% of programs) chose to have a pre-post or pre-post-delayed post design to evaluate the knowledge change of teachers over time. To enable these administration designs, the system contains multiple forms that are equated to generate comparable test scores across multiple test administrations. In addition, based on the training content and evaluation need, administrators could select a combination of test forms from the following elementary and middle-school assessments in mathematics: (a) Elementary number concepts and operations (ELNCOP); (b) Elementary patterns, functions, and algebra (ELPFA); (c) Grade 4 to 8 geometry (GEO); (d) Middle school number concept and operations (MSNCOP); (e) Middle school patterns, functions, and algebra (MSPFA). Along with the assessments, TKAS administers surveys at each testing administration to teachers and PD providers. These surveys collect self-report data on program characteristics, teacher background, teacher motivation for participating in the training, and teacher reflection on the training experiences.

To ensure that the sample only included inservice teachers enrolled in PD programs, we limited the sample in several ways. First, we dropped preservice teachers and preservice programs from the sample. We also excluded programs with fewer than 10 teachers, given concerns about the reliability of the estimated knowledge decay rates drawn from such small programs. Finally, we only included teachers who had both pretest and posttest scores. After these exclusions, our sample includes 3,652 inservice teachers from 161 programs across the nation.

Next, we used latency data on how long each teacher took to complete the assessment to identify teachers who did not take adequate time to consider answers to the assessment questions. We excluded teachers who completed an assessment in less time than it takes to simply read through the test form at an average speed. After removing teachers with random careless responses, the sample size dropped by approximately 10%, but the total number of programs remained unchanged. Summary statistics on teacher and program characteristics look similar across the full sample and the sample of teachers who completed the test with a good faith effort. For the remainder of the analysis, we only include the 3,340 teachers who made a good faith effort.

The final sample contains a mix of K-12 teachers who differ in education background and teaching experience (Table 1). Specifically, 27% of the teachers have a mathematics degree at either the undergraduate or graduate level. About 14% of the teachers have less than 3 years of teaching experience and 48% of them have more than 10 years of experience. In terms of program characteristics, approximately, half of the programs in the sample span both summer and school year. Summer institutes account for a slightly larger percentage compared with programs that occur during school year. In addition, the program dosage varies considerably. The number of sessions and total contact hours range from 1 to 32 sessions and from 2 to 120 hours. In terms of the uses of different tests, there are 14 test combinations in the sample. The five most frequently used combinations are ELNCOP (47 programs), MSPFA (30 programs), ELNCOP & ELPFA (16 programs), MSPFA & MSNCOP (15 programs), and ELPFA (12 programs). The complete list of test combinations used by the programs is available upon request.

Table 1.

Teacher and Professional Development Characteristics.

	Full sample		Good faith effort sample
	N	M (SD)	N	M (SD)
Teacher characteristics
Days waited to take pretest	3,652	12.16 (25.25)	3,340	11.76 (25.57)
Days waited to take posttest	3,652	11.10 (17.07)	3,340	10.61 (16.88)
Pretest score	3,652	0.15 (0.88)	3,340	0.19 (0.88)
Posttest score	3,652	0.27 (0.90)	3,340	0.30 (0.91)
Mathematics degree	3,484	0.28 (0.45)	3,191	0.27 (0.45)
Female	3,473	0.86 (0.35)	3,181	0.86 (0.35)
White	3,461	0.80 (0.40)	3,171	0.82 (0.39)
Years of teaching experience	3,399	10.60 (8.35)	3,109	10.82 (8.32)
Grade taught^a
Teaching grade K-5	3,316	0.58 (0.49)	3,049	0.60 (0.49)
Teaching grade 6-8	3,316	0.34 (0.47)	3,049	0.31 (0.46)
Teaching grade 9-12	3,316	0.16 (0.36)	3,049	0.16 (0.37)
Teaching credentials^a
All subjects credential	3,316	0.73 (0.44)	3,049	0.74 (0.44)
Mathematics credential	3,316	0.35 (0.48)	3,049	0.34 (0.47)
Science credential	3,316	0.05 (0.22)	3,049	0.05 (0.22)
Professional development characteristics
Number of sessions	2,268	10.36 (8.37)	2,161	10.55 (8.45)
Contact hours	2,293	52.12 (34.29)	2,170	53.32 (34.26)
Intensity	2,233	5.77 (2.62)	2,126	5.79 (2.63)
Program type
Summer institutes	3,652	0.26 (0.44)	3,340	0.27 (0.44)
School year programs	3,652	0.28 (0.45)	3,340	0.24 (0.43)
Programs that span both summer and school year	3,652	0.46 (0.50)	3,340	0.49 (0.50)

Note. The full sample has 3,652 teachers and 161 professional development programs. After excluding teachers who did not make an appropriate effort at the test, the new sample includes 3,340 teachers and 161 programs. All the categorical variables are coded as 0/1. For example, mathematics degree is coded as 1 if teachers have a math degree and coded as 0 otherwise. The mean of a categorical variable shows the percentage of teachers in a particular category.

The percentages of grade and credential do not add up to 100, as multiple options could be selected.

Empirical Strategy

The most common empirical approach to studying knowledge decay is to use pre, post, and delayed post designs to model growth trajectories as a result of an intervention (Goldschmidt & Phelps, 2010; Heyns, 1978). However, for TKAS data, and many other large-scale studies with a delayed post design, such a growth analysis often suffers from a series of issues associated with substantial missing data at the delayed posttest occasion. In TKAS, about 44 programs (27.3%) had a design with pretest, posttest, and delayed posttest occasions. In these programs, the percentage of teachers who have pretest and posttest but do not have a delayed posttest is 64%. As a result, the common approach of modeling growth would lead to significant reduction in sample size (89%) and potentially biased estimated effects.

In this analysis, we take a different approach to estimating knowledge decay that takes advantage of one of the administration features built into the TKAS system. When designing an administration plan for a PD program, it is possible to set up the TKAS system to give teachers the flexibility to choose when they would like to complete the assessment. In the study sample, 49.69% of the programs allow teachers more than 2 weeks to complete their posttest and 26.71% allow more than a month. More importantly, these wide administration windows create large variations in how long teachers wait to take the test, and therefore, it is possible to use this feature of TKAS to construct a measure for each teacher for the number of days that they wait to take the test.

Using a multilevel model with teachers nested in programs, we estimate the relationship between the number of wait days and posttest scores across teachers after accounting for program- and teacher-level differences. The model is specified below as follows:

\begin{array}{l} P o s t S o c r e_{i j} = β_{0 j} + β_{1} P r e S c o r e_{i j} + β_{2} P r e S c o r e_{i j}^{2} \\ + β_{3 j} W a i t_{i j} + β_{4} X_{i j} + ε_{i j} \end{array}

(1)

β_{0 j} = π_{00} + γ_{0 j} γ_{0 j} ~ N (0, σ_{γ_{0}}^{2})

(2)

β_{3 j} = π_{300} + γ_{30 j} γ_{30 j} ~ N (0, σ_{γ_{30}}^{2})

(3a)

β_{3 j} = π_{30} + π_{31} X_{3 j} + γ_{3 j} γ_{3 j} ~ N (0, σ_{γ_{3}}^{2})

(3b)

where Wait_ij is the key explanatory variable that indicates how long teachers waited to take the test. Specifically, it is 0 if teacher i is the first test-taker in program j. We use this relative measure of wait time across teachers so that the coefficient of Wait_ij reflects the change in posttest score across teachers having different wait times within a program.¹ In addition, we include quadratic and cubic function forms of the Wait_ij to test if the decay rate estimate is relatively stable across time.

To control for individual differences that are associated with both Wait_ij and PostScore_ij, we included PreScore_ij as well as a set of teacher characteristics (X_ij). The test scores are normed to the TKAS population, with a mean of zero and a standard deviation of one. Both the pretest and posttest scores used in the model are the average of these test scores from all assessments used by each program. These average scores should capture what teachers learned from the PD programs, assuming that the program administrators choose assessments aligned with their program content. PreScore_ij is included to account for differences in preexisting knowledge. In addition, our data suggests that the pretest and posttest scores have a quadratic relationship. To account for this, a square term of pretest score is added into the equation. The list of teacher characteristics as denoted by X_ij include demographics (race and gender dummy variables), teacher background variables (teaching experience and whether they have a math degree at either undergraduate or graduate level), and four composite variables that were built from Likert-type-scale items in teacher surveys. The composite variables include teachers’ reflections on the relevance and importance of the test questions, satisfaction with the PD program, and intrinsic and extrinsic sources of motivation for attending the programs.

This multilevel model allows the intercept and coefficient for Wait_ij to vary at the program level. As teachers are clustered within programs, a varying intercept (Equation 2) is required to capture the random program effect on teachers’ posttraining knowledge.² The random coefficient, as specified by Equation 3a, allows for program-specific estimates of teacher knowledge decay rates. In Equation 3a, π₃₀₀ represents the estimated average knowledge decay rate across all programs.

Furthermore, to address the second research question, Equation 3b includes program-level covariates that are associated with decay rate. Here, π₃₁ estimates differences in decay rate across different types of programs and is essentially the coefficient for the interaction term between wait time variable and program-level characteristics. Two kinds of program-level predictors (X_3j) for the random coefficient for Wait_ij are examined. The first set of variables representing the design features of the programs is constructed as follows. (a) Program type, defined using the starting month and the length of the program testing window. This variable categorizes programs into three types: summer institute, programs taking place during the school year (school year program), and programs that span both summer and school year (whole year program). (b) Number of training sessions, which indicates how many times teachers are required to meet. (c) Contact hours, which is based on providers’ reports of total hours of program training. (d) Intensity, defined as contact hours divided by the number of training sessions. (e) Uses of different tests, which characterize programs by different combinations of assessments they chose to use and, therefore, also reflect the content focus of the program. The second set of program-level variables is constructed by aggregating participants’ characteristics to the program level. These variables include: (a) average initial knowledge level measured by average pretest score; (b) average years of classroom teaching experience; and (c) percentage of teachers who have a mathematics degree.

Assessing Threats to the Validity of Knowledge Decay Estimate Using Robustness Tests

In the aforementioned multilevel model, the estimated coefficient for Wait_ij, if positive would be attributed to evidence of continued learning after PD programs, and if negative would be attributed to knowledge decay. However, as teachers are allowed to self-select, the effects of waiting to take the test could also reflect teachers with lower posttest scores selecting into later testing dates. This selection effect is an obvious threat to the inference that the effect for waiting to take an assessment can be attributed to knowledge decay.

We tested and controlled for the potential unwanted selection in the following three ways. First, we controlled for program-level differences to avoid having selection effects that are produced due to differences in program features. For example, good PD programs, which are more effective at improving teacher knowledge, could also motivate teachers to respond more promptly. Our multilevel model compares teachers who participated in the same PD program and thus could control for this particular source of self-selection.

Second, we controlled for a series of teacher-level covariates that are related to both the number of wait days and the posttest score to reduce any confounding effects on the knowledge decay estimator. For example, teachers who do well on the pretest might be more eager to take the posttest and are also more likely to perform better on the posttest. The knowledge decay estimate might be downward biased if the difference in pretest score is neglected. The teacher-level covariates included should be able to correct for the bias caused by observed individual differences.

Despite the rich set of covariates available, the list of control variables remains inevitably incomplete. As a result, the relationship between the test score and the number of wait days might still be driven by factors other than teacher knowledge decay. To further investigate, we conducted three additional analyses to rule out alternative hypotheses regarding the relationship between wait days and test scores. In the first analysis, we ran a series of sensitivity analyses to test if the decay rate estimate is driven by a small number of extreme values that come from teachers who waited for a long time to take the test. In the full sample, a small percentage (3.4%) of teachers waited for 50 days or more to take the test. This group of teacher may not represent the average teachers attending PD programs. To find out if our decay rates estimate changes after excluding extreme groups, we fit the same multilevel model with teacher-level covariates described earlier using samples that include teachers who completed the test within 50, 60, or 70 days.

In the second analysis, we ran a counterfactual test to estimate the effect of wait time at the pretest administration. Pretest data provides a powerful counterfactual test because these data are collected before the PD and because the treatment and associated learning has not taken place. Therefore, knowledge decay should not be observed at pretest. If a similar pattern is observed for wait days at pretest, then the observed effect of wait time could be attributable to teachers self-selecting into later testing according to a variety of unobserved personal characteristics. Conversely, if under this condition, there is no effect for wait days, then the counterfactual argument would indicate that any effect observed for wait days at posttest is more likely to be associated with the knowledge decay and not these unobserved characteristics. To estimate the effect of wait time at pretest administration, we used a similar multilevel model with pretest scores as the outcome variable. In this model, the key predictor is the number of wait days at pretest administration. The covariates used include gender, race, teaching experience, mathematics degree, and the four composite variables.

Finally, we tested whether our knowledge decay estimate is clouded by reverse causality. Reverse causality occurs if the amount of teacher learning affects when teachers take the test—that is, teachers who learned the most take the test first. This is problematic because the decay estimate from our model captures the true knowledge decay as well as this pattern caused by reverse causality. To test if teacher learning has an impact on wait time, we used a self-report measure of learning. Participating teachers, in addition to completing the preknowledge and postknowledge tests were asked to self-rate their learning after PD. While self-reports may not be as accurate as tests of knowledge, we think it is less likely that personal impressions of learning will decay. Self-reports therefore provide an alternative measure of learning that does not reflect knowledge decay in the same way as test scores. Specifically, we constructed a variable for teachers’ self-report learning by averaging two Likert-type scale³ items ranging from 1 to 6 and used this variable as the outcome measure. In addition, we used the wait time variable as the predictor and other teacher characteristics as the covariates. Results from this model could help us find out whether wait days and teacher learning are related, and a lack of a relation between wait time and self-report learning suggests that the knowledge decay rate estimate should not be biased due to reverse causality.

Results

Knowledge Decay Estimates

Table 2 presents the main results from the multilevel model. The first model specification only includes the number of days teachers wait to take the test as the independent variable and allows the intercept and the coefficient for the wait variable to vary at the program level (Column 1). Next, we add teacher-level covariates into the model to test if the estimated decay rate is driven by teacher characteristics (Column 2). The decay rate estimate does not change after adding teacher covariates into the model, indicating the estimate is robust to the inclusion of these teacher characteristics. For heterogeneous analyses, we chose to build on the model in column 2 because all model-fit statistics suggest that this model provides a better fit than the one without teacher-level covariates. In addition, the function form of the model has important implications for how the decay estimates should be interpreted. So, we tested the quadratic and cubic function form of the model, but found that the square and cubic terms are statistically insignificant. The model-fit statistics (i.e., log likelihood, Akaike information criterion [AIC], and Bayesian information criterion [BIC]) also suggest that the linear model fits the data better. Overall, the evidence suggests that the decay rate is relatively stable following the completion of the training.⁴

Table 2.

Estimates of Teacher Knowledge Decay Rate From Multilevel Models.

Outcome variable: Posttest score	Full sample	Full sample	<70 days	<60 days	<50 days
Outcome variable: Posttest score	(1)	(2)	(3)	(4)	(5)
Coefficient estimates
Wait at post (π₃₀₀)	−.0033**	−.0031**	−.0032**	−.0033**	−.0034**
Variance components (SD)
Random coefficient for Wait_ij $(σ_{γ_{30}})$	0.0010	0.0030	0.0029	0.0028	0.0042
Random intercept $(σ_{γ_{0}})$	0.3660	0.1997	0.2003	0.1951	0.1909
Model-fit statistics
Log likelihood	−4,151.02	−2,619.79	−2,571.75	−2,563.80	−2,534.02
AIC	8,314.05	5,269.57	5,173.50	5,157.59	5,098.05
BIC	8,350.73	5,359.29	5,262.91	5,246.94	5,187.23
Random intercept and coefficient	Yes	Yes	Yes	Yes	Yes
Teacher-level covariates	No	Yes	Yes	Yes	Yes
No. of programs	161	161	161	161	161
No. of observations	3,340	2,926	2,866	2,855	2,823

Note. Teacher-level covariates include pretest score and its square term, gender, race, teaching experience, mathematics degree, and composite measure of teachers’ reflection on the relevance and importance of the test questions, satisfaction with the professional development programs, intrinsic and extrinsic motivation for program participation. We include log likelihood, AIC, and BIC to inform the model selection. The difference among these indicators is that the log likelihood could be increased by simply adding covariates. Therefore, increased log likelihood does not necessarily reflect better model fit. In contrast, AIC and BIC are not sensitive to the number of covariates included because they introduce a penalty for including more parameters in the model. All the indicators suggest that the model in column 2 is a better fit than the model in column 1. AIC = Akaike information criterion; BIC = Bayesian information criterion.

p < .05. ** p < .01. *** p < .001.

A consistent pattern in both models is that the knowledge decay estimate is negative and significant at −.003 (p value < .01), indicating that on average, waiting one day to take the test would be associated with scores that are a 0.003 standard deviations lower.⁵ To understand the substantive implication of this estimate, we compare the estimated decay rate to pre–post teacher knowledge change as a result of the PD training. Specifically, we calculate the average difference of pretest and posttest score in the TKAS sample. The average observed change of .1109 would be lost in 37 days (.1109/0.003) with the decay rate being stable over time. To provide a more accurate benchmark on the magnitude of teacher learning during PD programs, we use estimates from a recent study (Phelps et al., 2016) that estimated pre–post teacher knowledge change using cross-classified hierarchical growth model and TKAS data. This study estimated the average change in scores for each of the TKAS assessments. As the test on ELNCOP is used by almost two thirds of the programs in TKAS, we use changes in teacher knowledge measured by this assessment as the proxy. In that study, the estimated average change in test scores in ELNCOP has a mean of 0.18 and a standard deviation of 0.17 at the program level. Again, assuming the decay rate is stable over time, the average knowledge gains for programs using this assessment would be lost in approximately 60 days (0.18/0.003). Overall, these findings suggest that some PD programs could have significant effects on improving teacher knowledge, but the effects do not sustain—our knowledge decay estimate shows that the average knowledge gains from the programs in our sample would be lost in just 37 days after teachers complete the training.

Robustness Tests

One piece of evidence demonstrating the robustness of the decay estimate is that the variation in estimated decay rate across samples that include different subsets of teachers is small (Table 2). Our results show that there is only a small number of teachers who waited 50 to 70 days after the completion of the trainings (43 teachers), and the estimated decay rate for teachers who completed the test within 70, 60, or 50 days are −0.0032, −0.0033, and −0.0034, respectively. This suggests that the estimated decay rate is not driven by a number of extreme cases and is relatively stable across samples.

In addition, the counterfactual test at the pretest administration generates insignificant estimates using a similar model, and the estimates for wait days are very close to zero in the model with or without teacher-level covariates (Table 3). Furthermore, the estimated program-level standard deviation for the coefficient of wait time variable is close to zero, indicating that there is no significant relationship between pretest score and the number of wait days at the pretest for all programs in the sample. These zero estimates suggest that any teacher characteristics that remain constant at pretest and posttest would not produce a relation between the number of wait days and the posttest scores. More broadly, the counterfactual test helps to rule out the possibility that the pattern captured at the posttest is attributed to any variation in observed or unobserved variables that stay constant across the two testing points. This is a good piece of evidence supporting the model in addition to the program-level and teacher-level covariates that account for possible self-selection.

Table 3.

Counterfactual Estimates Using Pretest Score as Outcome Variable.

Outcome variable: Pretest score	Model (1)	Model (2)
Coefficient estimates
Waited at Pre	.0000	−.0003
Variance components (SD)
Random coefficient for Wait_ij	0.0005	0.0000
Random intercept	0.3858	0.2194
Model fit statistics
Log likelihood	−4,075.49	−3,047.64
AIC	8,160.98	5,143.00
BIC	8,191.54	5,218.62
Random intercept and coefficient	Yes	Yes
Teacher-level covariates	No	Yes
No. of programs	161	161
No. of observations	3,340	2,926

Note. Teacher-level covariates include gender, race, teaching experience, mathematics degree, and variables of factor score for teachers’ reflection on the relevance and importance of the test questions, satisfaction with the professional development programs, and intrinsic and extrinsic motivation for program participation. We included log likelihood, AIC, and BIC to inform the model selection. The difference among these indicators is that the log likelihood could be increased by simply adding covariates. Therefore, increased log likelihood does not necessarily reflect better model fit. In contrast, AIC and BIC are not sensitive to the number of covariates included because they introduce a penalty for including more parameters in the model. AIC = Akaike information criterion; BIC = Bayesian information criterion.

p < .05. ** p < .01. *** p < .001.

Finally, we find that the knowledge decay estimate may not be biased by reverse causality. Specifically, we find an insignificant relationship between wait time and self-report learning (Table 4), indicating that there is no obvious pattern that teachers who self-report learning the most during the training take the test first. The result suggests that the estimates of knowledge decay rate is not influenced by this particular mechanism.

Table 4.

Estimates of Teacher Knowledge Decay Rate Using Self-Report Learning Measure.

Outcome variable: Self-report learning	Model (1)	Model (2)
Coefficient estimates
Wait at post (π₃₀₀)	−.0027	−.0026
Variance components (SD)
Random coefficient for Wait_ij $(σ_{γ_{30}})$	0.0045	0.0047
Random intercept $(σ_{γ_{0}})$	0.4439	0.4148
Model fit statistics
Log likelihood	−3,829.921	−3,321.9609
AIC	7,671.842	6,669.922
BIC	7,707.488	6,745.717
Random intercept and coefficient	Yes	Yes
Teacher-level covariates	No	Yes
No. of programs	161	161
No. of observations	3,340	2,926

Note. Teacher-level covariates include gender, race, teaching experience, mathematics degree, and composite measure of teachers’ reflection on the relevance and importance of the test questions, satisfaction with the professional development programs, and intrinsic and extrinsic motivation for program participation. We include log likelihood, AIC, and BIC to inform the model selection. The difference among these indicators is that the log likelihood could be increased by simply adding covariates. Therefore, increased log likelihood does not necessarily reflect better model fit. In contrast, AIC and BIC are not sensitive to the number of covariates included because they introduce a penalty for including more parameters in the model. All the indicators suggest that model 2 is a better fit than model 1. AIC = Akaike information criterion; BIC = Bayesian information criterion.

p < .05. ** p < .01. *** p < .001.

Knowledge Decay Estimates by Program and Teacher Characteristics

We find that there is substantial variation in the magnitude of the knowledge decay estimate across programs. The estimates range from −0.009 to 0.003 at 95% confidence level,⁶ although estimates for most of the programs fall within −0.004 to −0.002 (Figure 1).

In our analysis examining the heterogeneous knowledge decay rates, the variation in the decay rate is associated with program features and participants’ characteristics. The estimate of decay rate varies significantly by program types. Table 5 shows that the average knowledge decay rate for summer institute is −0.009, whereas the estimates for school year and whole year programs is −0.002.⁷ When comparing the decay rate of different types of programs to their corresponding gains, we find that summer institutes have the lowest gain yet the highest decay rate. The approximate knowledge gain is 0.05 standard deviations for summer institutes and 0.13 standard deviations for the other two types of program designs.

Table 5.

Heterogeneous Decay Rate by Programs and Teacher Characteristics.

	Coefficient estimates	Log likelihood	No. of Obs. (No. of programs)
Program type
Waited at post (summer institute) (π₃₀)^a	−0.0092**	−2617.57	2,926
Waited at post × whole year (π₃₁)^a	0.0068*		(161)
Waited at post × school year (π₃₁)^a	0.0072*
Intensity
Waited at post (π₃₀)	−0.0042**	−1402.19	1,597
Waited at post × intensity (π₃₁)	−0.0012*		(107)
No. of sessions
Waited at post (π₃₀)	−.0036**	−1408.56	1,621
Waited at post × no. of sessions (π₃₁)	0.0004		(108)
Contact hours
Waited at post (π₃₀ )	−.0034*	−1421.42	1,628
Waited at post × contact hours (π₃₁)	.0001		(109)
Average program-level pretest score
Waited at post (π₃₀)	−0.0035***	−2617.82	2,926
Waited at post × average pretest score (π₃₁)	0.0062*		(161)
Average teaching experience
Waited at post (π₃₀)	−0.0027*	−2145.10	2,483
Waited at post × average teaching experience (π₃₁)	−0.0002		(155)
Percentage of teachers with math degree
Waited at post (π₃₀)	−0.0025*	−2144.04	2,483
Waited at post × % math degree (π₃₁)	0.0054		(155)
Use of test combinations
Waited at post (π₃₀)	−0.0019	−2618.55	2,926
Waited at post × test combination indicator (π₃₁)	−0.0002		(161)

Note. Teacher-level covariates include pretest score and its square term, gender, race, teaching experience, mathematics degree and variables of factor score for teachers’ reflection on the relevance and importance of the test questions, satisfaction with the professional development programs, and intrinsic and extrinsic motivation for program participation. All the program and teacher characteristics variables (except for program type) are centered to the mean for the ease of interpretation.

π₃₀ represents the estimated knowledge decay rate for the reference group. π₃₁ is the estimate of difference in knowledge decay rate between the current group and the reference group. The estimated knowledge decay rate for whole year and school year program is −0.0024 (−0.0092 + 0.0068) and −0.002 (−0.0092 + 0.0072).

p < .05. **p < .01. ***p < .001.

In addition to the program types, we examine three other program feature variables: program intensity, number of session, and contact hour. All of these program-level variables are continuous variables and are centered to the mean to allow for examination of the heterogeneous effect at the average decay rate rather than at the extreme values. The effect of intensity on knowledge decay is negative and moderately significant, suggesting that the more compressed the training, the faster the acquired knowledge is lost. The effect of the number of sessions is only significant at .1 level (p value = .0680). The positive estimate indicates that more sessions are associated with higher knowledge retention. Finally, the results show that decay rates do not vary by the number of contact hours. Together, these findings suggest that contact hours might not reduce knowledge decay if training sessions were conducted within a short period of time. Teachers are more likely to sustain their knowledge after attending PD programs with longer duration and ongoing activities over time.

It is worth noting that the heterogeneous decay rates by program types might be partly explained by the fact that summer institutes are usually more intense with fewer sessions compared with school year programs. A group comparison using analysis of variance (ANOVA) shows that the intensity differs significantly across program types, with summer institutes having the highest intensity and whole year programs having the lowest (F value = 4.92, p value = .0091). As higher intensity is associated with higher knowledge decay rate, summer institutes with the highest intensity would be expected to have the highest decay rate among the three program types. Finally, the results also show that there is no significant variation in decay rate estimates across different test combinations. The interaction term is very small and statistically insignificant.

We next investigate how decay rate varies by participants’ characteristics. As the decay rate is estimated at the program level, we can only examine how it varies by the average characteristics of participants in a program. Results show that programs with a higher initial average score have a higher level of knowledge sustainability. A program with the lowest average level of initial knowledge has a decay rate that is twice as high as the rate for a program with an average knowledge level. However, neither the average teacher experience nor the percentage of teachers with a mathematics degree is found to be significantly associated with the decay rate.

To summarize, we find significant and substantial knowledge loss after teachers complete the PD programs. The estimated knowledge decay rate is −0.003, which indicates that the knowledge gain from pre to post for programs in this sample would be lost in 37 days, with decay rate being stable across time. Furthermore, the decay rate is affected by program features. Summer institutes have a much higher decay rate than programs that span a school year or the whole year. Programs with higher intensity or fewer number of training sessions are also associated with higher decay rate. These program features might affect the decay rate through a similar mechanism, as summer institutes are usually characterized by high intensity and small number of training sessions. Finally, programs with higher average teacher initial knowledge have less knowledge decay, but other teacher characteristics, such as average teaching experiences or having a math degree, are found to be unrelated to the decay rate in the context of teacher PD programs.

Discussion

This study makes use of a large-scale data set to empirically examine the extent to which teachers retain knowledge following PD program. The results show that teacher knowledge is not stable after PD. In fact, the loss of knowledge, or knowledge decay, does take place, and it may be quite substantial. If improving student learning is the goal of the PD programs, the focus of these programs should not only be on what teachers learn but also on whether the learning lasts long enough to make it to the classroom. In addition, the study shows that knowledge decay rate varies considerably across programs, and the decay rate is affected by teachers’ initial knowledge level as well as the design of PD programs (e.g., program type and intensity).

Implications

While there is more to learn, results from this study suggest several components of PD programs that are important for sustaining posttraining teacher learning. One important component is the duration of the program. Prior studies showed that longer duration of PD led to larger improvements in teacher knowledge or teaching practices (Copur-Gencturk & Papakonstantinou, 2016; Garet, Porter, Desimone, Birman, & Yoon, 2001). Likewise, our study found that programs with greater number of training sessions or longer duration are associated with lower knowledge decay rate, suggesting that longer program duration not only increases the impact of PD on teachers but also sustains the impact afterwards. Another component is the provision of follow-up activities after intensive training sessions. We found that summer programs have higher knowledge decay than whole year or school year programs. While alternative explanations are possible, a major difference between these programs is that unlike summer program, whole year or school year programs provide opportunities for teachers to immediately apply new knowledge in their own classrooms. School year programs also often provide follow-up support to teachers. These supports are important for teachers to internalize and apply key ideas learned from the training and, as a result, may reduce knowledge decay following the completion of PD. In designing or implementing teacher PD program, it could be important to consider these findings on knowledge decay together with those design components that improve program effects to increase teacher learning in the short and long run.

In addition to program design, the study informs the design of research studies evaluating the effectiveness of teacher PD. Policymakers might consider including the sustainability of program effects as another criterion for assessing program effectiveness. Existing research has found that teacher PD programs can have significant and substantial effects on teacher knowledge, supporting claims by policymakers and researchers that teacher PD training is an important lever toward improving instruction and student learning (Desimone, 2009). However, to date, research on the effectiveness of these programs has treated the program effects as if they are stable. This is against the viewpoint from cognitive scientists that knowledge or skills tend to be unstable and changing over time. The decay of knowledge gains is particularly salient when there is a lack of use of the knowledge acquired (Kim, Ritter, & Koubek, 2013). Those program evaluation studies that neglect the changing nature of teacher knowledge may provide incomplete or even misleading evidence on the effectiveness of the programs.

More broadly, knowledge decay has important implications not only in the context of teacher PD, but also throughout the entire continuum of teacher learning. For example, previous research suggests that teacher candidates and teachers often have gaps in their knowledge (Depaepe et al., 2015; Donna & Hick, 2017; Wilburne & Long, 2010). This occurs because candidates may not be adequately prepared by their teacher education programs and may not be able to keep deepening their knowledge after they start teaching. Other than that, teacher knowledge decay is another possible reason for the existing knowledge gaps. So far, there has been a lot of research discussing promising programs and strategies that prepare teachers and support them throughout their career (Abd-El-Khalick & Lederman, 2000; Alves et al., 2018; Christ, Arya, & Chiu, 2017; Darling-Hammond, 2012; Feiman-Nemser, 2001; Tatto & Menter, 2019). While these programs have focused more on teacher learning, our study suggests that it is worthwhile to consider knowledge decay as an equally important factor when designing, implementing, and evaluating programs that support teacher learning.

Limitations and Future Directions

This study estimates teachers’ knowledge decay rate after they complete the PD. While this approach does not require a second posttest score to estimate knowledge decay and thus does not suffer from missing data issues, it is limited in several other ways. The biggest concern is that the effects we observe might result from self-selection rather than true knowledge decay. However, as presented in the results section, our robustness test does not show evidence of self-selection. Furthermore, both the models we used to estimate the knowledge decay rate and the controls we entered in the models further guard against the possibility that the findings are due to selection. Still, we recognize that it is not possible to definitively claim that the effects of waiting to take a test are due to knowledge decay and not to other factors. To avoid this problem, future PD evaluation studies could collect data on treatment and control groups and use the standard pre, post, and follow-up design to examine the knowledge sustainability after the training. In practice, this design usually requires larger sample size and time commitments. In addition, the follow-up data collection is usually challenging with possible nonrandom sample attrition. For example, high performers are often more likely to respond to the follow-up, which leads to insignificant knowledge decay estimates. So, the choice of this design should be considered with its feasibility in the specific context as well as resources available to the study.

Second, while we know some aspects of the PD programs, it would be helpful if we have more information regarding the PD intervention, the teachers who participated, and also the school context in which these teachers work. For example, we do not know whether different decay rates for summer, school year, and whole year programs are a result of different timing, specific training activities, or other key measures of the program quality. In addition, prior research (Arthur et al., 1998) suggests that differences in decay rate might be attributed to whether there is a coordinated effort between schools and PD programs to train teachers on the most relevant knowledge or skill and allow them to apply it in schools. As it is suggested by our previous review of the literature (Roth et al., 2011; Yoon et al., 2007), details on different attributes of PD, including its context, content focus, and approach, would have been very useful to decipher how the key program features are connected to knowledge decay rates. Due to the unavailability of the information, our current analysis is unable to uncover what the key components of the programs are that lead to different knowledge decay rates.

Finally, the functional form of the model is another issue and a potential limitation of the study. In previous research, decay has typically been modeled with a power-form curve or log-linear learning model (Jaber, Kher, & Davis, 2003; Wright, 1936). However, we found that the relationship between the wait time and test score appears to be linear. For example, we tested a number of nonlinear specifications by adding polynomials of wait time, but none of these fit the data better than the simple linear model. Furthermore, the simple linear model allows for a straightforward interpretation of the average knowledge decay estimates. While these points explain the approach we took in our analyses, we acknowledge that our findings are limited by the lack of accurate prediction of knowledge decay rates at different points in the distribution of wait days. Furthermore, we do not have data on instruction quality and student outcome measures. Future studies with data on all these measures could provide a better understanding on the underlying mechanism of how decay of teacher knowledge is connected to the impact on teaching instruction and student outcomes.

Conclusion

The findings from this study strongly suggest that in studying the outcomes of PD and the effects of PD on teaching and learning, it is important to account for more than immediate teacher learning outcomes. A more comprehensive approach would consider a program’s true effect as the combination of learning and forgetting. From the way studies are currently conducted, we simply do not know to what extent learning is maintained, and consequently, we do not know the programs’ true value. Our findings suggest that it will be just as important to study both the stability of these learning effects and the conditions that support teachers in retaining knowledge. The importance of such studies becomes even more salient, given the current efforts to use teacher PD programs as one of the main policy levers for improving teacher knowledge and skill, teaching quality, and ultimately student achievement. Districts and states continue to devote substantial research funds to teacher education and PD programs; the true benefit of these investments will be difficult to ascertain without understanding both how teachers learn and forget critical knowledge and skills.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by a grant from the National Science Foundation (award no. 0927725). The opinions expressed herein are those of the authors and not the funding agency.

ORCID iD

Shuangshuang Liu

Notes

Author Biographies

Shuangshuang Liu is a research associate at the Educational Testing Service. Her research focuses on educational interventions and policies on K-12 teaching and learning. She is particularly interested in research related to teacher professional development, classroom observations, and ability-tracking practices.

Geoffrey Phelps is a managing senior research scientist at the Educational Testing Service. His research focuses on the measurement of teacher content knowledge, teacher development, and the relation of teacher knowledge and teaching quality.

References

Aaronson

Barrow

Sander

(2007). Teachers and student achievement in Chicago Public High Schools. Journal of Labor Economics, 24, 95-135.

Abdal-Haqq

(1995). Making time for teacher professional development (Digest 95-4). Washington, DC: ERIC Clearinghouse on Teaching and Teacher Education.

Abd-El-Khalick

Lederman

N. G.

(2000). Improving science teachers’ conceptions of nature of science: A critical review of the literature. International Journal of Science Education, 22(7), 665-701.

Alves

K. D.

Kennedy

M. J.

Kellems

R. O.

Wexler

Rodgers

W. J.

Romig

J. E.

Peeples

K. N.

(2018). Improving preservice teacher vocabulary instruction: A randomized controlled trial. Teacher Education and Special Education, 41(4), 340-356.

Arthur

W. J.

Bennett

W. J.

Stanush

P. L.

McNelly

T. L.

(1998). Factors that influence skill decay and retention: Quantitative review and analysis. Human Performance, 11, 57-101.

Ball

D. L.

Thames

M. H.

Phelps

(2008). Content knowledge for teaching: What makes it special? Journal of Teacher Education, 59(5), 389-407.

Battey

Llamas-Flores

Burke

Guerra

Kang

H. J.

Kim

S. H.

(2013). ELL policy and mathematics professional development colliding: Placing teacher experimentation within a sociopolitical context. Teachers College Record, 115(6), 1-44.

Baumert

Kunter

Blum

Brunner

Voss

Jordan

Tsai

Y. M.

(2010). Teachers’ mathematical knowledge, cognitive activation in the classroom, and student progress. American Educational Research Journal, 47, 133-180. doi:10.3102/0002831209345157

Bell

C. A.

Wilson

S. M.

Higgins

McCoach

D. B.

(2010). Measuring the effects of professional development on teacher knowledge: The case of developing mathematical ideas. Journal for Research in Mathematics Education, 41, 479-512.

10.

Birman

Desimone

L. M.

Porter

A. C.

Garet

M. S.

(2000). Designing professional development that works. Educational Leadership, 57, 28-33.

11.

Blank

R. K.

De las Alas

Smith

(2008). Does teacher professional development have effects on teaching and learning? Analysis of evaluation findings from programs for mathematics and science teachers in 14 states. Washington, DC: Council of Chief State School Officers.

12.

Boston

M. D.

Smith

M. S.

(2011). A ‘task-centric approach’ to professional development: Enhancing and sustaining mathematics teachers’ ability to implement cognitively challenging mathematical tasks. ZDM: The International Journal on Mathematics Education, 43(6-7), 965-977.

13.

Carney

M. B.

Brendefur

J. L.

Thiede

Hughes

Sutton

(2014). Statewide mathematics professional development teacher knowledge, self-efficacy, and beliefs. Educational Policy, 30, 539-572.

14.

Chetty

Friedman

J. N.

Rockoff

J. E.

(2013). Measuring the impacts of teachers II: Teacher value-added and student outcomes in adulthood (NBER Working Paper 19424). Cambridge, MA: National Bureau of Economic Research. Retrieved from http://www.nber.org/papers/w19424

15.

Christ

Arya

Chiu

M. M.

(2017). Video use in teacher education: An international survey of practices. Teaching and Teacher Education, 63, 22-35.

16.

Cochran-Smith

Lytle

S. L.

(2001). Beyond certainty: Taking an inquiry stance on practice. In Lieberman

Miller

(Eds.), Teachers caught in the action: Professional development that matters (pp. 45-58). New York, NY: Teachers College Press.

17.

Copur-Gencturk

Papakonstantinou

(2016). Sustainable changes in teacher practices: A longitudinal analysis of the classroom practices of high school mathematics teachers. Journal of Mathematics Teacher Education, 19(6), 575-594.

18.

Correnti

(2007). An empirical investigation of professional development effects on literacy instruction using daily logs. Educational Evaluation and Policy Analysis, 29(4), 262-295.

19.

Darling-Hammond

(2012). Powerful teacher education: Lessons from exemplary programs. San Francisco, CA: John Wiley.

20.

Darling-Hammond

Sykes

(Eds.) (1999). Teaching as the learning profession: Handbook of policy and practice. San Francisco, CA: Jossey-Bass.

21.

Depaepe

Torbeyns

Vermeersch

Janssens

Janssen

Kelchtermans

. . . Van Dooren

(2015). Teachers’ content and pedagogical content knowledge on rational numbers: A comparison of prospective elementary and lower secondary school teachers. Teaching and Teacher Education, 47, 82-92.

22.

Desimone

(2009). Improving impact studies of teachers’ professional development: Toward better conceptualizations and measures. Educational Researcher, 38, 181-199.

23.

Donna

J. D.

Hick

S. R.

(2017). Developing elementary preservice teacher subject matter knowledge through the use of educative science curriculum materials. Journal of Science Teacher Education, 28(1), 92-110.

24.

Farr

M. J.

(2012). The long-term retention of knowledge and skills: A cognitive and instructional perspective. New York, NY: Springer.

25.

Feiman-Nemser

(2001). From preparation to practice: Designing a continuum to strengthen and sustain teaching. Teachers College Record, 103, 1013-1055.

26.

Franke

M. L.

Carpenter

T. P.

Levi

Fennema

(2001). Capturing teachers’ generative change: A follow-up study of professional development in mathematics. American Educational Research Journal, 38(3), 653-689.

27.

Gaikhorst

Beishuizen

J. J.

Zijlstra

B. J.

Volman

M. L.

(2017). The sustainability of a teacher professional development programme for beginning urban teachers. Cambridge Journal of Education, 47(1), 135-154.

28.

Garet

M. S.

Heppen

Walters

Smith

Yang

(2016). Does content-focused teacher professional development work? Findings from three institute of education sciences studies. Washington, DC: Institute of Education Sciences.

29.

Garet

M. S.

Porter

A. C.

Desimone

Birman

B. F.

Yoon

K. S.

(2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38(4), 915-945.

30.

Gersten

Taylor

M. J.

Keys

T. D.

Rolfhus

Newman-Gonchar

(2014). Summary of research on the effectiveness of math professional development approaches (Regional Educational Laboratory 2014-010). Retrieved from https://ies.ed.gov/ncee/edlabs/projects/project.asp?projectID=391

31.

Goldschmidt

Phelps

(2010). Does teacher professional development affect content and pedagogical knowledge: How much and for how long? Economics of Education Review, 29(3), 432-439.

32.

Grider

(1993). Foundations of cognitive theory: A concise review. Retrieved from https://eric.ed.gov/?id=ED372324

33.

Guskey

T. R.

(2000). Evaluating professional development. Thousand Oaks, CA: Corwin Press.

34.

Guskey

T. R.

Yoon

K. S.

(2009). What works in professional development? Phi Delta Kappan, 90(7), 495-500.

35.

Healy

A. F.

Clawson

D. M.

McNamara

D. S.

Marmie

W. R.

Schneider

V. I.

Rickard

T. C.

Bourne

L. E.

(1998). The long-term retention of knowledge and skills (U.S. Army Research Institute Research Note 98-26). Retrieved from www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA349869

36.

Heller

J. I.

Daehler

K. R.

Wong

Shinohara

Miratrix

L. W.

(2012). Differential effects of three professional development models on teacher knowledge and student achievement in elementary science. Journal of Research in Science Teaching, 49(3), 333-362.

37.

Heyns

(1978). Summer learning and the effects of schooling. New York, NY: Academic Press.

38.

Hill

H. C.

Ball

D. L.

(2004). Learning mathematics for teaching: Results from California’s mathematics professional development institutes. Journal for Research in Mathematics Education, 35(5), 330-351.

39.

Hill

H. C.

Rowan

Ball

(2005). Effects of teachers’ mathematical knowledge for teaching on student achievement. American Educational Research Journal, 42(2), 371-406.

40.

Hill

H. C.

Umland

Litke

Kapitula

L. R.

(2012). Teacher quality and quality teaching: Examining the relationship of a teacher assessment to practice. American Journal of Education, 118, 489-519. doi:10.1086/666380

41.

Jaber

M. Y.

Kher

H. V.

Davis

(2003). Countering forgetting through training and deployment. International Journal of Production Economics, 85, 33-46.

42.

Kennedy

M. M.

(2016). How does professional development improve teaching? Review of Educational Research, 86(4), 945-980.

43.

Kim

J. W.

Ritter

F. E.

Koubek

R. J.

(2013). An integrated theory for improved skill acquisition and retention in the three stages of learning. Theoretical Issues in Ergonomics Science, 14(1), 22-37.

44.

Koellner

Jacobs

(2015). Distinguishing models of professional development the case of an adaptive model’s impact on teachers’ knowledge, instruction, and student achievement. Journal of Teacher Education, 66(1), 51-67.

45.

Kraft

M. A.

Blazar

Hogan

(2018). The effect of teacher coaching on instruction and achievement: A meta-analysis of the causal evidence. Review of Educational Research, 88(4), 547-588.

46.

McCutchen

Abbott

R. D.

Green

L. B.

Beretvas

S. N.

Cox

Potter

N. S.

. . . Gray

A. L.

(2002). Beginning literacy: Links among teacher knowledge, teacher practice, and student learning. Journal of Learning Disabilities, 35(1), 69-86.

47.

Phelps

Kelcey

Jones

Liu

(2016). Informing estimates of program effects for studies of mathematics professional development using teacher content knowledge outcomes. Evaluation Review, 40(5), 383-409.

48.

Polly

Neale

Pugalee

D. K.

(2014). How does ongoing task-focused mathematics professional development influence elementary school teachers’ knowledge, beliefs and enacted pedagogies? Early Childhood Education Journal, 42(1), 1-10.

49.

Putnam

R. T.

Borko

(2000). What do new views of knowledge and thinking have to say about research on teacher learning? Educational Researcher, 29(1), 4-15.

50.

Rail Safety and Standards Board. (2011). A model for optimising competence retention in the railway (skills fade): Optimising competence retention in the rail industry Appendices. London, England: BAE System. Retrieved from https://catalogues.rssb.co.uk/research-development-and-innovation/research-project-catalogue/t717

51.

Richardson

Placier

(2001). Teacher change. Handbook of Research on Teaching, 4, 905-947.

52.

Rivkin

S. G.

Hanushek

E. A.

Kain

J. F.

(2005). Teachers, schools and academic achievement. Econometrica, 73, 417-458.

53.

Roth

J. R.

Garnier

H. E.

Chen

Lemmens

Schwille

Wickler

(2011). Video-based lesson analysis: Effective science PD for teacher and student learning. Journal of Research in Science Teaching, 48, 117-148.

54.

Sandholtz

J. H.

Ringstaff

(2016). The influence of contextual factors on the sustainability of professional development outcomes. Journal of Science Teacher Education, 27(2), 205-226.

55.

Santagata

Kersting

Givvin

K. B.

Stigler

J. W.

(2010). Problem implementation as a lever for change: An experimental study of the effects of a professional development program on students’ mathematics learning. Journal of Research on Educational Effectiveness, 4(1), 1-24.

56.

Scher

O’Reilly

(2009). Professional development for K-12 math and science teachers: What do we really know? Journal of Research on Educational Effectiveness, 2(3), 209-249.

57.

Spiro

R. J.

Collins

B. P.

Thota

J. J.

Feltovich

P. J.

(2003). Cognitive flexibility theory: Hypermedia for complex learning, adaptive knowledge application, and experience acceleration. Educational Technology, 43(5), 5-10.

58.

Steinberg

R. M.

Empson

S. B.

Carpenter

T. P.

(2004). Inquiry into children’s mathematical thinking as a means to teacher change. Journal of Mathematics Teacher Education, 7(3), 237-267.

59.

Stothard

Nicholson

(2001). Skill acquisition and retention in training: DSTO support to the army ammunition study (No. DSTO-CR-0218). Salisbury, Australia: Electronics and Surveillance Research Laboratory, Defence Science & Technology Organisation.

60.

Supovitz

J. A.

Mayer

D. P.

Kahle

J. B.

(2000). Promoting inquiry-based instructional practice: The longitudinal impact of professional development in the context of systemic reform. Educational Policy, 14(3), 331-356.

61.

Tatto

M. T.

Menter

(2019). Understanding teacher education policy and practice cross-nationally. In Tatto

M. T.

Menter

(Eds.), Knowledge, policy and practice in teacher education: A cross-national study (Vol. 1) (pp. 3-8). London, England: Bloomsbury.

62.

Vermunt

J. D.

(2014). Teacher learning and professional development. In Krolak-Schwerdt

Glock

Böhmer

(Eds.), Teachers’ professional development (pp. 79-95). Rotterdam, The Netherlands: Sense Publishers.

63.

Vosniadou

(2007). The cognitive-situative divide and the problem of conceptual change. Educational Psychologist, 42(1), 55-66.

64.

Wilburne

J. M.

Long

(2010). Secondary pre-service teachers’ content knowledge for state assessments: Implications for mathematics education programs. Issues in the Undergraduate Mathematics Preparation of School Teachers, 1. Retrieved from https://files.eric.ed.gov/fulltext/EJ872484.pdf

65.

Wisher

R. A.

Sabol

M. A.

Ellis

(1999). Staying sharp: Retention of military knowledge and skills (U.S. Army Research Institute Special Report 39). Retrieved from https://apps.dtic.mil/docs/citations/ADA366825

66.

Wright

(1936). Factors affecting the cost of airplanes. Journal of Aeronautical Sciences, 3, 122-128.

67.

Yoon

K. S.

Duncan

Lee

S. W. Y.

Scarloss

Shapley

(2007). Reviewing evidence on how teacher professional development affects student achievement (Issues & Answers Report, REL 2007–No.033). Washington, DC: Regional Educational Laboratory Southwest, National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

68.

Zehetmeier

(2008). Zur Nachhaltigkeit von Lerher/innenfortbildung [The sustainability of teacher professional development] (Doctoral thesis). University of Klagenfurt, Klagenfurt, Austria.