The Effects of Teacher Fidelity of Implementation of Pathways to Health on Student Outcomes

Abstract

Previous research has demonstrated the importance of ensuring that programs are implemented as intended by program developers in order to achieve desired program effects. The current study examined implementation fidelity of Pathways to Health (Pathways), a newly developed obesity prevention program for fourth- through sixth-grade children. We explored the associations between self-reported and observed implementation fidelity scores and whether implementation fidelity differed across the first 2 years of program implementation. Additionally, we examined whether implementation fidelity affected program outcomes and whether teacher beliefs were associated with implementation fidelity. The program was better received, and implementation fidelity had more effects on program outcomes in fifth grade than in fourth grade. Findings suggest that implementation in school-based obesity programs may affect junk food intake and intentions to eat healthfully and exercise. School support was associated with implementation fidelity, suggesting that prevention programs may benefit from including a component that boosts school-wide support.

Keywords

translation obesity prevention implementation fidelity school-based

Introduction

Over the last decade, childhood obesity rates in the United States have increased dramatically (Spruijt-Metz, 2011). The significant psychosocial, health-related, and financial costs associated with obesity suggest a need for preventive interventions in childhood (Datar & Sturm, 2004; Koplan, Liverman, & Kraak, 2005; Oude Luttikhuis et al., 2009; Strauss, Rodzilsky, Burack, & Colin, 2001). Schools may be an ideal setting for delivering obesity prevention programs (Dietz & Gortmaker, 2001; Howerton et al., 2007; Story et al., 2000). Reviews and meta-analyses suggest that school-based obesity prevention interventions can be effective, at least in the short term, for reducing the prevalence of childhood obesity, improving protective factors (e.g., consumption of fruits and vegetables, increased physical activity) and creating a healthier environment (Dietz & Gortmaker, 2001; Dobbins, De Corby, Robeson, Husson, & Tirilis, 2009; Gonzalez-Suarez, Worley, Grimmer-Somers, & Dones, 2009). However, school-based obesity prevention programs are not universally successful (Shaya, Flores, Gbarayor, & Wang, 2008). Variation in success may be due in part to differences in program implementation, suggesting the need to focus on the quality of intervention delivery (Domitrovich & Greenberg, 2000).

Implementation Fidelity in Obesity Prevention

Implementation fidelity is the degree to which programs are implemented as intended by program developers in order to achieve desired program effects (Durlak & DuPre, 2008). Numerous school-based health promotion and disease prevention studies have shown that program implementation is associated with program mediators (Resnicow et al., 1998; Rohrbach, Graham, & Hansen, 1993; Rohrbach, Gunning, Sun, & Sussman, 2010) and program outcomes (Durlak & DuPre, 2008; Lillehoj, Griffin, & Spoth, 2004; McGraw et al., 1996; Pentz et al., 1990). Saunders, Ward, Felton, Dowda, and Pate (2006) found that a greater percentage of girls engaged in vigorous physical activity in high-implementing schools than in low-implementing schools. McGraw et al. (1996) found that the percentage of classroom sessions modified by the teacher, a proxy for implementation fidelity, was associated with positive changes in student dietary self-efficacy and knowledge. Authors hypothesized that modifications were in essence tailoring the lessons to the needs of their students, making the program more effective (McGraw et al., 1996). Resnicow et al. (1998) found an association between health knowledge and observed fidelity but no effects on fruit and vegetable intake or asking behaviors. These results are promising but inconclusive.

Contextual Factors

Contextual factors within the school structure are likely to influence the fidelity of implementation of prevention programs (Chen, 1998; Domitrovich et al., 2008). Schools are often strained to meet academic and policy-related priorities. Therefore, when a principal supports the implementation of an intervention, teachers are likely to implement the program with greater fidelity (Gregory, Henry, & Schoeny, 2007; Kam, Greenberg, & Walls, 2003; Ringwalt et al., 2003). Other factors, such as teachers’ beliefs about the benefits of a particular program and comfort implementing the program, have also been shown to increase implementation fidelity (Beets et al., 2008; Ennett et al., 2003; Kallestad & Olweus, 2003; Klimes-Dougan et al., 2009; Little, Sussman, Sun, & Rohrbach, 2013; Ringwalt et al., 2003). The influence of these types of contextual factors on the fidelity of prevention program implementation in schools is an integral, yet often overlooked, part of implementation research.

The Current Study

The present study uniquely contributes to the field of implementation research by exploring the associations between implementation fidelity, contextual factors, and program outcomes in a school-based obesity prevention program. The current study had three objectives: (a) explore the implementation fidelity of the Pathways program across the first 2 years of program implementation, (b) examine associations of implementation fidelity measures with program outcomes of the Pathways curriculum, and (c) explore the associations between teachers’ beliefs about school support for the Pathways program, benefits of the program, comfort implementing the program, and implementation fidelity. We hypothesized that children whose teachers implemented the Pathways program with a high degree of fidelity would demonstrate significantly less sedentary behavior and junk food intake and significantly greater fruit and vegetable intake, physical activity, and intentions to exercise and eat healthy.

Method

Study Design

Data for the present study are from the implementation of a newly developed school-based obesity prevention program, Pathways to Health (Pathways), for fourth- through sixth-grade children. Pathways was adapted from two nationally recognized evidence-based programs, The Midwestern Prevention Project (MMP or STAR; Pentz, Mihalic, & Grotpeter, 1997) for drug abuse prevention and Promoting Alternative THinking Strategies (Greenberg & Kusché, 1993) for violence prevention. A more detailed account of the translation process and curriculum development can be found in Sakuma, Riggs, and Pentz (2011). Data were collected from the 12 schools and 38 classrooms implementing the Pathways curriculum (Riggs, Spruijt-Metz, Chou, & Pents, 2012). Principals selected teachers who had a fourth-grade class for participation. Teachers attended a 1-day in-person training session using methodology adapted from a previous large-scale drug abuse prevention trial (Pentz & Trebow, 1997). The program was implemented in both fourth and fifth grades. Regular classroom teachers delivered the 15-session fourth-grade curriculum over the course of 7 weeks and the 10-session fifth-grade curriculum over the course of 5 weeks. Sessions lasted an average of 45 min; teachers were encouraged to implement the curriculum 2–3 times per week.

Subjects

Students

We approached 1,204 students for consent in the program schools in fourth grade (see Figure 1). Fifty-seven percent of students (N = 685) received parental consent for participation, 24% (N = 294) did not have parental consent and were treated as anonymous, and 19% declined to participate (N = 225). Anonymous students were not included in the current analyses because baseline and follow-up data could not be linked by individual. Of the 685 eligible students, 581 (85%) took the baseline survey. Five hundred and forty-two students completed the fourth-grade follow-up (93% retention rate) and 411 students completed the fifth-grade follow-up (76% retention rate). Students with incomplete teacher-level data (observations and self-reports) were excluded (N = 8 in fourth grade; N = 20 in fifth grade). The final analytic sample for the fourth-grade analyses was 534 students, and the final analytic sample for the fifth-grade analyses was 391 students. At baseline, students were 9.2 years old (SD = 0.5), on average, and a quarter of the sample received free or reduced lunch (24.9%). The sample was 51.5% female, 31.5% White, 3.6% African American, 25.9% Hispanic, 8.2% Asian, and 31.7% mixed ethnicity.

Figure 1.

Flow of study participants.

Teachers

A total of 32 teachers were trained in fourth grade, but 2 were excluded from this study because they did not have both observation and self-report data (N = 30, 94%). In fifth grade, 40 teachers were trained, but 1 was excluded due to incomplete observation data (N = 39, 98%). Individual teacher demographic characteristics were not evaluated in order to increase confidentiality and the validity of teacher self-reports.

Data Collection Procedures and Measures

Student Assessment

Baseline and follow-up measures were collected from students using paper-and-pencil surveys. At each wave of data collection, surveys assessed food intake, physical activity, sedentary behavior, and intentions to eat healthfully and exercise. Surveys took approximately 45 min to complete. Surveys were read aloud to students by research project staff, with a second staff member available to answer comprehension questions. Due to limited time, abbreviated scales were used. Construction of abbreviated scales proceeded through an extensive pilot testing process where full scales of food intake and physical activity were pilot tested and reduced to index items representing the highest loading items for each scale (Riggs, Chou, Spruijt-Metz, & Pentz, 2010).

Food intake

Fruit and vegetable intake was assessed through items adapted from the Youth Risk Behavior Survey (YRBS; Eaton et al., 2006). Three items asked about fruit intake (e.g., How often did you eat any fruit, fresh, or canned?) and 4 items assessed vegetable intake (7 items total, α = .74; Riggs et al., 2010). YRBS does not assess high fat or high sugar items, so 5 additional items were added from a previously validated food frequency questionnaire (Willett et al., 1985), which asked how often students consumed items such as candy, soda, potato chips, French fries, and pastries. Response choices ranged from 1 = less than once a week to 6 = two or more of these a day (5 items, α = .80).

Physical activity

Children’s level of physical activity was assessed using a shorter version of the Physical Activity Questionnaire for Children (Crocker, Bailey, Faulkner, Kowalski, & McGrath, 1997) that included 3 of the original 9 items (Riggs et al., 2010). These items assessed the level of activity outside of school (i.e., immediately after school, in the evenings, and on the weekend). Items asked, “In the last 7 days, (immediately after school, in the evenings, on the weekend) how often were you very active?” Response choices ranged from 1 = none to 5 = six or more times (3 items, α = .77).

Sedentary behavior

Sedentary behavior was assessed by 3 items on watching TV, playing videogames, and using a computer, adapted from the School-Based Nutrition Monitoring Student Questionnaire (Hoelscher, Day, Kelder, & Ward, 2003; Penkilo, George, & Hoelscher, 2008). Items asked, “On a regular school day, how many hours per day do you (a) usually watch TV or video movies at home or away from school, (b) spend on a computer at home or away from school and (c) play video games that you sit down to play like PlayStation, Xbox, GameBoy, or arcade games?” (3 items, α = .60; Riggs et al., 2010).

Intentions

Four items assessed students’ intentions to eat healthier and exercise more, adapted from a previous research trial (Pentz, Cormack, Flay, Hansen, & Johnson, 1986). These items are hypothesized to lead to behavior change based on the theory of planned behavior (Ajzen, 1991). Items asked, “When you get home today from school will you (e.g., choose to watch less TV than normal)?” Response choices were as follows: 1 = no, 2 = maybe and 3 = yes (α = .59).

Implementation Fidelity

Both observation and self-report methods assessed implementation fidelity. Project staff observed teachers during two sessions (fourth-grade curriculum Sessions #7 and #13; fifth-grade curriculum Sessions #6 and #9). These sessions were chosen because they were highly interactive and thought to be more difficult to implement (Durlak & DuPre, 2008). Self-reported implementation was assessed and measured using a standard form adapted from a previous research trial (Pentz et al., 1990). Teachers received the form prior to implementation and were instructed to complete it throughout the implementation period and return it in a prestamped envelope after program completion.

Observed implementation

A multidimensional implementation fidelity index was generated using two components: participant engagement (the extent of engagement of participants within the intervention, beyond exposure) and quality of delivery (as measured by teacher’s enthusiasm, integrity, and quality) (Dane & Schneider, 1998; Durlak & DuPre, 2008; Linnan & Steckler, 2000; Resnicow et al., 1998; Rohrbach et al., 2010). Three items assessed observed participant engagement (α = .80_{fourth grade}; α = .80_{fifth grade}). A sample item asked, “Approximately what percentage of the students actively participated in the discussions, role-plays or activities?” scored 1 = less than 10% to 5 = 76–100%. Six items assessed observed quality of delivery (α = .76_{fourth grade}; α = .76_{fifth grade}). A sample item asked, “How much enthusiasm did the teacher demonstrate for the program?” scored 1 = almost no enthusiasm, just went through the motions to 5 = very enthusiastic, seemed really involved and excited. Since these two indices were highly correlated (r = .73_{fourth grade}; r = .80_{fifth grade}), individual items were averaged to create a composite observed implementation fidelity score (α = .84_{fourth grade}; α = .89_{fifth grade}).

Self-reported implementation and beliefs

The self-report implementation instrument assessed implementation fidelity through three components: participant engagement, fidelity (as measured through quality of delivery), and dose delivered (Dane & Schneider, 1998; Durlak & DuPre, 2008; Linnan & Steckler, 2000). One item assessed self-reported participant engagement, “On average, when active participation of your students was required, what percentage of your students actively participated?” scored 1 = less than 25% to 4 = 76–100%. Self-reported quality was assessed by asking for each session, “How much do you think this lesson achieved its objectives?” scored 1 = not at all to 5 = very much (α = .95_{fourth grade}; α = .98_{fifth grade}). These two indices were correlated (r = .29_{fourth grade}; r = .67_{fifth grade}), and individual items were averaged to create a composite self-reported implementation fidelity score (α = .95_{fourth grade}; α = .98_{fifth grade}). To assess self-reported dose delivered, teachers reported for each session “Did you teach it?” scored 0 = no to 1 = yes. These items were summed to create an overall self-reported dose delivered index. However, this index was highly skewed (see Table 1) and was not included in the overall self-reported implementation fidelity score.

Table 1.

Implementation Fidelity Scores and Teacher Beliefs, M (SD) or %.

Characteristic	Fourth grade	Fifth grade
Characteristic	(n = 534)	(n = 391)
Observed implementation fidelity score	4.72 (0.46)	4.61 (0.55)*
Observed participant engagement^a	4.60 (0.42)	4.54 (0.54)
Observed fidelity^a	4.84 (0.56)	4.68 (0.63)*
Self-reported implementation fidelity score	3.52 (0.58)	3.87 (0.75)*
Self-reported participant engagement^b	3.44 (0.80)	3.63 (0.77)*
Self-reported quality^a	3.63 (0.65)	4.11 (0.89)*
Self-reported dose delivered^c	14.36 (1.41)	9.91(0.74)
Comfort implementing the program	3.17 (0.72)	3.28 (0.84)*
Perceived benefits of the program	3.36 (0.74)	3.38 (0.96)
Perceived school support for the program	3.69 (0.99)	4.06 (1.01)*

Note. ^aScale ranges from 1 to 5. ^bScale ranges from 1 to 4. ^cThere were 15 sessions in the fourth-grade curriculum and 10 sessions in the fifth-grade curriculum; t-tests were not performed for these items.

*p < .05.

Teacher beliefs about the program

Perceived school support for the Pathways program was assessed through 2 items (e.g., “Most administrators at my school were supportive of my teaching Pathways.”) scored 1 = strongly disagree to 5 strongly agree (α = .71_{fourth grade}; α = .78_{fifth grade}). Perceived benefits of the program were assessed through 3 items (e.g., “Pathways is an effective way to prevent unhealthy eating.”) scored 1 = strongly disagree to 5 strongly agree (α = .62_{fourth grade}; α = .68_{fifth grade}). Comfort implementing the program was assessed through 9 items (e.g., “Pathways fits well with the way I like to teach.”) scored 1 = strongly disagree to 5 strongly agree (α = .88_{fourth grade}; α = .92_{fifth grade}).

Data Analysis

The first objective was to explore the implementation fidelity of the Pathways program in the first 2 years of program implementation. We conducted Pearson product moment correlations to test associations between self-report and observed implementation fidelity scores. Next, we utilized t-tests to explore differences in implementation fidelity between the fourth- and fifth-grade implementation. The second objective was to examine associations between implementation fidelity measures and program outcomes of the Pathways curriculum. General mixed linear modeling (Murray & Hannan, 1990) was conducted to assess the relationship between implementation fidelity and program outcomes. Classroom and school were considered random factors, which allowed us to statistically account for the intraclass correlation within clustered units on computed significance levels. Analyses adjusted for student gender, ethnicity, reduced/free lunch status, the baseline value of the dependent variable, and a propensity-for-attrition score (to be described later). Fifth-grade implementation models controlled for fourth-grade fidelity scores because the program was implemented in both grades. βs and standard errors are reported. The third objective was to test the associations between teachers’ perceived school support, benefits, and comfort implementing the Pathways program and implementation fidelity. We ran Pearson product moment correlations to explore associations between contextual factors and implementation fidelity. Next, we utilized t-tests to explore differences in teachers’ beliefs between the fourth- and fifth-grade implementation. All analyses were conducted using Statistical Analysis System (SAS) statistical package (SAS Institute Inc. SAS/C Online Doc TM, 2000).

Results

Assessment of Attrition Bias

To assess the potential sample bias due to subject attrition at the 1-year follow-up, an attrition analysis was conducted on the analytic sample (n = 391) and the sample that was lost to follow-up (n = 143) on seven key baseline measures. The measures were age, gender, ethnicity, reduced or free lunch, and the five program outcomes. The comparisons used chi-square or t-tests to explore differences between the two groups. Analyses revealed statistically significant incomparability between lost to follow-up and retained subjects on gender, ethnicity, receiving reduced or free lunch, junk food intake, and sedentary behavior. Retained subjects were less likely to be female, 48.1% versus 60.8%; χ²(1, N = 534) = 6.82, p < .01; less likely to be Hispanic, 22.5% versus 32.2%, χ²(1, N = 534) = 5.20, p < .05; less likely to receive free lunch, 16.9% versus 46.9%, χ²(1, N = 534) = 50.30, p < .001; less likely to consume junk food, 2.33 versus 2.60, t(218) = −2.44, p < .05; and less likely to be sedentary, 2.49 versus 2.81, t(529) = −2.69, p < .01. To assess whether our results are generalizable to those students not included in the current study, an analysis was conducted between the consented students (N = 581) and the anonymous students (N = 198; i.e., students who did not have parental consent and were treated as anonymous) surveyed at baseline. The comparisons used chi-square or t-tests to explore differences between the two groups. Analyses revealed statistical incomparability between the consented students and the anonymous students on gender, ethnicity, receiving reduced or free lunch, junk food intake, and sedentary behavior. Consented students were more likely to be female, 51.6% vs. 42.4%; χ²(1, N = 779) = 5.01, p < .05; less likely to be Hispanic, 25.7% versus 33.5%, χ²(1, N = 779) = 4.53, p < .05; less likely to receive free lunch, 27.1% versus 41.1%, χ²(1, N = 779) = 13.65, p < .001; less likely to consume junk food, 2.38 versus 2.56, t(772) = −2.07, p < .05; and less likely to be sedentary, 2.57 versus 2.95, t(770) = −3.50, p < .01. Due to these differences, the extent of this generalizability is restricted to a population with baseline measurement access restrictions such as those experienced in this study.

To statistically adjust for possible bias introduced by nonrandom attrition at 1-year follow-up, a “propensity-for-attrition” score was computed for each subject retained at the follow-up, and adjusted for in each of the statistical models (Rosenbaum & Rubin, 1984). This score was computed among the entire baseline sample by associating the difference in the selected baseline measures (gender, ethnicity, receiving reduced or free lunch, junk food intake, and sedentary behavior) to the actual attrition status in a multiple regression analysis, and then assuming the association is also maintained among the subjects retained at the 1-year follow-up.

Implementation Fidelity in Fourth and Fifth Grade

Results comparing implementation fidelity items in fourth and fifth grade are presented in Table 1. In fourth grade, 81.1% of teachers reported implementing all 15 sessions, and in fifth grade, 97.6% of teachers reported implementing all 10 sessions. Both self-reported participant engagement and implementation quality were significantly higher in fifth than fourth grade (ps < .05), while observed fidelity was lower in fifth than fourth grade (p < .05). There was no significant difference across years for observed participant engagement. In fourth grade, observed implementation fidelity scores were negatively correlated with self-reported scores (r = −.10, p < .05) but, in fifth grade, were positively correlated (r = .72, p < .001).

Implementation Fidelity and Program Outcomes

Table 2 shows results of the mixed linear regression models exploring the relationship between observed and self-reported implementation fidelity and program outcomes in fourth and fifth grades. There were no significant effects in the fourth-grade models. In fifth grade, both observed and self-reported implementation fidelity were significantly associated with higher intentions to eat healthfully and exercise and lower junk food intake (ps < .05). There were no significant effects on fruit and vegetable intake, physical activity outside of school, or sedentary behavior.

Table 2.

Parameter Estimates From Mixed Effects Models for Implementation Effects.

	Fruit and vegetable consumption	Junk food	Intentions	PA out of school	Sedentary behavior
Fourth grade
Observed fidelity	0.03 (0.08)	0.01 (0.09)	0.00 (0.04)	0.04 (0.12)	−0.07 (0.11)
Self-reported	0.04 (0.06)	−0.06 (0.08)	0.02 (0.04)	−0.16 (0.10)	−0.11 (0.09)
Fifth grade
Observed fidelity	0.15 (0.09)	−0.38 (0.10)**	0.13 (0.05)*	0.15 (0.10)	−0.07 (0.12)
Self-reported	0.03 (0.07)	−0.30 (0.08)**	0.12 (0.04)*	0.00 (0.07)	−0.15 (0.09)

Note. Values are βs and standard errors. Two-tailed significance tests were used.

Random effects adjusted for in the models included school and classroom (teacher); Fixed effects controlled for in the models included propensity-for-attrition score, student gender, ethnicity, socioeconomic status, the baseline value of the dependent variable, and fourth-grade fidelity score (only in fifth-grade models).

*p < .01. **p < .001.

Implementation Fidelity and Teacher Beliefs

Perceived beliefs about school support for the Pathways program and comfort implementing the program were higher in fifth than fourth grade (ps < .05; see Table 1). There were no significant differences in perceived benefits of the program between grades. The results exploring associations between teacher beliefs and implementation fidelity in fourth and fifth grades are presented in Table 3. In both fourth and fifth grades, perceiving school support for the Pathways program, perceiving positive benefits of the program, and having comfort implementing the program were all significantly associated with increased self-report and observed implementation fidelity (ps < .05). In fifth grade, the associations were stronger than in fourth grade.

Table 3.

Correlation Matrix of Implementation Fidelity and Teacher Beliefs.

	Implementation fidelity
	Observed	Self-reported
Fourth grade
Comfort implementing the program	0.23	0.48
Perceived benefits of the program	0.13	0.46
Perceived school support for the program	0.22	0.21
Fifth grade
Comfort implementing the program	0.36	0.60
Perceived benefits of the program	0.49	0.58
Perceived school support for the program	0.36	0.54

Note. Pearson product moment correlations of data presented. All correlations are statistically significant at p < .05.

Discussion

The current study examined implementation fidelity of the Pathways program in the first 2 years, associations between self-reported and observed implementation fidelity, effects of implementation on health outcomes, and associations between teacher beliefs and implementation fidelity. While there was a high level of dose delivered across the fourth- and fifth-grade implementation (81.1% and 97.6%, respectively), it appears that fifth-grade implementation was associated with program outcomes.

In fourth grade, observed implementation fidelity scores were negatively correlated with self-reported scores (r = −.14, p < .01), revealing discrepant perceptions between staff observers and teachers. In contrast, in fifth grade, observed implementation fidelity scores were positively correlated with self-reported scores (r = .74, p < .0001), indicating that staff observers and teachers had similar perceptions of implementation fidelity. A limited number of studies have used multiple methods to assess implementation fidelity, and within these studies a wide range of agreement between observed and self-reported implementation has been reported (Hansen & McNeal, 1999; Lillehoj et al., 2004). Self-reported adherence may overestimate adherence, compared to objective assessments, possibly due to social desirability or interviewer bias (Adams, Soumerai, Lomas, & Ross-Degnan, 1999). Self-reports may be more inaccurate compared to observed data when respondents fear consequences from the results, but self-reports may be more valid measures for less sensitive information, such as height (Donaldson & Grant-Vallone, 2002). We used different measures and assessed slightly different components of implementation fidelity in self-report versus objective assessments. Also, different teachers implemented the curriculum in fourth and fifth grades. In addition, the program was tailored based on teacher feedback following fourth-grade implementation, and changes may have enhanced teachers’ understanding and improved their ability to align program delivery with the intended curriculum. Unfortunately, because we did not assess teacher background characteristics, such as teaching experience, it is impossible to rule out which, if any, of these factors had an effect on our results. Therefore, our results should be interpreted with caution but support the use of multimethod (i.e., self-report and observed) assessments to increase accuracy.

Consistent with previous research, we found that perceived school support, belief that the program was beneficial, and comfort with implementation were associated with higher levels of implementation fidelity (Beets et al., 2008; Ennett et al., 2003; Gregory et al., 2007; Kallestad & Olweus, 2003; Kam et al., 2003; Klimes-Dougan et al., 2009; Ringwalt et al., 2003). Given the considerable pressures teachers face to prepare students for standardized tests, prevention curricula are often seen as ancillary (Ringwalt et al., 2003). Therefore, it is important to ensure that teachers receive support from the principal and school staff. Principals can be central in promoting positive attitudes toward the program among parents and the community and in fostering a sense of collaboration among teachers and other staff to assist in the implementation of a prevention program (Kam et al., 2003). Adequate training also helps ensure that teachers feel comfortable implementing interactive sessions.

Interestingly, comfort with the program and perceived school support for the program were both higher among fifth-grade teachers than fourth-grade teachers, which may explain why fifth-grade teachers reported higher levels of implementation fidelity. It is unclear why observed implementation fidelity was lower in fifth grade compared to fourth grade. Because only two sessions were observed, one potential explanation is that the sessions observed in fifth grade were more challenging to implement. Previous studies have found that implementation fidelity is often lower in sessions that are more difficult to implement (Botvin, Baker, Filazzola, & Botvin, 1990; Hahn, Noland, Rayens, & Christie, 2002; Kallestad & Olweus, 2003). Since observed fidelity was relatively high among both fourth- and fifth-grade teachers, the question remains open. Surprisingly, we found that implementation fidelity affected two of the five program outcomes in fifth grade but had no effect in fourth grade. A number of potential explanations could explain these unexpected findings. In fourth grade, teachers reported making significantly more adaptations to the curriculum compared to fifth grade (p < .05). Although researchers expect teachers to implement the program without changing the core elements and internal logic of an intervention, often programs are adapted to meet the needs of the program recipients (Dusenbury, Brannigan, Hansen, Walsh, & Falco, 2005). Although we do not have specific details regarding the adaptations made, despite asking teachers to report the details of their adaptations on the self-report survey, it appears that these adaptations negatively affected program outcomes, which could be a reason we did not observe associations between implementation fidelity and program outcomes in the fourth-grade curriculum. Similarly, McGraw et al. (1996) found that teachers who implemented their program made significant adaptations to their curriculum, altering the effectiveness of the study. In that case, however, the adaptations were positive and made the more program more effective (McGraw et al., 1996). Based on the frequency with which adaptations are made in the adoption of prevention curriculum, program developers need to be aware of the reasons teachers make modifications and develop recommendations to ensure program goals are realized (Dusenbury et al., 2005). Additionally, it could be useful to provide additional support and coaching to teachers as they deliver core elements of the intervention. The provision of technical assistance has been associated with improved implementation of prevention interventions in schools (Fagan, Hanson, Hawkins, & Arthur, 2008; Gingiss, Roberts-Gray, & Boerm, 2006; Roberts-Gray, Gingiss, & Boerm, 2007; Rohrbach et al., 2010).

Another potential explanation for the unexpected findings is that multiple iterations of program delivery are required before the effects are observed. The Pathways program focuses on changing child impulse control, decision making, and social competence through interactive classroom lessons (Sakuma, Riggs, & Pentz, 2011). Altering these cognitive functions in children takes time, and changes may not be visible immediately, possibly explaining why we did not see changes in student behavior immediately after fourth-grade implementation but did after fifth-grade implementation. However, additional longitudinal analysis will be required to test whether the curriculum was able to change health behaviors over time. Additionally, the Pathways curriculum could be too mature for fourth graders, which could explain the lack of effects. However, future studies will need to replicate the program during more controlled conditions to disentangle these possibilities.

Strengths and Limitations

The current study contributes to the field of implementation research by examining the associations between implementation fidelity, contextual factors, and program outcomes over the course of 2 years of program implementation. The Pathways curriculum was taught in both fourth and fifth grades, focusing on different skills in each grade. Using feedback from the fourth-grade teachers, the fifth-grade curriculum was refined and strengthened. These improvements, however, may have made it difficult to disentangle the longitudinal effects of the fourth-grade curriculum on health outcomes from the proximate effects of the fifth-grade curriculum. One limitation of the study design is that students received both. However, the current analysis attempted to separate effects statistically by controlling for fourth-grade implementation fidelity scores when examining fifth-grade outcomes.

Another limitation is that two contextual factors that may have been especially valuable for assessing changes in school environment were not measured. First, this study occurred during the height of the 2009 economic collapse. Four participating schools closed between fourth- and fifth-grade implementation. Although these closures placed a large burden on research program staff to track students who switched to new schools, retention remained high in fifth grade (76%). However, a climate of school closings may have affected morale at schools that remained open. Second, many of the fourth-grade teachers received pink slips during the study, which could have affected the quality of program implementation. Unfortunately, lowered morale may be a common problem that researchers face when working in schools in the current economic climate.

Finally, the use of a single item indicator to assess self-reported participant engagement could have masked important findings. However, the αs for self-reported implementation fidelity are high in both fourth and fifth grade suggesting that the items constitute one index and using both self-reported and observed ratings of implementation fidelity is more reliable that using a single method.

Conclusion

Given the enormous burdens schools are under to meet academic and policy-related priorities and significant reductions in public funds, it is crucial that future studies include measures of school and community-level contextual factors (Durlak & DuPre, 2008). In the current study, these burdens may have had a significant impact on the implementation and effectiveness of the Pathways program. The program was better received and implementation fidelity had more effects on program outcomes in fifth grade compared to fourth grade, when much of the initial turmoil of the economic collapse had settled. Additionally, school support was associated with implementation fidelity, suggesting that prevention programs may benefit from including a component that boosts school-wide support.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by grant HD 052107 (NICHD, NIDA; Pentz, P.I.), and is registered at (#NCT00787709). ML was supported during the work on this project by a postdoctoral fellowship on grant R25 CA90956.

References

Adams

A. S.

Soumerai

S. B.

Lomas

Ross-Degnan

(1999). Evidence of self-report bias in assessing adherence to guidelines. International Journal for Quality in Health Care, 11, 187–192. doi:10.1093/intqhc/11.3.187

Ajzen

(1991). The theory of planned behavior. Organizational Behavior and Human Decision Process, 50, 179–211.

Beets

Flay

Vuchinich

Acock

K.-K.

Allred

(2008). School climate and teachers’ beliefs and attitudes associated with implementation of the positive action program: A diffusion of innovations model. Prevention Science, 9, 264–275.

Botvin

G. J.

Baker

Filazzola

A. D.

Botvin

E. M.

(1990). A cognitive-behavioral approach to substance abuse prevention: One-year follow-up. Addictive Behaviors, 15, 47–63.

Chen

H.-T.

(1998). Theory-driven evaluations. Advances in Educational Productivity, 7, 15–34.

Crocker

P. R.

Bailey

D. A.

Faulkner

R. A.

Kowalski

K. C.

McGrath

(1997). Measuring general levels of physical activity: Preliminary evidence for the physical activity questionnaire for older children. Medicine & Science in Sports & Exercise, 29, 1344–1349.

Dane

A. V.

Schneider

B. H.

(1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23–45. doi:10.1016/s0272-7358(97)00043-3

Datar

Sturm

(2004). Childhood overweight and parent- and teacher-reported behavior problems: Evidence from a progressive study of kindergartners. Archives of Pediatric and Adolescent Medicine, 158, 804–810.

Dietz

W. H.

Gortmaker

S. L.

(2001). Preventing obesity in children and adolescents. Annual Review of Public Health, 22, 337–353. doi:10.1146/annurev.publhealth.22.1.337

10.

Dobbins

De Corby

Robeson

Husson

Tirilis

(2009). School-based physical activity programs for promoting physical activity and fitness in children and adolescents aged 6-18. Cochrane Database of Systematic Reviews, 1. doi:10.1002/14651858.cd007651

11.

Domitrovich

C. E.

Bradshaw

C. P.

Poduska

J. M.

Hoagwood

Buckley

J. A.

Olin

… Ialongo

N. S.

(2008). Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Advances in School Mental Health Promotion, 1, 6–28.

12.

Domitrovich

C. E.

Greenberg

M. T.

(2000). The study of implementation: Current findings from effective programs that prevent mental disorders in school-aged children. Journal of Educational and Psychological Consultation, 11, 193–221.

13.

Donaldson

S. I.

Grant-Vallone

E. J.

(2002). Understanding self-report bias in organizational behavior research. Journal of Business and Psychology, 17, 245–260. doi:10.1023/a:1019637632584

14.

Durlak

J. A.

DuPre

E. P.

(2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. doi:10.1007/s10464-008-9165-0.

15.

Dusenbury

Brannigan

Hansen

W. B.

Walsh

Falco

(2005). Quality of implementation: Developing measures crucial to understanding the diffusion of preventive interventions. Health Education Research, 20, 308–313. doi:10.1093/her/cyg134

16.

Eaton

D. K.

Kann

Kinchen

Ross

Hawkins

Harris

W. A.

… Wechsler

(2006). Youth risk behavior surveillance—United States, 2005. MMWR Surveillance Summaries, 55, 1–108. doi:ss5505a1 [pii]

17.

Ennett

S. T.

Ringwalt

C. L.

Thorne

Rohrbach

L. A.

Vincus

Simons-Rudolph

Jones

(2003). A comparison of current practice in school-based substance use prevention programs with meta-analysis findings. Prevention Science, 4, 1–14.

18.

Fagan

A. A.

Hanson

Hawkins

J. D.

Arthur

M. W.

(2008). Bridging science to practice: Achieving prevention program implementation fidelity in the community youth development study. American Journal of Community Psychology, 41, 235–249.

19.

Gingiss

P. M.

Roberts-Gray

Boerm

(2006). Bridge-It: A system for predicting implementation fidelity for school-based tobacco prevention programs. Prevention Science, 7, 197–207.

20.

Gonzalez-Suarez

Worley

Grimmer-Somers

Dones

(2009). School-based interventions on childhood obesity: A meta-analysis. American Journal of Preventive Medicine, 37, 418–427.

21.

Greenberg

M. T.

Kusché

C. A.

(1993). Promoting social and emotional development in deaf children: The PATHS project. Seattle, WA: University of Washington Press.

22.

Gregory

Henry

D. B.

Schoeny

M. E.

(2007). School climate and implementation of a preventive intervention. American Journal of Community Psychology, 40, 250–260.

23.

Hahn

E. J.

Noland

M. P.

Rayens

M. K.

Christie

D. M.

(2002). Efficacy of training and fidelity of implementation of the life skills training program. The Journal of School Health, 72, 282.

24.

Hansen

W. B.

McNeal

R. B.

(1999). Drug education practice: Results of an observational study. Health Education Research, 14, 85–97. doi:10.1093/her/14.1.85

25.

Hoelscher

D. M.

Day

R. S.

Kelder

S. H.

Ward

J. L.

(2003). Reproducibility and validity of the secondary level School-Based Nutrition Monitoring student questionnaire. Journal of the American Dietetic Association, 103, 186–194. doi:10.1053/jada.2003.50031 [doi] S0002822302000354 [pii]

26.

Howerton

M. W.

Bell

B. S.

Dodd

K. W.

Berrigan

Stolzenberg-Solomon

Nebeling

(2007). School-based nutrition programs produced a moderate increase in fruit and vegetable consumption: Meta and pooling analyses from 7 studies. Journal of Nutrition Education and Behavior, 39, 186–196.

27.

Kallestad

J. H.

Olweus

(2003). Predicting teachers’ and schools’ implementation of the olweus bullying prevention program: A multilevel study. Prevention & Treatment, 6, 3–21.

28.

Kam

C.-M.

Greenberg

M. T.

Walls

C. T.

(2003). Examining the role of implementation quality in school-based prevention using the PATHS curriculum. Prevention Science, 4, 55–63.

29.

Klimes-Dougan

August

G. J.

Lee

C.-Y. S.

Realmuto

G. M.

Bloomquist

M. L.

Horowitz

J. L.

Eisenberg

T. L.

(2009). Practitioner and site characteristics that relate to fidelity of implementation: The Early Risers prevention program in a going-to-scale intervention trial. Professional Psychology: Research and Practice, 40, 467–475.

30.

Koplan

Liverman

C. T.

Kraak

V. I.

(2005). Preventing childhood obesity: Health in the balance. Washington, DC: National Academies Press.

31.

Lillehoj

C. J.

Griffin

K. W.

Spoth

(2004). Program provider and observer ratings of school-based preventive intervention implementation: Agreement and relation to youth outcomes. Health Education & Behavior, 31, 242–257. doi:10.1177/1090198103260514

32.

Linnan

Steckler

(2000). Process evaluation and public health interventions: An overview. In Steckler

Linnan

(Eds.), Process evaluation for public health interventions and research (pp. 1–23). San Francisco, CA: Jossey-Bass.

33.

Little

M. A.

Sussman

Sun

Rohrbach

L. A.

(2013). The effects of implementation fidelity in the towards no drug abuse dissemination trial. Health Education, 113, 281–296.

34.

McGraw

S. A.

Sellers

D. E.

Johnson

C. C.

Stone

E. J.

Bachman

K. J.

Bebchuk

… Edmundson

E. W.

(1996). Using process data to explain outcomes: An illustration from the child and adolescent trial for cardiovascular health (CATCH). Evaluation Review, 20, 291–312. doi:10.1177/0193841x9602000304

35.

Murray

D. M.

Hannan

P. J.

(1990). Planning for the appropriate analysis in school-based drug-use prevention studies. Journal of Consulting and Clinical Psychology, 58, 458–468.

36.

Oude Luttikhuis

Baur

Jansen

Shrewsbury

V. A.

O’Malley

Stolk

R. P.

Summerbell

C. D.

(2009). Interventions for treating obesity in children. Cochrane Database of Systematic Review, 1, CD001872. doi:10.1002/14651858.CD001872.pub2 [doi]

37.

Penkilo

George

G. C.

Hoelscher

D. M.

(2008). Reproducibility of the school-based nutrition monitoring questionnaire among fourth-grade students in Texas. Journal of Nutrition Education and Behavior, 40, 20–27.

38.

Pentz

M. A.

Cormack

Flay

Hansen

W. B.

Johnson

C. A.

(1986). Balancing program and research integrity in community drug abuse prevention: Project STAR approach. Journal of School Health, 56, 389–393. doi:10.1111/j.1746-1561.1986.tb05779.x

39.

Pentz

M. A.

Mihalic

S. F.

Grotpeter

J. K.

(Eds.). (1997). The midwestern prevention project. Boulder, CO: Center for the Study and Prevention of Violence, Institute of Behavioral Science, University of Colorado.

40.

Pentz

M. A.

Trebow

(1997). Implementation issues in drug abuse prevention research. Substance Use & Misuse, 32, 1655–1660.

41.

Pentz

M. A.

Trebow

E. A.

Hansen

W. B.

MacKinnon

D. P.

Dwyer

J. H.

Johnson

C. A.

… Cormack

(1990). Effects of program implementation on adolescent drug use behavior: The Midwestern prevention project (MPP). Evaluation Review, 14, 264–289. doi:10.1177/0193841x9001400303

42.

Resnicow

Davis

Smith

Lazarus-Yaroch

Baranowski

… Wang

D. T.

(1998). How best to measure implementation of school health curricula: A comparison of three measures. Health Education Research, 13, 239–250. doi:10.1093/her/13.2.239

43.

Riggs

Chou

C. P.

Spruijt-Metz

Pentz

M. A.

(2010). Executive cognitive function as a correlate and predictor of child food intake and physical activity. Child Neuropsychol, 16, 279–292. doi:919923832 [pii] 10.1080/09297041003601488 [doi]

44.

Riggs

N. R.

Spruijt-Metz

Chou

C. P.

Pents

M. A.

(2012). Relationships between executive cognitive function and lifetime substance use and obesity-related behaviors in fourth grade youth. Child Neuropsychology, 18, 1–11.

45.

Ringwalt

C. L.

Ennett

Johnson

Rohrbach

L. A.

Simons-Rudolph

Vincus

Thorne

(2003). Factors associated with fidelity to substance use prevention curriculum guides in the nation’s middle schools. Health Education & Behavior, 30, 375–391. doi:10.1177/1090198103030003010

46.

Roberts-Gray

Gingiss

P. M.

Boerm

(2007). Evaluating school capacity to implement new programs. Evaluation and Program Planning, 30, 247–257.

47.

Rohrbach

Gunning

Sun

Sussman

(2010). The project towards no drug abuse (TND) dissemination trial: Implementation fidelity and immediate outcomes. Prevention Science, 11, 77–88. doi:10.1007/s11121-009-0151-z

48.

Rohrbach

L. A.

Graham

J. W.

Hansen

W. B.

(1993). Diffusion of a school-based substance abuse prevention program: Predictors of program implementation. Preventive Medicine, 22, 237–260.

49.

Rosenbaum

P. R.

Rubin

D. B.

(1984). Difficulties with regression analyses of age-adjusted rates. Biometrics, 40, 437–443.

50.

Sakuma

K.-L. K.

Riggs

N. R.

Pentz

M. A.

(2011). Translating evidence based violence and drug use prevention to obesity prevention: Development and construction of the Pathways program. Health Education Research, 27, 343–358.

51.

SAS Institute Inc. SAS/C Online Doc TM (2000). Release 9.0. Cary, NC: Author.

52.

Saunders

R. P.

Ward

Felton

G. M.

Dowda

Pate

R. R.

(2006). Examining the link between program implementation and behavior outcomes in the lifestyle education for activity program (LEAP). Evaluation and Program Planning, 29, 352–364. doi:10.1016/j.evalprogplan.2006.08.006

53.

Shaya

F. T.

Flores

Gbarayor

C. M.

Wang

J. S.

(2008). School-based obesity interventions: A literature review. Journal of School Health, 78, 189–196. doi:10.1111/j.1746-1561.2008.00285.x

54.

Spruijt-Metz

(2011). Etiology, treatment, and prevention of obesity in childhood and adolescence: A decade in review. Journal of Research on Adolescence, 21, 129–152. doi:10.1111/j.1532-7795.2010.00719.x

55.

Story

Mays

R. W.

Bishop

D. B.

Perry

C. L.

Taylor

Smyth

Gray

(2000). 5-a-Day power plus: Process evaluation of a multicomponent elementary school program to increase fruit and vegetable consumption. Health Education & Behavior, 27, 187–200. doi:10.1177/109019810002700205

56.

Strauss

R. S.

Rodzilsky

Burack

Colin

(2001). Psychosocial correlates of physical activity in healthy children. Archives of Pediatric and Adolescent Medicine, 155, 897–902.

57.

Willett

W. C.

Sampson

Stampfer

M. J.

Rosner

Bain

Witschi

… Speizer

F. E.

(1985). Reproducibility and validity of a semiquantitative food frequency questionnaire. American Journal of Epidemiology, 122, 51–65.