Abstract
Purpose:
To examine whether deviation from fidelity in the implementation of an evidence-based program on safer sex education affected youth satisfaction and intention to avoid risky sexual behaviors.
Design:
Implementation evaluation.
Setting:
In-school and out-of-school settings in South Carolina.
Participants:
Three thousand seventy-three youths aged 10 to 14 years.
Intervention:
Making Proud Choices.
Measures:
Fidelity variables were implementation setting, program length, class size, gender composition, and curriculum adaptations. Outcome variables were youth program satisfaction and intentions to remain abstinent or avoid risky sexual behaviors postintervention.
Analysis:
Chi-square and t tests tested in-school and out-of-school comparisons. Multiple linear regression examined predictors of youth program satisfaction and intention to avoid risky behaviors.
Results:
Program duration (B = .002), class size (B = .074), program completion rate (B = .004), gender (B = .223), and race (B = .263) predicted program satisfaction (P < .05; R2 = 0.094). Longer program duration was negatively associated with planned abstinence (B = −.002), contraception use (B = −.004), and condom use (B = −.002). Participants in single-gender classes (B = .387) and females (B = .256) were more likely to practice safer sex. Low R2 showed limited impact on intention to practice safer sex (R2 = .030 and.015) and remaining abstinent (R2 = .033).
Conclusion:
Although fidelity deviations do impact youth satisfaction and intentions, the impact is not large. Implementations adapted to fit local settings can still be beneficial.
Keywords
Purpose
The United States has high teen birth rate (18.8 per 1000 15-19 years old women in 2017) and the highest teen pregnancy rate among developed countries. 1 -3 There were 57 pregnancies per 1000 US female teens in 2011 (pregnancy data are not available past 2011). Compared to the United States, teen pregnancy rate was almost half in other developed countries. 2,4,5 Although the US birth rate declined from 41.5 per 1000 females in 2007 to a record low of 18.8 in 2017, the rate remains exceptionally high. 3 South Carolina (SC), the location of this study, had the 17th highest teen birth rate in the United States (23.7 births per 1000 females) in 2016. 6,7
In addition to teen pregnancy, unsafe sex increases youths’ exposure to sexually transmitted diseases (STDs). There was significant increase in all STIs for youth aged 15 to 19 years in the United States in 2016. 8 As part of the Personal Responsibility Education Program funded through the US Federal Family Youth Services Bureau, a multiagency group implemented Making Proud Choices (MPC) in SC.
In evidence-based programs, implementation fidelity is a determinant for successful interventions. 9,10 Although a high degree of program fidelity could generate better results, many implementations stray from fidelity guidelines. 11 Programs might be expanded or truncated to fit into allotted class time or spread over a longer period of time (ie, several weeks) than intended. Activities may be deleted due to local circumstances or implemented with smaller groups than intended. It is important to understand how these alterations may affect youth perception of a program, and their intention to make positive choices. This work examines whether divergence from fidelity in the implementation of the evidence-based program MPC affected youths’ program satisfaction and intention to avoid sexual behaviors.
Methods
Intervention
The MPC curriculum was designed for middle school–aged adolescents to provide the knowledge, confidence, and skills necessary to reduce adolescent risk of STDs, HIV, and pregnancy via sexual abstinence or contraception. 12 The MPC curriculum also aims to increase adolescent skills in negotiation, refusal, and problem-solving. 12 Making Proud Choices is divided in 8 one-hour sessions. 12 Making Proud Choices was originally tested with groups of 6 to 8 African Americans, using 1-hour sessions over a 2-day period. 13 Our implementations, however, differed widely from that model and we hypothesized that this would lead to youth with less positive outcomes in knowledge, self-efficacy, and intention-to-use contraception.
Description
Making Proud Choices was implemented in 11 locations in SC where teen pregnancy and STI rates were high. The evaluation team used the federally provided exit surveys (Developed by Mathematica Policy Research, OMB Control number 0970-0398) for outcome evaluation, and fidelity logs for process evaluation. Fidelity logs tracked the location, time, and youth attendance for each session, the delivery of the planned activities, and content changes. The sample consisted of all youth who had a record of participation from 2013 to 2016.
Measures of Program Implementation and Fidelity
Program settings were divided as in-school and out-of-school. Program length was measured both as (1) total amount of instructional time in hours, and (2) number of calendar days to complete all lessons. Class size was calculated both as continuous and dichotomous variables (small; ≤12 [mean class size] vs large; >12). Class gender composition was single gendered or mixed gendered. Curriculum adaptations were measured by facilitator self-report via a program activity checklist. We also examined participant demographics. 14
Outcome variables
Satisfaction was measured with 4 items: (a) “did you feel interested in program sessions/classes?,” (b) “did you feel the material was clear?,” (c) “did discussions/activities help you to learn program lessons?,” and (d) “did you have a chance to ask questions about topics/issues that came up?.” Response options were from “All of the time = 5” to “None of the time = 1” (α = .7). A mean score ≥3 was categorized as “satisfied” and <3 was categorized as “not satisfied.”
Youth were asked if they intended to do any of the following in the next 6 months (1) have sex, (2) use any birth control method, (3) use a condom, and (4) abstain from sex. All items used the response categories “Much more likely = 5” to “Much less likely = 1”.
Analysis
Multiple linear regression was performed to test the predictive value of fidelity parameters on youth program satisfaction and intention to avoid risky sexual behavior. P values <.05 were considered statistically significant.
Human Participants
This project was reviewed by the institutional review board at the University of South Carolina and at the South Carolina Department of Health and Environmental Control. The original anonymous data collection (done for evaluation purposes) and the secondary data analysis were given exempt status, so informed consent was not required. Consent to participate in MPC was determined by the individual implementing organizations.
Results
A total of 3073 youth participated in the MPC program implemented in SC during 2013 to 2016. The mean age of participants was 13 (standard deviation = ±1.5). Sixty-seven percent of youth were African American, 54% were female, and 62% received the program in schools.
Youths’ Satisfaction Toward the Program
Several program attributes predicted satisfaction scores (F = 32.487; P = .000); however, the total variance accounted for by the regression model was low (R2 = 0.094). In-school setting and longer program duration were associated with lower mean satisfaction (b = −.123; P = .014 and b = −.002; P = .008, respectively). Smaller classes had greater overall program satisfaction (b = .074; P = .018). Total program hours and number of curriculum changes, however, were not statistically significant predictors.
Program satisfaction also differed significantly by gender and race. Females had higher satisfaction (b = .223; P = .000). African American youth also tended to have higher satisfaction in the program (b = .268; P = .000). Youth who were in classes with high attendance (measured as students attending at least 75% of lessons) also had higher satisfaction (b = .004; P = .001).
Youths’ Intentions to Have Sex, Abstain, or Engage in Safe Sex Practices Postintervention
Youths’ intention to have sex in 6 months
There was a significant association of the fidelity variables and demographic variables (F = 28.136; P = .000); however, the total variance accounted for by the model was low (R2 = 0.073). Longer total duration of the program and greater total hours of lesson delivery predicted a slightly lower intention to engage in sex 6 months post program (b = −.003; P = .001 and b = −.026; P = .005), as did higher student completion rate and single-gendered classes (b = −.007; P = .008 and b = −.171; P = .011). Higher participant age was associated with greater intention to have sex (b = .105; P = .000). Females were less likely to report intention to have sex in next 6 months (b = −.531; P = .000). African American youth reported a greater intention to have sex (b = .264; P = .000).
Youths’ intention to abstain from having sex
There was a significant association of demographic and fidelity variables with intention toward sexual abstinence (F = 14369; P = .000), but the overall R2 was low (R2 = 0.033). Females were more likely to intend to abstain (b = .428; P = .000), whereas African Americans and older youth were less likely to intend to abstain (b = −.158; P = .021 and b = −.062; P = .003). Longer program implementation predicted less intention to abstain (b = −.002; P = .035), and a higher number of curriculum adaptations surprisingly led to a slightly higher intention to abstain (b = .012; P = .007).
Youths’ intention to make safer choices when having sex
The trend of significant but small effects continued with intentions to use contraception (F = 17.829; P = .000; R2 = 0.030) and condoms (F = 6.769; P = .000; R2 = 0.015). Single-gendered classes were associated with greater intention-to-use contraception (b = .387; P = .000) or condoms (b = .256; P = .000). Students in programs that had longer duration were less likely to intend to use birth control methods (b = −.004; P = .000) or condoms (b = −.002; P = .035). Females showed greater intention-to-use a birth control method (b = .277; P = .000) and use condoms or make their partners use condoms when having sex (b = .256; P = .039).
Discussion
This study examined how different types of implementation parameters led to changes in youths’ satisfaction and intention to have sex. This is important because these implementation parameters often vary in response to local circumstances and can be a departure from the “ideal” program implementation. Although many of our independent variables are statistically significant, the R2 were quite low.
Fidelity adaptations had a consistent but limited impact on our outcome variables. Contrary to what we expected, the number of modifications in the lessons only affected intention to abstain and did so positively. Programs that took more days to complete showed lower satisfaction but contradictory impacts on youth intention as they were less likely to have sex in next 6 months but also less likely to abstain or use birth control. Class size had an effect on youth satisfaction but not on youth intention to engage in risky sexual behaviors. Finally, youth in gender-specific classes did not show differences in satisfaction levels but were more likely to report positive behaviors with respect to sex.
Further, females and African American youth tended to be more satisfied with the program. African American and older youth reported greater likelihood to engage in sex but were no different in intention-to-use contraception. Males were both more likely to engage in sex and less likely to report intention to engage in contraception.
Limitations
Youth were not randomly selected or assigned during program implementation. Further, some data were incomplete due to program facilitators’ noncompliance with fidelity checklists. The independent conditions—length, duration, gender composition, completion—were those that occurred during the course of implementation and were not randomly assigned or influenced in any way. Despite these limitations, overall sample was quite large and diverse and allowed us to address several different implementation conditions.
Conclusion
Evidence-based interventions tailored according to local contexts and specific to youth audiences can impact participants’ satisfaction and refrain them from risky behaviors. Our results suggest that, for MPC, implementation factors do have an impact on satisfaction and intention (and not always negatively) but the impact is not large; MPC implementations that are altered to fit local settings can still be effective.
So What? (Implications for Health Promotion Practitioners and Researchers)
What is already known on this topic?
Evidence-based programs (EBPs) succeeded in lowering teen pregnancy rate in different countries. Success of a program in generating positive outcomes is affected by how the program is implemented. Implementation fidelity is a determining factor in EBPs for successful translation of interventions into practice.
What does this article add?
This study evaluated implementation of an evidence-based safer sex education program in real-world settings in both schools and community-based organizations. Program implementation factors have an impact on satisfaction to the program despite diversity in settings. Program implementation factors have an impact on youth’s intention to abstain from sex or practice safer sex when having sex. Overall impacts are very small.
What are the implications for health promotion practice or research?
Findings have implications for safer sex education program implementation for adolescents in different settings, programs can be tailored considering the setting’s needs and maturity of the participants. Results suggest that implementations altered to fit local settings and maturity of participants still remain effective.
Footnotes
Authors’ Note
This research does refer to a program that is available commercially
Acknowledgments
The authors would like to acknowledge the South Carolina Department of Health and Environmental Control, who is the recipient of these grant funds, and Fact Forward of Columbia SC, who provided technical assistance and curriculum training to subgrantees. The authors would also like to acknowledge the PREP subgrantees and program facilitators, who delivered these programs to youth.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Family Youth Services Bureau via the Personal Responsibility Education Program (PREP) [grant number 1701SCPREP].
