How Do Contextual Factors Influence Implementation and Receipt of Positive Youth Development Programs Addressing Substance Use and Violence? A Qualitative Meta-Synthesis of Process Evaluations

Importance of cultural sensitivity, collaboration, and integration with ethnic minority communities

The importance of cultural sensitivity and collaboration when implementing programs in ethnic minority communities was a subtheme across 3 studies, ^30,33,36 all judged to be of low reliability but varying degrees of usefulness. These reported that formal and informal community engagement was a key factor in ensuring programs were culturally sensitive, accessible, and appealing to young people and their parents. This was particularly important when programs were targeting or situated within marginalized ethnic minority populations. For example, in a process evaluation judged to be of high relevance but low reliability, Armstrong and Armstrong ³⁰ reported from interviews with site coordinators delivering after-school programs in a southwestern US state that a program’s cultural relevance within an “ethnically diverse community” meant that it was “important to have a strong cultural awareness” and was essential to the program’s success. This included both “outreach projects with parents” and with schools in the local area and “liaison…with a trusted member of the community who could communicate with the parents, often times in Spanish.” ^30(p101)

Lee and colleagues’ ³⁶ study, though judged to be of both low reliability and relevance, corroborated this finding. In their evaluation of a PYD program targeting the aboriginal communities of the Northern Territory of Australia, they highlighted the importance of seeking and incorporating the views of the wider community, not just those of young people or parents. This, it was suggested, could provide support through the generation of ideas and allay fears among minority ethnic groups that the program was “a non-Indigenous solution so there is little ownership of it by Indigenous people.” ^36(p78) Community consultation led to activities that were youth oriented but culturally relevant, such as “bush hunting excursions and using computers to record traditional music.”^(p79) This study found that as the program progressed, staff became more active in encouraging community members to get involved, including through “engaging in regular formal meetings and informal discussion” with members of the community. ^36(p78)

Lee and colleagues ³⁶ also highlighted the importance of increasing both the cultural relevance and participation of the local community, addressing potential language barriers by “translating key proceedings,” and communicating with indigenous members “in their language.”^(p78) Armstrong and Armstrong ³⁰ also found instances where young people were only allowed to access and participate in the program, because “the parents were able to communicate with, and trusted the liaison”^(p101) officer connected with the program.

In some cases, programs actively recruited community members as staff. Lee and colleagues ³⁶ described this as “pivotal to the initiative’s success.”^(p79) Such actions could also be seen as providing the additional benefit of providing local role models. For example, after identifying a “lack of Chicano Latino adult role models”^(p51) that could “encourage, empower and develop leadership skills and qualities”^(p46) of local Chicano youth in Minnesota, program providers in the low-quality study by Bloomberg and colleagues ³³ trained local community members, as “facilitators” who could “work closely with youth in the initial 2-day training”^(p51) with the aim of establishing and maintaining a bond with them.

Challenges with community engagement and establishing trust

However, 2 studies ^18,37 of differing quality noted challenges in relying on volunteer community engagement and establishing trust of parents. These were studies of interventions that did not specifically target diverse ethnic populations but attempted to involve parents and local community members. For example, as reported by a study of high reliability and usefulness by Schwartz and colleagues, ³⁷ successful implementation of an intervention component relying on volunteer mentors was challenging when mentors were not always reliable in maintaining contact, leaving participants feeling “disappointed”; as 1 young person said, she hoped her mentor “would be there more than she was…, and she wasn’t”.

Building trusting and openly communicative relationships with parents could also be challenging. Maxfield and colleagues ¹⁸ studied the Quantum Opportunity Program, implemented in the United States, in a process evaluation judged as providing highly useful findings but with low methodological reliability. They found that trust and open communication were seen as important means of maintaining contact with young people and encouraging uptake of intervention activities. The case managers in this program reported parents who appeared “anxious to limit the exposure of family problems,” who seemed to experience case managers as “intrusive,” or may have “felt threatened” by the mentoring relationships that case managers established with their children, were subsequently the most “most difficult to reach” compared to parents who actively supported case managers and “reinforced the value of attending program activities.” ^{18 all quotes p.58}

Collaborating with and utilizing local community resources

Another subtheme concerned with collaborations with others in the local community was the importance of collaboration with other community agencies to enable program implementation. This was apparent in 3 studies ^18,30,32 of variable reliability and usefulness. Armstrong and Armstrong ³⁰ found that it was “extremely important for the site to utilize community resources from a programmatic standpoint”^(p101) in order to expand the range of activities offered, a critical element of PYD. For example, local libraries proved to be an “unplanned benefit” that could help deliver a reading program. Program providers cited being able to host “occasional large-scale events” by “taking advantage” of a nearby boys and girls club. ³⁰ Local funding bodies were considered another important local resource to support positive youth activities. This was the case in the study of medium reliability and relevance by Berg and colleagues ³² where the program received a grant that “enabled (young participants) to receive training in photography and show their work at a photography exhibit.”

The importance of collaboration was also apparent in Maxfield and colleagues’ ¹⁸ study of the Quantum Opportunity Program in which providers forged “partnerships with agencies that specialized (in a range of life skills training topics) such as substance abuse prevention, conflict resolution training, date rape, and sexual abuse.” The importance of being able to make use of other local services to maximize breadth of opportunities was regarded as particularly important to fill gaps in program providers’ expertise, such as when drawing on “student volunteers from the local university” to offer tutoring to support sites where case managers felt they lacked the skills to provide such services. ^18(p64)

Collaboration with schools

A final subtheme regarding community collaboration highlighted that collaboration with schools, while critical to implementation, could be time consuming and challenging. Three studies, ^18,30,38 2 based in the United States and 1 in the United Kingdom, examined the importance of liaising with schools to support the successful implementation of programs. All 3 studies were judged to be of high usefulness but variable in terms of methodological reliability. Site coordinators in Armstrong and Armstrong’s ³⁰ study of after-school programs in the southwestern US indicated that communicating with other community stakeholders to support the development of youth “such as schools, had an important impact on program implementation,”^(p101) particularly because they had a number of after-school programs located off schools’ sites. One way of dealing with barriers to communication arising from this was to designate a school liaison, who could work across program sites but who was an employee of a single school. The schools then also acted as a channel to disseminate information about program events to young people and their families in order to reach a wider audience and increase program reach.

Wiggins and colleagues’ ³⁸ study of after-school youth development targeting at-risk young people across England, which was judged to be of high reliability and usefulness, also found that “working with schools was crucial” for recruiting young people to programs, though negotiating “access and referral routes” was time consuming. In a context of providers aiming to meet challenging recruitment targets, some sites reformatted their program so that young people attended as an alternative rather than a supplement to their normal schooling, a major distortion of the intended intervention model. Maxfield and colleagues ¹⁸ also reported that collaborations with schools in the Quantum Opportunity Program were subject to logistical challenges. When case managers who transported young people to the school where tutoring services were provided found it “proved too burdensome,” participants’ uptake of tutoring plummeted.

Calm and authoritative program providers

One subtheme was the importance of program providers attending to young people in a calm and nurturing yet authoritative way, including in response to any challenging behavior exhibited by participants. Three studies of varying reliability and usefulness described provider attitudes and responses to young people in this context. The “Stand Up Help Out” program was evaluated by Bulanda and McCrea ³⁴ in a process evaluation judged as highly reliable and of medium usefulness. They reported that successful implementation was associated with staff signaling their continued commitment to providing “unconditional positive regard” when faced with challenging behavior from young participants. It was reported that this response style was acceptable to the young people, who did not feel they were treated “negatively.” ³⁴ Similarly, Maxfield and colleagues’ ¹⁸ evaluation found evidence supporting the need for case managers to engage with young people as individuals rather than collectively as a group. They found that “the most successful mentors used a balance between nurturing and discipline” when interacting with young people. ^18(p59)

In contrast, Cross and colleagues ³⁵ reported in what was judged a highly reliable and relevant study that staff struggled to respond to young people’s frequent challenging behavior with “very little redirection from staff members” and a disciplinary approach that “appeared capricious and confusing to youth.” ³⁵ In another site, the same evaluators, found staff to be “irritated and apathetic,” appearing to engage more with each other than interacting and addressing young people’s challenging behavior. ³⁵

Positive peer relations

A further subtheme was the importance of positive and supportive peer relations underpinned by staff and by program structure, as examined by 3 studies of differing reliability and usefulness. For example, a high-quality study by Bulanda and McCrea ³⁴ described a US after-school program where social differences, such as membership of different “street alliances” that could be a cause of conflict outside the program, did not necessarily prevent mutual collaboration and support within the program as long as participants were able to “prioritize their connectedness over the potential discord created by differences” and “recognize relationship problems and focus on relationship strengths.” However, Cross and colleagues ³⁵ argued that tensions among participants or between participants and staff could only be overcome in sites that were well managed. A lack of organization and high turnover of staff at 1 site within their study was a key factor in young people not seeming “to enjoy each other’s company” and that the positive outcomes observed in another site might be attributable to “the friendships among students, which were in part facilitated by stable site management.” ^35(p377)

Bloom et al ⁴⁰ described how the National Guard Youth Challenge Program separated participants who belonged to different gangs into different “squads” and removed gang symbols such as tattoos that could act as “physical reminders of past affiliations.” Although problems associated with gang membership were not always easily overcome, staff members did report that external problems were less likely to intrude during the residential phase of this program, where they “have them 24/7” and can instill values that young people can them take “home with them.” ^{40(both quotes p37)}

Staffing continuity essential to successful implementation

Within this, a key subtheme was the importance of staffing continuity to intervention delivery. As Armstrong and Armstrong ³⁰ noted, after-school program site coordinators felt that effective implementation and sustainability relied on minimizing staff turnover. This was a challenge for some programs. However, in the after-school program Cross and colleagues ³⁵ evaluated, “six of the original fourteen staff members quit or were fired before the end of the year.” Similarly, Lee and colleagues ³⁶ reported that turnover of staff “impacted significantly on program continuity and workloads.” Maxfield and colleagues ¹⁸ observed that they were “fortunate [that] turnover [in certain sites] was relatively low.” However, staff turnover led to a failure in sustaining mentoring relationships when unfilled staff positions resulted in participants not having a “primary mentor for as long as 2 or 3 months,” and when participants had multiple case managers.

Difficulty offering full-time posts in the youth work field

Across a number of studies, the lack of full-time positions increased the challenge of securing and retaining qualified staff. To overcome this, Armstrong and Armstrong ³⁰ report how 1 program aimed to recruit staff who were not looking for full-time work, such as college students interested in gaining experience of youth work. Difficulties with retaining trained employees could also mean that replacement staff were not well trained. Cross and colleagues ³⁵ report that youth workers who had been retained since program initiation “received more than 40 h[ours] of training on average” compared to those who had replaced them, who “received less than 6 hours” and that sites with high employee turnover were less likely to have staff who were highly trained because it was not possible to offer them the original training.

Creative attempts to compensate for lack of trained staff included drawing on existing skills that happened to be held by staff members and incorporating these opportunistically into program activities. For example, Armstrong and Armstrong ³⁰ observed that at 1 site an employee “with extensive orienteering skills” was encouraged “to organize camping trips and day hikes for youth,” and that at an another site, a staff-member “who enjoyed jazz dancing started a dance program.”

However, 2 studies ^18,37 reported that it was difficult to overcome limitations in skills due to a lack of training, leading to an inability to provide the range of activities normally expected of a PYD program. For example, based on a paper reporting on the study by Schwartz et al, ⁷ Bloom and colleagues ^{40(a paper reporting on the study by Schwartz et al7)} found that provision of individual tutoring was impossible to implement because of lack of tutor capacity and had to be “abandoned midway through the year.” The authors felt that despite providing an alternative academic activity, the lack of one-to-one tutoring may have “contributed to withdrawal of youths whose parents viewed tutoring as the main draw of the program.” ^40(p54) In addition, Maxfield and colleagues ¹⁸ reported that programs found it difficult to secure staff with expertise across the range of PYD domains. For example, programs expanded to include an educational component challenged staff “hired on the basis they could be case managers not tutors or teachers” and who “required extensive training and technical assistance.” Other sites that did not provide “extensive in-service training to improve case managers’ tutoring skills” relied on volunteer tutors instead, though these volunteers tended only to work for the program for “1 or 2 semesters.” ¹⁸

Young people determining their own engagement in activities

One subtheme within this relates simply to the extent to which young people were empowered to choose in which PYD activities to participate. This was described in 3 studies of variable reliability and usefulness. Berg and colleagues ³² described youth empowerment as a key component in their Youth Action Research for Prevention program and suggested that staff needed to be trained in “facilitation techniques” to halt the tendency for staff to determine decisions about how community engagement is undertaken. Young people’s decision-making processes were considered more important than their final choice of activity in Baker and colleagues’ ³¹ study of the South Baltimore Youth Centre. The evaluation, judged as being of low reliability and relevance, reported that when activities were “imposed (in a) top down (manner they) failed and were abandoned” and thus providers aimed to give young people authority in developing and executing activities. Schwartz and colleagues’ ¹⁹ study of youth-initiated mentoring found that when young people were able to choose their mentors, the mentoring relationship was more likely to be successful.

Limitations to choice provided

In contrast, 2 studies judged as highly reliable reported that young people in some interventions had very limited empowerment to shape and determine their involvement. For example, empowerment in the “All Stars” curriculum ³⁵ was highly restricted. In this study, also judged to be of high usefulness, young people’s choices were restricted to a list predetermined by the site director and program assistant at the start of each day. Empowerment was also restricted in the program evaluated by Schwartz and colleagues. ¹⁹

Tensions arising from choice

Another subtheme was the tensions that could arise when empowering young people to choose which activities in which to engage. Four studies, judged to be of high relevance with variable reliability, provided data on young people’s choice of activities, finding that some program components were often rejected by young people on the basis that they were unappealing. Sometimes these were activities with a learning component, such as “computer-assisted instruction” and “community service,” which were not received with “enthusiasm.” ^{18(all quotes p62)} This was also the case for “computerized job training” which was “ignored” ^31(p73) and academic assistance. ³⁵ Wiggins and colleagues ³⁸ argue that an academic style could alienate young people, particularly those whose lives are “chaotic and hard” and suggest that young people need to be able to get involved in activities at a level that is “most appropriate for them at any given time.” This might suggest the importance of a diversity of provision, not only to enable choice but also because different young people will have different preferences and developmental needs.

However, whereas some process evaluations, as well as much of the theoretical literature, suggested that young people’s empowerment to choose activities is central to PYD, process evaluation of the Quantum Opportunity Program ¹⁸ suggested that facilitating choice may in some cases deter engagement in the broad range of activities, which is also commonly regarded as a central feature of PYD. This study, judged to be of low quality but high relevance, reported that some sites offered more recreational activities (such as outings to the cinema, swimming, etc) because they attracted “more enrollees than did other activities.” However, as young people got older they resisted staff’s promotion of “activities with learning content” and continued to favor recreational activities that providers had originally used “to attract youth to the program.” Participants reported “that they missed doing “fun things” and that museum and other cultural activities were boring.” Similarly, when there was a scheduling clash between attending summer school and taking up summer employment, case managers were more likely to recommend summer school, but ultimately, they were “not able to prevent an enrollee from choosing (paid work).” This contrasted with program sites that provided a balanced combination but offered participants little choice, which appeared to have “less difficulty in maintaining interest” of young people. A lack of choice could be received positively by participants when it offered them something new. Participants in the National Youth Guard mentoring program “welcomed the small class size, tailored instruction, and self-paced approach” ^40(p48) of the high school completion program. The authors noted that a key element of the success of their educational component was that it was noticeably different from what young people were used to experiencing in school, as it combined both structure and individual support.

Maxfield and colleagues ¹⁸ reported that some sites provided financial incentives to increase engagement in specific educational activities, such as computer-assisted instruction and assessment tests. However, the 2 sites that took these approaches found that it was “effective for only short periods of time and only for students already inclined to spend time on the computer” and did not prove effective in motivating already resistant young people. The use of incentives was also reported as problematic by Cross and colleagues. ³⁵ To increase engagement in program activities, young people were randomly assigned to groups that accrued points for attendance. However, program staff thought the system unfair and decided to place high-attending youth together “to ensure the attending students would receive the maximum points” thus undermining the intended system. This “probably did not encourage attendance among the lower attending youth because they were placed in groups with very low probabilities for receiving points.” ^{15(all quotes pp52-53 a paper reporting on the study by Cross et al35)} In both of these programs using incentives, there was a tension in providers’ attitudes to empowerment. Although program providers wanted to enable choice, they also sometimes wanted to constrain choice to ensure young people engaged in the program overall or in specific activities staff regarded as important.