Fidelity After SECOND LIFE Facilitator Training in a Sexual Risk Behavior Intervention

Abstract

Background. Intensive facilitator training is a critical component to enable the use of Evidence-Based Interventions (EBIs) by communities. Such training ensures that program’s essential elements are implemented with fidelity. Many EBI training programs require resource- and time-intensive face-to-face training, eliminating participation from some community-based organizations due to significant costs and time away from work. The ¡CUÍDATE! (TAKE CARE OF YOURSELF!) program aimed at improving the sexual health of Latino youth is one such EBI limited in widespread dissemination by its current face to face format. The purpose of this study was to examine the efficacy of training by assessing whether facilitators who attended the virtual ¡CUÍDATE! facilitator training in SECOND LIFE™ implemented the ¡CUÍDATE! curriculum with fidelity (e.g. adherence to the curriculum and competent delivery) to adolescent youth.

Methods. Using a descriptive design, community-based facilitators (n=19) attended virtual SECOND LIFE™ ¡CUÍDATE! training. Post-training facilitators implemented two identical ¡CUÍDATE! sessions with two groups of adolescents and were instructed to audio record the sessions. The research team reviewed a random, stratified subset of the audio files (n=57) using the standardized curriculum as the standard to determine implementation fidelity adherence and fidelity competency.

Results. Fidelity adherence for the ¡CUÍDATE! training for both sessions was high (91.2%) (e.g. the activities were presented as intended). Fidelity competency of facilitators was also high, with the overall quality of the program sessions being 3.95 (SD 1.18, scale 1 to 5). Both fidelity measures (i.e., adherence and competency) improved from implementation of the first to the second session with 100% adherence to the curriculum reported for all facilitators in the second session. Overall quality of the program sessions also improved from the first (x=3.67, SD 1.37) to the second (x=4.43, SD 0.53) session.

Discussion. The efficacy of facilitator training using a virtual platform was shown to result in high levels of implementation fidelity, and thus may be a viable platform for widespread national training. Virtual training provides an avenue for reaching community-based agencies that work with adolescent youth and have limited resources for training expenses. Future studies will explore the use of virtual reality for other types of inter-professional education.

Keywords

capacity building community ¡CUÍDATE!education evidence-based interventions facilitator training fidelity health promotion risk reduction SECOND LIFE sexual education virtual environments virtual training

¡CUÍDATE! (Take Care of Yourself!), an Evidence-Based Intervention (EBI), is a sexual risk behavior prevention intervention for Latino Youth (Villarruel, Gal, Eakin, Wilkes, & Herbst, 2010; Villarruel, Jemmott, & Jemmott, 2012). The ¡CUÍDATE! curriculum consists of six-one hour modules focused on improving skills and self-efficacy in communication and negotiation of safe sex practices. Studies to date have provided evidence for the efficacy of this intervention in promoting healthy sex behaviors, including older age and condom use at first sex, fewer reports of sexual intercourse, and fewer numbers of sexual partners (Villarruel, Jemmott, & Jemmott, 2006; Villarruel, Zhou, Gallegos, & Ronis, 2010). Widespread dissemination of successful EBIs—such as ¡CUÍDATE!—is needed, but current face to face facilitator training of the interventions limit intervention scalability. ¡CUÍDATE! facilitator training (similar to other EBI training programs) requires resource-intensive, multiple day face-to-face training, a significant barrier for many community–based organizations due to significant costs and time away from work.

¡CUÍDATE! facilitator training was adapted to the multi-user virtual environment SECOND LIFE™ (Linden Labs) to improve accessibility of the curriculum and facilitator training (Valladares, Aebersold, Tschannen, & Villarruel, 2014; Villarruel, Aebersold, Valladares, Yeagley, & Tschannen, 2014; Villarruel et al., 2012). SECOND LIFE™ (SL), a web-based three-dimensional multi-user environment (MUVE), supports a high degree of interactivity, and thus was chosen as the platform for ¡CUÍDATE! facilitator training. Using the face to face ¡CUÍDATE! Training of Facilitators Manual, including objectives and approach as the framework for the adaption process (Villarruel et al., 2012), activities in the training were adapted to self-learning modules (e.g., activities required no interaction such as background on the ¡CUÍDATE! curriculum) or modified for implementation in SL (e.g., activities that required facilitator interaction) (Valladares, Aebersold, Tschannen, & Villarruel, 2014). Upon completion of the SL training, facilitators completed surveys and participated in formal debriefings to evaluate the feasibility and acceptability of the ¡CUÍDATE! SL facilitator training. The focus of this study was to examine the efficacy of training by assessing whether facilitators who attended the virtual ¡CUÍDATE! facilitator training in SL delivered the ¡CUÍDATE! curriculum with fidelity to adolescent youth (Figure 1). Specifically, this study explored implementation fidelity, referring to the degree a program is implemented as intended by the developer (Dusenbury, Brannigan, Falco, & Hansen, 2003). Audio taped ¡CUÍDATE! sessions implemented with adolescents were evaluated to determine the ability of facilitators trained in SL to implement the curriculum as intended (fidelity adherence) in a competent manner (fidelity competence).

Figure 1.

¡CUÍDATE! training facility within SL.

Literature Review

Second Life^TM

Second Life has been used in health training for decades, for activities such as identifying heart sounds, assessing patients, and engaging in reflective practice (Boulos, Hetherington, & Wheeler, 2007; Kemp, 2007). With over 36 million users, SL supports social networking and interactivity while providing flexibility in the environment for educators to design learning spaces in consideration of pedagogical frameworks. SL supports exploration, role play, simulations, interactions and experimentation via avatars (e.g., online, graphical representation of self in a virtual world) (Peterson, 2005). Additionally, virtual worlds provide greater access to learners, connecting participants who are geographically distant.

According to a recent systematic review of the use of virtual worlds in healthcare, research has been conducted in six primary areas: academic education, professional education, treatment, evaluation, lifestyle, and modeling. The majority of research has been completed in the area of professional education (Ghanbarzadeh, Ghapanchi, Blumenstein, & Talaei-Khoei, 2014). These studies have focused on improving learner knowledge, such as with on-line journal clubs and various educational information sessions related to cancer care and sexual education. Other professional education studies have been focused on skill development, such as with effectively communicating a poor prognosis to patients and families (Ghanbarzadeh et al., 2014).

A limited number of studies have focused on training healthcare professionals in the virtual environment. Heinrichs, Youngblood, Harter, and Dev (2008) undertook three virtual reality studies geared towards team training in emergency medicine for three distinct areas: response to trauma cases, disaster preparedness, and triage and treatment for mass casualties (Heinrichs, Youngblood, Harter, & Dev, 2008). All of the trainees found the virtual environment to be adequately realistic, providing an opportunity for repeated practice and engagement with team members geographically distant (Heinrichs, Youngblood, Harter, & Dev, 2008). Mitchell and colleagues (2011) used the virtual platform to deliver motivational interviewing training to physicians. The training environment of SL was rated highly by the participants, with 77% of physicians reporting SL to be an effective educational medium. Overall physician proficiency in motivational interviewing, as measured by coded audiotaped mock interviews with a standardized patient, improved pre- to post-training significantly (15% to 92% at or above proficiency level pre- to post-training, respectively) (Mitchell et al., 2011). Work has also been completed in SL related to sexual prevention, as noted by studies conducted by Boulos and Toth-Cohen (2007), who designed a sexual health project in SL in an effort to provide education about sexually transmitted diseases, unintended pregnancies, and improved sexual relations. SL was reported as being a viable venue for public health education, as 81% of participants noted the program to be fairly or very useful. Furthermore, participants reported learning something new (39%), possibly changing behavior (52%), with plans to return to the site in the future (54%) (Boulos & Toth-Cohen, 2009).

Gao, Noh, and Koehler (2009) compared patterns of communication during role play activities in the virtual and face to face training. Students communicated more often in SL, noting a significant increase in the number of ‘turns’ students took speaking during the SL role play (10.3 turns in SL vs. 4.5 turns during face to face, p<.001). Additionally, students reported having greater interest in SL, seeing the virtual environment to be less formal than face to face.

In summary, SL is the most mature and popular MUVE and provides a safe, simulated environment for health training (Warburton, 2009). Although authors reported benefits for using SL, including greater access to health information and real-world health resources, empirical research determining its overall efficacy is limited (Beard, Wilson, Morra, & Keelan, 2009). The virtual environment may be the optimal environment for health-related training to improve health.

Measures of Fidelity

A critical component of successful educational programs, including the use of EBIs in the community, is the intensive facilitator training needed to ensure that the program’s essential elements are implemented with fidelity. This differs from definitions of fidelity considered in simulation and gaming environments, where fidelity is defined as the accuracy of the virtual situation as compared to the real experience (Aldrich, 2009). Nehring and Lashley (2009), while reviewing nursing simulation literature for the past 40 years, describe fidelity as the degree that an object mimics reality (e.g., simulator design being low versus high fidelity). Enfield, Myers, Lara, and Frick (2012) evaluated model fidelity, or the extent to which the computational model is consistent with the underlying conceptual model. More in line with EBI studies is the definition of fidelity provided by Kyaw Tun, Alinier, Tang, and Kneebone (2015), “degree of accuracy to which simulation, whether it is physical, mental, or both, represents a given frame of reality in terms of cues and stimuli, and permissible interactions (pg 164).”

For the purpose of this study, implementation fidelity was defined as the degree to which a program is implemented as intended by the developers, which includes reality-based stimuli, cues, and appropriate interactions (Breitenstein, Gross, et al, 2010; Dusenbury et al., 2003; Mowbray, Holter, Teague, & Bybee, 2003). This definition of fidelity is similar to what has been described as simulation validation (e.g., the process of assessing that the conclusions reached from a simulation are similar to those reached in the real world system being modeled), although the focus is on the implemented curriculum rather than the simulation or game per se (Feinstein & Cannon, 2002).

Various methodologies have been used to determine implementation fidelity, including self-reports via surveys and interviews and observations either face to face or through audio/video recordings. Chen (2014) evaluated the fidelity of an emergency strategic flood response exercise—ATLANTIS. Fidelity was evaluated through questionnaires completed by participants. Participants (n=11) believed the in-world space was similar to the traditional exercise (54%), with 54% acknowledging the synergy in the responses to be similar in both the virtual in traditional environment (Chen, 2014). Durkin, Pollack, Star, and Rittle-Johnson (2012) compared the relationship between a survey of self-reported practices versus rater-coded videos in measuring fidelity of implementation and the effect on student outcomes. The two measures of fidelity (e.g., self-reported survey and coded videos) were more highly correlated among teachers in the intervention group than the control group. When comparing audio versus video recordings, Breitenstein, Fogg, et al. (2010) found audio recordings to be less intrusive than video recordings, and thus the methodology of choice. In line with these findings, this study evaluated fidelity through the review of audio recorded sessions of facilitators implementing the ¡CUÍDATE! curriculum with adolescents.

Methods

This study used a descriptive design with a convenience sample of community-based leaders interested in implementing the ¡CUÍDATE! curriculum with the youth they served in their community agency (Figure 2). The ¡CUÍDATE! curriculum is divided into six modules focused on increasing skills and self-efficacy in communication and negotiation of safe sex practices. Each module consists of multiple types of engaging activities for the adolescents—ages 13 to 18 years—classified as attitude-, knowledge-, or skill-based (Villarruel et al., 2012). ¡CUÍDATE! Training of Facilitators consisted of participants engaged in discussion, interactive activities, modeling of the curriculum through teachbacks, and peer/trainer feedback within SL. Activities not included in the virtual training but in the face to face training were those with similar format and structure as other activities, thus maximizing time and interactivity in SL (e.g. ice breaker activities that required discussion similar to other activities) (Villarruel et al., 2014). The study was reviewed by an Institutional Review Board and deemed exempt from future regulation and review.

Figure 2.

¡CUÍDATE! conference room.

Sample

Facilitators were recruited from community-based agencies throughout the United States using a variety of methodologies, including use of social media, distribution of study information via agencies focused on the sexual health of adolescent youth, and through an online portal that provided information about the training. Those interested in the study contacted the research team through the online portal. A thorough overview of the recruitment, enrollment, and SL orientation process is provided in a previous publication (Valladares et al., 2014). In total, five facilitator trainings were conducted in SL with a total of 35 facilitators (3-10 facilitators per training session).

¡CUÍDATE! Training

Each ¡CUÍDATE! facilitator training session was conducted by two Master Trainers who are experts in ¡CUÍDATE! and trained in the debriefing methods used for the training. Both Master Trainers have conducted numerous EBI trainings and were part of the ¡CUÍDATE! program from its inception (one of which was the original developer). Additional research team members—including an expert in virtual training—were present during each of the sessions to ensure smooth and efficient training. Each training consisted of three sessions conducted over three days: (1) a 2.5-hour overview of ¡CUÍDATE! key SL features, and the ‘teachback’ learning methodology (Kripalanti, Bengtzen, Henderson, & Jacobson, 2008; White, Garbez, Carroll, Brinker, & Howi-Esquivel, 2013) (2) 4-hour session with teachbacks from Modules 1-4; and (2) 3-hour session with teachbacks from Modules 5-6 in addition to a discussion on successful implementation (Figures 3 and 4) (Villarruel et al., 2014). The teachbacks—required a facilitator to model various activities within the curriculum while the remaining facilitators role-played adolescent youths—were an essential component to the training. The teachbacks mimicked the environment where the facilitators would eventually deliver the trainings.

Figure 3.

¡CUÍDATE! training room.

Figure 4.

¡CUÍDATE! facilitator orientation.

After each teachback, the Master Trainers would facilitate a debriefing of the activity using a reflective style debriefing method. Der Sahakian and colleagues (2015) identified six recommendations to ensure good debriefing. Table 1 provides an overview of the specific recommendations and the actions taken within this project in accordance with the recommendations. The debriefing style was most representative of the debriefing with good judgment model (Rudolf, Simon, Dufresene, & Raemer, 2006). Through the teachbacks and subsequent debriefing sessions, the research team was confident that the facilitators were knowledgeable in all components of the curriculum, a necessary component prior to evaluating implementation fidelity.

Table 1.

Good Debriefing Characteristics Aligned With the Study Protocol.

Good Debriefing Characteristic*	Study Activities
Reflect on their own performances as an instructor	1. Master trainers received initial training related to effective debriefing. 2. Following each training day, facilitators were asked to share their reflections on the day’s presentations (Master Trainer and facilitator-based), learning activities, and conduciveness of the environment.
Establish simulation ground rules	Facilitators participated in a pre-briefing session and orientation session where an overview of the project, needed SL skills, and expectations for the ¡CUÍDATE! training were reviewed.
Manage unexpected events and intended learning objectives by using a confederate during scenarios	Each training session in SL consisted of several research team members who performed different roles in the training session, including Master Trainer (provided curricular training), Facilitator (provided oversight for the training), technology support (the ‘go to’ person for any technological difficulties), and the recorder (took notes throughout the session).
Respect the steps of the debriefing process and good practice regarding learning psychology	Stages of the debriefing session, including exploration of (1) Reactions, (2) Understanding, and (3) Summarize of lessons learned was used in the debriefing following teachbacks. The Master Trainer provided an opportunity for the facilitator to reflect how they think the presentation went, identify what went well, and how they might do things differently in future presentations.
Maintain the balance between emotion and teaching by decontextualizing the experience from the participants during debriefing	Facilitators remained responsible for ensuring the environment remained conducive for learning.
Managing the input from the peers during the debriefing process	Debriefing with good judgment was used as the style for all debriefing sessions.

Recommendations for good debriefing came from Der Sahakian and colleagues (2015).

Fidelity Evaluation Procedure

Facilitators were evaluated for their ability to implement ¡CUÍDATE! with fidelity (e.g., with competency and in adherence to the curriculum) after training in SL. Implementation fidelity was evaluated by fidelity adherence, referring to the extent the trainer/facilitator delivers the intervention as specified in the Facilitator’s Curriculum Manual, and fidelity competence, defined as the level of skill of the person delivering the ¡CUÍDATE! curriculum.

To evaluate the fidelity of the ¡CUÍDATE! virtual training, facilitators were asked to record two implemented sessions with the adolescents (e.g. Modules 1-6) using a digital audio recording device provided by the research team post-virtual facilitator training. Only the first two sessions were audio recorded as that was set forth in the initial agreement with the facilitators upon participation in the larger study. Digital audio recording was chosen as a data collection method to ensure fidelity in implementation for several reasons. Today’s technology offers a variety of low cost recorders that are available at many retailers, making access to the recorders easy. Compared with the cost of travel and lodging of observers, the audio recording allows for a much more cost effective method for data collection. The recorders also offered a layer of anonymity to the facilitators who received the ¡CUÍDATE! Training. In addition, previous work has noted this methodology to be the methodology of choice (Breitenstein, Fogg, et al., 2010). Finally, the audio data were recorded in an mp3 format, allowing for multiple methods of secure transmission to the research team. Upon completion of each session, facilitators were asked to send the recorded session files through a secure file sharing service. Finally—at the end of the implementation—the facilitators mailed the recorders back to the research team with the files on them as a backup.

For the purpose of the analysis, the research team reviewed a random subset of the activities within each session—stratified by type of learning activity (e.g., knowledge, skill, attitude) and level of trainer exposure (e.g., included in the SL training or not included in the SL training)—to evaluate fidelity (Table 2). This methodology for fidelity analysis was used previously and was found to provide a representative sample of the practitioner and participant functioning (Barber et al., 2006). Specifically, 14 activities, consisting of all three types of activities (knowledge, 5; skill; 5; and attitude, 4), were chosen for review (Table 2). A total of three activities were reviewed for each facilitator. Actual activities reviewed by the research team were determined randomly and were stratified by activity type.

Table 2.

Overview of the ¡CUÍDATE! Facilitator Training Activities & Second Life Exposure.

Breakdown of the ¡CUÍDATE! Activities and Second Life Training Exposure
¡CUÍDATE! Activity	Type of Activity
Module 1: Introduction and Overview of the Program
Activity A: Conocimiento (Getting to Know You)
Activity B Talking Circle
Activity C: Creating Group Rules
Activity D: ¡CUÍDATE! Logo
Activity E: Discussing Pregnancy 15^t	Knowledge*
Activity F: Discussing STDs and HIV^
Module 2: Building Knowledge About Pregnancy, STDs and HIV
Activity A: What it Means to Be Latino/Latina Overview^t
Activity B: Cultural Values	Knowledge*
Activity C: What Latinos Think About HIV/AIDS: and Safer Sex^
Activity D: View DVD, ¡CUÍDATE!
Activity E: Myths and Facts Game^	Knowledge*
Module 3 Understanding Vulnerability to Pregnancy, STDs and HIV Infection
Activity A: Acknowledging the Risk of Unplanned Pregnancy, STDs and Safer Sex^
Activity B: View DVD, Demasiado Joven^t	Knowledge*
Activity C: A Romance^	Skill*
Activity D: Latino Cultural Attitudes and Safer Sex^	Attitude*
Activity E: La Lotería^ (Lottery/Bingo)
Module 4 Attitudes and Beliefs AboutPregnancy, STDs, HIV and Safer Sex
Activity A: ¿Quién Es Más Macho? ¿Quién Es Más Mujer?^ (Who is Macho?)	Attitude*
Activity B: Music and Discussion^t	Attitude*
Activity C: Adolescent Vulnerability to Unplanned Pregnancy, STDs and HIV^t
Activity D: La Zona Peligrosa (The Danger Zone)^	Knowledge*
Module 5 Building Condom-Use Skills
Activity A: Discussing Condoms^t
Activity B: Condom-Use Skills^	Skill*
Activity C: Overcoming Barriers to Condom Use^	Attitude*
Activity D: What Gets in the Way of Caring Behavior^
Activity E: Condom Line-up^	Skill*
Module 6 Building Negotiation and Refusal Skills
Activity A: No Hay Razón (There Is No Reason)^
Activity B: Introduction to the S.T.O.P. Technique Sofia and Miguel DVD and Discussion^t	Skill*
Activity C: S.T.O.P. Technique and Role-plays^	Skill*
Activity D: Safer Sex Jeopardy Game^
Activity E: Talking Circle

Activities reviewed for fidelity

Activities trained in all of the facilitator cohorts

Activities trained in some of the facilitator cohorts

The reviewers—who consisted of a PhD prepared researcher, a graduate research assistant, and 3 undergraduate research assistants—participated in a training session prior to the session reviews. The training included (1) an orientation to the ¡CUÍDATE! program by the developer; (2) an overview of concepts pertaining to fidelity monitoring (including adherence and competence); and (3) an overview of the specific evaluation forms to be used for the analysis. In alignment with previous work (O’Donnell, 2005), the developer of the ¡CUÍDATE! program outlined critical components of the intervention along with an indication of the range of variations for acceptable use. After the initial training, the reviewers rated a random sample of audio activity files (not part of the study), using the study evaluation forms independently. The reviewers then came together and compared and discussed results. This process was continued until consistency was achieved amongst the raters (≥ 90% agreement).

Fidelity Evaluation Instruments

Two instruments were used to evaluate the fidelity of the implemented activities. The Activity Observation Fidelity Instrument—which was developed for this study—evaluated the alignment of the recorded activity with the description of the activity in the standardized curriculum (e.g., fidelity adherence). The 3-item instrument required the reviewer to determine if the activity was carried out (yes/no); if the activity was carried out according to the directions in the facilitator curriculum (yes/no); and if changes were made to the activity (free text). Reviewers were also able to provide any additional comments they had related to the delivery of the activity. The Activity Observation Fidelity Instrument was completed for each of the three activities that were reviewed for the session by each reviewer, using Qualtrics©. Prior to analysis, the instrument was reviewed by experts in the ¡CUÍDATE! curriculum and fidelity measurement and found the tool to be valid and appropriate for data collection.

Upon review of all three activities, the reviewers then completed the Program Observation Form for Teen Pregnancy Prevention Grantees to determine fidelity competence (U.S. Department of Health and Human Services Office of Adolescent Health, 2011). The purpose of this 10-item instrument is to measure the fidelity and quality of implementation of the program delivery. Questions addressed in the instrument included the following: explanation clarity, time allocation, material presentation (e.g., “rushed or hurried”), adolescent understanding and, participation, facilitator characteristics (e.g., knowledge of program, level of enthusiasm, poise and confidence, rapport and communication, effectively addressed questions), and overall quality of the program session. In addition, the reviewers were able to note any problems encountered, facilitator strengths, and any additional comments (free text). The instrument has been used in the evaluation of several EBI programs through the US Department of Health & Human Services, Office of Adolescent Health. Reviewers completed the form, which was converted to an electronic form in Qualtrics©.

Data Analysis

Data from the Activity Observation Fidelity Instrument and the Program Observation Form in Qualitrics© were downloaded into an excel file. Data was reviewed to ensure accuracy and several descriptive statistics were computed. To determine overall fidelity adherence, the number of activities done with fidelity (e.g. evaluators reporting the activity adhered to the curriculum) divided by the total number of activities reviewed was computed. For fidelity competence, descriptive statistics, including means and standard deviations, were computed for each item within the Program Observation Form. Additionally, field notes were taken for all of the debriefing sessions. These were reviewed to determine the presence of commonalities and themes in the feedback.

Evaluation Findings

From all facilitators (n=35) in the larger study, 19 returned the required audio files and were able to be evaluated for implementation fidelity. Of the 19, one did not report any demographic information. Facilitators were relatively equally distributed in gender distribution (10 female and 8 male facilitators). The majority of facilitators reported being Hispanic (n=10), with the remaining being Caucasian (n=8). The mean age was 31.3 years old. The programs that the facilitators worked for were mainly community-based non-profit organizations; six of the facilitators came from Planned Parenthood, eight programs centered on Latino youth, three were community-based health centers, and one was a teen support organization.

Each of the training session cohorts was represented in the sample: cohort 1 (n=3 recordings), cohort 2 (n=2), and cohorts 3-5 each had four recordings submitted. Additionally, the majority of sessions reviewed for implementation fidelity (68%) were conducted by more than one facilitator trained in the ¡CUÍDATE! curriculum.

Fidelity Adherence

Fidelity adherence for the ¡CUÍDATE! Training for both Session I and II was 91.2% (n=52), with Session I fidelity (86.1%) being less than Session II fidelity (100%). As noted in Table 3, further analysis was conducted to determine if the type of activity (knowledge, skill, or attitude) impacted the facilitator’s ability to implement the activity with fidelity during the adolescent training. Knowledge activities (n=17), in total, were conducted with fidelity 88.2% of the time. Attitude activities (n=19) were found to be completed 84.2% of the time with fidelity. The final type of activity—skill (n=21)—was completed 100% of the time with fidelity.

Table 3.

Fidelity Adherence for Various Types of Activities.

Type of Activity	Session I		Session II		Total
Type of Activity	Number Reviewed	% Done with Fidelity (#)	Number Reviewed	# Done with Fidelity (%)	Number Reviewed	% Done with Fidelity (#)
Knowledge	11	9 (81.8%)	6	6 (100%)	17	15 (88.2%)
Skill	14	14 (100%)	7	7 (100%)	21	21 (100%)
Attitude	11	8 (72.7%)	8	8 (100%)	19	16 (84.2%)
Total	36	31 (86.1%)	21	21 (100%)	57	52 (91.2%)

Comparisons were made between the two sessions to determine variation in fidelity among the activities reviewed. In total, Session I and II activities were completed with 86.1% (n=36) and 100% (n=21) fidelity, respectively. In Session I, skill activities were conducted with 100% fidelity, whereas knowledge and attitude activities were completed with 81.8% and 72.7% fidelity, respectively. All three types of activities (e.g. knowledge, skills, and attitude) were done with 100% fidelity in Session II.

Fidelity Competency

Fidelity competency was identified for each Session that was completed by the facilitators (Table 4). Overall, facilitators explained activities (X=4.47, SD 0.70) with a clear understanding of the material (X=4.68, SD 0.58). Facilitators were able to track time during the activities (X=3.68, SD 1.20) and engage group members in discussion and curricular activities (X=4.53, SD 1.02). Overall, the reviewers determined that facilitators addressed questions/concerns effectively (X=4.58, SD 0.69) and had excellent rapport and communication with participants (X=4.26, SD 1.05). Facilitators had high levels of enthusiasm (X=4.11, SD 1.10) and displayed poise and confidence during curriculum implementation (X=4.47, SD 0.90). Overall quality of the program session, as identified by the reviewers, was high (X= 3.95, SD 1.18).

Table 4.

Fidelity Competence of Facilitator for Each Reviewed Session.

Item	Session 1 Average (SD)	Session 2 Average (SD)	Overall Average (SD)
In general, how clear were the program implementer’s explanations of the activities?(1. Not clear, 3. Somewhat Clear, 5. Very clear)	4.33 (0.78)	4.71 (0.49)	4.47 (0.70)
To what extent did the implementer keep track of time during the session and activities?(1. Not on time, 3. Some loss of time, 4. Well on time)	3.33 (1.37)	4.29 (0.49)	3.68 (1.20)
To what extent did the presentation of materials seem rushed or hurried?(1. Very rushed, 3. Somewhat rushed, 5. Not rushed)	3.58 (1.24)	4.29 (0.95)	3.84 (1.17)
To what extent did the participants appear to understand the material?(1. Little understanding, 3. Some understanding, 5. Good understanding)	4.50 (0.67)	5.00 (0.0)	4.68 (0.58)
How actively did the group members participate in discussions and activities?(1. Little participation, 3. Some participation, 5. Active participation)	4.25 (1.22)	5.00 (0.0)	4.53 (1.02)
The following scale was for the remaining items: 1. Poor, 3. Average, 5. Excellent
Knowledge of the program	4.08 (0.90)	4.86 (0.38)	4.37 (0.83)
Level of enthusiasm	4.00 (1.28)	4.29 (0.76)	4.11 (1.10)
Poise and confidence	4.25 (1.06)	4.86 (0.38)	4.47 (0.90)
Rapport and communication with participants	4.00 (1.21)	4.71 (0.49)	4.26 (1.05)
Effectively addressed questions/concerns	4.33 (0.78)	5.00 (0.0)	4.58 (0.69)
Rate the overall quality of the program session.	3.67 (1.37)	4.43 (0.53)	3.95 (1.18)

Additional quality comparisons were made overall between Session I and Session II. Average scores were higher for all items in Session II as compared to Session I, with the greatest improvement being in the facilitators’ ability to keep track of time (0.96 point improvement) and student engagement during sessions, discussions and activities (0.75 point improvement). In terms of the facilitator characteristics, improvements from Session I to Session II were greatest for facilitator knowledge of the program (0.78 point improvement) and for overall quality of the program session (0.76 point improvement). For Session II, facilitators were rated at the highest level (5.0) for the participants’ understanding of the material, the participants’ active engagement in discussions and activities, and the facilitators’ effectiveness of addressing adolescent questions and concerns.

Facilitator Debriefing

Facilitators reflected on personal performance at the debriefing sessions following their respective teachbacks, identifying strengths and areas for improvement. Many of the facilitators recognized opportunities for improvement, primarily relating to developing better connections with the “adolescents” (e.g., peers role-playing as adolescent avatars in the training). Facilitators found it difficult to determine the extent adolescents understood their teachback content, which would typically be indicated by non-verbal cues (e.g., making eye contact, shaking heads, etc.) in the SL environment. Facilitators indicated their presentation of the material might have changed (e.g., reiterating a difficult concept) had such non-verbal cues been assessed. Master Trainers provided expert opinion based on their experiences and provided additional support to the facilitators as they identified areas for improvement for future teachbacks.

In addition to the post-teachback debriefing, all facilitators participated in a debriefing following each session. During the debriefing, facilitators were asked to share their reflections on the day’s presentations (Master Trainer and facilitator-based), learning activities, and conduciveness of the environment. Facilitators reported that SL was an engaging and interesting tool for connecting with people across the country, and that it was both accessible and easy to hear. Overall, facilitators believed the training to be a great experience, with SL providing a relaxed, close to real environment to practice curricular implementation. However, while a few participants believed they were able to match people with their personalities, one of the major drawbacks—as noted above—resulted from the lack of face-to-face connection. Although SL provided aspects of social presence, the facilitators were unable to connect directly with the role-played adolescents (e.g., nodding, eye contact, etc.) to determine their content understanding. The other significant drawback, as reported by facilitators, was the technological issues experienced during the training. Primarily, technical issues revolved around wireless Internet connections and system load (e.g. lagging, loss of connection), headset failures, and in-world functionality (e.g. group chat). Despite having a research team member who was devoted solely to assisting facilitators with technical difficulties, the technical issues became a distraction to some of the facilitators.

Discussion

The purpose of this study was to examine whether facilitators who attended the ¡CUÍDATE! virtual training in SECOND LIFE™ implemented the program with fidelity to adolescent youth. Facilitators implemented ¡CUÍDATE! to adolescent youth successfully and with overall fidelity as evidence by high fidelity adherence and competency scores. Fidelity adherence of implementation was very high (91%), with Session II fidelity reaching 100% across facilitators. This finding is similar to what was reported for ¡CUÍDATE! Training after face to face sessions (87.8%) (Villarruel et al., 2006), indicating that use of SL may provide the platform needed for wide dissemination and scale up. Virtual training may be the necessary direction in order to reach the community-based agencies that work with adolescent youth and have limited resources.

Another finding of relevance is the improved fidelity scores that were noted from Session I to Session II. All of the activities reviewed for Session II were completed with fidelity, as compared to Session I (86% completed with fidelity). This is in line with Ericsson’s framework, which reinforces the importance of deliberate practice (Ericsson, 2004). As noted by Ericcson’s work, expertise improves with deliberate practice of skills (e.g. facilitating ¡CUÍDATE!). It might also be in part due to the types of activity. Skill-based activities—which were found to be done with fidelity 100% of the time in both sessions—may be easier for facilitators to conduct as it is a relatively structured flow of the activity (e.g. standardized steps in the process). Knowledge and attitude-based activities require a greater level of expertise and finesse on the part of the facilitator as the activities are more fluid, depending on the current knowledge level and attitude of the adolescents.

Overall fidelity competency of the facilitators was very high, especially in relation to the adolescents’ understanding of the material (X=4.68, SD 0.58) and their engagement in the discussions and activities (X=4.53, SD 1.02). Facilitators created a trusting environment where adolescents felt free to discuss the concepts in the ¡CUÍDATE! Curriculum. Although this may in part be related to the facilitators’ past experiences with working with adolescents, findings noted improvement in all items of fidelity competency that are related to explanations, time allocation, and participant engagement from Session I to Session II. Furthermore, facilitator knowledge, enthusiasm, confidence, rapport, ability to address questions, and overall quality of the program improved in Session II.

Debriefing sessions occurred at the conclusion of each training day and were important to the researchers, as modifications to the training implementation could be made to maximize learning for the facilitators. In line with previous studies, facilitators overall found SL fun, engaging, easy to use and enjoyed the opportunity to interact with others who were geographically distant (Boulos & Toth-Cohen, 2009; Mitchell et al., 2011). Facilitators reported the training to be a great experience, with SL providing a relaxed, close to real environment to practice curricular implementation. SL provided an avenue for bringing a diverse group of learners together to learn in a creative, interactive fashion. Although facilitator report does not necessarily equate to attaining learning outcomes, the high levels of fidelity provides evidence for knowledge uptake. Facilitators recognized the value of SL and were able to get beyond the challenges that the environment provided, including lack of face-to-face connection (e.g., non-verbal cues that indicate understanding). Although SL provided social presence, the facilitators were unable to connect directly with the role-played adolescent avatars (e.g., nodding, eye contact, etc.) to determine their content understanding. The other significant drawback, as reported by facilitators, was the technological issues experienced during the training. Although the technical issues reported by the first cohort (e.g., group chat malfunction) were fixed for future sessions, several issues relating to internet connectivity (e.g., bandwidth) were unable to be resolved for all facilitators. Further work is needed to evaluate the specific causes for these technical difficulties, which may in turn assist in determining minimum hardware and software standards for participation (Valladares et al., 2014).

A few limitations to the study must be considered. Due to the small sample size, it is difficult to generalize findings. Further research is needed to evaluate the efficacy of SL for training in other community-based organizations. Another limitation is the inability to review the audio recordings for each facilitator’s activities. Knowledge-, attitude- and skill-based activities were randomly selected for each facilitator, thus we may have obtained different results had all activities been evaluated. Also of importance is the fact that facilitators modeled only a few activities in the virtual setting. Revisions to the training could include increasing the number of activities that need to be modeled by each facilitator; these activities could be modeled either synchronously in the virtual environment or perhaps using software such as Voice Thread© (VoiceThread LLC).

Another limitation is the potential for subjectivity among the reviewers. Although a training sessions was conducted and greater than 90% agreement was achieved among the reviewers prior to analysis, possibility for variability in the evaluation may have occurred. A final limitation is the inability to link implementation fidelity with actual adolescent outcomes. Further work should consider evaluating adolescent outcomes (e.g., age at first sex, use of condoms, etc.) to determine the overall effectiveness of the training.

Conclusion

Many successful EBIs require intense face-to-face training, limiting the ability for many community-based organizations to participate due to limited resources. Strategies for efficient and effective dissemination are needed to maximize the potential impact of EBIs on the health of our nation. Use of a virtual platform, such as SECOND LIFE™, may be the answer for maximizing impact through greater dissemination among those working directly with adolescents. This study implemented a virtual facilitator training for ¡CUÍDATE! a sexual risk behavior prevention intervention for Latino Youth (Figure 5). Findings from this study provide evidence to support the use of virtual environments for training, without a negative impact on fidelity. Future work is needed to explore the use of virtual reality for other types of inter-professional education and its impact on implementation fidelity and learner outcomes.

Figure 5.

The ¡CUÍDATE! team.

Footnotes

Acknowledgements

We wish to express our appreciation and indebtedness to our reviewers for their tremendous help in improving this article. They are Barbara Sittner, Guillaume Henri Jean Alinier, Nicole Harder, and Amanda Wilford. In addition, we thank Glen Marian for his significant contribution in editing and formatting the manuscript.

Author Contributions

All authors contributed to this article, in content and in form. DT wrote the initial draft of the manuscript, with feedback from MA, JY and AV. All authors contributed equally to the editing of the manuscript.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by National Institute of Mental Health R21-MH095723.

Author Biographies

Dana Tschannen, PhD, RN—a clinical associate professor—is the director of the Post Masters Doctorate of Nursing Practice (DNP) Program and Vice Chair for the Department of Systems, Populations, and Leadership at the University of Michigan. The primary purpose of her research agenda is to positively impact safe patient care at the bedside through applying technology to improve knowledge acquisition, evidence based practice adoption, care delivery and workflow. This has included using technology (e.g. Clinical Care Daily, HANDS) to aid in point of care decision-making as well as for improved training and education around evidence-based practices and effective team functioning among students and multi-disciplinary staff. Dr. Tschannen has extensive experience in the design, development, and evaluation of educational trainings using virtual and face to face simulation methodologies.

Contact: djvs@umich.edu.

Joseph D. Yaksich, MS, RN, ACNP-BC earned his masters degree in Adult Acute Care form the University of Michigan School of Nursing. It was during that education that he was first exposed to the value of simulation. After graduation he stayed on at the University of Michigan to work in the Clinical Learning Center which utilizes low to high fidelity Simulation for nursing education. Currently Joseph is on faculty at Henry Ford College in Dearborn Michigan as the Nursing Simulation Faculty and Technology Instructor. There he is implementing a simulation program through the curriculum that is centered around nursing education. He also continues to practice part time as an Acute Care Nurse Practitioner.

Contact: jyaksich@umich.edu.

Michelle Aebersold, RN, PhD, CHSE, FAAN has been playing games since she got her first Atari game console as a kid playing asteroids. Now as a PhD prepared researcher she has works with virtual environments, learning technologies and high-fidelity simulation. She was a 2014 presenter at TEDxUofM-on the important role of simulation in nursing education. She is a member of the Standards Committee for the International Nursing Association for Clinical Simulation and Learning.

Contact: mabersol@umich.edu.

Antonia Villarruel, PhD, RN, FAAN is the Margaret Bond Simon Dean of Nursing at the University of Pennsylvania School of Nursing and Director of the Schools WHO Collaborating Center for Nursing and Midwifery Leadership. She has developed a keen interest in the digital world as a result of her extensive research in developing, testing, and disseminating evidence-based interventions to reduce Latino sexual risk behaviors. As a bilingual and bicultural nurse researcher, she developed an efficacious program to reduce sexual risk behavior among Latino youth – entitled Cuídate! This program is disseminated nationally by the CDC and has led to the study of Second Life, as a virtual facilitator training platform that has the potential to accelerate the dissemination of evidence-based interventions by increasing accessibility and affordability. Several of her recent studies involve web-based interventions. Dr. Villarruel has offered webinars on topics of diversity and hosted an open forum webinar on nursing issues. She also finds time to garden and listen to Siriusly Sinatra especially this in his 100th birthday year! Her favorite quote is: Energy and persistence conquer all things –Benjamin Franklin.

Contact: nursingdean@nursing.upenn.edu.

References

Aldrich

(2009). The complete guide to simulations and serious games: How the most valuable content will be created in the age beyond Gutenberg to Google. San Francisco, CA: John Wiley.

Barber

J. P.

Gallop

Crits-Christoph

Frank

Thase

M. E.

Weiss

R. D.

. . . Gibbons

M. C.

(2006). The role of therapist adherence, therapist competence, and alliance in predicting outcome of individual drug counseling: Results from the National Institute Drug Abuse Collaborative Cocaine Treatment Study. Psychotherapy Research, 16, 229-240.

Beard

Wilson

Morra

Keelan

(2009). A survey of health-related activities on Second Life. Journal of Medical Internet Research, 11(2), e17.

Boulos

M. N. K.

Hetherington

Wheeler

(2007). SECOND LIFE™: An overview of the potential of 3-D virtual worlds in medical and health education. Health Information & Libraries Journal, 24, 233-245.

Boulos

M. N. K.

Toth-Cohen

(2009). The University of Plymouth sexual health SIM experience in SECOND LIFE™: Evaluation and reflections after 1 year. Health Information and Libraries Journal, 26, 279-288.

Breitenstein

S. M.

Fogg

Garvey

C. A.

Hill

Resnick

Gross

(2010). Measuring implementation fidelity in a community-based parenting intervention. Nursing Research, 59, 158-165.

Breitenstein

S. M.

Gross

Garvey

C. A.

Hill

Fogg

Resnick

(2010). Implementation fidelity in community-based interventions. Research in Nursing & Health, 33, 164-173.

Chen

Y. F.

(2014). Evaluation of strategic emergency response training on an OLIVE platform. Simulation & Gaming, 45, 732-751.

DerSahakian

Alinier

Savoldelli

Oriot

Jaffrelot

Lecomte

(2015). Setting conditions for productive debriefing. Simulation & Gaming, 46(2), 197-208.

10.

Durkin

Pollack

Star

J. R.

Rittle-Johnson

(2012). Differences in fidelity of implementation measures. What videos and surveys reveal about algebra instruction. Paper presented at the SREE Spring 2012 Conference, March, Evanston, IL.

11.

Dusenbury

Brannigan

Falco

Hansen

W. B.

(2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research: Theory and Practice, 18, 237-256.

12.

Enfield

Myers

Lara

Frick

(2012). Innovation diffusion: Assessment of strategies within the diffusion simulation game. Simulation & Gaming, 43, 188-214.

13.

Ericsson

K. A.

(2004). Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Academic Medicine: Journal of the Association of American Medical Colleges, 79(Suppl. 10), S70-S81.

14.

Feinstein

A. H.

Cannon

H. M.

(2002). Constructs of simulation evaluation. Simulation & Gaming, 33, 425-440.

15.

Gao

Noh

Koehler

(2009). Comparing role-playing activities in Second Life to face to face environments. Journal of Interactive Learning Research, 20, 423-443.

16.

Ghanbarzadeh

Ghapanchi

A. H.

Blumenstein

Talaei-Khoei

(2014). A decade of research on the use of three-dimensional virtual worlds in health care: A systematic review. Journal of Medical Internet Research, 16(2), e47.

17.

Heinrichs

Youngblood

Harter

Dev

(2008). Simulation for team training and assessment: case studies of online training with virtual worlds. World Journal of Surgery 32(2), 161-170.

18.

Kemp

(2007). Second Life—Heart Murmur Sim video demonstration. Retrieved from https://youtube.com/watch?v=xJY2Iwbzop4

19.

Kripalanti

Bengtzen

Henderson

Jacobson

(2008). Clinical research in low-literacy populations: Using teach-back to assess comprehension of informed consent and privacy information. IRB: Ethics & Human Research, 30(2), 13-19.

20.

Kyaw Tun

Alinier

Tang

Kneebone

(2015). Redefining simulation fidelity for healthcare education. Simulation & Gaming, 46, 159-174.

21.

Mitchell

Heyden

Schroy

Andrew

Sadikova

Wiecha

(2011). A pilot study of motivational interviewing in training in a virtual world. Journal of Medical Internet Research, 13(3), e77.

22.

Mowbray

C. T.

Holter

M. C.

Teague

G. B.

Bybee

(2003). Fidelity criteria: Development, measurement, and validation. American Journal of Education, 24, 315-340.

23.

Nehring

W. M.

Lashley

F. R.

(2009). Nursing simulation: A review of the past 40 years. Simulation & Gaming, 40, 528-552.

24.

O’Donnell

(2005). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K-12 curriculum intervention research. Review of Educational Research, 78, 33-84.

25.

Peterson

(2005). Learning interaction in an avatar-based virtual environment: A preliminary study. PacCALL Journal, 1(1), 29-40.

26.

Rudolph

J. W.

Simon

Dufresne

R. L.

Raemer

D. B.

(2006). There’s no such thing as “nonjudgmental” debriefing: A theory and method for debriefing with good judgment. Simulation in Healthcare, 1, 49-55.

27.

SECOND LIFE^TM. Rosedale

(2003). San Francisco, CA: Linden Lab. (945 Battery Street, San Francisco, CA 94111, USA)

28.

U.S. Department of Health and Human Services Office of Adolescent Health. (2011). Program observation form for TPP grantees. Retrieved from http://www.performwell.org/index.php?option=com_mtree&;task=att_download&link_id=372&cf_id=24

29.

Valladares

A. F.

Aebersold

Tschannen

Villarruel

A. M.

(2014). Preparing facilitators from community-based organizations for evidence-based intervention training in SECOND LIFE™. Journal of Medical Internet Research, 16(9), e220.

30.

Villarruel

A. M.

Aebersold

Valladares

A. F.

Yeagley

Tschannen

(2014). Avatars travel for free: Virtual access to evidence-based intervention training and capacity building. AIDS Education and Prevention, 26, 445-458.

31.

Villarruel

A. M.

Gal

T. L.

Eakin

B. L.

Wilkes

Herbst

J. H.

(2010). From research to practice: The importance of community collaboration in the translation process. Research and Theory for Nursing Practice, 24, 25-34.

32.

Villarruel

A. M.

Jemmott

J. B.

Jemmott

L. S.

(2006). A randomized controlled trial testing an HIV prevention intervention for Latino youth. Archives of Pediatrics & Adolescent Medicine, 160, 772-777.

33.

Villarruel

A. M.

Jemmott

L. S.

Jemmott

J. B.

(2012). ¡CUÍDATE! A culturally-based program to reduce sexual risk behavior among Latino youth: Facilitator curriculum (2nd ed.). New York, NY: Select Media.

34.

Villarruel

A. M.

Zhou

Gallegos

E. C.

Ronis

D. L.

(2010). Examining long-term effects of Cuídate: A sexual risk reduction program in Mexican youth. Revista Panamericana de Salud Pública, 27, 345-351.

35.

Warburton

(2009). Second Life in higher education: Assessing the potential for and the barriers to deploying virtual worlds in learning and teaching. British Journal of Educational Technology, 40(3), 414-426.

36.

White

Garbez

Carroll

Brinker

Howi-Esquivel

(2013). Is “teach-back” associated with knowledge retention and hospital readmission in hospitalized heart failure patients? Journal of Cardiovascular Nursing, 28, 137-146.

Fidelity After SECOND LIFE Facilitator Training in a Sexual Risk Behavior Intervention

Abstract

Keywords

Literature Review

Second LifeTM

Measures of Fidelity

Methods

Sample

¡CUÍDATE! Training

Fidelity Evaluation Procedure

Fidelity Evaluation Instruments

Data Analysis

Evaluation Findings

Fidelity Adherence

Fidelity Competency

Facilitator Debriefing

Discussion

Conclusion

Footnotes

Acknowledgements

Author Contributions

Declaration of Conflicting Interests

Funding

Author Biographies

References

Second Life^TM