A Technology-Mediated Approach to the Implementation of an Evidence-Based Child Maltreatment Prevention Program

Abstract

Implementation of evidence-based parenting programs is critical for parents at-risk for child maltreatment perpetration; however, widespread use of effective programs is limited in both child welfare and prevention settings. This exploratory study sought to examine whether a technology-mediated approach to SafeCare^® delivery can feasibly assist newly trained providers in achieving successful implementation outcomes. Thirty-one providers working in child welfare or high-risk prevention settings were randomized to either SafeCare Implementation with Technology-Assistance (SC-TA) or SafeCare Implementation as Usual (SC-IU). SC-TA providers used a web-based program during session that provided video-based psychoeducation and modeling directly to parents and overall session guidance to providers. Implementation outcome data were collected from providers for six months. Data strongly supported the feasibility of SC-TA. Further, data indicated that SC-TA providers spent significantly less time on several activities in preparation, during, and in follow-up to SafeCare sessions compared to SC-IU providers. No differences were found between the groups with regard to SafeCare fidelity and certification status. Findings suggest that technology can augment implementation by reducing the time and training burden associated with implementing new evidence-based practices for at-risk families.

Keywords

evidence-based practice child welfare parenting prevention child maltreatment

In 2013, over four fifths of the estimated 679,000 confirmed abuse and neglect cases (91.4%) in the United States were perpetrated by either one or both parents (U.S. Department of Health and Human Services, Administration on Children, Youth, and Families, Children’s Bureau, 2015). Parents who are substantiated as perpetrators of child maltreatment are commonly referred to “parent training” programs with a goal of family preservation (Chaffin, Bonner, & Hill, 2001). Several behavioral parent training interventions have evidence for improving parent behaviors and reducing child maltreatment recidivism, including SafeCare^® (Chaffin, Hecht, Bard, Silovsky, & Beasley, 2012), Triple P (Prinz, Sanders, Shapiro, Whitaker, & Lutzker, 2009), Parent–Child Interaction Therapy (Chaffin et al., 2004), and Incredible Years (Hurlburt, Nguyen, Reid, Webster-Stratton, & Zhang, 2013; Webster-Stratton, 2014).

SafeCare is one of the most widely disseminated evidence-based parenting models in child welfare and prevention settings (23 states and 6 countries), with several studies indicating significant reductions in child maltreatment reports and recidivism for families who complete the intervention as part of child welfare or prevention services (Chaffin, Hecht, et al., 2012; Gershater-Molko, Lutzker, & Wesch, 2002; Silovsky et al., 2011). Additionally, prior research demonstrates that SafeCare leads to improvement in maternal depression (Chaffin, Bard, Bigfoot, & Maher, 2012) and child behavioral outcomes (Carta, Lefever, Bigelow, Borkowski, & Warren, 2013). SafeCare consists of three modules of six sessions each that target proximal risk factors for child physical abuse and neglect. The model structure is based on applied behavioral principles, and each module includes session protocols, structured assessments, and a training format including psychoeducation, modeling, and practice of the target skills as well as a structured goal-directed feedback approach that enhances parent skill mastery (Self-Brown et al., 2014).

Even with the increased spread of SafeCare and other evidence-based parenting models in the last decade, most of the parent-focused interventions delivered in child welfare and high-risk prevention settings have limited empirical support and generally focus on case management. A multitude of factors impact the broad-based dissemination of evidence-based programs, including provider attitudes toward evidence-based practice, inadequate training of providers, lack of organizational support for the implementation of evidence-based programs, and perceived fit with family needs (Aarons, 2004; Aarons & Palinkas, 2007; Schoenwald & Hoagwood, 2001). Research examining approaches that can enhance the reach of quality, evidence-supported programs within settings that can improve the lives and outcomes of vulnerable children and families is warranted.

Limited research to date has specifically explored how technological applications can reduce implementation barriers of evidence-based parenting programs in settings that serve families at-risk for maltreatment. In the field of child mental health, Beidas, Edmunds, Marcus, and Kendall (2012) evaluated the efficacy of a technology-based therapist training program for an evidence-based practice for child anxiety in comparison to live workshop training. Importantly, following training, all newly trained therapists were supported through video-teleconferencing consultation. The technology-based and live training methods produced small gains in therapist adherence, skill, and knowledge. Ultimately, however, it was the number of ongoing consultation hours after training that significantly predicted higher skill and adherence at follow-up (Beidas, Edmunds, Marcus, & Kendall, 2012).

Considerations for how to use technology in the implementation of evidence-based practices should not be limited strictly to provider training. Another option is to consider how technology can assist with intervention delivery directly to the consumer, such as the computer-mediated approach to service delivery described by Coyle, Doherty, Matthews, and Sharry (2007). Specifically, Coyle and colleagues propose that technology can assist in intervention delivery by including the computer, or relevant technology platform, as the third partner in the relationship between a provider and consumer. No known research has examined the impact of computer-mediated interventions on implementation outcomes. Such an approach could reduce evidence-based practice demands by assisting with (1) client assessment, (2) session delivery, (3) data-driven guidance of provider session tasks, (4) standardization and adherence of intervention protocol, and (5) session documentation.

This exploratory study examines a technology-mediated approach to session delivery of SafeCare, with newly trained providers delivering to families involved with child welfare or high-risk prevention community efforts. The technology tested is a tablet-delivered, web-based program (developed by the research team, entitled “SafeCare Takes Care”) designed to assist SafeCare providers during session delivery with parents. The primary objective of this randomized controlled trial is to examine the feasibility of the technology-assisted implementation of SafeCare (SC-TA), as compared to the usual SafeCare implementation (SC-IU), which is based on the best practices in implementation (Guastaferro, Lutzker, Graham, Shanley, & Whitaker, 2012; Whitaker, Lutzker, Self-Brown, & Edwards, 2008). Outcomes include SafeCare provider session fidelity, SafeCare provider certification status, SafeCare provider time burden, and the overall acceptability and feasibility of the technology assistance. Hypotheses are as follows: (1) providers in the SC-TA group will spend less time on preparation, delivery, and postsession activities compared to the SC-IU group, (2) providers in the SC-TA group will have higher fidelity scores and achieve certification at greater rates than SafeCare providers implementing without the technology assistance, and (3) providers in the SC-TA group will endorse the feasibility of the technology assistance.

Method

Participants

Study participants were 31 SafeCare providers who worked for 1 of 17 agencies that served parents involved in high-risk prevention or child welfare services across eight states. Specifically, 47% of providers were employed by agencies that serve families involved with child welfare, 41% worked for agencies that serve families at high risk for abuse or neglect, and 12% worked for agencies that serve families across both settings (e.g., substantiated and unsubstantiated families). Each agency participated in SafeCare training between February 2014 and October 2015. Providers were mostly female (87.1%) and were diverse in race and ethnicity (45% Caucasian, 23% Black, 32% Other, and 29% Latino). The majority of providers’ highest level of education was a bachelor’s degree (58%), though some had completed graduate degrees (23%). In terms of experience working with high-risk families, 53% of providers had 6 or more years’ experience.

The one study inclusion criteria for providers was that the provider employing agency was signed up for SafeCare workshop training. Agencies that contract for SafeCare training with the National SafeCare Training and Research Center (NSTRC) go through a mandatory implementation process, which includes Agency Readiness, SafeCare Provider Workshop Training, SafeCare Provider Initial Field Certification, and Active Implementation with Fidelity Maintenance. Agency directors that completed the Agency Readiness process and had enrolled providers in workshop training were asked about their willingness to participate in the current research project. An attempt was made to recruit 28 agencies, and 17 agencies (61%) agreed that the research team could invite the providers signed up for workshop training to participate in the study. Agencies that did not allow the invitation for study participation declined due to policy or funding issues, providing services to a predominantly Spanish-speaking population for which the web-based program was not available, or because the agency withdrew from SafeCare training after initially signing up and completing the readiness process.

Providers were invited to participate in the study via e-mail, which included study details and asked interested providers to attend a webinar where the study consent process and logistics were described. Following webinar participation, consent forms were e-mailed to providers, and those willing to participate signed the study consent form and completed a baseline assessment survey prior to or during the SafeCare Provider Workshop Training. Those in the SC-TA group participated in a webinar after workshop training, which provided them with the additional information on the use of the technology and navigation of the web-based program. Of the 59 SafeCare providers recruited for the study, 31 (53%) consented to participate in the project.

Procedures

The Georgia State University Institutional Review Board approved this study. Providers participated in the project for an average of 7.74 months, with a range of 4 to 10 months. The study period began at the time of the baseline assessment before or during the SafeCare Provider Workshop and concluded 6 months from the point at which the provider began delivering SafeCare with a family. Accordingly, the study period varied based on when the provider began implementing SafeCare with a family post-workshop. Providers were expected to complete three assessments (baseline, 3 months, and 6 months after starting SafeCare work with families) and weekly time diaries during the active phase of the project. Providers were incentivized US$30 per assessment and US$10 a month for time diaries. Each participating provider in the project was randomly assigned to either SC-IU or SC-TA. Because this study was focused on the feasibility of the SC-TA approach as compared to an SC-IU, a well-established implementation approach (Guastaferro et al., 2012; Self-Brown et al., 2014; Whitaker et al., 2008), a decision was made to use a randomized allocation design of 2:1. Thus, 20 providers were assigned to SC-TA and 11 were assigned to SC-IU.

Description of SafeCare

SafeCare is an 18-week parent training program delivered via home visitation. SafeCare contains three, six-session modules that target risk factors for child physical abuse and neglect, including (1) child health, (2) home safety, and (3) parent–child interaction. The focus of the child health module is to train parents to make effective decisions related to child health, to prevent child illness and injury, and assist parents in seeking appropriate health treatment for children when an illness or injury occurs. Parents are also taught to use health reference materials, including a validated SafeCare health manual. Home safety focuses on making the home a safe place where the risk for child injury is reduced. Parents identify and eliminate common household hazards and learn the appropriate levels of supervision for children across developmental stages. The parent–child interaction module focuses on the parent–child relationship, with the goal of improving interactions between the child and parent as well as child behavior by teaching parents structured skills to use in a consistent and predictable way with children across daily routine activities.

Each of the three SafeCare modules is structured according to an assess-train-assess approach, with six sessions each. Session 1 is a pretest of baseline skills and knowledge of the particular module. Sessions 2–5 are training sessions, which include explanation of target skills, modeling of target skills by the provider, practice of the target skills by the parent, and feedback from the provider about the parent’s mastery and competence in skills. Session 6 is a second assessment session, which allows the provider to examine parent mastery of target skills across each module. Providers train the parents in each module to a level of mastery, which is the demonstration of at least 80% of target skills at the Session 6 assessment. For a more detailed description of the SafeCare modules and structure, please refer to Self-Brown et al. (2014).

Implementation Study Conditions

SC-IU

The SC-IU condition followed the protocol used in standard SafeCare implementation, which includes the following phases: (1) Agency readiness—NSTRC reviews the requisite organizational communication, service system and SafeCare funding plan, implementation requirements, and budgetary information with interested agencies to ensure fit of SafeCare within the organizational context of the agency, (2) SafeCare workshop training—This includes a 4-day classroom-based training that involves didactic presentations, modeling of the instruction, and role-playing for SafeCare trainees as well as structured assessments of the skills taught, (3) SafeCare provider certification process—Providers receive support from NSTRC as they begin SafeCare delivery with families. The support includes the SafeCare trainers listening to audio recordings of the providers’ SafeCare sessions, assessing these sessions for fidelity, and then providing a follow-up coaching session to provide feedback to the provider. Once trainees demonstrate mastery of SafeCare, defined as meeting session fidelity of ≥85%, in three sessions per module (nine total), they are certified as a SafeCare provider, (4) SafeCare postcertification support and sustainability—Once a provider achieves certification, the provider moves into this phase that consists of monthly submissions of one SafeCare session recording monitored for fidelity by an NSTRC trainer or trained SafeCare coach at the implementing agency.

SC-TA

This implementation followed the four phases described above, but the delivery of SafeCare was adapted to include technology assistance delivered on a tablet via a web-based SafeCare program entitled SafeCare Takes Care. SafeCare Takes Care includes a combination of video, audio narration, and engaging questions and was developed through an alpha and beta testing process with parenting experts and parents similar in education and socioeconomic status to the parents served with the SafeCare program (Cowart-Osborne et al., 2014). The videos are presented in a manner similar to a talk show. For each module and session, the host of SafeCare Takes Care presents a new topic (i.e., the session content for that day) with video modeling of the skills from “at-home viewers.” For example, in the parent–child interaction module, a video begins with the talk show host explaining the skills being covered in the session, followed by a video of a parent (i.e., an at-home viewer) modeling these skills, and the host may take some questions from studio audience members or from fans on the “street cam.”

SafeCare Takes Care uses an open-source systems and languages to input text, picture, and video-related content into website interventions. All text was narrated to minimize literacy requirements. The architecture is based on the Python programming language using a Django web framework and Foundation (http://foundation.zurb.com), an advanced responsive front-end framework, to ensure mobile friendliness. This framework consists of Cascading Style Sheets and Javascript to ensure proper display of the web application across multiple devices with differing screen sizes and resolutions. SafeCare Takes Care was hosted at Oregon Research Institute (ORI) on a Linux server with MySQL, an open-source language for relational database development. Data collection components were securely transmitted to ORI servers using Secure Sockets Layer protocol. This platform has been successfully used for the delivery of other evidence-based parenting programs to high-risk parents (see Baggett et al., 2010).

SafeCare providers assigned to the SC-TA condition participated in the standard SafeCare workshop and also received training in the technology-mediated approach to SafeCare delivery. The technology training took approximately 2 hr and focused on how the provider utilizes the technology in each session. Specifically, after greeting the parent, the provider was instructed to connect the parent to the web-based program, during which the parent participates in the multimodal learning (e.g., explanation and modeling of skills) of SafeCare target skills. When the parent completes the web-directed portion of the session, the provider is prompted by the web-based program to take over the session delivery, revisit any explanation and modeling the parent has questions about, and then engage the parent in live practice of the skills presented in the web program. Lastly, the provider offers positive and constructive feedback about the practice and closes the SafeCare session.

In addition to the technology-mediated delivery, there were some slight adaptations to the scoring instructions of the SafeCare Fidelity checklist for the SC-TA to accommodate the use of the web-based program into the session (see Table 1). Specifically, fidelity items pertaining to explanation and modeling were scored as completed by a coach if it was clear in the audio recording of the session that the provider connected the parent participant to the web-based program. All other fidelity items on the SafeCare Fidelity Checklist remained the same. That is, SafeCare providers were fully expected to deliver the session opening, SafeCare target skills practice and feedback, and session closing. Additionally, if parents had questions about the explanation or modeling components reviewed in the web-based program, providers were trained to address these concerns, and fidelity was rated as it would be in standard implementation for these items. Lastly, SafeCare coaches were instructed to include the providers’ technology equipment under their scoring of “has materials ready” on the fidelity checklist. Providers in both groups participated in coaching calls with their assigned SafeCare coach following the coach’s scoring of fidelity, as is the protocol for SC-IU. These calls serve as an opportunity for the coach to provide positive and constructive feedback to the provider regarding their session delivery.

Table 1.

SafeCare Fidelity Checklist.

Has materials ready

1.Home visitor assessment document(s)

2.Parent training materials

3.Materials and other supplies^a

Opens the session

4.Exchanges appropriate initial greeting

5.Gives session overview

6.Discusses parent’s practice since last session

Demonstrates appropriate demeanor

7.Sits facing client

8.Communicates empathy, warmth, understanding

9.Maintains open posture

10.Has good eye contact

Uses active listening techniques

11.Uses words/expressions to encourage parent to talk

12.Uses open-ended questions

13.Uses reflecting or summarizing statements

Conducts formal assessments

14.Explains purpose and/or process of assessment

15.Assesses appropriate room/scenario/activity

16.Completes necessary assessment form(s)

Trains parent in skills/behaviors

17.Uses appropriate materials to train parent

18.Explains skills/behaviors to parent

19.Physically models skills/behaviors^a

20.Has parent practice skills/behaviors

21.Uses assessments form to document parent practice

22.Provides specific, positive feedback

23.Provides specific, corrective feedback

24.Repeats SafeCare 4 process until mastery/success or session time expires

25.Plans parent’s practice before next session

Addresses issues that arise during session

26.Encourages parent to ask questions/express concerns

27.Responds to parent questions/concerns

28.Uses problem solving when appropriate

Follows appropriate end of session sequence

29.Summarizes session

30.Asks for and answers parent questions

31.Gives general, positive feedback

32.Schedules meeting date/time for next session

^aItems in which scoring instructions were modified for SafeCare Implementation with Technology-Assistance.

Measures

Provider demographics and professional background

These factors were measured by a form developed for the project and asked questions regarding provider age, education, race/ethnicity, and field experience.

SafeCare delivery time demands

Time diaries were completed by providers who were delivering SafeCare to families. Providers were instructed to complete a time diary following each SafeCare session and submit them to the research team on a monthly basis. Information reported on the time diary form included the specific amount of time in minutes spent on SafeCare-related activities prior to, during, and following each session. These activities were determined by the research team in consultation with SafeCare trainers who are familiar with the common activities conducted by SafeCare providers.

Provider fidelity

Fidelity was measured utilizing the SafeCare Provider Fidelity Checklist (Lutzker & Bigelow, 2002). The checklist includes a number of concrete behaviors providers should perform during the SafeCare session. Providers submit audio recordings to NSTRC and expert coders rate fidelity using this checklist based on the verbal behaviors performed by the provider in the SafeCare session.

Provider implementation progress

SafeCare provider implementation progress was documented by research team members based on the implementation record review. Records were maintained by the NSTRC trainer who noted the progress of each provider through the training and certification process. To meet certification, the provider has to achieve 85% fidelity on three sessions in each SafeCare module (parent–child interaction, child health, and child safety), for a total of nine sessions. Provider implementation progress was coded as “workshop only” if the provider completed the training workshop but did not begin working with families. Providers were coded as “Began certification, SafeCare inactive,” if they began certification after workshop training but discontinued SafeCare delivery before reaching certification. Providers coded as “Began certification, SafeCare active” were still delivering SafeCare services at the end of the study period but had not yet achieved certification. Lastly, providers coded as SafeCare certified achieved at least 85% fidelity on nine SafeCare sessions during the study period.

SC-TA qualitative interview

A semi-structured qualitative interview was conducted upon completion of the study to gather feedback on satisfaction and recommendations for SC-TA and learn how the SC-TA implementation delivery approach compared to SC-IU. Questions included in the interview focused on specific feasibility topics as follows: (1) technology usability (e.g., Did the technology work well during SafeCare sessions? How do you feel about using the technology in session? Are there any challenges or things you don’t like about using the technology in session?); (2) SafeCare delivery effort (e.g., Can you speak to how the tech-assisted delivery added to/eased the burden of delivering a newly learned intervention? If given the choice to continue with tech-assistance vs. standard delivery would you? Would you recommend the tech-assisted delivery process to colleagues?); (3) progress through SafeCare implementation (e.g., Do you feel like the technology assistance impacted the fidelity of your sessions in anyway? Do you think the technology assistance impacted your progress through certification?); and (4) family acceptability (e.g., How do families seem to be responding to the technology? Did the technology impact rapport with parents? Are there other ways that you think technology could be useful in program delivery with families?).

Data Analysis Plan

Quantitative and qualitative data were analyzed using a convergence mixed-methods approach (Palinkas, Horwitz, Chamberlain, Hurlburt, & Landsverk, 2011). Quantitative data were analyzed using Fisher’s exact tests, chi-square tests, and independent samples t tests. Data for qualitative analyses included transcripts of audio recorded semi-structured interviews among providers. Thematic analysis was used to analyze all transcripts by the principal investigator and two other members of the research team. Derived codes from these transcripts were compared for consistency and overlap. Codes were grouped into themes. Differences in coding were explored and discussed until 95% agreement was reached across all transcripts. Periodic checks were made for intercoder agreement. Interrater discrepancies in coding were reviewed and discussed until 100% consensus was reached.

Results

SafeCare Delivery Time Demands

Independent samples t test were conducted to compare the average number of minutes for each activity and activity group (i.e., preparation, during session, and postsession activities) between SC-IU and SC-TA across all submitted time diaries (n = 400). Although the data are nested (i.e., time diaries within home visitor), the intraclass correlation coefficient was low (.11), suggesting that home visitors accounted for a trivial amount of variability in time reported across activities and so clustering was not accounted for in subsequent analyses.

Significant differences in the time reported were detected for most activities between home visitors delivering SC-TA and SC-IU. Significant differences were detected for most preparation activities, including the presession coaching call, t(87.3) = −4.64, p < .001, d = 0.732, organizing parent documents, t(332.4) = −3.53, p < .001, d = 0.367, and reviewing the session outline, t(374) = −3.58, p < .001, d = 0.375. Specifically, home visitors in the SC-TA condition reported spending less time on the coaching call, organizing parent documents, and reviewing the session outline but more time confirming the appointment. See Table 2 for average times reported for each activity, across condition.

Table 2.

Average Time in Minutes for Activity Duration Across Condition.

Activity Code	SC-TA (SD)	SC-IU (SD)	t(df)
Preparation
Presession coaching call	10.16 (7.65)	14.24 (1.81)	−4.64 (87.3)***
Organize assessment documents	9.83 (5.21)	10.20 (3.85)	−0.76 (301.2)
Organize parent documents	8.45 (5.50)	10.19 (3.81)	−3.53 (332.4)***
Test audio recorder	3.35 (4.87)	4.36 (5.37)	−0.93 (112)
Review session outline	12.14 (8.36)	15.75 (10.75)	−3.58 (374)***
During session
Initial greeting	12.51 (5.65)	9.56 (6.16)	4.67 (374)***
Problem-solving worksheet, if needed	17.92 (14.53)	30.83 (12.01)	−1.87 (16)^†
Baseline assessments, if applicable	21.42 (8.25)	29.74 (15.38)	−3.09 (49.3)**
Training portion, if applicable	28.16 (15.36)	39.49 (15.81)	−5.61 (265)***
End-of-module assessments, if applicable	16.86 (7.77)	34.44 (15.70)	−3.26 (9)*
Answer parent questions	13.51 (8.35)	7.78 (4.37)	8.21 (329.1)***
Postsession
Complete agency-specific notes	12.00 (6.15)	13.93 (6.09)	−2.92 (368)**
Organize and enter assessment data	8.93 (5.32)	9.82 (2.93)	−1.94 (320.5)^†
Upload session to SafeCare portal	16.27 (8.87)	23.13 (13.71)	−2.88 (55.6)**
Complete coaching calls	14.43 (12.14)	15.33 (5.18)	−0.61 (102.6)

Note. df = degrees of freedom; SC-TA = SafeCare Implementation with Technology-Assistance; SC-IU = SafeCare Implementation as Usual; SD = standard deviation.

^† p < .10. *p < .05. **p < .01. ***p < .001.

With regard to in-session activities, significant differences were found for almost all activities including the initial greeting, t(374) = 4.67, p < .001, d = 0.499, baseline assessments, t(49.3] = −3.09, p < .01, d = 0.674, training portion, t(265) = −5.61, p < .001, d = 0.727, end-of-module assessments, t(9) = −3.26, p < .05, d = 1.420, and answering parent questions, t(329.1) = 8.21, p <.001, d = 0.861. Home visitors delivering SC-TA reported they spent more time with the initial greeting and answering parent questions but spent less time conducting the baseline and end-of-module assessments and delivering the training portion.

A few significant differences were detected for postsession activities. Home visitors in the SC-TA condition reported spending significantly less time completing agency-specific notes, t(368) = −2.92, p < .01, d = 0.315, and uploading their session to the SafeCare Portal, t(55.6) = −2.88, p < .01, d = 0.593.

Considering all activities together by activity type (i.e., preparation, in-session, or postsession), SC-TA home visitors reported spending less time on preparation, t(389) = −4.52, p < .001, d = 0.458, and postsession, t(367.02) = −3.11, p < .01, d = 0.315, activities. Interestingly, no significant difference was detected for in-session activities overall. Further, across all activities involved in preparing, delivering, and following a session, home visitors in the SC-TA condition spent significantly less time than home visitors delivering SC-IU, t(397) = −3.10, p < .01, d = 0.323. Thus, home visitors delivering SC-TA were able to deliver SafeCare sessions in less time than home visitors delivering SC-IU. See Table 3 for overall differences by activity type.

Table 3.

Average Time in Minutes for Duration by Activity Types Across Condition.

Activity Type	SC-TA (SD)	SC-IU (SD)	t(df)
Preparation activities	9.75 (5.49)	12.37 (5.38)	−4.52(389)**
During session activities	17.85 (6.7)	19.17 (7.91)	−1.731 (379)
Postsession activities	11.94 (6.28)	13.62 (4.18)	−3.11 (367.02)*
Overall	13.23 (4.29)	14.66 (4.56)	−3.1 (397)*

Note. df = degrees of freedom; SC-TA = SafeCare Implementation with Technology-Assistance; SC-IU = SafeCare Implementation as Usual; SD = standard deviation.

*p < .01. **p < .001.

Provider Fidelity

Fidelity scores were collected throughout the participants’ duration in the study. Average scores for the first nine sessions (i.e., the sessions counting toward certification) were calculated and compared between the SC-TA and SC-IU groups. A total of 104 fidelity scores from 13 home visitors were observed in the SC-TA group; 65 scores from 9 home visitors were observed in the SC-IU group. Mean fidelity scores were 91.94 (standard deviation [SD] = 5.98) in the SC-IU group and 91.14 (SD = 8.81) in the SC-TA group. Because the intraclass correlation coefficient was low (0.19), clustering was not accounted for in the analysis of fidelity scores. Independent samples t tests found no difference between the groups in this regard, t(100.96) = −0.65, p = 0.52.

Provider Implementation Progress

Implementation status at Time 3 was obtained through implementation records. Group differences for implementation progress were assessed using Fisher’s exact test, and no statistically significant differences emerged. See Table 4 for a summary of the implementation progress of participants.

Table 4.

Certification Status Across Condition.

Certification Status	SC-TA, n (%)	SC-IU, n (%)
Completed workshop only	6 (31.6)	2 (18.2)
Started certification, SafeCare inactive^a	4 (21.1)	1 (9.1)
Started certification, SafeCare active^b	2 (10.5)	4 (36.4)
Certified during study duration	7 (36.8)	4 (36.4)

^aBegan certification and discontinued SafeCare delivery prior to achieving certification.

^bDid not achieve certification by the end of the study but still deliver SafeCare and work toward certification.

Feasibility/Acceptability of SC-TA

Semi-structured interview

All providers assigned to the technology-assisted group who had completed the implementation period with the technology were invited to participate in a qualitative interview. Interviews were completed with 71% of the providers in the SC-TA group who completed the study and returned to delivering SC-IU.

Technology usability

In regard to technology usability, the responses offered by SC-TA providers were overwhelmingly positive, with no providers reporting significant difficulty with technology use or delivery. All providers gave at least one example of how simple the technology was to use. For example, one participant stated, “[The] tablet makes everything easy. The tablet and the videos guided me through.”

The only noted challenge with the technology use was with the time it took for videos to load depending on the strength of the Internet services. These challenges were mostly noted in rural areas. One participant stated, “There were times where I was waiting for the videos to load and there was silence in the room…. You can use that time to discuss things with the family.”

Another stated, “We are in rural areas and the internet was slow, so that could be frustrating.”

Notably, because the implementation conditions were randomized at the provider level, there were providers within each agency utilizing both implementation approaches (SC-TA and SC-IU). One provider discussed that the SC-TA was so popular among providers that the SC-IU providers at the same agency acknowledged they would prefer the technology assistance, stating “The staff who didn’t use it [SC-TA] mentioned it would be nice to have. It’s what people were asking for. I’d be excited to use it more often.”

SafeCare delivery effort

In terms of SafeCare delivery, there was consistent agreement among the SC-TA providers that the SC-TA approach to SafeCare implementation offered many advantages for decreasing the overall effort for session planning and SafeCare delivery. For example, one participant stated, “[Technology] took care of some of the burden for getting ready [for sessions]. The technology guided us. I like not using ear buds so the technology is a shared experience between provider and client. It is a prompt for discussion. We would pause the videos to have discussions and for questions.”

Similarly, another participant described, “I didn’t have to prepare that much because everything was already being said. And I felt like the tablet, the videos, well, explained about each topic; it was really explained well. So whenever those were playing, it was good for the family and I didn’t have to explain anything else unless they asked.”

Progress through SafeCare implementation

SC-TA participants noted that having the technology assistance in the certification sessions for SafeCare positively impacted fidelity outcomes as well as their progress through certification. In regard to fidelity, all participants provided some comment about how the technology assisted them in maintaining focus in the session. For instance, one participant stated, “I think that it was helpful. I think that when I was using it when I first started SafeCare, it was nice to keep me in check…. Really helped me cover everything and keep me on track.”

Another participant indicated, “It made it easier for me to meet fidelity, when the computer is talking, there is less chance of me screwing up. I also used the tech time as a time to review session outline.”

Similarly, a participant stated, “The way the slides are formatted and the [technology-assistance] order is set up [helps you] stay on task and you go through the steps that you have to go through.…it kept me on track with what I needed to do. It was helpful and kept the session on track.”

In terms of progressing through implementation and achieving certification, most providers indicated that they would have had similar progression with or without the technology assistance but still noted that the technology assistance was helpful in making the implementation process a more positive experience. One participant stated, “I don’t know what I would of done without it, but I got through [certification] fast and very successfully and I think tech assisted was part.”

Another indicated, “When I stopped tech-assisted, some sessions were really difficult. It helped me a lot and I had to come up with some of my own resources to supplement once I stopped.”

Family acceptability

Providers noted that most parents were very accepting of the technology. Several providers offered examples of how the videos really assisted the parents in grasping the target SafeCare skills. For instance, one provider stated, “For safety, videos on YouTube, things on the news [provided through the technology-assistance] allowed for me and for families to see what could happen with hazards and if hazards weren’t removed. I still use them [the videos] as examples to show families what could happen and I think it has a bigger impact on them.”

Another indicated, “[The technology] is a prompt for discussion. We would pause the videos to have discussions and for questions. I was sitting there so I would pause if parent looked like they were struggling to get a concept…[The web-based program] gave a really clear explanation for the families what was expected.”

Additionally, one provider reported, “[The technology was] very good for people with trouble reading…. The audio and visual was very helpful.”

Providers did note that individual differences and preferences among parents matter and influence the overall acceptability of technology assistance. For instance, one provider noted, “[I worked with one parent] who was against TV and she did not like it. For parents who like TV a lot, they liked the videos and it really helped.”

Providers also noted that it is important to pay extra attention to rapport with technology assistance, stating, “It felt like the computer was doing everything for me. Sometimes I felt like I wasn’t making that relation with the family.”

Discussion

Limited research has examined how technology can improve the implementation of evidence-based programs targeting child maltreatment prevention and intervention. The purpose of this project was to examine the feasibility and influence of a technology-mediated implementation approach for an evidence-based program, SafeCare, which has been shown to reduce child maltreatment recidivism (Chaffin, Bard, et al., 2012; Gershater-Molko et al., 2002). Several hypotheses were proposed, with predictions of reduced time demands for SafeCare delivery, as well as improved SafeCare fidelity and implementation progress for providers utilizing the technology assistance. Hypotheses were partially supported.

Perhaps the most important finding from this exploratory study is the confirmed feasibility of the technology-assisted approach to SafeCare delivery. The technology was implemented across 17 new SafeCare agencies in eight states. There were notably few reported technological issues during the 2-year study, despite the fact that SC-TA providers had to bring computer tablets and Wi-Fi hot spots to each session in order to effectively access and deliver the web-based program, SafeCare Takes Care. It is anticipated that as Wi-Fi becomes more readily available and technological advances continue to improve exponentially, technology-mediated implementation and intervention will become increasingly feasible. Additionally, SafeCare Takes Care was very well received by families in most cases and extremely helpful in circumstances where parents had identified literacy and learning challenges. This was an unanticipated, but very important finding, as parents involved with child welfare and high-risk prevention programs commonly experience learning difficulties, and these difficulties have been tied to poor outcomes for family reunification (Booth, Booth, & McConnell, 2005; Booth, McConnell, & Booth, 2006).

Importantly, significant differences emerged between the two implementation conditions pertaining to the time demands involved with SafeCare preparation, session delivery, and postsession activities. Specifically, providers in the SC-TA condition reported spending significantly less time overall in preparation for and in follow-up to SafeCare sessions as well as reductions in assessment and parent training time during session delivery. While the reduction in delivery time demands may appear trivial when thinking about service delivery to one family, the public health impact of decreased time demands when delivering an evidence-based practice is significant. For instance, consistently briefer sessions would allow a provider to increase their caseload and serve more families at risk. Additionally, given that the technology can effectively deliver intervention components directly to the client, a technology-mediated approach may reduce the overall training burden and costs for new trainees. Furthermore, this approach may reduce the initial implementation barriers newly trained providers experience when implementing a new practice, which can enhance the likelihood of sustaining the new intervention. Clearly, more research is warranted to more fully explore the potential of technology-mediated approaches and their assistance in the implementation process.

There were two activities on which the SC-TA providers reported spending more time during the SafeCare sessions than SC-IU providers. Specifically, SC-TA providers spent more time on average with the initial greeting of their families and answering questions. Both of these findings are not surprising based on the qualitative data collected. SC-TA providers indicated that they paid significant attention to building rapport with families each week, given that the parent spent much of the session engaging with the technology. Consequently, these providers spent more time interacting with parents at the opening of SafeCare sessions and offered more time for questions related to both the content parents were exposed to in the web-based content and the skills practiced in partnership with the home visitor.

No differences emerged in the two implementation conditions for SafeCare provider fidelity or implementation progress. The lack of difference in fidelity between the two groups was somewhat surprising, given that the technology delivered several components of the SafeCare sessions that the SafeCare fidelity tool measures. Thus, for those components, fidelity was guaranteed as long as providers accessed SafeCare Takes Care during the client session. SafeCare provider fidelity in the implementation as usual group was extraordinarily high (on average, 91% fidelity was achieved), with very little variability in fidelity scores across all providers, which limited our ability to detect a significant difference. While no difference emerged in the quantitative data, the qualitative data suggested that SC-TA providers perceived that the technology made fidelity easier to achieve as well as reduced stress and increased confidence for providers when first delivering SafeCare to parents. This finding is commensurate with medical field research demonstrating that technology-based decision support systems improve physician confidence and ability to screen and identify health issues (Strayer et al., 2012).

The findings, or lack thereof, pertaining to provider implementation progress and certification are less surprising. There are numerous barriers documented in the current implementation science literature that impact overall implementation success (Aarons, 2004; Aarons & Palinkas, 2007; Schoenwald & Hoagwood, 2001). One evaluation of a statewide SafeCare roll out in a child welfare system found that only 23% of providers who completed free workshop training achieved SafeCare certification, mainly due to the lack of family referrals and policies establishing the exnovation of usual practice (Whitaker et al., 2012). Thus, further research that comprehensively examines implementation barriers, including agency culture and climate, service setting policies and infrastructure, and intervention implementation processes is warranted to ascertain the specific contribution of technological-based applications.

Limitations of the Current Study

This pilot study included a small sample and there was limited power to determine the differences on fidelity and certification between groups. Additionally, the technology used in this study, computer tablets, was provided to all SC-TA participants, which limits the generalizability of these results. A follow-up study focusing on whether the web-based program can be delivered with technology the providers have available to them in usual practice is warranted to maximize the feasibility of a technology-based approach. The assessment battery was limited in this pilot study, which restricted the information collected on other potential implementation barriers. The abbreviated study period was an additional limitation. Given that it often takes 6–9 months for providers to become certified and proficient with SafeCare, the 6-month study period does not provide information on feasibility for the technology-mediated approach once a provider is more experienced. Lastly, no family-level data were reported in this study. Family-level data collection is currently in progress that will allow us to examine if the technology-based SafeCare approach was equally effective on parental outcomes as traditional SafeCare. Given that there were no changes in the evidence-based content delivered to parents, just in the structure of delivery, there are no anticipated differences in the overall family outcomes. Interestingly, a recent review by Hall and Bierman (2015) suggests stronger effects in terms of both engaging parents and promoting positive outcomes for parents and children participating in blended intervention approaches that use technology along with professional support. Thus, it is possible that parental outcomes may improve to a greater extent with the technology assistance.

Strengths of the Current Study

Strengths of the study include the innovative focus of testing a technology-assisted approach to deliver an evidence-based child maltreatment intervention, for which limited prior research exists. Another strength is the data collection methods used in the study, which included self-report time diaries, independent fidelity-coded audio recordings of session delivery, and implementation record review for the 6-month study period. Additionally, the mixed methods approach allowed for the identification of themes related to session delivery and family acceptability that were not apparent from the quantitative data collected. Lastly, most providers trained to deliver SafeCare in this study had only achieved a bachelor’s degree, allowing for a conservative test of the technology-based implementation approach.

Conclusions and Future Directions

Technology is rapidly improving and there are numerous innovative technology-based applications that should be tested in order to increase our current understanding for how to effectively augment the implementation of evidence-based practices in the field of child maltreatment. The current study focused on one relatively small way that technology can augment implementation of an evidence-based practice. However, findings suggest that even a small change can have the potential for impact, especially when considering time demands for service delivery. To continue growing this line of research, it is imperative that child maltreatment researchers pursue collaborations with leaders in the technology field, to ensure that effective and sustainable technology-based approaches are a focus in developmental work (Kendall, Carper, Khanna, & Harris, 2015). For instance, in this study, the research team partnered with the ORI for the web-based technology platform that had been used in prior studies for delivery of a primary prevention parenting program (Baggett et al., 2010). This partnership saved development time and costs, which would have been insurmountable if the web-based platform had to be developed from scratch.

Future studies should explore how technology can more broadly impact evidence-based practice implementation. For instance, consider if the intervention workshop training could be delivered primarily to trainees through distance learning, and components of the intervention sessions could be delivered to the parent through technology without a provider present. This would substantially decrease training and program delivery costs, which could lead to increased uptake of effective practices by the child welfare workforce. Furthermore, as technology is increasingly available to service providers and parents served by child welfare and other prevention systems, the opportunity for real-time data collection will become increasingly possible, allowing for the continued assessment of evidence-based practices as they are implemented in real-world settings. Clearly, further technology-based research is warranted and critical to improve access to quality services for families involved with the child welfare system.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by NIMH 5 R21 MH098244-02 PI Self-Brown.

References

Aarons

G. A.

(2004). Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research, 6, 61–74.

Aarons

G. A.

Palinkas

L. A.

(2007). Implementation of evidence-based practice in child welfare: Service provider perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 34, 411–419. doi:10.1007/s10488-007-0121-3

Baggett

Davis

Feil

Sheeber

Landry

Carta

Leve

(2010). Technologies for expanding the reach of evidence-based interventions: Preliminary results for promoting social-emotional development in early childhood. Topics in Early Childhood Special Education, 29, 226–238.

Beidas

Edmunds

Marcus

Kendall

(2012). Training and consultation to promote implementation of an empirically supported treatment: A randomized trial. Psychiatric Services, 63, 660–665.

Booth

McConnell

(2005). The prevalence and outcomes of care proceedings involving parents with learning difficulties in the family courts. Journal of Applied Research in Intellectual Disabilities, 18, 7–17.

Booth

McConnell

Booth

(2006). Temporal discrimination and parents with learning difficulties in the child protection system. The British Journal of Social Work, 36, 997–1015.

Carta

J. J.

Lefever

J. B.

Bigelow

Borkowski

Warren

S. F.

(2013). Randomized trial of a cellular phone-enhanced home visitation parenting intervention. Pediatrics, 132, S167–S173.

Chaffin

Bard

Bigfoot

D. S.

Maher

E. J.

(2012). Is a structured, manualized, evidence-based treatment protocol culturally competent and equivalently effective among American Indian parents in child welfare? Child Maltreatment, 17, 242–252.

Chaffin

Bonner

B. L.

Hill

R. F.

(2001). Family preservation and family support programs: Child maltreatment outcomes across client risk levels and program types. Child Abuse & Neglect, 25, 1269–1289.

10.

Chaffin

Hecht

Bard

Silovsky

Beasley

(2012). A statewide trial of the SafeCare home-based services model with parents in child protective services. Pediatrics, 129, 509–515.

11.

Chaffin

Silovsky

J. F.

Funderburk

Valle

L. A.

Brestan

E. V.

Balachova

… Bonner

(2004). Parent-child interaction therapy with physically abusive parents: Efficacy for reducing future abuse reports. Journal of Consulting and Clinical Psychology, 72, 500–510.

12.

Cowart-Osborne

Jackson

Chege

Baker

Whitaker

Self-Brown

(2014). Technology-based innovations in child maltreatment prevention programs: Examples from SafeCare^® . Social Sciences, 3, 427–440.

13.

Coyle

Doherty

Matthews

Sharry

(2007). Computers in talk-based mental health interventions. Interacting with Computers, 19, 545–562.

14.

Gershater-Molko

R. M.

Lutzker

J. R.

Wesch

(2002). Using recidivism to evaluate Project SafeCare: Teaching bonding, safety, and health care skills to parents. Child Maltreatment, 7, 277–285.

15.

Guastaferro

Lutzker

Graham

Shanley

Whitaker

, (2012). SafeCare^®: Historical perspective and dynamic development of an evidence-based scaled-up model for the prevention of child maltreatment. Psychosocial Intervention, 21, 171–180.

16.

Hall

Bierman

(2015). Technology-assisted interventions for parents of young children: Emerging practices, current research, and future directions. Early Childhood Research Quarterly, 33, 21–32.

17.

Hurlburt

Nguyen

Reid

Webster-Stratton

Zhang

(2013). Efficacy of the Incredible Years group parent program with families in Head Start who self-reported a history of child maltreatment. Child Abuse & Neglect, 37, 531–543.

18.

Kendall

Carper

Khanna

Sue Harris

(2015). Computer technology and children’s mental health. Emerging Trends in the Social and Behavioral Sciences: An Interdisciplinary, Searchable, and Linkable Resource, 1–16. Retrieved from https://onlinelibrary-wiley-com.web.bisu.edu.cn/doi/10.1002/9781118900772.etrds0049/abstract;jsessionid=6EA530AC034C64D20999F58E5AA4C17A.f01t02.

19.

Lutzker

J. R.

Bigelow

K. M.

(2002). Reducing child maltreatment: A guidebook for parent services. New York, NY: Guilford.

20.

Palinkas

Aarons

Horwitz

Chamberlain

Hurlburt

Landsverk

(2011). Mixed method designs in implementation research. Administration and Policy in Mental Health and Mental Health Services Research, 38, 44–53.

21.

Prinz

Sanders

Shapiro

Whitaker

Lutzker

(2009). Population-based prevention of child maltreatment: The US Triple P system population trial. Prevention Science, 10, 1–12.

22.

Schoenwald

S. K.

Hoagwood

(2001). Effectiveness, transportability, and dissemination of interventions: what matters when? Psychiatric Services, 52, 1190–1197. doi:10.1176/appi.ps.52.9.1190

23.

Self-Brown

McFry

Montesanti

Edwards-Guara

Lutzker

Shanley

Whitaker

(2014). SafeCare: A prevention and intervention program for child neglect and physical abuse. In Reece

Hanson

Sargent

(Eds.), Treatment of child abuse: Common ground for mental health, medical and legal practitioners (2nd ed., pp. 50–58). Baltimore, MD: John Hopkins University Press.

24.

Silovsky

Bard

Chaffin

Hecht

Burris

Owora

… Lutzker

(2011). Prevention of child maltreatment in high risk rural families: A randomized clinical trial with child welfare outcomes. Children and Youth Services Review, 2033, 1435–1444.

25.

Strayer

Pelletier

Rollins

Heim

Ingersoll

Ritterband

Schorling

(2012). Evaluation of a screening and counseling tool for alcohol misuse: A Virginia Practice Support and Research Network (VaPSRN) trial. Journal of the American Board of Family Medicine, 25, 605–613.

26.

U.S. Department of Health and Human Services, Administration on Children, Youth, and Families, Children’s Bureau. (2015). Child maltreatment, 2013. Retrieved from http://www.acf.hhs.gov/programs/cb/research-data-technology/statistics-research/child-maltreatment

27.

Webster-Stratton

C. L.

(2014). Incredible Years^® parent and child programs for maltreating families. In Timmer

Urquiza

Timmer

Urquiza

(Eds.), Evidence-based approaches for the treatment of maltreated children: Considering core components and treatment effectiveness (pp. 81–104). New York, NY: Springer Science + Business Media. doi:10.1007/978-94-007-7404-9_6

28.

Whitaker

D. J.

Lutzker

J. R.

Self-Brown

Edwards

A. E.

(2008). Implementing an evidence-based program for the prevention of child maltreatment: The SafeCare program. Report on Emotional and Behavioral Disorders in Youth, 8, 55–62.

29.

Whitaker

D. J.

Ryan

K. A.

Wild

R. C.

Self-Brown

Lutzker

J. R.

Shanley

J. R.

… Hodges

A. E.

(2012). Initial implementation indicators from a statewide rollout of safecare within a child welfare system. Child Maltreatment, 17, 96–101.