Trauma-Informed Care in the Massachusetts Child Trauma Project

Implementing TIC in the Massachusetts Child Trauma Project

In Massachusetts, a multipronged TIC initiative and evaluation is currently underway, focused on improving the safety, permanency, and well-being of maltreated children, and that aligns closely with the aforementioned TIC implementation domains. The Massachusetts Child Trauma Project (MCTP) is a 5-year statewide systems-improvement initiative funded in 2011 by the Children’s Bureau (Administration for Children and Families and U.S. Department of Health and Human Services [USDHHS]). The goal of MCTP is to implement and sustain TIC within the CW and child mental health network (see Goldman Fraser et al., 2014). The key MCTP objectives are: (a) to improve identification and assessment of children exposed to complex trauma; (b) to build service provider capacity for the delivery of trauma-specific, evidence-based treatments (EBTs) in agencies serving CW involved children; (c) to increase linkages with and referrals of children to EBTs; and (d) to increase caregivers’ understanding about and sensitivity to child trauma.

MCTP focuses on three central activities: (1) training in CW; (2) EBT dissemination; and, (3) systems integration. The first set of activities most directly addresses the TIC implementation domains of training and workforce development and screening, assessment, and treatment services through basic and advanced child trauma trainings with CW staff and workshops for resource (foster) parents using the National Child Traumatic Stress Network (NCTSN) Child Welfare Training Toolkit and Caring for Children Who Have Experienced Trauma: A Workshop for Resource Parents (Grillo, Lott, & Foster Care Subcommittee of the Child Welfare Committee, NCTSN, 2010). Prior research has demonstrated the effectiveness of CW training curricula in improving TIC knowledge and practice (Conners-Burrow et al., 2013). As a sustainability strategy, CW staff and resource parents receive training to facilitate or cofacilitate curricula in the future.

MCTP’s second major activity addresses the TIC domain of assessment and treatment services through statewide dissemination of three trauma treatments with empirical support (e.g., Hodgdon, Kinniburgh, Gabowitz, Blaustein, & Spinazzola, 2013.; Mannarino, Cohen, Deblinger, Runyon, & Steer, 2012; Weiner, Schneider, & Lyons, 2009) via community-based mental health organizations: attachment, self-regulation, and competency (ARC; Blaustein & Kinniburgh, 2010), child–parent psychotherapy (CPP; Lieberman & Van Horn, 2005), and trauma-focused cognitive–behavioral therapy (TF-CBT; Cohen, Mannarino, & Deblinger, 2006). MCTP’s dissemination efforts employ comprehensive training and consultation in the form of a learning collaborative (LC) model, a promising approach to implementing empirically supported treatments in mental health (DeRosa, Amaya-Jackson, & Layne, 2013; Ebert, Amaya-Jackson, Markiewicz, & Fairbank, 2012: Institute for Healthcare Improvement, 2003; Nadeem, Olin, Hill, Hoagwood, & Horwitz, 2013). The LC model brings together mental health teams that comprise an administrator with authority to make policy and programmatic decisions (“senior leader” [SL]), clinical supervisors who monitor fidelity and provide support, clinicians who provide direct service, and a data manager. All members commit to a 1-year learning period, anchored by face-to-face learning sessions and intensive EBT consultation. MCTP also emphasizes the importance of leadership as a driver of effective implementation and sustainability, with a SL track focusing on EBT monitoring and continuous quality improvement (Fixsen, Naoom, Blasé, Friedman, & Wallace, 2005). Thus, it offers a platform for building governance and leadership in multiple TIC domains (e.g., policy, environment, and quality assurance).

A third MCTP component, Trauma-Informed Leadership Teams (TILTs), focuses on installing and supporting a structure for TIC systems integration at the community level. TILTs represent a promising means of actualizing the TIC domain of cross-sector collaboration (Conradi et al., 2011). They rely on leadership by CW management and participation by social workers, consumers, mental health providers, and other community service providers and stakeholders. The team process facilitates sharing of best practices across systems to increase awareness of the impact of trauma on children, creating consistency in TIC across service systems, addressing service gaps related to TIC, and reducing obstacles to accessing evidence-based services.

The Current Study

To illustrate key aspects of TIC developed under MCTP, we present data drawn from a mixed methods implementation and child outcome evaluation. Our earlier work (Goldman Fraser et al., 2014) describes the major components of MCTP and its implementation, including the activities, evaluation plan, preliminary outcomes, and lessons learned during the first year. We will stress the importance of widespread exposure to TIC concepts in both mental health and CW. We place particular emphasis on the importance of cross-system collaboration toward a shared language between systems. This study reports findings from the first year, evaluating TIC installation and improvements. Our two central research questions are (a) What improvements in TIC do TILTs, SLs, and clinicians report after 1 year of involvement in the project? and (b) Are EBTs disseminated through LCs associated with reductions in trauma symptoms and improvements in behavior among CW involved children at the first follow-up assessment, approximately 6 months into the treatment process? We hypothesized that MCTP’s TIC approach would have measurable benefits by 6 to 12 months. Our study approach emphasizes the collaborative efforts of the TIC system elements.

Method

Sample and Procedures

Data for this study were drawn from the larger evaluation of MCTP, a multisource, mixed method approach to assessing process and outcomes, as well as informing continuous quality improvement. The evaluation utilizes a combination of standardized and unstandardized surveys, key informant interviews, focus groups, CW records review, and child assessments. The design of this formative evaluation is primarily descriptive.

TILT sample and data collection

During the first year of implementation (October 2012 to September 2013), 16 of 17 area offices in the northern and western regions of the state developed TILTs. Most teams met monthly with a summer hiatus, and some held additional planning meetings. They had representation from CW workers, supervisors, and managers; community service providers (e.g., mental health workers, school staff, pediatricians, and court personnel); consumers (e.g., parents and youth); and resource parents. Evaluators conducted key informant interviews (October to December, 2014) with 32 TILT leaders from 14 (87.5%) of 16 teams. Two teams did not respond. Leaders held a variety of roles: director of areas (n = 2), area clinical managers (n = 10), area program managers (n = 6), managers (n = 4), supervisors (n = 6), social workers (n = 3), and an adoption worker. Most leaders (81.3%) were female.

SL sample and data collection

Twenty-seven SLs from 20 community mental health agencies participated in the LCs in the first year of implementation. The majority of SLs (70.4%; n = 19) were female. Nearly all were in a management role at their agency, and seven were responsible for overseeing multiple teams. In total, 40 teams participated in LCs. Agencies had one to four teams each, with half enrolling multiple teams. SLs participated in bimonthly online meetings with the MCTP project coordinator. Several agencies substituted supervisors. The project coordinator conducted key informant interviews with 25 SLs representing all 40 teams in 20 agencies 6 months after the LC began, in March 2013. In October 2012, leaders from 39 of the 40 teams participated in an online survey on trauma screening implementation, referral processes, and collaboration with CW. In October 2014, the nine agencies that continued with SL calls beyond the LC year (in a sustainability track) participated in an online questionnaire on TIC priorities.

Clinician sample and data collection

In 2012, prior to EBT training, clinicians and clinical supervisors (n = 190) completed an online survey assessing individual and agency TIC policies and practices, which they repeated 1 year later. Clinicians who completed the survey (n = 153; 80.5%) represented 20 mental health agencies and several disciplines: 45.4% (n = 69) psychology, 39.5% (n = 60) social work, 1.3% psychiatric nursing (n = 2), and 13.8% (n = 21) other. Less than half (46.1%) had up to 5 years of experience; 23.0% had 6–10 years; 19.7% had 11–20 years; 9.9% had 21–30 years; and 1.3% (n = 2) had 31 or more years. Over three quarters (77.1%; n = 118) had a master’s degree and 9.8% (n = 15) held a doctorate. Most were female (90.1%; n = 137). About 35.3% (n = 54) were 20–30 years old, 24.7% (n = 37) were 31–40 years, 20.7% (n = 32) were 41–50 years, and 4.0% (n = 6) were 61 years or older. All clinicians spoke English, 11.1% (n = 17) spoke Spanish, and 5.9% (n = 9) spoke other languages; and 79.1% (n = 121) of clinicians were White, 9.1% (n = 14) were Hispanic/Latino, 5.9% (n = 9) were African American, and 7.8% (n = 12) indicated other.

Child sample and data collection

Children in the first-year cohort (n = 326) and their parents, caregivers, or legal guardians who were enrolled in the evaluation received one of the three EBTs: 136 (57.63%) received TF-CBT, 108 (45.76%) received ARC, and 82 (34.75%) received CPP. Children’s mean age was 9.09 years (SD = 4.68; range = 0–18) at enrollment. Over half (56.0%; n = 183) of children were female and 44.0% (n = 143) were male. According to caregivers, just under one third (31.0%, n = 101) of children in the sample were Hispanic; the majority of children were White (73.3%, n = 239), 14.4% (n = 47) were African American, 1.5% (n = 5) were American Indian/Alaskan Native, 0.9% (n = 3) were Asian, and 11.0% (n = 36) did not respond (caregivers could select multiple race categories if applicable). Over one third (39.6%; n = 129) of children resided with their parents, 19.3% (n = 63) with other family members, 18.1% (n = 59) in regular foster care, 8.0% (n = 26) in treatment foster care, and 8.9% (n = 29) in another residence (residential treatment, shelter, and other). Less than half of children (44.8%; n = 146) were in state custody, 40.5% (n = 132) were in their parents’ custody, and 9.2% (n = 30) were in the custody of other family members at study entry. Two fifths of children (40.2%; n = 131) were on psychotropic medication. Clinicians identified eligible children, obtained consent from their caregivers, and enrolled children in the evaluation. Eligibility criteria included (a) referral for treatment related to trauma and (b) current open CW case. Clinicians administered assessments at baseline (i.e., at study enrollment, typically within the first two sessions), 6, 12, and 18 months, or until treatment was complete or the family terminated treatment. This study utilizes data from baseline and the first follow-up at 6 months or an earlier discharge, if the child left treatment for any reason prior to 6 months.

Measures

TILT measures

To assess functioning and implementation of TIC in TILTs, we reviewed meeting documentation (team membership, attendance, frequency and duration of meetings, and meeting content) and conducted key informant interviews on first-year implementation (recruitment, retention, activities, supports and challenges, and sustainability).

SL measures

SLs completed measures assessing trauma screening, referral, and outreach to CW; TIC priorities; and MCTP implementation. They also participated in key informant interviews.

Trauma screening, referral, and outreach to CW

We administered an online questionnaire to assess agency trauma screening and referral practices including (1) how CW referrals were triaged; (2) use of any type of trauma screening tool; (3) type of trauma screening tools; (4) process for assignment of referrals; (5) referral process for clients when an EBT clinician was not available; (6) clinician concerns about the flow of referrals and agency capacity to meet demand for EBTs; and, (7) outreach to local CW offices. We also assessed use of trauma screening during key informant interviews with SLs 6 months into the 1-year LC.

TIC priorities

SLs completed a brief online poll on their TIC priority domains. They ranked 7 items (lowest = 1; highest = 7): training and education in child trauma, availability and accessibility of trauma-focused treatment, parent/caregiver trauma, system integration/service coordination with child serving entities, screening and referral, understanding the impact of vicarious trauma on the workforce, and updating written policies.

MCTP implementation

Evaluators conducted semistructured key informant telephone interviews using prepared questions with SLs on their perceptions of the first 6 months of MCTP implementation. We focus here on the referral process and agency links with TILTs.

Clinician measure: Trauma-informed policy and practice

To evaluate TIC implementation among clinicians, we used the Trauma Informed System Change Instrument (TISCI; Richardson, Coryn, Henry, Black-Pond, & Unrau, 2012). The TISCI has 19 items answered on a 5-point scale (1 = not at all true; 5 = completely true). Higher scores represent more trauma-informed policies and practices. The three subscales, agency policy, agency practice, and individual practice have weighted scores (20–100). Agency policy refers to local, state, and federal policy that shape professional focus and action and assesses cooperation between and within agencies. Agency practice pertains to treatments or resources available to TIC and day-to-day agency practices that are trauma informed. Individual practice assesses the extent that individuals practice consistently in a trauma-informed manner. The TISCI’s internal consistency is adequate (Cronbach’s α = .53; Richardson et al., 2012).

Child measures

To evaluate child outcomes associated with receipt of any of the three EBTs, we used quantitative measures of posttraumatic stress symptoms and child behavior.

Posttraumatic stress symptoms in young children

To assess posttraumatic stress disorder (PTSD) symptoms in young children (aged 1–6), we used the Young Child PTSD Checklist (YCPC; Scheeringa, 2010). The YCPC is a 24-item caregiver report measure assessing traumatic events, trauma symptoms, and functional impairment in their children. Frequency of each symptom in the past 2 weeks is rated on a Likert-type scale (0 = not at all and 4 = everyday), with 19 items evaluating Diagnostic and Statistical Manual of Mental Disorders (Fourth Edition; DSM-IV) PTSD symptoms. Caregivers indicated how often each symptom in the child bothered them (startle response, intrusive memories, nightmares, physical distress, persistent negative emotions, withdrawal, clinginess, aggression, sleep problems, and lost skills). Item scores are summed with a “probable diagnosis” cutoff of >26. For functional impairment, caregivers indicate the extent to which symptoms “get in the way” of child functioning. Items are summed and the cutoff for functional impairment is >4. The instrument is relatively new, and no psychometric data were available, but it addressed relevant constructs in children as young as one 1 and was available at no cost, whereas the majority of existing measures that assess posttraumatic symptoms in young children are not appropriate for infants and toddlers and impose a financial burden.

Posttraumatic stress symptoms in older children

To assess trauma among older children (aged 7–18), we used the University of California, Los Angeles (UCLA) Posttraumatic Stress Disorder Reaction Index (UCLA PTSD-RI; Pynoos, Rodriguez, Steinberg, Stuber, & Frederick, 1998). The PTSD-RI is a 48-item semistructured interview assessing child exposure to 26 types of traumatic events and DSM-IV PTSD diagnostic criteria, including reexperiencing, avoidance/numbing, and arousal symptoms. The Parent Version was used for children under age 8, and both the Child and the Parent Versions were used for children aged 8–18 years. The PTSD-RI has good psychometric properties (Steinberg, Brymer, Decker, & Pynoos, 2004).

Child behavior

We used the Child Behavior Checklist (CBCL; Achenbach, 1992) for age 6–18 years (113 items) and 1.5–5 years (99 items). The CBCL is a standardized caregiver report measure of children’s emotional and behavioral problems. Subscales include internalizing (anxious, depressive and overcontrolled), externalizing (aggressive, hyperactive, noncompliant, and undercontrolled), and total problem behaviors. Items are rated on a 3-point Likert-type scale (0 = absent, 1 = occurs sometimes, and 2 = occurs often) for the past 6 months. Achenbach System of Empirically Based Assessment (ASEBA) calculates T-scores using a clinical cutoff of 63. The CBCL has high internal consistency (Cronbach’s α = .63–.97) and test–retest reliability (Pearson’s r = .80–.94; Thorvaldsen, 2005).

Data Analysis

Descriptive analysis of TILT data

Key informant interviews with TILT leaders were audio-recorded, hand-coded, and analyzed for themes. TILT meeting minutes were coded and analyzed thematically to generate descriptive statistics.

Descriptive analysis of SL and clinician data

To analyze data on SLs, we reviewed online survey results and analyzed key informant interviews on domains of TIC. We analyzed clinician data using t tests to assess changes in trauma-informed policies and practices prior to EBT training and 1 year later.

Multivariate analysis of child outcome data

We used a mixed effects approach to analyze data on the EBT’s impact on child outcomes (posttraumatic symptoms and problem behavior) by fitting a series of linear multilevel regression models to assess differences across study time points (baseline to first follow-up at 6 months or earlier discharge) separately for each outcome. To account for nonindependence of repeated observations on children, we allowed intercepts to vary across children, and to account for nesting of children with clinicians and clinicians within mental health agencies, we allowed intercepts to vary across clinicians and agencies. We used likelihood ratio tests to determine specification of random effects for each outcome. Covariates were child age, sex, number of trauma types, psychotropic medication, and custody (parent, other adult, state, and other). We excluded child race as a covariate due to missing data, as many caregivers chose not to provide this information. Controlling for child age and agency (each agency used a single EBT) largely accounted for variation by EBT, as we were not focused on differences in outcomes by EBT at this time, given the relatively small sample size. To address missingness, we fit multivariate models using maximum likelihood (ML) estimation, maximizing sample size by using all available data to compute ML estimates of model parameters. ML has nearly optimal statistical properties under the assumption of ignorability, allowing missingness of observations to depend on observed data (Allison, 2003).

Results

TILTs

TILTs engaged in a wide range of activities that reached a variety of audiences such as conducting a self-assessment; developing resources about child traumatic stress; organizing in-person trainings and group viewing of webinars about trauma for CW staff; developing webinars clarifying CW and mental health roles for the purpose of improved collaboration; conducting trainings about trauma for resource parents, local school systems, and community providers; creating a welcoming space for children and families; and holding wellness classes to address secondary stress among staff. Overall, findings from interviews with TILT leaders fell into five categories of TIC: (a) team membership recruitment and retention, (b) self-assessment, (c) communication and collaboration, (d) secondary traumatic stress, and (e) sustainability.

Team membership recruitment and retention

TILT leaders were able to recruit a wide range of professionals to join their teams, including mental health providers (n = 14), CW social workers (n = 14), supervisors (n = 13), managers (n = 13), alumni/youth consumers (n = 7), resource parents (n = 5), school personnel (n = 5), court personnel (n = 5), and others (e.g., court appointed special advocate, police, substance abuse provider, domestic violence representative, medical staff, immigrant center worker, child care staff, and state behavioral health representative). Member recruitment was one of the most challenging aspects of the first year. Several TILTs reported advantages of training in child trauma (e.g., Child Trauma Toolkit Training through MCTP; trauma certificate program at a local graduate school).

According to six leaders from different TILTs, a major success was developing and enhancing connections with the mental health community. All of the teams highlighted active participation by clinicians from local mental health agencies. Each of the teams sought participation from alumni consumers (youth, caregivers, and resource parents), agreeing that they were an important “voice,” but most leaders concurred that these were the most difficult roles to fill. Moreover, TILTs that were able to recruit consumers often found it challenging to retain them as members, citing the timing of the monthly meetings (during the work/school day) as a barrier. Some leaders reported that interest in TILTs was high in the beginning but waned over the course of the year. While the majority of teams reported 10–12 members early in the year, one team had only 3 to 4 core members by the year’s end. Leaders attributed low retention rates to factors including high rates of turnover among community professionals, increasing caseload demands, and turnover among CW agency staff due to organizational unrest—they felt that a high profile child death that year precipitated major changes in staff, leadership, policies, and practices within the agency, increased pressure on workers, and introduced competing commitments in the agency. One area program manager commented, “Some people’s attention needed to go other places – not because they didn’t think it was a good idea.” Another leader alluded to these shifts in organizational climate: “Staff dropped off, and I do attribute that to the changes going on in the agency at that time.” Nevertheless, several TILTs retained their members and developed a consistent core team. Most brought on new members as needed, though one team chose not to: “We used the group to do the group assessment and closed membership at that point. We desired to create a cohesive group.”

Self-assessment

TILT teams conducted an annual self-assessment using a tool designed to assist teams with taking stock of the systems and processes they currently have with respect to TIC. One TILT leader described the benefits: “The self-assessment was great because it gave us kind of a ‘strengths-and-needs’ look at our office.” Other leaders noted that self-assessment led to connections between CW workers and community providers.

Communication and collaboration

A number of TILTs developed strategies for improved communication about TIC and collaboration within the agency as well as with community providers. This work often began with finding a common language between CW workers and clinicians who were providing EBTs. According to one leader, “It was identified very early on that the language the Department speaks and the language the clinician speaks are completely not in the same world.” However, even within the CW agency, there was much work to be done in developing a shared understanding of child trauma. One leader highlighted the TILTs’ success in this endeavor, reporting that staff spoke much more often with each other and with families about TIC after their first year: “That’s a regular part of our vocabulary now … the trauma of parents and experience of children is more readily part of the conversation when talking to families.” In addition, TILTs focused their efforts in the first year on forming and strengthening partnerships with community providers in the school and juvenile justice systems.

Training and educational materials

A number of TILTs developed trainings and educational materials for professionals in the community (e.g., educators, court personnel, and resource parents). Each TILT took a unique approach to improving community responsiveness to child trauma. TILT efforts tended to focus on increasing awareness of the impact of trauma on children, improving professional’s sensitivity and responsiveness to trauma-impacted children, and providing information about their emotional needs. Leaders from six TILTs stated that resource parents were a priority in their work. They used a variety of methods for reaching out to them, including information packets and a “welcome book” for foster children and a lunch for social workers and resource parents to promote relationship building.

Referrals for child trauma treatment

Leaders cited an increase in appropriate EBT referrals as a direct result of TILT activities through word of mouth and staff training, practices that then spread throughout the CW agency. One leader explained, “It’s something that’s embedded in how we think … it’s always in our clinical discussions now.” The majority of leaders reported greater awareness of the impact of child trauma and the importance of EBTs, which facilitated referrals. In some instances, TILTs generated referrals during TILT meetings. However, they also identified circumstances in which referrals were problematic, such as a limited number of clinicians who work in certain regions or with young children.

Secondary traumatic stress

Another common theme raised by TILT leaders was secondary traumatic stress of staff and resource parents. As one TILT member said, the CW system itself is “systemic trauma.” Moreover, some TILT leaders felt that staff exposure to adversity had increased in recent years. One area clinical manager echoed this concern, emphasizing its impact on resource parents: “Secondary trauma for staff and resource parent’s trauma is a problem as well. We have a debriefing every other week now and we are reaching out individually if necessary.” Overall, TILTs developed many creative approaches to addressing secondary trauma, including wellness classes (e.g., mediation and yoga), support groups, a self-care committee, a “wellness room,” and a survey to screen workers for secondary trauma.

Sustainability

Leaders from the majority of teams expressed a desire to continue the TILT model of TIC. As one leader puts it, “I absolutely think it’s something worth sustaining!” Concurrently, they indicated a need for additional supports to sustain this mechanism of TIC beyond the life of MCTP. Nearly every leader identified the need for additional time to dedicate to trauma work on the TILT. Several leaders also felt that a stronger commitment from CW agency leadership would be critical to continuing their efforts, which would require higher prioritization of TILT work, additional allocation of resources for trauma-related work, and the development of internal policies to support statewide collaboration and the spread of TIC in CW practice across the state “to get everyone working with the same population on the same page.” Despite challenges, most leaders expressed a belief that TILTs serve an important function in CW and offered benefits to multiple stakeholders:

People feel like this is a valuable learning experience that they’ve gotten—that they’ve seen the benefits of being trauma-informed when they’re working with their clients; that the agencies we work with are seeing that they’re getting referrals from us; that the work that they’re doing with our clients is beneficial.

Many leaders emphasized that TILT members would need designated time to attend meetings and to engage in related work in the agency and community if their efforts were going to be successful in the long term, as they were already struggling with the existing demands of their jobs. They also expressed the need for a small amount of funding to develop and distribute TIC materials within the agency and community as well as to purchase food for monthly meetings.

Taken together, TILT leaders considered several achievements among their most successful in TIC implementation: the development and provision of trainings on TIC practices in CW and with outside systems partners, increased referrals for EBTs, and progress toward a shared language for child trauma among various providers. They also recognized value in MCTP support, including on-site visits from the project manager, free child trauma training and certification in EBTs, and events that brought TILTs together to share innovative practices. One challenge they encountered was the need for additional support, such as explicit guidance on how to develop and carry out TIC aims. However, two managers with expertise in child trauma expressed a preference to maintain freedom to pursue their own interests and to realize their vision of TIC. A significant challenge for TILTs was upheaval in the CW agency due to a highly publicized child death. Leaders felt that the ensuing turnover, high caseloads, shifting policies, and heightened stress led to problems maintaining participation in TILTs.

SLs in Mental Health LCs

Screening, referral, and outreach to DCF

SLs (n = 27) representing 44 of the 45 teams completed the survey on trauma screening, referral, and outreach to CW after the first month of the LC. When a SL represented more than one team, he or she responded to the survey separately for each team. The majority (64%; n = 28) reported using trauma screening in their agency. When their agency was unable to serve a family, 39% (n = 17) of SLs reported that they placed a family on their waitlist, 30% (n = 13) referred the client to another MCTP agency with EBT trained clinicians, 61% (n = 27) linked the client to a “trauma-informed” therapist in the organization, and 39% (n = 17) offered services from an EBT-trained clinician that was not a member of the LC team. During the first month of the LC, 49% (n = 22) of SLs reported that their agency had reached out to their CW area offices to build personal connections for receiving referrals.

MCTP implementation

Six months into the LC, 20 SLs representing 40 teams participated in a telephone interview gathering data on progress with MCTP implementation. The majority (80%; n = 16) reported that their agencies were actively involved in TILTs, 10% (n = 2) reported outreach to CW for membership with no response, and 10% (n = 2) reported they were not planning to be part of a TILT. Concerns about receiving referrals appeared to decrease from 34% (n = 15) at the beginning of the LC to 20% (n = 5) 6 months later, though there were fewer respondents at follow-up. We found a modest increase in agencies’ use of trauma screening at referral. According to online survey results at baseline, 64% (n = 28) of SLs reported that they were using some type of trauma screening. When asked about screening practices 6 months later, 70% (n = 14) of SLs reported use of screening at referral.

TIC priorities

In October 2014, SLs from nine agencies reported on their priorities in TIC. They were asked to make a forced choice on their highest and lowest priorities on elements of TIC. Of the seven priorities, 44% (n = 4) of the sample identified training and education in trauma and 33% (n = 3) identified available and accessible trauma-focused treatment services as the highest priorities. Approximately 66.6% (n = 6) rated “updating written procedures to be more trauma informed” as their lowest priority.

Clinicians in Mental Health LCs

Results from t tests examining changes in individual practices and agency policy and practices on the TISCI (Richardson et al., 2012) showed significant improvements at both the clinician and the agency levels. Mean scores for individual practices were higher at follow-up (M = 13.50, SD = 0.97) compared to baseline (M = 11.46, SD = 2.04), t(16) = 5.95, p = .001, as were mean scores for agency practices, M = 86.82, SD = 16.97 compared to M = 80.17, SD = 15.53, t(16) = 3.07, p = .007. Scores for agency policy were slightly higher at Time 2 (M = 26.01, SD = 6.20) versus Time 1 (M = 24.76, SD = 5.34), but the difference was not significant.

Children in Trauma-Focused EBTs

Table 1 presents the results of regression models examining change in child trauma symptoms and behavior from baseline to first follow-up. We describe specific findings subsequently.

OPEN IN VIEWER

Table 1.

Regression Models Examining Change in Trauma Symptomology and Child Behavior From Baseline to First Follow-up.

Measure		B	SE	df	p
UCLA PTSD Index Child Version	Re-experiencing	−3.25	0.48	170	<.001
	Avoidance/numbing	−2.06	0.63	166	.001
	Arousal	−1.07	0.44	165	.020
	Total severity	−6.56	1.30	161	<.001
UCLA PTSD Index Parent Version	Re-experiencing	−0.55	0.55	165	.320
	Avoidance/numbing	−1.58	0.58	161	.006
	Arousal	−0.52	0.45	165	.260
	Total severity	−2.82	1.26	160	.030
Young Child PTSD Checklist (YCPC)	Re-experiencing	−0.43	0.68	136	.520
	Avoidance/numbing	−0.73	0.78	130	.360
	Arousal	−1.39	0.74	133	.060
	Functional impairment	−2.42	0.93	135	.009
	Total severity	−2.44	1.96	126	.210
Child Behavior Checklist (CBCL)	Internalizing	−4.22	0.93	282	<.001
	Externalizing	−2.85	0.97	278	.003
	Total problems	−4.09	0.98	278	<.001

Note. n = 236. Parameter estimates (B) reflect change from T1 to T2, adjusting for child age, total trauma types, gender, psychotropic medication, custody status, clinician, and agency; SE = standard error, df = degrees of freedom, p = p value.

Posttraumatic stress symptoms in young children

Mean scores for young children on the YCPC were lower at first follow-up compared to baseline for reexperiencing (M = 5.58, SD = 4.99, n = 36 vs. M = 6.60, SD = 6.60, n = 136), avoidance/numbing (M = 4.43, SD = 4.73, n = 35 vs. M = 5.19, SD = 4.78, n = 129), arousal (M = 6.44, SD = 5.11, n = 36 vs. M = 7.55, SD = 4.86, n = 133), and functional impairment (M = 5.34, SD = 5.48, n = 35 vs. M = 8.05, SD = 6.16, n = 133). However, in multivariate analyses controlling for child age, sex, trauma types, psychotropic medication, and custody, EBT use predicted a significant reduction in posttraumatic symptoms only for functional impairment (B = −2.42, p = .009).

Posttraumatic symptoms in older children

Children aged 8–18 years self-reported reductions on all subscales of the UCLA PTSD index at first follow-up compared to baseline: reexperiencing (M = 6.13, SD = 4.43, n = 83 vs. M = 8.58, SD = 5.45, n = 165), avoidance/numbing (M = 8.96, SD = 5.37, n = 80 vs. M = 10.44, SD = 5.62, n = 156), arousal (M = 9.81, SD = 4.53, n = 80 vs. M = 10.85, SD = 4.15, n = 158), and total severity (M = 25.30, SD = 11.62, n = 77 vs. M = 30.22, SD = 12.77, n = 153). Mixed effects models showed that with control variables in the model, EBT participation significantly predicted reductions in symptoms for reexperiencing (B = −3.25, p < .001); avoidance/numbing (B = −2.06, p = .001); arousal (B = −1.07, p = .020); and total severity (B = −6.56, p < .001).

Caregivers also reported less posttraumatic symptomatology in children aged 7–18 for reexperiencing (M = 6.69, SD = 4.87, n = 80 vs. M = 7.48, SD = 4.98, n = 160), avoidance/numbing (M = 8.03, SD = 5.16, n = 80 vs. M = 9.78, SD = 5.38, n = 151), arousal (M = 9.10, SD = 3.68, n = 71 vs. M = 9.78, SD = 4.29, n = 160), and total severity (M = 24.30, SD = 11.92, n = 79 vs. M = 27.61, SD = 12.75, n = 160). In multivariate analyses controlling for other variables, there was a significant effect of EBT participation on avoidance/numbing (B = −1.58, p = .006) and total severity (B = −2.82, p = .030). Findings were not significant for caregiver report of reexperiencing or arousal symptoms.

Child behavior

Caregivers of children in the sample reported fewer behavior problems in their children from baseline to first follow-up, including internalizing (M = 60.30, SD = 8.28, n = 96 vs. M = 64.19, SD = 9.73, n = 269), externalizing (M = 60.89, SD = 10.67, n = 96 vs. M = 63.88, SD = 12.02, n = 269), and total problems (M = 61.72, SD = 9.58, n = 96 vs. M = 65.54, SD = 10.64, n = 269). Mixed effect models indicated significant effects of EBT participation on internalizing (B = −4.22, p < .001), externalizing (B = −2.85, p = .030), and total problems (B = −4.22, p < .001) holding all other variables in the model constant.

Discussion

Research demonstrates the deleterious impact of child abuse and neglect on children’s development over the lifespan, including increased risk for serious mental, behavioral, and physical problems (Child Welfare Information Gateway, 2013). When coupled with trauma experienced through family disruption and multiple experiences of separation and loss in the CW system, the toll on children’s well-being may be exacerbated (SAMHSA, 2011). Complex trauma experiences that adversely affect children and their caregivers also exert a negative impact on CW workers, clinicians, and other community service providers (Conrad & Kellar-Guenther, 2006). Large-scale, multipronged, systemic efforts are essential to creating a trauma-informed CW system that can effectively address these challenges, and yet there are few statewide initiatives, such as the MCTP. Unique to the MCTP approach is CW service delivery through TIC that targets multiple system components (e.g., CW staff and leadership, LCs in mental health agencies, relationships across community child and family service systems, and EBT for children and their families). We hypothesized that the MCTP approach to TIC dissemination would have substantial and sustainable benefits for children as well as for the larger system of care, which would be evident within the first 6 to 12 months of implementation. Our findings suggest that improvements across TIC domains can be realized across the CW system, mental health system, and other community partners. Among the many domains addressed in this study, we found empirical support for the effectiveness of MCTP’s strategies. Improvements are evident for improving awareness of the signs, symptoms, and impact of trauma on children, families, and service providers. These improvements are linked to the system’s use of evidence-based practices and treatments.

TIC in TILTs at 1 Year

Across the leadership, TILTs appear to be having a substantial impact on the system, including improving connections and communication between mental health providers and the CW system, creating a shared language between child mental health providers and CW workers, improving referrals for EBT, and developing numerous products and approaches for engaging community partners in TIC. One of the most striking accomplishments of TILTs is the relationship building that occurred across systems (e.g., CW, mental health, schools, and juvenile justice), particularly between mental health and CW. TILTs may have direct benefits for CW workers and clinicians by addressing secondary traumatic stress and numerous other creative efforts to address this critical issue. Barriers to progress include limited time due to overwhelming demands, recruitment and retention of consumers onto TILTs, and waning commitment over time, especially in the context of CW organizational challenges. Many of these issues are not unique to MCTP and have been reported in the literature previously (Aarons & Palinkas, 2007; Willging, Green, Gunderson, Chaffin, & Aarons, 2015) and thus likely constitute challenges that warrant consideration in any TIC initiative.

While many TILT leaders expressed that the knowledge base on trauma-focused EBTs had improved, others expressed that making referrals remained problematic, largely due to a lack of trained clinicians in their region or with a particular population (e.g., young children). The ongoing challenges of availability and accessibility of EBTs may undermine EBT reach and sustainability in mental health agencies (Foa, Gillihan, & Bryant, 2013). Still, our findings suggest that TILTs can be an effective TIC mechanism. Sustainability, however, will require the state to allocate adequate CW resources and to develop policies that prioritize TIC.

TIC in Mental Health LCs at 1 Year

SLs of LCs in mental health agencies identified training and education in trauma as well as available and accessible trauma-focused treatment for children and families, as their key priorities. Their emphasis on training and treatment underscores the need for continued, sustainable training programs in both TIC and trauma-focused EBTs. Interestingly, updating written procedures was the least prioritized activity, although arguably paramount to ensuring that shared, trauma-informed language and policies are both transparent and consistent at all levels of an agency (Health Resources and Services Administration, USDHHS, 2006). Not surprisingly, the clinicians in their agencies reported no significant improvements in trauma-informed agency policies 1 year into implementation. Given the multitude of priorities and pressures on mental health agencies, one might expect that updating written policies and procedures lagged behind other aspects of TIC, as updating these policies takes time, effort, and may not have the most obvious and immediate benefits to children and their families (i.e., compared to focusing resources and time on training and service delivery). On the other hand, accessibility of and referral to EBTs appear to have been effectively addressed, in part, by the activities of TILTs. The vast majority of SLs indicated that they were active on TILTs in their region. Although SLs originally identified referral issues as a significant concern, their concerns had dissipated 6 months later, due in part to stronger relationships, better communication, and shared language with CW staff and clinicians through the TILTs and LCs. Another important accomplishment that SLs identified was that the majority of mental health agencies had begun to integrate trauma screening at the beginning of the referral process. According to clinicians’ report on the TISCI, meaningful changes also occurred in trauma-informed practices at the individual level that they attributed to MCTP. This suggests that MCTP supported clinicians and agencies are becoming more equipped to engage in TIC practices. Future research might consider examining the impact of these changes on reducing providers’ secondary traumatic stress. As there were no significant changes in agencies’ TIC policies, a finding consistent with low prioritization by SLs, this suggests an area in need of attention by TIC interventions.

TIC and Children in EBT at 6 Months

Our findings suggest that the MCTP’s approach to TIC in general, and to trauma-informed EBT dissemination in particular, was associated with positive child outcomes across multiple symptom domains. From onset of an EBT to 6 months (or less) of treatment, children’s internalizing, externalizing, and total problem behaviors were significantly reduced. In addition, their posttraumatic stress symptoms had significantly decreased, with the most pronounced improvements among older children. Older children had fewer reexperiencing symptoms, avoidance and numbing symptoms, and reduced arousal symptoms. While caregivers endorsed reductions in avoidance/numbing symptoms and severity of posttraumatic symptoms overall in older children, our results suggest that they did not perceive improvements in their reexperiencing and arousal symptoms, whereas youth reported less symptomatology across all symptom clusters. Clearly, children between the ages of 8 and 18 felt they were less symptomatic than did their caregivers, although it is not clear why this was the case. Perhaps youth’s positive internal experiences were not easily detectable. Alternatively, caregivers may have had a lower threshold for children’s reexperiencing and arousal symptoms, which may appear especially disturbing to adults. We also did not account for EBT and EBT dosage, which may have differential effects for various symptom clusters. It is possible that EBTs for older children (TF-CBT and ARC) were not effective in these domains. This latter hypothesis is unlikely, however, as there are multiple randomized controlled trials that have demonstrated that TF-CBT, for example, is effective at reducing PTSD symptoms, including reexperiencing and arousal symptoms on youth and caregiver report measures (De Arellano et al., 2014).

Young children who received ARC, TF-CBT, or CPP also experienced significant reductions in functional impairment, and we found a strong trend for reduced arousal symptoms. However, caregivers did not report improvements in children’s reexperiencing symptoms, avoidance/numbing, or total severity. The inconsistent findings across symptom clusters may be related to measurement limitations, as the psychometric soundness of the YCPC is as yet unknown, and caregiver report has inherent biases. Insufficient dosage may also be a factor. For example, CPP is a longer term treatment model for which prior research shows benefit associated with an average of 32 sessions (Lieberman & Van Horn, 2005); thus, children may not have received a sufficient “dose” by 6 months or earlier discharge. Therefore, it will be important to examine these outcomes at later time points, when some children have had greater exposure to treatment and to examine the specific role of EBT dosage.

Conclusions and Limitations

A system of TIC appears to have taken hold through MCTP. Gains were made across a number of implementation domains in the first year, from modest improvements in child mental health at 6 months to enhanced practices in the larger system, including collaboration among CW, mental health, and community partners, and in trauma-informed individual and agency practices after 1 year. Saturation of TIC in both mental health and CW appears to improve cross-system collaboration, an essential element of a trauma-informed system. Ultimately, we found preliminary evidence of MCTP’s progress toward its four project goals: (a) improved identification and assessment of children exposed to complex trauma (e.g., TILT leaders’ report of identification of children exposed to trauma and referral to EBTs; SLs’ report of increased screening for child trauma at mental health agencies); (b) building service provider capacity for delivery of EBTs (e.g., EBT trained clinicians and associated positive child outcomes); (c) increased linkages with and referrals of children (e.g., TILT leaders’ report of EBT referrals and collaborative relationships with mental health); and (d) increased caregiver awareness and sensitivity about child trauma (e.g., significant increase in TIC mental health practices; TILT reports of TIC knowledge and practices). However, it is important to bear in mind several limitations of this study when interpreting our findings. First, we report on early findings, 1 year into a lengthier, large-scale statewide implementation effort. As successful implementation takes time and ongoing refinement, we will examine aspects of MCTP that we were not able to address here, such as differential effects of EBT models, the role of EBT dosage, changes in foster parents’ skills and knowledge after trauma-informed training, and variability in TIC at the system level over time (i.e., three different cohorts of TILTs and LCs phased in over 3 years).

Second, our findings on changes in TIC in the CW and mental health systems by the end of the first year were promising but modest. Leadership in mental health agencies reported a small increase in trauma screening; yet, changes in trauma-informed policies and practices in CW were not significant, and CW TILT leaders indicated that members’ engagement waned somewhat over time. This suggests that MCTP has room for improvement. Evaluation results from subsequent years of implementation may help clarify gaps in the MCTP approach and whether efforts require longer than 1 year to fully effect system change. In addition, the cost study that is now underway may begin to elucidate whether project expenditures (e.g., TILTs and EBTs) are reasonable and sustainable in light of TIC outcomes. Additional research on optimal methods for enhancing TIC linkages between the CW and mental health systems is needed.

Third, there are several study limitations related to the evaluation design. While we focused on evaluating TIC implementation and outcomes, we did not use nor could we identify an existing, validated tool to assess systemic improvements in TIC. Further, some of our evaluation methods (e.g., key informant interviews with TILTs and online surveys with SLs) do not provide objective empirical evidence of increased TIC. These shortcomings underscore the need for tools to assess collaboration across systems (e.g., CW and mental health) as well as means for evaluating whether or not a system is receiving the appropriate form and “dosage” of TIC to achieve the intended results. Developing such tools are of critical importance to rigorous evaluation of TIC efforts. An additional limitation was that although clinicians were provided with fidelity tools and adherence was emphasized during consultation and supervision, we did not collect data on fidelity (beyond program dosage) to assess model adherence. Research is still needed to clarify the role of EBT fidelity in achieving TIC system objectives. In addition, we did not have a precise measure of EBT referrals by CW workers to mental health agencies. The evaluation design was limited in part by the CW labor union, which restricts data collection with and by social workers, making it difficult to assess referrals in CW. Also, while we used an extensive mixed methods design, our study does not utilize an experimental design and we cannot attribute changes directly to MCTP. Lastly, the design of our study prevents us from determining empirically whether MCTP’s systemic efforts influence the majority of CW involved youth who do not receive EBTs. In the next phase of evaluation, we will employ a quasi-experimental design comparing placement stability for children in CW offices with and without TILTs, which may help us understand if MCTP’s systemic efforts have impacted CW involved youth regardless of whether or not they participated in an EBT.

Despite these limitations, findings on TIC in MCTP are encouraging. Mental health and CW systems have developed strong collaborations, practice changes are occurring at the clinician and agency level, and complexly traumatized children in EBTs provided by clinicians involved in LCs are demonstrating fewer behavior problems and experiencing fewer posttraumatic stress symptoms. TIC practice changes appear to lead to more cohesive, coordinated, and effective care with the potential to prevent the use of costly, higher level services for children in CW. These findings suggest that there may be even more robust outcomes at future time points and within other EBT cohorts after clinicians, supervisors, and agencies are fully trained and have experience implementing the treatment models, building on the bridges developed, and strengthened across child-serving systems within the culture of TIC.

The results of this study support the notion that a trauma-informed approach in CW necessitates coordination and changes at multiples levels of child and family serving systems that align across implementation domains (SAMHSA, 2014b). Policies that support and sustain evidence-based and evidence-informed TIC approaches such as MCTP are essential. Further, there is a need for policies and practices that address significant barriers to TIC, including secondary stress, burnout, and turnover in both CW and mental health (Aarons & Palinkas, 2007; Willging et al., 2015). A burgeoning interest in TIC among policy makers and practitioners, as well as initial efforts in cross-collaboration among federal agencies (e.g., Administration on Children, Youth, and Families [ACYF], Centers for Medicare and Medicaid Services [CMS], and SAMHSA), offers hope that there will be progress. However, additional efforts are needed to promote more widespread implementation of TIC in CW to enhance coordination among system components so that any single component does not limit the capacity of the larger system to promote the well-being of traumatized children and their families (Chadwick Trauma-Informed Systems Project, 2013), and to develop more effective tools for assessing TIC effectiveness with regard to achieving intended outcomes.