An International Effort to Develop a Fidelity Measure for Signs of Safety®

Abstract

Purpose:

This article describes how a worker fidelity assessment measure was developed for Signs of Safety®—an approach to assessing safety and risk in child protective services that is being implemented in the United States and over 11 other countries.

Methods:

We applied the Delphi Survey process with 70 experts from nine countries to identify key practice elements that could be assessed by supervisors. Then, 435 frontline staff were assessed by 285 supervisors from six countries.

Results:

Factor analyses of the 28 items yielded four distinct factors. These data were then used to refine the fidelity assessment.

Discussion:

The majority of supervisors reported that the assessment helped them to identify worker strengths and areas for refinement within the dimensions of Signs of Safety and overall child protective services practice. The value of developing similar tools for parents and workers was endorsed by the study participants.

Keywords

child abuse child welfare instrument development risk assessment

Fidelity to practices is an essential component of continuous quality improvement in child welfare. Fidelity can be defined as the extent to which the intervention delivered is true to the underlying principles on which it is based (Waltz, Addis, Koerner, & Jacobson, 1993). Measuring to what degree practices are implemented as intended is critical to determining whether an intervention improves outcomes for children and families. Fidelity assessment can help facilitate program change, improve service quality and effectiveness, help ensure accountability, and enable progress toward attaining the shared goals of providers, individuals, and families served within a system. In this article, we describe the development of a fidelity measurement tool for Signs of Safety—an approach to assessing safety and risk in child protective services.

The Importance of Measuring Practice Fidelity

While strategies such as the use of practice manuals can be utilized to enhance fidelity, they cannot guarantee effective implementation of an intervention (Forgatch, Patterson, & DeGarmo, 2005; Schoenwald & Henggeler, 2004). Instead, delivery of an intervention must be evaluated for fidelity to both intervention and application, so that one can explain if deviation from desired outcomes is a problem with the content or with the process of the intervention (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005; Forgatch et al., 2005). This information is an important part of an “improvement science” approach to studying, improving, and then reevaluating a program (Christie, Lemire, & Inkelas, 2017). As more research-supported practice models and interventions are being created in human services, model developers—and the organizations implementing the models—are identifying core aspects that need to be implemented well to achieve certain levels of quality outcomes. For example, research on the multisystemic therapy and wraparound services models suggests that client outcomes are more positive, when adherence to key aspects of the practice model is high:
Within children’s mental health services, this relationship [between treatment fidelity and improved client outcomes] has been found for multisystemic therapy (Henggeler et al., 2002), school mental health programmes (Greenberg et al., 1999), and many other models. Meanwhile, in wraparound, research has shown that the fidelity with which staff implement wraparound is associated with outcomes for the children they serve (Bruns, Suter, & Leverentz-Brady, 2006), and that system supports are indeed related to implementation fidelity as assessed. (Bruns, 2008, p. 9)
Thus, fidelity tool(s) become a critical first step in the development and deepening of any new human services approach. First, because child welfare, mental health, and other fields have increasingly asked practice innovators to document the effectiveness of their ideas through rigorous research, fidelity assessment provides at least an initial sense of what is being done differently in the program, so it is possible to say with some certainty whether the results or outcomes are connected to the change.

Second, recent developments in “implementation science” suggest the unfortunate frequency with which good ideas and practice innovations fail to make the “transport” from innovative pilots to large-scale field use, which means program funds and training efforts have therefore been wasted. Implementation researchers have begun creating different strategies that try to correct this failing—addressing everything from training, to enhancing ongoing coaching and conducting research on the importance of organizational culture and climate. For example, Fixsen discussed a more comprehensive approach to assessing fidelity in the context of implementation science that includes more than just worker adherence to a practice protocol (Fixsen et al., 2005). This definition has been expanded recently by Bruns, Suter, Force, and Burchard (2005) and others to conceptualize fidelity in three major ways:
Context fidelity refers to the structural aspects that encompass the framework for service delivery. Contextual aspects of fidelity might include professional development activities (e.g., pre- and in-service training), organizational climate, administrative practices, staff qualifications, supervisor/staff ratio, staff/client ratio, and inclusion/exclusion criteria. Contextual aspects of fidelity are different from compliance or competence in that they are not related to direct service delivery or the interaction between the practitioner and client.

Compliance fidelity refers to the extent to which the practitioner uses the core program components as intended by the developer. Compliance aspects of fidelity assess service delivery dosage and content as well as capture the interaction between the practitioner and client and might include frequency of service, intensity of service, service duration, and delivery of prescribed activities and/or curriculum.

Competence fidelity refers to practitioner skill level and the way in which the service is delivered. Competency aspects of fidelity assess the quality of service delivery and capture the interaction between practitioner and client. Competency aspects of fidelity might include type of strategies used by the practitioner (e.g., verbal, active, and modeling), reflective practice, relational work, trauma-informed work, and participant responsiveness (e.g., client report of their level of engagement or motivation; Rubiens, 2014, pp. 4–5; Sedlar, Bruns, Kerns, & Walker, 2015).

The act of constructing a fidelity assessment in human services should be more than just a technical exercise: It can be an organizational intervention in and of itself. Creation of a fidelity assessment tool requires that key stakeholders involved in an innovation come together in some way to form a community of practice. These tools can make the work of training, coaching, quality assessment, and overall implementation more effective.

Because of high staff turnover within child welfare, quality staff training, good supervisory coaching, and a strong, commonly understood practice culture are needed to support any practice initiative. Thus, attempts to implement an innovative practice need to distinguish between the effectiveness of the agency’s implementation capacity and the effectiveness of the intervention itself. Data on the organization fidelity and capacity to support practice are necessary to assure that practice has changed to align with the new intervention before that intervention can be tested for effects on children and families (Bellig et al., 2004). The systematic use of fidelity measures is a way of thinking about compliance and competence, detects variations in implementation, identifies major components of a program or approach, and refines interventions in the context of practice (Mowbray, Holter, Teague, & Bybee, 2003). Further, fidelity information can be used in feedback loops for practitioners and decision makers to monitor the status of implementation (Kaye & Osteen, 2011). Such monitoring can help to identify practice areas for targeted coaching and skill building, highlight areas of exemplary practice, and direct administrators toward organizational impediments to using the model with fidelity (Aarons, Sommerfeld, Hecht, Silovsky, & Chaffin, 2009; Fixsen et al., 2005).

Steps in Establishing Fidelity Measurement Tools

A number of scholars have proposed strategies for constructing fidelity assessments on new practice innovations including the major steps required in establishing fidelity criteria (e.g., Mowbray et al., 2003; Teague, Bond, & Drake, 1998):
define the model’s purpose as this will dictate the strategies used to develop the fidelity scaling,

assess the degree of model development, to help identify principles and ascertain whether a fidelity tool or something close already exists,

identify model dimensions and indicators of fidelity,

collect data to measure the indicators,

assess the psychometric properties of the tool.

While this seems straightforward, the actual application is challenging in many respects. For example, identifying the core principles and practices of a practice model that is multidimensional and used in many places can be difficult. Additionally, identification of appropriate data sources may be challenging. Certain structural kinds of data (e.g., presence of something in a written record, procedure codes, and frequency of home visits) are relatively easy to gather, while more process-oriented data (e.g., quality of client–staff interactions, quality of case consults, and organizational climate) can be harder to quantify. To construct fidelity assessments, it is necessary to pursue meaningful measures “not to just value what is easily measureable but also to seek to measure what is most valuable” (Decter & Pecora, 2011).

Creating an effective and efficient fidelity assessment is a process, not an event. It involves a community of practice coming together over a period of time and bringing together different skill sets to achieve the ultimate goals of furthering the legitimization of an approach people have excitement and faith in, while simultaneously refining and deepening the practice as the research process unfolds.

Purpose

Signs of Safety Is a Practice Model With No Formal Fidelity Assessment Tools

Signs of Safety® is a safety-organized and relationship-grounded approach to child welfare work created by Andrew Turnell and Steven Edwards. First developed in Western Australia, the approach has been refined over the past 23 years and has been adopted by Australia, Great Britain, Canada, Norway, the United States, and other countries. Signs of Safety is a strengths-based culturally responsive approach to determining safety and safety goals, risks, and strengths with families (Salveron et al., 2015; Turnell, 2011, Turnell & Murphy, 2017; Turnell & Edwards, 1997). Signs of Safety has four main objectives:
Providing strategies to foster effective working relationships between all parties (child, family, child welfare workers, community providers, etc.).

Minimizing workers bias through detailed “mapping” of what workers are worried about (harm and danger), what is going well (strengths and existing safety), and what needs to happen (goals and next steps).

Partnering with families and support people to build a safety planning process based on everyday behavioral changes that ensure child safety.

Supporting and ensuring that the family and its naturally connected networks enact the safety plan over time.

The Signs of Safety approach draws upon the participatory thinking and techniques of solution-focused (brief) therapy and strengths-based practices while always maintaining a rigorous focus on danger and harm (Turnell, 2007). The approach aims always to work in partnership with families and children to conduct risk assessments and produces safety plans together. It reduces danger by building from and deepening the strengths, existing success, resources, and networks that the family may already have. This is done by sustaining a rigorous focus on demonstrating and growing safety for children throughout the work.

Signs of Safety also seeks to create a more constructive culture around child protection organization and practice. Central to this are the three core framework principles:
establishing constructive working relationships and partnerships between professionals and family members and between professionals themselves

engaging in critical thinking and maintaining a position of inquiry by staying grounded in the everyday work of child protection practitioners (Turnell, 2017).

Studies of Signs of Safety have been completed in Australia, Canada Denmark, Finland, Netherlands, New Zealand, the United Kingdom, and the United States (see Turnell & Murphy, 2017; Bunn, 2013; Holmgård Sørensen, 2013; Lwin, Versanov, Cheung, Goodman, and Andrews, 2014; Salveron et al., 2015; Skrypek, Otteson, & Owen, 2010). This research has addressed training impact, worker interactions with families, decision-making and case planning behaviors, perceived “ease of practice,” child safety, foster care placements, staff retention and morale, and parent reports of practice changes.

Early outcome studies suggest that thorough and sustained implementation of the model is linked to lower rates of child maltreatment rereferral, fewer placements at birth, and fewer involuntary terminations of parental rights, as well as increased worker and supervisor job satisfaction, and reduced worker turnover (see Turnell & Edwards, 1997; Turnell & Murphy, 2017; Lwin et al., 2014; Munro, Turnell, & Murphy Child Protection Consulting, 2016; Skrypek et al., 2010). These preliminary findings need to be confirmed and refined through studies using random assignment or rigorous quasi-experimental designs and qualitative research. Signs of Safety is now being used widely, and initial evidence in the field supports its positive impact on families and agencies when fully implemented. Thus, it is important to understand what it takes to implement Signs of Safety with rigor for a sustained period of time.

The project reported in this article is a product of the Signs of Safety International Research Collaborative, an international work group of Signs of Safety practitioners, agencies, educators, and researchers. One of the collaborative’s first goals was to develop the supervisor completed fidelity measure reported in this article. This measure provides a tool for ensuring that the intervention is replicable and consistently adheres to the principles of Turnell and Edwards’ approach to child protection casework, trademarked as Signs of Safety. Concurrently, the collaborative is working on three additional Signs of Safety fidelity measures: a parent report, agency leader self-assessment, and organizational culture assessment aiming to create a suite of reliable and valid Signs of Safety fidelity measurement tools.

The tool’s purpose is to (1) capture the key practice elements of the Signs of Safety model and (2) distill these practice elements into a tool that can be systematically used across an organization to help assess and grow practice depth. In this article, we describe the development of the Signs of Safety Supervisor Practice Fidelity Assessment to be completed by supervisors and the results of the field testing process. However, organizational context and how it’s various dimensions affect supervisor behavior toward line staff also affects the supervisor’s ratings of line workers, no matter which model workers might be using in practice with their clients. Note that research on workforce has demonstrated repeatedly that supervisors have a profound impact on the working lives of direct practice child welfare workers.

Method

This section describes the steps taken to develop the Signs of Safety Supervisor Practice Fidelity Assessment and the results of the field testing process. All research pertaining to the tool was approved by the Human Subjects Review Committee at the author’s institution.

Step 1: Define the Purpose of the Assessment

As the Signs of Safety approach has been implemented by more jurisdictions, it is typical that agencies and managers ask questions about fidelity. For example, the child welfare leaders in Western Australia had been concerned that workers received the proper training, supervision, and support to learn and use the model, and they wanted to know whether workers were implementing it faithfully. At the same time, Sacramento County, CA, wanted to understand whether their early implementation efforts were improving outcomes and how to assess the quality of Signs of Safety practice. Thus, an international working group of members from Western Australia, Canada, and the United States was formed to address this measurement need.

Step 2: Assess Degree of Model Development

A review of the Signs of Safety literature was completed to identify how well the model principles had been articulated and to determine whether any fidelity tools already existed. This review helped to clarify model principles, identify potential practice dimensions, and confirm that no fidelity tool existed.

Step 3: Identify Model Indicators and Dimensions

In July 2012 through a Delphi Survey process, 126 Signs of Safety practitioners, trainers, and consultants from nine countries were invited to participate in the survey. The initial sample was later adjusted to 124 because two people did not receive the e-mail. A total of 70 of the 124 (56%) contributed narrative descriptions of (1) key elements of the Signs of Safety practice approach and (2) attributes and behaviors that best represent a skilled Signs of Safety practitioner, supervisor, or manager. People from nine countries participated in the survey (Australia, Canada, Denmark, Japan, Netherlands, New Zealand, Sweden, the United States, and the United Kingdom)—with a little over one third of the participants (36%) from the United States. Nearly, half of the participants (44%) identified themselves as practitioners and 39% were supervisors. Participants have been practicing, training, or consulting on Signs of Safety for an average of 4 years. Nearly, two thirds (64%) had between 2 and 5 years of experience and 29% had between 6 and 13 years of experience.

Responses were collated and distilled into a core list of attributes and behaviors relevant to Signs of Safety fidelity. The Delphi technique, developed by Dalkey and Helmer (1963), is a well-recognized method for achieving convergence of opinion concerning knowledge solicited from experts within certain topic areas or other key stakeholders. This technique has been applied in various fields such as program planning, needs assessment, policy determination, and resource utilization (Hsu & Sanford, 2007). This technique was used to identify items, develop scale anchors, and help design a range of possible scale structures. Note that while we identified context fidelity, compliance fidelity, and competence fidelity as the three ways of conceptualizing fidelity, during this initial work, we focused on creating a supervisor and parent completed fidelity assessment tools while other people associated with Signs of Safety have been more recently focusing on how to measure the context and compliance fidelity aspects in a large-scale study of Signs of Safety in New England (Munro et al., 2016).

Preliminary field test

The identified core list of items and a set of possible scale anchor designs were distributed back to the group of 126 international stakeholders in December 2012 to solicit feedback on the draft assessment tools. A variation of the Delphi method was again employed to obtain independent opinions about the content of the instrument, its face validity, and clarity of the text. A total of 126 participants were invited and 48 consented to participate in this supervisor survey. Of the 48 consented participants, 33 completed the survey, for a response rate of 69%. (Note that the overall response rate was 26% [33/126].) Seven countries participated in the survey. A little over one third (39%) of the participants were from the United States and 30% were from Canada. Thirty-seven percent of participants identified themselves as supervisors. Participants had been practicing Signs of Safety for an average of 6.1 years.

These second Delphi Survey respondents provided “expert” comments about the fidelity tool structure, scale anchors, and item wording. This phase helped the working group to identify the model dimensions and indicators, describe the source of data for each indicator, and develop operational definitions for the indicators or dimensions, including specifying the scale anchors for a rating scale. An 11-point anchor scale was initially selected because it complemented some of the clinical assessment scaling questions used in the Signs of Safety practice such as “on a scale of 0–10, how would you rate your ability to manage your son’s tantrums in public places?”

International prepilot

In March 2013, a second phase of preliminary testing for the Signs of Safety Supervisor Practice Fidelity Assessment began with a small international prepilot field test. The original tool included 27 items rated on an 11-point Likert-type scale ranging from 0 (never) to 10 (always). Supervisors were asked to select the rating that best fit the worker’s practice for that aspect of their work and were provided anchors for each item. For example, Item 1 asks supervisors to rate the worker’s capacity to listen and understand, with anchors “never listens to and understands the family” and “always listens to and understands the family.”

Supervisors made observations about additional aspects of worker practice gathered through supplemental questions. They also made comments about the fidelity tool structure and item wording. The Supervisor Practice Fidelity Assessment was refined during the summer of 2013 based on the supervisor feedback from this initial pilot.

Eleven supervisors throughout Canada, Western Australia, and the United States provided ratings for 19 workers. Supervisors reported the skill level of workers with respect to Signs of Safety on a scale that ranged from novice or beginning level to advanced level. Supervisors reported they had practiced Signs of Safety an average of 3.14 years (SD = 1.07) and that workers had practiced on average of 1.70 years (SD = 0.82).

Step 4: Collect a Large Sample of Data to Understand the Instrument Indicators and If They Clustered in Any Subgroups

Full-scale international pilot

The large-scale field test of the Signs of Safety Supervisor Practice Fidelity Assessment was conducted over a 14-month period with the collaboration of participating international jurisdictions and agencies. Partner organizations were recruited through a variety of means including in-person, e-mail contact, and written invitation. Once data sharing agreements were in place, participating agencies provided contact information for supervisors who fit the study eligibility criteria: The supervisor must have been currently supervising at least one caseworker relatively new to and one caseworker experienced in Signs of Safety and not have participated in Phase I of the study (the initial field test). Each supervisor was asked to rate two or more workers. Each assessment took approximately 20–30 min to complete.

Signs of Safety Supervisor Practice Fidelity Assessment

The Signs of Safety Supervisor Practice Fidelity Assessment (Supervisor Assessment) for the large-scale field test included 35 items rated on the same 11-point Likert-type scale. Observations about additional aspects of practice and fidelity tool and structure were gathered through two supplemental questions: What else did you see the worker do to achieve the key case outcomes? Having completed this survey, what do you think would be most helpful in supporting the worker to further strengthen their practice knowledge and skills? Information about supervisor and worker backgrounds and organization-level Signs of Safety practice were also ascertained.

Results

Participant Characteristics and Response Rate

A total of 326 supervisors across the jurisdictions were asked to complete the Supervisor Assessment on at least two different Signs of Safety workers. Two hundred and eighty-five supervisors (87%) participated. Four hundred and thirty-five forms were fully completed, averaging 1.5 forms (range 1–11) completed per supervisor. Based on the 570 expected possible forms, an overall response rate of 76% was achieved, with completion rates among all the pilot test agencies ranging from a low of 20% to a high of 100%. (Note that the overall response rate was 67% [435/652].)

Of the 435 workers rated, 173 (40%) were from Canada, 99 (23%) were from the United States, 87 (20%) were from Western Australia, 36 (8%) were from England, 22 (5%) were from Wales, and 18 (4%) were from the Netherlands. Supervisors reported that they had been supervising said workers an average of 2.60 (SD = 2.33) years and that workers had been practicing Signs of Safety an average of 2.35 (SD = 1.91) years (see Table 1). There was no missing quantitative data for any of the rating surveys, but some respondents did not answer all of the qualitative questions.

Table 1.
Supervisor and Worker Demographics.

Participant Mean Mode^a Range SD

Supervisors

Signs of Safety background

Years practicing Signs of Safety 3.46 3 0–12 2.11

Years organization has been practicing Signs of Safety 4.04 3 1–12 1.98

Years supervising the worker 2.60 1 0–15 2.33

Workers

Signs of Safety background

Years practicing Signs of Safety 2.35 1 0–15 1.91

Beginner 1.38 1 0–9 1.41

Early learner 1.64 1 0–6 1.03

Moderate 2.32 2 0–10 1.53

Substantial 3.05 2 0–8 1.74

Advanced 4.97 5 0–15 3.09

^aMost frequent response.

Step 5: Assess the Psychometric Properties

To screen for the most conceptually relevant items for inclusion in the final version of the tool, an exploratory factor analysis was conducted (principal component factoring followed by varimax rotation). A confirmatory factor analysis was then conducted using AMOS[version 18](Arbuckle, 2011). Internal consistency for each of the scales was examined using Cronbach’s α. Item response theory (IRT) was conducted to confirm the factor analysis as well as further establish psychometric properties of the proposed final Supervisor Assessment.

Factor analyses revealed a Supervisor Assessment tool containing 28 items yielding four factors or subscales. Factor 1, indicators of good child protective services (CPS) practice, includes core skills needed for CPS practice that are reinforced and taught in Signs of Safety training as well as tied to Signs of Safety model practices. The second through fourth factors Signs of Safety mapping, Signs of Safety family/support network engagement in safety planning, and Signs of Safety engagement with children include different Signs of Safety practice components that build upon and add to core CPS skills. A copy of the final Supervisor Assessment, along with factor loadings, is presented in Table 2.

Table 2.
Signs of Safety Supervisor Practice Fidelity Assessment.

Assessment Item Indicators of Good CPS Practice Signs of Safety Mapping Family/Support Network Engagement Engagement With Children

1. Listens to and understands the family. .85 .20 .14 .06

2. Treats the family as unique individuals. .83 .18 .22 .08

3. Did what they said they would do. .79 .10 .24 .05

4. Works with humility, accepting that their judgments may need to change. .78 .15 .19 .13

5. Is honest and direct with the family about what the Child Protection Service is worried about. .67 .24 .33 .04

6. Tells the family what they see is working well in the care and safety of the children. .69 .28 .31 .10

7. Involves the family in figuring out what needs to happen to get the child protection concerns addressed. .68 .30 .29 .25

8. Communicates clear judgments to the parents, children, network, and professionals about how safe the worker believes the children to be. .69 .26 .30 .19

9. Focuses on the specific caring behaviors the family must demonstrate so the children will be safe enough to close the case. .69 .31 .30 .25

10. Involves the children in the casework process. .56^a -.02 .25 .62

11. Helps children have their voice heard to influence decision-making. .61^a .08 .22 .61

12. Helps parents have their voice heard so their views influence decision-making. .75 .21 .25 .33

13. Asks family members whether what professionals are doing is helping. .59 .09 .32 .35

14. Uses Signs of Safety mapping to accurately reflect what is working well in the family. .34 .88 .27 .16

15. Uses Signs of Safety mapping to accurately reflect what we are worried about for the children. .24 .89 .25 .16

16. Uses Signs of Safety mapping to accurately reflect what needs to happen in the family. .25 .88 .26 .16

17. Uses the three houses, wizard or fairy, safety house, or other tool to bring out the voice of the child have their voices heard. .16 .58 .18 .54^a

18. Distinguishes between past harm and future dangers with family members and professionals. .38 .49 .44^a .24

19. Considers using the words and pictures to explain the child protection concerns to the child. .09 .48 .17 .72

20. Considers using the words and pictures with the parents to help them understand the child protection concerns. .09 .47 .18 .72

21. Uses a scaling question or process with family members and professionals to rate the safety of the child. .24 .48 .35 .35

22. Makes a distinction between strengths and acts of protection. .32 .35 .50 .34

23. Makes sure that the Signs of Safety mapping is understandable to family members. .25 .80 .32 .19

24. Creates safety plans with family members and professionals that reflect agreement about what will keep the child safe. .35 .30 .76 .18

25. Discusses with family members how proposed services will keep the child safe. .36 .26 .77 .18

26. Works with the parents to involve as many friends and family as possible to help keep the child safe. .38 .32 .74 .16

27. Checks with family members and the safety network to determine whether the safety plan is working. .37 .33 .75 .16

28. Works with family members and the safety network to refine the parts of the safety plan that are not working. .35 .29 .76 .19

Note. Items were originally rated on a scale ranging from never (0) to always (10). The updated assessment requires items be rated on a compressed scale ranging from never (0) to always (7).

^aFactor decisions for items 10, 11, 17, and 18 were made based on a group consensus based on clinical judgment of where the item would best fit.

A confirmatory factor analysis was then run and internal consistency for each of the subscales was examined using Cronbach’s α. The αs were high—between .89 and .95 (see Table 3). Overall, these analyses indicated that four distinct factors corresponding to the above defined subscales were underlying supervisor ratings of worker fidelity to Signs of Safety practice.

Table 3.
Descriptive Statistics for the Four Supervisor Assessment Factors.

Factor Number of Items M (SD) α

Indicators of good CPS practice 13 99.13 (19.88) .95

Signs of Safety mapping 5 29.31 (13.56) .95

Signs of Safety family/support network engagement 7 46.18 (15.33) .94

Signs of Safety engagement with children 3 13.17 (8.76) .89

Note. CPS = child protective services.

Using SPSS, an IRT analysis was completed to refine item selection and confirm the final proposed checklist structure. Analysis provided identical α reporting (.89 to .95) and nearly identical factor loadings. Supervisors tended toward providing positive ratings, however ratings did vary from low to high, with logits ranging from .14 to −2.36. No supervisor patterns in ratings were found, suggesting that a clear pattern in scores by rater was not present (i.e., no one was a more demanding vs. a more lenient overall rater). Analyses revealed little distinction and separation between response categories in the middle of the scale. Thus, moving forward the team decided to reduce the number of response rating categories from 11 to 7 and to provide definitions for each of the seven rating categories in order to facilitate tool completion.

Qualitative analysis

Qualitative data, consisting of open-ended responses to questions about tool structure and content, were reviewed by two researchers for identification of common themes. Atlas-TI [version 8.0] qualitative data analysis research software was used to code qualitative responses. Qualitative analysis yielded further support of the tool’s design and usefulness for practice improvement as described below.

The tool helped to identify areas of practice needing improvement

The majority of supervisors (81%) reported that the assessment tool helped them to identify weaknesses, both within dimensions of Signs of Safety practice (e.g., mapping, safety planning) and CPS practice (e.g., teaming, engagement skills). While most supervisors agreed that the tool helped to identify weaknesses, some did not see the utility in using the tool. Reasons given for lack of perceived utility included previous awareness of needs (i.e., the tool did not bring any new insight) and difficulty assessing weaknesses in worker practice without direct observation of practice. Some supervisors thought the tool is helpful for identifying areas of inexperience rather than areas of knowledge or skill limitation.

The tool helped to identify worker strengths

The majority of supervisors (83%) reported that the tool helped them to identify worker strengths. Supervisors reported the identification and reflection on strengths highlighted by the tool assisted mindfulness when assessing. Potential for the tool to help identify Signs of Safety practice champions who could act as leaders for the rest of their office was noted. Some supervisors used the assessment to clarify what they already knew, while others did not see the utility in using the tool. Reasons given include broadness of questions and the difficulty of giving quantitative ratings to qualitative practices.

Some gaps in assessing strengths and weaknesses in practice were noted

Supervisors stated that in some instances, ratings were difficult or not applicable based on worker experience (e.g., new worker that has not completed training in Signs of Safety) or case-specific considerations (e.g., how to properly assess engagement of young children).

The assessment provides a tool for reflection and planning

Thirty-one percent of supervisors reported that completing the assessment helped them reflect on the worker’s current practice strengths and areas for improvement specific to Signs of Safety. It also helped to identify what needs to happen from a supervisor’s standpoint to support workers to build their clinical skills.

Scaling of items was viewed as subjective

Some supervisors (8%) reported difficulties rating some items due to rating scale design. For example, a few worried that their rating of a particular score, such as 8, might not be equivalent to another respondent’s rating of an 8. It was also reported by supervisors that it was difficult to rate a worker’s performance on an item as a strength or as a weakness, when it might vary by practice context. Some questions are difficult to reflect accurately, for example, the use of Three Houses or other tools may not be appropriate for an older child, and therefore, the tool would not always be used.

Without direct observation, some practice items were difficult to answer

While aspects of Signs of Safety are modeled and practiced within supervision, it was difficult to rate the degree of practice fidelity in some areas when supervisors were not directly observing or listening to worker–family interactions. Some supervisors (10%) reported finding it difficult to rate a worker based on their assessment of a worker’s statements about what did or did not occur within the practice setting.

Research Limitations of the Full-Scale International Pilot

The jurisdictions included in this analysis are from various countries that have been implementing Signs of Safety for differing time periods (from months to several years) and have received varying amounts of training and follow-up coaching on protocol and procedures. Thus, each jurisdiction’s practice history with Signs of Safety is unique. While this contributes diversity for the field test, the extent to which this approach is being implemented consistently by caseworkers must be influenced by this and other “real-world” differences.

The Signs of Safety practice experts invited to participate in the Delphi Survey process were chosen because of their practice experience but in one sense “self-selected” themselves into the process by choosing whether or not to participate. In a similar vein, while the sample of workers who were assessed for the full-scale field test is not strictly representative of the entire group of Signs of Safety trained workers, they represent a diverse set of agencies and the ratings should provide some indications of the kinds of variance we might see in other studies of Signs of Safety worker fidelity.

Finally, direct observation of practice was not a routine expectation for supervisors prior to rating worker practice on the assessment tool. As part of the study, we encouraged selection of supervisors with direct observation experience of their staff, either in-person or using taped sessions. But not all of the supervisors had directly observed the workers they were rating. This is an important limitation of the study as supervisors need to have observed their staff—one way or another. These observations are important not only for the present study but also for home-based services and wraparound services (Author’s Own, 2003).

For these reasons, this study should be considered a pilot investigation. Additional research is needed to examine long-term use and benefit of the tool and to further validate the factor structure and revised tool proposed herein. Nonetheless, as the first attempt to create and validate a Signs of Safety assessment tool, it can provide valuable guidance to future research efforts. Such efforts should include a larger sample from an even broader range of jurisdictions where the model is being employed. Nevertheless, the current study implemented a careful approach to assessing fidelity to Signs of Safety practice that utilized Supervisor Assessments of CPS worker practice. The study also highlights the importance of utilizing a multistep process for creating and testing a fidelity instrument. Striving toward model fidelity while encouraging worker innovation is a critical step in maximizing the benefit of any standardized practice model.

Discussion and Application to Practice

To fully understand the effectiveness of Signs of Safety practice, additional studies of implementation fidelity are needed. Variations in program implementation can lead to a host of interpretive difficulties including inability to decipher whether an intervention is efficacious. While a child protection supervisor’s perspective of practice fidelity is important, findings suggest that a multidimensional assessment of fidelity is vital. Parent and child partnership is a distinguishing feature of Signs of Safety and hence an essential part of most any fidelity tool. Thus, combining the assessment of practice fidelity from the supervisor’s perspective with a worker self-assessment and feedback from parents and children served by the system would provide a more robust assessment of practice depth. This would also mitigate some of the concerns voiced by supervisors (e.g., subjectivity of items). It is a major challenge to find agencies with the resources and dedication to test a parent fidelity survey and a worker self-assessment. So while those two measures are slowly being pilot-tested, the supervisor-completed assessment is at least now available for use.

A unique aspect and strength of the present project is the comprehensive approach brought to the assessment of implementation fidelity of the Signs of Safety approach. Extensive reviews have been completed on strategies for constructing fidelity assessments on new practice innovations. Guided by research (e.g., Bond et al., 2000; Teague et al., 1998), the collaborative utilized a step-by-step approach to instrument construction, starting with defining the purpose of our tool, assessing the degree of model development, and identifying the model dimensions and indicators. The team then collected data and refined the tool using an iterative process prior to undertaking a large international full-scale pilot. This was followed by psychometric assessment of the tool’s properties to determine how best to refine the items and rating scale.

A second strength of the project concerns the attention given by the collaborative to implementing Signs of Safety fidelity within the context of CPS practice. Analyses revealed four factors within the tool: One measuring core aspects of good CPS practice and three additional clusters of skills related specifically to the model, namely, mapping, family engagement, and child engagement. This underscores the importance of measuring fundamental CPS practice skills, while highlighting the contribution of Signs of Safety practice skills above and beyond essential skills. Even with a relatively small number of items for practicality, we believe that the current set of items do address the four main practice objectives of Signs of Safety that were described in the introduction. But additional research to further validate the subscales is needed.

Signs of Safety Is a Practice Model With No Formal Fidelity Assessment Tools

Implementation science and other studies on organizational implementation (Arney et al., 2013; Fixsen et al., 2005; Rogers, 2003) demonstrate that implementation is a multiyear undertaking requiring whole organization engagement and commitment to the change process. Training practitioners in a practice model is a necessary condition, but training alone does not transform CPS delivery. Practice transformation requires comprehensive, integrated organizational endeavors that align leadership, organizational culture, learning, and meaningful measurement processes with the practice model. Any implementation must also be undertaken with a clear understanding that the practice model is a vehicle—not an end in itself. Rather, the goal is to improve the safety and well-being of vulnerable children. Any meaningful implementation, therefore, must involve an iterative learning process that deepens whole agency understanding of the practice model, along with organizational processes required to enable full use of the approach. Within the Signs of Safety literature, this understanding of implementation is represented in Figure 1.

Figure 1.
Signs of Safety implementation drivers.

The Signs of Safety approach seeks to create a culture of appreciative inquiry around frontline practice, where practitioners are supported to discuss good practice in order to establish a “what works” learning culture as the foundation of its organizational and practice transformation (Cooperrider & Whitney, 1999; Senge, 1990). While this “open” approach to learning and practice sounds straightforward, it can be a significant shift for staff and organizations that usually function within and perpetuate a risk averse culture of blame and fear. In contrast, Signs of Safety offers the potential of a deeper approach to working with children and families, with potentially greater satisfaction for staff and significantly improved outcomes.

Jurisdictions need to be mindful that they may not see significant outcome change systemwide for a few years, even with strong Signs of Safety fidelity. Early short-term outcomes may be spotted in key practice changes such as greater notation of parent strengths, more behaviorally specific case plans, and greater use of safety networks. Organizational alignment and model fidelity should be seen as mutually reinforcing processes.

If the jurisdictions who have higher fidelity to the model have better outcomes in critical areas such as lower child placement rates because of improved in-home safety plans, lower rereferral rates, and fewer involuntary terminations of parental rights, the benefit this understanding gives to the field and to other jurisdictions considering implementing Signs of Safety will be considerable.

Participant	Mean	Mode^a	Range	SD
Supervisors
Signs of Safety background
Years practicing Signs of Safety	3.46	3	0–12	2.11
Years organization has been practicing Signs of Safety	4.04	3	1–12	1.98
Years supervising the worker	2.60	1	0–15	2.33
Workers
Signs of Safety background
Years practicing Signs of Safety	2.35	1	0–15	1.91
Beginner	1.38	1	0–9	1.41
Early learner	1.64	1	0–6	1.03
Moderate	2.32	2	0–10	1.53
Substantial	3.05	2	0–8	1.74
Advanced	4.97	5	0–15	3.09

Assessment Item	Indicators of Good CPS Practice	Signs of Safety Mapping	Family/Support Network Engagement	Engagement With Children
1. Listens to and understands the family.	.85	.20	.14	.06
2. Treats the family as unique individuals.	.83	.18	.22	.08
3. Did what they said they would do.	.79	.10	.24	.05
4. Works with humility, accepting that their judgments may need to change.	.78	.15	.19	.13
5. Is honest and direct with the family about what the Child Protection Service is worried about.	.67	.24	.33	.04
6. Tells the family what they see is working well in the care and safety of the children.	.69	.28	.31	.10
7. Involves the family in figuring out what needs to happen to get the child protection concerns addressed.	.68	.30	.29	.25
8. Communicates clear judgments to the parents, children, network, and professionals about how safe the worker believes the children to be.	.69	.26	.30	.19
9. Focuses on the specific caring behaviors the family must demonstrate so the children will be safe enough to close the case.	.69	.31	.30	.25
10. Involves the children in the casework process.	.56^a	-.02	.25	.62
11. Helps children have their voice heard to influence decision-making.	.61^a	.08	.22	.61
12. Helps parents have their voice heard so their views influence decision-making.	.75	.21	.25	.33
13. Asks family members whether what professionals are doing is helping.	.59	.09	.32	.35
14. Uses Signs of Safety mapping to accurately reflect what is working well in the family.	.34	.88	.27	.16
15. Uses Signs of Safety mapping to accurately reflect what we are worried about for the children.	.24	.89	.25	.16
16. Uses Signs of Safety mapping to accurately reflect what needs to happen in the family.	.25	.88	.26	.16
17. Uses the three houses, wizard or fairy, safety house, or other tool to bring out the voice of the child have their voices heard.	.16	.58	.18	.54^a
18. Distinguishes between past harm and future dangers with family members and professionals.	.38	.49	.44^a	.24
19. Considers using the words and pictures to explain the child protection concerns to the child.	.09	.48	.17	.72
20. Considers using the words and pictures with the parents to help them understand the child protection concerns.	.09	.47	.18	.72
21. Uses a scaling question or process with family members and professionals to rate the safety of the child.	.24	.48	.35	.35
22. Makes a distinction between strengths and acts of protection.	.32	.35	.50	.34
23. Makes sure that the Signs of Safety mapping is understandable to family members.	.25	.80	.32	.19
24. Creates safety plans with family members and professionals that reflect agreement about what will keep the child safe.	.35	.30	.76	.18
25. Discusses with family members how proposed services will keep the child safe.	.36	.26	.77	.18
26. Works with the parents to involve as many friends and family as possible to help keep the child safe.	.38	.32	.74	.16
27. Checks with family members and the safety network to determine whether the safety plan is working.	.37	.33	.75	.16
28. Works with family members and the safety network to refine the parts of the safety plan that are not working.	.35	.29	.76	.19

Factor	Number of Items	M (SD)	α
Indicators of good CPS practice	13	99.13 (19.88)	.95
Signs of Safety mapping	5	29.31 (13.56)	.95
Signs of Safety family/support network engagement	7	46.18 (15.33)	.94
Signs of Safety engagement with children	3	13.17 (8.76)	.89

Footnotes

Authors' Note

Author Michael Caslor is now with Building Capacity Consulting Services, Manitoba, Canada.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Aarons

G. A.

Sommerfeld

Hecht

D. B.

Silovsky

J. F.

Chaffin

M. J.

(2009). The impact of evidence-based practice implementation and fidelity monitoring on staff turnover: Evidence for a protective effect. Journal of Consulting and Clinical Psychology, 77, 270–280.

Arbuckle

J. L.

(2011). Amos 20.0 user’s guide. Chicago, IL: IBM SPSS.

Arney

Lewig

Mildon

Shlonsky

Gibson

Bromfield

(2013). Spreading and implementing promising approaches in child and family services. In Arney

Scott

(Eds.), Working with vulnerable families (2nd ed., pp. 235–246). London, England: Cambridge University Press.

Bellig

A. J.

Borrelli

Resnick

Hecht

Minicucci

D. S.

Ory

,…Treatment Fidelity Workgroup of the NIH Behavior Change Consortium. (2004). Enhancing treatment fidelity in health behavior change Studies: Best practices and recommendations from the NIH behavior change consortium. Health Psychology, 23, 452–456.

Bond

G. R.

Evans

Salyers

M. P.

Willians

Kim

H. W.

(2000). Measurement fidelity in psychiatric rehabilitation. Mental Health Services Research, 2(2), 75–87.

Bruns

(2008). Measuring wraparound fidelity. In Bruns

E. J.

Walker

J. S.

(Eds.), The resource guide to wraparound (pp. 1–12). Portland, OR: National Wraparound Initiative, Research and Training Center for Family Support and Children’s Mental Health.

Bruns

E. J.

Suter

J. S.

Force

M. D.

Burchard

J. D.

(2005). Adherence to wraparound principles and association with outcomes. Journal of Child and Family Studies, 14, 521–534.

Bruns

E. J.

Suter

J. S.

Leverentz-Brady

(2006). Relations between program and system variables and fidelity to the wraparound process for children and families. Psychiatric Services, 57, 1586–1593.

Bunn

(2013). Signs of safety in England: An NSPCC commissioned report on the Signs of Safety model in child protection. London, England: NSPCC.

10.

Christie

C. A.

Lemire

Inkelas

(2017). Understanding the similarities and distinctions between improvement science and evaluation. New Directions for Evaluation, 153, 11–21.

11.

Cooperrider

D. L.

Whitney

(1999). Appreciative inquiry. Collaborating for change. San Francisco, CA: Berrett-Koehler.

12.

Dalkey

N. C.

Helmer

(1963). An experimental application of the Delphi method to the use of experts. Management Science, 9, 458–467.

13.

Decter

Pecora

P. J.

(2011). Preparing to create fidelity: Assessments for signs of safety. Document created for a work session on Signs of Safety implementation in North America, Chicago, IL.

14.

Fixsen

D. L.

Naoom

S. F.

Blase

K. B.

Friedman

R. M.

Wallace

(2005). Implementation research: A synthesis of the literature. Tampa: National Implementation Research Network, Louis de la Parte Florida Mental Health Institute, University of South Florida. Retrieved from http://nirn.fmhi.usf.edu/resources/publications/Monograph/index.cfm

15.

Forgatch

M. S.

Patterson

G. R.

DeGarmo

D. S.

(2005). Evaluating fidelity: Predictive validity for a measure of competent adherence to the Oregon model of parent management training. Behavior Therapy, 36, 3–13.

16.

Holmgård Sørensen

(2013). When parents network provides security for the child: an evaluation of 'security plans' in working with vulnerable families in Copenhagen commune. Københavns, Denmark: Socialforvaltnin-gen, Københavns Kommune.

17.

Hsu

C.-C.

Sanford

B. A.

(2007). The Delphi technique: Making sense of consensus. Research and Evaluation, 12, 1–8.

18.

Kaye

Osteen

(2011). Developing and validating measures for child welfare agencies to self-monitor fidelity to a child safety intervention. Children and Youth Services Review, 33, 2146–2151.

19.

Lwin

Versanov

Cheung

Goodman

Andrews

(2014). The use of mapping in child welfare investigations: A strength-based hybrid intervention. Child Care in Practice, 20, 81–97.

20.

Mowbray

C. T.

Holter

M. C.

Teague

G. B.

Bybee

(2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24, 315–340.

21.

Munro, Turnell, & Murphy Child Protection Consulting. (2016). Building sustainability and value for money. Appendix: Signs of Safety England Innovation Project—Second Wave. Perth, Australia: Author.

22.

Rogers

(2003). Diffusion of innovations (5th ed.). New York, NY: Free Press.

23.

Rubiens

(2014). Fidelity of preventive service models desk guide. New York: NYC Administration for Children’s Services.

24.

Salveron

Bromfield

Kirika

Simmons

Murphy

Turnell

(2015). Changing the way we do child protection: The implementation of Signs of Safety within the Western Australian Department for Child Protection and Family Support. Children and Youth Services Review, 48, 126–139.

25.

Schoenwald

S. K.

Henggeler

S. W.

(2004). A public health perspective on the transport of evidence based practices. Clinical Science and Practice, 11, 360–363.

26.

Sedlar

Bruns

E. J.

Kerns

S. E.

Walker

S. E.

(2015). Developing a quality assurance system for multiple evidence based practices in a statewide service improvement initiative. Administration and Policy in Mental Health and Mental Health Services Research. doi:10.1007/s10488-015-0663-8

27.

Senge

(1990). The fifth discipline: The art and practice of the learning organization. New York, NY: Doubleday.

28.

Skrypek

Otteson

Owen

(2010). Signs of safety in Minnesota: Early indicators of successful implementation in child protection agencies. Minneapolis, MN: Wilder Research. Retrieved from http://www.wilder.org/studies/Signs%20of%20Safety/925

29.

Teague

G. B.

Bond

G. R.

Drake

R. E.

(1998). Program fidelity and assertive community treatment: Development and use of a measure. American Journal of Orthopsychiatry, 68, 216–232.

30.

Turnell

(2007). Solution-focused brief therapy: Thinking and practicing beyond the therapy room. In Nelson

T. S.

Thomas

F. N.

(Eds.), Handbook of Solution-Focused Brief Therapy: Clinical Applications (pp. 295–314). Binghamton, NY: Haworth Press.

31.

Turnell

(2011) Preparing to Create Fidelity: Assessments for Signs of Safety. Document created for a work session on Signs of Safety implementation in North America, Chicago, IL.

32.

Turnell

Murphy

(2013). The Signs of Safety: A Comprehensive Briefing Paper v3.0. Retrieved from Signs of Safety www.signsofsafety.net/briefing-paper

33.

Turnell

Murphy

(2017). Signs of Safety Comprehensive Briefing Paper. Perth, Australia: Signs of Safety. Retrieved from https://www.signsofsafety.net/shop/

34.

Turnell

Edwards

(1997). Aspiring to partnership: The signs of safety approach to child protection. Child Abuse Review, 6, 179–190.

35.

Waltz

Addis

Koerner

Jacobson

N. S.

(1993). Testing the integrity of a psychotherapy protocol: Assessing therapist adherence and competence. Journal of Consulting and Clinical Psychology, 61, 620–630.