Thinking Differentially

Defining Differential Response

Our definition of DR stems from our lengthy experience in providing consultation to states, implementing the only national surveys on DR in CPS (Merkel-Holguin, Kaplan & Kwak, 2006; QIC-DR, 2009) in which core elements were defined and operationalized by state respondents. We also operate the National Quality Improvement Center on Differential Response in Child Protective Services, a 5-year grant where, with others, we are designing and conducting evaluation to rigorously study implementation, outcomes, and cost impact of DR, and we are building knowledge on the best practices on DR.

In our opinion, the article’s most unfortunate misunderstanding is what appears to be the authors’ characterization of DR as an evidence-based practice (EBP) or an intervention that is manualized. While some states refer to DR as a program or model, it has not been described as an EBP. Rather, DR is a method to restructure the CPS system to have multiple ways to respond to accepted (or the screened in) reports of child maltreatment. It is based on the notion that a system without choice cannot respond to all families and is poorly suited to respond to the various maltreatment or maltreatment risk-related conditions. In a DR-organized CPS system, there are a minimum of 2 response pathways—an investigation response (IR) pathway and an assessment response (AR) pathway. There is also the ability of agencies to reassign families between the pathways at any point in their involvement with child welfare, which the authors failed to note.

DR Implementation: Gaining Momentum and Increasing Variability

The authors accurately describe the phenomena of the increasing number of states implementing DR. The 2006 American Humane Association/Child Welfare League of America (AHA/CWLA) survey on DR (Merkel-Holguin et al., 2006), using a list of core elements of what constitutes a DR-organized CPS system, identified 15 states as implementers. In 2012, according to the Kempe Center for Prevention of Child Abuse and Neglect (2012), there are 16 states and the District of Columbia with statewide DR programs, 5 states with multiregion or multicounty DR programs, and 12 states actively developing a DR-organized CPS system, for a total of 31 states plus the District of Columbia engaged in this reform.

Despite the existence of the framework described above for DR, the utilization of the term DR has come to reflect a much broader set of CPS system reforms, so we are quite sympathetic to Hughes et al.’s (2013) view that what is meant by DR in practice is much less consistent from jurisdiction to jurisdiction. The authors credit various researchers, nonprofit organizations, and foundations as strategically promoting DR, resulting in the wide-scale adoption of this system reform. However, in the last few years more notable is the role of DR in Public Law 111-320, the Child Abuse and Prevention Treatment Act (CAPTA) Reauthorization Act of 2010. With an array of grants to states for programs to prevent, assess, investigate, prosecute, and treat child maltreatment, the CAPTA is the key federal legislation that guides the development and implementation of child protection policies at a state level. The 2010 Reauthorization amended the state plan requirements and assurances with governors having to certify that certain laws or statewide programs are in effect, including “differential response in triage procedures for the appropriate referral of a child not at risk of imminent harm to a community organization or voluntary preventive service.” It would seem to us that this reauthorization has already played and will continue to play a significant role in states’ adoption of DR.

The Administration for Children and Families’ Information Memorandum (ACYF-CB-IM-11-02; U.S. Department of Health and Human Services–Administration for Children and Families’ Information Memorandum, 2011) does not define DR, and as Hughes et al. observe as problematic, variability in DR implementation across states will be an ongoing feature of implementation. Given that federal laws provide states with significant latitude in defining and addressing child maltreatment whether with respect to DR and other aspects of CPS, the notion of a national CPS system with consistent standards and programs, which the authors alluded to in their criticism of DR variation, does not appear to be a national priority. One might also infer from the Hughes et al. (2013) article that there is consistency in states’ implementation of child maltreatment investigations. Nothing could be further from reality; states have different definitions, laws, and policies to guide investigations, and these differences are manifested in national data (U.S. Department of Health and Human Services–Administration on Children, Youth and Families, 2012). Finally, as Hughes et al. (2013) correctly point out, there are several approaches to DR that state and county agencies have adopted. In the discussion that follows, we consider those DR procedures and practices consistent with the definition offered above where cases are accepted as appropriate for CPS intervention, where an initial assignment of the referral is made to a response track as defined in state policy.

While there is variability across the states and counties implementing DR within the context of this more limited definition, from our knowledge there appears to be two consistent components of the AR pathway, which bear on the discussion in the article. The first is that, contrary to what Hughes et al. (2013) suggest, CPS systems typically have clear policies and procedures for assessing risk and safety for both IR and AR cases. In addition, while also apparently divergent from Hughes et al’s (2013) interpretation of interview data, it is a common practice for child welfare professionals to review prior reports of child maltreatment when determining pathway assignment (IR or AR) and how to approach the family.

The second component, consistent across the DR CPS system, is that the substantiation decision is eliminated for families receiving the assessment response in order to move away from a fault-finding orientation to one where the first response is to offer services. We agree with Hughes et al. (2013) that strength-based approaches are not dependent upon the implementation of DR reforms and represent practices that have been a part of traditional response systems. However, such approaches are by no means universal nationally, nor are they a clear choice for workers, their supervisors, or the systems that they operate within. Thus, while this is speculation, the values that accompany the AR pathway (reducing the adversarial nature, emphasizing service orientation) may actually be the impetus for permanently shifting the orientation of the entire CPS system in jurisdictions where it is implemented. In our experience, DR has propelled or created the need for the implementation of more uniform micro-child welfare practices, including enhanced screening protocols and screening decisions, safety organized practice, team decisions, and the use of family meetings as ways to build a more accountable and transparent child welfare system. And yes, many of these practices have also found their way into investigation response tracks complicating the evaluation picture for DR.

Under DR, workers report improved relationships with families and learn more about the family’s needs, and visit these families more than those receiving an investigation (Loman, Filonow, & Siegel, 2010). With regard to safety, experience with child protection cases leads us to think that one means to achieve child safety is through engagement of the family system coupled with a more comprehensive understanding of the family needs and strengths. Whether the conduct of an investigative fact-finding model enhances or diminishes the likelihood of successful engagement remains an open question from a research perspective. Nevertheless, for many CPS cases we believe such fact-focused processes can set up barriers to successful engagement, even for the most skilled worker. Previous studies on DR implementation provide a critical insight from family perspectives on investigations and assessments (Loman et al., 2010; Loman & Siegel, 2004), and the field could benefit from digesting what service recipients say, and how they experience the system and our responses to them. Finally, concern about the usefulness of substantiation is a persistent theme in the research literature (Drake, 1996; Fluke, 2009).

Comments on the Treatment of Research

The Hughes et al. (2013) article reflects some specific misunderstandings and mischaracterization of some of the research and evaluation that has taken place on DR, much of which has been addressed directly (cf. Loman & Siegel, 2004; Drake, 1996; Winokur and Gabel, 2013). We offer a few additional observations regarding the scientific content.

First, we agree that a systematic review of the research literature and a critical assessment of its quality would be a valuable addition to the field, and we appreciate the effort to attempt to do so by Hughes et al. (2013). Their article describes a qualitative interview methodology as well as a literature review used to support much of its content. For the interview aspect, what is known from the article is that the 28 respondents represent someone identified as qualified in DR in each state and, although this is not quite clear, that there is one respondent per state. It would be important to know whether respondents were from state-level agencies or county agencies, and similarly if from state or county administered systems, the longevity of DR implementation in the respondents’ states, or the specific roles in the agency of the participants (e.g., administrators, policy staff, training staff, etc.). It would also be useful to know the DR system structures these respondents represent, meaning whether their states implement the IR and AR pathways, and/or a pathway for screened out child maltreatment reports. Reading through the article, it is useful to recognize that, where research support is offered, many if not most of the assertions draw only on their literature review however the source of information is sometimes unclear. Thus, a more systematic and clear use of key informant data and presentation of the literature would be welcome.

The authors describe their criteria for the identification of the literature and efforts to identify both peer-reviewed and gray literature. There is no description of how the literature was analyzed, or if aspects of the literature were systematically classified, or how the relative quality of the research, where applicable, was assessed. We also invite readers of this volume to visit www.differentialresponseqic.org for a more comprehensive literature review (2011) that draws on the more than 50 research, evaluative, practice, and policy articles—from both a U.S. and an international context.

For “Findings #1 and 2” we are in agreement with the assertion that DR is not a unitary or entirely consistent practice. If DR were an EBP, then the concern would be important. Whether this consistency of design from jurisdiction to jurisdiction is relevant in terms of assessing the efficacy of DR in a specific jurisdiction is an open question in our view. However, given the recognized heterogeneous nature of CPS systems, successfully transplanting a fully formed let alone manualized DR initiative from one jurisdiction to another is unlikely. That said, the apparent lack of external validity begs the question about what is useful about our understanding of DR efficacy and effectiveness based on the research and evaluation efforts to date?

Like many other evaluations of system-level initiatives in child welfare (e.g., training effectiveness, risk and safety assessment, etc.) both adaptation and context are crucial considerations. Realist evaluation (Pawson & Tilley, 1997) provides one theoretical framework for considering how research and evaluation could be framed epistemologically to address practice, implementation, and outcome assessment. “In every case, the goal is to produce a tested theory about what works for whom in what circumstances and in what respects” (Pawson & Tilley, 2004, p 15). That is, even though the realist evaluation theoretical paradigm supports the concept that the evaluation of an initiative is valid only in its own context, the knowledge produced through the evaluations of these approaches may have applicability to similarly configured contexts. Ideally, jurisdictions will elect to carefully interpret findings and results from evaluations in other jurisdictions to assess applicability, and ensure that their own initiative is guided by strong implementation principles (e.g., Fixsen, Blase, Naoom, & Wallace, 2009), and rigorous evaluation.

In considering “Finding #2,” internal validity is an important concern, and as Hughes et al. (2013) suggest it is a limitation in virtually the entirety of the research literature in the child welfare field at the systems level. We certainly agree that authors of studies, including the authors of the article we are responding to, should be circumspect in identifying study limitations and the relative efficacy of findings in light of such limitations and threats. Nevertheless, the evaluations cited with random assignment (e.g., Loman et al., 2010; Loman & Siegel, 2004) represent an important advancement in being able to make clearer claims of efficacy, particularly in comparison with evaluations without the benefit of this stronger design.

Upcoming Research and Evaluation

It has been our privilege to participate in the Health and Human Services/Administration for Children and Families funded National Quality Improvement Center for Differential Response in Child Protective Services (QIC-DR) during the last 4 years. Among other activities, the QIC-DR has a focus on the evaluation of three state-level DR implementations; Colorado, Illinois, and Ohio. In addition, doctoral-level research grants have been provided through the QIC-DR to support five students to prepare DR-related dissertations. All of these research efforts are specifically aimed at improving the research and evaluation base for DR.

The key evaluation questions concern understanding the differences at each site between the IR and AR, medium-term child safety outcomes, and cost implications. A major focus is on understanding whether DR impacts the underlying factors that are hypothesized to tie together family engagement, services uptake, and safety-related outcomes. Common features of the evaluation are that each site implemented random trials based upon the determination of eligibility at CPS intake for the AR; and then these families were assigned randomly to either an AR or an IR. Families were followed for up to 18 months to assess safety outcomes, differences in service utilization, and to conduct cost comparison. Sample sizes in each group across sites are considerable, ranging from 1,000 to over 6,000 families per site. Sites made use of common instrumentation to support the cross-site evaluation including administrative data. Further, data on agency context, caseworker features, and other contextual features were obtained in order to understand some of the differences in model implementation within and between sites.

While it is too early to provide any results from the three local evaluations or the cross-site evaluation, we think that many of the findings will be instructive and reflect overall improvements in methodology and data analysis, drawing upon the experience of prior research and evaluation. Results are anticipated in the Fall 2013, and we hope the findings lead to further critical inquiry and discussion regarding DR efficacy and effectiveness.

We think that ongoing evaluation and research is a necessary part of ongoing DR development since there are many unresolved issues. Among the questions that we believe require further inquiry are:

Conclusion

The authors of this response do not necessarily share a common perspective regarding the efficacy of DR or the status of the research base. We do find ourselves in agreement with some of the criticisms by Hughes et al. (2013) regarding DR but think that the article mischaracterizes, overstates its criticisms, and fails to appropriately contextualize most of its “findings.” We are also aware of some important limitations within CPS systems and the current development of DR that are not addressed by the article. For example, implementation fidelity is highly problematic in many jurisdictions.

The notion that DR or any other systems level reform should not be implemented until the highest standard of research is conducted is unrealistic. The logical outcome of this perspective would have us suspending or eliminating many aspects of our approach to child protection including training programs, judicial review procedures, and perhaps the entirety of the CPS system. The California Evidence-Based Clearinghouse for Child Welfare currently recognizes five widely implemented child welfare initiatives, including DR, rated as having promising research evidence though none have achieved their highest-level research rating. We are not suggesting that this state is desirable but rather that the challenges are daunting. So, on behalf of children and families we must continue to up our “game” and we anticipate this will occur for DR.