Abstract
This article is a response to “Issues in Differential Response”, a review of the current evidence pertaining to differential response (DR) programs in child protective services (CPS). In my view, the Hughes, Rycus, Saunders-Adams, Hughes, and Hughes article suffers from several weaknesses. First, DR programs are critiqued as if they were a manualized intervention rather than a policy orientation, resulting in unhelpful conclusions. Second, the strength of existing evidence pertaining to child safety in DR is framed in an overly negative way. Finally, I attempt to place our understanding of DR in a broader perspective reflecting the continuing evolution of CPS and policies.
I would like to express my gratitude for the chance to comment on the recent article “Issues in Differential Response” (Hughes, Rycus, Saunders-Adams, Hughes, & Hughes, 2013). I found the article to be interesting and to bring up important issues. I will confine my comments to a few particular areas. First, I will take issue with the nature of the questions asked in the article, as I feel they might benefit from some reconceptualization. Second, I will directly discuss the issue of child safety, particularly as measured by rereport data. I do this because this issue is, in my view, the single most important part of any differential response (DR) evaluation—it is absolutely necessary that any new policy is not hurting children. Third, I will give my views of DR and DR research in the context of what I consider to be an ongoing shift in child welfare away from a criminal justice orientation and toward a public health orientation. From this perspective, DR might be seen as less of an isolated new intervention and more a logical, perhaps inevitable, result of larger changes in the conceptualization and execution of child welfare policy and practice. Finally, I will provide a brief perspective on the author’s assertion that proponents of DR are mischaracterizing current child protective services (CPS) practice as overly intrusive as a means of enhancing the attractiveness of DR. For the purpose of clarity, I will attempt to present my comments as a series of distinct points, beginning with a set of definitional issues.
DR is Not an Evidence Supported Treatment or an Evidence-Based Practice
The five questions at the core of this article are questions that strike me as those one might ask about an “empirically supported treatment” (EST). These questions are completely appropriate to the evaluation of a specific manualized program (e.g., Homebuilders, Triple P). These questions are not relevant to DR as a whole, because it is not a specific intervention. The authors produce a series of assertions, such as “Differential response programs do not adhere to a uniform, standardized practice model, nor are programs implemented consistently across sites.” These are not particularly relevant to understanding the utility of any state’s particular DR model. These statements are simply a clarification—DR is not a manualized EST, but a series of loosely similar policies being implemented in a number of different states and counties. So, if DR is not a specific intervention, what is it?
In my view, DR is most clearly described as a policy orientation. The single similarity between all DR programs is that there are always at least two “tracks,” usually including what we will term a traditional track and an alternative track, where more severe cases generally go to the traditional track. The simple fact is that DR programs in different states look very different from each other. These differences may include when the “track assignment” is made, on what criteria it is made, by whom it is made, how each track is staffed, what happens in each track, what provisions exist for “jumping” tracks, and how the presence of DR impacts other parts of the child welfare system.
It might be tempting, but misleading to say that the alternative track is differentiated by its preventative nature. However, all CPS interventions, without exception, are preventative—all have the sole purpose of reducing future harm to children. It might also be tempting to say that the tracks are discernible by the degree to which the interventions are voluntary. This also is wrong. Any child welfare case is eventually voluntary if the CPS worker fails to develop a sufficiently strong case to convince a judge to use coercive court action, irrespective of track status. Looking at it from this perspective, I found the first two questions asked in this article to be not very useful, except by way of a very general orientation to what DR is, confirming its already established chimeric nature. This variability between DR programs is not likely to change, even where there will to do so, as it partly results from a range of obdurate differences between locales in the form of state laws, administrative and practice structures, consent decrees, union contracts, geographic considerations, and so on.
DR Should Not be Studied as if it Were a Single Entity
While not a direct criticism of the Hughes et al. (2013) paper per se, I feel this is an area which deserves some clarification. Since DR is a general policy orientation which includes massive variability as it is implemented in different states, the whole idea that DR as a whole can be evaluated is, to my mind, an illusion. We cannot and will never be able to say “DR works” or “DR is safe” for the simple reason that there is no such thing as “DR” and the question is therefore not clear. To restate in more simple and forceful language—saying “DR is safe” is exactly the same as saying “any program calling itself DR (and perhaps meeting some vague criteria, like having at least two tracks) is safe” and is scientifically unsatisfying, to say the least. Conversely, the clear empirically determined failure of any given DR program (which has not yet occurred to my knowledge) might not tell us much about a different DR program in a different state.
Individual State or County Implementations of DR Policy Can Be Studied
So, how can science help us learn about DR? What can we say about it? As an example, Minnesota’s DR program is real, and can be described and studied, even if “DR” in general cannot. Studies of different cases of DR in different states can be performed, but that is about the best a conservative scientist should do, in my view. What we can say are things like “In state X, data suggest that the pilot DR program did not result in higher rates of child death compared to usual policy” or “Among the twelve DR programs reviewed, there is not strong evidence of increased or decreased cost among any program either short or long term.” Studies which span states might best be done from a multiple case study perspective—looking at each state individually and then coming to conclusions from what we learned about each program (e.g., Shusterman, Hollinshead, Fluke, & Yuan, 2005). If, at some future point, a convincing case can be made that two states have such very similar DR programs to allow them to be combined into a single, meaningfully homogeneous sampling frame, then that would be fine, but the burden is on the researcher to show that the states are implementing exactly, or almost exactly, the same intervention across a range of dimensions (intake procedures, staffing, practices, etc … ). Otherwise we risk introducing error to our findings and confusing ourselves.
While the authors seem to be in agreement with many of the above points, they also draw conclusions which, in my view, are seriously misguided. For example, they state on page 6 that “ … implementing jurisdictions often differed in significant ways from previously implemented reform efforts as well as among and within programs or agencies in a single jurisdiction. These differences were often celebrated rather than recognized as a fatal threat to effective model development.” Conclusions like these lead me to suspect that the authors may be attempting to understand DR as if it were a manualized EST. Programs like Incredible Years or Triple P are, in fact, manualized EST’s which are provided as service components and can be employed under an overall child welfare system. It is reasonable that programs such as these would be implemented and researched as the authors suggest. DR will probably never be implemented in a completely uniform way between states, as it must be fit to other currently existing policies (screening policies, judicial policies, etc … ) which will vary. This does not mean that each DR program has been remade from scratch as the authors suggest—“We did not find support in the DR literature that the model was shaped by empirical science and built incrementally based on the experiences of prior pilot sites.” This is simply untrue. Several states are implementing programs based on experiences of other states. For example, Ohio drew on the Minnesota experience (Loman, Filonow, & Siegel, 2010) and several states have scaled up their DR programs from smaller, internal pilots (e.g., Missouri).
Overall, my reaction is that Hughes and colleagues (2013) have evaluated DR from the wrong perspective and have drawn incorrect and unnecessarily harsh conclusions in answering their first two questions. I was consistently surprised by the negative tone of their conclusions and kept asking myself “compared to what?” My reading was that DR was being reviewed as if it were a manualized EST, and not as a new policy. If we step back and compare DR research to other research on changes in child welfare policy, it could arguably be seen as among the better empirically supported shifts in child welfare policy, not the worst. While I do not mean to suggest that poor prior research justifies poor future research, a sense of proportion is necessary. I believe a case could be made that DR research compares favorably, or at least equitably, to other domains, such as the impact of mandated reporting or any one of a number of foster care practices.
On the other hand, an interesting side issue is that DR research has often not supported by “pure research” grants per se, in the sense of federally awarded research projects carried out by universities, but has been paid for by states who are interested in improving their own practice, and implemented by nonacademic agencies such as the Institute for Applied Research. Such agencies have less pressure to publish in peer-reviewed journals than do tenure-track academic researchers. This has probably been a factor in the unfortunate situation obtaining currently, wherein much of the best quality work published on DR is not peer reviewed.
Safety
I will now explore the author’s conclusions in the area of child safety in some depth. Finding #3 is that “there is insufficient data to confirm the safety of children served in alternative tracks” (p. 500). I am guessing that the authors meant “There is insufficient data to show that the subpopulation of children who are served by alternative tracks are at least as safe as they would have been had they been served by a traditional service approach.” I found this part of the article to be disappointing. In my view, the studies reviewed, particularly the Minnesota and Ohio studies, were not clearly characterized and questionable and, again, overly pessimistic conclusions were drawn.
One traditional and very simple way to measure long-term child safety this is through rereports to CPS. We are told on page 10 that
even in those studies in which the subjects were randomly assigned to DR (experimental) and TR (control) groups, the study sample was not randomly selected: both study subgroups were drawn from an initial sample of lower risk families who had been identified as eligible for DR (citations in original include references to Loman et al. 2010 and Siegel & Loman, 2006 as specified in this document—Drake). Thus, one would expect to see comparable recurrence rates in the two comparison groups, regardless of whether they were served in the DR or TR track.
There is another question which could also be asked about safety, and that is “does the implementation of DR reduce safety among children not served in the alternative track.” Perhaps a somewhat counterintuitive proposition, this might happen, as the authors argue, should resources be drawn away from investigation cases under a DR system. To my knowledge, there is no evidence that this is happening, but it is something which should be monitored. I applaud the authors for bringing up this important issue.
A further issue correctly cited by the authors is that the evaluations done in Minnesota and Ohio essentially compared traditional services to an “alternative model plus money” condition, with families in the DR track eligible for up to $1,000 in material aid. Recent reanalysis of the data (Loman & Siegel, 2012) suggest that while the financial support was probably useful to families, the DR families probably still would have been safer than traditional services families even when the finding is absent. This finding should be considered preliminary and in my view, needs replication, preferably by an independent research team or through more studies on DR using conditions without the additional monetary support.
One thing that the authors seem to fail to understand is that the research they are criticizing is effectiveness research and not efficacy research. Efficacy research is a term used to refer to projects done in a tightly controlled setting, often with nonrepresentative samples, optimally trained, highly motivated and well-supported clinicians, and many other controls meant to emphasize intervention fidelity and exclude possible spurious factors. Effectiveness research generally emphasizes implementation of a program “on the ground” in real-world settings and realistic sampling, often with less control but more generalizability due to the less artificial context. Both types of research have strengths and weaknesses, primarily that efficacy research may better address internal validity issues whereas effectiveness research may provide better external validity. The work on DR has been of the “effectiveness” type, and should be critiqued as such. In concrete terms, this means that there should be some acknowledgment that the benefits gained from real-world trials generally come at the cost of the kind of control and precision available only in more purely academic trials. The question is not “are these designs perfect” but “are these designs good enough to give us useful information.”
Substantiation, Risk, and Recidivism
One very reasonable assumption which many people make is that substantiated cases are at high risk for recidivism compared to unsubstantiated cases. This assumption, unfortunately, appears to be false. Unsubstantiated cases have been found by many researchers to be at similar risk of rereport as substantiated cases (Hussey et al., 2005; Kohl, Jonson-Reid, & Drake, 2009). If we extend this logic slightly, we might question the author’s assertion that “investigation” track and “alternative response” tracks will necessarily have dramatically different baseline risks for rereport.
General Trends in Public Child Welfare and DR: Where Are We and Where Are We Going?
In my view, it is important to place any review of research on DR in a broader policy context. Hughes et al. (2013) did address this issue, to their credit, but I would take their discussion further, as I believe that DR research must be framed by an understanding of broader changes in the field. I believe that DR, or something like it, is almost an inevitable result of current general trends in public child welfare. If I am correct in this view, understanding implementation and outcomes of DR programs would be among the most pressing priorities in child welfare research.
There is evidence that future directions and policy innovations in child welfare may be increasingly focused on child well-being, prevention, community engagement, and other forms of outreach. The Adoption and Safe Families Act was notable in forwarding child well-being as one of the four core goals, in addition to safety, permanence, and accountability (Urban Institute, 2009). In short, it may be that public child welfare is gradually moving from a more criminal justice paradigm to a more public health paradigm. This has been especially evident internationally (Higgins & Katz, 2008; Scott, 2009), with calls for movement to a pure public health perspective far more radical than those contemplated in the United States (O’Donnell, Scott, & Stanley, 2008). These approaches share a primary emphasis upon child well-being (e.g., Scottish Government, 2012).
Public child welfare can be seen as a criminal justice or law enforcement entity or it can be seen as more of a preventative or public health entity. In fact, it is demonstrably a combination of both (Drake & Jonson-Reid, 2000). How these paradigms conflict with or complement each other is central both to our understanding of child welfare policy in general and to DR in particular. In particular, the concepts of safety and risk are important, as are the concepts of “investigation” and “assessment.”
What Are “Investigations” and Why Do CPS Workers Do Them?
Public child welfare is always preventative. Actions are always aimed at ensuring child safety in the future. Public child welfare never engages in punishment. In the instances when punishment for child maltreatment by a caregiver occurs, it is handled by the criminal justice system. Child welfare workers do perform investigations, however. These investigations can be of value in (or can damage) subsequent criminal investigations but their main purpose is to document past harm for the child welfare agency and the family courts. Such documentation can help the worker and future workers, understand the past harms the child may have experienced. This can support risk assessment and, very importantly, can provide the family court system with grounds for coercive action (e.g., immediate removal, later termination of parental rights) should such action prove necessary. We therefore have something of a paradox. Child welfare workers, who have only future-oriented goals (protection of children) routinely engage in past-oriented investigations which can be critical in how the case moves forward. In any case, investigations are focused on the past and are commonly, if not exclusively, focused on the particular events which precipitated the referral.
What Are “Risk Assessments” or “Safety Assessments” and Why Do CPS Workers Do Them?
Let us set investigations aside for a moment and think of something quite different—a risk or safety assessment. As far as I can tell, these terms seem to be often used interchangeably, although sometimes “safety” might have a more immediate connotation. For example, both the Ohio and Minnesota programs used the Structured Decision Making Family Risk Assessment tool. This is not an investigation of a specific prior event, it is an attempt to empirically categorize a family’s risk of hurting their children in the future across a broad spectrum of types of maltreatment. Risk or safety assessments, therefore, are aligned with the core mission of CPS—protecting children from future harm. The problem is that researchers and social scientists have not done a great job of creating particularly good risk assessment tools.
The authors consistently use phrases like “presuming that families are accurately assigned to tracks based on a valid assessment of risk level.” The unfortunate fact is that we do not have risk assessment tools which perform all that well. The Structured Decision Making Family Risk Assessment Instrument tool used in the Ohio and Minnesota studies could be termed state of the art, but the state of the art is not that strong (Bauman, Law, Sheets, Reid, & Graham, 2005; Broadhurst, Hall, Wastell, White, & Pithouse, 2010; Peters & Barlow, 2003; Taylor, Baldwin, & Spencer, 2008).
How is DR Different From Traditional Services With Regard to Investigation/Assessment?
This is a key issue raised by the authors. One of the primary differences between DR and traditional services is that DR does not do an “investigation.” This does not mean that DR is unconcerned with risk or safety. A necessary part of either a DR track or traditional services must be a comprehensive risk assessment. To the degree that understanding the event which precipitated the report is necessary to understand risks to the child, something like a traditional “investigation” should be a necessary part of DR practice. Understanding the degree to which DR cases involve necessary understanding of prior events is important and requires more study. Conversely, an investigation is not a risk assessment and investigations without a risk assessment are not sound practice either, no matter what track a child is in. For example, it is known that cross-type recidivism is the norm for nonneglect cases (e.g., Jonson-Reid, Drake, Chung, & Way, 2003), so any process which fails to assess for other types of maltreatment risk beyond that specified in the allegation (especially neglect) will be severely inadequate. Put simply, the question “was an investigation done” and “was the child’s risk assessed” are fundamentally different questions. Often, Hughes et al. (2013) seem to recognize this to their credit. I personally advocate for even more clearly separated treatments of these issues, however in current and future research.
In summary, I take it given that risk assessments should be performed in all cases. I also agree very much with the authors that risk assessments must include gaining an understanding of the event which generated the referral (p. 503). While this would constitute something of an “investigation,” it is a very different question if a formal investigation should be performed in all cases. While I believe understanding the precipitating event is always necessary, an in-depth and formal investigation emphasizing collection of evidence only serves one purpose—documentation for future action. If the precipitating event is not severe, then it is hard to see the value in such formalized evidence gathering or documentation, because juvenile or criminal court action will almost certainly not occur in any event. These distinctions may seem nitpicky to some, but they will have direct relevance to how workers carry out their tasks on a day-to-day basis.
Many factors may impact child safety. The authors spend a great deal of space in the discussion section under the safety section (Finding #3) giving reasons why DR would likely degrade child safety. The problem, in my view, is that the existing research is sufficiently strong in quality and quantity to show that child safety is not being degraded. This is a fundamental point of divergence between myself and Hughes and colleagues (2013). They believe that the research to date, including counts of rereferrals to CPS, is simply not convincing in demonstrating that children in DR programs are at least as safe as children served using traditional approaches. I disagree.
Does the DR Literature Misrepresent Traditional CPS to Enhance a DR Model?
The fifth conclusion in the article is that proponents of the DR have misrepresented “traditional CPS to enhance an alternative response model” (p. 504). My view is that the authors are both right and wrong. I agree and have written before that CPS are almost universally believed to be seen as offensively intrusive and unhelpful by clients and even other professionals, when this is not, in fact, the case (Drake & Jonson-Reid, 2007). Where I apparently diverge from Hughes et al. (2013) is that I think this misrepresentation is endemic rather than an intentional set of misrepresentations uniquely driven by proponents of DR. If Hughes et al. are implying intentional falsehood, that strikes me as both counterproductive and likely unfounded. In my view, the confusion about the core nature of CPS (intrusive vs. supportive or criminal justice vs. public health) remains one of the least discussed, little recognized, and most important areas in child welfare research and policy, and one which should be addressed.
Summary
This article raises a number of concerns about DR in a structured way. I was particularly pleased with the breadth of concerns raised, many of which could be profitably addressed in future research. I felt some of the questions asked in the article were misguided, particularly those framing DR as if it were a single, manualized intervention. I found the tone of the article to be unrealistically critical, casting one of the most well-researched areas in child welfare practice in undeservedly negative terms. Both the rigor of the research and the validity of the findings in DR are, in my view, considerably stronger than the authors of the current article suggest.
Footnotes
Editor’s Note
This article was invited and accepted at the discretion of the Editor.
Declaration of Conflicting Interests
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author received no financial support for the research, authorship, and/or publication of this article.