Book Review: From evidence to outcomes in child welfare: An international reader

Shlonsky and Benbenishty provide theoretical justification for evidence-based programs in child welfare services in addition to providing practical guidance toward the effective consumption, implementation, and evaluation of evidence in these programs. Multiple accessible frameworks are provided that emphasize critical components of evidence-based practices. These components include (among others) (1) increased research on clinician decision-making processes (beyond the role of standardized risk assessment), (2) increased focus on effectiveness over efficacy research, (3) balancing fidelity and adaptation in treatment dissemination, (4) workforce development, (5) collaboration with academic organizations, and (6) ongoing progress evaluation. Readers are also presented with multiple applied examples in which child welfare agencies successfully generated and utilized evidence for purposes of client needs assessment and program evaluation. The primary limitation of this text includes its exclusion of single-subject research as a viable tool for research in child welfare, limiting its applicability for frontline workers without advanced statistical training.

First, the authors clearly delineate the social and ethical imperatives for increasing the use of empirical methods in child welfare programs. Although the readership of this journal is well versed in the advantages of empirical approaches, the authors elucidate the balance administrators must achieve between treatment fidelity and adaptation to accommodate client context, values, and community resources. The authors also emphasize important distinctions between traditional approaches (e.g., selection of effective interventions and evaluations of “units of service delivered”) and a more functional empirical approach that facilitates timely feedback regarding meaningful client outcomes.

To enhance objectivity in clinical processes, the authors highlight the need to evaluate how and why caseworkers make specific placement decisions. Given the power caseworkers have over their clients’ lives, the urgency of this need is profound. After all, discrepancies and errors abound in the treatment of child welfare clients, and Chapter 6 illustrates the effects these discrepancies can have on families of color as a function of community socioeconomic factors. The authors present a useful model (hierarchical linear modeling) for evaluating the effects of resource allocation on individuals nested in groups, a model that could be applied to the effects of multiple client factors, services, and policies. Unfortunately, this approach (and chapter) is likely to overwhelm administrators who are not as statistically savvy as the investigators. Although multilevel modeling has gained great momentum among academic researchers, professionals outside academia would benefit from a more basic tutorial (e.g., what are fixed vs. time-varying predictors and what are random effects?) and/or reference to alternate sources where such information might be obtained.

The reader is advised that ignorance of caseworkers’ decision making will likely lead to ineffective service delivery and possibly even child fatalities. The authors stop short of emphasizing how failure to critically examine these processes can also lead to systematic errors that serve as the foundation of institutional discrimination and injustice. Indeed, significant resources that are currently applied to increasing cultural competence within child welfare agencies are of little worth if these agencies will not systematically evaluate the consequences their practices have for marginalized groups.

The authors call for greater research evaluating the effectiveness of treatments in minimally controlled settings as opposed to efficacy trials in laboratory settings. After all, the vast majority of research guiding evidence-based treatment stems from studies in which child welfare clients could rarely be included. Efficacy studies seek to isolate variables of interest, whereas child welfare clients often represent the sum of multiple risk factors and experimental confounds. Ironically, the clients that need effective services most desperately are the ones for whom the least evidence has been generated.

In thoughtful critiques of the American Psychological Association’s conception of “evidence,” Chorpita and colleagues (2003; Chorpita & Daleiden, 2009) have also called for varying levels of experimentation with increased attention to effectiveness and treatment dissemination. Rather than focusing exclusively on manualized treatment protocols, they advocate a functional approach that assesses “common elements” of effective interventions, providing clinicians with a more dynamic and flexible set of tools from which to choose in development treatments. The authors of Chapter 4 expand upon this call for a focus on common elements (specific practices that have been shown to be effective) and common factors (general approaches that have been associated with positive outcomes) over reliance on manualized treatments. Such approaches are highly popular insomuch as they facilitate both empirically supported and culturally competent services. Unfortunately, progress in this transition has been slow and researchers continue to rely on randomized clinical trials to the exclusion of quasi-experimental and single-subject research.

The ability to utilize research effectively also requires a level of education and training that surpasses the ability to implement manualized treatment protocols. Thus, the authors acknowledge that increased training of caseworkers is warranted if they are to espouse an integrated understanding of evidence and its component processes. Specifically, they recommend training to evaluate the evidence for specific practices to understand the different types of evidentiary questions (e.g., efficacy vs. effectiveness evaluations and evaluation of prevention programs vs. treatment comparisons) and how the results of each question can be best utilized. Readers are also provided with practical resources for identifying empirically supported treatments and common factors associated with positive outcomes.

Although the level of workforce development prescribed is challenging, the deprofessionalization of child welfare services and high turnover rates among caseworkers make increased training and organizational support essential. Further, the litigious nature of child protection may result in hesitation on the part of well-meaning administrators to invite potential critique. However, a commitment to empirically supported services provides workers with useful performance feedback that reinforces effective service provision in addition to shielding workers from unwarranted liability when tragic outcomes cannot be prevented. Even the best-trained professionals cannot predict human behavior with perfect accuracy, and reducing the stress associated with child welfare decisions can only lead to better retention of talented professionals. As the authors note in Chapter 1, the question is not “what works” but “what has a good chance of working for whom and in what circumstances?” (p. 23). When professionals provide the empirically supported treatments that are indicated for a family at risk, they cannot be held accountable when inevitable statistical errors exert their inherent uncertainty.

The authors encourage the creation of “an organizational culture that supports the use of evidence” (p. 158). Of course, such a commitment implies the collection of objective data that is user-friendly rather than cumbersome. In an environment that already involves excessive paperwork for the sake of record keeping, the authors could have provided more guidance into the development of data collection systems that might actually be utilized and not simply “maintained” for the sake of risk management. To be functional, administrators and caseworkers must appreciate how their contributions to research endeavors represent an investment in long-term productivity and effectiveness rather than simply additional paperwork.

To address the dearth of research evaluating child welfare outcomes, the authors advocate for collaborative efforts with universities and other nonprofit agencies to conduct large-scale evaluations of child welfare programs. It was disappointing that they did not highlight the applicability of single-subject designs for program evaluation purposes. Certainly, academic institutions have much to offer including talented researchers and students who are eager to gain training with minimal, if any, financial reimbursement. For example, Florida’s Behavior Analysis Service Project was formed as a mutually beneficial contract between the Florida Department of Children and Families and several local universities. The goal of this project included the provision of behavioral treatments to children in protective custody whose conduct resulted in frequent, costly placement disruptions. University faculty received access to research sites and participants, while the Department of Children and Families benefited from the expertise (and grant-writing abilities) of doctoral-level university faculty, students, and interns. Elsewhere, similar arrangements have consistently aided in promulgation of evidence-based practices (Gershater-Molko, Lutzker, & Wesch, 2003; Armstrong & Greeson, 2014; Akin, Bryson, McDonald, & Walker, 2012) and potentially enhance the recognition and perceived legitimacy of participating child welfare agencies.

Finally, the authors emphasize that evidence-based practice implies a commitment not just to the use of “effective” interventions but to ongoing progress evaluation and program improvement. Performance evaluation is at least as critical as the initial identification of effective interventions, yet it typically receives far less attention. Well-intended clinicians pride themselves on selecting interventions with empirical support and implementing these interventions with fidelity yet too often fail to measure their outcomes in any meaningful way. As such, they have no feedback regarding their clients’ response to intervention and could unknowingly cause harm to these individuals. This is especially true when practitioners work with clients and issues on which limited research exists (e.g., child welfare clients with complex problems who interact with multiple social welfare systems). Ongoing evaluation is an imperative that is often neglected due to inexperience, yet it need not be so intimidating.

Perhaps the greatest limitation of this text is the authors’ failure to identify single-subject research (SSR) as a tool for addressing many of the barriers facing child welfare programs. This oversight is understandable, given the field’s general reluctance to embrace SSR and the assumption (given its title) that it is better suited for research on individuals than systems. However, SSR represents a far more flexible and dynamic approach to evidence-based practice than the randomized clinical trial (RCT). As stated earlier, most program administrators are not trained and lack the resources needed to conduct sophisticated statistical analyses. Similarly, the individuals served by child welfare programs are not well suited to methods of subject selection or group assignment required in randomized controlled trials. RCTs require strict adherence to a predetermined methodology throughout the duration of the study, and this level of rigor is often impossible in child welfare settings. SSR alleviates these problems by assessing individuals and/or systems as a whole in a time-series analysis and adapting to changes as they arise (e.g., reversing conditions, withdrawing interventions, and applying interventions in staggered fashion across groups to demonstrate experimental control). Finally, SSR provides direct information regarding the effects of an intervention or program on the targeted population as opposed to a sample from which expected effects must be predicted. In this way, well-designed SSR simultaneously establishes both the efficacy and the effectiveness of an intervention or policy for its intended recipients. Numerous texts have been written about SSR (see Kazdin, 2010), and even a rudimentary understanding of the basic experimental logic could aide administrators in accomplishing the goals set forth by Shlonsky and Benbenishty. In regard to clinician decision making, data could be collected on any number of variables within an agency (e.g., recidivism, levels of service, and rates of placement) in response to specific interventions and/or policy changes. The ease and transportability of SSR allows for dynamic application to complex systems as well as systematic replication to examine the generality of study findings. Data collection and analyses can be as simple or complex as desired, facilitating adaptation to the abilities/resources of the workforce. Finally, SSR facilitates ongoing progress evaluation that does not require a statistician’s interpretation. Thus, even clinicians with limited power to make large-scale organizational changes can evaluate their work using empirical logic to the benefit of all involved. As such, frontline workers can be meaningfully engaged in evaluations of the outcomes in which they are so deeply invested.