Differential Response and the Reduction of Child Maltreatment and Foster Care Services Utilization in the U.S. From 2004 to 2017

Abstract

By 2014, the majority of U.S. states had implemented differential response (DR), a system policy that seeks to serve families of low-to moderate-risk for child maltreatment through family engagement, diversion from formal child protective services investigations, and service provision. However, the effects of DR programs on child welfare dynamics have yet to be evaluated nationally using causal methods. Using a quasi-experimental study design with data drawn from the National Child Abuse and Neglect Data System from 2004 to 2017, we found states with DR programs had approximately 19% fewer substantiated reports, 25% fewer children substantiated for neglect, and a 17% reduction in foster care services utilization when compared to states without DR programs. We find these estimates to be robust to the opioid epidemic and incarceration rates. Additional research is needed to better characterize DR programs and isolate the effects of DR programs geographically.

Keywords

child maltreatment differential response foster care

Introduction

In 1990, a report issued by the U.S. Advisory Board on Child Abuse and Neglect indicated that the state of child safety represented a national emergency. Subsequently, the advisory board recommended that the country’s child protection system shift to one that is child-centered and neighborhood-based (Guterman et al., 2014). Several approaches to child welfare system reform emerged after the publication of the advisory board’s report, including the implementation of differential response (DR), a system policy that promotes family engagement by allowing child protective services (CPS) to differentiate its response (e.g., investigation or assessment) to reports of child abuse and neglect based on multiple factors such as level of risk, child age, source of reporter, and type of reported maltreatment (Child Welfare Information Gateway, 2019). By 2014, the majority of states had at one point implemented some type of DR program (QIC-DR, 2014). Several experimental and quasi-experimental studies have sought to determine the safety of DR programs with generally positive results. However, to date, the effects of DR programs on child welfare caseload dynamics have yet to be evaluated nationally using causal methods. The current study examines the effects of DR programs on changes in rates of child maltreatment reports accepted for investigation, substantiated reports, and foster care service utilization by harnessing state level variation in DR implementation over time using a quasi-experimental design with data from the National Child Abuse and Neglect Data System (NCANDS) from 2004 to 2017.

Background

Subsequent to the release of the U.S. Advisory Board on Child Abuse and Neglect report in 1990, the roll out of state DR policies began slowly (Guterman et al., 2014). Initially, many states piloted DR implementation in a subset of specified counties or regions, later broadening to statewide programs based on pilot program results (Guterman et al., 2014). In response to wide variation in program content and implementation in the early years of DR, the American Humane Association and Child Welfare League of America developed a definition of DR in 2005. The definition resulted in a set of core components that states were asked to adhere to when reporting on the use of DR within their child welfare systems (Merkel-Holguin et al., 2006). Following issuance of the core components, efforts were made to develop national standards for DR definitions, implementation, and evaluation. In 2008, the U.S. Department of Health and Human Services established a 5 -year cooperative agreement that funded the National Quality Improvement Center on Differential Response in Child Protective Services (QIC-DR) (Guterman et al., 2014), with the purpose of supporting the implementation and rigorous evaluation of DR in multiple sites. In addition, statutory amendments were made to the Child Abuse Prevention and Treatment Act in 2010 to promote DR and other prevention approaches. Finally, in 2011 the U.S. Congress passed the Child and Family Services Improvement and Innovation Act, which provided a maximum of 30 states with a five year funding waiver to conduct demonstration projects that often included DR-based child welfare system reforms (Guterman et al., 2014).

According to the QIC-DR’s (2014) final report, 29 states had current DR policies in place (19 statewide and 10 at the regional/county-levels only); nine reported programs similar to DR that did not include DR core components; seven had discontinued DR programs, and some were in the planning stages of reinstating programs (Guterman et al., 2014). To date, more than 40 articles and papers have been published to identify DR utilization rates; to determine the safety of children in families assigned to alternative response tracks in DR programs when compared to children receiving traditional investigative responses; to determine track differences in family engagement, caseworker practices, and service provision; and to calculate the costs to CPS associated with DR implementation. The overall results from well-designed evaluations utilizing comparisons in experimental and quasi-experimental studies have demonstrated that child safety, typically measured by the re-report, substantiated re-report, or removal of a child from the home, is not compromised by DR (Fuller & Zhang, 2017; Fuller et al., 2013; Loman & Siegel, 2004, 2005, 2012, 2013, 2015; Loman et al., 2010; Lawrence et al., 2011; Ruppel et al., 2011; Murphy et al., 2013; Winokur et al., 2015). However, few studies have examined the effects of DR on child welfare system dynamics such as changes in reports accepted for investigation, rates of substantiation, or rates of foster care entry. Janczewski (2015) examined the relationship between DR implementation and rates of investigation, substantiation, and removal in neglect cases in 297 U.S. counties in 42 states using 2010 NCANDS data. In this cross-sectional study, counties implementing DR had significantly lower investigation and substantiation rates within county populations when compared to counties without DR. However, higher substantiation rates were observed among investigated cases in DR counties. In another cross-sectional study using 2010 NCANDS data, Janczewski & Mersky (2016) found neglect investigations in 284 counties in 39 states were 2.4 times more likely to be substantiated in DR counties than in non-DR counties. However, results from a mixed-effects longitudinal analysis of 2001–2010 neglect data from 269 counties suggested that while the rate of investigations fell dramatically within 3 years of DR implementation, substantiation rates did not change as a result of DR implementation (Janczewski & Mersky 2016).

In the last round of the Child Welfare IV-E Waiver Demonstration Projects that launched in 2012, three states—Arkansas, Nebraska, and Washington—implemented DR as one of their waiver interventions. Using a matched comparison analysis, a Washington State study found that DR significantly reduced the probability of foster care entries at 3, 6, 12, and 24 months after intake (TriWest Group, 2019). The Nebraska IV-E Waiver Evaluation found a significant difference in out-of-home placements at the individual level, with individuals assigned to DR placed less frequently than those assigned to the traditional investigatory response (University of Nebraska-Lincoln, 2019).

The current study contributes to the small number of studies that have examined the effects of DR programs on child welfare system dynamics outcomes by harnessing state level variation in DR implementation over time across U.S. states and the District of Columbia (DC) using NCANDS data. We examined the extent to which DR was utilized across U.S. states and DC from 2004 to 2017 and whether states that implemented a DR program during this period experienced changes in the number of child maltreatment reports accepted for investigation, substantiated reports, or foster care services utilization when compared to states that did not have a DR program, using a quasi-experimental design. Approximately three-quarters of all substantiated reports involve neglect (74.9%; U.S. Department of Health and Human Services, 2019), and neglect comprises the largest proportion of cases assigned to DR tracks (Shusterman et al., 2005). Therefore, we examined changes in reports, substantiated reports, and foster care service utilization for all types of maltreatment and specifically for child neglect. We hypothesized that DR systems (a) increased the number of reports screened in for investigation or assessment by CPS agencies given dedicated service availability for low-to moderate-risk cases; (b) decreased substantiated reports and reports substantiated for reasons of neglect by diverting families from traditional investigatory tracks; and (c) decreased foster care services utilization for all types of child maltreatment and specifically for reasons of neglect through a differentiated response.

Methods

Study Design

We used difference-in-differences (DID) models to compare changes in child welfare system outcomes before and after the introduction of DR programs in treatment states (i.e., states with DR programs) and control states (i.e., states without DR programs). The DID estimates the differential effect of a treatment on an outcome using a quasi-experimental design that compares the average change over time on an outcome variable for the treatment group compared to the average change in the outcome variable over time for the control group. To identify a causal effect, we assumed that prior to DR implementation, child maltreatment outcomes would have identical trends in treatment and control states, and we tested for that in our models. After a policy change such as the implementation of DR, the DID estimates how child maltreatment changes in the treated states compared to the control states that did not implement DR.

Data

Independent variables. NCANDS is a federally sponsored initiative that collects data on child maltreatment known to CPS agencies in the U.S. on an annual basis (USDHHS, 2015). States submit case-level data, called a Child File, by constructing an electronic file of child-specific records for each report of alleged child maltreatment that received a CPS action in the form of an investigation or an alternative response. A state was coded as utilizing a DR program if any child received a CPS determination of alternative response victim or alternative response non-victim. We triangulated the validity of each state’s use of the alternative response designations in NCANDS using longitudinal information collected by the QIC-DR regarding actual DR program implementation and direct outreach to states. Three states (Alaska, Nevada, and New Jersey) were eliminated from the treatment group and placed in the control group due to the incorrect use of alternative response codes or implementation of a program that was inconsistent with DR core components. States with 1% or less of total children served through a DR program over the study period were also moved from the treatment group to the control group (Utah; n = 1). New York and North Carolina were dropped from the main analysis due to missing foster care services data in NCANDS, resulting in a total of 23 DR states including the District of Columbia. The proportion of children served per year by the 23 DR states was calculated by dividing the total number of unduplicated children screened in for investigation by the total number of unduplicated children designated as alternative response victims or non-victims.

Dependent variables. We measured outcomes using NCANDS data including reported incidents of child maltreatment accepted for investigation or assessment (i.e., reports); total number of substantiated reports, by state and year for all forms of child maltreatment and for substantiated neglect specifically; foster care services resulting from CPS responses to reported allegations of child abuse and neglect for all children (i.e., foster care), children with substantiated reports, and children with reports substantiated for reasons of neglect. The following control group state/year pairs were missing information on reports and substantiated reports for the following years: Alabama, Alaska, and Georgia, 2004; Oregon, 2004 to 2011; and North Dakota, 2004 to 2009. Control group state/year pairs missing information on foster care services utilization included: Alabama 2004 to 2008; Alaska 2004 and 2005; Georgia and Oregon 2004 to 2011; Michigan, 2004 to 2007; and North Dakota, 2004 to 2009. These missing state/year pairs were excluded from the analyses. Pennsylvania was missing data foster care services utilization data from 2004 to 2005 and from 2008 to 2014 so was dropped from the main analyses.

Covariates. We used panel data from a variety of sources to account for state level differences that could affect DR utilization rates and child maltreatment outcomes from 2004 to 2017. Covariates were gathered from the Annual Social and Economic Supplement of the Current Population Survey (Flood & Pacas, 2017) and the University of Kentucky National Welfare Data (2018). Variables included the state log of population, child population, and personal income; share of children ages 3–4, 5–13, and 14–17; share of population that is non-Hispanic Black, non-Hispanic Asian, non-Hispanic other race, and Hispanic any race; share of immigrants in the state; state unemployment rate; share of children living below 75% of poverty line; share of mothers without a high school degree; log of the state minimum wage, the log of Supplemental Nutrition Assistance Program (SNAP) benefits for a family of three, the log of Temporary Assistance for Needy Families (TANF) benefits for a family of three, and a dummy variable for a state having a refundable Earned Income Tax Credit program. We adjusted for state legislative changes that either expanded or narrowed definitions of child maltreatment over time using data drawn from summary reports prepared by the Child Welfare Information Gateway (Children’s Bureau, n.d.).

In robustness checks we examined DID estimates for child maltreatment outcomes for all states with a DR program including North Carolina and New York (n = 25) to states without a DR program (Table 3). In a second robustness check, we considered whether our results were robust to the inclusion of social safety net programs that have been associated with reductions in child maltreatment in past research (Maguire-Jack et al., 2021) by including TANF and SNAP caseloads using data from the University of Kentucky Poverty Research Center (2018). In a third robustness check, we controlled for the nation’s opioid epidemic and its spillover effects on state child welfare systems (Crowley et al., 2019) by including state opioid deaths as a proxy for the opioid crisis using data on opioid overdose deaths for state and year from the Kaiser Family Foundation State Health Facts.¹ We further exploited differences in the timing of the opioid crisis across the top 11 states in terms of opioid deaths: Connecticut, Kentucky, Maine, Maryland, Massachusetts, Michigan, New Hampshire, Ohio, Pennsylvania, Rhode Island, and West Virginia. Research by Swann and Sylvester (2006) found increases in female incarceration were associated with increases in foster care placements in the 1980s. Therefore, we included the log of female incarceration by state and year in our models using data gathered from the Bureau of Justice Statistics National Prisoner Statistics program to account for increases during the study period².

Analytic Strategy

Given the treatment classification, DID can be estimated by the following regression model

l n {(O u t c o m e)}_{s t} = β_{s} + δ_{t} + λ (D R_{s} \times D R_Y e a r_{t}) + ϕ X_{s t} + E_{s t}

(1)

Equation (1) demonstrates that the natural logarithm of the child welfare system outcome is a function of state fixed effects ( $β_{s}$ ), year fixed effects ( $δ_{t}$ ), and an interaction term between the treated state (DR_s) and the year the treatment started (DR_Year_t) as well as demographic and socioeconomic characteristics in the states ( $X_{s t}$ ). The state fixed effects ( $β_{s}$ ) absorb time-invariant characteristics of the state, while the year fixed effects ( $δ_{t}$ ) absorb time-varying characteristics such as economic conditions not directly controlled for in the model. We use the natural logarithm of child welfare system outcomes for ease of interpretation. In equation (1), $λ$ is the DID coefficient, and it can be interpreted as the causal effect of DR programs on the percentage change in the child welfare system outcomes.

Difference-in-differences models assume that the estimated effect of the DR program on welfare system outcome variables in states are parallel (and essentially zero) before the introduction of a DR program. Thus prior to the implementation of the program, child welfare system outcomes would be the same in treatment and control states. We can evaluate the validity of this assumption by estimating a model that includes interactions between the treatment states in years leading up to program implementation.

l n {(O u t c o m e)}_{s t} = β_{s} + δ_{t} + \sum_{k = t - 3}^{t} λ_{k} (D R_{s} \times D R_Y e a r_{t}) + ϕ X_{s t} + E_{s t}

(2)

The parallel trends assumption indicates that the estimated coefficients on the interaction terms ( $λ_{k}$ ) will jointly be equal to zero in the years prior to the treatment. For the DID model and robustness checks, we tested three years of pre-trends for our six outcomes. This statistically tests whether there were other, potentially unobserved factors that might be associated with the DR policy prior to its implementation. We rejected the null hypothesis that pre-trends tests were equal to zero in the case of reports accepted for investigation. In this case, reports accepted for investigation in states that eventually adopted DR were falling prior to the implementation of the DR program. However, among our child welfare system outcomes, reports accepted for investigation had the highest variance. Notably, reports accepted for investigation were not directly related to the DR policy, which takes effect once a child is reported. We failed to reject the null hypothesis that the pre-tends coefficients were jointly equal to zero for the remaining child welfare system outcomes (see estimates in Table 3). Although Bertrand et al. (2004) and Donald and Lang (2007) argue that DID models can be misspecified as a result of serial correlation and intra-group correlation, our approach did not suffer from these problems because we estimated data aggregated to the state level. We adjusted for state and time fixed effects, and all cluster standard errors on state.

Results

Over the study period, 24 states and the District of Columbia (DC) used DR at some point in time (Table 1). We observed a gradual increase in state DR implementation from 2004 to 2015, when all 25 jurisdictions had a program. By 2017, two states discontinued utilization (Louisiana and Massachusetts). The proportion of children served through DR ranged from less than one percent (.2% in Wisconsin in its first year) to more than 50% (56.9% in Minnesota) in any given year with an average utilization rate of 26% over the study period.

Table 1.

Differential Response Utilization in U.S. States and the District of Columbia, 2004–2017.

	2004	2005	2006	2007	2008	2009	2010	2011	2012	2013	2014	2015, %	2016	2017
AR	—	—	—	—	—	—	—	—	—	3.1%	6.5%	6.3	7.5%	8.7%
CO	—	—	—	—	—	—	—	4.6%	7.5%	12.3%	15.2%	18.1	16.6%	17.7%
DC	—	—	—	—	—	—	—	—	1.6%	4.6%	16.7%	20.4	22.4%	23.9%
GA	—	—	—	—	—	—	—	—	22.1%	23.0%	22.9%	21.4	21.3%	34.7%
IA	—	—	—	—	—	—	—	—	—	—	20.7%	26.4	24.4%	20.5%
KY	24.6%	25.6%	25.3%	24.7%	22.6%	22.6%	23.1%	24.3%	24.4%	24.0%	18.1%	12.5	6.6%	2.6%
LA	1.9%	1.7%	0.8%	0.2%	2.1%	7.6%	7.9%	15.9%	16.7%	16.0%	9.4%	0.4	—	—
MD	—	—	—	—	—	—	—	—	—	0.5%	16.5%	28.7	27.3%	24.8%
MA	—	—	—	—	—	-	—	23.7%	26.6%	24.9%	14.9%	13.0	4.8%	—
MN	28.1%	32.6%	39.0%	42.5%	47.2%	49.8%	50.7%	53.0%	53.9%	54.7%	54.5%	52.0	46.7%	44.5%
MO	52.0%	50.7%	51.0%	48.3%	46.1%	43.2%	42.4%	47.2%	41.2%	48.4%	47.7%	49.5	50.2%	56.9%
NE	—	—	—	—	—	—	——	—	—	—	—	1.0	1.2%	2.1%
NY	—	—	—	—	—	1.1%	2.3%	3.7%	5.2%	6.2%	5.8%	5.4	5.6%	5.8%
NC	11.1%	20.5%	28.3%	37.4%	42.2%	42.7%	42.4%	42.3%	42.8%	42.7%	42.5%	43.0	45.2%	44.8%
OH	—	—	—	—	—	—	—	10.9%	14.8%	18.7%	24.5%	32.3	32.4%	33.2%
OK	14.4%	12.4%	11.9%	12.2%	13.9%	32.8%	35.1%	33.2%	16.6%	7.0%	3.6%	2.6	2.7%	2.1%
OR	—	—	—	—	—	—	—	—	—	—	0.4%	4.1	13.2%	12.3%
SC	—	—	—	—	—	—	—	—	13.1%	24.6%	19.5%	10.6	14.7%	14.4%
TN	—	—	0.8%	12.2%	27.0%	39.3%	41.2%	14.3%	14.3%	15.8%	13.9%	37.0	44.1%	46.8%
TX	—	—	—	—	—	—	—	—	—	—	—	1.2	5.4%	7.3%
VT	—	—	—	—	—	3.4%	24.1%	24.8%	24.4%	28.9%	30.9%	33.0	33.7%	30.8%
VA	—	—	—	41.7%	39.9%	37.4%	37.8%	38.3%	39.9%	38.7%	38.3%	37.9	38.8%	42.1%
WA	10.3%	10.7%	17.1%	18.1%	16.2%	21.5%	19.5%	17.3%	17.0%	11.6%	4.1%	24.9	34.1%	39.6%
WI	—	—	—	—	—	—	0.2%	2.1%	4.8%	7.8%	6.2%	5.9	9.6%	10.5%
WY	30.6%	29.6%	33.0%	34.0%	35.8%	37.8%	42.0%	43.8%	44.7%	45.7%	43.1%	42.7	42.2%	42.3%

To test our study hypotheses, we estimated the effects of DR programs in 23 states including DC (excluding New York and North Carolina) on child welfare system outcomes when compared to states that did not have a DR program, for all forms of maltreatment and for neglect specifically (Table 2). Differential response states had nearly 19% fewer substantiated reports (−.189, p < .01) and 25% fewer children with reports substantiated for neglect (−.245, p < .01) when compared to states without DR programs. Differential response states also had a 17% (−.16.5, p < .05) reduction in foster care services utilization when compared to states without DR including 16% (−.158, p < .05) fewer children in foster care with substantiated reports and 17% (−.172, p < .05) fewer children in foster care with substantiated reports of neglect.

Table 2.

Difference-in-Difference Estimates of State DR Utilization on Child Maltreatment and Foster Care Utilization.

Variables	Reports	Substantiated Reports	Substantiated Neglect	Foster Care	Foster Care Substantiated Reports	Foster Care Substantiated Neglect
State uses DR program	0.037 (0.046)	−0.189(0.051) ***	−0.245(0.058) ***	−0.165 (0.062) **	−0.158(0.059) ***	−0.172 (0.071) **
State expanded CAN definition	−0.010 (0.007)	−0.004 (0.015)	−0.007 (0.013)	0.023 (0.017)	0.028* (0.016)	0.025 (0.018)
State narrowed CAN definition	−0.018 (0.020)	0.008 (0.026)	0.032 (0.031)	−0.026 (0.036)	−0.015 (0.034)	−0.002 (0.039)
State has refundable state EITC	−0.131(0.045) ***	−0.013 (0.060)	0.018 (0.071)	−0.112* (0.060)	−0.022 (0.064)	0.006 (0.066)
Log state minimum wage	0.003 (0.115)	−0.023 (0.145)	−0.210 (0.160)	0.111 (0.166)	0.008 (0.135)	−0.006 (0.166)
Log of SNAP benefits	−1.466** (0.638)	−1.355 (0.970)	−1.280 (1.439)	−2.075* (1.089)	−1.361 (0.865)	−1.609 (1.212)
Log of TANF benefits	−0.034 (0.184)	0.111 (0.267)	0.100 (0.317)	0.548** (0.209)	0.248 (0.285)	0.200 (0.308)
State share of immigrants	1.700 (2.546)	−4.583 (4.780)	0.579 (5.407)	11.192* (6.295)	11.472* (6.448)	15.650** (7.071)
State unemployment rate	−0.019* (0.010)	−0.008 (0.020)	−0.014 (0.025)	0.002 (0.025)	0.013 (0.024)	0.006 (0.025)
State share in urban areas	0.244 (0.167)	0.075 (0.234)	0.320 (0.257)	−0.345 (0.359)	−0.367 (0.403)	−0.127 (0.429)
State share Black-non-Hispanic	−0.674 (0.629)	−0.387 (1.042)	−1.593 (1.191)	−0.327 (1.032)	0.098 (1.111)	−0.942 (1.259)
State share Asian-non-Hispanic	−0.636 (0.590)	0.518 (0.818)	0.127 (0.899)	−2.412** (1.111)	−2.087 (1.319)	2.285* (1.224)
State share other race non-Hispanic	−0.125 (0.474)	0.100 (0.621)	−0.449 (0.850)	−0.516 (0.718)	−0.367 (0.733)	−0.824 (0.844)
Person is Hispanic	−0.632 (0.396)	0.178 (0.745)	0.049 (0.938)	−0.969 (0.780)	−0.462 (0.998)	−0.593 (1.065)
Log state income	0.141 (0.308)	−0.691 (0.607)	−0.977 (0.775)	−0.511 (0.683)	−0.839 (0.747)	−1.107 (0.853)
State share of families <75% FPL	0.294 (0.345)	0.089 (0.357)	0.304 (0.434)	0.277 (0.613)	0.141 (0.535)	0.165 (0.606)
State share of mother <HS Ed	0.685* (0.374)	−0.667 (0.683)	−0.984 (0.841)	1.635** (0.739)	0.605 (0.943)	0.536 (1.072)
Log state population	0.840 (0.952)	0.352 (2.194)	0.321 (2.503)	1.359 (2.380)	2.260 (2.725)	2.260 (2.939)
Log state child population	−1.087 (0.731)	0.050 (1.243)	1.138 (1.421)	−1.832 (1.617)	−1.149 (1.580)	−0.272 (1.874)
State share children 3 to 4 yrs	−0.250 (0.575)	−0.664 (1.005)	−0.686 (1.017)	0.067 (1.192)	1.081 (1.249)	0.906 (1.359)
State share children 5 to 13 yrs	0.156 (0.453)	0.182 (0.702)	0.051 (0.706)	0.758 (0.712)	1.513** (0.745)	1.288 (0.818)
State share children 14 to 17 yrs	0.157 (0.365)	0.901 (0.721)	1.556* (0.786)	2.149***(0.716)	2.719***(0.906)	2.903***(0.977)
Year = 2005	0.064** (0.031)	0.085* (0.045)	0.085 (0.065)	0.210(0.070) ***	0.181** (0.072)	0.189** (0.076)
Year = 2006	0.034 (0.038)	0.059 (0.053)	0.085 (0.073)	0.253(0.093) ***	0.208** (0.093)	0.230** (0.105)
Year = 2007	0.046 (0.051)	0.020 (0.073)	0.054 (0.092)	0.176 (0.107)	0.073 (0.104)	0.107 (0.115)
Year = 2008	0.120 (0.074)	0.023 (0.097)	0.086 (0.126)	0.245*(0.142)	0.094 (0.143)	0.140 (0.159)
Year = 2009	0.349(0.126) ***	0.130 (0.185)	0.199 (0.248)	0.314 (0.243)	0.037 (0.247)	0.108 (0.286)
Year = 2010	0.494** (0.191)	0.270 (0.270)	0.357 (0.375)	0.606* (0.353)	0.237 (0.330)	0.370 (0.393)
Year = 2011	0.441** (0.180)	0.261 (0.252)	0.376 (0.359)	0.577 (0.345)	0.234 (0.327)	0.380 (0.385)
Year = 2012	0.422** (0.173)	0.275 (0.244)	0.388 (0.345)	0.590* (0.344)	0.280 (0.324)	0.414 (0.378)
Year = 2013	0.393** (0.170)	0.251 (0.243)	0.368 (0.341)	0.540 (0.331)	0.244 (0.326)	0.376 (0.377)
Year = 2014	0.312** (0.151)	0.186 (0.210)	0.310 (0.280)	0.476 (0.316)	0.226 (0.312)	0.350 (0.339)
Year = 2015	0.352** (0.167)	0.273 (0.230)	0.413 (0.315)	0.613* (0.354)	0.372 (0.345)	0.510 (0.377)
Year = 2016	0.364** (0.167)	0.290 (0.236)	0.397 (0.316)	0.589 (0.384)	0.314 (0.375)	0.408 (0.402)
Year = 2017	0.360** (0.168)	0.276 (0.249)	0.395 (0.328)	0.554 (0.376)	0.282 (0.386)	0.388 (0.411)
Constant	18.958** (9.107)	23.394 (14.768)	13.796 (19.628)	29.985 (20.793)	9.998 (22.408)	4.714 (23.961)
Pre-trends F-Test of three policy leads jointly equal to zero	3.14	1.82	1.29	0.52	1.05	1.95
p-value	0.03	0.16	0.29	0.67	0.38	0.13
Observations	654	654	654	637	637	637
R-squared	0.319	0.157	0.138	0.133	0.123	0.116
Number of jurisdictions	48	48	48	48	48	48

Notes: Standard errors in parentheses.

***p < .01, ** p < .05, * p < .10.

We ran a series of robustness checks to determine whether estimates were robust to the inclusion of New York and North Carolina given available NCANDS data for reports, substantiated reports, and reports substantiated for neglect (Table 3). We also examined whether initial estimates were robust to other important factors that might influence substantiation and foster care services utilization (Table 3). In the first check, estimates of substantiated reports and reports substantiated for neglect were slightly smaller than estimates in the main model in Table 2 (which excluded New York, North Carolina, and Pennsylvania) but retained their statistical significance. In the second check, which was limited to the 23 DR jurisdictions excluding New York, North Carolina, and Pennsylvania, estimates of substantiated reports, reports substantiated for neglect, and the three foster care services utilization outcomes were similar when we included SNAP and TANF caseloads in the full model. A similar pattern was observed in the third robustness check, with estimates that remained robust to state opioid deaths, the timing of the opioid epidemic across states and time, and state female incarceration rates. In the fourth robustness check, we added variables from the second and third checks (SNAP and TANF caseloads, opioid deaths, the timing of the opioid epidemic, and state female incarceration) to the full model and found estimates for all study outcomes to be slightly smaller but again, robust. The full model for each robustness check is available upon request.

Table 3.

Robustness Checks.

Variables	Reports	Substantiated Reports	Substantiated Neglect	Foster Care	Foster Care Substantiated Reports	Foster Care Substantiated Neglect
State has DR program (includes NY, NC, and PA)	0.028 (0.044)	−0.153(0.052)***	−0.243(0.062) ***
TANF and SNAP caseloads	0.048 (0.040)	−0.186(0.054) ***	−0.241(0.062) ***	−0.158 (0.074) **	−0.154 (0.070) **	−0.169 (0.081) **
AM	0.051 (0.045)	−0.177(0.051) ***	−0.235(0.061) ***	−0.160 (0.062) **	−0.161(0.060) ***	−0.175 (0.074) **
TANF and SNAP caseloads + opioids and incarceration	0.061 (0.040)	−0.173(0.054) ***	−0.229(0.064) ***	−0.147 (0.074) **	−0.148 (0.071) **	−0.163 (0.084) *

Notes: Standard errors in parentheses.

***p < .01, ** p < .05, * p < .10.

Discussion

Differential response systems seek to serve families considered low-to moderate-risk for child maltreatment through family engagement, diversion from formal CPS investigations, and service provision. We hypothesized that the presence of DR programs in states would (a) increase the number of reports screened in for investigation or assessment by CPS agencies given dedicated service availability for low-to moderate-risk cases; (b) decrease the number of children with substantiated reports and children substantiated for reasons of neglect by diverting families from the traditional investigatory track, and therefore, substantiation decisions; and (c) decrease foster care services utilization and foster care use specifically for reasons of neglect through diversion and preventive services receipt. We were unable to formally test the first hypothesis because the dependent variable of reports accepted for investigation failed to pass pre-trends tests, suggesting a high degree of variability in accepted reports across DR states. However, we found support for the second and third hypotheses. When compared to states without DR programs, DR states had significantly fewer children with substantiated reports and children utilizing foster care services. Notably, effect sizes were amplified for children with substantiated neglect reports and foster care services utilization among children with substantiated reports for reasons of neglect. These results were significant after accounting for a broad range of state level covariates and robust to the inclusion of state social safety net program participation, the opioid epidemic, and female incarceration rates. These findings comport with prior research that found DR programs to be associated with reduced probabilities for foster care entry among children in DR cases (TriWest Group, 2019; University of Nebraska-Lincoln, 2019) but contrast with findings from Janczewski (2015) and Janczewski & Mersky (2016) who found higher rates of substantiation among investigated neglect cases in counties with DR when compared to counties without DR. Notably, these latter findings were based on a single year of NCANDS data when only 13 states had implemented DR. Janczewski and colleagues also excluded several small counties and states from their analyses due to NCANDS confidentiality requirements for counties with less than 1000 cases.

While the use of county level data may offer greater precision for the study of DR in some cases, it also comes with trade-offs. For example, the county is not always the most relevant geographic unit of analysis for every state: sometimes it is the region or the entire state. In the current study, most DR states (76%) had state administered child welfare systems and dropping small counties from the analysis would have resulted in a considerable loss of data, as 40% of the NCANDS population cannot be attributed to a specific county due to the masking of small counties. Further, comparison of multi-year county aggregate results for the same statistical models for masked versus unmasked county data in NCANDS yields quite different results in terms of coefficient sign, magnitude, and variability (personal communication, [J. Fluke], September 30, 2021). Therefore, we chose the state as the level of analysis to maximize the use of available data. Notably, in this study, our estimates are conservative yet suggest DR significantly reduced substantiation and foster care services utilization during the study period, particularly in cases of neglect.

Limitations

This study’s strengths include the use of longitudinal data to create a quasi-experiment that allowed for the examination of the relationship between DR utilization and child welfare system outcomes over time across all U.S. states and DC. However, the study also has several limitations. While NCANDS is the most reliable source of DR utilization and child welfare services data available in the U.S., neglect estimates must be interpreted cautiously due to the measurement error associated with state definitions, the mapping of child abuse and neglect to NCANDS, and missing NCANDS data for multiple state/year combinations. Further, we were unable to examine parental risk factors, family needs, benefit participation, the types of services received by families, or other factors associated with child welfare decisions due to a lack of valid and reliable data in NCANDS. Moreover, we were unable to examine important differences between states with regard to the structure, practice features, and resources available to DR systems. Such limitations could be overcome in future studies through enhanced NCANDS data collection and dissemination efforts. Finally, assumptions may be made about the safety of children assigned to DR pathways; however, we were unable to ascertain whether actual abuse or neglect may have occurred among children served in DR cases.

Implications for Future Research

Differential response programs appear to represent a critical policy and programmatic strategy in the secondary prevention of child neglect and foster care service utilization. However, concerns have been raised about whether certain thresholds of utilization compromise child safety. For example, Fluke et al. (2019) examined differences in county DR utilization rates for six states that used DR systems from 2004 to 2013. Overall, higher utilization of DR in states was associated with lower re-reports. While overall re-reporting was associated with greater utilization, the rate of re-reporting for DR increased. In an earlier study of 13 states with full DR implementation, jurisdictions that assigned more than 33% of reports to DR experienced equal or greater levels of re-reporting of children assigned to DR when compared to children assigned to the traditional investigatory track (Piper, 2017). These latter findings comport with those of a Minnesota study where greater utilization of DR was associated with higher levels of risk and safety concerns in the DR track (Loman & Siegel, 2004). In some states, child deaths have occurred among children assigned to DR, leading to practices that have decreased utilization by assigning fewer children to DR. In other states, jurisdictions have discontinued DR programs entirely due to concerns related to the quality of implementation (American Professional Society on the Abuse of Children APSAC, 2016). Therefore, beyond the presence of DR implementation, future analyses should examine how utilization levels relate to child welfare system dynamics as well as the factors that influence utilization rates, such as state variability in program scaling, policy frameworks, community resources, and the degree of support from child welfare agency leaders, state legislatures, and the community, which impact sustainability. Future research should also examine how DR implementation interacts with other state level factors that were statistically significant or trending toward significance in our models such as definitions of child maltreatment, social safety net programs, and state sociodemographic characteristics. Ultimately, the gap in the research needed to address utilization variability is to better understand, at the point of initial screening decisions, what families are most likely to benefit from DR.

Implications for Policy and Practice

Results from this analysis indicate that the implementation of DR systems nationwide have led to reductions in the volume of children substantiated for maltreatment, particularly children substantiated for reasons of neglect. As a form of maltreatment, neglect is likely to result in serious long-term outcomes for impacted children (Widom, 2014), yet is the form of maltreatment for which CPS is least prepared to provide effective services (Trocmé et al., 2014), and most often occurs alongside poverty (Pelton, 2015). While speculative, the apparent reductions in foster care services utilization associated with DR implementation in this analysis may be due to reductions in the marginal use of out-of-home placements due to fewer children being exposed to decision makers who may be inclined to make removal decisions when decision making is uncertain (Doyle, 2013). From a policy perspective, DR may be an appropriate strategy for addressing high false positive rates in CPS screening decisions given little evidence that children assigned to DR are less safe than children receiving traditional investigatory CPS responses.

Further examination of the use of DR by child race and ethnicity is also needed to determine the degree to which DR may be an incremental strategy to reduce racial disproportionality and disparities in out-of-home care. Future studies might build on the work of Choi et al. (2021) who found that children of color and families receiving public assistance were less likely to be assigned to receive DR, and Semanchin Jones (2014) who similarly found that assignment to DR was generally less likely for African American, Native American, and multiracial children than white children, when controlling for poverty and other risk factors.

Conclusion

Study findings suggest the utilization of DR as an engagement enabling policy and practice model appears to reduce report substantiation and the utilization of foster care services, particularly among children who are the subjects of neglect reports. This effect appears to operate at multi-state and population levels. Despite these positive results, DR systems, like child welfare overall in the U.S., continues to be inconsistently practiced and in need of additional study to better isolate the effects of DR geographically and to enhance its effectiveness. Given the degree to which DR systems are deeply embedded in CPS systems nationwide, the degree of generally positive overall performance, and the extent to which the overall outcomes are consistent with the current philosophy of child welfare to minimize engagement while supporting families, DR appears to be an approach to child welfare policy that demonstrates societal value.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by the Centers for Disease Control and Prevention of the U.S. Department of Health and Human Services (HHS) (1R01CE003098). The contents are those of the author(s) and do not necessarily represent the official views of, nor an endorsement, by CDC/HHS, or the U.S. Government.

ORCID iDs

Michelle Johnson-Motoyama

Rebecca Phillips

Donna K. Ginther

Oliver W. J. Beer

Lisa Merkel Holguin

John Fluke

Notes

References

American Professional Society on the Abuse of Children (APSAC) . (2016). Special issue: Differential response. In APSAC Advisor (Vol. 28, pp. 1-34). Author.

Bertrand

Duflo

Mullainathan

(2004). How much should we trust differences-in-differences estimates? The Quarterly Journal of Economics, 119(1), 249–275.

Child Welfare Information Gateway . (2019). Definitions of child abuse and neglect in federal law. https://www.childwelfare.gov/topics/can/defining/federal/.

Children’s Bureau, Administration for children and families, & Child welfare information Gateway . (n.d.). Definitions of child abuse and neglect. Cited data from 2005, 2007, 2009, 2011, 2014, 2016. https://www.childwelfare.gov.

Choi

M. J.

Kim

Roper

LaBrenz

C. A.

Boyd

(2021). Racial disparities in assignment to alternative response. Children and Youth Services Review, 125. https://doi.org/10.1016/j.childyouth.2021.105988.

Crowley

D. M.

Connell

C. M.

Jones

Donovan

M. W.

(2019). Considering the child welfare system burden from opioid misuse: Research priorities for estimating public costs. The American Journal of Managed Care, 25(suppl 13), S256–S263.

Donald

S. G.

Lang

(2007). Inference with difference-in-differences and other panel data. The Review of Economics and Statistics, 89(2), 221–233. https://doi.org/10.1162/rest.89.2.221.

Doyle

J. J.

(2013). Causal effects of foster care: An instrumental variables approach. Children and Youth Services Review, 35(7), 1143–1151. https://hdl-handle-net.web.bisu.edu.cn/1721.1/120582.

Flood

S. M.

Pacas

J. D.

(2017). Using the annual social and economic supplement as part of a current population survey panel. Journal of Economic and Social Measurement, 42(3–4), 225–248. https://doi.org/10.3233/JEM-180447.

10.

Fluke

J. D.

Harlaar

Brown

Heisler

Merkel-Holguin

Darnell

(2019). Differential response and children re-reported to child protective services: County data from the national child abuse and neglect data system (NCANDS). Child Maltreatment, 24(2), 127–136. https://doi.org/10.1177/1077559518816381.

11.

Fuller

Nieto

Zhang

(2013). Differential response in Illinois: Final evaluation report. Urbana: Children and Family Research Center, School of Social Work, University of Illinois at Urbana-Champaign. https://cfrc.illinois.edu/pubs/rp_20131001_DifferentialResponseInIllinoisFinalEvaluationReport.pdf.

12.

Fuller

T. L.

Zhang

(2017). The impact of family engagement and child welfare services on maltreatment re-reports and substantiated re-reports. Child Maltreatment, 22(3), 183–193. https://doi.org/10.1177/1077559517709996.

13.

Guterman

Solarte

Myslewicz

(2014). The differential response (DR) implementation resource kit: A resource for jurisdictions considering or planning for DR. Casey Family Programs. http://www.ucdenver.edu/academics/colleges/medicalschool/departments/pediatrics/subs/can/DR/Documents/DifferentialResponse(DR)ImplementationResourceKit‐‐May2014%5B1%5D.pdf.

14.

Janczewski

C. E.

(2015). The influence of differential response on decision-making in child protective service agencies. Child abuse & neglect, 39, 50-60.

15.

Janczewski

C. E.

Mersky

J. P.

(2016). What’s so different about differential response? A multilevel and longitudinal analysis of child neglect investigations. Children and youth services review, 67, 123-132.

16.

Lawrence

C. N.

Rosanbalm

K. D.

Dodge

K. A.

(2011). Multiple response system: Evaluation of policy change in North Carolina’s child welfare system. Children and Youth Services Review, 33(11), 2355–2365. https://doi.org/10.1016/j.childyouth.2011.08.007.

17.

Loman

L. A.

Filonow

C. S.

Siegel

(2010). Ohio alternative response evaluation: Final Report. Institute of Applied Research. http://www.iarstl.org/papers/OhioAREvaluation.pdf.

18.

Loman

L. A.

Siegel

G. L.

(2004). Differential response in Missouri after five years: Final report. Institute of Applied Research. http://www.iarstl.org/papers/MODiffResp2004a.pdf.

19.

Loman

L. A.

Siegel

G. L.

(2005). Alternative response in Minnesota: Findings of the program evaluation. Protecting Children, 20(2/3), 78–92. http://doi.org/10.1.1.577.5639.

20.

Loman

L. A.

Siegel

G. L.

(2012). Effects of anti-poverty services under the differential response approach to child welfare. Children and Youth Services Review, 34(9), 1659–1666. https://doi.org/10.1016/j.childyouth.2012.04.023.

21.

Loman

L. A.

Siegel

G. L.

(2013). Ohio alternative response evaluation extension: Final report. Institute of Applied Research. http://www.iarstl.org/papers/OhioAREvaluation.pdf.

22.

Loman

L. A.

Siegel

G. L.

(2015). Effects of approach and services under differential response on long term child safety and welfare. Child Abuse & Neglect, 39, 86-97. https://doi.org/10.1016/j.chiabu.2014.05.014.

23.

Maguire-Jack

Johnson-Motoyama

Parmenter

(2021). A scoping review of economic supports for working parents: The relationship of TANF, child care subsidy, SNAP, and EITC to child maltreatment. Aggression and Violent Behavior, 101639.

24.

Merkel-Holguın

L. A.

Kaplan

Kwak

(2006). National study on differential response in child welfare. American Humane Association and Child Welfare League of America. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.498.5189&rep=rep1&type=pdf.

25.

Murphy

Newton-Curtis

Kimmich

(2013). Ohio SOAR project: Final report. http://www.hsri.org/files/uploads/publications/OHIO_SOAR_Final_Report.pdf.

26.

National Quality Improvement Center on Differential Response in Child Protective Services. (QIC-DR) . (2014)Final report: QIC-DR crosssite evaluation. http://www.ucdenver.edu/academics/colleges/medicalschool/departments/pediatrics/subs/can/QIC-DR/Documents/FinalCrossSiteEvaluationReport.pdf.

27.

Pelton

L. H.

(2015). The continuing role of material factors in child maltreatment and placement. Child Abuse & Neglect, 41, 30-39. https://doi.org/10.1016/j.chiabu.2014.08.001.

28.

Piper

K. A.

(2017). Differential response in child protection: How much is too much? Childrenand Youth Services Review, 82, 69-80. https://doi.org/10.1016/j.childyouth.2017.09.014.

29.

Ruppel

Huang

Haulenbeek

(2011). Differential response in child protective services in New York State: Implementation, initial outcomes, and impact of pilot project. http://www.ocfs.state.ny.us/main/reports/CPSDifferentialResponseEvaluationFinalReport_Jan2011.pdf.

30.

Semanchin Jones

(2014). Implementation of differential response: A race equity analysis. Child Abuse & Neglect, 39, 73-85. https://doi.org/10.1016/j.chiabu.2014.04.013.

31.

Shusterman

G. R.

Hollinshead

Fluke

J. D.

Yuan

Y. T.

(2005). Alternative responses to child maltreatment: Findings from NCANDS. U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation.

32.

Swann

C. A.

Sylvester

M. S.

(2006). The foster care crisis: What caused caseloads to grow? Demography, 43(2), 309–335. https://doi.org/10.1353/dem.2006.0019.

33.

The University of Nebraska-Lincoln, Center of children, Families, & The Law . (2019). Nebraska IV-E Waiver final report.

34.

TriWest Group . (2019). Washington state IV-E waiver demonstration project: Family assessment response (FAR) final evaluation report.

35.

Trocmé

Kyte

Sinha

Fallon

(2014). Urgent protection versus chronic need: Clarifying the dual mandate of child welfare services across Canada. Social Sciences, 3, 483-498. https://doi.org/10.3390/socsci3030483.

36.

University of Kentucky Center for Poverty Research . (2018). “UKCPR national welfare data, 1980-2014.” Gatton College of Business and Economics. http://www.ukcpr.org/data.

37.

U.S. Department of Health and Human Services, Administration for Children and Families, Administration on Children, Youth and Families, & Children’s Bureau . (2019). Child maltreatment 2018. http://www.acf.hhs.gov/programs/cb/research-data-technology/statistics-research/child-maltreatment

38.

U.S. Department of Health and Human Services, Administration for children and families, Children’s Bureau, & About NCANDS . (2015). http://www.acf.hhs.gov/programs/cb/resource/about-ncands.

39.

Widom

C. S.

(2014). Long-term consequences of child maltreatment. In Korbin

Krugman

(Eds.), Handbook of child maltreatment: Child maltreatment - Contemporary issues in research and policy (Vol. 2, pp. 225–247). Springer. https://doi.org/10.1007/978-94-007-7208-3_12.

40.

Winokur

Ellis

Drury

Rogers

(2015). Answering the big questions about differential response in Colorado: Safety and cost outcomes from a randomized controlled trial. Child Abuse & Neglect, 39, 98-108. https://doi.org/10.1016/j.chiabu.2014.06.005.