Abstract
Keywords
Poverty has consistently been shown to be strongly associated with child maltreatment and, relatedly, Child Protective Services (CPS) contact in the United States (Pelton, 2015). The association has been observed at the individual and neighborhood levels (Coulton et al., 2007; Freisthler et al., 2006; Irwin, 2009; Putnam-Hornstein & Needell, 2011), and over time (Kim & Drake, 2017). Given this association, it is critical to include some measure of poverty as a control when modeling child maltreatment or forms of involvement with the CPS system. Failing to do so will yield biased estimates (Wooldridge, 2013).
Among the most widely used sources of data in the analysis of child maltreatment is the National Child Abuse and Neglect Data System (NCANDS) Child File. It presents the only annually produced and universal national information on CPS reports and agency processes (Sedlak & Ellis, 2022). The basis for numerous peer-reviewed journal articles and government reports, NCANDS collects and combines data on all investigated incidents of child maltreatment from the CPS agencies of the 50 states, District of Columbia, and Puerto Rico (Sedlak & Ellis, 2022). NCANDS does not include a direct measure of socioeconomic status, such as income level or family status above or below the poverty line. It does include three measures related to poverty. These are binary variables noting whether the child's family has inadequate housing (FCHouse), whether the family has a financial problem (FCMoney), and whether the family is on public assistance (FCPublic). These variables are widely used (see Table 1) in the current empirical literature and are commonly employed at the national level. However, in most states, these variables are invalid and contrast sharply with any reasonable estimates. Crucially, the poor quality of these measures across states is not widely understood or routinely taken into account in the current literature.
Articles Using Any of the NCANDS Variables FCPublic, FCMoney, or FCHouse, in the Past 10 Years.
Note. NCANDS = National Child Abuse and Neglect Data System; FCPublic = family is on public assistance; FCHouse = family has inadequate housing; FCMoney = family has a financial problem; ANOVA = analysis of variance.
This paper has four parts. First, in order to illustrate why these three NCANDS poverty-related variables should not be relied upon at a national level, we establish benchmarks for child poverty and inadequate housing within the child welfare population using data from the second wave of the National Study on Child and Adolescent Well-Being (NSCAW II). We then compare estimates from the NCANDS Child file to these criteria. Second, we present a review of how these variables have been used in the literature, including a brief review of findings. Third, we examine the distributions of these variables in NCANDS data from 2010 to 2019, and offer a strategy for assessing the validity of each states’ data. Lastly, we suggest methods for working around the limitations in the NCANDS data through either limiting analyses to states that appear to have valid data or by incorporating county-level same-race economic variables from the American Community Survey.
The three variables we discuss are defined in the NCANDS Child File Codebook (Children's Bureau, 2019) as follows: FCPublic: “This field indicates if the child's family is receiving public assistance” and “Participation in any of the following social service programs such as TANF, General Assistance, Medicaid, SSI, SNAP, WIC, etc.” (p.57). FCMo ney: “A risk factor related to the family's inability to provide sufficient financial resources to meet minimum needs” and “This field indicates if the child's family has a financial problem” (p.56). FCHouse: “Housing facilities were substandard, overcrowded, unsafe or otherwise inadequate for the child to reside in, including homelessness” and “A caregiver risk factor related to substandard, overcrowded, or unsafe housing conditions, including homelessness” (p.55). We provide these definitions in full both for clarity and because they serve as the basis for our establishing benchmarks for purposes of assessing comparative validity.
Establishing a Poverty Benchmark
In order to evaluate FCPublic, FCMoney, and FCHouse, we require a reliable benchmark for purposes of comparison. NSCAW II is an attractive option because it consists of a fairly recent, nationally representative sample of CPS investigations. Although NSCAW II does not directly measure the precise concepts embodied in the NCANDS variables, it presents information that allows for the estimation of FCPublic. NSCAW II collected the caregivers’ self-reported household income, based on the combined income of all family members in the previous 12 months (Dolan et al., 2011). Households were then assigned one of four ordinal values relative to the federal poverty line based on the 2009 DHHS poverty level guidelines: < 50%, 50%–99%, 100%–199%, and >200% the poverty level. NSCAW II observed that 57% of children reported to CPS were below the poverty line, with 83% percent being below 200% of the poverty line (Dolan et al., 2011). This estimate of child poverty among CPS children is in line with other sources (Irwin, 2009; Putnam-Hornstein & Needell, 2011). Given that many public assistance programs have eligibility standards set higher than 100% of the poverty line (Congressional Research Service, 2015), we can conservatively expect that FCPublic should be at least 50%.
An important note is a difference between participation, which FCPublic measures, and eligibility, which we equate with poverty. Depending on the program, eligibility does not mean participation in services. Participation rates for select public assistance programs have been observed to vary, for example, from 20% to 75% (Macartney & Ghertner, 2021). However, NSCAW II also measured public assistance participation directly. Researchers observed that 72% of children in their CPS sample were receiving public insurance (such as Medicaid) near the time of their investigation and 74% of parents reported receiving either SNAP, WIC, SSI, or TANF, either of which would qualify them as having FCPublic = 1 under the codebook definition (Dolan et al., 2012).
A second important note is that not all public assistance programs set thresholds at or above the poverty line. However, a review of nine need-tested federal benefits found that only TANF, whose requirements are set by states, had eligibility criteria frequently under the poverty line (Congressional Research Service, 2015). Medicaid, which was not discussed in the review, has eligibility for children of at least 133% of the poverty line (Centers for Medicare & Medicaid Services, n.d.).
Given these data, and recognizing that variability surely exists between states, we conservatively set our benchmark criterion at 50%. These numbers can be crosschecked against other national figures. Kids Count data, using a definition of public assistance which is broadly similar to the Child File definition, finds that 24% of all children (not just children reported to CPS) received public assistance in 2018 (Kids Count, n.d.). Given the substantial overrepresentation of poor children in CPS reports (Coulton et al., 2007; Sedlak et al., 2010), with low-income children experiencing maltreatment at five times the rate of wealthier children, we would expect the number of children contacted by CPS to be considerably above the 24% figure. Again, 50% seems a conservative threshold.
Beyond the general work linking poverty and maltreatment, which often involves surveying caregivers regarding annual household income, there have been studies demonstrating significant associations between maltreatment and each of the three concepts measured by FCPublic (public assistance receipt), FCHouse (homelessness), and FCMoney (financial stress). Putnam-Hornstein and Needell (2011) found that public health insurance indicators from birth records significantly predicted future child maltreatment reports. Using data from the Fragile Families and Child Well-Being Study, Warren and Font (2015) constructed measures of housing unaffordability and housing instability, and found the latter to be associated with maltreatment risk. Liu and Merritt (2018) measured the perceived financial stress of caregivers using NSCAW II data. The authors observed that 80% of the caregivers were either “struggling to make it” or “just getting by” (p.327) financially, indicating high rates of financial stress in families involved with the child welfare system.
We use findings from Font and Warren (2013) to establish a criterion for FCHouse. Using NSCAW II data, they measure the rate of inadequate housing experienced by CPS-reported children with two indicators. The first is a measure of being doubled up, which they code as “1” if the caregiver reported at least two non-immediate family members (i.e., excluding partners and children) as residing in the same household as their family, and “0” otherwise. The second is a measure of homelessness, which they code as “1” if the caregiver reported having used emergency housing in the past 12 months, and “0” otherwise. Font and Warren (2013) observe a rate of being doubled up of 8.34%, and recent homelessness of 2.91%. The two measures are not mutually exclusive, so one would expect significant overlap. Taken together, these two measures underestimate the rate of FCHouse, which is far broader in the definition of inadequate housing. For example, it identifies safety, which is not directly taken into account in the measures used by Font and Warren (2013). Taking into account the author's estimates estimate, and the more expansive NCANDS definition, we conservatively set our criterion for FCHouse to be 10%.
FCMoney is far more challenging to benchmark, as it is loosely and subjectively defined. Two families might have the exact same economic context, but answer differently due to different interpretations of “sufficient financial resources” and “minimum needs.” FCMoney's relation to FCPublic is also unclear. One could argue that if a family is receiving public assistance, they necessarily have a demonstrable “financial problem,” implying that FCMoney should be found if FCPublic is. However, it is possible a family is able to meet their “minimum needs” because of public assistance receipt, implying FCMoney = 0 even though FCPublic = 1. Due to these challenges in operationalization inherent to the variable's definition, we do not provide a benchmark for FCMoney. However, along with FCPublic and FCHouse, we do examine how it is used in the literature and how it is distributed.
A final note on our benchmarks; they serve as very conservative markers that only function as a basis for comparison to demonstrate that the NCANDS poverty variables are underestimated. We do not endorse them as lower bound estimates of the given variables and assume they might lie considerably below the actual parameters.
Recent Usage of Poverty Variables in NCANDS Data
In order to evaluate how these variables are being used in the recent literature, we ran a search on Google Scholar in the Spring of 2022, limited to articles from 2013 forward and using the terms “NCANDS” and (“FCHouse” or “FCMoney” or “FCPublic”). The goal was not to conduct a sophisticated systematic review but to capture recent work using these variables so that their usage might be assessed. We were able to identify 19 primary sources (see Table 1) employing these variables. In most of the articles, these variables were covariates in regression models. Most articles used the three variables together. Ten of the articles used all three in their models, and two included a composite variable coded “1” if any of the three variables were noted as present. Only FCPublic was used alone, and this was done in three articles.
Fifteen of the 17 studies that used FCPublic included its rate in a descriptive statistics section. FCPublic fell below our benchmark criterion (was below 50%) in all 15. Five studies had rates below 20%, seven between 20% and 30%, and three between 40% and 50%. Out of 16 articles reporting FCMoney, four were below 10%, four between 10% and 20%, three between 20% and 30%, and three between 40% and 50%. Of the 16 articles including rates for FCHouse, five were below our criterion of 10%. Three articles reported rates between 10% and 20%, three were between 20% and 30%, two were between 40% and 50%, and one was above 50%. While none of the studies using FCPublic met our benchmark, several did meet the (much lower) benchmark for FCHouse.
Five of the articles reported rates for FCPublic or FCMoney above 40%. For FCPublic, these approach our benchmark. However, the rates reported in these particular articles are not national samples of the NCANDS population. Three of these articles reported on higher-risk subsets, including prior victims and those with medical neglect. Two of the articles reported only on single states (Texas and Rhode Island). The use of individual states rather than national averages is key, as we describe later because states vary radically with regard to these variables.
In our review of recent literature, the FCPublic variable appears to be generating estimates that are not credible. The FCHouse variable, for which we set a much lower bar, appears more credible, but several articles still failed to generate plausible estimates. Although we do not provide a specific benchmark for FCMoney, we believe the findings above indicate that it is not a valid measure of financial difficulty.
NCANDS Poverty Variable Distributions
We believe that a ready answer explaining low FCPublic, FCHouse, and FCMoney estimates is available when data are disaggregated at the state level. First, we look in detail at the distribution of FCPublic in 2019. We then look at four different measures (FCPublic, FCHousing, FCMoney, and a fourth variable that we derive, and name “FCAny”, which takes the value “1” if any of the other three are “1”), grading each as plausible, not plausible or all missing, across the 2010–2019 time frame.
Table 2 presents the distribution of FCPublic for each state in detail. The first four columns show the distribution of responses by the state as present in the 2019 NCANDS Child File. A value of “1” indicates that the family is on public assistance (yes), and a value of “2” indicates that the family is not on public assistance (no). A value of “9” indicates that the state does not know if the family is on public assistance (unknown), and a value of “99” indicates that the state entered no information (NULL) (Children's Bureau, 2019). It is immediately obvious that states have very different reporting standards, with a large number of states simply never using many of the available columns. For example, one state lists all children as not receiving public assistance (CO), some states list all children as “unknown” (NY, OH, OR) and some provide no data at all (AZ, HI, ID, IL, KS, KY, LA, MA, NV, TN, VA, VT). Some states do appear to have credible estimates, with Nebraska, for example, indicating that 77.3% of their children have an FCPublic value of “1.”
Distribution of FCPublic, 2019, By State.
1 = Yes; 2 = No; 9 = unknown or missing; 99 = null; FCPublic = family is on public assistance.
It appears that common practice in using the NCANDS data among most of the articles we reviewed is to count any response of “1” as “yes” and any other response as “no,” combining the “no,” “unknown” and “NULL” categories. When we apply this approach in the raw Child File, our estimates are consistent with published estimates (i.e., often around 10%). If we could assume that the missing values were missing completely at random, then these estimates would not be systematically biased, although they would still be low. It is immediately evident in Table 2, however, that the data are not missing at random, but that states have distinct and different patterns of reporting. The last column of Table 2 shows a calculated rate for FCPublic including the unknown and missing values in the denominator. This provides an estimate of the number of children measured as receiving public assistance divided by the number of children reported in that state, as appears to be common practice in the literature.
Along with the significant variation in reporting standards highlighted above, Table 2 also demonstrates significant variation in the rates reported.
We propose that the easiest way to understand the variation in reporting standards and rates is to break states into three groups for each poverty measure: (1) The measure is plausible (the rate is at least the value of our criterion value), (2) the measure is not plausible (the rate is lower than the value of our criterion value, or (3) the measure is always missing (either listed as “unknown” or “no data”).
Figures 1 to 3 identify these categories for each state from 2009 to 2019 for FCPublic, FCMoney, and FCHouse, respectively. Additionally, we include the “FCAny” variable, which is coded as “1” if any of the three variables are present in a single record, and zero otherwise in Figure 4. For FCHouse, we consider any rate over 10% to be plausible, while for the other three variables, we consider only values over 50% to be plausible. Note that while we do not provide a specific benchmark for FCMoney, we still present it in the scheme for illustrative purposes.
Family is on public assistance (FCPublic) categorizations (1 = plausible; 2 = not plausible; 3 = all missing).
Family has a financial problem (FCMoney) categorizations (1 = plausible; 2 = not plausible; 3 = all missing).
Family has inadequate housing (FCHouse) categorizations (1 = plausible; 2 = not plausible; 3 = all missing).
FCAny categorizations (1 = plausible; 2 = not plausible; 3 = all missing).
In reviewing Figure 4, it is clear that most states have data that are implausible or entirely missing, and data from these states should not be used, especially in creating national estimates in concert with states with more plausible data. Moving forward, we would like to offer these tables as guides for researchers interested in using NCANDS data in select years.
Fixing the Problem: Available Alternative Approaches
The overlap between poverty and maltreatment, and the extent to which poverty influences CPS decision-making, are longstanding research topics of public and scholarly interest. Similarly, accounting for poverty is necessary for generating meaningful insights into other high-priority areas, such as racial disparities in CPS reporting. Thus, identifying strategies for working around the limitations of the NCANDS fields is necessary. For example, raw Black/White reporting disparity is substantial, almost 2:1 (Children's Bureau, 2023), but studies incorporating eco nomic controls at the individual level, or using Census data to augment NCANDS data (see below) show that disparity drops to about 1:1 or lower after adjusting for poverty (Kim & Drake, 2017; Maloney et al., 2017; Pelton, 2015; Putnam-Hornstein et al., 2013). These concerns are not limited to racial themes, with other key factors (e.g., maltreatment type, rurality) also being related to poverty. Depending on the research questions and design, including poverty is necessary in many studies, particularly those exploring “front end” issues such as racial differences in initial reporting. To the degree that subsequent decision points (e.g., substantiation) may not be so strongly associated with income, inclusion of economic variables is less critical.
Remedy 1: Restricting the Sample
The easiest effective solution is to not use states (or counties, in county-administered states for which county data are available) with unacceptably low poverty estimates when using the NCANDS data. The advantage of restricting the sample of states is that the poverty variable achieves face validity (although we still lack precise knowledge as to whether families are accurately identified as poor) but the cost in external validity is high. Such an approach means that findings are only generalizable to the states sampled. We believe that such findings should not be aggregated but presented on a state-by-state basis, allowing a basis for generalization at the specific state level. This has been done in the past and is a viable solution when national generalizability is not required (Kim et al., 2018).
Remedy 2: Appending County-Level Data
The only way we can see to maintain the national character of the NCANDS data and to include a reasonable economic measure is to use county-level indicators. This is commonly done in studies using NCANDS county-level data, meaning the analysis dataset has one observation per county aggregated from NCANDS data (e.g., Kim & Drake, 2017). Prior work using state-level data similar to NCANDS has found that while smaller areas (such as zip codes and census tracts) provide better results than county-level indicators, county-level indicators are useful and predictive (Aron et al., 2010; Lery, 2009). It is also common practice in the health literature to use block group, tract, zip code, and county-level economic indicators to explain individual-level health outcomes (Moss et al., 2021). In these cases, county median income, or percentage poor, or other indicators are appended directly to individual-level data (one record representing one child or child/event). It is generally agreed that individual-level income data are preferable if obtainable (Hanley & Morgan, 2008). When area-derived socioeconomic values are used, although tract and zip code areal units are preferable, county-level economic data remain associated with many social outcomes, although it is not clear that they will always function as ideal control variables (Moss et al., 2021).
Censoring
Due to privacy concerns, NCANDS Child file data do not include county identifiers for counties with less than 1000 reports. This results in about 20% of children having a state identifier but no county identifier. This has been dealt with in the past by creating “pseudo-counties” (Kim & Drake, 2017), one for each state, aggregating the censored children in small counties in each state into a single statewide pseudo-county. This approach is a compromise and may undesirably collapse very different counties (for example low population poorer rural and wealthier suburban counties). It is also possible to simply discard these children from the analysis, accepting the sample loss and limiting generalizability to high-population counties. In such cases, researchers should carefully compare their reduced sample to the original sample in the interest of addressing generalizability concerns. Of note, because fatality records lack even state identifiers, it is impossible to use these sorts of adjustments to study child maltreatment fatalities.
Using Same-Race Economic Data
A rarely used (Jones et al., 2022; Kim & Drake, 2017; Wulczyn et al., 2013), but promising way to improve results using data at the county level is to break down county economic indicators by race. If the median income of a given county is, for example, $53,000, it is typical practice to apply that value to all individuals in that county, regardless of race. In this case, a Black resident would be assigned a contextual economic value of $53,000 as would a White or Hispanic resident. It is also possible, however, to use “same race” economic data. So if the median income were $53,000 but the Black median income were $45,000 and the White median income was $63,000, then Black subjects in that county could be assigned a contextual income value of $45,000 and White subjects in the same county could be assigned a value of $63,000. Such an approach can also be used when doing analyses at the county level, ideally by also disaggregating the sample into separate Black and White datasets, each using the appropriate race-specific income and child maltreatment rates. This approach has been found to provide improved model fit and findings that are more congruent with individual-level studies (Jones et al., 2022).
Remedy 3: Hybrid Approaches
It is possible to simultaneously employ the sample restriction and contextual income approaches in the same research design. Kim et al., (2018) appended census data to NCANDS data in their study of potential class bias in reporting. Their core analysis used NCANDS supplemented by Census data at the county level. They then replicated their study using state-specific data at the individual level without census data but employing NCANDS economic data for four states with reasonably high rates of economic stress indicated. The findings from both approaches were similar, providing increased confidence in the general findings from the national estimates incorporating county-level census data.
Finally, we would highlight the value of further research exploring subsets of the data where the NCANDS economic variables may be valid. For example, in county-administered states, it would be useful to parse these variables by county and see if some counties appear to have valid data. It is also possible that in some areas valid economic data exist only for particular groups of children, perhaps including substantiated children or children placed in foster care. In such cases, groups with credible data can be parsed and used in appropriate research designs and analyses.
Discussion and Applications to Practice
The FCPublic, FCMoney, and FCHouse variables in the NCANDS Child File cannot be used at a national level. They are at least three times too low and are systematically biased. Results from prior work using these variables should be treated with caution. Ideally, such work could be replicated using the approaches described above.
Given the centrality of economic factors in child maltreatment, the NCANDS Child File should not be analyzed without economic controls, unless a strong case can be made that economic controls are unimportant or unnecessary to the question, design and analysis.
We, therefore, believe that future analysis of NCANDS Child File data should incorporate economic factors using the strategies described above, or alternate approaches which yield similar results. The uncritical use of NCANDS Child File poverty variables at the national level certainly must stop.
As discussed above, we advocate merging county-level data onto NCANDS data as a means of including economic measures. However, one challenge brought forward by this approach is the censoring of county identifiers for counties with low child maltreatment report counts. This leaves approximately 20% of the child population in 80% of counties left out at the county level. These counties are mainly from rural areas, which can show differences from urban areas (e.g., Smith & Pressley, 2019). Also, given the differences in racial composition across low-population counties, this censoring creates a systematic bias in how child welfare researchers can examine already under-researched populations of color (Rowlands & Love, 2021). We recommend that NDACAN do either one of the following.
First, directly link county-level measures (perhaps with slight jittering to preserve confidentiality) into the child file for the de-identified counties to facilitate more generalizable research. Second, release county-aggregated datasets with all counties identified (i.e., removing the individual lens but allowing for better aggregate studies). We believe that both of these approaches retain confidentiality while being inclusive of the full NCANDS population.
Finally, we would emphasize the centrality of poverty to any nuanced analysis of child maltreatment reporting. At the state level, we feel states could derive substantial benefits from more accurately collecting poverty-related variables. At the federal level, perhaps there could be more support in this area. In the long term, it would improve the utility of NCANDS if other indicators of economic stress could be considered and incorporated.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
