Complex,Co-occurring Needs Patterns and Evidence-Based Service Planning for Families Involved in Foster Care: A Map for Research and Practice

Abstract

This study described the complexity of service need co-occurrence among foster care-involved families and identified prevalent patterns of needs to inform future evidence-based service planning research. We utilized state administrative child maltreatment records, and restricted data to cases where the child entered foster care in 2019 and the caseworker indicated the presence of at least one need from the Family Assessment of Needs and Strengths (FANS; n = 1631). We extracted all unique combinations of needs (i.e., needs profiles), and we used association rule mining to identify patterns within these profiles. A total of 780 unique needs profiles emerged among the 1631 cases, which we condensed into 78 patterns. Although the variability and complexity of needs profiles makes evidence-based service planning difficult, the present analysis mapped prevalent needs patterns to guide future research intended to assist caseworkers in this task. Identification of maltreatment determinants among families involved in foster care, and future research into the needs within different needs patterns that might undermine treatment effectiveness, may result in a better balance between parsimonious service plans and a full consideration of co-occurring service needs.

Keywords

foster care child maltreatment association rule mining machine learning evidence-based service planning

Introduction

Every year in the United States, approximately 3.5 million children come into contact with the child welfare system, over 600,000 children are involved in confirmed cases of child maltreatment, and around 200,000 children enter foster care (Child Maltreatment 2019, 2021; The AFCARS Report, 2020). The social and family-related vulnerabilities that place children in foster care—in addition to the added instability they experience in foster care—impede their optimal health, development, and well-being (Gypen et al., 2017; Zlotnick et al., 2012). For example, adults who were in foster care as children are more likely to be unemployed (Courtney et al., 2018), earn lower income (Courtney et al., 2018; Mersky & Janczewski, 2013), achieve lower educational attainment (Harris et al., 2009; Morton, 2018), experience more housing instability (Greeno et al., 2019), have worse mental and physical health outcomes (Kessler et al., 2008; Villegas et al., 2011), have substance use disorders (Greeno et al., 2019), and be involved in criminal activities (Mersky & Janczewski, 2013; Yang et al., 2021) compared to the general population. Addressing the circumstances under which children enter foster care is an important component of maltreatment prevention.

Evidence-based service planning

Unaddressed caregiver needs are risk factors for child maltreatment and foster care involvement (Dubowitz et al., 2011; Mersky et al., 2009) and re-entry into foster care (Mowbray et al., 2017). We define caregiver needs as: (1) Circumstances or characteristics that inhibit optimal youth development, such as parental substance use; or (2) Deficits in or absences of factors that promote optimal youth development, such as food insecurity. The existence of caregiver needs—and often multiple needs—increases the likelihood of maltreatment (Coulton et al., 2018; Thornberry et al., 2014; Vial et al., 2020; White et al., 2015) and, in the case of child removal, reduces the likelihood of family reunification (LaBrenz et al., 2020; Littell, 2001). Although these needs are risk factors for foster care entry, they may or may not be causes of foster care entry. Yet, the number of mandated treatment services increases along with the number of identified parental needs, regardless of if the needs are direct causes of foster care entry (D’Andrade & Chambers, 2012), and the presence of unaddressed needs makes it more difficult for parents to engage in these mandated services required for reunification (Littell, 2001). In response, experts have called for child welfare systems to increase the use of evidence-based service planning (EBSP; Berliner et al., 2015), or the use of focused, shorter-term, evidence-based interventions that address the factors directly causing maltreatment or posing clear risk for subsequent maltreatment, rather than broad and time-consuming multi-program service plans that focus on all needs.

In 2015, a task force of the American Professional Society on the Abuse of Children (APSAC) detailed an overarching framework for EBSP, and identified three core principles (Berliner et al., 2015). The first—service selection prioritizes effectiveness and efficiency—directs workers to develop service plans that match identified needs with the best available evidence-based interventions. The second principle—focus and parsimony—instructs workers to identify the most pressing needs that are thought to have caused past maltreatment or that pose a substantial risk for subsequent foster care placement, hereafter referred to as determinant needs. The goal is to mandate the minimal number of services needed to remediate the risk of maltreatment recurrence. And lastly, the third principle—triage and sequencing—takes up the issue of complex, co-occurring needs and asks workers to consider the temporal ordering of services and supports along with “stepped care.” That is, service plans should mandate the least restrictive and intensive services required to address high priority needs (Berliner et al., 2015).

These principles acknowledge the challenge for service planning generated by complex, co-occurring needs. Such a challenge has been a long-standing focus of child welfare research and evaluation, particularly because co-occurring needs are associated with a reduced likelihood of family reunification (LaBrenz et al., 2020; Littell, 2001). What remains less clear is which specific needs should be resolved to improve the odds of reunification (i.e., focus and parsimony) and the process for doing so (i.e., triage and sequencing).

Successful implementation of EBSP will therefore require ongoing guidance for foster care workers on how to handle some of the most prevalent patterns of co-occurring needs among families with children in foster care. As a first step in that process, the goal of the current study is to describe the general complexity of service needs profiles among foster care-involved families and identify a set of prevalent needs patterns that can guide subsequent research related to focus, parsimony, and sequencing of mandated service plans. We reason that to plan future research and develop successful child maltreatment prevention strategies, we need to understand the patterns of caregiver needs among families whose children enter foster care. To efficiently design interventions with the largest population-level effects, we need to identify which needs patterns appear most frequently and strongly within the population. We expect that identifying such patterns will inform the contents of evidence-based treatment services that parsimoniously address the determinants of child removal risk while concurrently responding to the existence of other needs important to caregivers. The goal of treatment services should be to ameliorate and/or remove the determinant(s) of foster care entry in a manner that is responsive to caregivers’ concurrent needs, rather than expect a caregiver to simultaneously address and remove all needs in the context of court-ordered services.

Common Approaches to Identifying Patterns of Parent Needs

Multiple needs often co-occur among families with children in foster care (Choi & Ryan, 2007; Jarpe-Ratner et al., 2015; Marsh et al., 2006; Victor et al., 2021). The task of describing and gaining insights from data that include a variety of caregiver needs pose both theoretical and analytic challenges. From a theoretical standpoint, we need to build a framework for understanding how different combinations of needs inter-relate and influence service outcomes. The analytic challenge of examining co-occurring needs is understood as a problem of combinations. We can compute the number of unique combinations (here referred to as profiles) of a set of n dichotomous indicators from the equation 2ⁿ-1. For example, assume we are studying only two service needs, {a} and {b}. The full set of unique profiles (i.e., combinations) is three (2²-1 = 3), or {a}, {b}, and {a,b}. A set of three dichotomously scored service needs ({a}, {b}, {c}) produces seven different profiles. The number of profiles grows rapidly when we consider a broader constellation of needs. Assessments containing either 5, 10, or 15 dichotomously scored service needs would produce 31, 1,023, and 32,767 unique profiles of service needs (respectively).

The combinatorial problem of co-occurrence requires simplifying strategies to make sense of the data and generate actionable insights. There are a few common analytic strategies in the literature. One strategy is to ignore the structure of co-occurrence and focus on a single need while controlling for other needs. Another option is to include interaction terms between needs. Though, as in the above description of the combinatorial problem, the number of interaction terms grows exponentially with the number of needs. Thus, the researcher must establish a priori which interaction terms to include. Another common approach for examining co-occurring needs is by creating a count of the total number of observed service needs. This approach assumes that the impact of a given need is fungible (i.e., can be replaced by a different service need) and additive (i.e., the influence of any given service need is equal to other service needs).

A strategy that has grown in popularity over the past decade has been latent class analysis (LCA). This person-centered approach has been used to explore or discover subgroups—known as ‘classes’ in the LCA framework—of individuals based on having common or shared patterns of service needs (Jarpe-Ratner et al., 2015), behavioral risk factors (Garraza et al., 2011; Hogue & Dauber, 2013), maltreatment profiles (Pears et al., 2008), and complex chronic conditions (Lindley et al., 2016). Researchers often validate latent class models by making systematic comparisons between the extracted classes on different outcomes. Advances in software have made these analyses relatively easy to perform, and the graphical representation of results facilitate model interpretation. Yet, the limitations of LCA are rarely discussed in the literature (Bauer & Curran, 2003). For example, a strict assumption of LCA is local independence. More specifically, in the context of LCA, local independence assumes that, for a given class, observed service needs are uncorrelated with each other, and their relationships are fully explained by the underlying latent variable. Thus, the underlying latent variable would cause changes in the level of service need, as opposed to one of the service needs directly influencing (or being influenced by) another service need. Many published LCA models also apply class labels that convey levels or amounts of service needs (e.g., “low need” and “high need”). This is problematic because LCA assumes classes differ only qualitatively (van Loo et al., 2018). Finally, the majority of LCA studies do not systematically describe the heterogeneity of observed service needs that exist within classes, instead focusing only on the predicted likelihood of service needs. This is problematic given the often indiscriminate and overlapping predicted likelihoods of service needs between different classes.

Association Rrule Mining

Association rule mining allows for the identification of needs patterns as they exist in the data. This is in contrast to ignoring the structure of co-occurrence and focusing on a single need, deciding a priori on which need co-occurrences to focus, reducing needs to simple frequency counts, or employing LCA. Association rule mining is a rule-based machine learning method for identifying observed patterns between variables (here, caregiver needs) in large databases (Hornik et al., 2005; Piatetsky-Shapiro, 1991). Each generated pattern—referred to as a rule in association rule terminology—contains an antecedent and a consequent determined by the algorithm. Antecedents and consequents can be interpreted as a series of if-then statements. For example, consider the following pattern: [substance use, mental health] → [parenting skills]. In this pattern, substance use and mental health are antecedents to parenting skills needs, and parenting skills needs is the consequent. The pattern implies that if substance use and mental health needs are present, then parenting skills needs are present. The terms antecedent and consequent describe positioning within the constructed association rule, but we cannot infer temporal ordering of service needs since all needs were identified at the same time point. Patterns extracted from association rule mining are not always true within a dataset. Additionally, patterns may appear frequently or infrequently, and they may connote a strong or weak relationship. To indicate the certainty, prevalence, and associational strength of a specific pattern, association rules commonly use metrics termed confidence, support, and lift.

Confidence refers to how often a given pattern is true, and mathematically this equates to the proportion of the cases that contain the antecedent which also contain the consequent (Hornik et al., 2005). Carrying forward the previous example, [substance use, mental health] → [parenting skills], confidence is the proportion of cases where parenting skill needs are present among all cases with both substance use and mental health needs. A confidence of 0.75 (i.e., 75%) would convey that in 75% of cases with substance use and mental health needs, parenting skills needs were also present.

Support refers to the probability that the antecedent and consequent appear together (Hornik et al., 2005). Stated another way, support is the prevalence of a pattern. For example, given the pattern [substance use, mental health] → [parenting skills], support is the proportion of cases that report substance use, mental health, and parenting skills needs together. A support of 0.20 (i.e., 20%) would convey that 20% of all cases contained a substance use need, mental health need, and parenting skills need.

Finally, lift is the ratio of the observed support to that expected if antecedent and consequent were independent. Mathematically, this equates to dividing support by the product of the probabilities of the antecedent and consequent. A lift of ‘1’ means that the probability of occurrence of the antecedent and that of the consequent are independent of each other. Hence, a higher lift indicates higher dependency between the consequent and the antecedent, and thus a stronger positive association (Hornik et al., 2005). Carrying forward the prior example with a support of 0.20, [substance use, mental health] → [parenting skills], let us assume the prevalence of the antecedent [substance use, mental health] is 0.25 and the prevalence of the consequent [parenting skills] is 0.40. The product of these probabilities (0.25*0.40) is 0.1. This would result in a lift of 0.20/0.10 = 2. Thus, this hypothetical antecedent and consequent appear together in the dataset more frequently than we would expect if they were completely independent. Patterns with higher lift indicate stronger positive associations between the antecedent and consequent, whereas patterns with lift closer to one indicate that the pattern does not exist much more within the dataset than would be expected by chance. For example, a lift of 2 would convey that the pattern appears in the data 2 times as often as it would be expected to appear by chance alone. Thus, lift provides the greatest indication of what patterns will co-occur together in future datasets.

As stated in the second principle of EBSP—focus and parsimony—caseworkers should identify the most pressing need or needs, known as determinant needs, that are thought to have caused foster care placement or that pose a substantial risk for subsequent foster care entry (Berliner et al., 2015). Ideally, caseworkers should identify the prioritized determinant need(s) and consider (1) if co-occurring needs reduce treatment effectiveness of the prioritized determinant need, and if so, (2) should services for co-occurring needs be sequenced. Yet evidence to inform this process is lacking. Patterns identified in this study, however, can serve as a guide for future research intended to direct the practice of thinking through treatment effectiveness in the presence of co-occurring needs, whereby the determinant need is the one associated with foster care entry, and the potentially co-occurring needs are those that might moderate the effectiveness of interventions designed to address the determinant need or that may be unrelated to foster care entry. We elaborate on these research implications in the Discussion section.

The Current Study

The objective of this study was to use descriptive analyses and association rule mining to identify the prevalence and patterns of caregiver needs under which children enter foster care. More specifically, to inform future research and evidence-based service planning, we sought to address the following regarding needs among families with children entering foster care:

1. Describe singular service needs with respect to prevalence and co-occurrence rates;

2. Identify service needs profiles (i.e., combinations of needs) present among families; and

3. Extract service needs patterns within these needs profiles using association rule mining.

Methods

Data

We obtained data through an official record-sharing agreement between a statewide child welfare agency in the Midwestern United States and the University of Michigan. Based on this agreement, the state agency provided the University of Michigan with a complete set of administrative records for all allegations of child abuse or neglect. The dataset contains a range of variables that document the timing and nature of each allegation. The dataset also contains information on the Child Protective Services (CPS) investigation related to each allegation in addition to strengths and needs assessments of families whose children ultimately enter foster care.

For the present study, the research team used all substantiated cases of abuse and neglect where a child was removed from the home and entered foster care in 2019 (N = 4752 cases), where caseworkers completed a strengths and needs assessment (n = 3847), and where caseworkers indicated a caregiver need within the assessment (n = 1631). Thus, the present study analyzes the data from these 1631 unique cases that indicated a caregiver need. The University of Michigan Institutional Review Board approved the present research.

Measures

We linked child welfare data information, including child and substantiated caregiver race and age, substantiated caregiver gender, and maltreatment type, to the initial Family Assessment of Needs and Strengths (FANS) completed by the foster care caseworker. The state requires caseworkers to complete FANS in all cases open for foster care services where parental rights have not been terminated within 30 days after the removal of the child, and every quarter thereafter. Caseworkers are exempt from completing FANS under the following circumstances: unresponsive caregiver; deceased caregiver; no legal, biological, or putative parent or legal guardian in the household; permanency placement goal of placement with a relative or another planned permanent living arrangement; parental rights terminated; caregiver refusal to participate in reunification services, or; reunification services no longer needed. Because FANS is used for treatment planning rather than maltreatment substantiation decisions, it is also possible that workers may not identify needs on the FANS among substantiated cases.

FANS evaluates the needs and strengths of caregivers and families along fourteen domains: mental health, parenting skills, substance use, intellectual capacity, literacy, physical health, sexual abuse of child, communication skills, domestic violence, social support, child characteristics, housing, financial strain, and employment (Table 1). Caseworkers score families in each of the domains on a scale of −5 to +2, where −5 indicates having a chronic problem or dysfunction in an area, 0 indicates an appropriate response or no problem, and +2 indicates having a strength in the area. Likewise, needs are domains scored with negative numbers whereas strengths are domains scored with zero or a positive number. Caseworkers collect information to complete FANS through interviews with families, collateral contacts, and document reviews. The psychometric properties of the FANS tool are unknown, but caseworkers must include narrative justifications for selected scores for each FANS domain. Presently, we used data from the initial FANS assessment completed within 30 days of child removal. We evaluated prevalence and patterns among needs only (i.e., domains with negative scores), inclusive of all domains assessed by FANS.

Table 1.

Assessed Needs from the Family Assessment of Needs and Strengths (FANS).

Need	Description
Mental health	Caregiver exhibits depression, low self-esteem, and/or emotional problems
Parenting skills	Caregiver needs improvement in parenting skills or exhibits abusive parenting
Substance use	Caregiver engages in substance use resulting in serious problem or dysfunction
Intellectual capacity	Caregiver has some intellectual impairment, limitation, or decision-making difficulty
Literacy	Caregiver is marginally literate or illiterate
Physical health issues	Caregiver has health problem or physical limitation that affects the family
Sexual abuse	Caregiver engaged in or failed to protect child from sexual abuse
Communication skills	Caregiver has limited, ineffective, isolating, hostile, or destructive communication
Domestic relations	There is domestic discord, lack of cooperation, or domestic violence within the home
Social support	Caregiver has limited social support, no social support, or destructive relationships
Child characteristics	Child has physical, emotional, or cognitive issues
Housing	The family experiences some housing instability or insecurity
Financial strain	Caregiver mismanages income or is in financial crisis
Employment	Caregiver in need of employment or is unemployed

Analysis

We used the R programming language version 4.0.2 for data management and analyses. We addressed our first research aim by calculating the prevalence of each need among families where a child was removed from the home. For any given need, we calculated the percentage of cases where the given need was the only one present and identified the average number of co-occurring needs per need (e.g., how many needs, on average, co-occurred with domestic violence). We further characterized cases by the total number of needs present.

Following these descriptive characterizations, we extracted all unique combinations of needs, referred to as needs profiles, and ranked profiles based on prevalence. With 14 different potential service needs included in the FANS instruments, there were 16,383 (i.e., 2¹⁴-1) potential different service needs profiles.

An important distinction is that between needs profiles and needs patterns. Whereas profiles reflect all needs present within a case, patterns reflect the co-occurrence of two or more needs. Each family has only one needs profile, but multiple needs patterns can exist within a given profile. For example, a family with the needs profile [mental health, domestic violence, and housing] could have the needs patterns of [mental health → domestic violence] and/or [mental health, domestic violence → housing] and/or [domestic violence → housing], and so on, given that these need occurrences emerged as patterns in the dataset. As opposed to profiles, patterns require that the need co-occurrence does not just appear once within the dataset, but rather is a persistent co-occurrence across observations whose frequency meets given support, confidence, and lift thresholds.

We conducted association rule mining using the arules R-package version 1.6-8 (Hahsler et al., 2021) to define patterns of service needs under which children enter foster care, our third research aim. For the present analyses, we determined minimum support (20%) and confidence (75%) thresholds to ensure that: (1) The patterns were prevalent (i.e., present in at least one-fifth of the population); and (2) Patterns were true more often than false (i.e., true for three out of every four applicable cases).

Results

Table 2 provides case-level characteristics for the 1631 unique cases in this study. The majority of children and substantiated caregivers were White (62%), average age of the youngest child was 6 years, average age of the youngest substantiated caregiver was 31 years, and the most common maltreatment type was supervisory neglect (75%). Overall race and age demographics for the cases included in the sample are similar to that of all children in foster care in the state (Child Maltreatment 2019, 2021).

Table 2.

Descriptive Summary of Cases (N = 1631).

	N (Percent) or Mean (sd)
Race of child(ren)
White	1011 (62%)
Black	359 (22%)
Multiracial	246 (15%)
American Indian, Asian, Native Hawaiian	11 (<1%)
More than one race	<5 (<1%)
Age of youngest child	6.0 (5.9)
Race of substantiated caregiver(s)
White	1013 (62%)
Black	319 (20%)
More than one race	201 (12%)
Multiracial	51 (3%)
American Indian, Asian, Native Hawaiian	13 (<1%)
Age of youngest substantiated caregiver	31.0 (8.3)
Gender of substantiated caregiver(s)
Women and men	976 (60%)
Women only	531 (33%)
Men only	110 (7%)
Maltreatment type
Supervisory neglect	1228 (75%)
Physical/medical neglect	805 (49%)
Physical abuse	460 (28%)
Threatened harm	431 (26%)
Failure to protect	169 (10%)
Other	118 (7%)
Sexual abuse	101 (6%)

Note. Maltreatment type is non-mutually exclusive.

Singular Service Needs

Parenting skills was the most prevalent need (87% of cases), followed by mental health (68%), substance use (59%), housing (49%), employment (47%), and financial strain needs (46%; Figure 1a). Need co-occurrence was common, with 94% of cases having two or more needs (Figure 1b).

Figure 1.

The prevalence of specific needs (a), and the prevalence of the number of needs per case (b). There were few cases with any one need that had no co-occurring needs (c), while average number of co-occurring needs associated with most needs was approximately 4–7 (d).

Social support system needs and communication skill needs never occurred in isolation. Parenting skill needs was the most common need type to occur in isolation, but occurred in isolation infrequently: Among cases where parenting skills needs were present, parenting skills needs were the only indicated need in 2.7% (Figure 1c). The average number of co-occurring needs that were associated with each individual need varied between approximately five and seven (Figure 1d).

Profiles of Service Needs

There was a total of 780 unique profiles of service needs among the 1631 cases. Table 3 presents the 25 most common needs profiles, which represent 21% of all cases. The most common profile was parenting skills in isolation, occurring in 2.3% of cases. The next most common profile included mental health, employment, housing, parenting skills, financial strain, and substance use needs, occurring in 1.5% of cases. Over 30% of cases had a unique needs profile that was not shared by any other case.

Table 3.

The 25 of 780 Most Prevalent Needs Profiles.

Needs Profile	N	Percent, %	Cumulative Percent, %
Parenting Skills	38	2.3	2.3
Mental Health, Employment, Housing, Parenting Skills, Financial Strain, Substance Use	25	1.5	3.9
Parenting Skills, Substance Use	23	1.4	5.3
Mental Health, Parenting Skills, Substance Use	21	1.3	6.6
Mental Health, Parenting Skills	16	1.0	7.5
Child Characteristics, Parenting Skills	15	0.9	8.5
Substance Use	15	0.9	9.4
Child Characteristics, Mental Health, Parenting Skills	14	0.9	10.2
Domestic Relations, Mental Health, Employment, Housing, Parenting Skills, Financial Strain, Substance Use	13	0.8	11.0
Mental Health, Employment, Parenting Skills, Substance Use	13	0.8	11.8
Child Characteristics	12	0.7	12.6
Child Characteristics, Mental Health, Employment, Housing, Parenting Skills, Financial Strain, Substance Use	12	0.7	13.3
Employment, Housing, Parenting Skills, Financial Strain, Substance Use	12	0.7	14.0
Child Characteristics, Communication Skills, Domestic Relations, Mental Health, Employment, Housing, Parenting Skills, Financial Strain, Social Support System, Substance Use	11	0.7	14.7
Domestic Relations, Mental Health, Parenting Skills, Substance Use	11	0.7	15.4
Mental Health	11	0.7	16.1
Domestic Relations, Mental Health, Employment, Housing, Parenting Skills, Financial Strain, Social Support System, Substance Use	10	0.6	16.7
Mental Health, Employment, Housing, Parenting Skills, Financial Strain, Social Support System, Substance Use	10	0.6	17.3
Mental Health, Employment, Parenting Skills, Financial Strain, Substance Use	10	0.6	17.9
Mental Health, Parenting Skills, Social Support System, Substance Use	10	0.6	18.5
Child Characteristics, Mental Health, Housing, Parenting Skills, Substance Use	9	0.6	19.1
Child Characteristics, Mental Health, Parenting Skills, Sexual Abuse	9	0.6	19.6
Communication Skills, Domestic Relations, Mental Health, Employment, Housing, Parenting Skills, Financial Strain, Social Support System, Substance Use	9	0.6	20.2
Domestic Relations, Mental Health, Parenting Skills	9	0.6	20.7
Child Characteristics, Mental Health, Parenting Skills, Substance Use	8	0.5	21.2

Patterns among Service Needs Profiles

We extracted 78 patterns (i.e., association rules) with measures of support greater than 20% and confidence greater than 75%. In total, six different needs appeared as consequents (i.e., the -then side of the statement): parenting skills, mental health, substance use, housing, employment, and financial strain. Figure 2 summarizes the full set of patterns. You can read the figure as a set of if-then statements. This is done by starting with an antecedent (i.e., if) displayed in the rows and moving across the columns to identify which consequents (i.e., then) are associated with the particular antecedent. Each pattern is represented by a colored rectangular cell, and darker colors signify stronger associations between the antecedent and consequent (i.e., a stronger if-then statement). Numeric values within the cells denote the patterns’ support (i.e., prevalence). For example, the first listed antecedent, [mental health, employment, parenting], is associated with financial strain. We interpret this pattern as: if mental health, employment, and parenting needs are present, then financial strain needs are likely present. The dark cell color of the rectangle representing the [mental health, employment, parenting] → [financial strain] pattern indicates the association between the antecedent and consequent is strong. The numeric value in this cell specifies that this pattern of mental health, employment, parenting skills, plus financial strain needs occurred in 25% of cases. The 78 identified patterns can serve as guides for future research intended to direct caseworkers’ practice of thinking through treatment effectiveness in the presence of co-occurring needs, whereby the determinant need is the consequent, and the potentially co-occurring needs are the antecedents. Thus, we next describe the patterns corresponding to each major consequent.

Figure 2.

Lift and support matrix of 78 extracted patterns from association rule analysis. Notes: Antecedents are displayed in the rows, and consequents are displayed in the columns. Each pattern is represented by a rectangular cell. Numbers within cells denote support (i.e., prevalence) of the pattern. Darker cell shadings signify higher lift (i.e., stronger associations between antecedents and consequents), whereas lighter cells shadings signify lift closer to one (i.e., weaker associations between antecedents and consequents). The solid outlined cell indicates the pattern with the highest lift (1.90), and the dotted outlined cell indicate the pattern with the highest support (0.63). The absence of shading and the absence of a support number indicates no pattern exists for the antecedent-consequent combination.

Parenting Skills Consequent

As displayed in Figure 2, parenting skills were consequents in 31 of the 78 patterns that met our criteria for extraction. The antecedents in these patterns included combinations of nine different needs: mental health, substance use, housing, employment, financial strain, child characteristics, domestic relations, social support, and communication. Each of these patterns had moderate to high support, ranging from 22% to 63% of cases. Despite the moderate to high prevalence of these patterns, they had relatively low lift, ranging from 1.01-1.10. This means the presence of the antecedent needs (i.e., the if- side of the statement) did not signal the likely presence of a parenting skill need. Instead, parenting skills antecedent needs and parenting skills needs were nearly independent, and they co-occurred about as frequently as would be expected by chance. Assessing the parenting skills consequent pattern with the highest support, the most prevalent pattern was that of [mental health] → [parenting skill need], present in 63% of cases (Figure 2, dotted cell). The low lift value of this pattern (lift = 1.05), however, suggests a very weak association exists between mental health needs and parenting skills needs.

Mental Health Consequent

Mental health needs were consequents in 26 of the 78 patterns. The antecedents in these patterns included combinations of nine different needs: parenting skills, substance use, housing, employment, financial strain, child characteristics, domestic relations, social support, and communication. Each of these patterns had moderate support, ranging from 22% to 37% of cases. Similar to the patterns for parenting skills consequents, the patterns had low lift ranging from 1.10-1.29. Thus, the needs that were antecedent to mental health needs are nearly independent from mental health needs, and mental health needs and their antecedents co-occurred about as frequently as would be expected by chance.

Substance Use Consequent

Substance use needs were consequents in three of the 78 patterns. The antecedents in these patterns included combinations of four different needs: parenting skills, mental health, employment, and financial strain. Each of these patterns had moderate support, ranging from 21% to 26% of cases. Similar to the patterns for parenting skills consequents, the patterns had low lift at 1.28. Thus, the needs that were antecedent to substance use needs are nearly independent from substance use needs.

Housing Consequent

Housing needs were consequents in only one of the 78 patterns: [mental health, employment, financial strain] → [housing]. This pattern had moderate support at 21% of cases and moderate lift at 1.54. Thus, the presence of mental health needs, employment needs, and financial strain with housing needs occurred more frequently than would be expected by chance.

Employment Consequent

Employment needs were consequents in nine of the 78 patterns. The antecedents in these patterns include combinations of five different needs: parenting skills, mental health, substance use, housing, and financial strain. Financial strain appeared as an antecedent need in all nine patterns. Each of these patterns had moderate support, ranging from 21% to 34% of cases, and high lift ranging from 1.59-1.67. The consistent presence of financial strain among the antecedents to employment needs and the high lift values indicate a strong association between the two needs.

Financial Strain Consequent

Financial strain needs were consequents in eight of the 78 patterns. The antecedents in these patterns included combinations of five different needs: parenting skills, mental health, substance use, housing, and employment. Employment needs appeared as an antecedent need in all eight patterns. Each of these patterns had moderate support, ranging from 21% to 27% of cases, and high lift ranging from 1.67-1.90. The pattern with the highest lift was [mental health, employment, housing] → [financial strain], and this pattern appeared 1.9 times more often than we would expect due to chance (Figure 2, bolded cell). This high lift value conveys that the presence of mental health, employment, and housing needs (i.e., the if- side of the statement) signals the likely presence of financial strain. This pattern was present in 21% of cases.

Discussion

The purpose of this study was to describe the complexity of service needs profiles among foster care-involved families and identify prevalent needs patterns that can guide future research related to the focus, parsimony, and sequencing of mandated service plans. We found that the expanse of service needs profiles among families with children entering foster care is large and variable, but we also identified meaningful patterns in these profiles using association rule mining. We view these extracted needs patterns as a map for future research and practice for achieving a better balance between parsimoniously addressing direct causes of foster care placement while also considering co-occurring needs that might threaten treatment effectiveness, as these patterns reflect the most prevalent and strongly associated patterns of needs.

The majority of families in this study had four or more needs. Thus, one-to-one service matching— whereby a caseworker requires a unique treatment for each identified need—would generate extensive and often difficult to attain requirements for the majority of the parent population included in our sample (D’Andrade & Chambers, 2012). Given a core EBSP principle is focus and parsimony (i.e., mandating the minimal number of services needed), intervention and service design and planning should consider how to address multiple needs within the fewest services. The presently extracted needs patterns can inform such service coordination and the contents of combined interventions. To reduce parent burden in treatment requirements, intervention design should explore how to integrate highly associated needs into a single intervention or treatment. For this purpose, we recommend focusing on needs patterns with high lift, as these needs are the most inter-dependent and likely to exist together in other samples. For example, coordinating services (and perhaps sequencing services) that address mental health, employment needs, housing needs, and financial needs (i.e., the pattern with highest lift in our analysis) could efficiently address needs for the one-fifth of parents in the study population who experienced these co-occurring needs. Because we cannot infer temporal ordering of service needs from the present analyses, additional research is needed to identify if such coordinated services should be delivered simultaneously or sequenced.

Accordingly, in addition to practice recommendations, our study provides a map for future research. Such research is central to the recently passed Family First Prevention Services Act (The Family First Prevention Services Act (FFPSA), 2018). The FFPSA encourages use of evidence-based approaches by child welfare systems to prevent children’s entry into foster care. Specifically, FFPSA allows systems to be reimbursed for programs on the Title IV-E Prevention Services Clearinghouse, an authorized list of evidence-based interventions maintained by the U.S. Children’s Bureau. Rigorous evaluations with experimental and quasi-experimental designs are needed to identify root determinants of foster care entry and guide parsimonious service planning, with the goals of expanding the services listed on the Title IV-E Prevention Services Clearinghouse and preventing youth foster care entry and re-entry. The present work provides a guide for such research.

As previously discussed, for any given determinant need, the presence of co-occurring needs may undermine treatment. In line with the third principle of EBSP—triage and sequencing— evaluating this effect modification (also known as heterogenous treatment effects or moderation) within effectiveness trials can help identify which needs should be addressed with a prioritized determinant need, and conversely, which needs are tangential to foster care entry. Our association rule mining results provide a map for the development of such trials. Given that a particular consequent need is the prioritized determinant of foster care placement, effectiveness trials should evaluate whether treatment for that prioritized determinant is less efficacious in the presence of particular antecedent needs patterns. For example, the mental health consequent was associated with the following antecedents: parenting skills, substance use, housing, employment, financial strain, child characteristics, domestic relations, social support, and communication. Future research should evaluate if the receipt of mental health services is less effective in reducing foster care re-entry based on the presence of the different combinations of these antecedents. The next generation of research would then test the ordering of such services to better define service sequencing. Given multiple antecedent patterns associated with a single consequent, we recommend prioritizing the needs patterns with highest lift, since high lift indicates stronger association between the needs, which we hypothesize results in a greater likelihood of an interaction effect being present.

In contrast to holding one need as the primary determinant, it may be unclear which need(s) is/are the direct cause of foster care placement and which need(s) are tangential to the placement. Comparative effectiveness research could help clarify the focus of EBSP and determine which services are minimally sufficient for reducing foster care placements among families with particular patterns of needs. As with our recommendations for effect modification studies, our present results provide a map for the development of such studies. For prioritizing the 78 patterns extracted in our analysis, we recommend first focusing on patterns with high support (i.e., prevalence) for the pragmatic reason of sample size requirements: participants in such studies would need to have all needs within the needs pattern. Comparative effectiveness trials could involve experimental factorial designs to compare services for the needs within our extracted needs patterns in isolation and in concert to identify the minimally sufficient treatment route to prevent maltreatment recurrence. Such a design, however, would require withholding treatment for particular needs for a portion of the study participants—a potential ethical violation and obstruction of best practices. Thus, we recommend observational studies that leverage administrative service planning records, cross-over experimental designs, or waitlist control designs as the next steps in elucidating the direct causes of foster care entry and temporal ordering of services. Taking the pattern with highest support from the present analysis [mental health] → [parenting skills], a comparative observational study would assess reunification and repeat foster care entry among four study arms, which would consist of parents with both mental health and parenting skills needs mandated to receive: (1) parenting skills intervention only (2) mental health treatment only; (3) mental health treatment + parenting skills intervention; or (4) neither parenting skills intervention nor mental health treatment. Alternatively, an experimental cross-over design families with this same set of needs could receive services in different orders to provide insights into the temporal sequencing of services.

Several patterns with parenting skills consequents had high support, and thus are good candidates for further comparative effectiveness research. Our present analysis found parenting skills needs to be widespread and co-occurring with many different needs. The high prevalence of parenting skills needs is perhaps unsurprising given that children enter foster care following perceived challenges in caregiving. The difficulty for service planners is to prescribe the right set of interventions to address those challenges. Given the weak networks in which parenting skills needs are embedded (i.e., low lift values), one interpretation of our findings is that parenting skills needs exist within too many different co-occurring needs patterns for a “one-size-fits-all” parenting skills services approach to effectively prevent foster care re-entry. That is, a wide range of parental needs such as mental health, financial strain, domestic violence, and substance misuse are linked with difficulties in parenting, raising the possibility of multiple determinants of maltreatment. A key question for comparative effectiveness research, therefore, is whether generalized parent education and skill development are sufficient to reduce ongoing risk for foster care involvement, or if the associated parental needs (e.g., domestic violence, substance misuse, etc.) are the determinant needs that must be remediated—independent of or in tandem with parenting skills—to lower the risk for foster care involvement.

The patterns of service needs identified in the present analyses may represent constellations of service needs that share a common underlying cause, such as poverty. At least two of the needs that the FANS assessed—financial strain and employment—are directly related to poverty. There are a number of studies demonstrating that increases in income supports reduce foster care entry rates (Biehl & Hill, 2018; Maguire-Jack et al., 2021). Likewise, states might consider policy adjustments to address underlying root causes of constellations of service needs in addition to individual-level services to address rates of foster care entry.

The use of association rule mining to identify meaningful patterns within the full set of caregiver needs profiles is an important first step in leveraging data science and artificial intelligence to improve the quality and effectiveness of child welfare services. This study lays the groundwork for future research designed to consider variations in service effectiveness based on patterns of co-occurring needs. Once needs have been classified, and the impact of co-occurrence has been more thoroughly documented, additional uses of data science methods and artificial intelligence tools could be deployed within child welfare systems to aid workers in need identification and service planning. Workers could be provided with an electronic assessment tool that relies on known consequential relationships with and among determinants to help cue the worker to potential unidentified service needs. For example, a worker may identify substance misuse and mental health as service needs. If these two needs are highly associated with another such as domestic violence, the system can suggest further, more in-depth assessment to rule out the other possible need. We think this would be useful for assessment purposes related to service need matching, not for the prediction of outcomes.

Strengths and Limitations

The present study had a number of strengths and limitations. First, we were able to leverage administrative data to identify patterns of needs that exist within the entire population of families involved in foster care in a Midwestern state and we were not limited by a convenience sample in our generalizations. Further, we used a sophisticated machine learning technique to address a direct service question, namely, what are the patterns of family needs under which children are removed from the home and enter foster care that multi-component maltreatment prevention interventions should address. We were, however, limited by the assessed needs available in the administrative dataset. The FANS tool does not comprehensively assess all family, caregiver, and child needs, and thus other important patterns of needs may exist in this population. Moreover, the FANS tool was developed to guide service planning, thus, the reliability and validity of the tool has not been rigorously evaluated. Given that the reliability and validity of the FANS tool is unknown, there is the potential for the introduction of caseworker bias into the data. Finally, only 34% of families whose children entered foster care indicated a need within the FANS assessment (i.e., 1631 of 4752 families). Nineteen percent (i.e., 905 families) did not complete the FANS, and another 47% of families did not have any need within the FANS. Because FANS does not guide decisions regarding whether or not to remove a child from the home (as this is determined by the maltreatment investigation), the absence of indicated needs among this majority of families may signal a true absence of social and material needs. However, this absence of indicated needs on FANS also may reflect the FANS tool’s inability to detect true and present needs. Accordingly, the present analyses could yield different needs patterns when based on different needs assessment tools. Likewise, repeating the present analyses in datasets from other states that include validated measures of caregiver and family needs, such as the North Carolina Family Assessment Scale (NCFAS)(Reed-Ashcraft et al., 2001) or the Family Assessment Form (FAF)(McCroskey et al., 1991), would provide useful insights into the validity of present findings. Regardless, the present analyses demonstrate that ARM can be a useful tool for identifying needs patterns and guiding future research and practice.

Conclusions

Families involved in foster care have multiple needs, and hundreds of unique combinations of needs exist. As child welfare systems continue to integrate evidence-based practices and evidence-based service planning, caseworkers must remain cognizant of the impact of particular patterns of need co-occurrence on service effectiveness as they seek to develop focused, parsimonious service plans. The present analysis identified patterns in these profiles and provides a map for future research to investigate which needs should be addressed together versus which needs can be sufficiently addressed in isolation to prevention maltreatment and maltreatment recurrence.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Rebeccah L. Sokol

Bryan G. Victor

References

Bauer

D. J.

Curran

P. J.

(2003). Distributional assumptions of growth mixture models: Implications for overextraction of latent trajectory classes. Psychological Methods, 8(3), 338–363. https://doi.org/10.1037/1082-989x.8.3.338

Berliner

Fitzgerald

M. M.

Dorsey

Chaffin

Ondersma

S. J.

Wilson

(2015). Report of the APSAC task force on evidence-based service planning guidelines for child welfare. Child Maltreatment, 20(1), 6–16. https://doi.org/10.1177/1077559514562066

Biehl

A. M.

Hill

(2018). Foster care and the earned income tax credit. Review of Economics of the Household, 16(3), 661–680. https://doi.org/10.1007/s11150-017-9381-1

Child Maltreatment 2019 . (2021). U.S. Department of health & human services, administration for children and families, administration on children, youth and families, children’s Bureau. Child Maltreatment 2019. https://www.acf.hhs.gov/cb/report/child-maltreatment-2019

Choi

Ryan

J. P.

(2007). Co-occurring problems for substance abusing mothers in child welfare: Matching services to improve family reunification. Children and Youth Services Review, 29(11), 1395–1410. https://doi.org/10.1016/j.childyouth.2007.05.013

Coulton

C. J.

Richter

F. G.-C.

Korbin

Crampton

Spilsbury

J. C.

(2018). Understanding trends in neighborhood child maltreatment rates: A three-wave panel study 1990–2010. Child Abuse & Neglect, 84, 170–181. https://doi.org/10.1016/j.chiabu.2018.07.025

Courtney

M. E.

Okpych

N. J.

Park

(2018). Report from CalYOUTH: Findings on the relationship between extended foster care and youth’s outcomes at age 21. Chicago: Chapin Hall at the University of Chicago.

D’Andrade

A. C.

Chambers

R. M.

(2012). Parental problems, case plan requirements, and service targeting in child welfare reunification. Children and Youth Services Review, 34(10), 2131–2138. https://doi.org/10.1016/j.childyouth.2012.07.008

Dubowitz

Kim

Black

M. M.

Weisbart

Semiatin

Magder

L. S.

(2011). Identifying children at high risk for a child maltreatment report. Child Abuse & Neglect, 35(2), 96–104. https://doi.org/10.1016/j.chiabu.2010.09.003

10.

Garraza

L. G.

Azur

Stephens

R. L.

Walrath

C. M.

(2011). Gender differences in patterns of child risk across programmatic phases of the CMHI: A multiple group latent class analysis (LCA). The Journal of Behavioral Health Services & Research, 38(2), 265–277. https://doi.org/10.1007/s11414-010-9219-6

11.

Greeno

E. J.

Lee

B. R.

Tuten

Harburger

(2019). Prevalence of substance use, housing instability, and self-perceived preparation for independence among current and former foster youth. Child and Adolescent Social Work Journal, 36(4), 409–418. https://doi.org/10.1007/s10560-018-0568-y

12.

Gypen

Vanderfaeillie

De Maeyer

Belenger

Van Holen

(2017). Outcomes of children who grew up in foster care: Systematic-review. Children and Youth Services Review, 76, 74–83. https://doi.org/10.1016/j.childyouth.2017.02.035

13.

Hahsler

Buchta

Gruen

Hornik

Johnson

Borgelt

Hahsler

M. M.

(2021). arules: Mining Association Rules and Frequent Itemsets. R package version 1.7-3. https://CRAN.R-project.org/package=arules

14.

Harris

M. S.

Jackson

L. J.

O’Brien

Pecora

P. J.

(2009). Disproportionality in education and employment outcomes of adult foster care alumni. Children and Youth Services Review, 31(11), 1150–1159. https://doi.org/10.1016/j.childyouth.2009.08.002

15.

Hogue

Dauber

(2013). Diagnostic profiles among urban adolescents with unmet treatment needs: Comorbidity and perceived need for treatment. Journal of Emotional and Behavioral Disorders, 21(1), 18–32. https://doi.org/10.1177/1063426611407500

16.

Hornik

Grün

Hahsler

(2005). Arules-A computational environment for mining association rules and frequent item sets. Journal of Statistical Software, 14(15), 1–25. https://doi.org/10.18637/jss.v014.i15

17.

Jarpe-Ratner

Bellamy

J. L.

Yang

D.-H.

Smithgall

(2015). Using child welfare assessments and latent class analysis to identify prevalence and comorbidity of parent service needs. Children and Youth Services Review, 57, 75–82. https://doi.org/10.1016/j.childyouth.2015.07.019

18.

Kessler

R. C.

Pecora

P. J.

Williams

Hiripi

O’Brien

English

White

Zerbe

Downs

A. C.

Plotnick

Hwang

Sampson

N. A.

(2008). Effects of enhanced foster care on the long-term physical and mental health of foster care alumni. Archives of General Psychiatry, 65(6), 625–633. https://doi.org/10.1001/archpsyc.65.6.625

19.

LaBrenz

C. A.

Fong

Cubbin

(2020). The road to reunification: Family-and state system-factors associated with successful reunification for children ages zero-to-five. Child Abuse & Neglect, 99, 104252. https://doi.org/10.1016/j.chiabu.2019.104252

20.

Lindley

L. C.

Mack

J. W.

Bruce

D. J.

(2016). Clusters of multiple complex chronic conditions: A latent class analysis of children at end of life. Journal of Pain and Symptom Management, 51(5), 868–874. https://doi.org/10.1016/j.jpainsymman.2015.12.310

21.

Littell

J. H.

(2001). Client participation and outcomes of intensive family preservation services. Social Work Research, 25(2), 103–113. https://doi.org/10.1093/swr/25.2.103

22.

Maguire-Jack

Johnson-Motoyama

Parmenter

(2021). A scoping review of economic supports for working parents: The relationship of TANF, child care subsidy, SNAP, and EITC to child maltreatment. Aggression and Violent Behavior. Advance online publication. https://doi.org/10.1016/j.avb.2021.101639

23.

Marsh

J. C.

Ryan

J. P.

Choi

Testa

M. F.

(2006). Integrated services for families with multiple problems: Obstacles to family reunification. Children and Youth Services Review, 28(9), 1074–1087. https://doi.org/10.1016/j.childyouth.2005.10.012

24.

McCroskey

Nishimoto

Subramanian

(1991) Assessment in family support programs: Initial reliability and validity testing of the Family Assessment Form. Child Welfare: Journal of Policy, Practice, and Program, 70(1), 19–33.

25.

Mersky

J. P.

Berger

L. M.

Reynolds

A. J.

Gromoske

A. N.

(2009). Risk factors for child and adolescent maltreatment: A longitudinal investigation of a cohort of inner-city youth. Child Maltreatment, 14(1), 73–88. https://doi.org/10.1177/1077559508318399

26.

Mersky

J. P.

Janczewski

(2013). Adult well-being of foster care alumni: Comparisons to other child welfare recipients and a non-child welfare sample in a high-risk, urban setting. Children and Youth Services Review, 35(3), 367–376. https://doi.org/10.1016/j.childyouth.2012.11.016

27.

Morton

B. M.

(2018). The grip of trauma: How trauma disrupts the academic aspirations of foster youth. Child Abuse & Neglect, 75, 73–81. https://doi.org/10.1016/j.chiabu.2017.04.021

28.

Mowbray

Victor

B. G.

Ryan

J. P.

Moore

Perron

B. E.

(2017). Parental substance use and foster care reentry. Journal of Social Work Practice in the Addictions, 17(4), 352–373. https://doi.org/10.1080/1533256x.2017.1361832

29.

Pears

K. C.

Kim

H. K.

Fisher

P. A.

(2008). Psychosocial and cognitive functioning of children with specific profiles of maltreatment. Child Abuse & Neglect, 32(10), 958–971. https://doi.org/10.1016/j.chiabu.2007.12.009

30.

Piatetsky-Shapiro

(1991). Discovery, analysis, and presentation of strong rules. In Knowledge discovery in databases (Vol. 248, pp. 255–264). AAAI/MIT Press.

31.

Reed-Ashcraft

Kirk

R. S.

Fraser

M. W.

(2001). The reliability and validity of the North Carolina family assessment scale. Research on Social Work Practice, 11(4), 503–520. https://doi.org/10.1177/104973150101100406

32.

The AFCARS Report . (2020). U.S. Department of health and human services, administration for children and families, administration on children, youth and families, children’s Bureau. The AFCARS Report. https://www.acf.hhs.gov/sites/default/files/documents/cb/afcarsreport27.pdf

33.

Family First Prevention Services Act. . (2017). H.R.253, 115th Congress, 1. https://www.congress.gov/bill/115th-congress/house-bill/253/text?q=%7B%22search%22%3A%5B%22familyfirstpreventionservicesact%22%5D%7D&r=1

34.

Thornberry

T. P.

Matsuda

Greenman

S. J.

Augustyn

M. B.

Henry

K. L.

Smith

C. A.

Ireland

T. O.

(2014). Adolescent risk factors for child maltreatment. Child Abuse & Neglect, 38(4), 706–722. https://doi.org/10.1016/j.chiabu.2013.08.009

35.

van Loo

H. M.

Wanders

R. B.

Wardenaar

K. J.

Fried

E. I.

(2018). Problems with latent class analysis to detect data-driven subtypes of depression. Molecular Psychiatry, 23(3), 495–496. https://doi.org/10.1038/mp.2016.202

36.

Vial

van der Put

Stams

G. J. J.

Kossakowski

Assink

(2020). Exploring the interrelatedness of risk factors for child maltreatment: A network approach. Child Abuse & Neglect, 107, 104622. https://doi.org/10.1016/j.chiabu.2020.104622

37.

Victor

B. G.

Resko

S. M.

Ryan

J. P.

Perron

B. E.

(2021). Identification of domestic violence service needs among child welfare–involved parents with substance use disorders: A gender-stratified analysis. Journal of Interpersonal Violence, 36(5–6), NP2908–NP2930. https://doi.org/10.1177/0886260518768569

38.

Villegas

Rosenthal

J. A.

O’Brien

Pecora

(2011). Health outcomes for adults in family foster care as children: An analysis by ethnicity. Children and Youth Services Review, 33(1), 110–117. https://doi.org/10.1016/j.childyouth.2010.08.020

39.

White

O. G.

Hindley

Jones

D. P.

(2015). Risk factors for child maltreatment recurrence: An updated systematic review. Medicine, Science and the Law, 55(4), 259–277. https://doi.org/10.1177/0025802414543855

40.

Yang

McCuish

Corrado

(2021). Is the foster care-crime relationship a consequence of exposure? Examining potential moderating factors. Youth Violence and Juvenile Justice, 19(1), 94–112. https://doi.org/10.1177/1541204020939643

41.

Zlotnick

Tam

T. W.

Soman

L. A.

(2012). Life course outcomes on mental and physical health: The impact of foster care on adulthood. American Journal of Public Health, 102(3), 534–540. https://doi.org/10.2105/ajph.2011.300285