Postadoption and Guardianship

Abstract

Objectives:

This study evaluated the effects of the Adoption Preservation, Assessment, and Linkage (APAL) postpermanency program.

Method:

A quasi-experimental, posttest-only design was used to estimate the program’s effects on youth discharged from foster care to adoption or legal guardianship. A random sample was surveyed (female = 44.7%; African American = 94%) and tracked with administrative data. The primary analysis estimated the program’s effect of being assigned to the intervention, whereas a supplementary analysis estimated the effects of actually receiving the services.

Results:

The APAL program was associated with higher levels of caregiver commitment, fewer youth behavior problems, and lower odds of placement discontinuity.

Conclusion:

Offering postpermanency outreach services to families might help reinforce caregiver commitment and promote children’s well-being.

Keywords

postadoption implementation integrity quasi-experiment instrumental variables placement discontinuity

Recent changes in federal child welfare policy encourage states to place special emphasis on achieving timely permanence for children in out-of-home care. Since the implementation of the Adoption and Safe Families Act (ASFA) in 1997, many states have succeeded in accelerating permanency timelines for foster children waiting to be adopted or discharged to the legal guardianship of relatives and foster parents. Nationwide, approximately 52,000 children were adopted from public foster care systems in 2012 (U.S. Department of Health and Human Services [DHHS], 2013), a 41% increase over the 37,000 children adopted before the passage of ASFA. Another 16,000 foster children were discharged to the guardianship of relatives or foster parents (U.S. DHHS, 2013), a 167% increase since ASFA. In many states including Illinois, Michigan, Missouri, New Jersey, and New York, the number of children in subsidized adoptive and guardianship placements now exceeds the total number of children in foster care (Barth, Wulczyn, & Crea, 2005; Testa, 2004).

Achieving a permanent home through adoption or guardianship does not always mark the end of a child’s involvement with the child welfare system. Maintaining a permanent family connection is a lifelong process that is beset by new challenges and unforeseen difficulties stemming from caregiver burden, financial strains, and limited social support (Berry, Propp, & Martens, 2006; Festinger, 2002; Howard & Smith, 2003; Reilly & Platz, 2003). The traumatic effects of being removed from one’s own home and placed, sometimes repeatedly, in alternative care settings do not miraculously disappear after an adoption or guardianship is finalized. Approximately 10–25% of intended adoptions disrupt prior to finalization, and between 1% and 10% of adoptive homes result in the child’s temporary or long-term replacement into foster care after the adoption has been legally finalized (Child Welfare Information Gateway, 2012). Children who are adopted at an older age and exhibit multiple behavioral problems are at high risk for reentering foster care (Barth, Berry, Yoshikami, Goodfield, & Carson, 1988; Festinger, 2002; Smith, Garnier, Howard, & Ryan, 2006). Because an increasing number of older children are attaining permanence through adoption and guardianship, federal and state governments are placing a higher priority on designing, testing, and disseminating evidence-supported, postpermanency services and support interventions that can assist families in sustaining their original permanency commitments and prevent children from reentering foster care. To help build the empirical evidence for the efficacy and effectiveness of postpermanency services, this study employs a quasi-experimental design and advanced methods of statistical analysis to evaluate the effects of a postpermanency intervention for older youth discharged from foster care to adoption or legal guardianship in the state of Illinois.

Factors Influencing Postpermanency Adjustment

The majority of studies on postpermanency continuity have focused on identifying children’s and families’ characteristics influencing their adjustment and adaptation. For example, child age is consistently reported as one of the strongest sociodemographic predictors of adoption disruption or dissolution, as older children tend to live a longer period in maltreating families, experience more placements in out-of-home care, or retain deeper emotional attachments to their birth parents (Barth et al., 1988; Festinger, 2002; McDonald, Propp, & Murphy, 2001; Webster, Barth, & Needell, 2000). Children with physical, mental, emotional disabilities or behavioral problems have been associated with higher levels of child externalizing behaviors and parenting burdens, which lead to placement instability (Barth et al., 1988; Leung & Erich, 2002; Smith et al., 2006). Conversely, positive behavioral, emotional, or educational functioning prior to adoption leads to better postadoption adjustment (Goldman & Ryan, 2011). Longitudinal analysis establishes a strong linkage between previous multiple placements and an increased likelihood of subsequent moves in long-term care (Webster et al., 2000). However, findings on relationships between other sociodemographic characteristics of adopted children and adoption outcomes are inconsistent and inconclusive. For example, the association between race/ethnicity and placement outcomes remains ambiguous, and studies have produced conflicting results (e.g., Smith et al., 2006; Webster et al., 2000). Findings on sibling group adoption also yield a mixed picture (e.g., Leung & Erich, 2002; Smith et al., 2006).

In terms of family characteristics, studies find evidence of relationships between a variety of caregiver or family characteristics and postpermanency outcomes. Adoption by strangers, as opposed to foster care adoption, and by families lacking experience caring for children with special needs increase the risk of disruption (Barth et al., 1988). The higher achievement expectations that permanent caregivers have for the children, which are also correlated with caregivers’ own levels of educational attainment, especially the mother’s, and higher family income, the more likely disruption or dissolution will occur (Haugaard & Hazan, 2003; McDonald et al., 2001). Married adoptive parents tend to report more positive adjustment than do single parents (McDonald et al., 2001). Mothers’ regular attendance at religious services is associated with increased positive family functioning (Erich & Leung, 1998). Houston and Kramer (2008) demonstrate that the higher the level of contact with the adoption agency, the less family conflict parents experience and the more satisfied they feel about the adoption. Finally, Howard and Smith (2003) find that kin parents or guardians are less likely to report troubles and express service needs. Kin adopters are more willing to adopt the same child if they had to do over it again and more satisfied with the overall adoption than nonkin adopters (Ryan, Hinterlong, Hegar, & Johnson, 2010).

Program Evaluations of Postpermanency Services

Few of postpermanency services and programs have been rigorously evaluated for their efficacy and effectiveness (Barth & Miller, 2000). In spite of the paucity of rigorously evaluated postpermanency interventions, we were able to retrieve 15 studies, 11 of which used the rate of disruption, dissolution, or postpermanency discontinuity as the main indicator to evaluate program effectiveness. Smith’s (2006) evaluation of the Illinois adoption preservation services showed that at the conclusion of services, 87% of children receiving services were still living at home. Similarly, the evaluation of a postpermanency program in San Diego showed that 88% of families retained physical custody of the children after receiving the services (Tibbitts & Mike, 2002). A study of the Missouri Intensive In-Home Services Program (Berry et al., 2006) found that after receiving the intervention, 83% of adoptive families avoided out-of-home placements of children with special needs after a year of follow-up. The evaluation of Oregon’s Postadoption Family Therapy Project reported the highest continuity rate: 92% of the families remained intact after finishing family therapy (Prew, Suter, & Carrington, 1990). Although other studies have reported lower rates of family continuity in the 70–73% range (Avery, 2004; Groze, Young, & Corcran-Rumppe, 1991), the variation most likely reflects differences in the duration of surveillance and selection biases that arise from the intake of children assessed to be at high risk for out-of-home placement.

We found five studies that used the Child Behavior Checklist (Achenbach, 1991) to compare children’s social and emotional well-being before and after participating in the services. Two studies that evaluated postadoption services for children with severe behavioral, emotional, and medical problems reported that children achieved a positive adjustment at home and at school after participating in the services (Goldsmith, 2002; Tibbitts & Mike, 2002). The evaluation by Lenerz, Gibbs, and Barth’s (2006) of a postadoption program provided to 400 adoptive families in Connecticut showed that even short-term services led to significant improvements in children’s behaviors. Howard and Smith’s (1995) evaluation of an adoption preservation program in Illinois showed that at the end of the first year of the preservation services, there was a significant decline in children’s total behavior problems scores, externalizing behavior scores, and internalizing behavior scores. Dhami, Mandel, and Sothmann (2007) evaluated a postadoption support program in Canada. Adoptive parents indicated that the use of postadoption services had a positive impact on children’s behaviors and emotions as well as their relations with siblings and peers.

In summary, these studies exhibit some limitations in the validity of their design and analysis. For example, the lack of an experimental design makes it difficult to detect the true program effects (Shadish, Cook, & Campbell, 2002). With one exception (see Lahti, 2006), the remaining reviewed studies used either a one-group posttest-only design or one-group pretest–posttest design, which cannot confidently rule out confounding factors that might pose a threat to the internal validity of the findings. In addition, none of these studies provided an implementation integrity assessment, but instead assumed that all participants fully complied with treatment assignment.

This study uses a more rigorous study design than prior evaluations of postpermanency programs and applies advanced statistical analysis to generate undiluted estimates of program effects. It aims to answer the following well-built research question framed in PICO terms (Testa, 2010)—Population; Intervention; Comparator, and Outcome:
Do former Illinois foster youth ages 12 to 17 years old who are in adoptive or guardianship arrangements (P) have fewer unmet service needs, fewer behavior problems, caregivers with higher levels of commitment, and lower rates of family discontinuity (O) if they were assigned to or received the Adoption Preservation, Assessment and Linkage (APAL) program (I) compared with similar youth who were assigned to or received services as usual (C)?
The theory of change that guided the choice of the Adoption Preservation, Assessment, and Linkage (APAL) intervention focused on the reduction of unmet service needs as the primary outcome. When a child exhibits difficult behavior problems, the “costs” of placement stability may start to feel excessive to caregivers (Timmer, Sedlar, & Urquiza, 2004). In the absence of accessible and acceptable postpermanency services (Freundlich & Wright, 2003), the chances that children with unmet service needs will be retained in the permanent placement are hypothesized to be low. By linking children to postpermanency services, the assumption is that the volume of unmet service needs would be reduced, which could directly reduce child behavior problems and secondarily fortify a family’s permanency commitment to the child. Together these proximal outcomes are hypothesized to reduce the distal outcome of postpermanency discontinuity compared to children whose postpermanency service needs were not met.

Method

Populations

Drawing on the findings from Illinois Postpermanency Survey I, which showed Illinois postpermanency families with older children reported substantial service needs (see Fuller et al., 2006), the Illinois Department of Children and Family Services (IDCFS) developed a postadoption program called APAL with the goal of providing needs assessments and referral services with families to address their service needs and helping prevent out-of-home placement and increase long-term placement stability. The covered population was 4,050 families who were adopted or taken into guardianship between July 1997 and June 2004 and resided in the Chicago area, had an active subsidy case between October 2007 and September 2008, and had ever been assigned to the Illinois title IV-E Subsidized Guardianship Waiver Demonstration.

Intervention Description

The APAL intervention is a needs assessment and referral outreach program funded by IDCFS and implemented by three private agencies in the Chicago area. Youth aged either 13 or 16 years at the time of assignment (October 19, 2007) were eligible to receive the APAL intervention and youth aged 12, 14, 15, or 17 were provided services as usual offered by the DCFS postadoption unit (13 is the median of 12 and 14; 16 is the median of 15 and 17). A letter inviting families to participate in the APAL project was sent out to the families when IDCFS was conducting a recertification of caregiver eligibility to renew the annual medical subsidy. Caregivers were interviewed about whether they needed a set of services in the areas of health and mental health, education, and other support services; whether the services were included in the adoption/guardianship agreement; and whether they had tried to obtain those services on their own. If caregivers expressed a need for any of the services and also indicated they needed assistance in obtaining the services, caseworkers would then refer them to the DCFS postadoption unit or the Maintaining Adoption Connections program. As soon as the needs assessments and/or service referral was finished, the case was closed. Because these three child welfare agencies providing the APAL services adhered to the same standardized treatment protocol to assess need, there should be minimal variation in the treatment provided across these three agencies.

Research Design and Sample

Six months after the APAL program was implemented, the Illinois Postpermanency Survey II was conducted by telephone survey to evaluate the effectiveness of the APAL program. This study presents findings to the evaluation. One of the authors was the principal investigator of the original evaluation study. The evaluation used a quasi-experimental, posttest-only design with a comparison group. A stratified random sample was drawn from each study stratum that was fixed by APAL treatment assignment and eligibility of subsidized guardianship (vs. adoption). Of the families who were assigned to receive the APAL intervention, 225 households were randomly chosen from the guardianship-eligible group and 110 from the guardianship-ineligible (adoption) group for the posttreatment interview for the evaluation. Of the families that were assigned to receive the services as usual, 225 households were randomly chosen from the guardianship-eligible group and 110 from the guardianship-ineligible group for comparison purposes. The participant flowchart is presented in Figure 1. The unit of assignment is family, but only one target child was selected for the APAL assessment (with the earliest case opening date determining which child was the focal child). As a consequence, there is no clustering effect within each family. The primary data collection was approved by the University of Illinois Institutional Review Board and reanalysis of the study data was approved by the University of North Carolina Institutional Review Board. Permission to link survey responses to administrative data was limited to respondents who gave informed consent for the data linkage. Of the 447 respondents (intervention, n = 232; comparison, n = 215) who completed the survey, 10 interviewed cases were dropped from the follow-up analysis because eight did not consent to data linkage and two were subsequently determined to have been ineligible for inclusion in the analysis sample.

Figure 1.
Participant flowchart.

Because unequal probabilities of selection were used, sample weights and Taylor series estimation of standard errors were applied in the analysis to compute confidence limits and to represent the population of adoptions and legal guardianships in Cook County, Illinois. The completed interviews consisted of 437 families (intervention, n = 224, 66.9%; comparison, n = 213, 63.6%) and represented 65% of the overall families selected into the sample. As none of the variables, including dependent variables, had missing values for more than 5% of the sample, we chose list-wise deletion for cases with incomplete data in the inferential statistics.

Implementation Integrity

The effectiveness of a program is a product of both the validity of the intervention and the integrity of its implementation (Testa & White, 2014). Failure to achieve a program’s intended outcomes may reflect either a problem with implementation integrity or a problem with intervention validity. Although consensus is still building around the meaning of implementation integrity (also called fidelity, Dusenbury, Brannigan, Falco, & Hansen, 2003), Dane and Schneider (1998) identify five dimensions that provide a useful starting point, that is, adherence, the degree to which program components are delivered as planned; exposure, the amount of program content received by participants; quality, assessment of the skill and competence of the program deliverers; responsiveness, the level of acceptance or rejection of program content by the participants; and program differentiation, the uniqueness of the intervention compared to other programs, especially those components received by the comparison group.

Of the five dimensions, only exposure and program differentiation were measureable in this study with available data. All participants were asked to recall whether they had been contacted by a service provider who wanted to talk to them about service needs for the target youth. To correct for memory errors and to validate respondents’ answers, caregivers’ survey responses were linked to the APAL administrative data that tracked the families to whom the APAL worker reported they delivered the services. As indicated in Figure 1, slightly over half of the families (n = 120, 53.5%) in the intervention group were exposed as intended to the treatment by being contacted by APAL caseworkers (treatment compliers). A significant portion of the intervention group (n = 104, 46.5%), however, did not receive any of the APAL services either because they refused, were not locatable, or were not offered the services by APAL workers (treatment no shows). With respect to program differentiation, almost all participants (n = 208, 97.6%) assigned to the comparison group did not have access to any of the APAL services (comparison compliers), but a few of them (n = 5, 2.4%) did crossover to the treatment condition and received the full APAL service (treatment crossovers). This imperfect implementation of the APAL intervention argues for supplementing the analysis of estimating the effect of being assigned to the APAL intervention with an analysis of estimating the effect of actually receiving the APAL services.

Measures

Unmet service needs

The primary proximal outcome investigated in this study was the difference in the average level of unmet need between the intervention and the treatment groups. Caregivers were presented with a list of postpermanency services and asked whether the target child needed any of them, and if so, how successful they were in obtaining the needed services and supports. The variable was the number of identified child needs for which the caregiver reported not receiving services, ranging from 0 to 5 (M = .27, SD = .71).

Caregiver commitment

To gauge caregivers’ subjective attitudes toward maintaining the adoption or guardianship arrangement, we developed a caregiver commitment scale (White & Liao in progress) based on the attachment concept which is closely related to the extent to which the caregiver is motivated to invest in an enduring relationship with the child. This scale included 7 items and assessed how likely the parent would end the relationship if he or she could, how often the parent thinks of ending the relationship, how close the parent and child are, and how the relationship affects the parent and the child. Responses to each question were based on a 5-point Likert-type scale, with higher scores representing higher levels of caregiver commitment to the youth. All the responses were summed to create a composite score which ranged from 12 to 31 (M = 27.14, SD = 3.28). Although the caregiver commitment scale has not been validated by other studies, the Cronbach’s α for the scale is .73, which is considered acceptable for a short instrument (Rubin & Babbie, 2013).

Behavior problems index (BPI)

The standardized instrument, Behavior Problems Index (BPI), was used to measure the frequency and range of child problem behaviors. The instrument was developed by Peterson and Zill (1986) for children aged 4 years and older. The overall score is based on caregivers’ answers to 28 items. Caregivers were asked to rate as “often true,” “sometimes true,” or “not true” the occurrence of each behavior of the youth over the past 3 months. Following the scoring instructions for the instrument, BPI items were dichotomized into two categories, where “yes” category (coded as 1) indicates “often true” or “sometimes true” and “no” category (coded as 0) stands for “not true.” In addition, 1 item (cries too much) was deleted when computing the score for youth aged 12 and older based on the scoring guidelines of the instrument. All items were standardized and summed to create a composite score of overall BPI which ranged from 0 to 27 (M = 10.14, SD = 7.54). The Cronbach’s α for the scale is .91.

Postpermanency discontinuity

The distal outcome of placement discontinuity was created using the placement history records obtained from the IDCFS Integrated Database that tracks each child’s placement event since they were brought into care. In this study, a child is coded as having experienced discontinuity if any of the following conditions were met: (1) the child reentered state custody after adoption or guardianship or (2) the adoption or guardianship subsidy payment ended prior to the child’s 18th birthday. To calculate the long-term impact of the program on discontinuity, children were tracked through December 31, 2012. This variable was coded as 1 if the youth met at least one of the criteria of discontinuity listed previously after the APAL program was implemented. A total of 28 youth experienced placement discontinuity after adoption or guardianship finalization.

Covariates

Covariates were selected based on a review of previous postpermanency studies and included additional variables that showed a statistically significant difference between the APAL and the comparison group (see Table 1). Child’s age at interview was measured as a continuous variable in years. Child’s race was represented by two categorical variables, including White (=1) and Hispanic (=1), and African American was the reference group. Disability was a dichotomous variable and defined as the child having a physical health problem, mental/emotional disorder, or receiving special education (1 = had disability; 0 = had no disability). The number of placement changes after finalization but prior to the interview is a continuous variable. Adoption is a dichotomous variable (1 = adoption; 0 = guardianship). Caregiver’s relation to the child was represented by two categorical variables, including distant kin (1 = cousins, great-grandparents, or great aunts and uncles; 0 = other) and nonkin foster parents (1 = biologically unrelated foster caregivers; 0 = other), and close kin (grandparents, uncles, or aunts) was the reference group. Received monthly subsidy and medical insurance were both measured as a dichotomous variable (1 = the family was receiving a monthly subsidy or medical card; 0 = the family was not receiving a monthly subsidy or medical card). Caregivers were also asked whether they felt the subsidy adequately covered the expenses they incurred for the child (1 = yes; 0 = no). The family income variable was a dichotomous variable (1 = annual income above US$30,000; 0 = annual income equal to or under US$30,000).

Table 1.
Characteristics of Study Participants in the Intervention and Comparison Groups.

Variables Intervention Group (n = 224) Comparison Group (n = 213) Missing Cases

Weighted Mean/% SE Weighted Mean/% SE

Child characteristics

Age at interview 15.34 .11 15.91 .14 0

Age at finalization 7.99 .19 8.19 .22 0

Female 48.1% 41.8% 0

African American† 96.2% 92.1% 0

Hispanic 1.4% 2.8% 0

Non-Hispanic white 2.1% 3.9% 0

Child has a disability† 42.0% 51.9% 0

Number of child needs 2.29 .18 2.55 .21 0

Child has contact with birth family 92.0% 95.7% 0

Number of placement changes after finalization .25 .04 .31 .05 0

Abuse allegation at initial removal 4.4% 8.0% 0

Neglect allegation at initial removal 44.2% 45.0% 0

Substance exposed at birth 9.3% 7.2% 0

No prior indicated allegation 42.0% 39.9% 0

Adoption* 58.0% 45.2% 0

Guardianship* 42.0% 54.8% 0

Caregiver characteristics

Age at interview 55.80 .86 57.02 .93 21

Educational level 13.56 .22 12.86 .22 3

Married couple 28.5% 22.5% 0

Unmarried caregiver with other adults 22.9% 24.9% 0

Lone caregiver 48.0% 51.8% 0

Nonkin foster parent† 12.5% 19.9% 0

Close kin 76.8% 69.5% 0

Distant kin 10.8% 10.6% 0

Received monthly subsidy* 99.4% 90.9% 5

Amount of subsidy is adequate 32.5% 31.8% 19

Medical insurance for child*** 99.1% 92.5% 3

Income above US$30,000* 35.1% 24.7% 0

Size of informal support network 28.55 .86 27.58 .86 5

Note. SE = standard error.

†p < .10. p < .05. p < .01. p < .001 (two-tailed test).

Analysis Strategies

The primary analysis based on group assignment estimates the effect of being assigned to the APAL services, in which all 224 participants allocated to the APAL intervention group regardless of whether they received the intervention or not are compared with the entire comparison group of 213 children (see Figure 1). When there is incomplete service receipt, however (as indicated by the treatment no shows in this study), a supplementary analysis based on implementation integrity may be used to estimate the effect of actually receiving the full APAL services. In the past, the estimation of treatment received might have been attempted simply by comparing the 120 recipients (compliers) in the treatment group to the 208 nonrecipients in the comparison group (compliers). This straightforward approach suffers from a potentially serious selection bias in that the compliers with the assigned treatment are unlikely to be a random subset of all participants who were originally assigned to receive the treatment (Shadish et al., 2002; Testa, 2010). To account for selection biases that arise from incomplete implementation, instrumental variables (IVs) can be used for estimating treatment effects under conditions of incomplete compliance with assigned treatment intentions. Two-stage least squares (2SLS) procedure in STATA 10.1 was used to conduct the IVs estimation using youth age as an instrument.

The youth age dummy variable (13, 16 or 12, 14, 15, and 17) which was used to assign the two groups is considered as an instrument. There are two key criteria for identifying a valid instrument (called Z here). The first is that Z has a nonzero effect on the APAL-received variable; that is, receipt of APAL services is substantially higher for the intervention than the comparison group. As previously noted, half (53.5%) of the families in the intervention group received APAL services as intended compared to only 2.4% that crossed-over from comparison group. This difference in treatment rates can be expressed in a linear probability model (LPM) as follows: (APAL–received)_i = α₀+α₁ Z _i. Substituting the previously mentioned proportions on the right side of the equation yields 2.4% + 51.1%Z. Clearly, the assumption that Z has a nonzero effect on receipt of treatment is satisfied in this study. The second assumption is that Z is uncorrelated with any remaining influences on the outcome of interest left over in the “error box” (Freedman, 2010) after emptying out all other systematic influence on the outcome. This assumption is called the exclusion restriction, which means any effect of Z on the outcome is indirectly exerted solely through its effect on the treatment received (Angrist, Imben, & Rubin, 1996). It is plausible to make the assumption that age at treatment assignment has no direct effect, after other age-related variables, such as youth age at interview and number of placement changes after finalization, are taken into account.

Results

Characteristics of APAL Intervention and Comparison Group

Results of t-tests and χ² tests of group differences across a variety of sociodemographic characteristics are presented in Table 1. As indicated, the mean age of youth at interview was about 6 months older than in the comparison group. Nearly all of the youth in the intervention group were African American (96.2%), which was slightly higher than the comparison groups (92.1%). A substantial proportion (42%) in the intervention group had a disability such as a physical health problem, a mental/emotional disorder, and/or a learning disability. This proportion was nominally higher in the comparison group (51.9%). There were significantly more adoptions and fewer guardianships in the APAL group than in the comparison group. In addition, the difference for nonkin parents was nominally significant: 12.5% in the APAL group and 19.9% in the comparison group. Finally, a higher proportion of families received permanency subsidies and health coverage benefits for the children in the APAL group than in the comparison group. There was a significantly larger proportion of families with annual household incomes above US$30,000 in the intervention group. Other than these characteristics, the two groups were balanced on the rest of the covariates.

Bivariate Analyses of Outcome Differences

Results of the preliminary test of outcome differences are displayed in Table 2. Youth assigned to the APAL group had lower levels of unmet service needs than youth in the comparison group, but the difference was not statistically significant (p = .21). Caregivers in the intervention group showed significantly higher levels of posttreatment commitment to the permanence of the relationship than caregivers in the comparison group (p < .01). The effect size was Cohen’s d = .316 (95% CI: [.125, .507]) for this outcome. In addition, youth in the intervention group had significantly lower reported BPI scores by their caregivers than youth in the comparison group (p < .01). The effect size was Cohen’s d = −.293 (95% CI: [−.482, −.105]) for the behavior problems outcome. Youth in the APAL group also experienced marginally significant rates of placement discontinuity that were lower than youth in the comparison group (p < .10). The effect size was OR (odds ratio) = .506 (95% CI: [.228, 1.123]) for placement discontinuity.

Table 2.
Preliminary Analyses of Outcome Differences.

Variables Intervention Group Comparison Group

Weighted Mean/% SE n Weighted Mean/% SE n

Proximal and distal outcomes

Number of unmet service needs .22 .04 224 .33 .06 213

Caregiver commitment 27.65 .22 221 26.54 .27 206

Child BPI 9.03 .52 224 11.34 .55 213

Postpermanency discontinuity† 4.46 224 8.45 213

Note. BPI = behavior problems index; SE = standard error.

†p < .10. p < .05. p < .01. p < .001 (one-tailed test).

APAL’s Main Effects

Results for the effects of being assigned to APAL intervention on three proximal and the distal outcome when controlling for the covariates are presented in Table 3. As indicated in Model 1, the APAL assignment was associated with lower levels of posttreatment unmet service needs in the expected direction, but the difference was not statistically significant (B = −.083, p = .27, 95% CI: [−.230, .064]). Model 2 shows that being assigned to the APAL group was significantly associated with higher levels of posttreatment caregiver commitment (B = .859, p < .01, 95% CI: [.206, 1.513]), suggesting the caregiver commitment scores for youth assigned to the APAL group were .859 points higher than those assigned to the comparison group. In addition, Model 3 indicates that APAL assignment was significantly associated with lower levels of BPI scores (B = −1.421, p < .05, 95% CI: [−2.835, −.007]), indicating the BPI scores for youth assigned to the APAL group were 1.421 lower than those assigned to the comparison group. Finally, Model 4 shows that APAL assignment had a marginally significant association with lower odds of placement discontinuity (OR = .401, p < .10, 95% CI: [.151, 1.064]), showing youth assigned to the APAL intervention were 60% less like to have family discontinuity as of December 2012 than the comparison group did. In the next section, we examine how these findings compare with the analysis that focuses on the nonrandom subset of families that actually received the APAL intervention.

Table 3.
Weighted Multivariate Analyses of the Effects of Being Assigned to APAL Program on Outcomes.

Variables Model 1: Unmet Needs (N = 413) Model 2: Caregiver Commitment (N = 407) Model 3: Child BPI (N = 413) Model 4: Postpermanency Discontinuity (N = 387)

B SE B SE B SE OR SE

Intervention variable

APAL assignment −.083 .075 .859** .332 −1.421* .719 .401† .199

Covariates

Age at interview −.029 .022 −.059 .088 −.359† .208 .544*** .078

White .374 .313 .352 .581 1.040 1.758 — —

Hispanic −.186† .097 .782 .880 −4.149* 1.595 — —

Child has a disability .032 .075 −.468 .338 3.441*** 740 .607 .290

Number of placement changes after finalization .176* .076 −2.353* .405 4.255* .686 2.244* .724

Adoption .001 .074 .830* .337 −1.259† 732 .489 .234

Distant kin .243 .197 −.951 .641 .262 1.237 2.143 1.456

Nonkin foster parent .282* .127 .380 .449 2.786 1.056 .771 .546

Received monthly subsidy 1.047 .845 −2.90† 1.553 7.223 2.707 — —

Amount of subsidy is adequate −.152* .071 −.162 .346 −1.016 .713 .807 .434

Medical insurance for child −1.520 .574 1.316 1.251 −6.891 2.519 .250 .298

Income above US$30,000 .084 .083 .360 .357 −1.293† .752 1.275 .642

Constant 1.111 .736 29.516* 1.860 14.289* 3.976 — —

R ² .13 .24 .26 .15

Note. APAL = adoption preservation, assessment, and linkage; BPI = behavior problems index; OR = odds ratio; SE = standard error. The variable White, Hispanic, and received monthly subsidy were dropped automatically by the STATA program in Model 4.

†p < .10. p < .05. p < .01. p < .001 (two-tailed test).

APAL’s Effects of Receiving the Services

Table 4 shows the results of the IV estimation of receiving the APAL program on the three posttreatment, proximal outcomes of unmet service needs, caregiver commitment, and BPI scores. Column 1 presents results of the first stage, which is the LPM of the regression of APAL receipt on the age match without considering any covariates. Column 2 shows the results of the LPM controlling for the same covariates as those in Table 3. Both coefficients indicate that those children who were 13 or 16 years old assigned to APAL had a 53 percentage point higher probability of receiving the APAL services than those who were 12, 14, 15, or 17 years old. This is roughly the difference between the percentages of APAL compliers (53.5%) and the APAL crossovers (2.4%) noted previously. The similarity of the difference in APAL receipt rates with and without controls indicates that the assignment mechanism is plausibly ignorable even without conditioning on all the covariates included in the primary analysis. A rule of thumb for avoiding instruments that exert a weak effect on treatment receipt is that the F-statistic should be greater than 10 (Stock, Wright, & Yogo, 2002). The F-statistics in Columns 1 and 2 were above 10, indicating that age of 13 or 16 years old at the time of APAL assignment was a strong instrument.

Table 4.
Weighted IV Estimation Analysis of the Effects of Receiving APAL Program on Proximal Outcomes.

Outcome $⟶$ APAL Received Model 1: Unmet Needs Model 2: Caregiver Commitment Model 3: Child BPI

(1) (2) (3) (4) (5) (6) (7) (8)

Estimator OLS OLS OLS 2SLS OLS 2SLS OLS 2SLS

Age dummy: 13 or 16 at assignment 0.529 (0.038) 0.535* (0.039) −.083 (.075) .859** (.332) −1.421* (.719)

APAL received −0.155 (0.142) 1.618* (0.635) −2.658* (1.353)

F-statistic 195.05 15.63 3.23 3.19 6.54 6.15 12.24 12.11

N 437 413 413 413 407 407 413 413

Note. APAL = adoption preservation, assessment, and linkage; BPI = behavior problems index; IV = instrumental variable; OLS = ordinary least squares; 2SLS = two stage least squares. Model 1 includes no covariates. Standard errors are in parenthesis. Covariates being controlled for in Column 2–8 are the same as those in Table 3.

†p < .10. p < .05. p < .01. p < .001 (two-tailed test).

Column 3 displays the effect of being assigned to the services on unmet needs, which is the same as the coefficient estimate from Model 1 in Table 3. The coefficient doubles in magnitude when 2SLS is used to estimate the effect of actually receiving the APAL services. The same occurs for the two other estimates of the effects on caregiver commitment and child behavior problems. The message from these results is a simple one: Estimates based on treatment group assignment understate the magnitude of the APAL effect because of its dilution from mixing together compliers with no shows and crossovers from the comparison groups. This is a product of imperfect implementation integrity. But with a strong and valid instrument like the ignorable assignment mechanism of the youth age, it is possible to recover estimates of undiluted effects of the treatment on the treated. Ideally, we would have liked to use a 2SLS method to generate an estimate of receiving APAL on postpermanency continuity. However, most of the available software, such as IV-LPM or IV probit, is not consistent and reliable when both the outcome and the IVs are binary variables.

Discussion and Applications to Social Work

The findings of the APAL’s effect on caregiver commitment suggest that reassuring caregivers of the availability of ongoing support and conveying to them that they are not forgotten or left out may help moderate parenting stress and anxiety. APAL intervention is likely to fortify parents’ original sense of commitment to their children and can cultivate an empathetic awareness of the impact of adoption and legal guardianship on their children’s behaviors. These services can also potentially improve parenting skills and family functioning; promote family cohesion and adaptability; and reinforce caregivers’ commitment and motivation to maintain a lasting legal relationship with the children they have adopted or taken under their permanent guardianship.

The finding with respect to the outcome of child behavior problems echoes previous studies, which demonstrate that postpermanency services might assist parents in better understanding children’s needs and elevating awareness and abilities to access community resources to improve children’s behavior problems (Dhami, Mandel, & Sothmann, 2007; Goldsmith, 2002; Howard & Smith, 1995; Lenerz, Gibbs, & Barth, 2006). Although the effect size of −.293 was considered small statistically based on Cohen’s criteria (Cohen, 1988), the effect of APAL program has substantial practical meaning, given that 0.1 point improvement in the BPI score could make a huge difference for the well-being of nearly 50,000 adopted children and 17, 000 children taken into guardianship (DHHS, 2013) yearly.

Perhaps the more striking finding, considering the lengthy follow-up period after APAL intervention, is its potential effectiveness in reducing placement discontinuity for former foster children during adolescence. Five years after APAL implementation, age-eligible children assigned to the intervention registered significantly lower odds of postpermanency discontinuity of care (effect size was .506). The coefficient could be twice as large among treatment compliers. These findings are consistent with existing studies that have identified a positive role of postpermanency services in preventing out-of-home placement and placement disruption (Avery, 2004; Berry et al., 2006; Smith, 2006). Given that 52,000 children were adopted in 2012 and the discontinuity rate was estimated at 10% that will translate into preventing approximately 2,600 adopted children from eventually being replaced out of home each year.

In addition to showing possible positive program effects, this study also illustrated an approach for examining both the effects of being assigned to an intervention and the effects of actually receiving the treatment. Findings from the primary analysis can be of interest to policy makers who want a better estimate of the impact of rolling out a postpermanency program under real-world implementation conditions. For example, if the APAL program were to be implemented on a population-wide scale, it would be effective in reinforcing caregiver commitments even if only half of the families received full treatment as intended. The same beneficial program inference can be extended to child behavior problem and postpermanency continuity of care. On the other hand, relying solely on the treatment group assignment to estimate program effects understates the effect of the intervention on families who actually engage in the service. Since an estimated effect is always much less when implementation integrity is only partial or incomplete (Carroll et al., 2007), there is interest in discovering the potential impact if the program were to be fully implemented with integrity. This study demonstrated how a traditional program effect estimation based on group assignment could be supplemented with IVs methods to estimate the impact of receiving APAL under conditions of full implementation. The use of youth age at assignment as an instrument satisfied the first assumption of the IVs method that the instrument nontrivially boosts receipt of treatment.

The findings from this study offer some helpful insights to child welfare practitioners and postpermanency program evaluators. The APAL program appears to be conducive to motivating caregivers to sustain their commitments to raise adopted children and former wards to adulthood and increase postpermanency continuity of care. The incomplete implementation of the program, however, suggests that postpermanency services may need to be offered in a different way. The lack of full coverage may be an encouraging sign that most families are able to manage well on their own. There may even be simpler and less expensive ways of reaching caregivers. Conveying public appreciation for the sacrifices these families make and offering postpermanency services and support interventions through periodic contact by phone, the postal service, or e-mail may prove just as efficacious as more expensive in-home assessments.

The findings also alert program evaluators to the importance of measuring implementation integrity in order to generate accurate inferences about program impacts. This study confirms that full compliance with treatment assignment is unlikely to be the norm in most implementations of postpermanency services. This suggests there will always be a need to examine separately both the overall effect of simply offering the intervention to a target population and the specific effect of the population’s fully receiving the treatment as intended. By using an IVs approach, the estimated APAL program effect approximately doubled in magnitude. The results suggest that the estimates of effect sizes can be improved by identifying and applying ignorable assignment mechanisms that can be used as strong instruments in IVs estimation even with quasi-experimental and other observational studies. An excellent example of the IVs approach is Doyle’s (2013) study of the causal effects of foster care.

Future studies may consider using other methods of quasi-experimental evaluation, such as regression discontinuity design, to determine whether results are similar. Propensity score matching could be also used as a post hoc strategy to match the treatment group compliers with comparison group compliers to minimize observed and unobserved differences between the two groups. Future research testing the mediating effects through which the APAL program leads to the outcomes is warranted. In addition, it would also be interesting to detect whether the intervention works better for certain groups. Finally, given the research design evaluating the effects of APAL program was a two-group posttest only design, future research may need to incorporate a pretest and a randomization process to create two statistical equivalent groups, which aim to make a causal inference of the APAL intervention and enhance the internal validity of the study.

Limitations and Strengths

This study fills in some of the research gaps in evaluations of postpermanency services. Some limitations of the study should be acknowledged. First, the study design did not use pretests to measure the baseline of the outcomes, and it didn’t use random assignment to assign the two groups. Therefore, there may be other nonequivalences in both unmeasured pretreatment and unobserved factors that could be confounded with the estimated APAL effect and explain the observed outcome differences. The limitation embedded in this quasi-experimental design and the failure to control for the baselines of outcomes statistically might pose significant threats to the internal validity of the results. Second, the assumption that youth age at assignment is a valid IV is open to question. Because the study was not a randomized control trial, it is still possible that even after controlling for chronological age, the developmental stages of being 13 or 16 years old could still exert a direct effect on the outcomes of interest. The violation of this assumption also poses a threat to the validity of conclusions and interpretations about the efficacy of the APAL intervention and is generalizability to other populations.

Despite these limitations, this study contributes to evidence on the impact of APAL program on unmet service needs, caregiver commitment, behavior problems, and placement discontinuity of youth in adoptive or guardianship arrangements. This study conducted program fidelity measurement and two types of analysis were used to examine the effect of being assigned to and actually receiving the APAL program. The IV approach used in the supplementary analysis significantly reduced selection bias in the treatment-received variable, which generates a less diluted estimate of the program effect. Finally, tracking of the continuity of permanency placements for up to 5 years after the APAL intervention was implemented allows the examination of long-term impact of the program.

Variables	Intervention Group (n = 224)	Comparison Group (n = 213)	Missing Cases
Child characteristics
Age at interview**	15.34	.11	15.91	.14	0
Age at finalization	7.99	.19	8.19	.22	0
Female	48.1%		41.8%		0
African American†	96.2%		92.1%		0
Hispanic	1.4%		2.8%		0
Non-Hispanic white	2.1%		3.9%		0
Child has a disability†	42.0%		51.9%		0
Number of child needs	2.29	.18	2.55	.21	0
Child has contact with birth family	92.0%		95.7%		0
Number of placement changes after finalization	.25	.04	.31	.05	0
Abuse allegation at initial removal	4.4%		8.0%		0
Neglect allegation at initial removal	44.2%		45.0%		0
Substance exposed at birth	9.3%		7.2%		0
No prior indicated allegation	42.0%		39.9%		0
Adoption***	58.0%		45.2%		0
Guardianship***	42.0%		54.8%		0
Caregiver characteristics
Age at interview	55.80	.86	57.02	.93	21
Educational level	13.56	.22	12.86	.22	3
Married couple	28.5%		22.5%		0
Unmarried caregiver with other adults	22.9%		24.9%		0
Lone caregiver	48.0%		51.8%		0
Nonkin foster parent†	12.5%		19.9%		0
Close kin	76.8%		69.5%		0
Distant kin	10.8%		10.6%		0
Received monthly subsidy***	99.4%		90.9%		5
Amount of subsidy is adequate	32.5%		31.8%		19
Medical insurance for child***	99.1%		92.5%		3
Income above US$30,000*	35.1%		24.7%		0
Size of informal support network	28.55	.86	27.58	.86	5

Variables	Intervention Group	Comparison Group
Proximal and distal outcomes
Number of unmet service needs	.22	.04	224	.33	.06	213
Caregiver commitment**	27.65	.22	221	26.54	.27	206
Child BPI**	9.03	.52	224	11.34	.55	213
Postpermanency discontinuity†	4.46		224	8.45		213

Variables	Model 1: Unmet Needs (N = 413)	Model 2: Caregiver Commitment (N = 407)	Model 3: Child BPI (N = 413)	Model 4: Postpermanency Discontinuity (N = 387)
Intervention variable
APAL assignment	−.083	.075	.859**	.332	−1.421*	.719	.401†	.199
Covariates
Age at interview	−.029	.022	−.059	.088	−.359†	.208	.544***	.078
White	.374	.313	.352	.581	1.040	1.758	—	—
Hispanic	−.186†	.097	.782	.880	−4.149*	1.595	—	—
Child has a disability	.032	.075	−.468	.338	3.441***	740	.607	.290
Number of placement changes after finalization	.176*	.076	−2.353***	.405	4.255***	.686	2.244*	.724
Adoption	.001	.074	.830*	.337	−1.259†	732	.489	.234
Distant kin	.243	.197	−.951	.641	.262	1.237	2.143	1.456
Nonkin foster parent	.282*	.127	.380	.449	2.786**	1.056	.771	.546
Received monthly subsidy	1.047	.845	−2.90†	1.553	7.223**	2.707	—	—
Amount of subsidy is adequate	−.152*	.071	−.162	.346	−1.016	.713	.807	.434
Medical insurance for child	−1.520**	.574	1.316	1.251	−6.891**	2.519	.250	.298
Income above US$30,000	.084	.083	.360	.357	−1.293†	.752	1.275	.642
Constant	1.111	.736	29.516***	1.860	14.289***	3.976	—	—
R ²	.13	.24	.26	.15

Outcome $⟶$	APAL Received	Model 1: Unmet Needs	Model 2: Caregiver Commitment	Model 3: Child BPI
Age dummy: 13 or 16 at assignment	0.529** (0.038)	0.535*** (0.039)	−.083 (.075)		.859** (.332)		−1.421* (.719)
APAL received				−0.155 (0.142)		1.618* (0.635)		−2.658* (1.353)
F-statistic	195.05	15.63	3.23	3.19	6.54	6.15	12.24	12.11
N	437	413	413	413	407	407	413	413

Footnotes

Acknowledgment

The authors would like to thank Dr. Mary Keegan Eamon for her insightful comments on the earlier drafts of this article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Achenbach

. (1991). Manual for the child behavior checklist/4—18 and 1991 profile. Burlington, VT: University of Vermont Department of Psychiatry.

Angrist

Imbens

Rubin

(1996). Identification of causal effects using instrumental variables. Journal of American Statistical Association, 91, 444–472.

Avery

R. J

. (2004). Strengthening and preserving adoptive families: A study of TANF-funded post adoption services in New York State. Retrieved November 29, 2010, from http://nysccc.org/wp-content/uploads/tanfaverypasrpt.pdf

Barth

R. P.

Berry

Yoshikami

Goodfield

R. K.

Carson

M. L

. (1988). Predicting adoption disruption. Social Work, 33, 227–233.

Barth

R. P.

Miller

J. M

. (2000). Building effective post-adoption services: What is the empirical foundation? Family Relations, 49, 447–455. doi:10.1111/j.1741-3729.2000.00447.x

Barth

R. P.

Wulczyn

Crea

. (2005). From anticipation to evidence: Research on the Adoption and Safe Families Act. The Virginia Journal of Social Policy and the Law, 12, 371–399.

Berry

Propp

Martens

. (2006). The use of intensive family preservation services with adoptive families. Child and Family Social Work, 12, 43–53. doi:10.1111/j.1365-2206.2006.00426.x

Carroll

Patterson

Wood

Booth

Rick

Balain

. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 1–9. doi:10.1186/1748-5908-2-40

Child Welfare Information Gateway. (2012). Adoption disruption and dissolution. Washington, DC: U.S. Department of Health and Human Services, Children’s Bureau. Retrieved August 15, 2013, from https://www.childwelfare.gov/pubs/s_disrup.pdf

10.

Cohen

. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.

11.

Dane

A. V.

Schneider

B. H

. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23–45. doi:10.1016/S0272-7358(97)00043-3

12.

Dhami

M. K.

Mandel

D. R.

Sothmann

. (2007). An evaluation of post-adoption services. Children and Youth Services Review, 29, 162–179. doi:10.1016/j.childyouth.2006.06.003

13.

Doyle

J. J

. (2013). Causal effects of foster care: An instrumental-variables approach. Children and Youth Services Review, 35, 1143–1151. doi:10.1016/j.childyouth.2011.03.014

14.

Dusenbury

Brannigan

Falco

Hansen

. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18, 237–256. doi:10.1093/her/18.2.237

15.

Erich

Leung

. (1998). Factors contributing to family functioning of adoptive children with special needs: A long term outcome analysis. Children and Youth Services Review, 20, 135–150. doi:10.1016/S0190-7409(97)00070-4

16.

Festinger

. (2002). After adoption: Dissolution or permanence? Child Welfare, 81, 515–533.

17.

Freedman

D. A

. (2010). Statistical models and causal inference: A dialogue with the social sciences. New York, NY: Cambridge University Press.

18.

Freundlich

Wright

. (2003). Post-permanency services. Washington, DC: Casey Family Programs.

19.

Fuller

T. L.

Bruhn

Cohen

Lis

Rolock

Sheridan

. (2006). Supporting adoptions and guardianships in Illinois: An analysis of subsidies, services, and spending. Final Report. Urbana, IL: Children and Family Research Center.

20.

Goldman

G. D.

Ryan

S. D

. (2011). Direct and modifying influences of selected risk factors on children’s pre-adoption functioning and post-adoption adjustment. Children and Youth Services Review, 33, 291–300. doi:10.1016/j.childyouth.2010.09.012

21.

Goldsmith

. (2002). Post Legal Adoption Network (PLAN) Program Summary (pro-gen). New York, NY: New Alternatives for Children.

22.

Groze

Young

Corcran-Rumppe

. (1991). Post Adoption Resources for Training, Networking and Evaluation Services (PARTNERS): Working with special needs adoptive families in stress. Washington, DC: Prepared with Four Oaks, Cedar Rapids, Iowa, for the Department of Health and Human Services, Adoption Opportunities.

23.

Haugaard

J. J.

Hazan

. (2003). Adoption as a natural experiment. Development and Psychopathology, 15, 909–926. doi:10.1017/S0954579403000427

24.

Houston

Kramer

. (2008). Meeting the long-term needs of families who adopt children out of foster care: A three-year follow-up study. Child Welfare, 87, 145–171.

25.

Howard

J. A.

Smith

S. L

. (1995). Adoption preservation in Illinois: Results of a four year study. Normal: Illinois State University.

26.

Howard

J. A.

Smith

S. L

. (2003). After adoption: The needs of adopted youth. Washington, DC: CWLA Press.

27.

Lahti

M. F

. (2006). Maine adoption guides: A study of postlegalization services. In Dore

M. M.

(Eds.), The postadoption experience (pp. 111–134). Washington, DC: CWLA Press.

28.

Lenerz

Gibbs

Barth

R. P

. (2006). Postadoption services: A study of program participants, services, and outcomes. In Dore

M. M.

(Eds.), The postadoption experience (pp. 95–110). Washington, DC: CWLA Press.

29.

Leung

Erich

. (2002). Family functioning of adoptive children with special needs: Implications of familial supports and child characteristics. Children and Youth Services Review, 24, 799–816. doi:10.1016/S0190-7409(02)00240-2

30.

McDonald

T. P.

Propp

J. R.

Murphy

K. C

. (2001). The postadoption experience: Child, parent, and family predictors of family adjustment to adoption. Child Welfare, 80, 71–94.

31.

Peterson

J. L.

Zill

. (1986). Marital disruption, parent-child relationships, and behavior problems in children. Journal of Marriage and the Family, 48, 295–307. doi:10.2307/352397

32.

Prew

Suter

Carrington

. (1990). Post adoption family therapy: A practice manual. Salem, OR: Children’s Services Division.

33.

Reilly

Platz

. (2003). Characteristics and challenges of families who adopt children with special needs: An empirical study. Children and Youth Services Review, 25, 781–803. doi:10.1016/S0190-7409(03)00079-3

34.

Ryan

S. D.

Hinterlong

Hegar

R. L.

Johnson

L. B

. (2010). Kin adopting kin: In the best interest of the children? Children and Youth Services Review, 32, 1631–1639. doi:10.1016/j.childyouth.2010.06.013

35.

Rubin

Babbie

. (2013). Essential research methods for social work. 3rd ed. Belmont, CA: Books/Cole.

36.

Shadish

W. R.

Cook

T. D.

Campbell

D. T

. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin Company.

37.

Smith

. (2006). A study of the Illinois adoption/guardianship preservation program. In Dore

M. M.

(Eds.), The postadoption experience (pp. 67–94). Washington, DC: CWLA Press.

38.

Smith

Garnier

Howard

Ryan

. (2006). Where are we now: A post ASFA examination of adoption disruption. Adoption Quarterly, 9, 19–44. doi:10.1300/J145v9n04-02

39.

Stock

Wright

Yogo

. (2002). A survey of weak instruments and weak identification in GMM. Journal of Business and Economic Statistics, 20, 518–529. doi:10.1198/073500102288618658

40.

Testa

M. F

. (2004). When children cannot return home: Adoption and guardianship. The Future of Children, 14, 114–129.

41.

Testa

M. F

. (2010). Logic of results-oriented accountability: Population, intervention, comparisons, and outcomes. In Testa

Poertner

(Eds.), Fostering accountability: Using evidence to guide and improve child welfare policy (pp. 75–100). New York, NY: Oxford University Press.

42.

Testa

M. F.

White

K. R

. (2014). Insuring the integrity and validity of social work interventions: The case of the subsidized guardianship waiver experiments. Journal of Evidence-Based Social Work, 11, 157–172. doi:10.1080/15433714.2013.847260

43.

Tibbitts

Mike

. (2002). San Diego youth and community services post-legal adoption support program. San Diego, CA: San Diego Youth and Community Services.

44.

Timmer

S. G.

Sedlar

Urquiza

A. J

. (2004). Challenging children in kin versus nonkin foster care: Perceived costs and benefits to caregivers. Child Maltreatment, 9, 251–262. doi:10.1177/1077559504266998

45.

U.S. Department of Health and Human Services. (2013). The AFCARS report #20. Washington, DC: Administration for Children and Families, Children’s Bureau. Retrieved August 15, 2013, from http://www.acf.hhs.gov/sites/default/files/cb/afcarsreport20.pdf

46.

Webster

Barth

R. P.

Needell

. (2000). Placement stability for children in out-of-home care: A longitudinal analysis. Child Welfare, 79, 614–632.

47.

White

Liao

. (in progress). Testing the validity and reliability of a caregiver commitment scale for post-permanency population.

Variables	Intervention Group (n = 224)		Comparison Group (n = 213)		Missing Cases
Variables	Weighted Mean/%	SE	Weighted Mean/%	SE	Missing Cases
Child characteristics
Age at interview**	15.34	.11	15.91	.14	0
Age at finalization	7.99	.19	8.19	.22	0
Female	48.1%		41.8%		0
African American†	96.2%		92.1%		0
Hispanic	1.4%		2.8%		0
Non-Hispanic white	2.1%		3.9%		0
Child has a disability†	42.0%		51.9%		0
Number of child needs	2.29	.18	2.55	.21	0
Child has contact with birth family	92.0%		95.7%		0
Number of placement changes after finalization	.25	.04	.31	.05	0
Abuse allegation at initial removal	4.4%		8.0%		0
Neglect allegation at initial removal	44.2%		45.0%		0
Substance exposed at birth	9.3%		7.2%		0
No prior indicated allegation	42.0%		39.9%		0
Adoption***	58.0%		45.2%		0
Guardianship***	42.0%		54.8%		0
Caregiver characteristics
Age at interview	55.80	.86	57.02	.93	21
Educational level	13.56	.22	12.86	.22	3
Married couple	28.5%		22.5%		0
Unmarried caregiver with other adults	22.9%		24.9%		0
Lone caregiver	48.0%		51.8%		0
Nonkin foster parent†	12.5%		19.9%		0
Close kin	76.8%		69.5%		0
Distant kin	10.8%		10.6%		0
Received monthly subsidy***	99.4%		90.9%		5
Amount of subsidy is adequate	32.5%		31.8%		19
Medical insurance for child***	99.1%		92.5%		3
Income above US$30,000*	35.1%		24.7%		0
Size of informal support network	28.55	.86	27.58	.86	5

Outcome $⟶$	APAL Received		Model 1: Unmet Needs		Model 2: Caregiver Commitment		Model 3: Child BPI
	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)
Estimator	OLS	OLS	OLS	2SLS	OLS	2SLS	OLS	2SLS
Age dummy: 13 or 16 at assignment	0.529** (0.038)	0.535*** (0.039)	−.083 (.075)		.859** (.332)		−1.421* (.719)
APAL received				−0.155 (0.142)		1.618* (0.635)		−2.658* (1.353)
F-statistic	195.05	15.63	3.23	3.19	6.54	6.15	12.24	12.11
N	437	413	413	413	407	407	413	413