Challenges in Using Comparison Data in Child Welfare Evaluations

Abstract

Objective:

This article describes the evaluation of permanency roundtables, an intervention to help youth in foster care achieve legal permanency and the challenges evaluators faced in finding and using appropriate comparison data.

Method:

In 2009, permanency roundtables were conducted for 496 children in Georgia, most of whom had spent extended time in care. Half (50%) achieved legal permanency within 24 months of the roundtables.

Results:

Five potential sources of comparison data, none of which provide an ideal comparison, are presented and discussed.

Conclusions:

Ideally, evaluators should plan for one or more comparison groups at the very beginning of the evaluation. However, post hoc comparisons should not be dismissed altogether. The challenge for evaluators is to generate valid comparison groups to determine whether an intervention made a significant improvement, despite challenges such as limited funding, tight timelines, and unmeasured contextual differences between groups.

Keywords

child welfare foster care evaluation comparison data permanency

To most accurately determine whether an intervention has made a significant improvement in a given population, it is necessary to include a comparison or control group in the evaluation. Further, for those choosing an intervention, an important consideration is how the intervention group’s outcomes compared to a comparison group. Providing valid comparison data is a key step in moving interventions up the evidence ladder. While a randomized control study has historically been considered the gold standard in evaluation, implementing such a study design can be challenging, given issues such as the intended or unintended spread of the intervention (e.g., clinicians applying the intervention to clients who are not part of the intervention group under study) and the ethics of withholding treatment from a group that could truly benefit from it (Sibald & Roland, 1998). In the absence of a randomized control study, using comparison data is a common alternative (Gribbons & Herman, 1997; Shadish, Cook, & Campbell, 2001). However, locating and using valid comparison data can be challenging. This article describes the evaluation of an intervention designed to help youth in foster care achieve legal permanency and the challenges faced by the evaluators in finding and using appropriate comparison data. The evaluators wanted to determine (1) whether permanency rates for youth whose cases were included in permanency roundtables (PRTs) were higher than those for youth whose cases were not included in PRTs, (2) whether statewide permanency rates increased appreciably after the statewide spread of the permanency roundtable intervention, and (3) how permanency rates in the current study compared to other evaluations of the same intervention. To address these questions, five potential sources of comparison data were considered.

PRTs in Georgia

A PRT convenes a team of professionals to expedite permanency planning for children and youth in foster care by planning and implementing specific actions to be taken to overcome obstacles to achievement of legal permanency, defined as adoption, reunification, or guardianship. PRTs are unique in that their sole focus is on expediting legal permanency for youth. The PRT team members include the case manager, the case supervisor, a master practitioner (a permanency expert from within the agency), a permanency consultant (a permanency expert external to the agency), and a facilitator. The PRT team meeting follows a specific structure, which includes presentation of the case, clarifying questions, brainstorm of solutions to overcome permanency obstacles, and development of an action plan with specific steps to be taken in the following 6 months to achieve permanency. After the PRT, the case manager and supervisor are expected to meet monthly with the master practitioner to review progress toward permanency and to adjust action plans as needed. For a complete description of the process, see Rogg, Davis, and O’Brien (2009).

In Georgia, PRTs were conducted at the beginning of 2009 for 496 children and youth who were considered to be stuck in foster care. The vast majority of these children and youth were from Fulton and DeKalb counties in metropolitan Atlanta and were part of the Kenny A. consent decree, which was the result of a class action lawsuit brought forth because of the length of time they had spent in care. Some children from other counties who were experiencing difficulty achieving permanency were also included in the roundtables. The evaluation team sought to find suitable comparison data against which to compare permanency outcomes.

Challenges Finding Comparison Data for the PRTs in Georgia

It was challenging to find appropriate comparison data for the PRTs in Georgia for several reasons. First, due to the speed with which the intervention was planned and implemented, the evaluation team did not have time to plan for and incorporate comparison data at the beginning of the study. The roundtables were announced in November 2008 and began in January 2009, allowing for approximately 1 month of planning time. Time and resources dedicated to the evaluation were spent developing content for forms, planning for secure transmission of data, and other logistical issues.

Second, at the same time that the initial PRTs were being implemented, Georgia Division of Family and Children Services (DFCS) began emphasizing the importance of permanency statewide through discussions at leadership meetings and presentations throughout the state. Although the vast majority of cases included in the roundtables were youth included in the Kenny A. consent decree (Fulton and DeKalb counties), Georgia DFCS intentionally included youth from other counties; in keeping with the statewide emphasis on permanency, the purpose was to provide experience in conducting PRTs to case managers in all 17 of the state’s service delivery systems. Although intended to expedite permanency for youth whose cases are included in the roundtable, PRTs are also intended to help staff expand their skills in helping all youth achieve permanency. In other words, contamination was desirable from a practice standpoint but hampered the ability to draw a comparable comparison group. As a result, any within-state comparison groups were contaminated from the outset. After the initial set of roundtables was conducted in January and February 2009, roundtables occurred statewide; nearly 3,000 roundtables occurred statewide in 2009, and 1,266 occurred in 2010.

The third challenge was that the cases included in the initial set of roundtables included all of the youth who were required to be included because of their Kenny A. status. There were no comparable youth in Fulton or DeKalb counties in terms of age and length of stay because all youth whose length of stay exceeded a specified threshold were included in the PRTs, which were an approved response to the consent decree.

Finally, it was difficult to find a comparison group elsewhere within the state of Georgia, not only because each county had experience as part of the initial set of PRTs but also because of the differences in the foster care populations in Fulton and DeKalb counties (urban areas in which 92% of the foster care population is African American, for example) and in populations in other counties (more rural and suburban areas in which a smaller percentage of the foster care population is African American). Even if the spreading of practice had not resulted in contamination in other counties, the differences in demographic characteristics and differences among urban, suburban, and rural settings were a limiting factor.

Method

Participants

Of the 496 children and youth whose cases were included in the roundtables, 57% were male and 43% female. Most (92%) were African American and 7% were White. At the time of the roundtables, their median age was 13 (range 1–18 years) and the median length of time in care was 52 months (range 1–214 months). The vast majority of the youth (91%) were from Fulton and DeKalb counties in metropolitan Atlanta and were part of the Kenny A. class action lawsuit.

Measure and Procedure

The evaluation included numerous predictors of legal permanency, including child characteristics, family information, child welfare experience, and factors related to the PRTs. A full description of the measures can be found in O’Brien, Davis, Morgan, Rogg, and Houston (2012). The primary outcome of interest in the current article is achievement of legal permanency within 24 months of the roundtable.

Roundtables occurred in January and February 2009, and outcomes were tracked 12 and 24 months later (for more information about the 12- and 24-month outcomes, see O’Brien, Davis, Morgan, Rogg, & Houston, 2012; Rogg, Davis, & O’Brien, 2011). Data on predictors and outcomes came from databases and information collected from case managers at four time points, namely, prior to the roundtable, at the roundtable, and 12 and 24 months after the roundtable. The project was deemed exempt from IRB review by the Georgia DFCS. A full description of the procedures can be found in Davis et al. (2013).

The following five alternative comparison groups were examined in this article:
matched comparison data,

statewide population of youth in foster care (Kenny A. monitoring reports),

social services quality reviews (using Child and Family Services Review [CFSR] standards),

state and county data on length of time in care and permanency rates, and

other evaluations of PRTs

One comparison aimed to determine whether 496 youth whose cases were included in the initial intervention achieved legal permanency at a higher rate than youth who did not receive a roundtable. Three comparisons aimed to determine whether statewide rates of achieving legal permanency rose after the roundtables, given the immediate spread of the practice throughout the state. Finally, the last comparison examined the rates of permanency achieved by the intervention in Georgia compared to rates of permanency achieved by the same intervention in other jurisdictions. The procedure for forming and analyzing each comparison group is described along with the results subsequently.

Results

Permanency Outcomes

Twelve months after the PRTs, 31% of the sample had achieved legal permanency, and 24 months after the roundtables, 50% had achieved legal permanency (Table 1). Twenty-four months after the PRTs, the rates of achievement of permanency differed by age group, with the youngest children (age 0–6) achieving a permanency rate of 76.3% and the oldest youth (age 13–18) achieving a rate of 34.2%. A fuller description and discussion of outcomes can be found in Davis et al. (2013).

Table 1.
Legal Status 24 Months After the Roundtables for the 496 Children by Age Group.

Outcome Age Group at the Time of Roundtable Total

0–6 7–12 13–18

Legal permanency achieved Adoption 39.8% 30.6% 11.3% 21.8%

Guardianship 20.4% 19.4% 17.5% 18.5%

Reunification 16.1% 12.7% 5.6% 9.5%

Any legal permanency 76.3% 62.7% 34.2% 49.8%

Emancipation (aging out of foster care after age 18) n/a n/a 36.4% 19.8%

Custody termination (e.g., court relieved agency of custody due to child’s runaway status) 0% 0% 1.9% 1.0%

Remaining in care 23.7% 37.3% 27.5% 29.4%

Total number of children 93 134 269 496

Comparison Groups

Five potential comparison groups are discussed. Table 2 lists each potential comparison group and its intended purpose.

Table 2.
Comparison Groups for the 496 PRT Youth.

Comparison Group Purpose

1. Matched comparison data Compare permanency rates for PRT youth with non-PRT youth

2. Statewide population of youth in foster care (Kenny A. monitoring reports) Examine statewide permanency rates over several years to determine whether roundtables have impacted permanency rates statewide

3. Social services quality reviews (using Child and Family Services Review standards)

4. State and county data on length of time in care and permanency rates

5. Other evaluations of permanency roundtables Compare permanency rates in the current study with other evaluations of the same intervention

Note. PRT = permanency roundtable.

Matched Comparison Data

Description

The evaluation team sought to create a matched comparison group to determine whether youth who had received a PRT achieved permanency at a higher rate than youth who did not receive a PRT. Propensity score matching (using caliper matching; see Caliendo & Kopeinig, 2008) was used to develop a data set of matched cases from available statewide data of children in care in Georgia. An event history analysis using these matched data was planned to compare the roundtabled children with the non-roundtabled children on the length of time to permanency and percentage achieving legal permanency.

Expectations

It was anticipated that these analyses would yield differences in days to permanency (survival days) between January 1, 2009, and December 31, 2010 (approximately 2 years after the roundtables), for youth whose cases were and were not roundtabled. Specifically, it was expected that youth who were roundtabled would experience higher rates of legal permanency and spend fewer days in care than youth who were not roundtabled.

Procedure

A data set of matched records (N = 19,003) was created from three separate data sets (2008, 2009, and 2010) of children under 18 and in care in Georgia on January 1 of each year. From this data set, a subset of cases (n = 9,484) was created that excluded cases with:
most recent exit before January 1, 2009,

date of birth before January 1, 1991 or after January 1, 2009 (over 18 or not yet born on January 1, 2009),

earliest removal or entry into care after January 1, 2009, and

a child’s case possibly included in the roundtables (while this was not possible to confirm, those cases appearing on the state’s roundtables spreadsheet were excluded from the data set).

An indicator variable was created to designate whether a child’s case was included in the PRTs as part of original roundtable project in January and February 2009 (n = 493).

Because of disparities in such variables as age, race/ethnicity, geographic region, length of stay, presence of a disability, and placement settings between children whose cases were and were not included in the roundtables, propensity score matching was used to select cases for comparison analyses of time to permanency and achieving legal permanency (reunification, guardianship, or adoption). Youth who received roundtables were more likely to be older, live in the metropolitan Atlanta area, be African American, have a disability, not to be living in a family foster home, and to have been in care longer.

The variables listed above (plus gender) were included in the logistic model as covariates to calculate the propensity score (i.e., the propensity to receive the roundtable intervention), which was then used to select matched cases. Table 3 presents the cases used in the logistic regression analysis. From the original 9,484 cases considered for matching, 7,141 had complete data and were therefore used in the analysis. All model covariates were significant, except for gender and type of placement setting.

Table 3.
Case Counts for Propensity Score Logistic Regression.

Analysis of Children Included in Initial Roundtable Project Number of Cases

Total number of cases considered for matching 9,484

Deleted due to missing values or possible roundtable involvement (missing data; also includes those roundtabled after June 30, 2009) 2,343

Used 7,141

Roundtabled 493^a

Not roundtabled 6,648

^aThree children achieved permanency immediately before their roundtable occurred; their data are not included in this regression.

The model predicted actual intervention status (i.e., whether a child was roundtabled) for 96% of the cases. Unfortunately, given that the youth who participated in the roundtables represented such a specific population, it was difficult to find reasonable matches in the comparison data set. As a result, the matching process yielded matches for only about half (53%) of the children roundtabled as part of the original roundtable project. The number of matched cases identified was 518 (259 from the PRT group and 259 matches). In addition, the PRT (intervention) and control groups were different on several of the variables on which they were matched.

Findings

Because only 259 of the 493 roundtabled cases resulted in matches and because the groups were not sufficiently matched with the key variables, analyses were not performed.

Discussion and limitations

While propensity score matching is an effective way to develop a comparison group that is similar on given characteristics (e.g., demographic characteristics, risk factors, and placement history), it was not used in this evaluation because of the limited number of youth matches. Further, some researchers have indicated that propensity score analysis does not result in substantially different results compared to using traditional multivariate modeling (Freedman & Berk, 2008; Stürmer et al., 2006).

Statewide Population of Youth in Foster Care (Kenny A. Monitoring Reports)

Description

As part of the consent decree, monitoring reports were produced every 6 months to assess compliance with the terms of the decree. Permanency outcomes were tracked according to length of time in care. Table 4 summarizes findings from the reports for permanency-related outcomes over 10 reporting periods (from January–June 2007 through July–December 2011). For most outcomes, a target goal is included.

Table 4.
Kenny A. Monitoring Report Data: 2007 Through 2011.

Kenny A. Monitoring Report Data Target 2007 2008 2009 2010 2011

Jan–Jun Jul–Dec Jan–Jun Jul–Dec Jan–Jun Jul–Dec Jan–Jun Jul–Dec Jan–Jun Jul–Dec

Entry on or after consent decree (October 27, 2005; Outcome 8 group) n= 2,803 3,395 4,031 4,679 5,171 5,603 6,085 6,475 7,074 7,465

Positive permanency (reunification, permanent relative placement, adoption, guardianship, or permanent legal custody) within 12 months of entry for children who had been in care 12 months or less at the end of the prior reporting period (Outcome 8a) 40% 45%^a d = −0.11 (−0.16 to 0.06) 48% d = −0.04 (−0.09 to 0.00) 49% d = −0.02 (−0.07 to 0.02) 51% d = −0.04 (−0.09 to 0.00) 50% 53%^b d = 0.07 (0.02 to 0.11) 54%^b d = 0.09 (−0.05 to 0.13) 54%^b d = 0.09 (0.05 to 0.13) 54%^b d = 0.09 (0.05 to 0.13) 55%^b d = 0.11 (0.07 to 0.15)

Positive permanency (reunification or permanent relative placement within 12 months of entry; permanent legal custody, adoption or guardianship within 24 months of entry) (Outcome 8b) 74% 46%^a d = −0.22 (−0.27 to 0.17) 50%^a d = −0.13 (−0.18 to 0.09) 53%^a d = −0.07 (−0.11 to 0.02) 56% d = 0.00 (−0.04 to 0.04) 56% 59%^b d = 0.07 (0.03 to 0.11) 61%^b d = −0.11 (0.07 to 0.16) 62%^b d = 0.14 (0.10 to 0.18) 61%^b d = 0.11 (0.07 to 0.15) 62%^b d = −0.14 (−0.10 to 0.18)

Positive permanency (regardless of time frame) n/a 47%^a d = −0.41 (−0.46 to 0.36) 54%^a d = −0.25 (−0.30 to 0.20) 57%^a d = −0.19 (−0.23 to 0.14) 62%^a d = −0.07 (−0.12 to 0.03) 65% 69%^b d = 0.10 (−0.06 to 0.14) 73%^b d = 0.21 (0.16 to 0.25) 75%^b d = 0.26 (0.22 to 0.31) 75%^b d = 0.26 (0.22 to 0.31) 76%^b d = 0.29 (0.25 to 0.34)

Other exits^c n/a 6%^a 6%^a 7%^a 8% 9% 10% 10% 10% 10% 10%

In care up to 24 months at time of decree and still in care at beginning of time period (Outcome 9 group) n= 625 470 334 247 183 140 106 74 49 32

Positive permanency 40% 20% d = 0.04 (−0.20 to 0.27) 20% d = 0.04 (−0.20 to 0.27) 19% d = 0 (−0.25 to 0.25) 21% d = 0.07 (−0.20 to 0.23) 19% 19% d = 0 (−0.31 to 0.31) 24% d = 0.16 (−0.16 to 0.48) 24% d = 0.16 (−0.19 to 0.52) 27% d = 0.25 (−.15 to 0.65) 9% d = 0.48 (−1.17 to 0.22)

Other exits^c n/a 6% d = 0.11 (−0.30 to 0.52) 5% d = 0 (−0.43 to 0.43) 14%^a d = 0.62 (−0.22 to 1.03) 14%^a d = 0.52 (0.13 to 0.91) 5% 6% d = 0.11 (−0.42 to 0.64) 7% d = 0.20 (−0.35 to 0.75) 9% d = 0.35 (−0.22 to 0.92) 8% d = 0.28 (−0.40 to 0.95) 19%^a d = 0.82 (0.21 to 1.43)

In care more than 24 months at time of decree and still in care at beginning of time period (Outcome 10 group) n= 414 349 281 219 161 129 95 66 43 35

Positive permanency 45% 9% d = −0.06 (−0.40 to 0.28) 9% d = −0.06 (−0.41 to 0.28) 14% d = 0.21 (−0.13 to 0.55) 15% d = 0.26 (−0.09 to 0.60) 10% 16% d = 0.30 (−0.09 to 0.68) 18% d = 0.38 (−0.03 to 0.78) 23%^b d = 0.55 (−0.12 to 0.97) 9% d = 0.06 (−0.56 to 0.69) 6% d = −0.31 (−1.13 to 0.51)

Other exits^c n/a 7% d = −0.15 (−0.51 to 0.21) 8% d = −0.07 (−0.44 to 0.30) 24%^a d = 0.64 (0.31 to 0.97) 27%^a d = 0.73 (0.39 to 1.07) 9% 11% d = 0.12 (−0.30 to 0.55) 13% d = 0.23 (−0.22 to 0.67) 12% d = 0.18 (−0.33 to 0.68) 9% d = 0 (−0.65 to 0.65) 17% d = 0.40 (−0.17 to 0.97)

Note. The reporting period for the first half of 2009 (January through June; italicized) is used as the comparison group because it is the period during which the roundtables took place. Effect sizes (d) and associated 95% confidence intervals are reported for each comparison. ^aThe rate is significantly less during a given reporting period compared to the first half of 2009, p < .05. ^bThe rate is significantly more during a given reporting period compared to the first half of 2009, p < .05. ^cIncludes emancipation and transfer to other county or state.

Source. Monitoring Reports for Periods III through VII, Kenny A. v. Perdue (Accountability Agents James T. Dimas and Sarah A. Morrison), available at aysps.gsu.edu/KennyA.

Expectations

It was expected that permanency outcomes for youth included in the consent decree would improve in the reporting periods after the roundtables.

Procedure

Monitoring reports were located and outcomes before and after the roundtables were summarized.

Findings

The Kenny A. Monitoring Reports track results for youth in the “Outcome 8” group, which consists of youth who entered foster care on or after October 27, 2005, the date of the consent decree. A total of 113 youth whose cases were included in the PRTs entered care on or after this date. Results for Outcome 8a (Positive permanency [reunification, permanent relative placement, adoption, guardianship, or permanent legal custody] within 12 months of entry for children who had been in care 12 months or less at the end of the prior reporting period) consistently exceeded the target of 40%. The rate of achieving positive permanency as defined by Outcome 8a was significantly higher during each of the five reporting periods after the roundtables (range 53–55%) compared to the reporting period during which the roundtables took place (50%). In contrast, results for Outcome 8b (Positive permanency [reunification or permanent relative placement within 12 months of entry; permanent legal custody, adoption, or guardianship within 24 months of entry]) have yet to attain the target of 74%. The rate of achieving positive permanency as defined by Outcome 8b was significantly higher during each of the five reporting periods after the roundtables (range 59–62%) compared to the reporting period during which the roundtables took place (56%).

The “Outcome 9” group includes youth who were in care for up to 24 months at the time of the decree and were still in care at the time of a given reporting period. As would be expected and desired, the number of youth in this group has decreased over time—from 625 in the first half of 2007 to 32 in the second half of 2011. The target for the percentage of youth in this group attaining permanency (40%) has not been met in any reporting period to date. However, the percentage of youth in this group achieving permanency was highest (ranging from 24% to 27%) in three of the reporting periods after the roundtables occurred. Only 9% of youth exiting care from this group in July–December 2011 achieved permanency.

The “Outcome 10” group includes youth who were in care for more than 24 months at the time of the decree and were still in care at the time of a given reporting period. As with Outcome 9, the number of youth in this group has decreased over time—from 414 in the first half of 2007 to 35 in the second half of 2011. The target for the percentage of youth in this group attaining permanency (45%) has not been met in any reporting period to date. As with Outcome 9, the percentage of youth in this group achieving permanency was highest (ranging from 16% to 23%) in three of the post-roundtable reporting periods. Only 6% of the youth exiting care from this group in July–December 2011 achieved permanency.

Discussion and limitations

Overall, examination of statewide data using Kenny A. monitoring reports suggests that the roundtables may have helped to improve outcomes for some youth in the 24 months after the roundtable. The percentage of youth in the Outcome 8 group (entry on or after the date of the consent decree) achieving permanency was higher in the reporting periods during and after the roundtables compared to earlier reporting periods. In addition, the rise in the percentage of youth in Outcomes 9 and 10 achieving permanency after the roundtables took place is noteworthy, particularly since it is likely that the youth who continued to remain in care were cases for which permanency was more difficult to attain. This comparison data, however, does not include individual-level data, which would allow for a comparison of outcomes by important demographic characteristics such as age. In addition, youth who were included in the roundtables are represented in the statewide data, and it is not possible to disentangle their outcomes from those of youth who were not included in the roundtables.

Social Services Quality Reviews Using CFSR Standards

Description

As a potential comparison group, the evaluation team examined Georgia’s social services quality reviews from 2007 to 2011. CFSRs are periodic reviews required by the Children’s Bureau to monitor outcomes in state child welfare systems.¹ The two-stage reviews, which included a statewide assessment and an on-site visit, encompassed three major outcome areas, namely, safety, permanency, and well-being.

The CFSR permanency area includes the following two outcomes: (1) Permanency 1: children have permanency and stability in their living situations and (2) Permanency 2: the continuity of family relationship and connections is preserved for children.

Because of the relevance to PRTs, the Permanency 1 outcome and specific relevant items within that outcome are presented in Table 5. The state is regarded as in “substantial conformity” with each item if at least 90% of the reviewed cases have the item rated as a strength. It is important to note that Fulton and DeKalb counties are excluded from the report because of the Kenny A. consent decree.

Table 5.
Georgia Social Services Quality Review Outcomes Tracking.

Georgia Social Services Quality Review Outcomes Tracking Percentage of Cases Reviewed in Which Item is Rated as a “Strength'

Federal Standard FFY 2007 FFY 2008 FFY 2009 FFY 2010 FFY 2011

Children have permanency and stability in their living situations (Permanency 1) 95% 44% 31% 29% 34% 41%

Permanency goal for the child (Item 7) 90% 58% 44% 49% 52% 64%

Reunification, guardianship, or permanent placement with a relative (Item 8) 90% 61% 48% 43% 48% 52%

Adoptions (Item 9) 90% 38% 29% 29% 38% 47%

Not having a permanency goal of other planned permanent living arrangement (Item 10) 90% 68% 74% 79% 82% 86%

Note. Georgia changed to a regional case review process in federal fiscal year 2007. Data collected prior to that change were based on case samples pulled from selected counties. Beginning federal fiscal year (FFY) 2007, case samples were pulled by region so that all counties were potentially included in the review process, with the exception of Fulton and DeKalb counties, which fall under the Kenny A. Consent Decree review process for foster care cases. Federal fiscal years are from October through September; for example, FFY 2007 is October 1, 2006, through September 30, 2007. Because sample sizes were not available, effect sizes are not reported.

Expectations

It was expected that permanency outcomes would improve in the reporting periods after the roundtables because of the statewide adoption of the intervention.

Procedure

Social services quality review reports were located and outcomes before and after the roundtables were summarized.

Findings

As Table 5 indicates, Georgia has not achieved substantial conformity for the overall Permanency 1 outcome area, though the percentage of cases achieving the outcome increased from 2009 to 2011. In addition, Georgia has not achieved substantial conformity for most of the individual permanency items. Permanency goal for the child (Item 7); reunification, guardianship, or permanent placement with a relative (Item 8); and adoptions (Item 9), after decreasing from 2007 to 2008, increased from 2009 to 2011. Adoptions, in particular, made a substantial gain from 29% in 2008 and 2009 to 47% in 2011. Not having a permanency goal of other planned permanent living arrangement (Item 10) has shown steady progress over the 5 years. Note that it was not possible to do tests of statistical significance because the sample sizes were not available.

Discussion and Limitations

The outcomes reported in the social services quality review reports do not correspond directly with the outcome of interest in the current study, that is, achievement of legal permanency within 24 months of an intervention. However, overall examination of the social services quality review reports suggests that the roundtables may have helped to improve outcomes for youth statewide. The reports indicate improved outcomes—that is, increased rates of cases for which the federal CFSR permanency items were rated as strengths—beginning in 2009, when the statewide spread of roundtables began. It is not possible to directly tie the statewide spread of roundtables to these improved outcomes but the data trend in a positive direction concomitant with the spread of roundtables statewide. In addition, as with the Kenny A. monitoring reports, this comparison data does not include individual-level data, which would allow for a comparison of outcomes by important demographic characteristics such as age.

State and County Data on Length of Time in Care and Permanency Rates

Description

Because the state began implementing the roundtables statewide almost immediately after the first set of roundtables (in January and February 2009), it was not possible to compare permanency rates in other counties to those in Fulton and DeKalb counties to assess the project’s impact. As such, the evaluation team decided to examine 3 years’ data of state and county data to determine whether the roundtables had caused permanency rates to increase throughout the state.

Expectations

It was expected that the length of time in care would decrease and permanency rates would increase after the PRTs.

Procedure

The evaluation team received data from Georgia DFCS on children in care in 2008 (the year prior to PRT implementation), 2009 (the year in which PRTs were first implemented), and 2010. These data included the child’s demographic characteristics (age, gender, and race/ethnicity), date of removal, placement type, disability status, and outcome. However, since the state system did not track which youth had been included in roundtables, it was not possible to compare permanency outcomes for youth who were and were not included in roundtables.

From the data provided by the state, separate data sets were created for children in care as of January 1 for each of the 3 years, and event history analyses were conducted to examine the differences, if any, in the time to permanency for all children in care at the beginning of the year for each of the 3 years. Analyses by age-group were run for the state as a whole and for Fulton and DeKalb counties separately.

Findings

The characteristics of the children in care statewide were nearly identical in 2008 and 2009. However, as seen in Table 6, the children in care in 2010 were significantly more likely to have a disability (specifically, emotional disturbance). They were also more likely to be in an institutional or hospital placement setting.

Table 6.
Child Characteristics by Year Statewide.

Child Characteristics Year

2008 2009 2010

Age 0–6 42.1% 42.6% 42.2%

7–12 24.7% 24.2% 23.9%

13–17 33.3% 33.2% 33.9%

d = 0.01(−0.02 to 0.02) d = −0.01(−0.04 to 0.02)

Gender Male 50.9% 51.4% 51.9%

Female 49.0% 48.6% 48.1%

d = 0.01(−0.02 to 0.04) d = 0.01(−0.02 to 0.04)

Race Black 48.7% 48.8% 47.6%^b

White 46.5% 45.6% 47.0%^b

Black−White mixed 3.9%^a 4.6% 4.7%

d = −0.01(−0.04 to 0.02) d = −0.02(−0.06 to 0.01)

Ethnicity Hispanic 6.3%^a 7.1% 7.0%

d = 0.03(0.00–0.06) d = 0.00(−0.03 to 0.03)

Disability Has a disability 50.4% 49.3% 58.6%^b

d = −0.02(−0.05 to 0.05) d = 0.19(0.16 to 0.21)

Emotionally disturbed Is emotionally disturbed 39.0% 38.5% 48.0%^b

d = −0.01(−0.04 to 0.02) d = 0.19(0.16 to 0.22)

Placement type Family home (foster/adoptive/relative) 79.9%^a 75.8% 78.4%^b

Institution/hospital 7.5%^a 11.6% 15.1%^b

Group home 6.6%^a 6.4% 4.0%

Detention 0.3%^a 0.7% 0.6%

Parent/trial home visit 1.7% 2.0% 0.0%^b

Other/missing/runaway 4.0% 3.6% 2.0%^b

d = −0.03(−0.06 to 0.01) d = −0.16(−0.20 to 0.13)

Length of stay Mean number of months in care, removal to January 1 (SD) 22.8 (26.3) 22.7 (26.0) 22.2 (25.9)

d = 0.00(−0.02 to 0.03) d = −0.02(−0.05 to 0.01)

Total 11,142 9,471 7,451

^aFor 2008 vs. 2009, p < .05. ^bFor 2009 vs. 2010, p < .05. Note. Georgia Division of Family and Children Services provided data on children in foster care as of January 1 each year (2008 and 2009 data were provided in August 2010; 2010 data were provided in June 2011). Effect sizes are calculated for 2008 vs. 2009 and for 2009 vs. 2010 at the category level (e.g., for placement type as a whole rather than for individual comparisons). Effect sizes (d) and associated 95% confidence intervals are reported for each comparison.

Statewide, time to permanency, and the percentage of youth achieving permanency were significantly different from year to year, but the findings were in different directions: while the 2008 to 2009 comparison showed reduced time to permanency and higher percentages of children achieving permanency, the 2009 to 2010 comparison showed increased time to permanency and/or lower percentages of children achieving permanency (see Figure 1 and Table 7).

Figure 1.
Percentage of children under 18 and in care as of January 1 achieving legal permanency statewide.

Table 7.
Statewide and County Comparisons of Average Days in Care and Percentage Achieving Legal Permanency by Age Group for Children Under Age 18 and in Care as of January 1, 2008, 2009, and 2010.

Statewide DeKalb County Fulton County

All ages* 0–6* 7–12* 13–17* All ages* 0–6* 7–12 13–17* All ages 0–6 7–12 13–17

2008 Average days in care during 2008 (SE) 275^b (1.14) d = 0.11 (0.09 to 0.14) 259^b (1.79) d = −0.13 (0.08 to 0.17) 270^b (2.29) d = 0.10 (0.04 to 0.15) 302^b(1.84) d = 0.10 (0.06 to 0.15) 283 (4.78) d = 0.10 (−0.01 to 0.21) 282 (7.16) d = 0.10 (−0.06 to 0.27) 264 (12.57) d = 0.11 (−0.14 to 0.36) 294 (7.34) d = 0.09 (−0.08 to 0.26) 287 (3.30) d = 0.02 (−0.06 to 0.10) 281 (5.24) d = 0.08 (−0.04 to 0.21) 289 (6.49) d = 0.09 (−0.08 to 0.25) 292^a (5.63) d = −0.16 (−0.30 to 0.02)

Percentage achieving legal permanency 45%^a d = −0.13 (−0.16 to −0.10) 56%^a d = −0.11 (−0.16 to −0.07) 49%^a d = −0.11 (−0.17 to −0.05) 29%^a d = −0.15 (−0.21 to −0.10) 39%^a d = −0.18 (−0.30 to −0.06) 47% d = −0.11 (−0.29 to 0.07) 47% d = −0.27 (−0.55 to 0.01) 29%^a d = −0.20 (−0.40 to −0.01) 39% d = 0 (−0.09 to 0.09) 43% d = −0.07 (−0.21 to 0.07) 43% d = 0 (−0.18 to 0.18) 32% d = 0.11 (−0.06 to 0.27)

Number of children 11,143 4,687 2,748 3,708 658 260 116 282 1,241 492 311 438

2009 Average days in care during 2009 (SE) 261 (1.28) 243 (1.99) 258 (2.59) 290 (2.10) 271 (4.57) 270 (6.83) 250 (11.11) 283 (7.33) 285 (3.74) 271 (5.89) 279 (7.36) 310 (5.59)

Percentage achieving legal permanency 51% 61% 54% 35% 47% 52% 59% 37% 39% 46% 43% 28%

Number of children 9,473 4,038 2,292 3,143 699 294 125 280 1,073 461 256 356

2010 Average days in care during 2010 (SE) 266 (1.45) d = 0.04 (0.01 to 0.07) 249^b (2.26) d = 0.05 (0.00 to 0.09) 261 (2.97) d = 0.02 (−0.04 to 0.09) 292 (2.33) d = 0.02 (−0.04 to 0.07) 256^a (5.62) d = −0.12 (−0.24 to −0.01) 251 (8.73) d = −0.16 (−0.34 to 0.02) 239 (13.99) d = −0.08 (−0.33 to 0.17) 269 (8.57) d = −0.11 (−0.29 to 0.07) 285 (3.90) d = 0 (−0.09 to 0.09) 280 (5.79) d = 0.07 (−0.06 to 0.21) 280 (8.52) d = 0.01 (−0.18 to 0.19) 295 (6.64) d = −0.13 (−0.29 to 0.02)

Percentage achieving legal permanency 47%^a d = −0.09 (−0.12 to −0.05) 57%^a d = −0.09 (−0.14 to −0.04) 51% d = −0.07 (−0.13 to 0.00) 32%^a d = −0.07 (−0.14 to −0.01) 52% d = 0.11 (−.02 to 0.24) 59% d = 0.16 (−0.04 to 0.36) 62% d = 0.07 (−0.24 to 0.38) 41% d = 0.09 (−0.11 to 0.29) 39% d = 0 (−0.10 to 0.10) 46% d = 0 (−0.15 to 0.15) 42% d = −0.02 (−0.23 to 0.19) 29% d = 0.03 (−0.16 to 0.21)

Number of children 7,451 3,146 1,778 2,527 500 203 87 210 906 393 196 317

Note. Georgia Division of Family and Children Services provided data on children in foster care as of January 1 each year (2008 and 2009 data were provided August 2010; 2010 data were provided June 2011). Effect sizes (d) and associated 95% confidence intervals are reported for each comparison. ^aNumber of days in care/percentage achieving legal permanency is significantly lower than in 2009. ^bNumber of days in care/percentage achieving legal permanency is significantly higher than in 2009.

p < .05 (log-rank test of equality of survival distributions for the different years).

For the year-to-year time to permanency comparisons in the two counties on which the roundtable intervention was initially focused, the comparisons between years for DeKalb County showed significant differences for all ages combined and for the 0–6 and 13–17 age groups. The DeKalb County year-to-year comparisons showed a reduced* time to permanency and a higher percentage of children achieving permanency for both the 2008–2009 and 2009–2010 comparisons. None of the comparisons between years for Fulton County showed significant differences (see Table 7).²

Statewide results showed that the rate of achieving legal permanency in 2009 was significantly higher than in 2008 for all ages; however, the 2009 rate was also significantly higher than the 2010 rate. (Differences were significant for the total under 18 and for each of the three age-groups: 0–6, 7–12, and 13–17.) The probability that a child in care on January 1 would remain in care at the end of the year was 52% in 2008, dropping to 46% in 2009 and rising to 50% in 2010.

Discussion and limitations

The initial roundtables focused primarily on children from DeKalb and Fulton counties in the metro Atlanta area, who had been in care for longer periods of time. Year-to-year comparisons for those two counties showed significant improvement from year to year across the 3-year period for DeKalb County and no significant improvement from year to year for Fulton County. As documented in the 12-month technical report (Rogg, Davis, & O’Brien, 2011), there were significant differences in the caseloads between these two counties.

Differences between 2008 and 2009 statewide and from 2008 to 2010 in DeKalb were likely due at least in part to the roundtable project (including orientation, training, consultation, and follow-up), the statewide implementation of roundtables, and the associated statewide emphasis on permanency values and practice. However, it is not possible to confirm that increases in the permanency rate were due to the roundtables, given that information as to which cases were included in the roundtables is not available.

Statewide results for 2010 (indicating a decline in the permanency rate compared to 2009) and in Fulton County suggest that it is more difficult to achieve permanency for the more difficult cases, such as the cases of children and youth who are emotionally disturbed. These cases require more intensive efforts beyond roundtables to address child needs and find appropriate permanency resources capable of handling those needs.

As noted above, since the state system did not track which youth had been included in roundtables, it was not possible to compare permanency outcomes for youth who were and were not included in roundtables. This is a major limitation of this comparison group.

Other Evaluations of PRTs

Description

A very broad comparison, suitable for mention but not rigorous enough to be utilized as a study-specific comparison group, comes from other evaluations of PRTs.

Expectations

The evaluation team did not have any expectations about this comparison data but sought the information for benchmarking.

Procedure

A small number of reports have been released, describing outcomes of PRTs in jurisdictions throughout the country. These were gathered and summarized.

Findings

For example, 27 (26%) of 104 youth whose cases were roundtabled in Kentucky in 2009 achieved permanency within 1 year (Huebner, James, Hagenbuch, & Cordell, 2010). However, these youth were on average younger than the youth in Georgia (mean age 10.7 years, in Kentucky) and had been in care for a shorter length of time.

A more recent study on outcomes for 726 youth in 11 jurisdictions in four states (Alabama, Colorado, Florida, and Ohio) found an overall permanency rate of 9% within 1 year, with rates ranging from 0% to 26% across jurisdictions (White, Corwin, Buher, & O’Brien, 2013). The most similar comparison data from the Georgia study is the 12-month rate of achievement of legal permanency (31%). This four-state evaluation lacks comparability to the Georgia study, however, in many ways; for example, criteria for inclusion in the roundtables and demographic characteristics of youth whose cases were included in roundtables dffered. The mean age of the youth in the four-state study was 16 years, which is significantly older than the youth in the Georgia study (13 years).

Discussion and limitations

While other evaluations of the same intervention can be helpful for benchmarking, in this case, there were no comparable studies (e.g., participating youth were of different age, and the follow-up periods in the comparison studies were shorter).

Discussion and Applications to Social Work

Of the 496 children and youth whose cases were included in PRTs at the beginning of 2009, half (50%) achieved legal permanency within 24 months. Given that many of these children and youth were considered to be “stuck” in foster care, the permanency achievement rate was notable. The evaluators wanted to determine (1) whether permanency rates for roundtabled youth were higher than those for youth who were not roundtabled, (2) whether statewide permanency rate increased appreciably after the statewide spread of the roundtable intervention, and (3) how permanency rates in the current study compared to other evaluations of the same intervention. To address these questions, five potential comparisons were considered.

The matched comparison group, intended to address whether permanency rates for roundtabled youth were higher than those for youth who were not roundtabled, did not result in a sufficient sample of matched cases, so these analyses were not conducted. The Kenny A. monitoring reports, social services quality reviews, and state and county data on the length of time in care and permanency rates provided some evidence that rates of achieving permanency increased statewide for youth following implementation of the PRTs, but the improved outcomes cannot be linked directly to the roundtable intervention. Lastly, other evaluations of PRTs lack comparability in terms of the selection criteria and participant demographics.

When comparing results from other jurisdictions, it can be difficult to measure the impact of a specific intervention—even when the study population, entry criteria, staffing, and caseloads are similar—because of the differences that may affect implementation of the intervention (and therefore its success). These include factors such as state laws and policies; agency structure and funding; agency leadership, focus, and communication; staff stability; staff training and support; and differences in the “regular” practice model. To the extent possible, staff and leadership surveys can be administered to measure and control for these differences.

Ideally, evaluators should plan for one or more comparison groups at the very beginning of the evaluation. Randomized control studies provide what is considered to be the most reliable evidence of causation and effectiveness. However, they are expensive, time consuming, and may have limited external validity.

Post hoc comparison group data should not be dismissed altogether. Because one comparison group is unlikely to be ideal, it may be helpful to include more than one comparison group and look for patterns across groups. For each comparison, it is important to be clear about the comparability, discussing each one’s relative strengths and weaknesses and the limits of any conclusions that can be drawn. One limitation of using post hoc comparison data, for example, is the inability to infer causality.

In summary, the roundtables helped many children achieve permanency and improved permanency practice statewide. The challenge for Georgia is to effectively address the needs and well-being of children coming into and remaining in care (particularly those with disabilities and emotional and behavioral disorders) and to sustain the improved practice and focus on permanency while simultaneously ensuring safety and well-being. The challenge for evaluators is to design and plan for valid comparison groups so they can determine whether an intervention has made a significant improvement, despite challenges such as limited funding, tight timelines, and often unmeasured (or difficult to measure) contextual differences between groups.

Outcome	Age Group at the Time of Roundtable	Total
Legal permanency achieved	Adoption	39.8%	30.6%	11.3%	21.8%
Guardianship	20.4%	19.4%	17.5%	18.5%
Reunification	16.1%	12.7%	5.6%	9.5%
Any legal permanency	76.3%	62.7%	34.2%	49.8%
Emancipation (aging out of foster care after age 18)	n/a	n/a	36.4%	19.8%
Custody termination (e.g., court relieved agency of custody due to child’s runaway status)	0%	0%	1.9%	1.0%
Remaining in care	23.7%	37.3%	27.5%	29.4%
Total number of children	93	134	269	496

Comparison Group	Purpose
1. Matched comparison data	Compare permanency rates for PRT youth with non-PRT youth
2. Statewide population of youth in foster care (Kenny A. monitoring reports)	Examine statewide permanency rates over several years to determine whether roundtables have impacted permanency rates statewide
3. Social services quality reviews (using Child and Family Services Review standards)
4. State and county data on length of time in care and permanency rates
5. Other evaluations of permanency roundtables	Compare permanency rates in the current study with other evaluations of the same intervention

Analysis of Children Included in Initial Roundtable Project	Number of Cases
Total number of cases considered for matching	9,484
Deleted due to missing values or possible roundtable involvement (missing data; also includes those roundtabled after June 30, 2009)	2,343
Used	7,141
Roundtabled	493^a
Not roundtabled	6,648

Kenny A. Monitoring Report Data	Target	2007	2008	2009	2010	2011
Entry on or after consent decree (October 27, 2005; Outcome 8 group)	n=	2,803	3,395	4,031	4,679	5,171	5,603	6,085	6,475	7,074	7,465
Positive permanency (reunification, permanent relative placement, adoption, guardianship, or permanent legal custody) within 12 months of entry for children who had been in care 12 months or less at the end of the prior reporting period (Outcome 8a)	40%	45%^a d = −0.11 (−0.16 to 0.06)	48% d = −0.04 (−0.09 to 0.00)	49% d = −0.02 (−0.07 to 0.02)	51% d = −0.04 (−0.09 to 0.00)	50%	53%^b d = 0.07 (0.02 to 0.11)	54%^b d = 0.09 (−0.05 to 0.13)	54%^b d = 0.09 (0.05 to 0.13)	54%^b d = 0.09 (0.05 to 0.13)	55%^b d = 0.11 (0.07 to 0.15)
Positive permanency (reunification or permanent relative placement within 12 months of entry; permanent legal custody, adoption or guardianship within 24 months of entry) (Outcome 8b)	74%	46%^a d = −0.22 (−0.27 to 0.17)	50%^a d = −0.13 (−0.18 to 0.09)	53%^a d = −0.07 (−0.11 to 0.02)	56% d = 0.00 (−0.04 to 0.04)	56%	59%^b d = 0.07 (0.03 to 0.11)	61%^b d = −0.11 (0.07 to 0.16)	62%^b d = 0.14 (0.10 to 0.18)	61%^b d = 0.11 (0.07 to 0.15)	62%^b d = −0.14 (−0.10 to 0.18)
Positive permanency (regardless of time frame)	n/a	47%^a d = −0.41 (−0.46 to 0.36)	54%^a d = −0.25 (−0.30 to 0.20)	57%^a d = −0.19 (−0.23 to 0.14)	62%^a d = −0.07 (−0.12 to 0.03)	65%	69%^b d = 0.10 (−0.06 to 0.14)	73%^b d = 0.21 (0.16 to 0.25)	75%^b d = 0.26 (0.22 to 0.31)	75%^b d = 0.26 (0.22 to 0.31)	76%^b d = 0.29 (0.25 to 0.34)
Other exits^c	n/a	6%^a	6%^a	7%^a	8%	9%	10%	10%	10%	10%	10%
In care up to 24 months at time of decree and still in care at beginning of time period (Outcome 9 group)	n=	625	470	334	247	183	140	106	74	49	32
Positive permanency	40%	20% d = 0.04 (−0.20 to 0.27)	20% d = 0.04 (−0.20 to 0.27)	19% d = 0 (−0.25 to 0.25)	21% d = 0.07 (−0.20 to 0.23)	19%	19% d = 0 (−0.31 to 0.31)	24% d = 0.16 (−0.16 to 0.48)	24% d = 0.16 (−0.19 to 0.52)	27% d = 0.25 (−.15 to 0.65)	9% d = 0.48 (−1.17 to 0.22)
Other exits^c	n/a	6% d = 0.11 (−0.30 to 0.52)	5% d = 0 (−0.43 to 0.43)	14%^a d = 0.62 (−0.22 to 1.03)	14%^a d = 0.52 (0.13 to 0.91)	5%	6% d = 0.11 (−0.42 to 0.64)	7% d = 0.20 (−0.35 to 0.75)	9% d = 0.35 (−0.22 to 0.92)	8% d = 0.28 (−0.40 to 0.95)	19%^a d = 0.82 (0.21 to 1.43)
In care more than 24 months at time of decree and still in care at beginning of time period (Outcome 10 group)	n=	414	349	281	219	161	129	95	66	43	35
Positive permanency	45%	9% d = −0.06 (−0.40 to 0.28)	9% d = −0.06 (−0.41 to 0.28)	14% d = 0.21 (−0.13 to 0.55)	15% d = 0.26 (−0.09 to 0.60)	10%	16% d = 0.30 (−0.09 to 0.68)	18% d = 0.38 (−0.03 to 0.78)	23%^b d = 0.55 (−0.12 to 0.97)	9% d = 0.06 (−0.56 to 0.69)	6% d = −0.31 (−1.13 to 0.51)
Other exits^c	n/a	7% d = −0.15 (−0.51 to 0.21)	8% d = −0.07 (−0.44 to 0.30)	24%^a d = 0.64 (0.31 to 0.97)	27%^a d = 0.73 (0.39 to 1.07)	9%	11% d = 0.12 (−0.30 to 0.55)	13% d = 0.23 (−0.22 to 0.67)	12% d = 0.18 (−0.33 to 0.68)	9% d = 0 (−0.65 to 0.65)	17% d = 0.40 (−0.17 to 0.97)

Georgia Social Services Quality Review Outcomes Tracking	Percentage of Cases Reviewed in Which Item is Rated as a “Strength'
Children have permanency and stability in their living situations (Permanency 1)	95%	44%	31%	29%	34%	41%
Permanency goal for the child (Item 7)	90%	58%	44%	49%	52%	64%
Reunification, guardianship, or permanent placement with a relative (Item 8)	90%	61%	48%	43%	48%	52%
Adoptions (Item 9)	90%	38%	29%	29%	38%	47%
Not having a permanency goal of other planned permanent living arrangement (Item 10)	90%	68%	74%	79%	82%	86%

Child Characteristics	Year
Age	0–6	42.1%	42.6%	42.2%
7–12	24.7%	24.2%	23.9%
13–17	33.3%	33.2%	33.9%
		d = 0.01(−0.02 to 0.02)		d = −0.01(−0.04 to 0.02)
Gender	Male	50.9%	51.4%	51.9%
Female	49.0%	48.6%	48.1%
		d = 0.01(−0.02 to 0.04)		d = 0.01(−0.02 to 0.04)
Race	Black	48.7%	48.8%	47.6%^b
White	46.5%	45.6%	47.0%^b
Black−White mixed	3.9%^a	4.6%	4.7%
		d = −0.01(−0.04 to 0.02)		d = −0.02(−0.06 to 0.01)
Ethnicity	Hispanic	6.3%^a	7.1%	7.0%
		d = 0.03(0.00–0.06)		d = 0.00(−0.03 to 0.03)
Disability	Has a disability	50.4%	49.3%	58.6%^b
		d = −0.02(−0.05 to 0.05)		d = 0.19(0.16 to 0.21)
Emotionally disturbed	Is emotionally disturbed	39.0%	38.5%	48.0%^b
		d = −0.01(−0.04 to 0.02)		d = 0.19(0.16 to 0.22)
Placement type	Family home (foster/adoptive/relative)	79.9%^a	75.8%	78.4%^b
Institution/hospital	7.5%^a	11.6%	15.1%^b
Group home	6.6%^a	6.4%	4.0%
Detention	0.3%^a	0.7%	0.6%
Parent/trial home visit	1.7%	2.0%	0.0%^b
Other/missing/runaway	4.0%	3.6%	2.0%^b
		d = −0.03(−0.06 to 0.01)		d = −0.16(−0.20 to 0.13)
Length of stay	Mean number of months in care, removal to January 1 (SD)	22.8 (26.3)	22.7 (26.0)	22.2 (25.9)
		d = 0.00(−0.02 to 0.03)		d = −0.02(−0.05 to 0.01)
Total	11,142	9,471	7,451

	Statewide	DeKalb County	Fulton County
2008	Average days in care during 2008 (SE)	275^b (1.14) d = 0.11 (0.09 to 0.14)	259^b (1.79) d = −0.13 (0.08 to 0.17)	270^b (2.29) d = 0.10 (0.04 to 0.15)	302^b(1.84) d = 0.10 (0.06 to 0.15)	283 (4.78) d = 0.10 (−0.01 to 0.21)	282 (7.16) d = 0.10 (−0.06 to 0.27)	264 (12.57) d = 0.11 (−0.14 to 0.36)	294 (7.34) d = 0.09 (−0.08 to 0.26)	287 (3.30) d = 0.02 (−0.06 to 0.10)	281 (5.24) d = 0.08 (−0.04 to 0.21)	289 (6.49) d = 0.09 (−0.08 to 0.25)	292^a (5.63) d = −0.16 (−0.30 to 0.02)
Percentage achieving legal permanency	45%^a d = −0.13 (−0.16 to −0.10)	56%^a d = −0.11 (−0.16 to −0.07)	49%^a d = −0.11 (−0.17 to −0.05)	29%^a d = −0.15 (−0.21 to −0.10)	39%^a d = −0.18 (−0.30 to −0.06)	47% d = −0.11 (−0.29 to 0.07)	47% d = −0.27 (−0.55 to 0.01)	29%^a d = −0.20 (−0.40 to −0.01)	39% d = 0 (−0.09 to 0.09)	43% d = −0.07 (−0.21 to 0.07)	43% d = 0 (−0.18 to 0.18)	32% d = 0.11 (−0.06 to 0.27)
Number of children	11,143	4,687	2,748	3,708	658	260	116	282	1,241	492	311	438
2009	Average days in care during 2009 (SE)	261 (1.28)	243 (1.99)	258 (2.59)	290 (2.10)	271 (4.57)	270 (6.83)	250 (11.11)	283 (7.33)	285 (3.74)	271 (5.89)	279 (7.36)	310 (5.59)
Percentage achieving legal permanency	51%	61%	54%	35%	47%	52%	59%	37%	39%	46%	43%	28%
Number of children	9,473	4,038	2,292	3,143	699	294	125	280	1,073	461	256	356
2010	Average days in care during 2010 (SE)	266 (1.45) d = 0.04 (0.01 to 0.07)	249^b (2.26) d = 0.05 (0.00 to 0.09)	261 (2.97) d = 0.02 (−0.04 to 0.09)	292 (2.33) d = 0.02 (−0.04 to 0.07)	256^a (5.62) d = −0.12 (−0.24 to −0.01)	251 (8.73) d = −0.16 (−0.34 to 0.02)	239 (13.99) d = −0.08 (−0.33 to 0.17)	269 (8.57) d = −0.11 (−0.29 to 0.07)	285 (3.90) d = 0 (−0.09 to 0.09)	280 (5.79) d = 0.07 (−0.06 to 0.21)	280 (8.52) d = 0.01 (−0.18 to 0.19)	295 (6.64) d = −0.13 (−0.29 to 0.02)
Percentage achieving legal permanency	47%^a d = −0.09 (−0.12 to −0.05)	57%^a d = −0.09 (−0.14 to −0.04)	51% d = −0.07 (−0.13 to 0.00)	32%^a d = −0.07 (−0.14 to −0.01)	52% d = 0.11 (−.02 to 0.24)	59% d = 0.16 (−0.04 to 0.36)	62% d = 0.07 (−0.24 to 0.38)	41% d = 0.09 (−0.11 to 0.29)	39% d = 0 (−0.10 to 0.10)	46% d = 0 (−0.15 to 0.15)	42% d = −0.02 (−0.23 to 0.19)	29% d = 0.03 (−0.16 to 0.21)
Number of children	7,451	3,146	1,778	2,527	500	203	87	210	906	393	196	317

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Caliendo

Kopeinig

(2008). Some practical guidance for the implementation of propensity score matching. Journal of Economic Surveys, 22, 31–72.

Davis

C. W.

O’Brien

Rogg

C. S.

Morgan

L. J.

White

C. R.

Houston

(2013). 24-month update on the impact of roundtables on permanency for youth in foster care. Children and Youth Services Review, 35, 2128–2134.

Freedman

D. A.

Berk

R. A.

(2008). Weighting regressions by propensity scores. Evaluation Review, 32, 392–409.

Gribbons

Herman

(1997). True and quasi-experimental designs. Practical Assessment, Research & Evaluation, 5. Retrieved from http://pareonline.net/getvn.asp?v=5&n=14

Huebner

R. A.

James

Hagenbuch

Cordell

(2010). Kentucky Permanency Roundtables: Phase one program evaluation one-year follow-up results. Frankfort: Kentucky Department of Community Based Services.

O’Brien

Davis

C. W.

Morgan

L. J.

Rogg

C. S.

Houston

(2012). The impact of roundtables on permanency for youth in foster care. Children and Youth Services Review, 34, 1915–1921.

Rogg

C. S.

Davis

C. W.

O’Brien

(2009). Permanency Roundtable Project: Process evaluation report. Seattle, WA: Casey Family Programs.

Rogg

C. S.

Davis

C. W.

O’Brien

(2011). Permanency Roundtable Project: 12-Month outcome evaluation report. Seattle, WA: Casey Family Programs.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2001). Experimental and quasi-experimental designs for generalized causal influence. Boston, MA: Houghton Mifflin.

10.

Sibald

Roland

(1998). Understanding controlled trials: Why are randomised controlled trials important? BMJ, 316, 201.

11.

Stürmer

Joshi

Glynn

R. J.

Avorn

Rothman

K. J.

Schneeweiss

(2006). A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. Journal of Clinical Epidemiology, 59, 437–447.

12.

White

C. R.

Corwin

Buher

O’Brien

(2013). The Multi-site Accelerated Permanency Project technical report: 12-month permanency outcomes. Seattle, WA: Casey Family Programs.

Kenny A. Monitoring Report Data	Target	2007		2008		2009		2010		2011
Kenny A. Monitoring Report Data	Target	Jan–Jun	Jul–Dec	Jan–Jun	Jul–Dec	Jan–Jun	Jul–Dec	Jan–Jun	Jul–Dec	Jan–Jun	Jul–Dec
Entry on or after consent decree (October 27, 2005; Outcome 8 group)	n=	2,803	3,395	4,031	4,679	5,171	5,603	6,085	6,475	7,074	7,465
Positive permanency (reunification, permanent relative placement, adoption, guardianship, or permanent legal custody) within 12 months of entry for children who had been in care 12 months or less at the end of the prior reporting period (Outcome 8a)	40%	45%^a d = −0.11 (−0.16 to 0.06)	48% d = −0.04 (−0.09 to 0.00)	49% d = −0.02 (−0.07 to 0.02)	51% d = −0.04 (−0.09 to 0.00)	50%	53%^b d = 0.07 (0.02 to 0.11)	54%^b d = 0.09 (−0.05 to 0.13)	54%^b d = 0.09 (0.05 to 0.13)	54%^b d = 0.09 (0.05 to 0.13)	55%^b d = 0.11 (0.07 to 0.15)
Positive permanency (reunification or permanent relative placement within 12 months of entry; permanent legal custody, adoption or guardianship within 24 months of entry) (Outcome 8b)	74%	46%^a d = −0.22 (−0.27 to 0.17)	50%^a d = −0.13 (−0.18 to 0.09)	53%^a d = −0.07 (−0.11 to 0.02)	56% d = 0.00 (−0.04 to 0.04)	56%	59%^b d = 0.07 (0.03 to 0.11)	61%^b d = −0.11 (0.07 to 0.16)	62%^b d = 0.14 (0.10 to 0.18)	61%^b d = 0.11 (0.07 to 0.15)	62%^b d = −0.14 (−0.10 to 0.18)
Positive permanency (regardless of time frame)	n/a	47%^a d = −0.41 (−0.46 to 0.36)	54%^a d = −0.25 (−0.30 to 0.20)	57%^a d = −0.19 (−0.23 to 0.14)	62%^a d = −0.07 (−0.12 to 0.03)	65%	69%^b d = 0.10 (−0.06 to 0.14)	73%^b d = 0.21 (0.16 to 0.25)	75%^b d = 0.26 (0.22 to 0.31)	75%^b d = 0.26 (0.22 to 0.31)	76%^b d = 0.29 (0.25 to 0.34)
Other exits^c	n/a	6%^a	6%^a	7%^a	8%	9%	10%	10%	10%	10%	10%
In care up to 24 months at time of decree and still in care at beginning of time period (Outcome 9 group)	n=	625	470	334	247	183	140	106	74	49	32
Positive permanency	40%	20% d = 0.04 (−0.20 to 0.27)	20% d = 0.04 (−0.20 to 0.27)	19% d = 0 (−0.25 to 0.25)	21% d = 0.07 (−0.20 to 0.23)	19%	19% d = 0 (−0.31 to 0.31)	24% d = 0.16 (−0.16 to 0.48)	24% d = 0.16 (−0.19 to 0.52)	27% d = 0.25 (−.15 to 0.65)	9% d = 0.48 (−1.17 to 0.22)
Other exits^c	n/a	6% d = 0.11 (−0.30 to 0.52)	5% d = 0 (−0.43 to 0.43)	14%^a d = 0.62 (−0.22 to 1.03)	14%^a d = 0.52 (0.13 to 0.91)	5%	6% d = 0.11 (−0.42 to 0.64)	7% d = 0.20 (−0.35 to 0.75)	9% d = 0.35 (−0.22 to 0.92)	8% d = 0.28 (−0.40 to 0.95)	19%^a d = 0.82 (0.21 to 1.43)
In care more than 24 months at time of decree and still in care at beginning of time period (Outcome 10 group)	n=	414	349	281	219	161	129	95	66	43	35
Positive permanency	45%	9% d = −0.06 (−0.40 to 0.28)	9% d = −0.06 (−0.41 to 0.28)	14% d = 0.21 (−0.13 to 0.55)	15% d = 0.26 (−0.09 to 0.60)	10%	16% d = 0.30 (−0.09 to 0.68)	18% d = 0.38 (−0.03 to 0.78)	23%^b d = 0.55 (−0.12 to 0.97)	9% d = 0.06 (−0.56 to 0.69)	6% d = −0.31 (−1.13 to 0.51)
Other exits^c	n/a	7% d = −0.15 (−0.51 to 0.21)	8% d = −0.07 (−0.44 to 0.30)	24%^a d = 0.64 (0.31 to 0.97)	27%^a d = 0.73 (0.39 to 1.07)	9%	11% d = 0.12 (−0.30 to 0.55)	13% d = 0.23 (−0.22 to 0.67)	12% d = 0.18 (−0.33 to 0.68)	9% d = 0 (−0.65 to 0.65)	17% d = 0.40 (−0.17 to 0.97)

Georgia Social Services Quality Review Outcomes Tracking	Percentage of Cases Reviewed in Which Item is Rated as a “Strength'
Georgia Social Services Quality Review Outcomes Tracking	Federal Standard	FFY 2007	FFY 2008	FFY 2009	FFY 2010	FFY 2011
Children have permanency and stability in their living situations (Permanency 1)	95%	44%	31%	29%	34%	41%
Permanency goal for the child (Item 7)	90%	58%	44%	49%	52%	64%
Reunification, guardianship, or permanent placement with a relative (Item 8)	90%	61%	48%	43%	48%	52%
Adoptions (Item 9)	90%	38%	29%	29%	38%	47%
Not having a permanency goal of other planned permanent living arrangement (Item 10)	90%	68%	74%	79%	82%	86%

		Statewide				DeKalb County				Fulton County
		All ages*	0–6*	7–12*	13–17*	All ages*	0–6*	7–12	13–17*	All ages	0–6	7–12	13–17
2008	Average days in care during 2008 (SE)	275^b (1.14) d = 0.11 (0.09 to 0.14)	259^b (1.79) d = −0.13 (0.08 to 0.17)	270^b (2.29) d = 0.10 (0.04 to 0.15)	302^b(1.84) d = 0.10 (0.06 to 0.15)	283 (4.78) d = 0.10 (−0.01 to 0.21)	282 (7.16) d = 0.10 (−0.06 to 0.27)	264 (12.57) d = 0.11 (−0.14 to 0.36)	294 (7.34) d = 0.09 (−0.08 to 0.26)	287 (3.30) d = 0.02 (−0.06 to 0.10)	281 (5.24) d = 0.08 (−0.04 to 0.21)	289 (6.49) d = 0.09 (−0.08 to 0.25)	292^a (5.63) d = −0.16 (−0.30 to 0.02)
	Percentage achieving legal permanency	45%^a d = −0.13 (−0.16 to −0.10)	56%^a d = −0.11 (−0.16 to −0.07)	49%^a d = −0.11 (−0.17 to −0.05)	29%^a d = −0.15 (−0.21 to −0.10)	39%^a d = −0.18 (−0.30 to −0.06)	47% d = −0.11 (−0.29 to 0.07)	47% d = −0.27 (−0.55 to 0.01)	29%^a d = −0.20 (−0.40 to −0.01)	39% d = 0 (−0.09 to 0.09)	43% d = −0.07 (−0.21 to 0.07)	43% d = 0 (−0.18 to 0.18)	32% d = 0.11 (−0.06 to 0.27)
	Number of children	11,143	4,687	2,748	3,708	658	260	116	282	1,241	492	311	438
2009	Average days in care during 2009 (SE)	261 (1.28)	243 (1.99)	258 (2.59)	290 (2.10)	271 (4.57)	270 (6.83)	250 (11.11)	283 (7.33)	285 (3.74)	271 (5.89)	279 (7.36)	310 (5.59)
	Percentage achieving legal permanency	51%	61%	54%	35%	47%	52%	59%	37%	39%	46%	43%	28%
	Number of children	9,473	4,038	2,292	3,143	699	294	125	280	1,073	461	256	356
2010	Average days in care during 2010 (SE)	266 (1.45) d = 0.04 (0.01 to 0.07)	249^b (2.26) d = 0.05 (0.00 to 0.09)	261 (2.97) d = 0.02 (−0.04 to 0.09)	292 (2.33) d = 0.02 (−0.04 to 0.07)	256^a (5.62) d = −0.12 (−0.24 to −0.01)	251 (8.73) d = −0.16 (−0.34 to 0.02)	239 (13.99) d = −0.08 (−0.33 to 0.17)	269 (8.57) d = −0.11 (−0.29 to 0.07)	285 (3.90) d = 0 (−0.09 to 0.09)	280 (5.79) d = 0.07 (−0.06 to 0.21)	280 (8.52) d = 0.01 (−0.18 to 0.19)	295 (6.64) d = −0.13 (−0.29 to 0.02)
	Percentage achieving legal permanency	47%^a d = −0.09 (−0.12 to −0.05)	57%^a d = −0.09 (−0.14 to −0.04)	51% d = −0.07 (−0.13 to 0.00)	32%^a d = −0.07 (−0.14 to −0.01)	52% d = 0.11 (−.02 to 0.24)	59% d = 0.16 (−0.04 to 0.36)	62% d = 0.07 (−0.24 to 0.38)	41% d = 0.09 (−0.11 to 0.29)	39% d = 0 (−0.10 to 0.10)	46% d = 0 (−0.15 to 0.15)	42% d = −0.02 (−0.23 to 0.19)	29% d = 0.03 (−0.16 to 0.21)
	Number of children	7,451	3,146	1,778	2,527	500	203	87	210	906	393	196	317