Abstract
Objective:
The purpose of this study is to examine the correlates of child behavior change over time in a replication of the KEEP intervention.
Method:
The study sample was drawn from the treatment group of the Maryland replication of KEEP (n=65). Change over time was analyzed using multilevel linear mixed modeling.
Results:
Parents’ use of positive reinforcement relative to discipline was associated with the rate of child behavior change among program participants; parents with the lowest initial levels of reinforcement reported the greatest decrease in child problem behaviors. Other participant characteristics were not associated with child behavior change during the study period.
Conclusions:
The results indicate the efficacy of an evidence-based foster parent training program for reducing child problem behaviors and underscore the utility of teaching parents to use more positive responses relative to discipline as a robust path to improved child outcomes.
According to the most recent wave of data available from the National Survey of Child and Adolescent Well-Being, more than 40% of children in foster care display clinical levels of externalizing behavior problems (Casanueva, Tueller, Smith, Dolan, & Ringeisen, 2014). High frequencies of externalizing behavior problems in children have been linked to poor mental health, academic and employment outcomes for the child, and depression and anxiety among the adults who care for them (Barlow, Coren, & Stewart-Brown, 2002; Burke, Loeber, & Birmaher, 2002; Dretzke et al., 2009; Gross, Shaw, Burwell, & Nagin, 2009; Harden & Whittaker, 2011). Externalizing behaviors in children are also likely to influence the nature of interactions between a caregiver and their child. Increased attention provided by the caregivers may unwittingly reinforce negative child behaviors which, in turn, may escalate harsh parenting (Chamberlain et al., 2008; Patterson, DeBaryshe, & Ramsey, 1990). For children in foster care, worsening behavior can lead to unstable and restrictive placements (Fisher, Stoolmiller, Mannering, Takahashi, & Chamberlain, 2011; James, Landsverk, & Slymen, 2004). To disrupt this cycle, researchers have suggested implementing parent training programs that help caregivers reduce the attention paid to negative behaviors and increase rewards for positive behaviors (Barth et al., 2005).
Background
A sizable field of empirical literature has explored the benefits of parenting interventions. A Cochrane review conducted by Furlong et al. (2013) assessed the evidence for cognitive and cognitive–behavioral-based parenting programs delivered in a group format. The authors identified 13 randomized and quasi-experimental trials of cognitive and cognitive–behavioral-based group parenting programs for primary caregivers of children aged 3 to 12 years with clinical levels of conduct problems (n = 1,078). Furlong et al. (2013) found significant reductions in child conduct problems for treatment group participants. Positive results were consistent whether child behaviors were measured by parents report or independently assessed. The authors concluded that cognitively oriented group-based parenting programs were effective, for parents of youth with significant behavior concerns. This coheres with the findings reported in other meta-analyses of parent management programs (Dretzke et al., 2009; Michelson, Davenport, Dretzke, Barlow, & Day, 2013).
A smaller but growing body of literature has evaluated the effectiveness of parenting interventions delivered to foster and kinship caregivers. A Cochrane review (Turner, MacDonald, & Dennis, 2007) identified just six randomized or quasi-experimental studies of cognitive–behavioral-based parent training programs delivered to foster parents. For foster parents (n = 463) in these studies, the authors identified no overall evidence for the effectiveness of the cognitive–behavioral training programs on child behavior problems, psychological, or interpersonal functioning. Turner, Macdonald, and Dennis (2007) concluded that there was insufficient evidence in this area to guide best practices, highlighting the need for additional studies with larger samples, more precise measures, diverse sampling strategies, and varied program designs.
Subsequent to the Turner et al. (2007) review, increasingly robust evaluations have provided support for the efficacy of group-based foster parent training programs in reducing problem behaviors among children in foster care (Dorsey et al., 2008; Festinger & Baker, 2013). Although the current evidence base for foster parent training programs is uneven, recent studies of group-based foster parent trainings using standardized curricula and robust research designs that measure parent and/or child behavior change have emerged with promising results (Dorsey et al., 2008; Festinger & Baker, 2013). One such program is Keeping Foster and Kinship Parents Supported (KEEP), a 16-week training and support group for foster and kinship caregivers of children between the ages of 4 and 12. In KEEP, caregivers learn to implement research-supported behavior management strategies aimed at reducing the frequency of externalizing problem behaviors exhibited by the children in their care (Chamberlain et al., 2008).
A pilot study of the KEEP model found a significant reduction in child behavior problems among the parents who received the intervention compared to parents who received treatment as usual or an additional stipend with no training (Chamberlain, Moreland, & Reid, 1992). These results were later replicated in a randomized control trial (n = 700) that found significant reductions in the number of externalizing child behavior problems and increased placement stability for the KEEP participants compared with the control group (Chamberlain et al., 2008; Price et al., 2008). In addition, Chamberlain et al. (2008) found that the intervention’s effect on child problem behaviors was partially mediated by parents’ increased use of positive reinforcement relative to discipline. This effect was present among children whose caregivers reported greater than six child problem behaviors in a 24-hr period before starting the intervention (high risk) but not for children below that threshold (low risk). In addition, the children in the high-risk group experienced a greater decrease in problem behaviors compared to children in the low-risk group during the study period. KEEP has since been replicated in multiple geographically diverse settings including Maryland, New York City, the United Kingdom, and Sweden.
In KEEP, parents are taught to reduce the attention paid to negative behaviors and reward positive behaviors in order to reduce the frequency of problem behaviors among the children in their care. On average, caregivers who participate in KEEP report a decline in child problem behaviors, but there are participants for whom this is not true. Hurlburt, Chamberlain, DeGarmo, Zhang, and Price (2010) used growth mixture models to identify four trajectory classes that described the child behavior patterns of KEEP participants over the 16 weeks of treatment. Most children in the sample had a low rate of problem behaviors that remained low throughout the study (69%), about 15% started high and decreased, and 12% started high and stayed high. The remaining 5% started low but saw behaviors increase during the study period. Overall, studies of KEEP have consistently demonstrated a favorable intervention effect on externalizing problem behaviors, placement stability, and permanency outcomes (Buchanan, Chamberlain, Price, & Sprengelmeyer, 2013; Greeno et al., Under Review; Price, Roesch, & Walsh, 2012). Still the findings reported by Hurlburt et al. (2010) highlight the need to better understand the process and correlates of change in parenting interventions, so that caregivers in need of additional supports or alternate programming could be identified and served.
The Present Study
The present study was a replication of the KEEP program delivered in accordance with the original KEEP model with a minor revision, made at the suggestion of the model developers, that the minimum age for inclusion in the study was age 4 instead of age 5 (see Price, Chamberlain, Landsverk, & Reid, 2009 for model description). Previous evaluations of both the KEEP model and foster parent training programs in general have used pre–post data to understand the impact of group participation on child outcomes. The present study provides a unique contribution to the literature by examining the course and correlates of child behavior change among participants in a parenting intervention. The following two research questions were addressed: (1) How do reports of child behavior problems change between pretreatment and 2 months post-intervention for KEEP participants? and (2) Are participant characteristics associated with individual differences in the patterns of child behavior change reported by KEEP participants?
Method
Participants
The study sample was drawn from the Maryland replication of KEEP (Greeno et al., Under Review). The sample consisted of foster and kinship parents providing care for children between the ages of 4 and 12. Children were required to have been in the current placement for a minimum of 1 month and not be considered medically fragile. Seventy-five foster and kinship caregivers, representing 65 households, participated in 9 cohorts of KEEP between 2010 and 2012. Only one caregiver per home was included in the evaluation. All foster and kinship parents who consented to participate in the KEEP replication were included in the analysis (N = 65). Selected demographic characteristics of participants and the target children are presented in Table 1.
Sample Demographics for the Maryland KEEP Evaluation Participants.
Note. N = 65. KEEP = Keeping Foster and Kinship Parents Supported; CBCL = Child Behavior Checklist; PDR = Parent Daily Report.
aCalculated according to the Chamberlain et al. (2006) standard: low risk = PDR ≤ 6; high risk = PDR > 6.
Measures
Child behaviors were measured using the Parent Daily Report (PDR; Chamberlain & Reid, 1987), a 31-item measure of parent-reported behavior problems (e.g., arguing, complaining, and back talk). Caregivers were asked to recall the past 24 hr and respond yes or no to whether the identified foster or kinship child displayed the behavior. At pretreatment and the 2-month follow-up time points, the PDR score was informed by two administrations within close succession (usually 1 week) and averaged to obtain a stable measure of functioning. The internal consistency of the PDR in this full sample was sufficient (Cronbach’s α = .82 pretreatment and .85 posttest).
Initial risk status was assessed based on prior research by Chamberlain et al. (2006) that identified an increased risk of placement disruptions for children with greater than six behaviors on the PDR. This standard has been used in multiple studies as a moderator of KEEP treatment effect (Chamberlain et al., 2008; Price et al., 2012). A dichotomous variable was created using the pretreatment value of the PDR, so that a child with a pretreatment PDR score less than or equal to 6 was considered low risk (0) and a child with a pretreatment PDR score greater than 6 was considered high risk (1).
The proportion of positive reinforcement is a composite scale developed by the Oregon Social Learning Center (OSLC) to measure parent’s use of behavior management techniques (Chamberlain et al., 2008; Price et al., 2012). Caregivers are asked about the use of specific discipline and positive reinforcement techniques in the previous 24 hours. In addition, caregivers are asked to report their overall use of parent management techniques (positive reinforcement and discipline) on a 7-point Likert-type scale ranging from “3+ times a day” to “less than once a month” to “never.” The item scores are combined to create a scale score that describes the use of positive reinforcement techniques in relation to the total use of parent management techniques. A higher score means more positive reinforcement and less discipline.
Children’s externalizing problems were measured using the Externalizing Scale of the Child Behavior Checklist (CBCL; Achenbach & Rescorla, 2001). The CBCL was completed on the telephone by caregivers once at pretreatment. For the analysis, scores were dichotomized as being in the clinical range compared to the subclinical/normal range. The internal consistency of the CBCL in this sample was sufficient (Cronbach’s α = .91 at pretreatment and .93 at posttest).
Child age in years was measured at the time the parent consented to participate. Attendance rates were calculated as the proportion of group sessions attended in person. Child gender (male or female) and placement type (kinship or nonrelative foster care) were also measured.
Procedures
The present study was approved by the University of Maryland Institutional Review Board. A single-group longitudinal cohort design was used. State child welfare workers contacted foster and kinship families on their caseloads and encouraged them to participate in the program. Facilitators then visited the families in their homes prior to the trainings to answer questions about the program and obtain written consent for participation in the evaluation.
Pretest data on parenting practices and child behaviors were collected approximately 1 week prior to the start of the KEEP training. Additional data on child behaviors were collected weekly during the 16-week intervention and again approximately 2 months after the completion of the KEEP training. Only one caregiver and one child per home were eligible for the study. For consistency, the same foster or kinship parent was asked to answer all questionnaires. Time was measured in weeks (max = 25), with the start date set as 0; pretreatment measures were negative numbers and the weeks during and after treatment commenced were positive numbers. A maximum of 18 time points were analyzed for each youth. Weekly behavior scores were nested within each participant. Out-of-home placement type was defined using administrative data provided by the state child welfare services agency.
Facilitators were trained by consultants from the OSLC over a period of 5 days. Trainees were coached on the KEEP curriculum, group logistics, recruitment, and facilitation. After the initial training period, facilitators received weekly supervision from an OSLC consultant. To ensure fidelity to the model, all of the group sessions were recorded for review by the OSLC consultant. Feedback from the video review was provided during the weekly supervision sessions. Facilitators could be certified to lead groups without supervision after having facilitated three full cycles of the 16-week groups. To ensure fidelity, facilitators continued to submit video recordings of each sessions for intermittent review.
Analysis
Exploratory analysis found sufficient within- and between-subjects variation in PDR scores across time points to justify the use of multilevel statistical methods that can account for the nesting of the dependent variable (PDR scores) within subjects (Singer & Willett, 2003). Therefore, a series of multilevel linear mixed models (growth models) were used to assess the trajectory of externalizing child behaviors (PDR) from pretreatment to follow up and model whether participant characteristics affect the trajectory of child behavior change. All analyses were conducted using SPSS v 21 (IBM, 2012).
All assumptions of multilevel growth models were met. Maas and Hox (2005) suggest using a sample size greater than 50 at the second level of nesting in multilevel models to ensure unbiased estimates of standard errors. The present study exceeded this criteria. Polynomial and quadratic transformations of the weekly PDR scores were tested. The linear model showed superior fit and was used in the analyses. A model-building approach was used as suggested by Singer and Willett (2003). Model fit was assessed by comparing −2 log likelihood (deviance), Akaike’s Information Criterion (AIC), and Schwarz’s Bayesian Information Criterion (BIC) between each successive model.
To gain a better understanding of the differential impact of the intervention, individual trajectories of child behavior were identified (Donaldson, 2008). The data file was split by case ID and a regression was run for each participant using a 50% confidence interval. If the confidence interval contained zero, the slope was classified as flat. If the confidence interval did not contain zero, the regression coefficient was used to determine the direction of the slope. A participant with a positive slope was classified as having increasing frequency of problem behaviors over time. Participants with a negative slope were classified as having a decreasing frequency of problem behaviors over time. Frequencies were calculated to assess the number of participants who had increasing, decreasing, or flat slopes, and bivariate analyses were run to identify correlates of exhibiting a particular trajectory. Finally, a binomial logistic regression assessed whether any participant characteristics were associated with exhibiting decreasing behaviors versus no change or increasing behaviors.
Missing Data
Of the 65 parents who agreed to participate, 56 (86%) completed the intervention. Four parents consented to the study and completed the pretest but never attended the group. Three parents dropped out after attending fewer than three group sessions and two parents dropped out of the study after attending half or more sessions due to issues unrelated to the intervention. Among parents who completed the group, parents attended 78% (12.5 of 16) of the sessions and completed 74% of PDR interviews (a mean of 13.4 of a possible 18 measurement points; SD = 2.94). All participants completed the pretreatment PDR interviews (n = 65) and 43 caregivers completed the follow-up interviews. The proportion of positive reinforcement was missing for only one case in the entire sample.
All participants who consented to take part in the evaluation were included in the analysis. Longitudinal multilevel models are able to handle data in which participants have missing or unevenly spaced observations (Shek & Ma, 2011; Singer & Willett, 2003). Among the nine participants who did not complete the study, five had pretreatment data only. For the four participants with additional data points, there were no clear observable patterns or similarities in the PDR profiles or demographic characteristics that differentiated them from those who completed the group. Sensitivity to missing data was assessed by running analyses that included and excluded dropouts. Model estimates were not affected by the inclusion of subjects who dropped out of the study.
Results
A series of multilevel models were conducted to measure child behavior change (using PDR score) across the 16 weeks of the intervention and a 2-month follow-up. First, the unconditional means model was estimated, which identifies the amount of variation in PDR scores due to within and between participants differences (Table 2). The intra-class correlation of the model was .69, indicating that 69% of the variance in PDR scores was accounted for by differences between participants, justifying the use of a multilevel model (Singer & Willett, 2003).
Summary of Results for the Multilevel Growth Model Fitted to Evaluate the Contributions of Participant Characteristics on the Trajectory of Child Behavior Changea for KEEP Group Participants.
Note. N = 65. KEEP = Keeping Foster and Kinship Parents Supported; AIC = Akaike Information Criterion; BIC = Bayesian Information Criterion; PDR = Parent Daily Report.
aParent Daily Report.
bHigh initial risk compared to low initial risk group calculated according to the Chamberlain et al. (2006) standard: low risk = PDR ≤ 6; high risk = PDR > 6.
*p < .05; **p < .01; *** p < .001.
Time was included as an independent variable to estimate the amount of change in PDR scores that is due to the passage of time. The mean pretreatment PDR for the sample was 6.39 (SE = .049), which would be considered high risk for placement disruption (Chamberlain et al., 2006). The mean slope for the model with time as the only predictor was −0.13 (SE = .03). At follow-up (i.e., 8 weeks posttreatment), the average KEEP participant would report a decrease of more than three child problem behaviors on the PDR.
Next, we tested the effect of participant characteristics on both pretreatment mean PDR scores and the rate of change. Participant characteristics were tested individually to assess their impact on the pretreatment mean and then with an interaction with time to assess their impact on the mean rate of behavior change (slope). When model fit was improved by inclusion of the item or the associated interaction term, the item was retained in subsequent models. When an item or interaction term was no longer significant after the inclusion of additional items, the nonsignificant item would be removed and a decision to retain the item would be made according to the impact on model fit. Kinship status, child age, and child gender did not improve model fit and were therefore removed from subsequent models. Attendance and CBCL Externalizing Scale scores were significant predictors of pretreatment mean PDR scores and the rate of change in intermediate models, but detracted from model fit when initial risk status was included in the model; attendance and CBCL were, hence, removed from subsequent models.
The final multilevel model for measuring the trajectory of child behavior change included time, initial risk status (high vs. low), proportion of positive reinforcement, and an interaction term for the impact of proportion of positive reinforcement and time as independent variables. The intercept for initial risk status was significant, indicating the mean pretreatment difference between high-risk (pretreatment PDR > 6) versus low-risk (pretreatment PDR ≤ 6) groups was not zero, which was expected. The slope for initial risk was nonsignificant, indicating that the rate of behavior change for participants who started out in the high-risk group was not different from participants who started in the low-risk group. Parents who reported using less discipline and more positive reinforcement at pretreatment reported a significantly lower initial PDR scores (b = −16.78; p < .001). Finally, the interaction between proportion of positive reinforcement and time was significant, indicating that the pretreatment proportion of positive reinforcement affected the rate of change in PDR scores over time (b = .94; p < .05).
An illustration of the impact of the proportion of positive reinforcement on the trajectory of child problem behaviors for children in the high-risk category is presented in Figure 1. Regression lines were calculated and plotted using the value for proportion of positive reinforcement at the mean, 1 SD above the mean, and 1 SD below the mean (Cohen, Cohen, West, & Aiken, 2002). Parents with a lower proportion of positive reinforcement experienced the steepest decrease in the frequency of behavior problems on the PDR over time (b = .065; p < .01). As the 16-week intervention neared its end, the regression lines changed rank order, so that by the 2-month follow-up, parents with the lowest pretreatment proportion of positive reinforcement relative to discipline reported the lowest PDR behavior counts. Approximately 54% of the variance in the mean pretreatment PDR value and 18% of the variance in the PDR rate of change was explained by the inclusion of proportion of positive reinforcement and initial risk status.
Impact of caregivers use of positive reinforcement relative to discipline at pretreatment on the trajectory of Parent Daily Report (PDR) behavior counts for children with a high initial risk of disruption (PDR > 6). Note. Within 1 SD of mean proportion of positive reinforcement (n = 44); ≥ 1 SD above mean proportion of positive reinforcement (n = 9); ≤ 1 SD below mean proportion of positive reinforcement (n = 11).
To better understand the differential experiences of KEEP group participants, individual trajectories of child behavior change were identified for 60 of the participants (see Table 3). The slope could not be determined for five participants who had less than two recorded PDR scores. More than half of the participants (53%; n = 32) had a negative slope, indicating that the number of behavior problems decreased during the course of the study. About 17% (n = 10) of the participants had positive slopes, indicating an increase in PDR counts over time, and about 30% (n = 18) of the participants experienced no change in their PDR scores during the study period. Bivariate analyses, as well as a binomial logistic regression, were conducted in an attempt to identify factors that could differentiate the participants that improved during the intervention period from those who plateaued or got worse. No participant characteristics or behaviors were significantly associated with belonging to the group with improving behaviors.
Child Behavior Changea by Initial Risk of Disruption.b
Note. PDR = Parent Daily Report.
aParent Daily Report.
bCalculated according to the Chamberlain et al. (2006) standard: low risk = PDR ≤ 6; high risk = PDR > 6.
Discussion and Applications to Social Work
The study aimed to identify the patterns and correlates of behavior change for participants in a research-supported foster parent training program. Participants in the KEEP group reported a significant reduction in child problem behaviors over time; however, some participants reported no change and a small minority reported a worsening of child behaviors during the study period. In the final model placement type, child age, child gender, group attendance, and initial CBCL externalizing score were not significantly associated with the initial PDR score or the change in child problem behaviors reported by participants during the study period. Only the interaction between the proportion of positive parenting and time affected the trajectory of behavior change in this study sample.
Parents who reported using less positive reinforcement and more discipline before attending the KEEP group reported the greatest decreases in child behavior change over the course of the study. This coheres with earlier research that found the proportion of positive reinforcement employed by caregivers to partially mediate the KEEP intervention effect (Chamberlain et al., 2008). Together these findings suggest that KEEP’s greatest strength may be in changing how parents reinforce youth and implement discipline, especially among parents who start with a low proportion of positive reinforcement relative to discipline. The primary drivers of child behavior change in this study are parenting style and time, providing additional support for the theory of change prominent in parenting interventions, and that parents are the key lever for change in child behavior.
Previous studies have found a greater decrease in problem behaviors for children in the high initial risk group (PDR > 6 at pretreatment) compared to children in the low-risk group (Chamberlain et al., 2008; Price et al., 2012). In the current study, behavior change over time was not differentiated by initial risk after controlling for the proportion of positive reinforcement. These findings suggest parent training programs like KEEP may be beneficial even for parents with children who are at lower risk of placement disruption.
Despite findings from an older meta-analysis, Turner et al. (2007) who suggested no evidence for the effectiveness of cognitive–behavioral foster parent training programs, the majority of participants in the current study reported improved child behaviors over time. Almost one third of participants reported no change in child behavior problems during the study period and 17% actually got worse during the course of the intervention. Although the goal of the intervention was to reduce the incidence of child behavior problems, children in foster care tend to exhibit increased problem behaviors over time (Aarons et al., 2010). Thus the maintenance of pretreatment levels of problem behaviors among a large portion of the group participants may be interpreted as programmatic success.
The patterns of child behavior change identified in this sample are similar to those found by Hurlburt et al. (2010); however, the proportion of the sample in each group is quite different. In this study, 63% of the sample began with high PDR scores, compared to about 26% in Hurlburt et al. (2010). Additionally, 36% of the current sample started with low PDR scores that remained low, compared to almost 70% in Hurlburt et al. (2010). The inclusion of more challenging youth in the current study is likely a reflection of the recruitment strategy that focused on the behavior change potential of KEEP for families with very difficult children.
Additional analyses identified no significant predictors that could differentiate the participants who improved during the intervention period from those who plateaued or got worse. More study is needed to better understand whether there are subgroups of participants in research-supported foster parent training programs who need alternate or supplemental training. The results from this study confirm previous findings that participation in group-based foster parent training programs is associated with improved child behaviors and that the proportion of positive reinforcement impacts the rate of change (Chamberlain et al., 2008). Additional research that consider the correlates of change in parenting interventions are needed to understand the differential effectiveness of program participation and improve services for those who are not experiencing the expected decrease in child problem behaviors.
The current analysis is limited by the lack of a control group to provide a counterfactual for participation in the KEEP group. In addition, a larger sample size would decrease the possibility of Type II errors and expand the types of analyses that are possible. Any nonsignificant findings should be confirmed with a larger sample size. Due to sample size at the cohort level, we could not include facilitator or cohort characteristics in the multilevel model, which may have provided important insight into between-group differences. Despite these limitations, the present study provided important insight into the process of change for participants in a research-supported foster parent training program.
In summary, the present study found that on average, participation in the KEEP intervention resulted in decreased frequency of child behaviors problems. In addition, the rate of change was dependent on the parent’s use of positive reinforcement, supporting the findings from the original KEEP trial (Chamberlain et al., 2008). Parents whose proportion of positive reinforcement relative to discipline was low at pretreatment reported the greatest decrease in child behavior problems during the study period. This relationship held even for parents of children in the high initial risk group, strengthening the argument for the effectiveness of research-supported parenting programs in reducing child problem behaviors among diverse samples (Huey & Polo, 2008), in this case, of foster children. Teaching parents to use more positive responses relative to discipline with their children appears to be an important and robust path to improve child outcomes, regardless of pretreatment levels of problem behaviors. Importantly, children of the foster parents who had the least favorable amalgam of parenting strategies before the intervention benefited the most during and after participating in the group. The findings presented here provide additional support for the utility of programs such as KEEP that train foster and kinship parents to manage difficult child behaviors.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed the reciept of the following financial support for the research, authorship, and/or publication of this article: This research was supported by funding from the Department of Health and Human Services - Administration for Children and Families - Children’s Bureau award HHS-2008-ACF-ACYF-CO-0046 and the University of Maryland School of Social Work Dean’s Research Fund.