Assessment of mental health difficulties in children and young people in care attending a specialist mental health service

Abstract

The purpose of this study is to test the validity of the Assessment Checklist measures in assessing complex mental health and behavioural difficulties of children and young people in care attending a specialist mental health service in Queensland, Australia. Fifty-eight consumers (53% male) with an average age of 8 years were assessed by carers on the Assessment Checklist for Children—Short Form and Strengths and Difficulties Questionnaire, and 44 consumers (36% male) with an average age of 13 years were assessed by carers on the Assessment Checklist for Adolescents—Short Form and Strengths and Difficulties Questionnaire. Results showed that the Assessment Checklist for Children—Short Form total score correlated with the Strengths and Difficulties Questionnaire total score. There were some correlations between subscales on both the measures. The Assessment Checklist for Adolescents—Short Form and Strengths and Difficulties Questionnaire correlated on very few domains. Domains of the Assessment Checklist measures that assess emotional dysregulation, trauma, interpersonal/attachment styles, sexual behaviour and food maintenance appear to provide additional clinical information about consumers that the standard Child and Youth Mental Health Service carer-report measure (Strengths and Difficulties Questionnaire) does not. It is recommended that the Assessment Checklist measures be used as an additional measure to assess the complexity of the children and young people in care who attend specialist mental health services.

Keywords

Children/young people mental health assessment children in care childhood trauma carers SDQ ACC/ACA

Introduction

Children and young people (C/YP) in care represent one of the most vulnerable and disadvantaged groups in Western societies, including Australia (Osborn & Bromfield, 2007; Tarren-Sweeney, 2008). There is significant empirical evidence that indicates C/YP in care are at an increased risk for developmental and mental health disorders across all facets of their life and lifespan. Negative outcomes include attachment difficulties, anxiety, depression, post-traumatic stress, conduct problems (including defiance, anger and aggression), sexual reactive problems, inattention/hyperactivity, and suicidal behaviour (Briere et al., 2001; Leve et al., 2012; Osborn et al., 2008; Oswald et al., 2010; Sawyer et al., 2007; Tilbury et al., 2007). Negative long-term outcomes include drug and alcohol use/misuse, poor physical health, homelessness, criminality and incarceration (Nathanson & Tzioumi, 2007; Norman et al., 2012; Richardson, 2005) highlighting the devastating costs that a history of complex trauma can have across an individual’s entire lifespan.

Psychopathology in children in care has been measured by parent-report rating scales such as the Rutter Questionnaires (Elander & Rutter, 1996), the Child Behaviour Checklist (CBCL) (Achenbach & Rescorla, 2001) and the Strengths and Difficulties Questionnaire (SDQ) (Goodman, 2001). Several studies have published CBCL scores and Rutter scores for children in care (Armsden et al., 2000; Heflinger et al., 2000; Horan et al., 1993; Pilowsky, 1995; Roy et al., 2000; Rushton et al., 1997; St. Claire & Osborne, 1987; Vorria et al., 1998). The SDQ has also been used in many published studies with children in care (Egelund & Lausten, 2009; Francis et al., 2017; Goemans et al., 2018; Goodman et al., 2004; Hiller & St. Clair, 2018; Larsen et al., 2018; Marquis & Flynn, 2009; Migliorini et al., 2016; Whyte & Campbell, 2008). The SDQ is based on the Rutter Questionnaires that were developed in the 1960s. Goodman updated the items of the Rutter Questionnaires to align with the DSM-IV and ICD-10 (Stone et al., 2010). The SDQ is relatively short compared to the CBCL. The CBCL has long been viewed as the gold standard in assessing child problems and is a solid instrument in doing an in-depth assessment of child psychopathology, while the SDQ is suitable for screening and treatment outcome monitoring. However, the Rutter Questionnaires, CBCL and SDQ do not adequately measure a range of problems manifested by children in care, particularly domains related to attachment and relationship difficulties, trauma responses, self-injury and sexual behaviour (Tarren-Sweeney, 2007). To compensate for the inadequacy of other measures, a carer-rated measure was developed to cover domains specific for children and adolescents in care (Tarren-Sweeney, 2007, 2013c) followed by development of the short form of these measures (Tarren-Sweeney, 2014).

The purpose of this study is to determine the validity of the Assessment Checklist for Children—Short Form (ACC-SF)/Assessment Checklist for Adolescents—Short Form (ACA-SF) in comparison to the SDQ in measuring mental health difficulties within a vulnerable Queensland population. Further, to clarify the clinical utility of the ACC-SF/ACA-SF as a supplementary diagnostic and treatment outcomes instrument as assessed by clinicians of the specialist mental health service. This research is relevant to assess the efficacy of the ACC-SF/ACA-SF in providing clinically meaningful information to clinicians working with children in care in a specialist mental health service.

It is hypothesized that the ACC-SF/ACA-SF will correlate with the SDQ.

In addition, it is hypothesized that the results obtained from the ACC-SF/ACA-SF will hold clinical utility given that the measures capture a range of difficulties that are not evaluated by the SDQ.

Method

Ethics approval was obtained from the Children’s Health Queensland Health and Hospital Services Human Research Ethics Committee (HREC/16/QRCH/8). Research governance approval was obtained from each of the three Hospital and Health Services where the study took place. Permission to use the Assessment Checklist measures was obtained from the developer Michael Tarren-Sweeney.

Specialist mental health service—Program description

In 2004, concerns regarding a clear unmet need for therapeutic services within the out-of-home care population was identified within the Queensland, Australia Crime and Misconduct Commission (CMC) report “Protecting Children: An Inquiry into Abuse of Children in Foster Care.” At the time it was identified that up to 17% of those in the care system were identified as having particularly high levels of complex and extreme psychological and behavioural problems requiring urgent intervention. As such, the CMC report recommended that “more therapeutic treatment services are made available to children with severe psychological and behavioural problems” and that the implemented service should be evaluated (CMC, 2004, p. 194). Following the report recommendations, Evolve Therapeutic Services (ETS) was established as a tertiary level mental health intervention service.

The key focus of ETS is to provide planned and coordinated therapeutic and behaviour supports to C/YP in care, aimed at improving their emotional wellbeing and the development of skills to enhance participation in school and in the community. Eligibility criteria includes: the child is under 18 years of age, presents with severe and/or complex psychological and/or behavioural problems (i.e. a chronic trauma history, extreme behavioural problems across multiple settings, at risk of harming self/others and multiple placement breakdowns), and is in care on interim or finalized Child Protection Orders. ETS is a trauma-informed program grounded in well-established theoretical perspectives (child development, systemic theory, trauma, attachment, psychodynamic theory and grief and loss) that encapsulates a collaborative “wrap-around” model of service. Provision of service is achieved through a flexible use of appropriate evidence-informed individual and systemic therapeutic interventions and a coordinated and sustainable partnership with key government, non-government and private sector agencies. Clinical mental health interventions include an initial and ongoing comprehensive assessment of the bio/psycho/social/cultural aspects of the C/YP and their significant others; risk management and safety planning; attachment and/or trauma focused therapies, which may include dyadic work, individual therapy, family-based intervention or the use of other treatment modalities; and, stakeholder coordination and support. ETS is an evidence based program that offers individual and systemic interventions that lead to improved consumer and carer wellbeing, placement stability, interpersonal relations, and stakeholder engagement (Eadie, 2017; Klag et al., 2016).

Currently there are 11 ETS teams located throughout Queensland. The vast majority of funded staff are frontline qualified and registered allied health workers (e.g. Psychiatry, Psychology, Social Work, Nursing, Occupational Therapy, Speech Pathology). All teams have a Professional Development Coordinator to manage inter-sectorial capacity building, and several teams have an Indigenous Program Coordinator.

Participants

Participants were foster carers, kinship carers or residential care workers for C/YP attending three Evolve Therapeutic Services’ sites in Queensland, Australia.

Fifty-eight carers (55 foster/kinship carers and 3 residential care workers) completed the ACC-SF and SDQ for the child in their care. Fifty-three percent (n = 31) of the consumers were male and the average consumer age was 7.9 years (S.D. = 1.5).

Forty-four carers (37 foster/kinship carers and 7 residential care workers) completed the ACA-SF and SDQ for the young person in their care. Thirty-six percent (n = 16) of the consumers were male and the average consumer age was 12.8 years (S.D. = 1.8).

Measures

Strengths and Difficulties Questionnaire

Two measures were used in this study. The SDQ is used as an outcome measure as part of standard practice in all child and youth mental health services in Queensland. The SDQ (Goodman, 1999) is a brief behavioural screening questionnaire about C/YP aged 4–17 years. There is a carer, teacher and self-report version of the measure. The SDQ includes 25 items relating to attributes of the target C/YP, some positive and some negative. The 25 items are divided between 5 sub-scales: Emotional symptoms, Conduct problems, Hyperactivity/Inattention, Peer relationship problems and Pro-social behaviours. The first four scales are also summed to yield a Total Difficulties score (Australian Mental Health Outcomes and Classification Network [AMHOCN], 2005). The SDQ has been shown to have good validity and reliability (Goodman, 2001). It has been used successfully with children in OOHC (Egelund & Lausten, 2009; Goemans et al., 2018; Goodman et al., 2004; Hiller & St. Clair, 2018; Marquis & Flynn, 2009; Whyte & Campbell, 2008), and norms have been verified in the Australian population (Hawes & Dadds, 2004), including an Aboriginal population (Zubrick et al., 2006).

Assessment Checklist for Children/Adolescents

The Assessment Checklist measures (Assessment Checklist for Children/Adolescents) began development in 2007 (Tarren-Sweeney, 2007). Clinical surveys of the mental health of children in foster care have failed to account for a range of problems manifested by such children largely because measurement has been restricted to standard parent-report checklists (Achenbach & Rescorla, 2001; Elander & Rutter, 1996; Goodman, 2001) and not measures specifically normed to children and young people in care. The measure has domains that are more specific to C/YP that have experienced trauma related to child maltreatment. The Assessment Checklist measures are completed by foster carers, kinship carers, residential care workers or adoptive parents.

There were several steps to the development of the clinical content (behaviours and mental health problems) of the ACC and ACA. These included reviews of clinical assessment reports, clinician surveys, a literature review, development of a conceptual framework, and a review of the draft instrument by foster carers and clinicians (Tarren-Sweeney, 2007, 2013a).

The ACC was designed to measure those problems manifested by children in care aged 5–10. The aim was to identify all clinically significant problems experienced by children in alternate care, which are not adequately measured by the CBCL. The long form of the instrument has good content, construct and criterion-related validity, and internal consistency (Tarren-Sweeney, 2007). The internal reliability of the ACC-SF total score and subscales were high (Tarren-Sweeney, 2014). The ACC-SF has 44 items and the following subscales, Sexual behaviour, Pseudomature, Non-reciprocal, Indiscriminate, Insecure, Anxious-distrustful, Abnormal pain response, Food maintenance, and Self injury.

The ACA was designed to measure those problems manifested by children in care aged 11–18. The long form of the ACA has good content, construct and criterion-related validity, as well as high internal reliability (Tarren-Sweeney, 2013c). The internal reliability of the ACA-SF total score and subscales were high (Tarren-Sweeney, 2014). The ACA-SF has 37 items and the following subscales, Non-reciprocal, Social instability, Emotional dysregulation, Dissociation/trauma symptoms, Food maintenance, and Sexual behaviour.

There have been several studies that have utilized the ACC/ACA measures, including the Brief Assessment Checklist for Children (BAC-C) and Brief Assessment Checklist for Adolescents (BAC-A), with some assessing the psychometric properties of the measures (Goemans et al., 2018; Tarren-Sweeney, 2007, 2013b, 2013c) and others conducting research studies (Chambers et al., 2010; Tarren-Sweeney, 2010, 2013a, 2017; Tarren-Sweeney et al., 2019; Tarren-Sweeney & Hazell, 2006; Tucker & Mares, 2013). Psychometric properties of the BAC compare with other existing screening instruments, including the SDQ and CBCL (Goemans et al., 2018; Tarren-Sweeney, 2013b). Therefore, the ACC-SF and ACA-SF instruments were chosen for the current study to assess clinically meaningful mental health difficulties across the various subscales of the instruments.

Procedure

Between April 2016 and June 2017, at three ETS sites, the ACC-SF/ACA-SF was completed by the carer at entry to the service, every 3 months and at discharge, in line with the SDQ completion. Carers included foster and kinship carers and residential workers. The ACC-SF/ACA-SF takes approximately 10 minutes to complete. The carer (or residential worker) was offered the written consent form, ACC-SF/ACA-SF and SDQ for completion by the Administration Officer (AO) while in the waiting room of the clinic. For clinicians attending a home visit, the consent form and measures were offered to the carer for completion. The completed measures were returned to the AO. The AO would scan and email the documents to the Research Officer each week. Each consumer’s ACC-SF/ACA-SF scores and profile was provided by the Research Officer to the clinician for clinical purposes. Clinicians were able to monitor progress or deterioration with subsequent completion of the measures, to assist in treatment/service provision.

Data analysis

Data from all carer types was combined as the development and normative data for the ACC/ACA included data combined from foster and kinship carers, and residential workers (Tarren-Sweeney, 2007). Data was analyzed using SPSS version 24. Descriptives were performed on ACC-SF/ACA-SF and SDQ subscales and the percentage of consumers allocated to clinical bands was calculated. Validity of the ACC-SF/ACA-SF against the SDQ was assessed using Pearson’s R analysis. Only paired data, where the ACC-SF/ACA-SF and SDQ were completed at the same time by a carer, was used for analysis. There was no missing data on any of the completed measures.

Results

Assessment Checklist for Children—Short Form

Descriptive analyses were performed on the ACC-SF subscales and total score and SDQ subscales and Total Problems Index (n = 58). Mean and standard deviation, and clinical bands for each subscale and total score are in Table 1.

Table 1.

Assessment Checklist for Children—Short Form & Strengths and Difficulties Questionnaire—Mean scores and clinical bands.

Clinical Scale	Mean (SD)	Clinical Bands (%)
Clinical Scale	Mean (SD)	Normal	Borderline	Clinical
ACC Sexual behaviour	1.04 (1.96)	69	2	29
ACC Pseudo-mature interpersonal behaviour	3.42 (2.39)	26	24	50
ACC Non-reciprocal interpersonal behaviour	4.25 (2.98)	19	26	55
ACC Indiscriminate interpersonal behaviour	3.52 (2.46)	29	31	40
ACC Insecure interpersonal behaviour	4.31 (2.71)	16	17	67
ACC Anxious-distrustful	3.42 (2.85)	33	10	57
ACC Abnormal pain response	1.00 (1.55)	93	5	2
ACC Food maintenance behaviour	1.81 (2.28)	71	12	17
ACC Self-injury	1.64 (2.24)	60	9	31
ACC Total score	24.42 (13.31)	14	5	81
SDQ Emotional symptoms scale	3.69 (2.50)	50	16	34
SDQ Conduct problems scale	5.07 (2.11)	12	16	72
SDQ Hyperactivity scale	5.17 (1.59)	45	38	17
SDQ Peer problems scale	4.28 (1.50)	9	24	67
SDQ Pro-social scale	5.60 (2.42)	52	17	31
SDQ Total Problems Index	18.21 (5.21)	17	24	59

Note: SDQ—This broad classification is based on information from the http://www.sdqinfo.org/ website © R Goodman (2012) and is derived from British norms. SDQ clinical bands are normal, borderline and clinical.

ACC-SF—Classification is based on information from The Assessment Checklist measures website, http://www.childpsych.org.uk/ © M Tarren-Sweeney (2013) and is derived from Australian norms. ACC-SF clinical bands are normal, elevated and clinical. For the purpose of SDQ comparison for this study, borderline was used for the elevated band.

The results in Table 1 provide an overall profile of the mental health difficulties that children experienced as rated by their carer on the ACC-SF and SDQ. Low mean scores were evident for ACC-SF subscales sexual behaviour, abnormal pain response, food maintenance behaviour and self injury. Eighty-one percent of the sample fell into the clinical range on the ACC-SF total score, and approximately two-thirds were in the clinical range for insecure interpersonal behaviour. Over half the sample were in the clinical range for pseudo-mature interpersonal behaviour, non-reciprocal interpersonal behaviour and anxious-distrustful. Over two-thirds of the sample fell into the clinical range on SDQ subscales conduct problems and peer problems. Fifty-nine percent of the sample fell into the clinical range on the SDQ total problems index.

Pearson’s R analysis was performed on the ACC-SF subscales and total score and SDQ subscales and Total Problems Index (n = 58). Results are in Table 2.

Table 2.

Correlation matrix for ACC-SF scales and SDQ scales.

	SDQ Emotional Symptoms	SDQ Conduct Problem	SDQ Hyperactivity	SDQ Peer Problems	SDQ Prosocial	SDQ Total Problems Index
ACC Sexual Behaviour	.380** (p = .003)	.115 (p = .389)	−.014 (p = .915)	−.131 (p = .325)	.109 (p = .413)	.187 (p = .160)
ACC Pseudomature	.389** (p = .003)	.366** (p = .005)	.078 (p = .558)	.103 (p = .442)	−.034 (p = .802)	.389** (p = .003)
ACC Non-reciprocal	.353** (p = .007)	.631* (p = .000)	.316*** (p = .016)	.011 (p = .933)	−.562* (p = .000)	.524* (p = .000)
ACC Indiscriminate	.146 (p = .274)	.155 (p = .244)	.158 (p = .235)	−.009 (p = .947)	−.060 (p = .655)	.179 (p = .179)
ACC Insecure	.648* (p = .000)	.143 (p = .283)	.056 (p = .677)	−.135 (p = .312)	.119 (p = .373)	.347** (p = .008)
ACC Anxious Distrustful	.585* (p = .000)	.272*** (p = .039)	.199 (p = .134)	−.169 (p = .204)	−.130 (p = .331)	.403** (p = .002)
ACC Abnormal pain response	.100 (p = .457)	.209 (p = .116)	.090 (p = .503)	.079 (p = .556)	−.123 (p = .358)	.183 (p = .170)
ACC Food maintenance	.278*** (p = .035)	.459* (p = .000)	.376** (p = .004)	.028 (p = .834)	−.155 (p = .247)	.442** (p = .001)
ACC Self injury	.396** (p = .002)	.268*** (p = .042)	.287*** (p = .029)	.257 (p = .052)	−.007 (p = .957)	.461* (p = .000)
ACC total	.646* (p = .000)	.511* (p = .000)	.305*** (p = .020)	−.005 (p = .972)	−.177 (p = .184)	.609* (p = .000)

* p < .001, **p < .01, ***p < .05.

Weak correlation 0.1–0.29.

Moderate correlation 0.3–0.69.

Strong correlation 0.7–1.0.

The ACC-SF total score and SDQ Total Problems Index (TPI) had a moderate positive correlation, r = .609 (p = .000), indicating that the instruments overall are measuring a similar construct.

ACC-SF Indiscriminate interpersonal behaviour and Abnormal pain response subscales did not correlate with any SDQ subscales, indicating that these domains are not being measured by the SDQ and may provide unique information about consumer behaviour. ACC-SF Sexual behaviour subscale had a weak positive correlation with SDQ Emotional symptoms, r = .380 (p < .01). ACC-SF Pseudomature interpersonal behaviour had a weak positive correlation with SDQ Emotional symptoms, r = .389 (p < .01), SDQ Conduct problems, r = .366 (p < .01) and SDQ Total Problems Index, r = .389 (p < .01). ACC-SF Non-reciprocal interpersonal behaviour had a moderate positive correlation with SDQ Emotional symptoms, r = .353 (p < .01), SDQ Hyperactivity, r = .316 (p < .05), SDQ Conduct Problems, r = .631 (p < .001), a moderate negative correlation with SDQ Prosocial, r = −.562 (p < .001) (a negative correlation would be expected from a clinical perspective), and a moderate positive correlation with SDQ Total Problems Index, r = .524 (p < .001).

ACC-SF Insecure interpersonal behaviour had a moderate positive correlation with SDQ Emotional symptoms, r = .648 (p < .001), and SDQ Total Problems Index, r = .347 (p < .01). ACC-SF Anxious- distrustful had a moderate positive correlation with SDQ Emotional symptoms, r = .585 (p < .001), a weak positive correlation with SDQ Conduct Problems, r = .272 (p < .05), and a moderate positive correlation with Total Problems Index, r = .403 (p < .01). ACC-SF Food maintenance behaviour had a weak positive correlation with SDQ Emotional symptoms, r = .278 (p < .05) and a moderate positive correlation with SDQ Conduct Problems, r = .459 (p < .001), SDQ Hyperactivity, r = .376 (p < .01), and Total Problems Index, r = .442 (p < .01). ACC-SF Self-injury had a moderate positive correlation with SDQ Emotional symptoms, r = .396 (p < .01), a weak positive correlation with SDQ Conduct Problems, r = .268 (p < .05), and SDQ Hyperactivity, r = .287 (p < .05), and a moderate positive correlation with the SDQ Total Problems Index, r = .461 (p < .001). As well as the moderate correlation the ACC-SF total had with the Total Problems Index, it correlated moderately with SDQ Emotional symptoms, r = .646 (p < .001), SDQ Conduct Problems, r = .511 (p < .001), and SDQ Hyperactivity, r = .305 (p < .05).

ACC-SF non-reciprocal interpersonal behaviour and self injury appear to be the most correlated subscales with the majority of SDQ subscales.

Assessment Checklist for Adolescents—Short Form

Descriptive analyses were performed on the ACA-SF subscales and total score and SDQ subscales and Total Problems Index (n = 44). Mean and standard deviation, and clinical bands for each subscale and total score are in Table 3.

Table 3.

Assessment Checklist for Adolescents—Short Form & Strengths and Difficulties Questionnaire—Mean scores and clinical bands.

Clinical Scale	Mean (SD)	Clinical Bands (%)
Clinical Scale	Mean (SD)	Normal	Borderline	Clinical
ACA Non-reciprocal interpersonal behaviour	4.60 (2.59)	5	16	79
ACA Social instability/behavioural dysregulation	7.23 (3.19)	5	20	75
ACA Emotional dysregulation/distorted social cognition	5.60 (3.00)	2	11	87
ACA Dissociation/trauma symptoms	1.92 (2.30)	55	32	13
ACA Food maintenance behaviour	2.64 (2.78)	68	16	16
ACA Sexual behaviour	1.15 (2.32)	66	18	16
ACA Total score	23.15 (9.48)	5	2	93
SDQ Emotional symptoms scale	4.86 (2.23)	25	16	59
SDQ Conduct problems scale	5.25 (1.83)	2	18	80
SDQ Hyperactivity scale	4.36 (2.00)	71	13	16
SDQ Peer problems scale	4.73 (1.48)	5	18	77
SDQ Pro-social scale	5.66 (2.18)	50	23	27
SDQ Total Problems Index	19.20 (4.65)	7	20	73

Note: SDQ—This broad classification is based on information from the http://www.sdqinfo.org/web site © R Goodman (2012) and is derived from British norms. SDQ clinical bands are normal, borderline and clinical.

ACA-SF—Classification is based on information from The Assessment Checklist measures website, http://www.childpsych.org.uk/ © M Tarren-Sweeney (2013) and is derived from Australian norms. ACA-SF clinical bands are normal, elevated, indicated clinical and marked clinical. For the purpose of SDQ comparison for this study, indicated and marked clinical were merged into the clinical band.

The results in Table 3 provide an overall profile of the mental health difficulties that adolescents experienced as rated by their carer on the ACA-SF and SDQ. Low mean scores were evident for ACA-SF subscales sexual behaviour and dissociation/trauma symptoms. Ninety-three percent of the sample fell into the clinical range on the ACA-SF total score, and over three-quarters of the sample fell into the clinical range on ACA-SF subscales social instability/behavioural dysregulation, non-reciprocal interpersonal behaviour and emotional dysregulation/distorted social cognition. Over three quarters of the sample fell into the clinical range on SDQ subscales conduct problems and peer problems. Seventy-three percent of the sample fell into the clinical range on the SDQ total problems index.

Pearson’s R analysis was performed on the ACA-SF subscales and total score and SDQ subscales and Total Problems Index (n = 44). Results are in Table 4.

Table 4.

Correlation matrix for ACA-SF scales and SDQ scales.

	SDQ Emotional Symptoms	SDQ Conduct Problem	SDQ Hyperactivity	SDQ Peer Problems	SDQ Prosocial	SDQ Total Problems Index
ACA Non-reciprocal	.242 (p = .114)	.127 (p = .410)	.059 (p = .704)	.089 (p = .567)	−.332*** (p = .028)	.220 (p = .152)
ACA Social instability	−.097 (p = .532)	.052 (p = .736)	−.009 (p = .954)	−.198 (p = .198)	.142 (p = .356)	−.093 (p = .548)
ACA Emotional dysregulation	.142 (p = .356)	.161 (p = .297)	−.041 (p = .791)	.043 (p = .782)	−.097 (p = .529)	.128 (p = .409)
ACA Dissociation and trauma	.279 (p = .067)	−.100 (p = .519)	−.006 (p = .970)	.013 (p = .932)	−.117 (p = .450)	.096 (p = .534)
ACA Food maintenance	−.238 (p = .120)	.542* (p = .000)	.051 (p = .745)	−.148 (p = .337)	.144 (p = .351)	.073 (p = .636)
ACA Sexual behaviour	−.058 (p = .711)	.313*** (p = .039)	.263 (p = .085)	.032 (p = .839)	−.208 (p = .174)	.218 (p = .154)
ACA total	.076 (p = .622)	.360*** (p = .016)	.083 (p = .594)	−.072 (p = .642)	−.127 (p = .413)	.191 (p = .215)

* p < .001, **p < .01, ***p < .05.

Weak correlation 0.1–0.29.

Moderate correlation 0.3–0.69.

Strong correlation 0.7–1.0.

ACA-SF subscales correlated with very few of the SDQ subscales, indicating that it is measuring unique domains for this adolescent population in OOHC. This is a different picture to the correlation between the ACC-SF and SDQ.

ACA-SF Non-reciprocal interpersonal behaviour had a moderate negative correlation with SDQ Prosocial, r = −.332 (p < .05). ACA-SF Food maintenance behaviour had a moderate positive correlation with SDQ Conduct problems, r = .542 (p < .001). ACA-SF Sexual behaviour had a moderate positive correlation with SDQ Conduct problems, r = .313 (p < .05). ACA-SF Total had a moderate positive correlation with SDQ Conduct problems, r = .360 (p < .05).

There was a weak non-significant correlation between the ACA-SF Total and SDQ Total Problems Index. This indicates that the instruments overall are measuring a different construct.

Discussion

This study assessed the validity of the ACC-SF/ACA-SF in measuring complex mental health and behavioural difficulties of C/YP in OOHC attending a specialist mental health service in Queensland, Australia. Three ETS sites took part in the study. Carer-report measures (ACC-SF/ACA-SF & SDQ) were completed by foster/kinship carers or residential workers and domains from each measure were correlated.

High mean scores and high numbers of children falling in the clinical band on several of the ACC-SF and SDQ subscales indicate that the tools are measuring the mental health concerns of a complex cohort of children attending the specialist mental health service. As a measure of total mental health problems, over 80% of the sample fell into the clinical range on the ACC-SF total score, while 59% of the sample fell into the clinical range on the SDQ total problems index. Low mean scores for ACC-SF subscales sexual behaviour, abnormal pain response, food maintenance behaviour and self injury are reasonable for children aged 5 to 10.

The ACC-SF total and SDQ Total Problems Index correlation indicates that the two measures are looking at similar constructs for this population. The hypothesis was achieved with regard to the total scores of the two measures but not across all the subscales. There are a number of subscales for both measures that correlate, some stronger than others. The ACC-SF correlated strongly with the SDQ Emotional Symptoms and Conduct Problems subscales and the overall SDQ Total Problems Index. The ACC-SF Indiscriminate interpersonal behaviour and Abnormal pain response subscales, however, did not correlate with any SDQ subscales. Overall, the ACC-SF clinical scales did not correlate with the SDQ Peer Problems and Prosocial scales. These results indicate that upon examination of the ACC-SF and SDQ measures, they capture similar and uniquely different information about consumer behaviour. In comparison to the SDQ, the ACC-SF explores attachment (pseudo mature, non-reciprocal, indiscriminate, insecure and distrust clinical scales), sexual and food maintenance behaviours in more detail.

High mean scores and high numbers of adolescents falling in the clinical band on several of the ACA-SF and SDQ subscales indicate that the tools are measuring the mental health concerns of a complex cohort of adolescents attending the specialist mental health service. As a measure of total mental health problems, over 93% of the sample fell into the clinical range on the ACA-SF total score, while 73% of the sample fell into the clinical range on the SDQ total problems index. Low mean scores were evident for ACA-SF subscales sexual behaviour and dissociation/trauma symptoms which was unexpected for this cohort.

In contrast to the ACC-SF, the ACA-SF subscales correlated with very few of the SDQ subscales. The hypothesis was not achieved for the ACA-SF. Statistically significant results between the measures was captured on 4 of the 42 possible subscale comparisons: SDQ Conduct problems with the ACA-SF Food maintenance and Sexual behaviour clinical scales and the ACA-SF total; and SDQ Prosocial and ACA-SF Non-reciprocal scales. These surprising results indicate two things. Firstly, both measures capture observable externalizing and pro-social behaviours. Secondly the ACA-SF may measure more significantly unique domains relevant to adolescent populations in care when compared to the ACC-SF.

Examination of the measure items supports this. Like the ACC-SF, the ACA-SF and SDQ are consistent in measuring externalizing behaviours whereas they diverge in relation to exploration of internalizing behaviours. Further, where the SDQ and ACA-SF are not statistically significantly, the ACA-SF explores attachment (pseudo mature, non-reciprocal, indiscriminate, insecure and distrust clinical scales), dissociation/trauma and food maintenance behaviours in more detail.

It is also worth considering, although the ACC-SF and the ACA-SF both have the same or similarly worded items, there are considerable differences in the age groups they evaluate. This is particularly important considering that the measures used in this study were carer-rated. With younger children, they are often more communicative and/or amenable to the care provider asking questions about their externalizing and internalizing presentation. Thus, the care providers may be more informed regarding the child’s presentation. The adolescent population however are often not as forthcoming and reluctant to engage in personally reflective conversations which may elicit feelings and/or show a level of vulnerability.

Feedback from clinicians supported the efficacy of this measure for the ETS population. Clinicians were asked if the Assessment Checklist measures were useful beyond the standard measures used in ETS and how the ACC-SF/ACA-SF might contribute to clinical practice. The ACC-SF/ACA-SF was reported by clinicians to be more complex trauma and attachment specific than the SDQ. For clinical practice, the ACC-SF/ACA-SF can act as a clinical screening tool that better captures areas of concern unique to the ETS consumer population that can guide more comprehensive mental health assessment and recovery-orientated treatment planning. Clinicians also reported that the measures were more effective in capturing symptoms of complex trauma and attachment challenges over time. Thus, the ACC-SF/ACA-SF was identified as being clinically more effective and sensitive when screening and monitoring challenges potentially faced by those with a complex trauma and care experience compared to the SDQ. Therefore, the hypothesis was achieved.

Use of the Assessment Checklist tools allow a better understanding of the factors influencing the mental health of children and adolescents in care. Complex trauma impacts individuals differently, however there are some common symptoms and behaviours for populations in the child protection context that can manifest in multiple ways. The ACC-SF and ACA-SF screen for behaviours that are known to impact C/YP in care including indiscriminate or insecure attachment, sexually reactive behaviours and food maintenance behaviours. The standard SDQ measure that is used within ETS is focused on emotional related symptoms, conduct problems, hyperactivity/inattention, peer relation problems and pro-social behaviours, therefore it does not screen for known behaviours of concern that are unique to trauma populations.

The information gathered by the ACC-SF/ACA-SF can be used to inform risk assessments and the development of individualized safety plans and recovery-focused interventions to address the specific challenges of concern. This was supported by clinicians who had experience with the measures during the study. Completing the ACC-SF/ACA-SF aided in facilitating discussions with carers and other stakeholders, such as school staff, foster support staff and child safety officers, around current functioning of consumers. This generated constructive therapeutic conversations that may not have occurred through the SDQ results alone.

The study has a number of limitations. Firstly, the study included two metropolitan and one rural clinic however did not include populations from remote areas of Queensland. Geographical location may have biased the results of the study. Secondly, the study focused on the use of the care provider response format. This format assumes that the respondent is aware of most, if not all of, the consumer’s presentation/behaviour/challenges. A large proportion of young people accessing the ETS program reside within residential services. Thus, there could be gaps in the respondent’s knowledge of the young person due to staff changeover in residential settings resulting in either under or over reporting.

Thirdly, the study did not consider or measure the level of stress being experienced by the care providers. Carer stress may impact on the accuracy of the care provider responses on measures. Lastly, the study was conducted in English through a westernized medical model. This is a limitation as over a third of those accessing the ETS service are from Aboriginal & Torres Strait Islander and Culturally & Linguistically Diverse populations. This may have implications should the study be replicated in non-western or non-English speaking populations as neither of the study measures have been language adapted.

This study looked at the validity of the Assessment Checklist Short Form measures in assessing complex mental health and behavioural difficulties of C/YP in care attending a specialist mental health service. Domains of the ACC-SF/ACA-SF that assess emotional dysregulation, trauma, interpersonal/attachment styles, sexual behaviour and food maintenance appear to provide additional clinical information about consumers that the standard Child and Youth Mental Health Service carer-report measure does not.

A further study will be conducted with all 11 ETS sites across Queensland to capture outcome data using the ACC-SF/ACA-SF which may be used for clinical and research purposes. It may also be of interest to determine if care provider stress and/or residing in a residential service may be influencing variables in terms of statistical significance between the measures. Further, research on the impact of placement history and duration of placement on the mental health difficulties of this cohort would be useful. Given that at the time of this study a third of the ETS consumer population are from Aboriginal and Torres Strait Islander or Culturally and Linguistically diverse populations, there is scope for further research to ensure that the ACC-SF and ACA-SF are both culturally sensitive and appropriate.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Kathryn Eadie

References

Achenbach

Rescorla

(2001). Manual for ASEBA school-age forms and profiles. University of Vermont, Research Center for Children, Youth, and Families.

Armsden

Pecora

P. J.

Payne

V. H.

Szatkiewicz

J. P.

(2000). Children placed in long-term foster care: An intake profile using the child behavior checklist/4–18. Journal of Emotional and Behavioral Disorders, 8(1), 49–64.

Australian Mental Health Outcomes and Classification Network (AMHOCN). (2005). Strengths and Difficulties Questionnaire Training Manual. Commonwealth of Australia. http://amhocn.org/static/files/assets/d666a3f8/SDQ_Manual.pdf

Briere

Johnson

Bissada

Damon

Crouch

Gil

Hanson

Ernst

(2001). The Trauma Symptom Checklist for Young Children (TSCYC): Reliability and association with abuse exposure in a multi-site study. Child Abuse and Neglect, 25, 1001–1014.

Chambers

M. F.

Saunders

A. M.

New

B. D.

Williams

C. L.

Stachurska

(2010). Assessment of children coming into care: Processes, pitfalls and partnerships. Clinical Child Psychology and Psychiatry, 15(4), 511–527.

Crime and Misconduct Commission (CMC). (2004). Protecting children: An inquiry into abuse of children in foster care. CMC.

Eadie

(2017). Evolve Therapeutic Services: Outcomes for children and young people in out-of-home care with complex behavioural and mental health problems. Children Australia, 42(4), 277–284.

Egelund

Lausten

(2009). Prevalence of mental health problems among children placed in out-of-home care in Denmark. Child & Family Social Work, 14, 156–165.

Elander

Rutter

(1996). Use and development of the Rutter parents’ and teachers’ scales. International Journal of Methods in Psychiatric Research, 6(2), 63–78.

10.

Francis

Y. J.

Bennion

Humrich

(2017). Evaluating the outcomes of a school based Theraplay® project for looked after children. Educational Psychology in Practice 33(3), 308–322.

11.

Goemans

Tarren-Sweeney

van Geel

Vedder

(2018). Psychosocial screening and monitoring for children in foster care: Psychometric properties of the Brief Assessment Checklist in a Dutch population study. Clinical Child Psychology and Psychiatry, 23(1), 9–24.

12.

Goodman

(1999). The extended version of the Strengths and Difficulties Questionnaire as a guide to child psychiatric caseness and consequent burden. Journal of Child Psychology and Psychiatry, 40(5), 791–798.

13.

Goodman

(2001). Psychometric properties of the Strengths and Difficulties Questionnaire. Journal of the American Academy of Child and Adolescent Psychiatry, 40, 1337–1345.

14.

Goodman

(2012). SDQ: Information for researchers and professionals about the Strengths and Difficulties Questionnaires. https://www.sdqinfo.org/

15.

Goodman

Ford

Corbin

Meltzer

(2004). Using the Strengths and Difficulties Questionnaire (SDQ) to screen looked after children for psychiatric disorders. European Child and Adolescent Psychiatry, 13(Supplement 2), 1125–1131.

16.

Hawes

D. J.

Dadds

M. R.

(2004). Australian data and psychometric properties of the Strengths and Difficulties Questionnaire. Australian and New Zealand Journal of Psychiatry, 38(8), 644–651.

17.

Heflinger

C. A.

Simpkins

C. G.

Combs-Orme

(2000). Using the CBCL to determine the clinical status of children in state custody. Children and Youth Services Review, 22(1), 55–73.

18.

Hiller

R. M.

St. Clair

M. C.

(2018). The emotional and behavioural symptom trajectories of children in long-term out-of-home care in an English local authority. Child Abuse & Neglect, 81, 106–117.

19.

Horan

Kang

Levine

Duax

Luntz

Tasa

(1993). Empirical studies on foster care: Review and assessment. Journal of Sociology and Social Welfare, 20(1), 131–154.

20.

Klag

Fox

Martin

Eadie

Bergh

Keegan

Turner

Raeburn

(2016). Evolve Therapeutic Services: A 5-year outcome study of children and young people in out-of-home care with complex and extreme behavioural and mental health problems. Children and Youth Services Review, 69, 268–274.

21.

Larsen

Baste

Bjorknes

Myrvold

Lehmann

(2018). Services according to mental health needs for youth in foster care?—A multi-informant study. BMC Health Services Research, 18(1), 634.

22.

Leve

L. D.

Harold

G. T.

Chamberlain

Landsverk

J. A.

Fisher

P. A.

Vostanis

(2012). Practitioner review: Children in foster care—Vulnerabilities and evidence-based interventions that promote resilience processes. Journal of Child Psychology and Psychiatry, 53(12), 1197–1211.

23.

Marquis

R. A.

Flynn

R. J.

(2009). The SDQ as a mental health measurement tool in a Canadian sample of looked-after young people. Vulnerable Children and Youth Studies, 4(2), 114–121.

24.

Migliorini

Rania

Cavanna

Cardinali

Guiducci

(2016). Comparing maltreating and foster families in Italy. Journal of Child and Family Studies 25(3): 746–755.

25.

Nathanson

Tzioumi

(2007). Health needs of Australian children living in out of home care. Journal of Paediatric Child Health, 43, 695–699.

26.

Norman

R. E.

Byambaa

Butchart

Scott

Vos

(2012). The long-term health consequences of child physical abuse, emotional abuse, and neglect: A systematic review and meta-analysis. PLoS Medicine, 9(11), e1001349.

27.

Osborn

Bromfield

(2007). Young people leaving care (Research Brief No. 7) . Australian Institute of Family Studies.

28.

Osborn

Delfabbro

Barber

J. G.

(2008). The psychosocial functioning and family background of children experiencing significant placement instability in Australian out-of-home care. Children and Youth Services Review, 30(8), 847–860.

29.

Oswald

S. H.

Heil

Goldbeck

(2010). History of maltreatment and mental health problems in foster children: A review of the literature. Journal of Paediatric Psychology, 35, 462–472.

30.

Pilowsky

(1995). Psychopathology among children placed in family foster care. Psychiatric Services, 46(9), 906–910.

31.

Richardson

(2005). Social costs: The effects of child maltreatment (Resource Sheet No. 9) . http://www.aifs.gov.au/nch/pubs/sheets/rs9/rs9.html

32.

Roy

Rutter

Pickles

(2000). Institutional care: Risk from family background or pattern of rearing? Journal of Child Psychology and Psychiatry and Allied Disciplines, 41(2), 139–149.

33.

Rushton

Quinton

Dance

Mayes

(1997). Preparation for permanent placement: Evaluating direct work with older children. Adoption and Fostering, 21(4), 41–48.

34.

Sawyer

M. G.

Carbone

J. A.

Searle

A. K.

Robinson

(2007). The mental health and wellbeing of children and adolescents in home-based foster care. Medical Journal of Australia, 186(4), 181–184.

35.

St. Claire

Osborne

(1987). The ability and behaviour of children who have been “in-care” or separated from their parents. Early Child Development and Care, 28, 187–354.

36.

Stone

Otten

Engels

Vermulst

Janssens

(2010). Psychometric properties of the parent and teacher versions of the Strengths and Difficulties Questionnaire for 4- to 12-year olds: A review. Clinical Child and Family Psychology Review, 13, 254–274.

37.

Tarren-Sweeney

(2007). The Assessment Checklist for Children—ACC: A behavioural rating scale for children in foster, kinship and residential care. Children and Youth Services Review, 29, 672–691.

38.

Tarren-Sweeney

(2008). The mental health of children in out-of-home care. Current Opinion in Psychiatry, 21, 345–349.

39.

Tarren-Sweeney

(2010). Concordance of mental health impairment and service utilization among children in care. Clinical Child Psychology and Psychiatry, 15(4), 481–495.

40.

Tarren-Sweeney

(2013). The Assessment Checklist measures. http://www.childpsych.org.uk/

41.

Tarren-Sweeney

(2013a). An investigation of complex attachment and trauma-related symptomatology among children in foster and kinship care. Child Psychiatry and Human Development, 44(6), 727–741.

42.

Tarren-Sweeney

(2013b). The Brief Assessment Checklists (BAC-C, BAC-A): Mental health screening measures for school-aged children and adolescents in foster, kinship, residential and adoptive care. Children and Youth Services Review, 35(5), 771–779.

43.

Tarren-Sweeney

(2013c). The Assessment Checklist for Adolescents—ACA: A scale for measuring the mental health of young people in foster, kinship, residential and adoptive care. Children and Youth Services Review, 35, 384–393.

44.

Tarren-Sweeney

(2014). Clinician’s guide to the Assessment Checklist Series: Specialized mental health measures for children in care. Routledge.

45.

Tarren-Sweeney

(2017). Rates of meaningful change in the mental health of children in long-term out-of-home care: A seven- to nine-year prospective study. Child Abuse & Neglect, 72, 1–9.

46.

Tarren-Sweeney

Goemans

Hahne

A. S.

Gieve

(2019). Mental health screening for children in care using the Strengths and Difficulties Questionnaire and the Brief Assessment Checklists: Guidance from three national studies. Developmental Child Welfare, 1(2), 177–196.

47.

Tarren-Sweeney

Hazell

(2006). Mental health of children in foster and kinship care in New South Wales, Australia. Journal of Paediatrics and Child Health, 42(3), 89–97.

48.

Tilbury

Osmond

Wilson

Clark

(2007). Good practice in child protection. Pearson Education Australia.

49.

Tucker

Mares

(2013). Establishing a mental health service for young children in out-of-home care: The Gumnut Clinic for 0 to 5 year olds in Western Sydney. Children and Youth Services Review, 35, 205–212.

50.

Vorria

Rutter

Pickles

Wolkind

Hobsbaum

(1998). A comparative study of Greek children in long-term residential group care and in two-parent families: I. Social, emotional, and behavioural differences. Journal of Child Psychology and Psychiatry and Allied Disciplines, 39(2), 225–236.

51.

Whyte

Campbell

(2008). The Strengths and Difficulties Questionnaire: A useful screening tool to identify mental health strengths and needs in looked after children and inform care plans at looked after children reviews? Child Care in Practice, 14, 193–206.

52.

Zubrick

S. R.

Lawrence

D. M.

De Maio

J. A.

Biddle

(2006). Testing the reliability of a measure of Aboriginal children’s mental health: An analysis based on the Western Australian Aboriginal Child Health Survey. Australian Bureau of Statistics.