The effect of the Development and Well-Being Assessment as an adjunct to standard referral letters on referral decisions by child and adolescent mental health services

Abstract

Background: High rejection rates for referrals to child and adolescent mental health services (CAMHS) are common. The most cited reasons for rejection are that the child does not have a clinical need for assessment and poor quality of the referrals. However, studies of interventions aimed at improving appropriateness of referrals are sparse. Methods: In this randomized feasibility trial, we tested if the Development and Well-Being Assessment (DAWBA) as an adjunct to referral letters could improve accuracy of referral decisions made by CAMHS. The primary outcome of the study was the proportion of “correct” referral decisions. Results: The study included 160 children referred to CAMHS. Almost all (95.6%) participants fulfilled criteria for a mental disorder and 82.1% also reported high impact of symptoms. Compared to the group who did not complete the DAWBA, referral decisions for the DAWBA group showed higher sensitivity (0.63 vs. 0.83), specificity (0.30 vs. 0.42), and negative predictive value (0.14 vs. 0.36) as well as slightly higher positive predictive value (0.81 vs. 0.86). Conclusions: The use of the DAWBA as an adjunct to standard referral letters could lead to more correct referral decisions and reduce the proportion of wrongful rejection referrals to CAMHS.

Keywords

child psychiatry adolescent psychiatry referral service use development and well-being assessment Kiddie schedule for affective disorders and schizophrenia-COMP

Introduction

The referral process to child and adolescent mental health services (CAMHS) is complex (Hartveit et al., 2011) and is influenced by characteristics of both the referring professional and the receiving specialized services as well as by patient characteristics (Forrest et al., 2006). Common barriers to the referral process are time constrains among referring professionals, uncertainties about referral criteria, and poor communication between primary and specialty services (Hartveit et al., 2011; O’Brien et al., 2016). In Denmark, referrals to CAMHS can be carried out by educational psychologist, caseworkers in social services, general practitioners (GPs), or other medical doctors. GPs are responsible for the majority of referrals to CAMHS in Denmark and many other countries (Hansen et al., 2021; Lambert et al., 2020). GP satisfaction with CAMHS is often reported as low due to challenges with accessing specialist help (Lambert et al., 2020) and referrals from GPs have been shown to be three times more likely to be rejected than referrals from other sources (Hinrichs et al., 2012). Studies on interventions aimed at improving the referral process to CAMHS are therefore highly relevant. However, research on appropriateness of outpatient referrals is scarce (Akbari et al., 2008; Mehrotra et al., 2011). To the best of our knowledge, only two studies with small sample size have previously reported on the effect of interventions aimed at the referral process to CAMHS (Neira-Munoz & Ward, 1998; Seierstad et al., 2017). Both investigated the effect of joint consultations with CAMHS specialists and found a positive effect on quality of referrals and a reduction in numbers of referrals (Neira-Munoz & Ward, 1998; Seierstad et al., 2017).

Ideally, the decision to refer should be appropriate and the referral letter should include a transfer of all relevant information (Mehrotra et al., 2011). The Danish National Health Authorities recommend that milder child mental disorders are managed in primary care settings (Sundhedsstyrelsen, 2017). Only children with more complex and severe symptoms with impaired functioning where previous interventions in primary settings have not had sufficient effect should be referred to CAMHS (Sundhedsstyrelsen, 2017).

The most commonly stated reason for rejection of referrals by CAMHS is that the referred child does not fulfill the criteria for assessment by CAMHS (Rambøll Management and Center for Kvalitetsudvikling, 2008; Seierstad et al., 2017; Smith et al., 2018).

However, systematic investigations of the proportion and characteristics of rejected children who do in fact fulfill assessment criteria have not previously been done. Another commonly reported reason for rejection is missing information in the referral letter (Rambøll Management and Center for Kvalitetsudvikling, 2008; Smith et al., 2018). A potential consequence of deficient information in referral letters could be rejection by CAMHS for children who are in fact in need of more specialized services which makes studies such as the present one of utmost importance.

Referral guidelines can provide an important foundation for improving the quality of referral letters (Mehrotra et al., 2011). However, passive dissemination of referral guidelines do not lead to improvements in referral behavior (Akbari et al., 2008) and for CAMHS in Denmark rejection rates have remained around 20–25% in the last decade (Hansen et al., 2021; Rambøll Management and Center for Kvalitetsudvikling, 2008), despite the introduction of referral guidelines. Another tool which could have the potential to improve the quality of referral is the use of a self-report diagnostic interview as an adjunct to standard referral letter which could provide systematic information on symptoms of mental disorders and their impact on the child’s functioning. The Development and Well-Being Assessment (DAWBA) could be a candidate for such a diagnostic interview. The DAWBA is a semi-structured diagnostic interview assessing child mental health symptoms (R. Goodman et al., 2000), which is easy to understand for parents (Last et al., 2014) and feasible to complete online (Last et al., 2014). It has been suggested that the DAWBA could be a valuable tool in the referral process (Ford et al., 2013). However, a systematic literature search of Embase and PubMed using the search term DAWBA yielded no previous studies investigating the usefulness of the DAWBA as part of the referral process.

This study aimed to investigate the effect of the DAWBA as an adjunct to referral letters on referral decisions by CAMHS. In addition, we wanted to investigate what proportion of children referred to CAMHS fulfills referral criteria.

We hypothesized that the use of the DAWBA would lead to a decrease in rejection rates, specifically for referrals from GPs, as referrals from GPs have previously been shown to most frequently lack relevant information (Rambøll Management and Center for Kvalitetsudvikling, 2008).

Methods

Design

The study was a randomized feasibility trial investigating the effect of the DAWBA in a parallel design comparing referred children randomized to completing the DAWBA with a group randomized to completing the extended version of the Strengths and Difficulties Questionnaire (SDQ) in a 1:1 allocation.

Participants

Eligibility criteria: The study included children aged 6–17 referred to outpatient CAMHS a random week out of every month from March 2019 to March 2020, regardless of referral reason. Children under 6 years of age were excluded, as the computerized Kiddie Schedule for Affective Disorders and Schizophrenia (KSADS-COMP) which was used to assess diagnostic criteria for a psychiatric disorder has not been validated for children younger than 6 years (Townsend et al., 2020).

Setting: The study was conducted at the Child and Adolescent Psychiatric Department of the North Denmark Region, which provides multidisciplinary specialist mental health services to children aged 0–17 with moderate to severe mental disorders. There are approximately 2300 referrals to the outpatient service each year (Hansen et al., 2021) and 20–25% of these are rejected (Hansen et al., 2021). Referrals are generally processed within 3 days of being received in order to ensure that children accepted for psychiatric assessment are seen within 30 days, which is a requirement for public hospitals in Denmark. This was the reason for the short window for including participants in the study.

Recruitment: Eligible participants were identified based on referral letters and invited to participate in the study at the time of referral. Information about the study was sent to a parent of the referred child using a secure email routinely used for all communication between hospitals and citizens in Denmark. If the child was placed in care, the responsible case worker was contacted in order to receive contact information for either the biological parent or other primary caregiver who then received the information. In few cases, the case worker could not be reached, or the case worker declined that the child could be invited to participate due to circumstances related to the placement.

Intervention

Participants were randomized to two groups (SDQ or DAWBA):

(1) SDQ: Families were asked to complete the SDQ. The CAMHS specialist making the referral decision (accept/reject) had access to the referral letter and the SDQ.

(2) DAWBA: Families were asked to complete the DAWBA which also included the SDQ. The CAMHS specialist making the referral decision had access to the referral letter and a clinician rating of the DAWBA.

The comparison group was asked to complete the SDQ to minimize differences in inclusion requirements and to ensure information on impact of the mental health problems for both groups. For both groups, the CAMHS specialist also had access to information from previous assessments in the electronic patient record if the child had previously been assessed.

Procedure: All parents were asked to complete the assigned assessment instrument (SDQ/DAWBA), and children ≥11 years received a self-report version of the SDQ/DAWBA. To be included in the study, at least one informant had to complete the SDQ and for the DAWBA group additionally at least one diagnostic section of the DAWBA interview before the referral decision was made. The family had approximately three days to fill out the DAWBA/SDQ from the time of the invitation for the study. All non-responders were sent a reminder about the study 24 hours after the invitation to participate in the study. All participants had access to telephone support if they had any problems with completing the assigned assessment instrument.

A diagnostic interview was conducted with included participants using the KSADS-COMP (Townsend et al., 2020). The KSADS-COMP interviews were conducted face-to-face, except for participants included in March 2020 (n=11) where interviews were conducted via secure video calls due to COVID-19 restrictions.

If the referral of the child was rejected by CAMHS, the family was offered a written summary of the KSADS-COMP interview including recommendation for what services to contact to access relevant support for their child.

Diagnostic measures

The DAWBA is a self-completed diagnostic interview (R. Goodman et al., 2000), developed to identify mental disorders as defined by ICD-10 (World Health Organization, 2004) and DSM-5 (American Psychiatric Association, 2013) applicable to 2–17 year-olds (Youth in Mind, 2010). It has high interrater reliability (Aebi et al., 2012) and there is a considerable agreement between diagnoses generated by the DAWBA and diagnoses assigned based on standard clinical assessment (Aebi et al., 2012; R. Goodman et al., 2000). The DAWBA includes a parent interview, and for children aged 11–17 there is also a self-report interview. A shorter teacher version exists which was not used in this study. The DAWBA consists of a mix of structured closed questions and open-ended questions about different mental health symptoms and their impact (A. Goodman et al., 2011). Following completion of the DAWBA interview, a clinical rating was carried out by a CAMHS clinician summarizing the information in the DAWBA and assigning diagnostic codes. This clinician’s summary and free text responses to any open questions asked were made available to the CAMHS specialist making the referral decision.

The extended version of the SDQ (R. Goodman, 1999) is a brief 25-item behavioral and emotional assessment tool with an impact supplement for 2–17 year-olds (R. Goodman, 1999). The SDQ has satisfactory to strong psychometric properties (Niclasen et al., 2012; Stone et al., 2010), is well validated (Stone et al., 2010) and useful in clinical research (Becker et al., 2004), and Danish norms for SDQ scores exist (Arnfred et al., 2019).

KSADS-COMP: The K-SADS-Present and Lifetime Version (K-SADS-PL) is a semi-structured diagnostic interview (Kaufman et al., 1997) which identifies mental disorders according to the DSM-V (Nishiyama et al., 2020). The KSADS-COMP is a computerized version of the K-SADS-PL (Townsend et al., 2020). The KSADS-COMP has demonstrated good convergent validity (Townsend et al., 2020). This study utilized the clinician administered version of the KSADS-COMP that has recently been translated into Danish by our research group.

Training of raters: All raters were specially trained physicians and psychologists at PhD/PhD student level with a minimum of four years of clinical experience in CAMHS. The DAWBA and the KSADS-COMP were never rated by the same person. All raters completed a two-day training with an expert in the relevant interview (K-SADS and/or DAWBA) prior to the initiation of the study and all raters received supervision by the same expert every three months throughout the duration of the study.

Randomization

Participants were randomized to two groups (SDQ or DAWBA) in a 1:1 computer-generated randomization stratified by sex using the statistical package R (R Core Team, 2013). Randomization of study participants was conducted by a research secretary involved in study management prior to participants being invited to participate in the study. An invitation to participate in the study with log-in for the DAWBA or SDQ was generated in accordance with the randomization. The clinicians conducting the KSADS-COMP interviews were not informed about randomization, content of the SDQ/DAWBA, or referral decision prior to the interview; however, in some cases, the families revealed parts of this information during the KSADS-COMP interview.

Outcomes

The primary outcome for the study was the proportion of referred children with a “correct” referral decision. A correctly accepted referral in the study was defined as a referral for a child who had one or more DSM-5 diagnoses based on the KSADS-COMP interview and a high impact score on the SDQ. Scores above the 90th percentile are classified as “High” (Arnfred et al., 2019). Secondly, we wanted to investigate what proportion of children referred to CAMHS fulfilled criteria for assessment by CAMHS.

Sample size

We included participants every month of the year to account for seasonal variation in referrals, and to ensure that the group of participants was as representative of the clinical population of referred children as possible. The decision to only include one week of every month was a pragmatic decision based on available resources for the study. Due to the sparsity of the literature in the field, it was not possible to make accurate assumptions of how big an effect the DAWBA would have on CAMHS referral decisions, and therefore not possible to make a justified sample size calculation.

Statistical methods

Descriptive statistics are presented as frequencies and percentages for categorical variables and medians and interquartile range (IQR) for continuous variables. For comparison of descriptive statistics between groups, t-test was applied for continuous variables and Fisher’s exact test for categorical variables.

To investigate representativeness of the study sample, we compared the sample to the general clinical outpatient population of all referrals to the study center in 2018.

We evaluated the ability of CAMHS to make correct referral decisions with or without the DAWBA by comparing referral decisions to our constructed KSADS-SDQ reference standard for clinical need of assessment. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were reported for the two groups. A subgroup analysis was conducted to test the accuracy of CAMHS referral decisions specifically for referrals from GPs compared to referrals from other referral sources.

Ethics approval

The study was reported to the Danish Data Protection Agency (ref-number 2018–186). No further ethics approval was necessary for performing this study. The study was conducted in concordance with the Helsinki Declaration. Consent to participate in the study was secured from the parent(s) with legal custody of the participating child even if only one parent actively participated in the study. For children placed in care, permission to invite the family was provided by the responsible case worker, who secured the consent from the legal guardian of the child and written consent to participate was provided by the participating primary caregiver(s). For all children aged 15–17 years of age, written consent was also secured from the child in accordance with Danish legislation.

Results

Study sample

Figure 1 shows the recruitment to the study. 570 children were eligible to participate in the study. For the SDQ group, 26.5% (n=76) filled out the SDQ in time to be included in the study. For the DAWBA group, 29.7% (n=84) filled out the DAWBA before a referral decision was made. In the DAWBA group, 90.5% of parents (n=76) completed the entire interview. Of the 62 children ≥11 years of age in the DAWBA group, 64.5% (n=40) completed the entire DAWBA. The KSADS-COMP was completed by 76.9% (n=123) of the included participants. There was no difference between the two groups with regards to how many completed the KSADS-COMP interview (p=1.00) (Table 1).

Figure 1.

Recruitment to the study.

Table 1.

Baseline characteristics.

	Full sample							Completed kiddie schedule for affective disorders and schizophrenia-COMP
	Total sample		DAWBA		SDQ		p-value	Total sample		DAWBA		SDQ		p-value
	N = 160		N = 84		N = 76			N = 123		N = 65		N = 58
Sociodemographic characteristics
Boys, n (%)	92	(57.1)	49	(58.3)	42	(55.3)	0.75	74	(60.2)	41	(63.1)	33	(56.9)	0.58
Median age (IQR)	13	(10–15)	13	(9–15)	13	(11–15)	0.42	14	(10–16)	14	(10–16)	14	(11–15)	0.46
Placement in care, n (%)	11	(6.8)	2	(2.4)	9	(11.8)	0.03	9	(7.3)	≤4	(≤6.2)	8	(13.8)	0.01
Placed in special education program, n (%)	16	(10.9)	10	(11.9)	6	(9.5)	0.79	14	(11.4)	9	(13.9)	5	(8.6)	0.41
Referral source
General practitioner, n (%)	98	(61.3)	53	(63.1)	45	(59.2)	0.63	77	(62.6)	42	(64.6)	35	(60.3)	0.71
Other (Pediatrician, Edu. Psychologist, case worker), n (%)	63	(38.8)	31	(36.9)	31	(40.8)	0.63	46	(37.4)	23	(35.4)	23	(39.7)	0.71
Primary referral diagnosis
Emotional disorder^a, n (%)	56	(35.0)	35	(41.7)	21	(27.6)	0.07	38	(30.9)	25	(38.5)	13	(22.4)	0.06
Neurodevelopmental disorder^b, n (%)	86	(53.8)	44	(52.4)	42	(55.3)	0.75	64	(52.0)	35	(53.9)	29	(50.0)	0.67
Other^c, n (%)	18	(11.3)	5	(6.0)	13	(17.1)	0.04	21	(17.1)	5	(7.7)	16	(27.6)	0.01
Symptom severity
Total SDQ problem score n (% with high/very high score)	124	(77.5)	65	(77.4)	59	(77.6)	1.00	99	(80.5)	51	(78.5)	48	(82.8)	0.65
SDQ impact score, n (% with high/very high score)	135	(84.4)	71	(84.5)	64	(84.2)	1.00	105	(85.4)	55	(84.6)	50	(86.2)	1.00
Referral rejected, n (%)	43	(26.7)	18	(21.2)	25	(32.9)	0.11	35	(28.5)	14	(21.5)	21	(36.2)	0.11
Completed K-SADS-COMP interview, n (%)	123	(76.4)	65	(76.5)	58	(76.3)	1.00	—	—	—	—	—	—	—

Includes affective disorders (DF30-39), anxiety disorders (DF40-42+DF93), and eating disorders (DF50).

Includes autism spectrum disorders (DF84), attention deficit disorders (DF90+DF98.8c), and tic disorders (DF95).

Includes all primary referral diagnoses (incl. non-specified/unknown) not specified in emotional disorder or neurodevelopmental disorder.

Note. SDQ = Strengths and Difficulties Questionnaire; DAWBA = Development and Well-Being Assessment

Characteristics of the study population

Table 1 shows the demographic and clinical characteristics of the participants in the two groups. The only statistically significant differences were that there were more children placed in care and more children with “other” as the primary referral diagnosis included in the SDQ group compared to the DAWBA group. Although fewer referrals in the DAWBA group were rejected by CAMHS, this was not statistically significant (p=0.11), nor was the difference significant when looking only at referrals from GPs (46.7% (SDQ) versus 28.3% (DAWBA), p=0.09).

With respect to representativeness of the study population, the study sample differed with regards to fewer being placed in special education programs at the time of referral (p=0.02) and fewer having previously been assessed for a mental disorder (p<0.01) compared to all referrals to the study center in 2018 (Table 2). Statistically significantly more children in the study sample were also referred for an emotional disorder compared to all referrals in 2018 (p=0.04) (Table 2).

Table 2.

Representativeness of the study sample.

	All referrals		Study sample		p-value
Time period	Jan–Dec 2018		Mar 2019–Mar 2020
	N2233	(%)	N160	(%)
Sociodemographic characteristics
Boys	1251	(56.0)	91	(56.9)	0.87
Median age (IQR)	13.6	(9.7–16.0)	13.8	(10.5–15.7)	0.21
Placement in care	184	(8.2)	11	(6.9)	0.65
Placed in special education program	403	(18.1)	17	(10.6)	0.02
Previously assessed for a mental health disorder	840	(37.6)	41	(25.6)	<0.01
Referral source
General practitioner	1252	(56.1)	98	(61.3)	0.22
Other medical doctor	271	(12.1)	18	(11.3)	0.90
Educational psychologist	595	(26.7)	35	(21.9)	0.20
Case worker social services	115	(5.2)	9	(5.6)	0.71
Primary referral diagnosis
Neurodevelopmental disorders	1218	(54.6)	86	(53.8)	0.87
Emotional disorders	604	(27.1)	56	(35.0)	0.04
Other	411	(18.4)	18	(11.3)	0.02
Referral rejected	536	(24.0)	43	(26.9)	0.44

Proportion of children fulfilling referral criteria for CAMHS

Almost all children who completed the KSADS-COMP fulfilled criteria for at least one mental disorder (95.9%, n = 118), and four out of five (79.7%, n = 98) fulfilled criteria for two or more diagnoses. There were no statistically significant differences in the proportion of children with comorbid disorders between the SDQ and the DAWBA group (p = 0.50) or when comparing those accepted to those rejected by CAMHS (p = 1.00).

There was no statistically significant difference between the SDQ group and the DAWBA group with regards to children fulfilling diagnostic criteria for a neurodevelopmental disorder (p = 0.68) or an emotional disorder (p = 0.70) (Table 3). Half of the children who completed KSADS-COMP (50.4%, n = 62) fulfilled criteria for both a neurodevelopmental and an emotional disorder.

Table 3.

Proportion of children with neurodevelopmental disorders and emotional disorders.

	Total sample		SDQ		DAWBA		p-value
	N	%	N	%	N	%
	123		58		65
Neurodevelopmental disorder	92^a	(74.8)	42	(72.4)	50	(76.9)	0.68
Emotional disorder	87^a	(70.7)	40	(69.0)	47	(72.3)	0.70

according to the KSADS-COMP.

Combined n for neurodevelopmental disorders and emotional disorder greater than the total n, due to co-morbidity.

Note. SDQ = Strengths and Difficulties Questionnaire; DAWBA = Development and Well-Being Assessment.

When combining KSADS-COMP diagnosis with a high impact score on the SDQ, 82.1% (n = 101) of the children who completed the KSADS-COMP fulfilled criteria for clinical need for assessment by CAMHS.

The effect of DAWBA as an adjunct to referral letters on referral decisions

Table 4 compares the sensitivity, specificity, PPV, and NPV of referral decision of the two groups for participants who completed the KSADS-COMP.

Table 4.

Sensitivity, specificity, PPV, and NPV with and without the use of DAWBA when making referral decisions.

	Full sample		Referred from GP		Other referral source
	SDQN = 58	DAWBAN = 65	SDQN = 35	DAWBAN = 42	SDQN = 23	DAWBAN = 23
Sensitivity	0.63	0.83	0.53	0.76	0.78	0.95
Specificity^a	0.30	0.42	0.50	0.50	-	0.14
Positive predictive value	0.81	0.86	0.89	0.87	0.74	0.86
Negative predictive value^a	0.14	0.36	0.18	0.33	—	0.50

Specificity and negative predictive value cannot be calculated for referrals from other referral sources.

Note. SDQ = Strengths and Difficulties Questionnaire; DAWBA = Development and Well-Being Assessment.

In the SDQ group, 63% (n = 30) of children with a clinical need for assessment were accepted by CAMHS in contrast to 83% (n = 44) in the DAWBA group. The sensitivity of the CAMHS referral decision was also markedly higher in the DAWBA group in the subgroup analysis for both referrals from GPs and from other referral sources. Of the referrals accepted by CAMHS, 81% (n = 30) in the SDQ group fulfilled the defined criteria for CAMHS assessment (PPV) compared to 86% (n = 44) in the DAWBA group.

The specificity of the referral decision by CAMHS was much lower than the sensitivity and only 30% (n≤4) of the rejected referrals in the SDQ group and 42% (n = 5) of rejected referrals in the DAWBA group did not fulfill the defined referral criteria. When evaluating referral decisions for referrals from GPs only, there was no difference in specificity between the two groups. However, there was a higher NPV for the referral decision in the group assigned to the DAWBA for both the full sample and for referrals from GPs.

Discussion

In this randomized feasibility trial, we tested the effect of the DAWBA as an adjunct to standard referral letters on referral decisions by CAMHS. To our knowledge, this is the first study to ever examine this use of the DAWBA.

Referral decisions in the group of children assigned to the DAWBA showed higher sensitivity and specificity and a higher NPV compared to the SDQ group indicating a decrease in the proportion of children incorrectly rejected without increasing the proportion of wrongfully accepted referrals. Other relevant findings of this study are that almost all the referred children (95.9%) fulfilled the criteria for a mental disorder and four out of five had comorbid psychiatric disorders.

There has been an ongoing discussion about the proportion of inappropriate referrals to CAMHS (Rambøll Management and Center for Kvalitetsudvikling, 2008) as a high proportion of referrals are rejected by CAMHS (Rambøll Management and Center for Kvalitetsudvikling, 2008; Socialstyrelsen, 2019; Weisser et al., 2019). From a commissioner’s point of view, there could be an interest in limiting the amount of referrals to CAMHS (Mallett et al., 2012) but for children with mental disorders and their families there are potentially great costs to not accessing services. The discussion of inappropriate referrals is further complicated by lack of evidence about appropriate referral rates (Davies & Elwyn, 2006).

Appropriateness of referrals is assessed using three dimensions; 1) necessity of the referral, 2) correct destination of referral, and 3) quality of the referral (Blundell et al., 2010). This study mainly evaluated the third dimension, namely, if improved quality of the referral letters as a result of the DAWBA being used as an adjunct would affect referral decisions by CAMHS, but the study findings in relation to the other dimensions are also briefly discussed.

With regards to necessity of referral to specialty services, we found that almost all referred children fulfilled criteria for at least one mental disorder at the time of referral but only 82.1% had both a mental disorder and reported high impact on functioning, and thus almost one fifth of referrals did not meet the criteria of the constructed reference standard for necessity of referral. This could indicate that just under a fifth of referrals are in fact inappropriate justifying the high rejection rate by CAMHS, but the NPV found in this study was low indicating that referrals for children in need of CAMHS assessment are also rejected. Also, the reference standard used in this study is a simplification that might not fully capture the complex concept of clinical need (Angold et al., 1999; Druss et al., 2007).

The relatively low specificity and NPV of the referral decision found in this study may be explained by the fact that assessment of correct destination of the referral was not included in the reference standard. According to stepped cared policies for child mental health, children should only be referred to CAMHS when assessment or intervention options are exhausted in primary care and more specialized service is needed (Akbari et al., 2008; Grimshaw et al., 2005). Referrals not indicating severe mental disorder might therefore be rejected by CAMHS if no other interventions have been attempted prior to referral. This approach is sensible in that it ensures that CAMHS resources are not being used on milder cases, which could be sufficiently helped by interventions in primary settings. However, there is a risk that professionals in primary setting do not have the knowledge or skills to correctly identify mental disorders and deliver evidence-based treatments. A recent study by Reardon et al. found that only 3% of children with anxiety disorder had received evidence-based treatment. In addition, pharmacological treatment is the most effective treatment for core symptoms of moderate to severe ADHD (Catalá-López et al., 2017), but in Denmark pharmacological treatment should only be initiated by CAMHS specialists (Retsinformation.dk, 2019) and thus the stepped care approach might delay initiation of relevant treatment.

This study aimed to assess if referral decisions would be more correct if the DAWBA was used as an adjunct to standard referral letters. The results indicate that the DAWBA is useful for this application and for referrals from GPs could have the potential to improve sensitivity of the referral decision to the level of referrals from other referral sources. We hypothesized that the use of the DAWBA as an adjunct to referral letters would lead to a decrease in rejection rates, particularly for referrals from GPs. Although we did see a trend toward lower rejection rates in the group randomized to the DAWBA compared to the SDQ group, this did not reach statistical significance. A potential explanation is that higher quality of referrals with regards to description of symptomatology does not affect the proportion of referrals rejected due to lack of prior interventions in primary settings. In the latter case, the DAWBA might be a more useful tool if it is applied in the process before referring professionals decide to refer a child to CAMHS as opposed to being applied after the decision to refer has already been made. This would require CAMHS specialists being available to make clinical ratings of completed DAWBA interviews and consult with referring professionals on how to proceed based on the assessment of the DAWBA.

Strengths and limitations

This study adds to the very limited existing research on interventions to improve appropriateness of referrals to CAMHS (Akbari et al., 2008; Seierstad et al., 2017). A strength of the study is that the intervention was tested in a randomized study design in a clinical setting resembling real world conditions. All instruments used in the study (DAWBA, K-SADS, and SDQ) are well-validated instruments that have been broadly applied in both research and clinical settings (Marzocchi et al., 2004; Matuschek et al., 2016; Zeinoun et al., 2013). Another strength of the study was that all clinical raters had several years of clinical experience working in CAMHS.

However, there are several limitations to the study. The reference standard used to determine correct referral decision was a simplified model for clinical need and did not account for inappropriate destination of referrals. However, to the best of our knowledge, this study is the first study investigating appropriateness of referrals to CAMHS which utilized a reference standard as a guide to evaluate accuracy of the referral decision. Another limitation was that we did not include the teacher’s version of the SDQ. Thus, high impact of symptoms only in school settings might have been missed by not including impact scores from the teacher’s SDQ in the reference standard.

We were not able to blind participants or the clinical raters to the randomization within the design of this study. This could have been achieved if all participants had completed the DAWBA and randomization following clinical rating had instead been made to disclosure or non-disclosure of the DAWBA to the CAMHS specialist making referral decisions.

The participation rate was low, and we cannot rule out the risk of selection bias with regards to participants. Children included in the study were less likely to have previously been assessed for a mental disorder and fewer were in special needs educational programs compared to all referrals to the study center. These two factors are most likely linked as having a diagnosed mental disorder is associated with placement in special education (Kommunernes Landsforening, 2018). The low participation rate could partly be a result of the very short timeframe (3 days) for filling out the assigned assessment instrument. As a result of the low participation rate, the sample size of the study was also smaller than intended and this affected the statistical power of the study with regards to showing an effect of the DAWBA on rejection rates. Based on the results of this feasibility trial, future RCTs investigating the DAWBA as an intervention to improve the referral process should include a minimum of 276 referrals from GPs specifically in order to show a statistically significant reduction in rejection rates for referrals from GPs of 15%.

Also, the study was conducted at a single CAMHS and referral decisions might be made differently at other sites, and the results of this study are therefore not necessarily generalizable to other CAMHS. However, high rejection rates are found nationally in Denmark (Rambøll Management and Center for Kvalitetsudvikling, 2008) and have also been reported from other countries (Hinrichs et al., 2012; Socialstyrelsen, 2019; Weisser et al., 2019). To address this limitation, future studies investigating the effect of DAWBA as an adjunct to referral letters on referral decisions by CAMHS could advantageously include multiple CAMHS centers.

Conclusion

The DAWBA could be a useful aid to improving accuracy of referral decisions by CAMHS for referrals from all referral sources. However, the DAWBA might be most valuable as an adjunct to referrals from GPs who generally have less time for assessment in daily practice compared to other referral sources. This feasibility trial focused on assessing the DAWBA’s value in addressing inappropriateness in the quality of referrals to CAMHS, which is only one part of the complex referral process. If a clinician rated DAWBA was instead used as a decision-making aid for GPs or other referral sources prior to making the decision to refer a child to CAMHS, it might also impact other dimensions of appropriateness of referrals. This would be a relevant intervention to test in future studies.

Footnotes

Acknowledgments

We are grateful to the participating families. We would like to thank Ulla Schierup Nielsen and Jens Richardt Møllegaard Jepsen for invaluable support with clinical ratings of the DAWBA and the KSADS-COMP and Heidi Hattmann Hjortshøj for coordinating the data collection. In addition, we would like to thank Professor Charlotte Ulrikka Rask and Professor Niels Bilenberg for training and supervision during the study. Lastly, we would like to thank the Referral Unit at the Child and Adolescent Psychiatric Department of Aalborg University Hospital, particularly Anne Randers, for collaborating on this study.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the study received funding from TrygFonden, Aase og Ejner Danielsens Fond, Helsefonden, and Den Obelske Familiefond. The funders had no influence on the design of the study, the conduction of the study, or the analysis and reporting of the data.

Data Availability

Data as well as the statistical code used for data analysis is available upon reasonable request directed to the corresponding author.

ORCID iD

Anna Sofie Hansen

References

Aebi

Kuhn

Metzke

C. W.

Stringaris

Goodman

Steinhausen

H. C.

(2012). The use of the development and well-being assessment (DAWBA) in clinical practice: A randomized trial. European Child and Adolescent Psychiatry, 21(10), 559–567. https://doi.org/10.1007/s00787-012-0293-6

Akbari

Mayhew

Al-Alawi

M. A.

Grimshaw

Winkens

Glidewell

Pritchard

Thomas

Fraser

(2008). Interventions to improve outpatient referrals from primary care to secondary care. The Cochrane Database of Systematic Reviews, 20(3), CD005471. https://doi.org/10.1002/14651858.CD005471.pub2

American Psychiatric Association, D.-5 T. F (2013). Diagnostic and statistical manual of mental disorders: DSM-5^TM. In: Diagnostic and statistical manual of mental disorders: DSM-5^TM (5th ed.). American Psychiatric Publishing, Inc. https://doi.org/10.1176/appi.books.9780890425596

Angold

Costello

E. J.

Farmer

E. M.

Burns

B. J.

Erkanli

(1999). Impaired but undiagnosed. Journal of the American Academy of Child and Adolescent Psychiatry, 38(2), 129–137. https://doi.org/10.1097/00004583-199902000-00011

Arnfred

Svendsen

Rask

Jeppesen

Fensbo

Houmann

Obel

Niclasen

Bilenberg

(2019). Danish norms for the strengths and difficulties questionnaire. Danish Medical Journal, Jun, 66(6), A5546.

Becker

Woerner

Hasselhorn

Banaschewski

Rothenberger

(2004). Validation of the parent and teacher SDQ in a clinical sample. European Child and Adolescent Psychiatry, 13(2), 11–6. https://doi.org/10.1007/s00787-004-2003-5

Blundell

Clarke

Mays

(2010). Interpretations of referral appropriateness by senior health managers in five PCT areas in England: A qualitative investigation. Quality and Safety in Health Care, 19(3), 182–186. https://doi.org/10.1136/qshc.2007.025684

Catalá-López

Hutton

Núñez-Beltrán

Page

M. J.

Ridao

Macías Saint-Gerons

Catalá

M. A.

Tabarés-Seisdedos

Moher

(2017). The pharmacological and non-pharmacological treatment of attention deficit hyperactivity disorder in children and adolescents: A systematic review with network meta-analyses of randomised trials. PloS One, 12(7), e0180355. https://doi.org/10.1371/journal.pone.0180355

Davies

Elwyn

(2006). Referral management centres: Promising innovations or Trojan horses? BMJ (Clinical Research), 332(7545), 844–846. https://doi.org/10.1136/bmj.332.7545.844.

10.

Druss

B. G.

Wang

P. S.

Sampson

N. A.

Olfson

Pincus

H. A.

Wells

K. B.

Kessler

R. C.

(2007). Understanding mental health treatment in persons without mental diagnoses: Results from the national comorbidity survey replication. Archives of General Psychiatry, 64(10), 1196–1203. https://doi.org/10.1001/archpsyc.64.10.1196

11.

Ford

Last

Henley

Norman

Guglani

Kelesidi

Martin

A. M.

Moran

Latham-Cork

Goodman

(2013). Can standardized diagnostic assessment be a useful adjunct to clinical assessment in child mental health services? A randomized controlled trial of disclosure of the development and well-being assessment to practitioners. Social Psychiatry and Psychiatric Epidemiology, 48(4), 583–593. https://doi.org/10.1007/s00127-012-0564-z

12.

Forrest

C. B.

Nutting

P. A.

Von Schrader

Rohde

Starfield

(2006). Primary care physician specialty referral decision making: Patient, physician, and health care system determinants. Medical Decision Making, 26(1), 76–85. https://doi.org/10.1177/0272989X05284110

13.

Goodman

(1999). The extended version of the strengths and difficulties questionnaire as a guide to child psychiatric caseness and consequent burden. J. Child Psychol. Psychiat, 40(5), 791–799. https://doi.org/10.1111/1469-7610.00494

14.

Goodman

Ford

Richards

Gatward

Meltzer

(2000). The development and well-being assessment: Description and initial validation of an integrated assessement of child and adolescent psychopathology. Journal of Child Psychology and Psychiatry and Allied Disciplines, 41(5), 645–655. https://doi.org/10.1017/S0021963099005909

15.

Goodman

Heiervang

Collishaw

Goodman

(2011). The “DAWBA bands” as an ordered-categorical measure of child mental health: Description and validation in British and Norwegian samples. Social psychiatry and psychiatric epidemiology, 46(6), 521–532. https://doi.org/10.1007/s00127-010-0219-x

16.

Grimshaw

J. M.

Winkens

R. A. G.

Shirran

Cunningham

Mayhew

Thomas

Fraser

(2005). Interventions to improve outpatient referrals from primary care to secondary care. The Cochrane Database of Systematic Reviews, 3(3), CD005471. https://doi.org/10.1002/14651858.CD005471

17.

Hansen

A. S.

Kjaersdam Telléus

Lauritsen

M. B.

(2021). Changes in referral patterns to outpatient child and adolescent psychiatric services from 2005–2018. Nordic Journal of Psychiatry, 75(6), 437–446. https://doi.org/10.1080/08039488.2021.1880636

18.

Hartveit

Biringer

Vanhaeht

Haug

Aslaksen

(2011). The Western Norway mental health interface study: A controlled intervention trial on referral letters between primary care and specialist mental health care. BMC Psychiatry, 11(177). https://doi.org/10.1186/1471-244X-11-177.

19.

Hinrichs

Owens

Dunn

Goodyer

(2012). General practitioner experience and perception of Child and Adolescent Mental Health Services (CAMHS) care pathways: A multimethod research study. BMJ Open, Nov, 2(6), e001573. https://doi.org/10.1136/bmjopen-2012-001573

20.

Kaufman

Birmaher

Brent

Rao

Flynn

Moreci

Williamson

Ryan

(1997). Schedule for affective disorders and schizophrenia for school-age children-present and lifetime version (K-SADS-PL): Initial reliability and validity data. Journal of the American Academy of Child and Adolescent Psychiatry, 36(7), 980–988. https://doi.org/10.1097/00004583-199707000-00021.

21.

Kommunernes Landsforening . (2018). Børns diagnoser og skoletyper (Children’s diagnoses and types of schooling. [In Danish]. https://www.kl.dk/media/14395/ojyf-ezrxvgiuwyfuxvd.pdf

22.

Lambert

A. K.

Doherty

A. J.

Wilson

Chauhan

Mahadevan

(2020). GP perceptions of community-based children’s mental health services in Pennine Lancashire: A qualitative study. BJGP Open, 4(4), bjgpopen20X101075. https://doi.org/10.3399/bjgpopen20x101075.

23.

Last

Henley

Norman

Goodman

Ford

(2014). Innovations in Practice: Feasibility of the development and well-being assessment as an adjunct to clinical assessment in child and adolescent mental health services. Child and Adolescent Mental Health, 19(2), 142–146. https://doi.org/10.1111/camh.12017

24.

Mallett

Halligan

Matthew Thompson

G. P.

Collins

G. S.

Altman

D. G.

(2012). Interpreting diagnostic accuracy studies for patient care. BMJ (Online), 345, e3999. https://doi.org/10.1136/bmj.e3999

25.

Marzocchi

G. M.

Capron

Di Pietro

Duran Tauleria

Duyme

Frigerio

Gaspar

M. F.

Hamilton

Pithon

Simões

Thérond

(2004). The use of the Strengths and Difficulties Questionnaire (SDQ) in Southern European countries. European Child & Adolescent Psychiatry, 13(2), ii40–ii46. https://doi.org/10.1007/s00787-004-2007-1

26.

Matuschek

Jaeger

Stadelmann

Dölling

Grunewald

Weis

von Klitzing

Döhnert

(2016). Implementing the K-SADS-PL as a standard diagnostic tool: Effects on clinical diagnoses. Psychiatry Research, Feb 28(236), 119-124. https://doi.org/10.1016/j.psychres.2015.12.021.

27.

Mehrotra

FORREST

C. B.

Lin

C. Y.

(2011). Dropping the baton: Specialty Referrals in the United States. The Milbank Quarterly, 89(1), 39–68. https://doi.org/10.1111/j.1468-0009.2011.00619.x

28.

Neira-Munoz

Ward

(1998). Child mental health services. Side by side. The Health Service Journal, 108(5617), 26–27

29.

Niclasen

Teasdale

T. W.

Andersen

A. M. N.

Skovgaard

A. M.

Elberling

Obel

(2012). Psychometric properties of the Danish strength and difficulties questionnaire: The SDQ assessed for more than 70,000 raters in four different cohorts. PLoS ONE, 7(2), e32025. https://doi.org/10.1371/journal.pone.0032025

30.

Nishiyama

Sumi

Watanabe

Suzuki

Kuru

Shiino

Kimura

Wang

Lin

Ichiyanagi

Hirai

(2020). The kiddie schedule for affective disorders and schizophrenia present and lifetime version (K-SADS-PL) for DSM-5: A validation for neurodevelopmental disorders in Japanese outpatients. Comprehensive Psychiatry, Jan(96), 152148. https://doi.org/10.1016/j.comppsych.2019.152148.

31.

O’Brien

Harvey

Howse

Reardon

Creswell

(2016). Barriers to managing child and adolescent mental health problems: A systematic review of primary care practitioners’ perceptions. British Journal of General Practice, 66(651), e693–707. https://doi.org/10.3399/bjgp16X687061

32.

R Core Team (2013). A language and environment for statistical computing. http://www.r-project.org/.

33.

Rambøll Management & Center for Kvalitetsudvikling, R. M (2008). Henvisningsprojektet i børne- og ungdomspsykiatrien (in Danish)-. 978-87-7601-255-7. https://sum.dk/Aktuelt/Publikationer/∼/media/Filer-Publikationer_i_pdf/2008/Henvisningsprojektet.ashx

34.

Retsinformation, dk . (2019). Vejledning om medikamentel behandling af børn og unge med psykiske lidelser. (in Danish). https://www.retsinformation.dk/eli/retsinfo/2019/9733.

35.

Seierstad

T. G.

Brekke

Toftemo

Haavet

O. R.

(2017). GPS’ and child and adolescent psychiatry specialists’ experiences of joint consultations in the GP’s office: A qualitative study. BMC Research Notes, 10(1), 458. https://doi.org/10.1186/s13104-017-2766-7

36.

Smith

Kyle

R. G.

Daniel

Hubbard

(2018). Patterns of referral and waiting times for specialist child and adolescent mental health services. Child and Adolescent Mental Health, 23(1), 41–49. https://doi.org/10.1111/camh.12207

37.

Socialstyrelsen (2019). Psykiatrisk vård och behandling till barn och unga. (in Swedish) https://www.socialstyrelsen.se/globalassets/sharepoint-dokument/artikelkatalog/oppna-jamforelser/2019-12-6475.pdf.

38.

Stone

L. L.

Otten

Engels

R. C. M. E.

Vermulst

A. A.

Janssens

J. M. A. M.

(2010). Psychometric properties of the parent and teacher versions of the strengths and difficulties questionnaire for 4- to 12-Year-olds: A review. Clinical Child and Family Psychology Review, 13(3), 254–274https://doi.org/10.1007/s10567-010-0071-2

39.

Sundhedsstyrelsen (2017). Forløbsprogram for børn og unge med angst og/eller depression. https://www.sst.dk/da/udgivelser/2017/∼/media/E62CECC2AE9440818723CADE96438A7B.ashx

40.

Townsend

Kobak

Kearney

Milham

Andreotti

Escalera

Alexander

Gill

M. K.

Birmaher

Sylvester

Rice

Deep

Kaufman

(2020). Development of three web-based computerized versions of the kiddie schedule for affective disorders and schizophrenia child psychiatric diagnostic interview: Preliminary validity data. Journal of the American Academy of Child and Adolescent Psychiatry, 59(2), 309–325. https://doi.org/10.1016/j.jaac.2019.05.009

41.

Weisser

K. H.

Diseth

T. H.

Boye

Faerden

Ekeberg

Ø.

(2019). Examining the organization and quality of the psychiatric consultative service in Norway. Nordic Journal of Psychiatry, 73(1), 9–15. https://doi.org/10.1080/08039488.2018.1525426

42.

World Health Organization (2004). ICD-10: International statistical classification of diseases and related health problems: Tenth revision (2nd ed). World Health Organization. https://apps.who.int/iris/handle/10665/42980

43.

Youth in Mind (2010). Development and well-being assessment. https://youthinmind.com/products-and-services/dawba/.

44.

Zeinoun

Bawab

Atwi

Hariz

Tavitian

Khani

Nahas

Maalouf

F. T.

(2013). Validation of an Arabic multi-informant psychiatric diagnostic interview for children and adolescents: Development and well being assessment-Arabic (DAWBA-Arabic). Comprehensive Psychiatry, 54(7), 1034–1041. https://doi.org/10.1016/j.comppsych.2013.04.012

The effect of the Development and Well-Being Assessment as an adjunct to standard referral letters on referral decisions by child and adolescent mental health services — a randomized feasibility trial

Abstract

Keywords

Introduction

Methods

Design

Participants

Intervention

Diagnostic measures

Randomization

Outcomes

Sample size

Statistical methods

Ethics approval

Results

Study sample

Characteristics of the study population

Proportion of children fulfilling referral criteria for CAMHS

The effect of DAWBA as an adjunct to referral letters on referral decisions

Discussion

Strengths and limitations

Conclusion

Footnotes

Acknowledgments

Declaration of Conflicting Interests

Funding

Data Availability

ORCID iD

References