Opening the Black Box

Abstract

Objective:

The Taxonomy of Care for Youth was developed to gather information about the care offered to children and adolescents with behavioral and emotional problems in various care settings. The aim was to determine similarities and differences in the content of care and thereby to classify the care offered to these children and youth within and across care organizations.

Method:

Interventions (N = 56) offered in primary health care, child and youth care, and mental health care were assessed by using descriptors. Professionals scored the degree of applicability of these descriptors and interventions with similar descriptors were merged.

Results:

As a result, the interventions could be classified into 7 main types of support and the total number of interventions reduced to 27.

Conclusion:

The descriptors used in this study were able to classify the various kinds of care offered based on their content, thereby creating an overview of distinct interventions.

Keywords

mental health care child and youth care primary health care behavioral problems emotional problems

Introduction

The characteristics of care and treatment for children with behavioral and emotional problems are of major interest to those concerned with improving the effectiveness of care (Abraham & Michie, 2008; Ballinger, Asburn, Low, & Roderick, 1999; Garland et al., 2010; Libby et al., 2005; Lloyd-Evans, Johnson, & Slade, 2007). Practitioners and policy makers alike require an understanding of these characteristics to make well-founded decisions concerning the kind of care to be offered (Chorpita & Daleiden, 2009; Cjaza, Schulz, Lee, & Belle, 2003; Ezell et al., 2011; Harden & Klein, 2011; Knorth et al., 2011; Lee & Barth, 2011; Marsh, Angell, Andrews, & Curry, 2012; Miller & Row, 2009; Yohalem & Wilson-Ahlstrom, 2010). To date, there has been a lack of consensus concerning how to gather information on the characteristics of care and treatment. In addition, the labeling of interventions does not seem to accurately reflect the actual content of care. In fact, identical labels might be used for completely different care modules or treatments, while similar treatments may be given different labels (Van Yperen, Van Rest, & Vermunt, 1999; Lloyd-Evans et al., 2007).

The number of interventions offered within child and youth care (CYC) organizations has increased tremendously (Kazdin, 2000; Loeffen, Ooms, & Wijgergangs, 2004; Veerman, Janssens, & Delicat, 2005). A study by Loeffen, Ooms, and Wijgergangs (2004) demonstrated that within the Netherlands there were approximately 1,500 different labels for interventions within CYC. Even for one specific type of support, known as Family Preservation Services, there were 92 different labels (Veerman et al., 2005). These studies suggest that the greater part of these interventions actually entail the same type of care. Therefore, more knowledge is urgently needed about the characteristics of care and treatment.

In addition, information about the characteristics of care may also enable researchers to determine associations between problem behavior, the care delivered to children and their families, and the outcomes (DeJong, Horn, Gassaway, Slavin, & Dijkers, 2004; Fein, 2002; Maschi, Hatcher, Schwalbe, & Rosato, 2008). Until recently, none of the few taxonomic systems available in the field of CYC were capable of recording information about the most salient aspects of the care process (Evenboer, Huyghen, Tuinstra, Reijneveld, & Knorth, 2012a). The Taxonomy of Care for Youth (TOCFY) was thus developed within the context of the Collaborative Center on Care for Children and Youth (C4Youth; Evenboer, Huyghen, Tuinstra, Knorth, & Reijneveld, 2012b).

TOCFY contains six domains regarding the content, judicial context, duration, intensity, and recipients of care, and the expertise of the professionals most involved (Evenboer, Huyghen, Tuinstra, Knorth, & Reijneveld, 2012b). The terminology used in the domains of judicial context, duration, intensity, recipients, and professional expertise are similar across organizations, thus enabling a comparison of interventions across organizations in these domains. However, the first TOCFY domain, content of care, primarily contains a list of organization-specific labels of interventions. While this might facilitate manageability for the professionals using TOCFY (Evenboer, Huyghen, Tuinstra, Knorth, & Reijneveld, 2012b; Evenboer, Huyghen, Tuinstra, Knorth, & Reijneveld, 2014), these organization-specific labels also limit our ability to compare the care offered within and across care organizations.

Therefore, before being able to compare the care offered, the content of these interventions must be further assessed to determine whether interventions with different labels actually differ in content. This can be achieved by asking experts to evaluate the content of the interventions they are familiar with using lists of standardized descriptors (cf. Abraham & Michie, 2008; Michie, Hyder, Walia, & West, 2011). We used the database “Effective Youth Interventions” (EYI) of the Netherlands Youth Institute (2013) to provide a list of descriptors. For each intervention included in this database, a wide variety of phrases is used to indicate methodical or technical aspects of an intervention. We provided an overview of the most frequently used phrases or “techniques” and reformulated them into a standardized format that reflects the action orientation of the care worker or therapist. An example is the following expression—“behavioral instructions.” It was reformulated as—“prompting client to practice behavioral instructions.” Finally, the list of standardized descriptors was applied for categorizing the various types of interventions offered to children with behavioral and emotional problems. Such assessment provides the opportunity to determine the nature of any similarities or differences concerning the content of the care offered within and across care organizations.

The aim of this study is therefore to determine the similarities and differences in content of care, that is to classify the care offered to these children and adolescents within and across care organizations, thereby using a specific assessment procedure. In this study, “care and treatment” entails the activities and techniques carried out by professionals to help children and adolescents with behavioral and emotional problems within the field of primary health care (PHC), CYC, and mental health care (MHC; see also Fonagy, Target, Cottrell, Phillips, & Kurtz, 2002; Wade, Biehal, Farrelly, & Sinclair, 2011).

Method

Sample

Four care organizations from the field of PHC (offering N = 7 interventions), CYC (N = 42), and MHC (MHC-A, N = 31; MHC-B, N = 11) participated in the C4Youth study. The manuals concerning the interventions offered by these four care organizations (in total N = 91) were used to obtain more detailed information about the content of care.

Procedure

We first assessed whether the 91 interventions were conceptually and empirically founded. We used four criteria—formulated and applied by an independent committee of national experts evaluating interventions in the context of accreditation (Zwikker, Van Dale, & Kuunders, 2009)—to determine (1) whether a protocol description was available, (2) whether the intervention was theoretically well founded, (3) whether research had been done on the intervention, and (4) whether scientific literature had been published on the intervention (cf. Veerman & Van Yperen, 2008). In total, 56 interventions did not meet all four criteria and were labeled as “poorly defined.” The other 35 could be labeled as “well-defined.” In the current study, we focused on the poorly defined interventions, considering this to be the most difficult group to categorize. Thereafter, the method could and shall also be used in an additional study to characterize the contents of the group of well-defined interventions. The procedure that was followed consisted of four steps (Figure 1).

Figure 1.

Overview of the procedure used to identify distinct interventions.

First, we categorized these 56 interventions in terms of main types of support, a term indicating the whole range of activities carried out to improve the functioning and development of children and adolescents (and their families). The categorization was made based on the names of the interventions and the explanations in the manuals available. Thereby we used an inductive approach, with the terminologies and descriptions in the treatment manuals as the leading indicators for defining these main types of support. In this way, we can assume that these main types of support cover the range of interventions offered to children and adolescents with behavioral and emotional problems in our study.

Second, we collected descriptors for each main type of support using the interventions included in the EYI database (Netherlands Youth Institute, 2013). The manuals of the selected interventions from this database were analyzed and the 20 most frequently used descriptors across these intervention manuals were put together, resulting in separate lists for each type of support (Appendix). A “descriptor” refers to a short sentence that describes a care component or activity undertaken in the intervention (e.g., providing parents with information about the situation, prompting a child to express emotions, teaching a young person how to deal with setbacks and frustrations, etc.). A set of 20 descriptors was collected for each main type of support, which was found to be a feasible amount for enabling a general categorization of the content of interventions (cf. Abraham & Michie, 2008).

Third, professionals working at the participating care organizations scored the descriptors on a 7-point Likert-type scale ranging from (1) very poor to (7) very good. Very poor meant that a descriptor was very inaccurate in representing a methodical aspect, while very good meant that the descriptor precisely represented the methodical or technical aspect that was to be carried out. The professional background of the coders varied from psychiatry and psychology to behavioral science and family counseling. Each intervention was assessed by two professionals to decrease the potential bias. Fourth, the scores of the professionals were analyzed and those interventions that scored similarly on content were merged into one distinct intervention.

Analysis and Reporting

The method used in this study was developed in order to examine whether and to what degree we could distinguish profiles of descriptors characterizing the differentially labeled interventions. We take as a starting point the “mean profile” of descriptors per main type of support. This profile is used as a framework with which the interventions could be compared within a specific main type of support. The interventions were compared to others within the same type of support category in order to identify similarities and differences concerning content. First, we computed mean scores per descriptor, by using the individual scores of the two raters and compared these to the overall mean scores per descriptor for each intervention and main type of support. In deciding to merge interventions, 60% of these mean scores per descriptor were allowed to differ from the overall mean score per descriptor up to a maximum of 0.5 of a rating point. The total overall mean score of an intervention was allowed to differ by a maximum of 0.5, compared to the overall mean score for the main type of support. Interventions that did not meet both criteria—and thus were not merged with other interventions—were then compared pairwise using the mean scores per descriptor rather than the overall mean scores per descriptor. For each remaining pair that was compared, no more than 25% of the descriptors were allowed to deviate from each other by more than 1.0 point if they were to be merged.

Results

In total, 56 interventions derived from the PHC organization (n = 1), the CYC organization (n = 29) and the 2 MHC organizations (MHC-A, n = 20; MHC-B, n = 6) were analyzed. The first step (Figure 1) resulted in seven main types of support—“family support,” “parenting support,” “individual child support,” “trauma support,” “foster care support,” “experiential learning support,” and “independent living support.” The number of interventions before and after the four steps can be seen in Table 1, including examples of descriptors for each main type of support.

Table 1.

Number of Interventions for Each Main Type of Support, Before And After Analysis.

Main Type of Support	n (Before Analysis)	n (After Analysis)	Examples of Descriptors per Main Type
FCS	2	1	Providing information about the situation Providing feedback Addressing feelings of guilt
TRS	3	2	Teaching client how to set rules Stimulating client to relive an event Prompting client to express emotions
ELS	4	2	Showing client how to deal with unfamiliar situations Facilitating positive experiences Teaching client how to deal with setbacks and frustrations
ICS	6	3	Instruction in cognitive restructuring Stimulating motivation Prompting client to express emotions
ILS	3	2	Training in self-regulation Analyzing the client’s social environment Providing positive reinforcement
PAS	16	9	Showing clients how to use positive reinforcement Stimulation of interaction Teaching clients to use disciplinary rules
FAS	22	8	Stimulating of interaction Giving behavioral instructions Prompting clients to practice behavioral instructions

Note. FCS = foster care support; TRS = trauma support; ELS = experiential learning support; ICS = individual child support; ILS = independent living support; PAS = parenting support; FAS = family support.

After analyzing the contents, the number of different interventions was much lower than the total at the start of the analysis, especially within family support and parenting support (Table 1). The results reveal that even interventions in the same care organization had similar content, as did interventions used by different types of care organizations (Figure 2). The three interventions by the MHC-B classified under parenting support could all be considered similar in content, while the interventions by the MHC-A classified under parenting support had to be considered for the greater part as different types of parenting intervention. Interventions offered by the CYC organization could for the greater part be considered similar concerning content within each of the seven main types of support. Ultimately, the original 56 interventions by the four care organizations participating in the C4Youth study could be reduced to 27 distinct interventions.

Figure 2.

Number of distinct interventions within the seven main types of support.

Discussion

The aim of this study was to determine the similarities and differences in content of care, that is to classify the care offered to these children and adolescents within and across care organizations, thereby using a specific assessment procedure. The descriptors that were used enabled us to compare the content of the interventions after classifying them into seven main types of support. As a result, a sample of 56 interventions—which were poorly defined—could be reduced to 27 distinct interventions across the four care organizations participating in the study.

The interventions used by the different care organizations showed a high degree of similarity concerning the content of care, suggesting that the number of interventions could be considerably reduced (>50% reduction), especially in the parenting support and family support categories, resulting in a substantial decrease in the number of distinct interventions overall. As Loeffen et al. (2004) and Veerman, Janssens, and Delicat (2005) have proven, the number of interventions offered within CYC has increased enormously in recent years, especially with respect to family and parenting interventions. The results of our study demonstrate that despite the various labels used to designate these parenting and family interventions, the content was similar in a substantial number of cases. One explanation for the use of multiple labels to describe the same intervention might be that the manuals used by professionals working in different departments within the same care organization were not the same. It may also be possible that these intervention manuals were compiled without consultation across departments.

However, it was not only within, but even more so between care organizations that a lack of standardization could be found (Lloyd-Evans et al., 2007). This explains why it is often difficult to compare these interventions (Abraham & Michie, 2008; DeJong et al., 2004; Michie et al., 2011) and to link the care received to client characteristics and outcomes across organizations (Ezell et al., 2011). Not being able to research these kinds of associations is a serious threat for collecting scientific knowledge that serves the quality enhancement of our care systems (Michie et al., 2011).

Our study also showed that in the participating CYC organization in particular, the number of distinct interventions could be readily reduced. This might be the result of a less systematic approach to interventions in comparison to professionals working in MHC organizations. Especially within the main types parenting support and family support, a number of CYC interventions were assessed as belonging to the same type of support. In contrast, the interventions by the MHC organizations were more likely to be clearly distinct. This might be due to the relatively strong impact of the “evidence-based practice revolution” in the MHC field (Van der Linden & De Graaf, 2010; Hibbs, 2001) or to the demands of insurance companies, especially in that part of the care sector.

Using descriptors of care components or activities to characterize the content of interventions is relatively new in the fields of PHC, CYC, as well as MHC. Abraham and Michie (2008) and Michie, Hyder, Walia, and West (2011) used a set of labels concerning “behavior change techniques” to characterize interventions that were aimed at physical activity, healthy eating, or smoking cessation. The method that we used in our study was somewhat similar but focused on characterizing the content of interventions offered to children in relation to behavioral or emotional problems and the child or youth’s relationship with their families. Although we did not assess inter-rater reliability in terms of total agreement on the scores of a descriptor, we analyzed the amount of cases in which the differences between the scores of both raters on a descriptor was more than 2 points on the 7-point Likert-type scale. It appeared that the number of such differences was very small (3.4%), indicating that the professionals involved in our study were highly in agreement concerning the descriptors that mostly characterize an intervention.

The results of our study suggest that the method is able to distinguish similarities and differences between the content of interventions offered by the four care organizations participating in the C4Youth study. When the applicability of descriptors of two or more interventions within a main type of support varied to a large extent, the interventions were found to be distinct types. On the other hand, when the interventions within a main type of support displayed a high amount of similarities concerning the applicability of descriptors, they were merged to the same distinct type. This indicates that for categorizing interventions it is not only important to focus on those descriptors that characterize interventions the best but also to look at descriptors that do not very accurately display the content of an intervention.

Strengths and Limitations

One strength of this study is that the descriptors used to classify the content of the interventions were defined independently of the organization-specific labels used by the four care organizations themselves. This provided a standard which allowed greater insight into the content of the care offered. Another strength is that we were able to compare the content of the interventions used by different types of care organizations in the field of PHC, CYC, and MHC. In addition, the interventions were characterized by two professionals, decreasing the likelihood of bias during the assessment of the interventions.

One limitation of the study may be that despite the use of the EYI databank’s thesaurus, we may have overlooked some important descriptors of the care offered—only the 20 most frequently used descriptors were selected. Nevertheless, 20 is a relatively large number of descriptors to characterize an intervention. Moreover, the scores of the descriptors by the professionals were generally rather high, suggesting they were good representations of the activities and operations carried out as part of an intervention.

Implications

The outcomes of our study have implications for future research and practice concerning the care offered to children and youth with behavioral and emotional problems. First, the descriptors that were used to classify the set of poorly defined interventions could also be used to classify the set of interventions which were considered well defined.

Second, the interventions offered by the four care organizations could be reduced to a set of distinct interventions. This suggests that care organizations should describe their interventions more clearly and structure them accordingly. In this way, a more transparent and systematic overview of the distinct interventions used by each care organization can be established, which could encourage better communication not only between professionals within care organizations but also across them, and may actually support clients and their caregivers in choosing the appropriate type of care (Jager, Reijneveld, Metselaar, Knorth & De Winter, submitted). However, there might be reasons why a care organization is not so much interested in the most transparent picture of its care and treatment offered. First, a more transparent, “profiled” overview of the types of care might exclude clients who do not fit with the profile of services provided. Second, an assessment of intervention contents could also display that within a care organization the greater part of the differentially labeled interventions in fact could be considered as similar, implying a less correct picture of services to the outside world. In these situations, organizations are vulnerable and might resist a transparent assessment of the care offered.

Third, by using the TOCFY to classify the most salient aspects of the care offered to children with behavioral and emotional problems, we can investigate the connection between the problem behavior initially presented by these young clients, the care they receive, and the outcomes that become apparent after leaving care (Miller & Row, 2009; Ten Brink, Veerman, De Kemp, & Berger, 2004). The method used to develop TOCFY is also applicable in other care settings, countries or cultural contexts. Categories and subcategories as formulated in TOCFY are universal, except for the “content” domain. The content domain, which is the most exhaustive domain, should be fine-tuned depending on the specific organization or setting in which the instrument is about to be used (Evenboer, Huyghen, Tuinstra, Knorth, & Reijneveld, 2012b). Although our findings show that the TOCFY instrument is promising, we recommend testing it in other areas or contexts as well.

Conclusion

The descriptors used in this study allowed the classification of the content of care offered to children with behavioral or emotional problems and their families within and across care organizations. On this basis, we were able to distinguish similarities and differences in the content of interventions and create a systematic overview of distinct interventions.

Footnotes

Appendix

Table A1.

Descriptors Defined For The Seven Main Types Of Support

Providing information about the situation (FCS, ELS, ICS, and FAS)

Addressing feelings of guilt (FCS)

Stimulating empathic behavior (FCS)

Prompting client to take responsibility (FCS and ELS)

Stimulating mutual acceptance (FCS)

Stimulating client to perform new behavior (FCS)

Stimulating affective behavior (FCS)

Providing feedback (FCS, TRS, ELS, ICS, PAS, and FAS)

Positive rephrasing (FCS)

Positioning differences and similarities of clients (FCS)

Giving recognition (FCS)

Discussing problematic behavior (FCS)

Showing client how to use positive reinforcement (FCS and PAS)

Providing structure and space (FCS)

Teaching client discipline (FCS and FAS)

Giving homework and provide an evaluation (FCS, TRS, ICS, and PAS)

Stimulating contact with the biological parents (FCS)

Teaching client how to set rules (TRS, PAS, and FAS)

Stimulating client to relive an event (TRS)

Prompting client to express emotions (TRS, ICS, ELS, and PAS)

Prompting client to regulate emotions (TRS, ICS, ELS, and PAS)

Teaching relaxation exercises (TRS)

Teaching touching exercises (TRS)

Prompting client how to recognize sexual exceeding behavior (TRS)

Prompting client how to rebuilt confidence (TRS)

Prompting client how to rebuilt social relationships (TRS)

Giving sexual education (TRS)

Practicing role stereotyping with client (TRS, ICS, and PAS)

Organizing meetings for fellow sufferers (TRS)

Prompting client to adjust expectations (TRS)

Using a rewarding system for the client (TRS and FAS)

Prompting client to maintain (telephonic) contact with parent(s) (TRS)

Showing client how to deal with unfamiliar situations (ELS)

Facilitating positive experiences (ELS, TRS, and FAS)

Teaching client how to deal with setbacks and frustrations (ELS and ICS)

Prompting client how to make choices (ELS)

Prompting client how to reflect on their own behavior (ELS, ICS, ILS, and PAS)

Building a relationship of trust with the client (ELS)

Prompting client how to work together in a group (ELS)

Providing the client with protection (ELS)

Providing a plan for school and work (ELS and ILS)

Prompting client to learn communicative skills (ELS)

Providing client with instructions and directions (ELS)

Using nonverbal communication (ELS)

Prompting client to increase self-reliance (ELS, ICS, and ILS)

Instruction in cognitive restructuring (ICS and FAS)

Stimulating motivation (ICS, TRS, ELS, ILS, PAS, and FAS)

Prompting client to make contact with official authorities (ILS)

Accomplish agreement with client on substance abuse (ILS)

Prompting client to more often explicitly name positive experiences (ILS)

Prompting client to arrange their own housing (ILS)

Having solution-oriented conversations (ILS)

Supporting client by using tutors (ILS)

Prompting client to set up a “take off” plan (ILS)

Prompting client to actively search for leisure activities (ILS)

Teaching client how to plan activities and make choices (ILS)

Prompting client how to use a rewarding system (PAS)

Stimulating parental involvement (PAS and FAS)

Prompting client to monitor the child’s behavior (PAS)

Training problem-solving ability of the client (PAS)

Prompting client to ignore undesirable behavior (PAS and FAS)

Prompting client to punish undesirable behavior (PAS and FAS)

Prompting client to appeal to the social environment (FAS)

Prompting client to give feedback (FAS)

Prompting client to negotiate (FAS)

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study received funding from the Netherlands Organization for Health Research and Development (ZonMw), the Province of Groningen, the University Medical Center Groningen, the University of Groningen, the health insurance company Menzis, and the youth care providers Accare and Elker.

References

Abraham

Michie

(2008). A taxonomy of behaviour change techniques used in interventions. Health Psychology, 27, 379–387.

Ballinger

Asburn

Low

Roderick

(1999). Unpacking the black-box of therapy. A pilot study to describe occupational therapy and physiotherapy interventions for people with stroke. Clinical Rehabilitation, 13, 301–309.

Cjaza

S. J.

Schulz

Lee

C. C.

Belle

S. H.

(2003). A methodology for describing and decomposing complex psychosocial and behavioral interventions. Psychology and Aging, 18, 385–395.

Chorpita

B. F.

Daleiden

E. L.

(2009). Mapping evidence-based treatments for children and adolescents: Application of the Distillation and Matching Model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology, 77, 566–579.

DeJong

Horn

S. D.

Gassaway

J. A.

Slavin

M. D.

Dijkers

M. P.

(2004). Towards a taxonomy of rehabilitation interventions: Using an inductive approach to examine the “black box” of rehabilitation. Archives of Physical Medical Rehabilitation, 85, 678–686.

Evenboer

K. E.

Huyghen

A. M. N.

Tuinstra

Reijneveld

S. A.

Knorth

E. J.

(2012). Taxonomic systems in the field of health care, family care, and child and youth care: A systematic overview of the literature. Children and Youth Services Review, 34, 2304–2310.

Evenboer

K. E.

Huyghen

A. M. N.

Tuinstra

Knorth

E. J.

Reijneveld

S. A.

(2012). A taxonomy of care for youth: Results of an empirical development procedure. Research on Social Work Practice, 22(6), 637–646.

Evenboer

K. E.

Huyghen

A. M. N.

Tuinstra

Knorth

E. J.

Reijneveld

S. A.

(2014). Looking into care: The reliability and feasibility of a Taxonomy of Care for Youth. Accepted for publication in the Journal of Social Work.

Ezell

Spath

Zeira

Canali

Fernandez

Thoburn

Vecchiato

(2011). An international classification system for child welfare programs. Children and Youth Services Review, 33, 1847–1854.

10.

Fein

(2002). The black box. Accounting for program inputs when assessing outcomes. In Maluccio

A. N.

Canali

Vecchiato

(Eds.), Assessing outcomes in child and family services. New York, NY: Walter de Gruyter.

11.

Fonagy

Target

Cottrell

Phillips

Kurtz

(2002). What works for whom? A critical review of treatments for children and adolescents. New York, NY: The Guilford Press.

12.

Garland

A. F.

Brookman-Frazee

Hurlburt

M. S.

Accurso

E. C.

Zoffness

R. J.

Haine-Schlagel

Ganger

(2010). Mental health care for children with disruptive behavior problems: A view inside therapists’ offices. Psychiatric Services, 61, 788–795.

13.

Harden

B. J.

Klein

(2011). Infants/toddlers in child welfare: What have we learned and where do we go from here? Children and Youth Services Review, 33, 1464–1468.

14.

Hibbs

E. D.

(2001). Evaluating empirically based psychotherapy research for children and adolescents. European Child and Adolescent Psychiatry, 10, 3–11.

15.

Jager

Reijneveld

S. A.

Metselaar

Knorth

E. J.

De Winter

A. F.

(2014), Discrepancies between adolescents’ attributed relevance and experiences regarding communication are associated with poorer client participation and learning processes in psychosocial care. Patient Education and Counseling. doi:10.1016/j.pec.2014.08.020

16.

Kazdin

A. E.

(2000). Psychotherapy for children and adolescents. Directions for research and practice. New York, NY: Oxford University Press.

17.

Knorth

E. J.

Reijneveld

S. A.

Van Eijk

L. M.

Noordik

F. W.

Tuinstra

(2011). Sticking closely to the social–emotional development of three thousand children inside and outside care. Kind en Adolescent Praktijk, 10, 192–194 (in Dutch).

18.

Lee

B. R.

Barth

R. P.

(2011). Defining group care programs: An index of reporting standards. Child and Youth Care Forum, 40, 253–266.

19.

Libby

A. M.

Saranga Coen

Price

D. A.

Silverman

Orton

H. D.

(2005). Inside the black box: What constitutes a day in a residential treatment centre? International Journal of Social Welfare, 14, 176–183.

20.

Lloyd-Evans

Johnson

Slade

(2007). Assessing the content of mental health services: A review of measures. Social Psychiatry and Psychiatric Epidemiology, 42, 673–682.

21.

Loeffen

Ooms

Wijgergangs

(2004). An organizational model is a must for effective care of children and youth. Nederlands Tijdschrift voor Jeugdzorg, 7, 102–113 ( in Dutch ).

22.

Marsh

J. C.

Angell

Andrews

C. M.

Curry

(2012). Client–provider relationship and treatment outcome: A systematic review of substance abuse, child welfare, and mental health services research. Journal of the Society for Social Work and Research, 12, 233–267.

23.

Maschi

Hatcher

S. S.

Schwalbe

C. S.

Rosato

N. S.

(2008). Mapping the social service pathways of youth to and through the juvenile justice system: A comprehensive review. Children and Youth Services Review, 30, 1376–1385.

24.

Michie

Hyder

Walia

West

(2011). Development of a taxonomy of behaviour change techniques used in individual behaviour support for smoking cessation. Addictive Behaviors, 36, 315–319.

25.

Miller

Rowe

(2009). Cracking the black box: What makes an arts intervention program work? Best Practices in Mental Health: An International Journal, 5, 52–64.

26.

Netherlands Youth Institute. (2013). Dutch youth institute. Retrieved August 14, 2013, from https://www.nji.nl

27.

Ten Brink

L. T.

Veerman

J. W.

De Kemp

R. A. T.

Berger

M. A.

(2004). Implemented as intended? Recording family worker activities in a Families First program. Child Welfare, 83, 197–215.

28.

Van der Linden

De Graaf

(2010). State of the art. Acquaintance with and use of evidence-based interventions in child and youth mental health care services. Utrecht, The Netherlands: Trimbos Institute (in Dutch).

29.

Van Yperen

T. A.

Van Rest

Vermunt

(1999). Programmes in child and youth care: The definition of core concepts. Utrecht, The Netherlands: NIZW Publishers (in Dutch).

30.

Veerman

J. W.

Janssens

J. M. A. M.

Delicat

J. W.

(2005). Effectiveness of intensive home-based child and family care: A meta-analysis. Pedagogiek, 25, 176–196 (in Dutch).

31.

Veerman

J. W.

Van Yperen

(2008). A perspective on effectiveness. Handbook on practice-driven outcome research in care for youth. Delft, The Netherlands: Eburon (in Dutch).

32.

Wade

Biehal

Farrelly

Sinclair

(2011). Caring for abused and neglected children. London, England: Jessica Kingsley Publishers.

33.

Yohalem

Wilson-Ahlstrom

(2010). Inside the black box: Assessing and improving quality in youth programs. American Journal of Community Psychology, 45, 350–357.

34.

Zwikker

Van Dale

Kuunders

(2009). Committee accreditation of interventions: Method and procedure. Utrecht, The Netherlands: Netherlands Youth Institute/National Institute for Public Health and Environment (in Dutch).