Going Global

Abstract

The spread of evidence-based practice throughout the world has resulted in the wide adoption of empirically supported interventions (ESIs) and a growing number of controlled trials of imported and culturally adapted ESIs. This article is informed by outcome research on family-based interventions including programs listed in the American Blueprints Model and Promising Programs. Evidence from these controlled trials is mixed and, because it is comprised of both successful and unsuccessful replications of ESIs, it provides clues for the translation of promising programs in the future. At least four explanations appear plausible for the mixed results in replication trials. One has to do with methodological differences across trials. A second deals with ambiguities in the cultural adaptation process. A third explanation is that ESIs in failed replications have not been adequately implemented. A fourth source of variation derives from unanticipated contextual influences that might affect the effects of ESIs when transported to other cultures and countries. This article describes a model that allows for the differential examination of adaptations of interventions in new cultural contexts.

Keywords

evidence-based interventions transportability adaptation globalization evidence-based practice

The global spread of evidence-based practice relies, in part, on the degree to which empirically supported interventions (ESIs) have the capacity to produce desirable outcomes and equivalently large effects in cultures and contexts different from those in which they were originally developed. Although the ESI knowledge base is rapidly growing, the number of ESIs is still relatively small in comparison to the global need. Thus, the import of ESIs from other cultural contexts appears to be warranted (cf. Ferrer-Wreder, Adamson, Kumpfer, & Eichas, 2012). For instance, of the 134 Swedish outcome studies of behavioral and social interventions conducted between 1995 and June 2010, 41% involved the evaluation of ESIs originating from other countries (Swedish National Board of Health and Welfare, 2011). In many countries, imported family-based ESIs with wide ranging intervention-related effects in their validation studies are being tested in randomized controlled trials (RCTs) that attend to the cultural translation of program content and measures. Findings from these studies suggest that the generalization of ESIs to new contexts, including settings and populations, may be more challenging than previously assumed.

This article was written as a description of and reflection on empirical studies from the Swedish and wider European research literature on imported interventions. The approach used is similar to a case-study approach in which selected nonrepresentative exemplars provide the context for a commentary on imported intervention research areas with room for improvement. The focus is primarily on replication studies of family-based ESIs imported to Sweden and within Europe. We make an argument for the need for culturally nuanced research strategies in adapting and testing imported ESIs. This need is motivated by effectiveness trials where expected outcomes were not observed and where research methodologies did not make it possible to disentangle alternative reasons to account for failures.

The focus of this article is timely in light of recent scholarly debates on the need to bridge the gap between science and practice as well as the pressing need for replication research as a means to advance prevention science (e.g., Emshoff, 2008; Valentine et al., 2011). The article concludes by describing a preliminary model for developing culturally congruent -content in ESIs and, through successive experimentation, developing empirical support for imported culturally adapted interventions, the Planned Intervention Adaptation (PIA) protocol. It is important to note that, although our commentary is empirically informed, it is not based on a systematic review of the research literature. Thus, the examples given are not representative of adaptation and translation efforts globally.

Mixed Findings From International Replications

The findings from international replications of family-based ESIs are mixed. To be sure, well-designed replication studies suggest that many ESIs confer benefits comparable to those observed in the studies that warranted their initial designation as promising or effective. From the U.S. Blueprints Model and Promising Programs (BMPP; http://www.colorado.edu/cspv/blueprints), for example, these include replications in RCTs of functional family therapy in Sweden (Hansson, Cederblad, & Höök, 2000), multidimensional treatment foster care in Sweden (Hansson & Olsson, 2012; Kyhle Westermark, Hansson, & Olsson, 2011) as well as the incredible years in Canada (Taylor, Schmidt, Pepler, & Hodgins, 1998), England (Gardner, Burton, & Klimes, 2006), Norway (Fossum, Mørch, Handegård, Drugli, & Larsson, 2009), and Sweden (Axberg & Broberg, 2012).

However, other international trials of BMPP ESIs have not replicated the benefits observed in the original trials or they have produced mixed results. One example is a test of U.S.-developed multisystemic therapy (MST). MST is an intensive family- and community-based treatment for adolescents with serious adjustment difficulties that often include criminal behavior, violence, substance abuse, and serious emotional disturbances. MST has an extensive evidence base in the United States (Schoenwald, Heiblum, Saldana, & Henggeler, 2008) and has been imported to several countries and has undergone controlled evaluations in five of them. Although MST intervention benefits were replicated versus a control group in Norway (Ogden & Hagen, 2006), the Netherlands (Deković, Asscher, Manders, Prins, & van der Laan, 2012), and the United Kingdom (Butler, Baruch, Hickey, & Fonagy, 2011), the same benefits relative to a control group were not demonstrated in Sweden (Andrée Löfholm, Olsson, Sundell, & Hansson, 2009) or Canada (Leschied & Cunningham, 2002).

In the same vein, the well-known Australian parenting training program Triple-P has been successfully exported to Germany (Hahlweg, Heinrichs, Kuschel, & Feldmann, 2007), Hong Kong (Leung, Sanders, Leung, Mak, & Lau, 2003), Japan (Matsumoto, Sofronoff, & Sanders, 2010), Switzerland (Bodenmann, Cina, Ledermann, & Sanders, 2008), and the United States (Prinz, Sanders, Shapiro, Whitaker, & Lutzke, 2009). However, findings from controlled trials in Switzerland (Eisner, Nagin, Ribeaud, & Malti, 2012; Malti, Ribeaud, & Eisner, 2011), and Canada (McConnell, Breitkreuz, & Savage, 2011) failed to replicate the positive effects.

Similar results are observed in an international replication of the Strengthening Families Program for parents and youth 10-14 (SFP) for school-based populations. SFP is an American developed family-focused universal drug prevention program. The results of a RCT conducted in Sweden with a universal version of SFP 10-14 yielded no statistically significant differences between youth taking part in SFP relative to adolescents in a treatment as usual (TAU) control group (Skärstrand, 2010). The example of the SFP trial in Sweden is returned to in greater detail later in this article.

The difficulties encountered in transporting ESIs within Europe can be exemplified by the Örebro Prevention Program (ÖPP) trial in the Netherlands. ÖPP is a Swedish-developed intervention aimed at encouraging parental rule-setting concerning alcohol consumption by adolescents (Koutakis, Stattin, & Kerr, 2008). Delivered to parents through schools, ÖPP is designed to provide information on the negative consequences of underage drinking. A Swedish efficacy study showed that ÖPP was associated with increased parental disapproval of underage drinking and, moreover, reduced both youth drunkenness and delinquency (Koutakis et al., 2008). When ÖPP was exported to the Netherlands and evaluated in a RCT, ÖPP by itself did not decrease the onset of drinking (but ÖPP in combination with a student intervention was effective; Koning et al., 2009). In speculating on these findings, Koning and colleagues observed that parents may be less influential in delaying the onset of alcohol use in countries such as the Netherlands, where lower legal drinking ages and more lenient alcohol policies prevail. To complicate matters, a recent independent cluster-randomized effectiveness study of ÖPP in Sweden (Bodin & Strandberg, 2011) did not replicate the original Swedish findings. One possible explanation for the null results is according to Bodin and Strandberg that, since the first ÖPP study was conducted, Swedish media campaigns aiming to increase restrictive parenting practices about alcohol have changed popular views regarding underage alcohol consumption, diminishing the potential for ÖPP to demonstrate differences between the experimental and TAU control groups.

Explanations of International Replication Failures

At least four explanations appear plausible for how to account for the contradictory results from studies of imported ESIs (cf. Hansen, 2011; Valentine et al., 2011). One has to do with methodological differences of the outcome trials. A second deals with ambiguities in the cultural adaptation process. A third potential explanation is that ESIs in failed replications have not been adequately implemented. A fourth source of variation may lie in unobserved contextual influences that have potential to moderate the effects of ESIs when imported to new settings.

This classification of explanations influencing successful translation is somewhat arbitrary because there is an overlap between the concepts. For instance, adapting an ESI to a new context can affect the treatment fidelity as originally conceptualized, because the intervention itself may have been altered. Our reason to differentiate the two is that the adaptation often precedes the actual implementation, thus leaving one or the other to account for failures in outcomes. In the same vein, the choice of TAU as a control is part of research methodology, but the selection of a TAU could be conditioned by cultural contexts, that is, it should be informed by an understanding of pubic policies, media, social welfare practices, and available routine services.

Research Methodology

Several aspects of research methodology have been related to variation in effect sizes observed in trials of social and health interventions. First, efficacy trials in which program developers supervise the provision of experimental services often produce larger effect sizes than effectiveness trials that take place in the context of routine services where program developers are less involved (e.g., Curtis, Ronan, & Borduin, 2004; Emshoff, 2008; Petrosino & Soydan, 2005). In addition, effect sizes tend to be larger in studies with passive control conditions, such as a wait-list control or placebo, compared to active control conditions such as TAU (Baldwin, Christian, Berkeljon, & Shadish, 2012; Grissom, 1996; Magill & Ray, 2009; Shadish, 2011). Furthermore, strategies for handling attrition are related to the size of observed effects. Inclusion only of participants who follow through with an intervention or treatment (i.e., treatment of the treated) normally produces greater effect sizes relative to intention to treat analysis, where dropouts and participants who fail to comply with treatment are included also in parameter estimation (Wright & Sims, 2003). In addition, effect sizes are influenced by the heterogeneity of samples (Grissom & Kim, 2012), the fit of outcome measures to the theory of change in an intervention (Donaldson, 2007; Frechtling, 2007), and the length of the follow-up time (e.g., Andrée Löfholm et al., 2009; Magill & Ray, 2009).

A review of 13 MST RCTs exemplifies the problems of comparing outcome studies using different control conditions (Andrée Löfholm, Brännström, Olsson, & Hansson, 2013). Of these RCTs, 10 are from the United States and 1 each was from Norway, Sweden, and Canada. Although the investigators across these studies sought to design rigorous evaluations, no two studies used the same research methodology and the TAU conditions included a wide variety of treatment alternatives. For instance, the control conditions included individual therapy/counseling, cognitive behavioral therapy, TAU within the juvenile justice system, TAU within child and adolescent mental health care, and TAU within the child welfare system. Within TAU conditions, the most common alternative was that the comparison group received a mixture of various interventions that could include individual or group treatment with different emphases and theoretical bases, placements in out-of-home care as well as placement prevention (i.e., community-based) interventions. There was little information about what interventions the individuals in the TAU condition actually received, and while the dose, duration, and services breakdown often was described for the experimental group, no such information was available regarding the TAU condition. Not surprisingly, the effects observed in these trials varied greatly and a meta-analysis indicated no significant overall difference between TAU and MST (Andrée Löfholm et al., 2013). Estimated treatment effects on recidivism suggested that the various TAU conditions seemed to contain a greater variation in change producing mechanisms than experimental conditions, supporting the hypothesis that content of TAU conditions could affect the outcomes. This result is consistent with that of a Cochrane review (Littell, Popa, & Forsythe, 2005), including 8 of the 13 RCTs. However, other meta-analysis (e.g., Aos et al., 2011; Curtis et al., 2004) identify significant overall effects of MST. One possible reason for the different conclusions is the inclusion and exclusion of studies. Applied to the Swedish trial of MST, the lack of program effects (Andrée Löfholm et al., 2009) might have been because the Swedish TAU was more effective than its counterpart in United States. For example, a comparison of Child Behavior Checklist (CBCL) change scores from the Swedish, Norwegian, and two American MST evaluations showed that the average decrease in CBCL T scores for the Swedish MST group was similar to that of the Norwegian MST study and larger than or equal to that of American studies (Sundell et al., 2008). This cross-study comparison of change scores indicated that the Swedish MST treatment was effective. The CBCL change scores in the Swedish and Norwegian TAU groups, however, decreased considerably more than the scores in the two American TAU groups. This indicates that the Swedish TAU, and to a somewhat lesser extent also the Norwegian TAU, were more potent relative to their counterparts in the United States. Thus, the mixed findings across international replications of MST may in part be due to variation in type of control group across the trials. If this is the case, it points to the need for more culturally nuanced methodology in outcome research, particularly replications of imported ESIs.

Adaptation

When an ESI is imported to a new culture, program materials must often be translated and the content of program activities is often screened for cultural relevance. Typically, some type of adaptation or modification is needed. Unfortunately, there is no consensus about the criteria for determining when cultural adaptation is needed (Cardemil, 2010; Ferrer-Wreder, Sundell, & Mansoory, 2012). Although adaptation might be necessary in order to implement an ESI (e.g., Allen, Linnan, & Emmons, 2012; Damschroder et al., 2009), its value is contrasted with the need for high treatment fidelity (e.g., Bernal, Jiménez-Chafey, & Rodríguez, 2009; Castro, Barrera, & Holleran Steiker, 2010; Durlak & DuPre, 2008). One solution to these conflicting views is to restrict adaptation to what Resnicow, Soler, Braithwaite, Ahluwalia, and Butler (2000) referred to as surface structure and stress fidelity to the so-called deep structure. An intervention’s deep structure is similar to what other scholars have called an intervention’s theory of change, program theory, or internal logic (McKleroy et al., 2006). The deep structure refers to the causal model that specifies the empirical and theoretical relations between intervention activities, mediators of change (e.g., changes in skills, knowledge, or attitudes), and ultimate outcomes (e.g., changes in behavior or health status). Examples of deep structure components in empirically supported family-skills training interventions include program activities that strengthen parental skills in communication and child supervision or monitoring, which in turn are posited to yield adjustment benefits for adolescents in those altered families. The knowledge base on intervention deep structure is not extensive (e.g., United Nations Office on Drugs and Crime [UNODC], 2009), but will hopefully expand through descriptive research literature on determinants and outcomes as well as dismantling studies and/or mediation-outcome analyses of the intervention’s active ingredients.

The surface structure of an intervention consists of aspects of the intervention that improve the face validity of the intervention with participants (Resnicow et al., 2000). Participants have to understand what is being communicated to them in order to benefit from it. Surface structure also includes how the intervention resonates with participants’ lives and how the intervention appeals to people, through intervention images, intervention activities, behavioral preferences around the target behaviors, and appropriate channels of communication (Resnicow & Braithwaite, 2001; Resnicow, Diorio, & Davis, 2009). Like high-quality implementation, adequate surface structure is also necessary but it does not in itself lead to intervention success. Across different intervention cultural adaptation models, there seems to be less disagreement about the need for surface structure changes (e.g., language, images, and activities not key to intervention success) when implementing an ESI with a new cultural group dissimilar from the validation samples (Ferrer-Wreder, Adamson et al., 2012; UNODC, 2009). Clearly, an intervention must be presented in a culturally grounded metric.

The need for and extent of adaptation varies (Allen et al., 2012). In some imported ESI trials, translation at a semantic level may be particularly difficult because the concepts within intervention content are not readily interpretable and words or characters may scarcely exist for them. In these circumstances, translation alone can require a sequential process of translation, back translation, retranslation, and review by culture and program experts. Perhaps, the most enigmatic issue in the cultural adaptation process involves making the determination of when an adaptation is necessary, including when an adaptation may compromise the core or deep elements of an intervention.

A case in point, the Swedish parent management training program entitled Comet (Kling, Forster, Sundell, & Melin, 2010) was based on behavioral parent-training components developed in the United States by Barkley (1997), Webster-Stratton (1996), and Bloomquist and Schnell (2002). From a pilot test with Swedish parents, the developer of Comet decided to reduce program content on time-out (normally an essential part of parent management training), because it was perceived to be offensive to many Swedish parents. Two succeeding RCTs comparing COMET to a waiting list control group showed that COMET produced the expected intervention benefits relative to the control group (Enebrink, Högström, Forster, & Ghaderi, 2012; Kling et al., 2010). In the case of Comet, the omission of what may represent a core feature of parenting training in the originating country appeared to have been appropriate, although a different design would be needed to make a causal inference about the effect of dropping time-out.

Making these kinds of cultural adaptations while preserving other features of ESIs is emerging as a crucial challenge in the globalization of evidence-base practice. A transported intervention with a thorough description of the adaptation process is the Swedish trial of the school-based drug prevention program SFP 10-14 (Skärstrand, Larsson, & Andreasson, 2008). This study involved the adaptation of the SFP 10-14 (March 2001 edition) for Swedish conditions. With support from the program’s first author, Dr. Virginia Molgaard, changes included translation of program content from English to Swedish, the creation of new film resources using Swedish actors, alterations in the program delivery format, and the modification of some program activities. The process of adaptation of SFP 10-14 in the Swedish trial involved the formation of two reference groups, one consisting of researchers, and one representing teachers to investigate the interest of agencies in an alcohol prevention family program and to identify any possible barriers to participation. All material including manuals and manuscripts for SFP 10-14 videos were translated into Swedish. The quality of the Swedish videos was improved by using a professional film team and trained actors. The videos were shot in real-life settings, not in a studio. Altogether, 11 new videos plus one demonstration video were made.

Prior to the implementation and evaluation, two sixth-year classes in two schools in Stockholm were chosen to participate in a pilot study. On a regular basis throughout the pilot study, SFP 10-14 group leaders, teachers, and the research team met to discuss program content and implementation issues. In addition, the group leaders and the teachers completed checklists after each session, which included questions about the activities from each session such as if there had been enough time to do all the activities; if there had been enough preparation for the session; and how well the parents and children responded to the activities. Following the completion of the first seven sessions, a focus group was conducted with attending parents.

One major change in the Swedish version of SFP 10-14 concerned the program format. From discussions with the two reference groups, it became clear that the most appropriate setting for a family program in Sweden would be public schools, using the classroom teacher as one of the leaders for youth sessions. Consequently, the youth sessions were held during the daytime on ordinary school days. Instead of having a family session at every meeting as required in the American SFP 10-14 protocol, the choice was made to have a total of two family sessions: one in the seventh session to finish part one of the program and the second in the last session of the second part of the program.

Another difference between the original SFP 10-14 and the Swedish version was the size of the groups. The former was designed for 8–13 families and 3 facilitators, 2 for youth sessions, and 1 for parent sessions. The Swedish version was provided to a classroom of 25–30 students and all parents who were interested in participating. Two or three facilitators implemented the youth sessions. Facilitators also conducted the two parent sessions. In addition, more emphasis on alcohol and other drugs was added to part two and an additional session focusing on explicit drug and alcohol use prevention was developed. This increased focus was interactive in nature with video vignettes on young people and alcohol (e.g., young people asking their parents to buy them alcohol), followed by peer group discussions and homework assignments. The student responses were used as a foundation for discussions among the parents. An example of an activity change in the Swedish adaptation of SFP 10-14 involved the closing circle. In the U.S.-based SFP 10-14, the parents formed a closing circle and recited a creed at the end of each session. This was considered to be culturally inappropriate in a Swedish context and was omitted.

In a cluster randomized trial with multiple assessments, the Swedish SFP showed no intervention-related benefits compared to a TAU control group (Skärstrand, 2010). This trial raises several potential problems associated with design and analysis standards often used to test imported interventions in effectiveness trials. For example, the adaptation of SFP to a Swedish context conformed to many of the recommendations for the international dissemination of SFP published by its program developer (Kumpfer, Pinyuchon, Teixeira de Melo, & Whiteside, 2008), the researchers in the Swedish trial argued that they intended to avoid major changes to SFP’s deep structure (Skärstrand et al., 2008), and changes made in program format (i.e., reduction of the number of parent sessions) were approved by one of the program developers (Dr. Molgaard). The trial showed no intervention-related benefits, and the design used in this case did not allow for a determination of whether intervention effectiveness was compromised by the adaptations (highlighting the difficulty of determining whether an adaptation is of a surface or deep structure) or, alternatively, whether the adaptations were insufficient and more changes might have been necessary in order to yield benefits among Swedish families. Research processes that provide for tests of major adaptations, such as the substantial abbreviation of parent involvement in the SFP trial, appear warranted in replication research on ESIs.

Implementation

A third consideration in explaining replication failures has to do with implementation. Implementation is a multidimensional construct, consisting of “fidelity, dosage, quality, participants responsiveness, program differentiation, monitoring of control conditions, program reach, and adaptation” (Berkel, Mauricio, Schoenfelder, & Sandler, 2011, p. 23). Fidelity involves adherence to the program curriculum, competence in using the intervention, and differentiation from alternative services (Berkel et al., 2011). Fidelity is a key variable among implementation markers, and it has been found to modify intervention benefits (Durlak & DuPree, 2008). If an imported ESI is implemented with poor fidelity, an otherwise well-conducted outcome study might falsely produce findings of no effect.

In the Swedish trial of MST, treatment adherence was measured monthly by caregiver reports on the MST therapist adherence measure (TAM). The TAM is a structured interview that is designed to assess several aspects of implementation. The interviews are used to ensure that therapists are faithful to MST principles and practices. Although the delivery of MST was supervised by MST Services Inc., TAM data suggested that fidelity was more than a full standard deviation below the mean reported in the American studies (Sundell et al., 2008). This raises the possibility that MST may not have been implemented with sufficient fidelity to provide a fair test of its effectiveness. This interpretation is supported by Swedish data on the development of treatment adherence over a period of 6 years (Andrée Löfholm, Eichas & Sundell, submitted). Multilevel structural equation modeling indicated a significant time effect on aggregated team-level therapist adherence, and that the individual adherence measure predicted the outcomes at the termination of therapy. If this result is applicable to more ESIs, it has profound implications to the planning and the execution of an effectiveness trial.

Context

Cultural context is complex and can be conceptualized in a variety of ways (Hofstede, 2001; Super & Harkness, 1999; Sussman, Unger, & Palinkas, 2008). In this article, we use the term context to connote the broad scope of circumstances and characteristics that include factors that surround a particular implementation effort. Examples of contextual factors include the local and national policies about the system and the services that are given, a provider’s perception of the evidence supporting the use of an ESI, and characteristics of the individuals involved in the implementation effort. The exact effects of these cultural differences are widely acknowledged but have rarely been the focus of systematic research in translating ESIs to new cultures (Castro et al., 2010; Ferrer-Wreder et al., 2012).

Exporting ESIs across cultures and within cultures, where access to resources, the prevalence of problems, and the constellation of risk and protective factors affecting the incidence of problems vary, presents numerous challenges. One of these appears to be developing a clearer understanding of how social services are organized cross-nationally. Variations in standard social and health services may alter the interpretation of whether a newly imported ESI can be considered more effective than TAU or not. For example, in some countries intensive case management is a service for persons with severe mental illness. Typically, intensive case management provides a nurse, social worker, or other clinician as a case manager who carries a small caseload of between 10 and 20 patients. The case manager takes primary responsibility for keeping contact with patients, assessing their needs, and ensuring that needs are met. A systematic review of outcomes from 29 RCTs of intensive case management demonstrated culture-based patterns across studies (Burns et al., 2007). Meta-regression showed that intensive case management was more effective in societies where the use of hospital care by individuals with severe mental illness was high and case management was less effective in countries where hospital use for serious mental illness was already low. The interpretation was that where community services are good, hospital care was used sparingly and only when absolutely necessary (Burns et al., 2007). Under such circumstances, a case manager may find it difficult to have an impact on hospital use, if that is the outcome by which this type of intervention is judged as useful. When community services are poor, it is comparatively more common for patients to spend long periods of time in hospitals, and a case manager may find it easier to reduce hospitalization. Thus, low levels of hospital use can be seen as a proxy for good community services, which has been shown to modify the effects of case management interventions for individuals experiencing severe mental illnesses (Burns et al., 2007). In short, the services environment may produce ceiling and floor effects in outcomes measures, which have relevance for the way the effectiveness of ESIs is interpreted across cultural contexts.

In the same vein, consider the correctional context for juvenile offenders. In Sweden and Norway, youth offenders receive, almost without exception, dispositions to the child welfare system (Ginner-Hau & Smedler, 2011; Levin, 1998). The standard procedure for prosecutors and criminal courts is to refer offending youths to child social services, which typically provides in-home services and supervision (Sundell, Vinnerljung, Andrée Löfholm, & Humlesjö, 2007). As a consequence, in-home interventions such as MST, which was designed originally for juvenile offenders, might not produce the same relative improvement in case outcomes as observed in the United States, where youth offenders are often institutionalized and processed within juvenile or criminal justice systems which are differentially committed to rehabilitation (Lipsey, 1999). According to Sundell et al. (2008), there are two differences that may explain the significant MST effect in Norway and the nonsignificant findings in Sweden. The first is that the implementation of MST in Norway was guided by the Ministry of Child and Family Welfare, implemented nationally and sponsored by a research unit to support and evaluate the quality of the implementation. In contrast, the Swedish implementation was guided by local initiatives without a national supporting framework, a difference that might explain the comparably low-MST treatment fidelity scores in the Swedish study. The second difference favors the Swedish TAU. Fewer youths received residential care in the Swedish (18%) study when compared with the Norwegian (50%) study. This might have disfavored the Norwegian TAU group given that residential care is an intervention with well-known risks for iatrogenic effects. These two differences may explain the significant MST effect in Norway but not in Sweden.

Beyond important contextual variations in TAU, different sociodemographic contexts appear also to moderate the relevance and potential effect sizes of ESIs. For example, research on risk and resilience (for reviews, see, Fraser, 2004; Rutter, 1979) suggests that neighborhood poverty is related to antisocial behavior in childhood and adolescence, either directly through low social cohesion and informal social control or indirectly whereby poor parenting reduces supervision and provides models of aggressive or coercive problem-solving (e.g., Grant et al., 2003). Other contextual stressors include high-crime rate and the presence of gangs, which afford opportunities for delinquent involvement and rewards for antisocial attitudes. The success of treatment for antisocial behaviors is likely to depend on the number of these risk factors that confront youths and the degree to which these risk factors may be counterbalanced by the presence of protective factors, such as a competent prosocial adults who provide opportunities for conventional commitments (Jaffe, Caspi, Moffitt, Polo-Tomás, & Taylor, 2007). Because aggregate vulnerability is related to the net balance between individual and environmental risk and protective factors, the potential effectiveness of ESIs will be context sensitive. Variation in outcomes is likely to be partially a function of how risk and protective factors, both individual and contextual, may pattern themselves in a given cultural setting. Across cultures, the connections between risk and protective factors and outcomes of interest may differ. If variations in risk processes differ in meaningful ways, the deep structure of an imported ESI may be compromised because culturally irrelevant risk relationships are addressed or because important culturally relevant risk relationships are not addressed. Additionally, intervention science will optimally progress if replication studies are as concerned about building an evidence base for programs as it is for testing the generalizability of our understanding of the phenomena we seek to change through interventions on a global scale (Hansen, 2011). In other words, “outcomes mania” (Emshoff, 2008, p. 396) should be tempered by an equivalent passion for confirming etiological knowledge and building theory through experimentation (i.e., intervention trials).

The PIA Protocol

From attempts to replicate ESIs across and within countries, a number of models for cultural adaptation are beginning to emerge (for a review, see Ferrer-Wreder et al., 2012). Typically, these models prescribe a series of steps or decision-making guidelines for adapting, implementing, and evaluating an intervention for a new context. Model development in the cultural adaptation of interventions has outstripped systematic and controlled empirical work. Yet, progress in this field depends, at least in part, on developing, testing, and then refining ideas guidelines for how to most effectively go about conducting and testing the effects of culturally oriented intervention adaptations.

One of the few programmatically generic perspectives designed for imported ESIs, the PIA protocol is based on an elaboration of an adaptation model first developed by Resnicow and colleagues (2000; see also Ahluwalia, Harris, Catley, Okuyemi, & Mayo, 2002; Harris et al., 2001). Programmatically generic intervention adaptation models are not common in the cultural adaptation literature (Castro et al., 2010; Ferrer-Wreder et al., 2012). PIA is designed to fill this gap. PIA does not presently have a dedicated empirical evidence base and the presentation of this model in this article is not prescriptive. However, even in the absence of a direct evidence base, PIA illustrates the challenges incumbent in launching rigorous research on the cultural adaptation of imported ESIs. In addition, PIA incorporates features of other adaptation models and, in this sense, it represents a synthesis of the latest thinking (see e.g., Castro et al., 2010; Ferrer-Wreder et al., 2012; Kumpfer et al., 2008; Kumpfer, Xie, & O’Driscoll, 2012; UNODC, 2009). The PIA protocol is designed for a situation in which an ESI is to be imported to a new country or cultural context (Table 1). To promote sustainability, the protocol is structured for use collaboratively with program developers (including certified program agents or program experts), intervention scientists, and other program stakeholders (e.g., policy makers, program participants, practitioners, or end users). That is, the responsibility to carry out PIA is not considered the sole responsibility of any one stakeholder group, and collaboration among stakeholders is emphasized in PIA. Briefly summarized, PIA provides a tool for determining the need, and if warranted, the scope and nature of intervention adaptations that may improve an imported ESI. Across two phases with substeps, PIA makes use of concepts from a cultural adaptation model developed by Resnicow and colleagues (2000), who distinguished the deep from surface structural elements of interventions.

Table 1.

Planned Intervention Adaptation (PIA): A Protocol for Culturally Tailoring an Imported Empirically Supported Intervention.

Phase I	Phase II
Step 1: Select a promising empirically supported intervention (ESI) for importation. Program developers and program stakeholders in the new context decide to work together to adapt and test the selected ESI. A Formative Research Participant Pool is created. “Adaptation as Usual” (e.g., translating surveys and intervention materials) takes place	Step 6: conduct a three-arm effectiveness trial of the imported ESI with three experimental conditions: ESI_Min = A minimally adapted edition of the imported ESI—from Phase I Step 1 (language changes and minor surface structure changes only to intervention) ESI_Full = A fully adapted edition of the imported ESI—from Phase I Steps 1–5 Treatment as usual control condition
Step 2: Create three subsamples from the Formative Research Participant Pool with Subsample 1 completing back translated instruments and those already in use in the new implementation context. The instruments are then changed on the basis of the psychometric analysis that examined the cultural equivalence and appropriateness of the evaluation instrumentation	Outcome of phase II: Effectiveness trial results assist in additional ESI adaptations The effectiveness trial’s study results may point to the need for additional changes to the ESI and another effectiveness trial or point toward the need for a larger scale dissemination trial with the edition of the intervention found to be most effective (e.g., ESI_Min or ESI_Full)
Step 3: Subsample 2 completes revised instruments from Step 2. Researchers then test the generalizability of the ESI’s theory of program change (i.e., connections between theory-based mediators of intervention change and ultimate intervention outcomes) in Subsample 2 (i.e., a deep structure inquiry)
Step 4: Subsample 3 takes part in focus groups designed to examine the ESI’s materials and activities for cultural relevance and appropriateness (i.e., a surface structure inquiry)
Step 5: Deep and surface structure inquiries guide adaptations made to the imported ESI
New stakeholders and program developers make changes to the imported ESI according to the results of the formative research studies (steps 1–4). A minimally adapted and fully adapted editions of the ESI are pilot tested, which informs another round of intervention refinement

PIA Phase I—Preintervention Trial Studies

Phase I begins by conjoining program developers and program stakeholders in a collaborative agreement. The agreement should specify the implementation context and affirm commitment to carry out the PIA protocol, which involves a series of small studies. If relevant, resources to provide for the translation and back translation of study surveys and program materials should be identified. The rest of the steps in Phase I involve conducting formative studies (e.g., needs assessment and pilot studies) intended to inform adaptation activities. Phase I studies are expected to take approximately 1½ to 2 years.

The Phase I studies require a sample of participants who are similar to those individuals who will ultimately take part in the imported ESI. Adequate sample sizes should be used in order to meet the assumptions of the statistical tests to be used as well as to generate reasonably detectable effect sizes. The PIA protocol is ideally suited for an effectiveness trial that sets the stage for a larger scale dissemination trial. Thus, samples should be selected in such a fashion as to preserve the heterogeneity expected in practice, and sites should be selected in order to represent the conditions expected in routine implementation. For example, it might not be appropriate to exclude from the sample complex cases with comorbidities because such cases could be expected under routine implementation.

The sample is randomly divided into two smaller subsamples. Step 2 in Phase I focuses on the development of culturally congruent measures. One subsample is invited to complete the newly translated study surveys. The evaluation team then conducts validity and reliability analyses on survey responses and modifies the study surveys accordingly. Modifications should be based on tests of the cross-cultural structural equivalence of constructs and relationships (e.g., Byrne & van De Vijver, 2010). When the structure of constructs is found to vary, a process of scale development that is beyond the scope of this article should be initiated (for a discussion, see DeVellis, 2012; Wu, Li, & Zumbo, 2007).

Step 3 in the PIA protocol involves a preliminary inquiry into the risk and protective factors on which an imported ESI is based. In short, Step 3 involves a test of the relevance of the deep structure or foundational features of an intervention. In this step, Subsample 2 completes the revised study surveys generated from Step 2 in a cross-sectional study design. The evaluation team then estimates relations between the ESI’s targets for change and the intervention’s intended outcomes. Step 3 is not a common part of the cultural adaptation literature, although it has been recommended on several occasions (e.g., Castro et al., 2010; Kumpfer et al., 2008). It addresses the relevance of the mediating mechanisms targeted in an ESI or, alternatively, the question of whether the “active ingredients” in an ESI are likely to operate comparably in a new cultural context. Step 3 is taken in order to test the potential generalizability of an imported ESI’s deep structure in the new implementation context. If archival descriptive data sets or prior research provide information comparable to that which would be generated in Step 3, then this is an alternative way to make this preliminary test of the imported ESI’s deep structure (Kumpfer et al., 2008).

Step 4 of Phase I involves recruiting a smaller group of participants from Subsample 2 to take part in a focus group that would screen the surface structure of the imported ESI in order to determine whether intervention materials and activities are acceptable and culturally relevant. Step 4 is a common part of cultural adaptation and would be based on traditional focus group methods as well as models of these types of focus groups conducted as part of cultural adaptation studies (e.g., Skärstrand et al., 2008). Testing the cultural relevance of an intervention’s surface structure through expert panels or participant focus groups or interviews is not a novel suggestion in the intervention cultural adaptation field. However, we view this step as particularly useful when it is integrated into a systematic adaptation process dealing with both surface and deep structure aspects of an intervention.

The results from Phase I should provide evidence to guide to deep and surface structure changes to an imported ESI prior to its implementation in an effectiveness trial. In Step 5, program stakeholders in the new implementation context and the imported ESI’s developers work together to determine the nature and extent of needed changes to the imported ESI and create an empirically informed adaptation of the imported ESI. The final products of Phase I should include a minimally adapted edition of the ESI with only language and minor surface structure changes made to intervention content (program and training materials) and a fully adapted edition of the imported ESI that is empirically informed by Steps 1 through 5. A small-scale pilot test of the newly generated interventions should inform more finalized intervention revisions (minimal and fully adapted) which are then tested in Phase II. Here, it is suggested that the pilot test be a short-term trial of the actual interventions (minimal and fully adapted editions of the intervention) and data collected are process oriented and qualitative in nature (focus groups of pilot participants and interventionists). Thus, Phase I of the PIA protocol sets the stage for Phase II.

Phase II—Three-Arm Effectiveness Study

Phase II of the PIA protocol involves the use of a three-arm experimental design to test the minimally and fully adapted editions of an imported ESI versus an active or passive control condition. The participants in the intervention trial should complete relevant surveys at pre, post, and at a 6-month follow-up, with additional follow-up desirable to detect longer term intervention-related change. Intervention-related changes should be determined either through more traditional outcome analyses or with structural equation modeling of mediation-outcome and moderation-outcome effects (Fairchild & MacKinnon, 2009). Although more complicated than traditional outcome analyses, mediation and moderation analyses allow for additional tests of the imported ESI’s deep structure by examining relations between theory-based mediators (typically malleable risk and protective factors) and distal outcomes. While there certainly can be a choice in the approach to the analyses in Phase II, what is arguably critical is the comparative nature of the experimental design used in Phase II. A three-arm randomized trial provides for a rigorous comparison between alternative adaptations (Castro et al., 2010). It is a design feature that has been absent in many replications of imported ESIs.

Potential limitations of PIA are yet to be empirically documented but could include such as garnering an adequate research funding for both formative research and a conceptually and temporally connected intervention effectiveness trial, ensuring adequate statistical power to identify intervention effects in a multiarmed intervention trial as well as navigating new intervention stakeholder and program developer collaborations in cases in which there are more than one program developer or in a situation in which the program developer is not able to participate in PIA. However, even in the context of several challenges, conducting imported intervention effectiveness research with little systematic plan toward cultural adaptation also poses already demonstrated limits on knowledge development. Hence, the positing of intervention cultural adaptation models and empirical tests of such models is a promising future research direction for intervention science. PIA represents one of many potential models that warrant empirical exploration, as the field works to develop the ideas and tools needed to transport useful interventions on an international scale.

Problems and Current Best Practices in the Cultural Adaptation of Interventions

The specific parts of an ESI that represent its surface or deep structure and the degree to which structures should be allowed to change are often matters of debate. Interventions well-suited to cultural adaptation are those that have an explicit deep structure, which is rooted in descriptive research, theory, and/or validation studies. Findings from mediation analyses are particularly useful in specifying deep structures.

The cultural adaptation of imported ESIs should be conducted—in our view—in the context of rigorous effectiveness trials, rather than in large-scale dissemination studies. In the rush to improve routine services by adopting ESIs, there can be a mind-set and even political pressure to begin with dissemination studies. This perspective tends to underestimate the challenges of adaptation, and, while adaptation of course is still present in dissemination studies, calibrating the optimal degree of adaptation is more likely to be accomplished in smaller scale effectiveness research.

Initial decisions about adaptations should be based on literature review and formative research, the kind of which has been advanced by several cultural adaptation models including PIA (e.g., etiological studies, focus groups, and pilot testing). New end users and program developers should use this formative research as a basis for consensus building around which decisions regarding adaptations should be undertaken. In our experience, difficulties can arise in building consensus for adaptations. Findings from formative research and the effort to systematically consider adaptations may better elucidate what, if anything, is essential to modify. Lacking this, some less savory alternatives to building consensus are foregoing all forms of adaptation, other than those of language translation, or leaving adaptation decisions to end users, perhaps relying on the judgment of administrators who make decisions based on implementation problems as they emerge.

Carefully documenting what is considered a modification (e.g., add-ons, omissions, or modifications) as well as the extent and nature of the actual adaptations are likely to be essential in the ensuring the long-term viability of an imported intervention in a new context. A record of the adaptation process provides a guide both for future end users and for future adaptations (e.g., Kumpfer et al., 2008; UNODC, 2009). It is important to utilize experimental designs that allow for the isolation of cultural adaptations in relation to outcomes (e.g., Castro et al., 2010). In public health, Resnicow, Dilorio, and Davis (2009, p. 213) argued that “more research is needed to determine the independent contribution of cultural tailoring on key consumer variables such as perceived relevance and salience, and, ultimately, behavioral impact.” More generally, a research-driven approach to cultural adaptation has potential to inform implementation science, where fidelity is one of a host of major concerns and adaptation is an open, and sometimes nagging, empirical question (see, e.g. Damschroder et al., 2009; Grimshaw, Eccles, Lavis, Hill, & Squires, 2012).

Conclusion

The spread of evidence-based practice throughout the world has resulted in an increased interest in ESIs and a growing number of controlled trials of imported and culturally adapted interventions. Evidence from selected case examples of replication trials of family-based U.S. BMPP appears mixed. We are beginning to learn from these successes and failures that features of both ESIs and the research designs used to test them may contribute to outcomes, that is whether transport from one cultural context to another is successful in terms of program implementation and observed outcomes. In this respect, one single negative case indicates a problem in transportability, thus warrants reflection. The factors that influence the successful translation and implementation of ESIs include variation in aspects of research design, the degree of adaptation made to both the surface and deep structures of an intervention, and the interaction of program content with the political, sociodemographic, and cultural context.

The next generation of imported ESI trials should incorporate research designs that allow for a differential examination of surface and deeper adaptations. The connection between culture, the core premises of interventions, and program effectiveness is strong and often underestimated. It should be the focus of substantial research attention (Castro et al., 2010). Data from recent replications suggest that carefully controlled effectiveness research is warranted before an ESI is recommended for dissemination in a new cultural context. Also, much can be learned from domestic intervention adaptations for ethnic or racial subgroups, especially if risk processes differ within subgroups. An emerging challenge is the identification of aspects of adaptation that may be unique to specific contexts—and should remain conceptually and practically separated—versus those aspects of adaptation that may have universal application.

The disparate evidence base on imported ESIs, including the family-based interventions that we have reviewed here, serves as both a catalyst for and a guide to building knowledge about successful transportation. This type of work is in its infancy. Advances have been hindered by a lack of systematic research on cultural factors that may be related to intervention benefits and a dearth of evidence supporting the models used to drive the cultural adaptation of ESIs.

Strengthening the knowledge base of information on the transportation of ESIs from one cultural context to another has the potential to advance implementation science. Unquestionably, recent replication failures have focused the wider field of evidence-based practice on issues of adaptation and implementation (Grimshaw et al., 2012; Valentine et al., 2011). Advances in the adaptation field may contribute to the development of standards for implementation (e.g., development of culturally sensitive measures and use of factorial designs to compare minimal versus more comprehensive adaptation of program material). Implementation science in general and the field of cultural adaptation in particular hold the promise of spreading useful interventions across the globe. Although challenges clearly loom large, the potential for evidence-based practice rests nontrivially on developing scientific approaches to the translation and adaptation of promising and effective interventions.

Footnotes

Authors’ Notes

This article is based on a chapter by Sundell, K. & Ferrer-Wreder, L. (in press). The transportability of empirically-supported interventions. In A. Shlonsky & R. Benbenishty (Eds.), Using Evidence in Child Welfare. Publisher under negotiation. Relative to the chapter, this article is a commentary intended for a more diverse disciplinary audience.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Ahluwalia

J. S.

Harris

K. J.

Catley

Okuyemi

K. S.

Mayo

M. S.

(2002). Sustained-release bupropion for smoking cessation in African Americans: A randomized controlled trial. Journal of the American Medical Association, 288, 468–474.

Allen

J. D.

Linnan

L.-A.

Emmons

K. M.

(2012). Fidelity and its relationship to implementation effectiveness, adaptation and dissemination. In Brownson

R. S.

Colditz

G. A.

Proctor

E. K.

(Eds.), Dissemination and implementation research in health translating science to practice (pp. 281–304). New York, NY: Oxford University Press.

Andrée Löfholm

Brännström

Olsson

Hansson

(2013). Treatment-as-usual in effectiveness studies: What is it and does it matter? International Journal of Social Welfare, 22, 25–34.

Andrée

Löfholm C.

Eichas

Sundell

(submitted). Treatment fidelity in Multisystemic Therapy: Association with time, therapist and team experience.

Andrée Löfholm

Olsson

Sundell

Hansson

(2009). Multisystemic therapy with conduct disordered youth: Stability of treatment outcomes two years after intake. Evidence & Policy, 5, 373–397.

Aos

Lee

Drake

Penucci

Kilma

Miller

… Burley

(2011). Return on investment: Evidence-based options to improve statewide outcomes (Document No. 11-07-1201). Olympia: Washington State Institute for Public Policy.

Axberg

Broberg

A. G.

(2012). Evaluation of ‘The Incredible Years’ in Sweden: The transferability of an American parent-training programme to Sweden. Scandinavian Journal of Psychology, 53. 224–232.

Baldwin

S. A.

Christian

Berkeljon

Shadish

W. R.

(2012). The effects of family therapies for adolescent delinquency and substance abuse: A meta-analysis. Journal of Marital and Family Therapy, 38, 281–304.

Barkley

(1997). Defiant children. A clinician's manual for assessment and parent training (2nd ed.). New York, NY: The Guilford Press.

10.

Berkel

Mauricio

A. M.

Schoenfelder

Sandler

I. N.

(2011). Putting the pieces together: An integrated model of program implementation. Prevention Science, 12, 23–33.

11.

Bernal

Jiménez-Chafey

M. I.

Rodríguez

M. M. D.

(2009). Cultural adaptation of treatments: A resource for considering culture in evidence-based practice. Professional Psychology: Research and Practice, 40, 361–368.

12.

Bloomquist

M. L.

Schnell

S. V.

(2002). Helping children with aggression and conduct problems. New York, NY: The Guilford Press.

13.

Bodenmann

Cina

Ledermann

Sanders

M. R.

(2008). The efficacy of the Triple P-Positive Parenting Program in improving parenting and child behavior: A comparison with two other treatment conditions. Behaviour Research and Therapy, 46, 411–427.

14.

Bodin

M. C.

Strandberg

A. K.

(2011). The Örebro prevention program revisited: A cluster-randomized effectiveness trial of program effects on youth drinking. Addiction, 106, 2134–2143.

15.

Burns

Catty

Dash

Roberts

Lockwood

Marshall

(2007). Use of intensive case management to reduce time in hospital in people with severe mental illness: Systematic review and meta-regression. British Medical Journal, 335, 336–341.

16.

Butler

Baruch

Hickey

Fonagy

(2011). A randomized controlled trial of multisystemic therapy and a statutory therapeutic intervention for young offenders. Journal of the American Academy of Child and Adolescent Psychiatry, 50, 1220–1235.

17.

Byrne

B. M.

van De Vijver

F. J. R.

(2010). Testing for measurement and structural equivalence in large-scale cross-cultural studies: Addressing the issue of nonequivalence. International Journal of Testing, 10, 107–132.

18.

Cardemil

E. V.

(2010). Cultural adaptations to empirically supported treatments: A research agenda. The Scientific Review of Mental Health Practice, 7, 8–21.

19.

Castro

F. G.

Barrera

Holleran Steiker

L. K.

(2010). Issues and challenges in the design of culturally adapted evidence-based interventions. Annual Review of Clinical Psychology, 6, 213–239.

20.

Curtis

N. M.

Ronan

K. R.

Borduin

C. M.

(2004). Multisystemic treatment: A meta-analysis of outcome studies. Journal of Family Psychology, 18, 411–419.

21.

Damschroder

L. J.

Aron

D. C.

Keith

R. E.

Kirsh

S. R.

Alexander

J. A.

Lowery

J. C.

(2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4, 50. doi:10.1186/1748-5908-4-50

22.

Deković

Asscher

J. J.

Manders

W. A.

Prins

P. J. M.

van der Laan

(2012). Within-intervention change: Mediators of intervention effects during multisystemic therapy. Journal of Consulting and Clinical Psychology, 80, 574–587. doi:10.1037/a0028482

23.

DeVellis

R. F.

(2012). Scale development: Theory and application (3rd ed.). Thousand Oaks, CA; Sage Press.

24.

Donaldson

S. I.

(2007). Program theory-driven evaluation science: Strategies and applications. Mahwah, NJ: Lawrence Erlbaum.

25.

Durlak

DuPre

(2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350.

26.

Eisner

Nagin

Ribeaud

Malti

(2012). Effects of a universal parenting program for highly adherent parents: A propensity score matching approach. Prevention Science, 13, 252–266. doi:10.1007/s11121-011-0266-x

27.

Enebrink

Högström

Forster

Ghaderi

(2012). Internet-based parent management training: A randomized controlled study. Behaviour Research and Therapy, 50, 240–249. doi:10.1016/j.brat.2012.01.006

28.

Emshoff

J. G.

(2008). Researchers, practitioners, and funders: Using the framework to get us on the same page. American Journal of Community Psychology, 41, 393–403.

29.

Fairchild

A. J.

MacKinnon

D. P.

(2009). A general model for testing mediation and moderation effects. Prevention Science, 10, 87–99.

30.

Ferrer-Wreder

Adamson

Kumpfer

K. L.

Eichas

(2012). Advancing intervention science through effectiveness research: A global perspective. Child and Youth Care Forum, 41, 109–117. doi:10.1007/s10566-012-9173-y

31.

Ferrer-Wreder

Sundell

Mansoory

(2012). Tinkering with perfection: Theory development in the intervention cultural adaptation field. Child and Youth Care Forum, 41, 149–171. doi:10.1007/s10566-011-9162-6

32.

Fossum

Mørch

W-T.

Handegård

B. H.

Drugli

M. B.

Larsson

(2009). Parent training for young Norwegian children with ODD and CD problems: Predictors and mediators of treatment outcome. Scandinavian Journal of Psychology, 50, 173–181.

33.

Fraser

M. W.

(Ed.) (2004). Risk and resilience in childhood: An ecological perspective (2nd ed.). Washington, DC: NASW Press.

34.

Frechtling

J. A.

(2007). Logic modeling methods in program evaluation. San Francisco, CA: Jossey-Bass.

35.

Gardner

Burton

Klimes

(2006). Randomised controlled trial of a parenting intervention in the voluntary sector for reducing child conduct problems: Outcomes and mechanisms of change. Journal of Child Psychology and Psychiatry, 47, 1123–1132.

36.

Ginner-Hau

Smedler

A. C.

(2011). Different problems—same treatment. Swedish juvenile offenders in community-based rehabilitative programmes. International Journal of Social Welfare, 20, 87–96.

37.

Grant

K. E.

Compas

B. E.

Stuhlmacher

A. F.

Thurm

A. E.

McMahon

S. D.

Halpert

J. A.

(2003). Stressors and child and adolescent psychopathology: Moving from markers to mechanisms of risk. Psychological Bulletin, 129, 447–466.

38.

Grimshaw

J. M.

Eccles

M. P.

Lavis

J. N.

Hill

S. J.

Squires

J. E.

(2012). Knowledge translation of research findings. Implementation Science, 7, 50. doi:10.1186/1748-5908-7-50

39.

Grissom

R. J.

(1996). The magical number .7 + .2: Meta-meta-analysis of the probability of superior outcome in comparisons involving therapy, placebo, and control. Journal of Consulting and Clinical Psychology, 64, 973–982.

40.

Grissom

R. J.

Kim

J. J.

(2012). Effect sizes for research: Univariate and multivariate applications (2nd ed.). New York, NY: Routledge.

41.

Hahlweg

Heinrichs

Kuschel

Feldmann

(2007). Therapist-assisted, self-administered bibliotherapy to enhance parental competence short- and long-term effects. Behavior Modification, 32, 659–681.

42.

Hansen

W. B.

(2011). Was Herodotus correct? Prevention Science, 12, 118–120.

43.

Hansson

Cederblad

Höök

(2000). Funktionell familjeterapi. Socialvetenskaplig tidskrift, 3, 231–243.

44.

Hansson

Olsson

(2012). Effects of multidimensional treatment foster care (MTFC): Results from a RCT study in Sweden. Children and Youth Services Review, 34, 1929–1936.

45.

Harris

K. J.

Ahluwalia

J. S.

Okuyemi

K. S.

Turner

J. R.

Woods

M. N.

Backinger

C. L.

Ken

(2001). Addressing cultural sensitivity in a smoking cessation intervention: Development of the Kick It at Swope Project. Journal of Community Psychology, 29, 447–458.

46.

Hofstede

(2001). Culture’s consequences. Comparing values, behaviors, institutions and organizations across nations. Thousand Oaks, CA: Sage.

47.

Jaffe

S. R.

Caspi

Moffitt

T. E.

Polo-Tomás

Taylor

(2007). Individual, family, and neighborhood factors distinguish resilient from non-resilient maltreated children: A cumulative stressor model. Child Abuse & Neglect, 31, 231–253.

48.

Kling

Å.

Forster

Sundell

Melin

(2010). A randomized controlled effectiveness trial of parent management training with varying degrees of therapist support. Behavior Therapy, 41, 530–542.

49.

Koning

I. M.

Vollebergh

W. A. M.

Smit

Verdurmen

J. E. E.

van den Eijnden

R. J. J. M.

ter Bogt

T. F. M.

Engels

R. C.

(2009). Preventing heavy alcohol use in adolescents (PAS): Cluster randomized trial of a parent and student intervention offered separately and simultaneously. Addiction, 104, 1669–1678.

50.

Koutakis

Stattin

Kerr

(2008). Reducing youth alcohol drinking through a parent-targeted intervention: The Örebro Prevention Program. Addiction, 103, 1629–1637.

51.

Kumpfer

K. L.

Pinyuchon

Teixeira de Melo

Whiteside

H. O.

(2008). Cultural adaptation process for international dissemination of the Strengthening Families Program. Evaluation & the Health Professions, 31, 226–239.

52.

Kumpfer

K. L.

Xie

O’Driscoll

(2012). Effectiveness of a culturally adapted Strengthening Families Program 12-16 Years for high risk Irish families. Child and Youth Care Forum, 41, 173–195. doi:10.1007/s10566-011-9168-0

53.

Kyhle Westermark

Hansson

Olsson

(2011). Multidimensional treatment foster care (MTFC): Results from an independent replication. Journal of Family Therapy, 33, 20–41.

54.

Leschied

A. W.

Cunningham

(2002). Seeking effective interventions for serious young offenders—Interim results of a fouryear randomized study of multisystemic therapy in Ontario, Canada. Centre for Children & Families in the Justice System. Retrieved February 15, 2010, from www.lfcc.on.ca/mst_final_results.html

55.

Leung

Sanders

M. R.

Leung

Mak

Lau

(2003). An outcome evaluation of the implementation of the Triple P-Positive Parenting Program in Hong Kong. Family Process, 42, 531–544.

56.

Levin

(1998). Uppfostringsanstalten. Om tvång i föräldrars ställe [The borstal home]. (Lund studies in social welfare XX). Lund: Arkiv förlag.

57.

Lipsey

M. W.

(1999). Can intervention rehabilitate serious delinquents? The Annals of the American Academy of Political and Social Science, 564, 142–166.

58.

Littell

J. H.

Popa

Forsythe

(2005). Multisystemic therapy for social, emotional, and behavioural problems in youth aged (pp. 10–17). Chichester, England: John Wiley.

59.

Magill

Ray

L. A.

(2009). Cognitive-behavioral treatment with adult alcohol and illicit drug users: A meta-analysis of randomized controlled trials. Journal of Studies on Alcohol and Drugs, 80, 516–527.

60.

Malti

Ribeaud

Eisner

(2011). The effectiveness of two universal preventive interventions in reducing children's externalizing behavior: A cluster-randomized controlled trial. Journal of Clinical Child and Adolescent Psychology, 40, 677–692.

61.

Matsumoto

Sofronoff

Sanders

M. R.

(2010). Investigation of the effectiveness and social validity of the Triple P positive parenting program in Japanese society. Journal of Family Psychology, 24, 87–91.

62.

McConnell

Breitkreuz

Savage

(2011). Independent evaluation of the Triple P Positive Parenting Program in family support service settings. Child & Family Social Work, 17, 43–54.

63.

McKleroy

Galbraith

Cummings

Jones

Gelaude

Carey

… Carey

J. W.

(2006). Adapting evidence based behavioral interventions for new settings and target populations. AIDS Education Prevention, 18, 59–73.

64.

Ogden

Hagen

K. A.

(2006). Multisystemic therapy of serious behavior problems in youth: Sustainability of therapy effectiveness two years after intake. Child and Adolescent Mental Health, 11, 142–149.

65.

Petrosino

Soydan

(2005). The impact of program developers as evaluators on criminal recidivism: Results from a meta-analysis of experimental and quasi-experimental research. Journal of Experimental Criminology, 1, 435–450.

66.

Prinz

R. J.

Sanders

M. R.

Shapiro

C. J.

Whitaker

D. J.

Lutzke

J. R.

(2009). Population-based prevention of child maltreatment: The U.S. triple p system population trial. Prevention Science, 10, 1–12.

67.

Resnicow

Braithwaite

R. L.

(2001). Cultural sensitivity in public health. In Braithwaite

R. L.

Taylor

S. E.

(Eds.), Health issues in the Black community (pp. 516–541). San Francisco, CA: Jossey-Bass.

68.

Resnicow

Diorio

Davis

(2009). Cultural sensitivity in culture and the development of HIV prevention and treatment programs. In Edgar

T. M.

Noar

S. M.

Freimuth

V. S.

(Eds.), Communication perspectives on HIV/AIDS for the 21st century (pp. 193–220). New York, NY: Lawrence Erlbaum.

69.

Resnicow

Soler

Braithwaite

R. L.

Ahluwalia

J. S.

Butler

(2000). Cultural sensitivity in substance use prevention. Journal of Community Psychology, 28, 271–290.

70.

Rutter

(1979). Protective factors in children’s responses to stress and disadvantage. In Kent

M. W.

Rolf

J. E.

(Eds.), Primary prevention of psychopathology: Social competence in children (pp. 49–74). Hanover, NH: University Press of New England.

71.

Schoenwald

S. K.

Heiblum

Saldana

Henggeler

S. W.

(2008). The international implementation of Multisystemic Therapy. Evaluation and the Health Professions, 31, 211–224.

72.

Shadish

W. R.

(2011). Randomized controlled studies and alternative designs in outcome studies: Challenges and opportunities. Research on Social Work Practice, 21, 636–643.

73.

Skärstrand

(2010). Prevention of alcohol and drug preblems among adolescents: Evaluating Swedish version of the strengthening families programme (Doctoral dissertation , Karolinska institutet, Stockholm, Sweden). Retrieved from http://publications.ki.se/jspui/handle/10616/40344

74.

Skärstrand

Larsson

Andreasson

(2008). Cultural adaptation of the strengthening families programme to a Swedish setting. Health Education, 108, 287–300.

75.

Sundell

Hansson

Andrée Löfholm

Olsson

Gustle

L-H.

Kadesjö

(2008). Multisystemic therapy and traditional services for antisocial adolescents in Sweden. Results from a randomized controlled trial after six months. Journal of Family Psychology, 22, 550–560.

76.

Sundell

Vinnerljung

Andrée Löfholm

Humlesjö

(2007). Child protection in Stockholm: A local cohort study of childhood prevalence. Children and Youth Services Review, 29, 180–192.

77.

Super

C. M.

Harkness

(1999). The environment as culture in developmental research. In Friedman

S. L.

Wachs

T. D.

(Eds.), Measuring environment across the life span (pp. 279–323). Washington, DC: American Psychological Association.

78.

Sussman

Unger

J. B.

Palinkas

(2008). Country prototypes and translation of health programs. Evaluation & the Health Professions, 31, 110–123.

79.

Swedish National Board of Health and Welfare. (2011). Svensk och internationell forskning om sociala interventioners effekter [Swedish and international research on the effects of social interventions]. Stockholm, Sweden: Socialstyrelsen.

80.

Taylor

T. K.

Schmidt

Pepler

Hodgins

(1998). A comparison of eclectic treatment with Webster-Stratton’s parents and children series in a children's mental health center: A randomized controlled trial. Behavior Therapy, 29, 221–240.

81.

United Nations Office on Drugs and Crime. (2009). Guide to implementing family skills training programmes for drug abuse prevention. Vienna, Austria: Author.

82.

Valentine

J. C.

Biglan

Boruch

R. F.

Castro

F. G.

Collins

L. M.

Flay

B. R.

… Schinke

S. P.

(2011). Replication in prevention science. Prevention Science, 12, 103–117.

83.

Webster-Stratton

(1996). Early intervention with videotape modeling: Programs for families of children with oppositional defiant disorder or conduct disorder. In Hibbs

E. S.

Jensen

P. S.

(Eds.), Psychosocial treatments for child and adolescent disorders: Empirically-based strategies for clinical practice (pp. 435–474). Washington, DC: American Psychological Association.

84.

A. D.

Zumbo

B. D.

(2007). Decoding the meaning of factorial invariance and updating the practice of multi-group confirmatory factor analysis: A demonstration with TIMSS data. Practical Assessment Research and Evaluation, 12, 1–26.

85.

Wright

C. C.

Sim

(2003). Intention to treat approach to data from randomized controlled trials: A sensitivity analysis. Journal of Clinical Epidemiology, 56, 533–842.