Cross-Cultural Validation of the Modified Practice Attitudes Scale: Initial Factor Analysis and a New Factor Model

Abstract

Objective: The objective of this study was to create the Korean version of the Modified Practice Attitudes Scale (K-MPAS) to measure clinicians’ attitudes toward evidence-based treatments (EBTs) in the Korean mental health system. Method: Using 189 U.S. therapists and 283 members from the Korean mental health system, we examined the reliability and validity of the MPAS scores. We also conducted the first exploratory and confirmatory factor analysis on the MPAS and compared EBT attitudes across U.S. and Korean therapists. Results: Results revealed that the inclusion of both “reversed-worded” and “non–reversed-worded” items introduced significant method effects that compromised the integrity of the one-factor MPAS model. Problems with the one-factor structure were resolved by eliminating the “non–reversed-worded” items. Reliability and validity were adequate among both Korean and U.S. therapists. Korean therapists also reported significantly more negative attitudes toward EBTs on the MPAS than U.S. therapists. Conclusions: The K-MPAS is the first questionnaire designed to measure Korean service providers’ attitudes toward EBTs to help advance the dissemination of EBTs in Korea. The current study also demonstrated the negative impacts that can be introduced by incorporating oppositely worded items into a scale, particularly with respect to factor structure and detecting significant group differences.

Keywords

evidence-based treatments cross-cultural assessment attitudes South Korea dissemination factor analysis

Evidence-Based Treatments

Over the past several decades, efficacious treatments have been developed for various emotional and behavior problems for both youth (Chorpita et al., 2011; Kazdin & Weisz, 1998, 2003) and adults (Ayers, Sorrell, Thorp, & Wetherell, 2007; Ponniah & Hollon, 2008). These effective treatments span different theoretical orientations, including but not limited to, cognitive behavioral therapy for depression (Hofmann & Smits, 2008; Ost, 2008; Siev & Chambless, 2007), behavioral therapy for anxiety (Ekers, Richards, & Gilbody, 2008; Norton & Price, 2007), interpersonal psychotherapy for depression (Cuijpers, van Straten, Andersson, & Van Oppen, 2008), short-term psychodynamic therapy (Leichsenring, 2001; Shedler, 2010), and family therapy for conduct and eating disorder problems (Carr, 2009; Cottrell & Boston, 2002). The “dodo bird verdict”—that all treatments are equally effective—no longer seems to be true (Beutler, 2002; Crits-Christoph et al., 1991), and psychosocial interventions appear to have differential effectiveness depending on the therapeutic content delivered. These treatments with demonstrated research support are known as evidence-based treatments (EBTs).

Dissemination Efforts of EBTs

Despite these EBT developments, a large portion of mental health providers in the field continue to deliver treatments that have little to no empirical support (Baumann, Kolko, Collins, & Herschell, 2006; Weersing & Weisz, 2002; Zima et al., 2005)—and when tested against EBTs, these non-EBTs delivered by community clinicians tend to do worse than EBTs (Weisz et al., 2013; Weisz, Jensen-Doss, & Hawley, 2006). This is unfortunate, particularly given the amount of distress and long-term impairment associated with mental health and behavioral problems when left unresolved (e.g., Compton, Burns, Egger, & Robertson, 2002; Langley, Bergman, McCracken, & Piacentini, 2004; Rapaport, Clary, Fayyad, & Endicott, 2005). Given this problem, researchers have begun to focus on how to disseminate EBTs more effectively to practicing clinicians in the field (McHugh & Barlow, 2010). Efforts taken so far to increase the dissemination and implementation of EBTs have included (a) increasing accessibility of effective treatments (e.g., through publishing lists of effective practices online, such as via the American Psychological Association Division 12 Society of Clinical Psychology’s “Website on Research-Supported Psychological Treatments,” www.div12.org/PsychologicalTreatments/index.html), (b) increasing the flexibility of EBT approaches (e.g., Chorpita, Daleiden, & Weisz, 2005), and (c) funding agencies increasing support for training therapists to deliver EBTs in “real world” clinical settings (Clark et al., 2009).

EBTs in Cross-Cultural Contexts

EBTs have also begun to be tested and developed for implementation in non-English speaking settings and countries, including in Cambodia (Hinton et al., 2005; Otto et al., 2003), Vietnam (Hinton et al., 2004), Iran (e.g., Ebesutani, Helmi, Fierstein, Taghizadeh, & Chorpita, 2016; Jaberghaderi, Greenwald, Rubin, Zand, & Dolatabadi, 2004), Japan (Asukai, Saito, Tsuruta, Kishimoto, & Nishikawa, 2010), and South Korea (Park, Oh, Oh, Jung, & Na, 2014). Wide-scale adoption and implementation of EBTs in these countries have however been also limited, including in South Korea (e.g., Ebesutani & Shin, 2014). Several of these countries are in dire need of more effective mental health services. For example, anxiety and depression are highly prevalent among Korean adolescents (Ahn, Ebesutani, & Kamphaus, 2014; Ahn & Shin, 2012), and suicide rates are cited as the most common causes of death among Korean youth (Kim, 2003).

Measurement of EBT Attitudes

Across the world, uptake and implementation of EBTs in “real-world” settings have been slow and have faced with obstacles (Strupp & Anderson, 1997; Westen, Novotny, & Thompson-Brenner, 2004). Researchers have thus recently begun to step back and ask more basic questions regarding why clinicians are not adopting and using EBTs, despite their increased availability, flexibility, and training opportunities. Diffusion of innovation theory (Rogers, 2003) suggests that positive attitudes toward the “new technology” is important for successful dissemination and implementation. Assessment instruments have therefore been recently developed to measure clinician’s attitudes toward EBTs. The Evidence-Based Practice Attitude Scale (EBPAS; Aarons, 2004) was one measure developed for this purpose. The EBPAS asks questions about EBTs, such as “appeal” of EBTs, with a particular emphasis on manualized treatments. Through this measure, researchers found that therapists tend to view manualized EBTs as rigid and inflexible. Borntrager, Chorpita, Higa-McMillan, and Weisz (2009) modified the EBPAS and created the Modified Practice Attitudes Scale (MPAS), which does not specifically ask about treatment manualization, per se, but rather about the empirically supported nature of EBTs in general.

The MPAS has since been used in a variety of contexts, including examining changes in attitudes following professional trainings of EBTs (Borntrager et al., 2009) and university courses on evidence-based practices (Bearman, Wadkins, Bailin, & Doctoroff, 2015). The MPAS has also been used to understand beliefs held by clinicians hindering their adoption of EBTs (Lilienfeld, Ritschel, Lynn, Cautin, & Latzman, 2013) and also significant correlates of EBT attitudes, such as readiness to change and perceived client barriers to implementation (Lewis & Simons, 2011).

Additionally Needed Psychometric Validation of the MPAS

Despite increased use of the MPAS across a variety of settings to measure this important aspect of behavior change (i.e., attitudes; Rogers, 2003), relatively little work has been published on its psychometric properties. Borntrager and colleagues (2009) reported on its reliability (α = .80) and that the MPAS correlated significantly with the EBPAS scores (i.e., another attitude measure; r = .77) supporting its construct validity. However, no studies to date have examined the factor structure of the MPAS. This is somewhat concerning, particularly given that the MPAS includes both reversed-worded and non–reversed-worded items—a feature of self-report scales that can be problematic given that it may introduce wording “method effects” that can complicate factor structures and scoring procedures (Brown, 2003; Marsh, 1996). For example, van Sonderen, Sanderman, and Coyne (2013) found that the inclusion of both non–reversed-worded and reversed-worded items can contribute to respondent confusion while not adequately preventing response acquiescence. It also may produce spurious factors in exploratory factor analysis (EFA) whereby results often show a “2-factor structure,” when in reality, the data are unifactorial in nature (albeit with method effects; see Brown, 2006). Studies are thus needed to investigate the psychometric properties of the MPAS (among both U.S. and non-U.S. therapists), particularly while exploring for potential wording “method effects.”

The Present Study

The purpose of the present study was thus twofold. First, we sought to create a Korean version of the MPAS and examine its psychometric properties for use in the Korean mental health system. Second, we sought to examine for the first time the factor structure underlying the MPAS.¹ We were particularly interested in whether any modifications would be needed due to the presence of “method effects” introduced by the reversed- and non–reversed-worded items.

We predicted that wording “method effects” would have to be considered to achieve a well-fitting model with adequate psychometric properties. Even though the MPAS is theoretically posited to be a unidimensional scale (reflecting attitudes toward EBTs), not appropriately controlling for “method effect” could lead to a poorly fitting unidimensional model making it appear as though multiple factors are present, when in fact only a single attitude factor is responsible for meaningful item variation (as commonly found in the literature; Bagner, Storch, & Roberti, 2004; Fresco, Heimberg, Mennin, & Turk, 2002). When comparing attitudes of EBTs between Korean and U.S. therapists, we also hypothesized that Korean therapists would report significantly more negative attitudes toward EBTs compared with U.S. therapists. This is because dissemination of EBTs in Korea has begun relatively more recently compared to the United States. We also hypothesized that the methods effects associated with the original MPAS would lead to biased total scores in these group comparisons when and when not accounting for these method effects.

Method

Participants

Korean Sample

The Korean sample included 283 participants from the fields of clinical, counseling, health, and school psychology who completed the Korean version of the MPAS (K-MPAS) as part of a survey related to evidence-based practices in psychology in Korea. This sample included 256 females (90.5%) and 25 males (8.8%). Two participants did not report their gender. Ages ranged from 22 to 50 years (M = 30.0 years, SD = 7.73). All participants were Korean and fluent in Korean. Additional background information on the participants may be seen in Table 1.

Table 1.

Participants Background Information.

	Korean graduate students (n = 114); M (SD)	Korean direct service professionals (n = 143); M (SD)	Total Korean sample (n = 283); M (SD)
Age (years)	26.81 (5.73)	32.15 (8.06)	30.05 (7.73)
Years of clinical experience	0.88 (1.77)	4.18 (4.81)	2.68 (4.04)
Average number of assessment cases per month	3.39 (9.45)	11.25 (10.23)	8.38 (10.59)
Average number of therapy cases per month	5.44 (19.06)	6.58 (11.28)	6.48 (14.85)
Average number of therapy sessions per month	6.44 (14.67)	13.57 (18.79)	10.77 (17.34)
Number of hours of supervision per week	1.74 (3.85)	3.06 (3.46)	2.36 (3.59)

Note. The sample size of the total Korean sample does not equal the sum of the Korean graduate student sample and the Korean direct service professionals sample because there were 23 participants who responded as “other” and 3 participants who did not provide this information.

U.S. Sample

The U.S. sample included 189 therapist participants from the Montana mental health system. The gender composition of this therapist sample was 60% female and 40% male. A total of 77 therapists reported having a master’s degree (40.7%), 45 reported having a PhD (23.8%), and 1 reported having an “other” degree (0.5%); 66 therapists (34.9%) did not report their degree. The most common reported primary theoretical orientation was cognitive behavioral (n = 78, 41.3%), followed by eclectic (n = 38, 20.1%) and then psychodynamic (n = 18, 9.5%). Other primary theoretical orientations at rates of less than 5% included humanistic, interpersonal, client-centered, solution-focused, family systems, and Gestalt. Years of training ranged from 1 year to 35 years (M = 6.7 years, SD = 5.42). Participants reported an average of 17.4 years in practice (SD = 10.53), with a range of 0.5 to 45 years. All participants of this U.S. sample were American and fluent in English.

Procedure

Korean Sample Procedures

We sought to obtain responses from participants from a variety of areas in the mental health field in South Korea. We therefore first administered the MPAS measure (along with other related questionnaires) to psychology graduate students and direct service professionals attending a clinical psychology conference. We then created an online version of this survey and sent it to a listserv of school counselors. Last, we sent the same online survey to the Korean Clinical Psychology Association listserv. Participants who consented to participate completed the questionnaires anonymously. A random subset of participants was rewarded with gift certificates for their participation. All study procedures were approved by the university institutional review board.

U.S. Sample Procedures

U.S. therapists were identified through lists of psychologist and professional counselor licensees in the Montana mental health system. One thousand-five hundred anonymous surveys were sent to these individuals. A total of 191 participants returned their packets (response rate = 12.7%). In the mailed survey packet, participants were given an informed consent form explaining the full purpose of the study, that their participation was voluntary, that their return of the packet means that they agree to participate, and that their information will remain anonymous unless they chose to provide their name and phone number to be included in a $50 gift card raffle.

Measures

The Modified Practice Attitudes Scale

The MPAS (Borntrager et al., 2009) was developed in the United States to measure therapists’ attitudes toward EBTs. The original MPAS includes eight items, among which five items are reversed scored (Items 2, 3, 5, 7, and 8). Respondents are asked to rate the extent to which they agree with each item on a 5-point Likert-type scale, ranging from 0 (not at all) to 4 (to a very great extent). Reliability of the MPAS total scale score (estimated via internal consistency) based on a U.S. sample of therapists was reported to be .80 (Borntrager et al., 2009). No studies to date have reported on its factor structure.

The Korean Version of the Modified Practices Attitude Scale

The K-MPAS was based on the original 8-item English MPAS (Borntrager et al., 2009). For the translation of the MPAS from English to Korean, we followed the recommended back-translation procedures outlined by Brislin (1970). First, we had a bilingual MA graduate student in clinical psychology translate the MPAS items into Korean. We then had another bilingual MA graduate student in clinical psychology back-translate the Korean MPAS items into English. The research team (which included a PhD-level American clinical psychologist and a PhD-level Korean clinical psychologist) compared the back-translation with the original version to identify and resolve discrepancies. The eight MPAS items translated into Korean may be seen in the appendix.

Korean Version of the Attitudes Toward Standardized Assessment Scales

The Korean Version of the Attitudes Toward Standardized Assessment Scales (K-ASA; Ebesutani & Chung, 2014) is a 12-item measure developed to measure therapists’ attitudes toward standardized assessment scales. The K-ASA is based on the U.S. version of the ASA (Jensen-Doss & Hawley, 2010), and the K-ASA yields a total score and two subscales: “Benefit over Clinical Judgment” and “Practicality” of standardized assessment scales. Respondents are asked to rate items on a 5-point Likert-type scale, ranging from 1 (strongly disagree) to 5 (strongly agree). Higher scores indicate more negative attitudes toward standardized clinical assessment. The K-ASA demonstrated a well-fitting factor structure and acceptable reliability and validity indices (Ebesutani & Chung, 2014). Internal consistency of the ASA total score in the present sample was acceptable at .70.

Korean Version of the Knowledge of Evidence-Based Services Questionnaire

The Korean version of the Knowledge of Evidence-Based Services Questionnaire (K-KEBSQ; Ebesutani, 2014) was based on the original KEBSQ developed in the U.S. by Stumpf, Higa-McMillan, and Chorpita (2009). The KEBSQ measures knowledge of EBT practices for youth emotional and behavioral disorders. The KEBSQ contains 40 evidence-based and non-EBTs practices for the areas of anxious/avoidant, depressed/withdrawn, disruptive behavior, and attention/hyperactivity. Respondents are asked to indicate all problem areas where the practice has empirical support for effectiveness. All correct answers are summed to create a total knowledge score ranging from 0-160, with higher scores indicating more knowledge of EBTs. This questionnaire was reported as a stable and valid instrument to measure knowledge of EBTs (Stumpf et al., 2009). The total score was used in the present study as our criterion measure of EBT knowledge.

Data-Analytic Plan

Subsamples

We randomly divided our Korean sample (N = 289) into a random EFA subsample (n = 142) and a random confirmatory factor analysis (CFA) subsample (n = 141) for analyses described below.

Factor Analyses and Fit Indices

We used Mplus Version 7.11 (L. K. Muthén & Muthén, 2010) to conduct all factor analyses. Since the MPAS data were ordinal in nature, we treated these data as categorical and computed matrices based on polychoric correlations (Jöreskog, 1994; Olsson, 1979). We also used the robust weighted least squares with mean and variance adjustment (WLSMV) estimator, which has been recommended when conducting analyses on ordinal data (Flora & Curran, 2004; B. Muthén, du Toit, & Spisic, 1997). We used the following fit indices to evaluate model fit. Comparative fit index (CFI) values greater than .90 and .95 were used as benchmarks for acceptable and good fit, respectively (Hu & Bentler, 1999). Root mean square error of approximation (RMSEA) values less than .08 and .05 were used as benchmarks for acceptable and good fit, respectively (Browne & Cudeck, 1993). We also evaluated chi-square, which assesses the magnitude of discrepancy between the sample and fitted covariances matrices (Hu & Bentler, 1999). Good model fit is supported when the chi-square value is not significant (given its degrees of freedom; Kline, 2005).

Exploratory Factor Analysis

We first conducted EFA using Mplus Version 7.11 (Muthén & Muthén, 2010) to examine the factor structure of the K-MPAS data. We instructed Mplus to extract (a) a one-factor model (based on the original measure positing a single total score; Borntrager et al., 2009), (b) a two-factor model, given that the MPAS has both reversed-worded (e.g., “Evidence-based treatments do not allow me to tailor my therapy to each client’s individual needs”) and non–reversed-worded items (e.g., “I like using evidence-based treatments because of the structure they provide”),² and (c) a three-factor model to check for more than two factors.

We used multiple criteria to determine the number of factors underlying the MPAS. First, we examined eingenvalues; in particular, we examined the scree plot and the point at which the eingenvalues begin to “level-off” as indicative of the number of underlying factors (Costello & Osborne, 2005). Second, we examined the fit indices of each model to examine how well each model fit the data. Third, we considered the interpretability of the factor structures based on the item content of each factor. For example, if in the two-factor model, one of the factors consists of only the non–reversed-worded items, and the second factor consists of only the reversed-worded items, then the interpretation of the item-to-factor content mapping would suggest that this apparent “two-factor” model actually represents “method effects” due to the reversed-worded items—a common phenomenon in factor analytic research (Brown, 2006). Last, each latent factor had to have at least three items with significant factor loadings. Although “significance” of factor loadings in EFA used to be inferred based on the size of the factor loadings (e.g., >.30; cf. McDonald, 1999), Mplus now offers the ability to estimate the significance of each factor loading in an EFA context (L. K. Muthén & Muthén, 2010).

Confirmatory Factor Analysis

Standard One-Factor Model

We conducted CFA in the second half of the Korean sample to examine the fit of the original 8-item one-factor model (Borntrager et al., 2009). We then examined a one-factor model while controlling for method effects, as described directly below.

One-Factor Model, Controlling for Method Effects

In the context of CFA, “method effects” (i.e., systematic sources of item variance—such as due to the presence of reversed-worded and non–reversed-worded items) can be controlled for via two main techniques. The first is called (a) the correlated traits–correlated uniqueness (CT-CU) modeling approach. This CT-CU approach involves correlating error terms among the reversed-worded (or non–reversed-worded) items of a measure (see Corwyn, 2000; Marsh, 1996; Marsh, Scalas, & Nagengast, 2010). The theory underlying this approach is that error terms, under normal situations, are uncorrelated with each other. However, due to the same wording nature of either the reversed-worded (or non–reversed-worded) items, each set of reversed-worded (or non–reversed-worded) items are presumed to be affected by a portion of (nonrandom) error that is correlated with each other. This is modeled by correlating their error terms. The other way to control for method effects is called (b) the latent method factor (LMF) approach, which is also called the correlated traits–correlated method approach. This LMF approach involves creating and modeling (a) a LMF to account for the reversed-worded items and (b) an LMF to account for the non–reversed-worded items, or both (Bagozzi, 1993; Marsh & Grayson, 1995). Both CT-CU and correlated traits–correlated method approaches have been recommended and offer unique advantages (see Bagozzi, 1993; Brown, 2006; Tomas, Hontangas, & Oliver, 2000). For the present study, we used the CT-CU modeling approach to control for and model method effects (via correlating their error terms).

U.S. and Korean Samples

To examine the cross-cultural applicability of the newly identified model, we examined the CFA model in both the Korean sample and U.S. sample. We used the same fit index cutoffs noted above to examine model fit.

Reliability

We then examined reliability of the resulting items via internal consistency. We used .70 as the general cutoff score for acceptable reliability.

Validity

To examine convergent validity, we examined bivariate correlations between the K-MPAS total score and the K-ASA total score. This analysis examined whether attitudes toward EBTs converged significantly with attitudes toward evidence-based assessments (EBAs). We hypothesized that these correlations would be positive and significant. As an additional validity test, we also examined bivariate correlations between the K-MPAS total score and the K-KEBSQ total score. This analysis examined whether attitudes toward EBTs converged significantly with knowledge of EBTs. Significance was set at p < .05 for these analyses.

Cross-Cultural Comparison of Means

We then compared whether the U.S. and Korean therapists differed with respect to their attitudes toward EBTs across these two countries. We conducted a one-way analysis of variance to examine for significant differences across groups (p < .05).

Results

Exploratory Factor Analysis

The eigenvalues based on the Korean sample were as follows: 2.30, 1.69, 1.04, 0.83, 0.66, 0.55, 0.51, and 0.43. The scree plot can be seen in Figure 1. Based on the scree plot, it appears as though two factors are present. Examination of fit indices also suggested the presence of two-factors. Specifically, and consistent with a “method effects” model, fit indices for the one-factor EFA model were poor (RMSEA = .146, CFI = .689), and fit indices for the two-factor EFA model were excellent (RMSEA = .063, CFI = .962). The correlation between the two “factors” was nonsignificant (r = .15, p = .194). Although these fit indices suggest the presence of a two-factor model underlying the MPAS items, examination of the loading patterns in Table 2 clearly shows the presence of reversed-worded “method effects” contributing to the creation of the apparent two-factor structure, as predicted. All five of the reversed-worded items (i.e., MPAS Items 2, 3, 5, 7, and 8) loaded on the first factor, and all three of the non–reversed-worded items (MPAS Items 1, 4, and 6) loaded on the second factor. This is consistent with a “method effects” model, which may occur when reversed-worded items are present (e.g., Bagner et al., 2004; Fresco et al., 2002). In such cases, it may be the case that a single factor (e.g., “EBT Attitudes”) is largely responsible for item variation, albeit with wording-based method effects leadings to the statistical appearance of “two factors.” In such cases, however, the structure would still be unifactorial. This however cannot be tested in EFA. CFA is needed to be able to control for such method effects to then test the fit of a one-factor model (described below).

Figure 1.

Scree plot.

Table 2.

Exploratory Factor Analytic Results of the Modified Practice Attitudes Scale.

			Two-factor “wording method effects” model
		One factor	Factor 1	Factor 2
Items	Abbreviated/paraphrased item content	Factor loading (SE)	Factor loading (SE)	Factor loading (SE)
MPAS 1	I am willing to use EBTs.	−.16 (.107)	−.26 (.107)	.45 (.103)*
MPAS 4	I like [the structure] of EBTs.	−.12 (.078)	−.25 (.097)	.53 (.103)*
MPAS 6	EBTs allow clinicians to respond to important events in therapy.	.11 (.083)	.02 (.061)	.73 (.127)*
MPAS 2^a	EBTs do not allow me to tailor my therapy.	.61 (.084)*	.63 (.082)*	−.08 (.122)
MPAS 3^a	Clinical experience and judgment are more important than EBTs.	.36 (.079)*	.35 (.079)*	.03 (.109)
MPAS 5^a	EBTs require too much learning of different treatments for each problem area.	.47 (.080)*	.43 (.086)*	.27 (.11.)
MPAS 7^a	I dislike EBTs because they are too inflexible.	.69 (.069)*	.69 (.069)*	.00 (.006)
MPAS 8^a	EBTs are not designed to handle clients with multiple problems.	.65 (.073)*	.63 (.076)*	.08 (.114)

Note. Significant factor loadings (p < .01) are indicated in boldface and an asterisk. MPAS = Modified Practice Attitudes Scale; EBT = evidence-based treatment.

Reverse-worded items are Items 2, 3, 5, 7, and 8.

Confirmatory Factor Analysis

Results of the CFA analyses based on the Korean subsample may be seen in Table 3.

Table 3.

Fit Statistics for the Confirmatory Factor Analytic Models of the Modified Practice Attitudes Scale.

Model	CFI	RMSEA	χ²	df	p
Model A: Original 8-item, one-factor model (Korean data)	.753	.142	76.54	20	.000
Model B: 8-item CT-CU model with method effects^a (Korean data)	.923	.085	34.51	17	.007
Model C: Modified 5-item, one-factor model (Korean data)	1.000	.000	4.70	5	.453
Model D: Modified 5-item, one-factor model (U.S. data)	.998	.030	5.83	5	.324

Note. CT-CU = correlated traits–correlated uniqueness; CFI = comparative fit index; RMSEA = root mean square error of approximation.

Covariances were included among the errors of the reversed-worded items.

Standard One-Factor Model

Results revealed poor model fit for the original 8-item, one-factor MPAS model, which did not control for method effects (i.e., Model A; RMSEA = .142, CFI = .753). Three of the eight items also had nonsignificant factor loadings: Item 1 (.17), Item 4 (.15), and Item 6 (.07). The original 8-item, one-factor MPAS model clearly did not fit well in the Korean sample.

One-Factor Model, Controlling for Method Effects

Model fit based on the CT-CU one-factor model may be seen in Table 3. This model evidenced acceptable fit (i.e., Model B; CFI = .923, RMSEA = .085). However, all loadings associated with the three non–reversed-worded items remained nonsignificant and small (.03, .10, and .13; see Table 4).

Table 4.

Confirmatory Factor Analysis Model Parameters of the (CT-CU) One-Factor Model (Controlling for Method Effects).

		One-factor
Items	Abbreviated/paraphrased item content	Factor loading	SE	p value
MPAS 1	I am willing to use EBTs.	.13	.091	.142
MPAS 2^a*	EBTs do not allow me to tailor my therapy.	.54*	.072	.000
MPAS 3^a*	Clinical experience and judgment are more important than EBTs	.35*	.084	.000
MPAS 4	I like [the structure] of EBTs	.10	.091	.253
MPAS 5^a*	EBTs require too much learning of different treatments for each problem area	.59*	.076	.000
MPAS 6	EBTs allow clinicians to respond to important events in therapy	.03	.083	.759
MPAS 7^a*	I dislike EBTs because they are too inflexible.	.85*	.057	.000
MPAS 8^a*	EBTs are not designed to handle clients with multiple problems.	.62*	.061	.000

Note. Significant factor loadings (p < .01) are indicated in boldface and an asterisk. MPAS = Modified Practice Attitudes Scale; EBT = evidence-based treatment.

Reversed-worded items are Items 2, 3, 5, 7, and 8.

Another related recommendation for dealing with wording-related method effects is to simply remove either the reversed-worded or non–reversed-worded items (see Brown, 2003; Marsh, 1996). This is because (as noted above) the inclusion of both reversed-worded and non–reversed-worded items can introduce significant complexity with respect to factor structure and scoring procedures.

Following these recommendations (i.e., Brown 2003; Marsh 1996), we removed the three non–reversed-worded items from the scale and examined the fit of the 5-item one-factor CFA model. As shown in Table 3 (i.e., Model C: “Modified 5-item, one-factor model (Korean data)”), the fit of this revised model was now excellent (RMSEA < .05; CFI > .95). The factor loadings for the 5-item one-factor model appears in Table 5. All factor loadings were also significant (p < .001), ranging from .34 to .89. Collectively, these results provide support for this 5-item one-factor MPAS model in the Korean sample.

Table 5.

Confirmatory Factor Analysis Model Parameters of the 5-Item Modified Practice Attitudes Scale Based on the Korean and U.S. Data.

		One-factor model
Item/sample	Abbreviated/paraphrased item content	Factor loading	SE	p value
Korean data
MPAS 2	EBTs do not allow me to tailor my therapy	.53*	.075	.000
MPAS 3	Clinical experience and judgment are more important than EBTs	.34*	.082	.000
MPAS 5	EBTs require too much learning of different treatments for each problem area	.56*	.077	.000
MPAS 7	I dislike EBTs because they are too inflexible.	.89*	.062	.000
MPAS 8	EBTs are not designed to handle clients with multiple problems.	.62*	.061	.000
U.S. data
MPAS 2	EBTs do not allow me to tailor my therapy.	.75*	.040	.000
MPAS 3	Clinical experience and judgment are more important than EBTs.	.60*	.056	.000
MPAS 5	EBTs require too much learning of different treatments for each problem area.	.52*	.059	.000
MPAS 7	I dislike EBTs because they are too inflexible.	.82*	.036	.000
MPAS 8	EBTs are not designed to handle clients with multiple problems.	.78*	.038	.000

Note. Significant factor loadings (p < .01) are indicated in boldface and an asterisk. MPAS = Modified Practice Attitudes Scale; EBT = evidence-based treatment.

U.S. and Korean Samples

To examine whether this model also fit well in the U.S. sample, we examined this 5-item, one-factor MPAS model in a sample of U.S. therapists (n = 189). The fit of this model (based on the U.S. therapists’ data) appears in Table 3 (i.e., Model D: “Modified 5-item, one-factor model (U.S. data)”). This model also evidenced excellent model fit (e.g., RMSEA = .030, CFI = .998). All factor loadings for the 5-item one-factor model were also significant, ranging from .52 to .82 (see Table 5). Based on this sample of U.S. therapists, fit of the 5-item, one-factor model (RMSEA = .030, CFI = .998) was substantially better than the fit of the 8-item, one-factor model (RMSEA = .084, CFI = .964). These results provide support for the factor structure of this new (5-item) MPAS model across countries.

Reliability

Internal consistency of the 5-item MPAS total score was .78 based on the U.S. sample of therapists. Internal consistency of the 5-item K-MPAS total score was .65 based on the full Korean subsample.

Convergent Validity

EBT Attitudes With EBA Attitudes

Based on the Korean sample, the K-MPAS total score (i.e., attitudes toward EBTs) was significantly correlated with (a) the K-ASA total score (i.e., attitudes toward EBAs in general), r(283) = .45, p < .01; (b) the K-ASA Practicality subscale score (i.e., attitudes toward EBAs with respect to practicality), r(283) = .39, p < .01; and (c) the K-ASA Benefits Over Clinical Judgment subscale score (i.e., attitudes related to benefits over clinical judgment), r(270) = .40, p < .01. This indicates that the more positive one’s attitudes toward EBTs, the more positive are their attitudes are toward EBA.

MPAS Attitudes With Knowledge

Based on the Korean sample, (positive) attitudes toward EBTs (as measured by the K-MPAS total score) were positively and significantly correlated with K-KEBSQ total score (i.e., knowledge of EBTs), r(246) = .14, p < .05. Although this is associated with a small effect (Cohen, 1988), these results indicate that positive EBTs attitudes are somewhat related to greater knowledge of EBTs.

Cross-Cultural Comparison of Means

We compared attitudes toward EBTs based on the 5-item MPAS one-factor model total score across Korean and U.S. therapists.³ Based on the revised model, Korean therapists (n = 67) reported significantly more negative attitudes toward EBPs (i.e., significantly greater MPAS total scores; M = 11.96, SD = 2.94) than the 187 U.S. therapists (M = 10.95, SD = 3.67). The magnitude of the difference was small but significant (p = .04).

We then compared means between these same two groups based on the original 8-item scale. When based on this 8-item total score, these two groups were no longer associated with significant differences in means. These analyses revealed that the U.S. therapists (M = 29.63, SD = 5.11) and the Korean therapists (M = 28.37, SD = 3.24) did not significantly differ on EBT attitudes when using the full 8-item scale, F(254) = 3.55, p = .061, ns. These results suggest that removal of the three non–reversed-worded items from the scale may not be a trivial modification, and that this refinement can lead to significant differences in results and interpretations of central study questions, such as differences in EBT attitudes between groups.

Discussion

In the present study, we sought to develop a Korean version of the MPAS to measure attitudes toward EBTs among Korean therapists. This is an important pursuit because the field has been focused on determining how to most effectively disseminate EBTs to practicing community clinicians in the field to enhance care delivered to patients. Researchers have thus begun to measure clinician’s attitudes toward EBTs. In the present study, we added to this movement by translating the MPAS into Korean and examining the psychometric properties of its scales so that the K-MPAS may be used within the Korean mental health system. More specifically, we developed a modified, 5-item K-MPAS that performed well and was associated with good model fit, adequate reliability, and acceptable convergent validity.

The present study was also the first to examine the factor structure of the MPAS. Although reversed-worded items have traditionally been recommended in scale development (to protect against response acquiescence), researchers more recently have found that they do not protect well against response acquiescence and may actually contribute to respondent confusion and inattention (van Sonderen et al., 2013). Similarly, researchers have begun to wonder whether the inclusion of oppositely worded items do more harm than good (Brown, 2003; Marsh, 1996). For example, researchers examined this issue via item response theory and found that the non–reversed-worded items of the Loneliness Questionnaire were associated with significantly better psychometric properties than the reversed-worded items (Ebesutani et al., 2012b). Consequently, researchers have more recently begun to recommend removing the poorly performing, reversed-worded (or non–reversed-worded) items from the measure entirely. In the present study, we followed this recommendation and removed the non–reversed-worded items to measure EBT attitudes. This led to a salient one-factor model that still assessed a relatively broad range of attitudes related to EBTs, including asking about training burden associated with EBTs, the ability for EBTs to be tailored to individual client needs, the relative importance of clinical judgment, research-informed clinical decision making, EBT flexibility, and comorbidity coverage achieved by EBTs. It is however worth noting that the reliability of this modified measure fell just below the cutoff for good reliability. Reliability of the original MPAS was reported to be .80 (Borntrager et al., 2009), and so more research is needed to understand if there are ways to increase the reliability of the K-MPAS. For example, given that internal consistency is known to reduce as items are removed (as was the case in the present study), future work in this area may strongly consider replacing the eliminated items with items that query the same content domains (and perhaps other content domains as well) yet in the same wording direction as the items included in the scale. The scores from the K-MPAS nonetheless significantly predicted EBT knowledge and attitudes toward EBAs, and it was associated with significant differences between U.S. and Korean therapists, demonstrating the validity and utility of this modified scale.

Clinical and Research Implications

The K-MPAS developed in the present study has the potential for important clinical and research application—particularly with respect to helping disseminate EBTs in the Korean mental health system. Through administering the K-MPAS to members in the Korean mental health system, the nature of attitudes toward EBTs may be determined specific to different groups (e.g., clinic-based psychologists, hospital-based assessors, school-based counselors, etc.). This can inform which populations are in more need of trainings and education programs to increase their (positive) attitudes toward EBTs to support subsequent EBT implementation. In education contexts, the K-MPAS can also be used to measure and monitor the attitudes of graduate students and trainees as they progress through their training programs to ensure that positive attitudes are fostered and maintained. In research, the K-MPAS can also be used to identify predictors of positive and negative attitudes toward EBTs, complementing work that has already begun in the United States (e.g., Nakamura, Higa-McMillan, Okamura, & Shimabukuro, 2011).

The present study also has implications with respect to the development of attitude measures in both Eastern and Western contexts. Specifically, the inclusion of both reversed-worded and non–reversed-worded items in the K-MPAS contributed to structural problems, supporting the recommendations in the literature to include either reversed-worded or non–reversed-worded items, but not both. Furthermore, significant group differences in attitudes were found between the Korean and U.S. therapists only when based on the total score that did not include both reversed-worded and non–reversed-worded items. These results suggest that including both non–reversed-worded and reversed-worded items in a single scale may distort the total score.

Limitations

Despite the strengths of the current article—including being the first to examine the factor structure underlying the MPAS—there were limitations worth noting. First, the present study was based on self-reports of attitudes. Attitudes however may also be assessed via other paradigms, including the Implicit Association Task, which has been shown to be a powerful assessment paradigm to assess attitudes (Greenwald, McGhee, & Schwartz, 1998; Karpinski & Hilton, 2001). Another limitation in the present study was the relatively small number of school-based counselors in the Korean sample. This is noteworthy because counselors have begun to be placed in schools in Korea to provide mental health services to students. In the present study, we only obtained responses from 36 school counselors. Given the importance of this service provider population with respect to delivering effective evidence-based services in school settings in Korea, it will be important for future studies to collect data from more school counselors to better understand their attitudes and readiness to deliver EBTs to students. Last, although the U.S. and Korean samples differed significantly on the 5-item MPAS total score, but not on the 8-item MPAS total score (thus supporting differential performance between these two versions of the MPAS), their p values were not drastically different (i.e., .06 vs. .04). Some caution should be taken when interpreting these results. The overall set of psychometric indices however do support the 5-item version over the 8-item version, so this too should be considered when interpreting this differential pattern regarding the significant differences of the total scores across samples.

Despite these limitations and areas for future research, the present study found support for the K-MPAS as a well-performing measure to assess attitudes toward EBTs. The adoption and delivery of EBTs will be an important focus through the upcoming decades of dissemination research, and we hope the K-MPAS can be used to assess and understand how mental health providers think about EBTs when providing care in the Korean mental health system. The present study also conducted the first factor analytic investigation to understand the structure underlying the MPAS to measure attitudes toward EBTs. Based on the factor analytic results, we recommended removing the reversed-worded items and instead retain only the non–reversed-worded items to measure EBT attitudes. In both Eastern and Western contexts, there are several measures that also include both reversed-worded and non–reversed-worded items. As demonstrated in the present study, this issue should be closely examined to identify bias and distortions in scale scores resulting from such method effects. This is not a trivial matter, and not making the necessary adjustments and modifications to eliminate score contamination can lead to undesirable outcomes, such as not being able to detect differences between groups. There may be other problems associated with such oppositely worded items, and future studies should carefully consider these findings when pursuing scale development and scale refinement.

Footnotes

Appendix

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Aarons

G. A.

(2004). Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research, 6, 61-74.

Ahn

C. M.

Ebesutani

Kamphaus

(2014). A psychometric analysis and standardization of the behavior assessment system for children-2, self-report of personality, child version among a Korean sample. School Psychology Quarterly, 29, 198-212.

Ahn

C. H.

Shin

H. S.

(2012). Maternal psychological control and child depression: A mediation model. Korean Journal of Youth Studies, 19, 227-253.

Asukai

Saito

Tsuruta

Kishimoto

Nishikawa

(2010). Efficacy of exposure therapy for Japanese patients with posttraumatic stress disorder due to mixed traumatic events: A randomized controlled study. Journal of Traumatic Stress, 23, 744-750.

Ayers

C. R.

Sorrell

J. T.

Thorp

S. R.

Wetherell

J. L.

(2007). Evidence-based psychological treatments for late-life anxiety. Psychology and Aging, 22, 8-17.

Bagner

D. M.

Storch

E. A.

Roberti

J. W.

(2004). A factor analytic study of the loneliness and social dissatisfaction scale in a sample of African-American and Hispanic-American children. Child Psychiatry & Human Development, 34, 237-250.

Bagozzi

R. P.

(1993). Assessing construct validity in personality research: Applications to measures of self-esteem. Journal of Research in Personality, 27, 49-87.

Bagozzi

R. P.

Heatherton

(1994). A general approach to representing multifaceted personality constructs: Application to state self-esteem. Structural Equation Modeling, 1, 35-67.

Baumann

B. L.

Kolko

D. J.

Collins

Herschell

A. D.

(2006). Understanding practitioners’ characteristics and perspectives prior to the dissemination of an evidence-based intervention. Child Abuse & Neglect, 30, 771-787.

10.

Bearman

S. K.

Wadkins

Bailin

Doctoroff

(2015). Pre-practicum training in professional psychology to close the research–practice gap: Changing attitudes toward evidence-based practice. Training and Education in Professional Psychology, 9, 13-20.

11.

Beutler

(2002). The dodo bird is extinct. Clinical Psychology: Science and Practice, 9, 30-34.

12.

Borntrager

Chorpita

Higa-McMillan

Weisz

(2009). Provider attitudes toward evidence-based practices: Are the concerns with the evidence or with the manuals? Psychiatric Services, 60, 677-681.

13.

Brislin

(1970). Back-translation for cross-cultural research. Journal of Cross-Cultural Psychology, 1, 185-216.

14.

Brown

(2003). Confirmatory factor analysis of the Penn State Worry Questionnaire: Multiple factors or method effects? Behaviour Research and Therapy, 41, 1411-1426.

15.

Brown

(2006). Confirmatory factor analysis for applied research. New York, NY: Guilford Press.

16.

Browne

Cudeck

(1993). Alternative ways of assessing model fit. In Bollen

K. A.

Long

J. S.

(Eds.), Testing structural equation models (pp. 136-159). Newbury Park, CA: Sage.

17.

Carr

(2009). The effectiveness of family therapy and systemic interventions for adult-focused problems. Journal of Family Therapy, 31, 46-74.

18.

Chorpita

B. F.

Daleiden

E. L.

Ebesutani

Young

Becker

K. D.

Nakamura

B. J.

. . . Starace

(2011). Evidence-based treatments for children and adolescents: An updated review of indicators of efficacy and effectiveness. Clinical Psychology: Science and Practice, 18, 154-172.

19.

Chorpita

B. F.

Daleiden

E. L.

Weisz

(2005). Identifying and selecting the common elements of evidence based interventions: A distillation and matching model. Mental Health Services Research, 7, 5-20.

20.

Clark

D. M.

Layard

Smithies

Richard

D. A.

Suckling

Wright

(2009). Improving access to psychological therapy: Initial evaluation of two UK demonstration sites. Behaviour Research and Therapy, 47, 910-920.

21.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

22.

Compton

Burns

Egger

Robertson

(2002). Review of the evidence base for treatment of childhood psychopathology: Internalizing disorders. Journal of Consulting and Clinical Psychology, 70, 1240-1266.

23.

Corwyn

R. F.

(2000). The factor structure of global self-esteem among adolescents and adults. Journal of Research in Personality, 34, 357-379.

24.

Costello

A. B.

Osborne

J. W.

(2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10. Retrieved from http://pareonline.net/getvn.asp?v=10&n=7

25.

Cottrell

Boston

(2002). Practitioner review: The effectiveness of systemic family therapy for children and adolescents. Journal of Child Psychology and Psychiatry, 43, 573-586.

26.

Crits-Christoph

Baranackie

Kurcias

J. S.

Beck

Carroll

Perry

(1991). Meta-analysis of therapist effects in psychotherapy outcome studies. Psychotherapy Research, 1, 81-91.

27.

Cuijpers

van Straten

Andersson

Van Oppen

(2008). Psychotherapy for depression in adults: A meta-analysis of comparative outcome studies. Journal of Consulting and Clinical Psychology, 76, 909-922.

28.

Ebesutani

(2014). The validation of the Korean version of the Knowledge of Evidence-Based Services Questionnaire. Korean Journal of Health Psychology, 19, 119-146.

29.

Ebesutani

Chung

(2014). Attitudes Toward Standardized Assessment Scales–Korean Version: Scale development and psychometric investigation controlling for wording “method effects”. Manuscript submitted for publication.

30.

Ebesutani

Drescher

Reise

Heiden

Hight

Damon

Young

(2012b). The Loneliness Questionnaire-Short Version: An evaluation of reverse-worded and non-reverse-worded items via item response theory. Journal of Personality Assessment, 94, 427-437.

31.

Ebesutani

C. K.

Helmi

Fierstein

Taghizadeh

Chorpita

(2016). A pilot study of modular cognitive-behavioral therapy and cognitive-behavioral hypnotherapy for treating anxiety in Iranian girls. International Journal of Cognitive Therapy. Advance online publication. doi:10.1521/ijct_2016_09_01

32.

Ebesutani

C. K.

Shin

(2014). Knowledge, attitudes, and usage of evidence-based assessment and treatment practices in the Korean mental health system: Current status and future directions. Korean Journal of Clinical Psychology, 33, 875-901.

33.

Ekers

Richards

Gilbody

(2008). A metaanalysis of randomized trials of behavioural treatment of depression. Psychological Medicine, 38, 611-623.

34.

Flora

Curran

(2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9, 466-491.

35.

Fresco

D. M.

Heimberg

R. G.

Mennin

D. S.

Turk

C. L.

(2002). Confirmatory factor analysis of the Penn State Worry Questionnaire. Behaviour Research and Therapy, 40, 313-323.

36.

Greenwald

McGhee

Schwartz

(1998). Measuring individual differences in implicit cognition: The implicit association test. Journal of Personality and Social Psychology, 74, 1464-1480.

37.

Hinton

D. E.

Chhean

Pich

Safren

Hofmann

Pollack

(2005). A randomized controlled trial of cognitive-behavior therapy for Cambodian refugees with treatment-resistant PTSD and panic attacks: A cross-over design. Journal of Traumatic Stress, 18, 617-629.

38.

Hinton

D. E.

Pham

Tran

Safren

S. A.

Otto

M. W.

Pollack

M. H.

(2004). CBT for Vietnamese refugees with treatment-resistant PTSD and panic attacks: A pilot study. Journal of Traumatic Stress, 17, 429-433.

39.

Hofmann

S. G.

Smits

J. A. J.

(2008). Cognitive behavioral therapy for adult anxiety disorders: A meta-analysis of randomized placebo-controlled trials. Journal of Clinical Psychiatry, 69, 621-632.

40.

L. T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.

41.

Jaberghaderi

Greenwald

Rubin

Zand

Dolatabadi

(2004). A comparison of CBT and EMDR for sexually abused Iranian girls. Clinical Psychology & Psychotherapy, 11, 358-368.

42.

Jensen-Doss

Hawley

K. M.

(2010). Understanding barriers to evidence-based assessment: Clinician attitudes toward standardized assessment tools. Journal of Clinical Child & Adolescent Psychology, 39, 885-896.

43.

Jöreskog

K. G.

(1994). On the estimation of polychoric correlations and their asymptotic covariance matrix. Psychometrika, 59, 381-389.

44.

Karpinski

Hilton

J. L.

(2001). Attitudes and the implicit association test. Journal of Personality and Social Psychology, 81, 774-788.

45.

Kazdin

A. E.

Weisz

J. R.

(1998). Identifying and developing empirically supported child and adolescent treatments. Journal of Consulting and Clinical Psychology, 66, 19-36.

46.

Kazdin

A. E.

Weisz

J. R.

(2003). Evidence-based psychotherapies for children and adolescents. New York, NY: Guilford Press.

47.

Kim

H. S.

(2003). The present state of a suicidal act of the young people and confrontation a plan-centering around a middle school boy (Unpublished master’s thesis). Keimyung University, Daegu, South Korea.

48.

Kline

R. B.

(2005). Principles and practice of structural equation modeling (2nd ed.). New York, NY: Guilford Press.

49.

Langley

Bergman

McCracken

Piacentini

(2004). Impairment in childhood anxiety disorders: Preliminary examination of the child anxiety impact scale-parent version. Journal of Child and Adolescent Psychopharmacology, 14, 105-114.

50.

Leichsenring

(2001). Comparative effects of short term psychodynamic psychotherapy and cognitive behavioural therapy in depression. Clinical Psychology Review, 21, 401-419.

51.

Lewis

C. C.

Simons

A. D.

(2011). A pilot study disseminating cognitive behavioral therapy for depression: Therapist factors and perceptions of barriers to implementation. Administration and Policy in Mental Health and Mental Health Services Research, 38, 324-334.

52.

Lilienfeld

S. O.

Ritschel

L. A.

Lynn

S. J.

Cautin

R. L.

Latzman

R. D.

(2013). Why many clinical psychologists are resistant to evidence-based practice: Root causes and constructive remedies. Clinical Psychology Review, 33, 883-900.

53.

Marsh

H. W.

(1996). Positive and negative global self-esteem: A substantively meaningful distinction or artifactors? Journal of Personality and Social Psychology, 70, 810-819.

54.

Marsh

H. W.

Grayson

(1995). Latent variable models of multitrait-multimethod data. In Hoyle

R. H.

(Ed.), Structural equation modeling: Concepts, issues and application (pp. 177-198). Thousand Oaks, CA: Sage.

55.

Marsh

H. W.

Scalas

L. F.

Nagengast

(2010). Longitudinal tEBTs of competing factor structures for the Rosenberg Self-Esteem Scale: Traits, ephemeral artifacts, and stable response styles. Psychological Assessment, 22, 366-381.

56.

McDonald

R. P.

(1999). Test theory: A unified approach. Mahwah, NJ: Lawrence Erlbaum.

57.

McHugh

Barlow

D. H.

(2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist, 65, 73-84.

58.

Muthén

du Toit

S. H. C.

Spisic

(1997). Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes. Retrieved from https://www.statmodel.com/download/Article_075.pdf

59.

Muthén

L. K.

Muthén

B. O.

(2010). Mplus user’s guide. Los Angeles, CA: Muthén & Muthén.

60.

Nakamura

Higa-McMillan

Okamura

Shimabukuro

(2011). Knowledge of and attitudes towards evidence-based practices in community child mental health practitioners. Administration and Policy in Mental Health and Mental Health Services Research, 38, 287-300.

61.

Norton

Price

(2007). A meta-analytic review of adult cognitive-behavioural treatment outcome across the anxiety disorders. Journal of Nervous and Mental Diseases, 195, 521-531.

62.

Olsson

(1979). Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika, 44, 443-460.

63.

Ost

L-G.

(2008). Cognitive behaviour therapy for anxiety disorders: 40 years of progress. Nordic Journal of Psychiatry, 62, 5-10.

64.

Otto

Hinton

D. E.

Korbly

N. B.

Chea

Gershuny

Pollack

M. H.

(2003). Treatment of pharmacotherapy-refractory posttraumatic stress disorder among Cambodian refugees: A pilot study of combination treatment with cognitive-behavior therapy vs sertraline alone. Behavior Research and Therapy, 41, 1271-1276.

65.

Park

S. C.

H. S.

D. H.

Jung

S. A.

K. S.

(2014). Evidence-based, non-pharmacological treatment guideline for depression in Korea. Journal of Korean Medical Science, 29, 12-22.

66.

Ponniah

Hollon

(2008). Empirically supported psychological interventions for social phobia in adults: A qualitative review of randomized controlled trials. Psychological Medicine, 38, 3-14.

67.

Rapaport

M. H.

Clary

Fayyad

Endicott

(2005). Quality-of-life impairment in depressive and anxiety disorders. American Journal of Psychiatry, 162, 1171-1178.

68.

Rogers

E. M.

(2003). Diffusion of innovations (5th ed.). New York, NY: Free Press.

69.

Shedler

(2010). The efficacy of psychodynamic psychotherapy. American Psychologist, 65, 98-109.

70.

Siev

Chambless

D. L.

(2007). Specificity of treatment effects: Cognitive therapy and relaxation for generalized anxiety and panic. Journal of Consulting and Clinical Psychology, 75, 513-522.

71.

Strupp

H. H.

Anderson

(1997). On the limitations of therapy manuals. Clinical Psychology: Science and Practice, 4, 76-82.

72.

Stumpf

Higa-McMillan

Chorpita

B. F.

(2009). Implementation of evidence-based services for youth: Assessing provider knowledge. Behavior Modification, 33, 48-65.

73.

Tomas

J. M.

Hontangas

P. M.

Oliver

(2000). Linear confirmatory factor models to evaluate multitrait-multimethod matrices: The effects of number of indicators and correlation among methods. Multivariate Behavioral Research, 35, 469-499.

74.

van Sonderen

Sanderman

Coyne

J. C.

(2013). Ineffectiveness of Reverse Wording of Questionnaire Items: Let’s learn from cows in the rain. PLoS One, 8(7), e68967.

75.

Weersing

V. R.

Weisz

J. R.

(2002). Community clinic treatment of depressed youth: Benchmarking usual care against CBT clinical trials. Journal of Consulting and Clinical Psychology, 70, 299-310.

76.

Weisz

J. R.

Jensen-Doss

Hawley

K. M.

(2006). Evidence-based youth psychotherapies versus usual clinical care: A meta-analysis of direct comparisons. American Psychologist, 61, 671-689.

77.

Weisz

J. R.

Kuppens

Eckshtain

Ugueto

A. M.

Hawley

K. M.

Jensen-Doss

(2013). Performance of evidence-based youth psychotherapies compared with usual clinical care: A multilevel meta-analysis. JAMA Psychiatry, 70, 750-761.

78.

Westen

Novotny

C. M.

Thompson-Brenner

(2004). The empirical status of empirically supported psychotherapies: Assumptions, findings, and reporting in controlled clinical trials. Psychological Bulletin, 130, 631-663.

79.

Zima

B. T.

Hurlburt

M. S.

Knapp

Ladd

Tang

Duan

. . . Wells

K. B.

(2005). Quality of publicly-funded outpatient specialty mental health care for common childhood psychiatric disorders in California. Journal of the American Academy of Child and Adolescent Psychiatry, 44, 130-144.