Abstract
Objectives: There is a growing literature indicating that organizational and individual worker-level factors affect decisions about whether or not empirically supported treatments (ESTs) are adopted within health care agencies. The purpose of this pilot study is to further investigate and measure worker’s attitudes within a community organization. Method: A small organization participated in the study due to their diversity in services offered. Of the 92 workers eligible for participation in the study, 66 (72%) completed the Evidence-Based Practice Attitude scale survey. Results: Multivariate analyses revealed that female workers scored higher on both Openness and total score; workers with nursing, education, or psychology majors scored lower than workers with other (excluding social work) majors on both Divergence and total score; and that older workers scored higher on Divergence. Conclusion: Although small, this study identifies individual characteristics that are most likely to fit the profile of an EST adopter.
Background
Although there have been efforts to advance the use of empirically supported treatments (ESTs) into community-based mental health organizations (CBMHO) clinical practices (Aarons & Sawitzky, 2006; Abrahamson, 2001; Burns, 2003; Essock et al., 2003; Glisson, 2002; Goldman et al., 2001; Ringeisen & Hoagwood, 2002), the limited successes of dissemination and poor implementation of efficacious treatments within these organizations are well documented (Hoagwood, Burnas, Kiser, Ringeisen, & Schoenwald, 2001; Weisz & Jensen, 1999). The National Institutes of Health’s Bridging Science and Service (National Institutes of Health, 1999) indicated that clinical effectiveness and utility of any new treatments are just as important as efficacy issues in controlled clinical trials when evaluating treatment strategies. There are ongoing efforts to study and understand the limited success of implementing and adopting ESTs throughout CBMHOs which this study investigated.
There is a growing literature suggesting that CBMHO’s worker characteristics affect decisions about whether or not ESTs are adopted and implemented within health care agencies (Aarons, 2005; Glisson & Hemmelgarn, 1998; Glisson & James, 2002; Hemmelgarn, Glisson, & Dukes, 2001; Hemmelgarn, Glisson, & James, 2006). Worker’s years of work experience has been investigated (Aarons, 2004), educational attainment (Aarons, 2004; Ogborne, Wild, Braun, & Newton-Taylor, 1998; Stahmer & Aarons, 2009), educational discipline (Aarons, 2004; Stahmer & Aarons, 2009), and students completing an internship (Aarons, 2004; Garland, Kruse, & Aarons, 2003), all of which have seem to shape attitudes toward the use of ESTs.
Individual-Level Barriers
It also must be noted that there is a developing literature focusing on worker attitudes along with their demographics characteristics. Adoption of ESTs into practice settings may be hampered or made easier by providers’ attitudes toward that new treatments, interventions, and practices. A quick measure of workers’ attitudes toward adopting ESTs was developed and attitudes were investigates related to a set of individual differences (Aarons, 2004; Aarons & Sawitzky, 2006; Garland et al., 2003; Pignotti & Thyer, 2009; Stahmer & Aarons, 2009). According to Aarons (2004), worker’s attitudes can be reliably measured and vary in relation to individual differences. Any EST implementation plans in the workplace should include provider attitudes. These attitudes have the potential to improve the process and effectiveness of dissemination efforts (Aarons, 2004).
Worker Demographics
While the primary purpose of Aarons’ (2004) study was to develop a brief EST attitude measure, there were other hypotheses tested. Of particular concern for this study was to further test the hypothesis that participants with higher educational levels, primary discipline, and professional status would have a more open attitude toward EST adoption. The original study found no differences in attitudes toward adoption of ESTs across disciplines (e.g., social work, Marriage and Family Therapy [MFT], psychology, psychiatry, and other). There were however, individual differences across higher educational levels and professional status (Aarons, 2004; Aarons & Sawitsky, 2006).
Other studies measuring workers’ attitudes about ESTs have had mixed outcomes. For instance, Pignotti and Thyer (2009) found significant differences related to subject’s age and years of experience which is contrary to earlier studies (Aarons, 2004). There also seems to be inconsistent findings between studies pertaining to subjects’ educational attainment. Those studies reporting educational attainment (Aarons, 2004; Loy, 1968; Ogborne, et al., 1998) found that higher degreed workers reported more positive attitudes compared to those with less education. However, a more recent study found differences between workers’ attitudes although they had equal levels of educational attainment (Stahmer & Aarons, 2009). Aarons’ original 2004 EBPAS validation study resulted in no significant differences between worker’s educational discipline and EST attitudes. However, in a later study, (Stahmer & Aarons, 2009), attitudinal differences were found between workers with different educational disciplines.
Because this is a developing, broad area of study, the lack of direct evidence pointing toward primary EST barriers requires a staged approach investigation. This study builds upon the dearth of knowledge about individual worker-level characteristics that have shown to be important issues when attempting to implement and adopt ESTs throughout CBMHOs. The overall purpose of this pilot study is to further investigate and measure individual characteristics such as worker demographics and individual attitudes toward EST within a CBMHO. This will further contribute to the growing body of knowledge regarding individual worker-level factors and EST adoption.
Method
The Community-Based Mental Health Organization
A small CBMHO was approached to participate in the study due to their diversity in services offered, such as addiction, mental health, and children and family services. At the time of the study the CBMHO employed 134 persons at eight locations, one of which houses administrative worker only, in the Buffalo, NY metropolitan area. Ninety two employees provide direct client services with the remaining 42 employees making up the support or administrative worker. The CBMHO provides a range of health and mental health services such as addiction counseling, mental health assessments and treatments, and dual disorder services all with multiple funding streams.
Worker Inclusion Criteria
The subject population (a) were all persons 18 years or older, (b) employed by the CBMHO during the baseline assessment, (c) could read and understand English sufficiently to complete informed consents and data collection forms, and (d) had direct clinical contact with clients, and (e) agreed to voluntary participate. Employees having dual responsibilities, such as program oversight along with carrying an active client caseload, were invited to participate. However, employees having only managerial responsibilities were excluded from the study.
Participants
Of the 92 workers eligible for participation in the study, 66 (72%) completed the survey. The 66 respondents ranged in age from 25 to 71 (mean [M] = 45, median [Mdn] = 47, standard deviation [SD] = 12) had worked full time in a human service position from 2 to 40 years (M = 16, Mdn = 14, SD = 10), and had worked in their present position from less than a year to 29 years (M = 8, Mdn = 6, SD = 8). The majority of workers had either a bachelor (35%) or a master degree (39%). Although a majority of workers had a social work background (43%), other workers had education (6%), nursing (8%), or psychology (9%) backgrounds. However, a third (34%) described the educational background as “other.” Over three-quarters (82%) identified themselves as “White” and nearly three-quarters (71%) were female. Aggregate demographic data for the clinical worker were not available from the agency.
Measures
The measures used in this study consisted of the Evidence-Based Practice Attitude scale (EBPAS: Aarons, 2004) and the worker demographics questionnaire from the organizational social context (Glisson, 2002). The EBPAS consists of 15 items assessing four dimensions of attitudes toward adoption of evidence-based practices. A 5-point response format (0 = not at all, 1 = to a slight extent, 2 = to a moderate extent, 3 = to a great extent, and 4 = to a very great extent) is used to respond to each item. Scale scores were computed as the mean of items comprising the scale. The 4 scales (and their α values for these data) are as follows. Requirements is a 3-item scale (α = .87) that assesses the likelihood the worker would adopt a new EST if it were required. Appeal is a 4-item scale (α = .80) that measures the likelihood the worker would adopt a new EST if colleagues were happy with it or it was intuitively appealing, made sense, and could be used correctly. Openness is a 4-item scale (α = .80) that assesses the “Openness” of the worker to consider trying or actually adopting new interventions. Divergence is a 4-item scale (α = .63) that assesses the worker’s assessment of the clinical value of research-based interventions versus clinical experience. A higher score indicates “more” of the scale name, except for Divergence where a higher score indicates a valuing of clinical experience and knowledge over research-derived knowledge. In addition, a total (mean) score (α = .78) was computed for the 15 items in the measure. The α (internal consistency) values found here are equal to or, in the case of Divergence, slightly better than those reported by Aarons (2004).
The worker demographic questionnaire consisted of eight questions: gender, race, Hispanic/Latino ethnicity (not used), level of education completed, educational major of highest degree, age, full time years of experience in human services work, and years worked in present agency.
Procedures
Participants were invited to participate in the study during an agency-wide appreciation day retreat. There was an existing research relationship with the CBMHO and the EBPAS was added during a previously scheduled longitudinal survey completion date. The EBPAS was completed along with other scales regarding organizational factors. The completed surveys were entered into SPSS, scored, and processed by investigator. The study was reviewed and approved by the university’s Institutional Review Board (IRB). All authors certify responsibility for this study and article.
Results
Although we used statistical tests to test the significance levels of relationships, we emphasize observed effect sizes since the study population is that of all eligible employees in this one agency. Table 1, then, presents the means and SDs for the EBPAS scores for the total sample and by demographic group.
Summary Statistics for EBPAS Scale by Selected Worker Demographic Characteristics
Note. EBPAS = Evidence-Based Practice Attitude scale. Means with different subscripts differ at p < .05 by a Tukey honest significant difference test.
The analysis of variance (ANOVA) results showed that females had higher scores than males only on total score, t(54) = 2.20, p = .032; 90% confidence interval (CI) = [0.07, 0.49]; d = .65. Due to the small numbers of workers with an educational major of nursing (n = 3), education (n = 4), or psychology (n = 5), these three majors were combined into a single category (NEP) for the analysis, which has the unfortunate effect of obscuring any major-specific effects of these majors on attitudes toward ESTs. Only the ANOVA for Divergence, F(2, 50) = 8.84, p < .001, was significant. Post hoc tests for Divergence found that Nursing, Education, and Psychology (NEP) majors had significantly higher Divergence scores than social work majors (M1 − M2 = 0.77, p < .002, 90% CI = [0.42, 1.12], d = 1.20) or “other” majors (M1 − M3 = 0.87, p < .001, 90% CI = [0.51, 1.24], d = 1.46) by a Tukey Honestly Significant Difference (HSD) test. Thus, workers with educational, nursing, or psychology majors placed a significantly higher value on clinical knowledge than did either social work or other majors.
To analyze worker-educational level, high school graduates (n = 3) were combined with associates degree (n = 7) worker and PhD worker (n = 1) were combined with master’s degree (n = 26) worker. However, none of the ANOVAs were significant.
Correlations were computed between age, years of full-time human service work, years in the present agency, and the EBPAS. Table 2 reports these correlations. As shown there, the correlations (absolute value) between age and the scales ranged between .04 (Appeal) and .16 (Divergence). Years of full-time human service work had similar correlations (absolute value) with Openness, Divergence, and total (.15 to .17) but smaller correlations with requirements and Appeal. Finally, correlations with years in this agency ranged between .16 (Divergence) to .09 to .07 (Requirements, Appeal, and total) to a low of .01 with Openness. None of the correlations, however, were significant.
Correlations Between Age, Years of Full-Time Human Service Work, Years in Current Agency, and EBPAS (N = 58)
Note. EBPAS = Evidence-Based Practice Attitude scale.
Discussion and Applications to Social Work
This study investigated the relationship between worker demographic characteristics and scores on the EBPAS measure in a small CBMHO. The results showed that of the worker demographic characteristics examined, gender was related significantly to total score at the medium effect size level, and educational major was related significantly to Divergence at the large effect size level.
Several limitations should be kept in mind with respect to this study. First, the results are based on the views of a large proportion of the clinical service staff from one relatively small CBMHO offering a range of programs to children, adults, and families. Although this agency and its workers may be similar to other CBMHOs of its general size and with a similar list of services and service population, the similarities (and differences) cannot be known. Thus, the results may not be generalizable beyond this agency. Although, a large majority of workers did elect to participate, a sizable percentage did not and nonparticipation may not have been random. A sample of service workers from a large sample of organizations might yield different results. While some of the relationships were moderate to large in magnitude, particularly for Divergence, the smallness of the analysis sample means large sampling error and therefore, wide CIs. In addition to sampling error, small cell sizes contribute to imprecision in estimating correlations, particularly point biserials, since one additional person would make a relatively larger contribution to the correlation than in a larger sample. That said, the relationships found here parallel, to a fairly large extent, those reported in other studies relating worker characteristics to EST attitudes, thus giving us confidence in these results.
It is possible, even likely, that attitudes to ESTs are shaped by personal background, education and employment backgrounds, and working experiences, as well as by organizational characteristics such as the services or programs provided, size, and cultural beliefs about service innovation. Although this study cannot address organizational-level characteristics—indeed, to our knowledge no study yet has, this study, as well as others like it, shed light on the relationships between attitudes toward evidence-based practices and worker demographics.
The impetus for the adoption of ESTs typically comes from agency administrators in response to a number of internal and external factors. These can be factors related to the agency’s own internal drive to continuously improve the quality and outcomes of services being provided, as well as to external forces including funders who are increasingly clear in their expectations that agencies they contract with demonstrate they have adopted and are delivering EST-based services. This can be a direct expectation or indirect by virtue of the fact that funders are not only expecting agencies to document their outcomes, but funders, faced with shrinking budgets, are increasingly making decisions as to which agencies they will continue to fund based on who is achieving the highest outcomes for the dollars provided. Many administrators understand that the adoption of ESTs in their agencies have the potential to improve agency outcomes significantly over their current non-EST routine practices. In either case, the challenge for administrators is to overcome the barriers at an organizational and individual level that impede the adoption of ESTs. The results of this study give limited, but important, insights for agency administrators to assist them with developing effective strategies for promoting the adoption of specific ESTs within their agencies.
In addressing some organizational-level barriers, the results of this study would suggest that a successful strategy for implementing a selected EST within an agency should include: clarity for all workers from agency administration that the adoption of an EST may not be an elective effort but rather an organizational requirement from the funding source; proactively utilizing Aarons EBPAS to recruit and identify like-minded existing workers who show evidence of having higher scores on the Appeal and Openness scales and a lower score on the Divergence scale, and who have a higher scores on understanding of organizational requirements to conduct within the agency a small scale trial of implementing a EST; support this voluntary team by designing preparatory practice sessions and early trials to create psychological safety within the above teams that results in encouraging new team behaviors; and promote shared meaning and process improvement through reflective practices.
Further, this study’s results indicate that individual-level characteristics that are most likely to fit the profile of an effective EST small scale trial team member are: female and those who have a degree in other than education, psychology, or nursing. As small scale trials of an EST have success in their implementation, it provides the opportunity to take the lessons learned in the implementation process and successes experienced by trial team members and apply them to broadening and deepening the EST to other workers in the agency who may be individually less amenable to EST adoption as reflected in their EBPAS scores and behaviors. Successful small scale trial volunteers are in an excellent position to be both positive opinion leaders promoting adoption of ESTs and potential trainers within the agency.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.