Abstract
BACKGROUND:
Vocational rehabilitation for people with severe mental illnesses has reached a critical stage in its development of evidence-based practices. A recent meta-analysis has suggested that good program quality measured by fidelity scales is necessary but not sufficient to achieve good employment outcomes. It is likely therefore that there are other aspects of program implementation quality that are not addressed by the current scales.
OBJECTIVE:
The aim of this study was to identify new dimensions of program quality potentially applicable to improving the effectiveness of evidence-based vocational rehabilitation.
METHODS:
A review and synthesis of the theoretical and empirical literature in the broader science of psycho-social program implementation was conducted to identify the core dimensions which could be applied to evidence-based vocational rehabilitation.
RESULTS:
Twenty reviews and four single studies identified the common dimensions of a generic implementation framework. Empirical findings from eight of these, and 19 additional papers built a conceptual framework for improving evidence-based practices in vocational rehabilitation. The resulting framework consists of nine dimensions of implementation quality and twenty-three contextual factors.
CONCLUSIONS:
This conceptual framework will assist program funders, researchers, practitioners, and policymakers to consider a wider range of program influences to advance the science and practice of vocational rehabilitation for people with severe mental illness.
Keywords
Introduction
The science of vocational rehabilitation for people with severe mental illnesses has advanced significantly over the past thirty years and has reached a critical stage in the development of evidence-based practices. There is now a widely agreed set of principles and practices, known collectively as Individual Placement and Support (IPS) which has proven to be more effective than other approaches to vocational rehabilitation (Drake, Bond, and Becker, 2012, p. 59). This IPS approach has been adopted and implemented across four continents and for different populations (Drake and Bond, 2014). These different populations have included young adults and older adults, people with a homelessness history, post-traumatic stress disorder, substance-use disorders, and involvement with the criminal justice system (Drake and Bond, 2014).
The evidence of IPS effectiveness includes 24 randomized controlled trials (Drake and Bond, 2017), 12 systematic reviews (Marshall et al., 2014) and a Cochrane review (Kinoshita et al., 2013). This body of evidence has consistently found IPS is more effective than alternative approaches to vocational rehabilitation in terms of the numbers of participants commencing competitive employment. A recent systematic review and meta-analysis of 17 randomized controlled trials comparing IPS to alternative types of vocational rehabilitation, found IPS increased the odds of participants commencing employment by more than two times (pooled risk ratio 2.40, 95% CI 1.99–2.90) (Modini et al., 2016). This review covered IPS implementation in 10 countries.
Despite this body of evidence there is still room for improvement (Drake et al., 2012, p. 4). For instance, a loss of effectiveness has been observed for IPS implementations outside the United States of America (USA). This loss of effectiveness has been attributed to reduced adherence to the original principles and practices of the program (Bond, Drake, and Becker, 2012). Faithful adherence to the way a program was originally delivered in research trials, is referred to as program fidelity (Stith et al., 2006). Currently the main strategy advocated to improve outcomes from IPS program implementations is to increase fidelity as measured by the latest standardized fidelity scale (Drake et al., 2012, p. 112). However, evidence from a recent systematic review and meta-analysis suggests that this strategy may not be adequate to improve intended program outcomes. This is because good program fidelity is necessary, but not sufficient, for good employment outcomes to be achieved (Lockett, Waghorn, Kydd, and Chant, 2016). Hence it is possible that other aspects of program implementation quality, other than fidelity, influence program performance. If these other aspects can be identified, and if causally linked to program outcomes, attention to these could further develop vocational rehabilitation practices for people with severe mental illnesses.
Identifying factors which influence the quality of program implementation is a problem that is not unique to vocational rehabilitation. A research to practice effectiveness gap has been noted in the literature concerned with improving the effectiveness of a wide range of psycho-social programs. In fact, the discipline of implementation science emerged in response to this widely reported type of problem (Nilsen, 2015). The premise underpinning implementation science is that improving program performance first requires improving the quality of program implementation (Durlak and DuPre, 2008). Durlak (2015) has argued that “…it is not evidence-based programs that are effective, but it is well-implemented evidence-based programs that are effective” (Durlak, 2015, p. 1124). This implies the need for a measure of program implementation quality, coupled with an understanding of the wider ecological factors which influence the quality of implementation in a particular context (Damschroder et al., 2009). Such a wider conceptual framework would assist researchers, practitioners, and policymakers to systematically map factors outside fidelity that might influence program outcomes (Menear and Briand, 2014). It could also help to identify which implementation factors appear to be influential across different settings, or ones that are unique to cultures or contexts (Roe, 2017).
The purpose of this literature review and evidence synthesis was to construct a broader conceptual framework for improving the implementation of evidence-based practices in vocational rehabilitation for people with severe mental illnesses. This involved examining the broader implementation science literature to identify factors besides fidelity, influencing the quality of health and social science program implementation. The intention was to highlight the most promising of these as potential candidates for enhancing the effectiveness of psychiatric vocational rehabilitation programs.
Methods
The literature review and evidence synthesis was conducted in three stages. The first stage aimed to identify the common dimensions of implementation quality and contextual factors known to influence program performance, in order to develop a generic conceptual framework which could then be applied to vocational rehabilitation. In stage two, the literature was revisited, along with literature on IPS implementation, to explore the strength of the evidence for all the identified dimensions of implementation and contextual factors. The third stage was to use the synthesized evidence to construct a conceptual implementation framework for vocational rehabilitation, for people with severe mental illnesses in particular.
Literature review: Stage one
In stage one a broad approach was taken exploring the implementation science literature from across the health and social sciences. To manage the volume, papers were included in the search if they were published in English in a peer reviewed journal between 1997 and 2016.
The first author undertook searches of Medline, PubMed, PsycINFO and the Cochrane library, using the following search terms combined with the Boolean operators ‘OR’ and ‘AND’; implement*, evidence-based practice*, evidence-based treatment, evidence-based medicine, vocational rehabilitation, mental illness, mental health, review. Reference lists of selected articles were also searched for other articles which appeared relevant to the review. In addition, contents pages of two key journals were searched: the Journal of Vocational Rehabilitation, and Implementation Science.
Papers of any study design were permitted (e.g. cross-sectional survey, longitudinal studies, randomized controlled trials) and were not restricted by quantitative or qualitative emphasis. The focus of this first stage was primarily on systematic reviews or meta-analyses. Single studies were included if particularly relevant, or if they were missed by reviews. The literature search was considered to be complete when data saturation was reached and additional publications did not contribute new information to the dimensions and contextual factors identified.
This first stage of the review aimed to broaden out the scope of existing implementation frameworks for vocational rehabilitation. The included papers were classified as being conceptual, or based on empirical evidence, or both. The papers were examined for all the factors identified as influencing implementation quality and intervention performance. Factors were included if they were found at least twice in different studies, no other criteria were applied. The literature examined in this first stage was used to identify a set of common dimensions of quality and contextual factors in a generic conceptual framework for program implementation.
Literature review: Stage two
In stage two, the literature was revisited to explore the strength of the evidence for each of the identified dimensions of implementation and contextual factors. The focus in this second stage was on empirical papers identifying influences on program performance and the performance of evidence-based practices in vocational rehabilitation, for people with severe mental illness in particular. For this reason, published papers from IPS implementation studies were also examined for possible influences on program performance, outside those measured by the current fidelity scales. To be included in this second phase of the review, studies needed to report empirical evidence for a direct relationship between a dimension of quality, or contextual factor in the implementation framework, and a measure of program performance. Given the focus of this review to identify influences on program performance other than fidelity, studies which only examined the influence of fidelity were excluded.
Each of the included empirical papers was re-examined and the evidence in each paper mapped against one or more dimension of quality or contextual factor from the generic conceptual framework. Each paper was also rated according to its target relevance to vocational rehabilitation for people with severe mental illness. Papers from vocational rehabilitation or general psychiatry were rated A. Papers from the general health sector but outside psychiatry, were rated B. Papers outside the health sector were rated C. For example, a paper showing evidence of an association between staff experience and program performance in illness-prevention programs, was mapped against the contextual factor in the generic framework staff expertise, and given a relevance rating of B.
Mapping the evidence against the generic conceptual framework was an iterative process. It involved a gradual refinement and re-naming of some of the dimensions of quality and contextual factors for clarity, and applicability to vocational rehabilitation. Re-naming was informed by established evidence-based practices in vocational rehabilitation.
Once this mapping of all the included papers was completed, each dimension of quality and each contextual factor was classified according to whether it was a direct or an indirect influence. Where there was no evidence, or there was only one study with evidence of a direct influence on program performance, the relationship was classified as indirect. Where evidence from two or more studies converged, this was classified as a direct influence.
Literature review: Stage three
The third stage involved rating each dimension of quality for strength of evidence and feasibility to measure. This rating was based on the identified evidence, examples of measures of the dimension or contextual factor used in the empirical papers, and the first author’s judgements about how the phenomenon could be measured. The highest rating was two: indicating both good evidence of a direct influence on program performance, and the dimension was considered sufficiently operationalized to be feasible to measure. Good evidence was defined as three or more studies from the vocational rehabilitation or general psychiatry literature providing evidence of a direct influence. The next highest rating was one: limited evidence of a direct influence, but the dimension was considered sufficiently operationalized to be feasible to measure. Limited evidence was defined as less than three studies from the vocational rehabilitation or general psychiatry literature providing evidence of a direct influence. The lowest rating was zero: limited evidence for a direct influence, and the dimension appeared insufficiently operationalized and therefore difficult to measure.
Once rated, the direct influences were grouped together as new dimensions of quality, and the indirect influences were grouped as contextual factors in the conceptual framework for vocational rehabilitation.
Results
A generic conceptual implementation framework to enhance program performance
Twenty-four studies were retained in the first phase of the literature review, twenty reviews and four single studies. These studies represented published reports from the domains of public health, general health, health promotion and prevention, education, communications, and marketing. The studies included interventions focused on children, adolescents and adults. The largest review, Fixsen, Naoom, Blase, Friedman, and Wallace (2005) covered 743 studies, of which 22 were empirical. The review covered multiple research areas including health, mental health, social services, justice, business, and engineering. The largest empirically-based review examined 581 studies on prevention and health promotion programs for children and adolescents (Durlak and DuPre 2008).
The concepts and constructs influencing implementation quality and intervention performance were extracted from each included review or study. Although different terminology was used across these studies, it was possible to identify a common set of influences on program performance and to categorize these influences. These were used to develop a generic conceptual implementation framework for improving program performance (see Table 1, column 1). This framework consists of the core dimensions of implementation quality including program fidelity measurement and four groups of contextual factors. These groups were: (1) the socio-economic and political context, (2) the implementation process, (3) characteristics of the participating organizations, and (4) characteristics of the participating individuals. The characteristics of the intervention were also identified in the reviewed literature as a common influence on implementation quality. This was not included in the framework because the focus of this review was on a single type of intervention, namely evidence-based vocational rehabilitation.
The common dimensions in a generic implementation framework mapped to vocational rehabilitation
The common dimensions in a generic implementation framework mapped to vocational rehabilitation
Notes: 1. Twenty reviews and four single studies identified the common dimensions of a generic implementation framework; 2. Each dimension of quality and each contextual factor was classified according to whether it was a direct or indirect influence on program performance. Where there was no evidence or there was only one study with evidence of a direct influence the relationship was classified as indirect. Where two studies or more cited evidence this was classified as a direct influence; 3. Each influence was rated based on the strength of evidence and feasibility to measure it. ** = good evidence (three or more studies from the vocational rehabilitation or general psychiatry literature providing evidence of a direct influence on program performance), and the dimension was considered sufficiently operationalized to be feasible to measure; * = limited evidence (less than three studies from the vocational rehabilitation or general psychiatry literature providing evidence of a direct influence), but the dimension was considered sufficiently operationalized to be feasible to measure. No asterisk = limited evidence for a direct influence and the dimension appeared insufficiently operationalized and therefore difficult to measure.
Sixteen of the included studies from phase one were excluded from the second review phase. This was because they focused on intermediate goals of improving program implementation, for example program fidelity, rather than on the end goal of improving program outcomes attained by program participants. The eight remaining studies that examined direct influences on program outcomes were retained for examination in the second stage of the literature review. These were the reviews by Berkel, Mauricio, Schoenfelder, and Sandler, 2011; Dane and Schneider, 1998; Dorr et al., 2007, Durlak and DuPre, 2008; Fixsen et al., 2005; Kitson, Harvey, and McCormack, 1998; Schell et al., 2013; Woltmann et al., 2012. In addition to these eight reviews, a further 19 studies (one review and 18 single studies) were identified as meeting the inclusion criterion (see Appendix 1).
Data from the included studies were synthesized to build a new conceptual framework for improving evidence-based vocational rehabilitation. This involved mapping the empirical evidence into the generic implementation framework to adapt and refine the dimensions of quality and the relevant contextual factors. For example, the dimension of quality, monitoring of comparison health conditions in the generic framework, was further refined to quality of mental health treatment. Program differentiation was refined into the removal of non-evidence based practices. The contextual factor described as government policy in the generic framework was divided into two policy forms: health policy and social security policy. Each dimension of quality and each contextual factor were then classified as direct or indirect, depending on the evidence for their influence on vocational rehabilitation program performance (see Table 1).
Five additional dimensions of implementation quality, other than program fidelity, in the generic implementation framework were classified as having a direct influence. Three contextual factors were also considered to have a direct influence: technical assistance, program evaluation and feedback, and staff expertise (see Table 1). Together these eight dimensions of implementation quality were expected to have the strongest direct influence on improving practices in vocational rehabilitation. The new conceptual framework shows both direct and indirect influences (see Fig. 1). The direct influences are shown within the dimensions of implementation quality ellipse (see Fig. 1), and represent new candidates for dimensions of program implementation quality relevant to psychiatric vocational rehabilitation. They are: program intensity, quality of delivery, employment specialist expertise, program evaluation and feedback, technical assistance, quality of mental health treatment, removal of non-evidence based practices, and participant responsiveness. These dimensions were determined from evidence of program performance based on the proportion of program participants commencing competitive employment.
A conceptual framework for improving the effectiveness of evidence-based practices in vocational rehabilitation. Notes: There are eight dimensions of implementation quality in the ellipse which along with fidelity, are hypothesized to have the strongest direct influence on achieving high program performance. There are 23 contextual factors in the quadrants, these are hypothesized as having a mostly indirect influence mediated through fidelity and the other dimensions of implementation quality in the ellipse. These contextual factors may also have a direct influence, but this is considered to be either weaker or harder to measure. Any possible direct influence of these contextual factors is not discussed in this review. There are also likely to be multiple inter-relationships between all the dimensions of quality and contextual factors in the framework. These are not depicted in Fig. 1. Each direct influence was rated based on the strength of evidence and feasibility to measure it. ** = good evidence (three or more studies from the vocational rehabilitation or general psychiatry literature providing evidence of a direct influence on program performance), and the dimension was considered sufficiently operationalized to be feasible to measure; * = limited evidence (less than three studies from the vocational rehabilitation or general psychiatry literature providing evidence of a direct influence), but the dimension was considered sufficiently operationalized to be feasible to measure. No asterisk = limited evidence for a direct influence and the dimension appeared insufficiently operationalized and therefore difficult to measure.
Twenty-three contextual factors were identified and are shown in the quadrants outside the ellipse in Fig. 1. These were classified using the four categories in the generic implementation framework: the socio-economic and political context, the implementation process, characteristics of the participating organizations, and characteristics of the participating individuals. These contextual factors are expected to indirectly influence program effectiveness, mediated through one or more of the dimensions of implementation quality. For example, funding levels and resourcing is likely to indirectly influence the proportion of program participants commencing employment, mediated through potential impacts on program fidelity, program intensity, quality of mental health treatment, the availability of technical assistance, and employment specialist expertise.
This conceptual framework for implementing evidence-based vocational rehabilitation considers program fidelity along with other aspects of implementation quality. Eight additional dimensions of implementation quality are identified which are expected to have a direct influence on improving program performance. This framework is anticipated to be useful to program funders, practitioners, administrators, evaluators, and researchers to consider a broader range of direct and indirect influences on program performance.
Measuring program fidelity is an important part of this framework. A recent systematic review has shown that a focus on program fidelity is necessary during program implementation for developing program performance (Lockett et al., 2016). However, it is also clear from this review that it is possible for a program to have high fidelity yet be a poor performer in terms of the primary program outcome. Of the 52 cohorts reporting fidelity according to the IPS-15 scale, 40 were classified as high fidelity, a total score of 61 out or more, out of 75, on the IPS-15 scale. Of these 15/40 (37.5%) were low performing (Lockett et al., p. 268). If high fidelity is achieved during program implementation, yet program performance remains below expectations, attention can shift to other dimensions of implementation quality that may be limiting program performance. For programs that achieve high fidelity and high performance, it is possible that attention to these other dimensions of implementation quality may further improve performance.
New dimensions of implementation quality in vocational rehabilitation
The structure of the IPS fidelity scales (IPS-15 and IPS-25) recognize the organizational context in which practices are examined. Principles of good practice are considered in both scales with respect to: staffing, services, and organization. However, most other dimensions of program implementation quality are not covered by these scales (Bond, Drake, McHugo, Rapp, and Whitely, 2009). For example, fidelity assessors are instructed to observe employment specialists in meetings with program participants to observe the presence or absence of particular practices, such as the provision of financial guidance, or how disclosure issues are managed. The fidelity assessors are not required to observe other employment specialist attributes and behaviors, such as goal focus, knowledge of employers, or sensitivity and empathy shown to participants. This is a notable limitation of the current fidelity scales because employment specialist attitudes and knowledge have been found to be directly associated with program outcomes (Glover and Frounfelker, 2011; Gowdy, Carlson, and Rapp, 2004).
Employment specialist expertise
Staff expertise describes the skills, attitudes, knowledge, and competencies of expert, or particularly successful staff, implementing evidence-based practices. This expertise has been identified as an important dimension of implementation quality (Berkel et al., 2011; Dusenbury, Brannigan, Hansen, Walsh, and Falco, 2004). This is particularly salient in the IPS approach to vocational rehabilitation, because the employment specialist working with individual program participants, can directly influence their motivation, behavior, engagement with the program, and subsequent program outcomes.
Employment specialist expertise can be measured in terms of role experience, and through observational ratings of their behaviors and skills against pre-determined competency criteria (Guydish et al., 2014). The behaviors, attitudes, and knowledge of employment specialists have been measured using a standardized scale (Corbière, Brouwers, Lanctôt, and van Weeghel, 2014). This measure, named the BAKES (behaviors, attitudes, knowledge in employment specialists) scale was tested with 153 employment specialists working in a range of supported employment services. A positive relationship was found between employment specialist attributes and program performance. Those who took a supportive and client-centered approach, and who had relationships with employers and supervisors, had better program performance as measured by employment commencements and job tenure. Studies of other aspects of employment specialist expertise have found that knowledge of IPS principles is not as important as an ability to apply those principles (Glover and Froundelker, 2011; Taylor and Bond 2014). This is a promising research direction where further studies are needed to identify other aspects of employment specialist expertise which may further improve program outcomes.
Program intensity
Program intensity is the concentration of a program and contributes directly to program performance (Dulak and DuPre, 2008). It can also be measured in vocational rehabilitation programs (McGuire, Bond, Clendenning, and Kukla, 2011). Measurable aspects of intensity include the length of time participants are in the program, the numbers of face to face contacts between the participants and the employment specialist, the frequency of contact between the employment specialist and the clinical teams, and between the employment specialists and employers (Drake et al., 2012, p. 42).
Low program intensity represented by contact frequency between participants and the employment specialist, explains the unusually low proportion of participants commencing employment in a United Kingdom (UK) randomized controlled trial of IPS compared with other randomized controlled trials. The UK IPS intervention achieved more than 80% adherence to fidelity (67/75 and 69/75) yet only 22% of participants obtained competitive employment (Heslin et al., 2011). A mean of 15 contacts between program participants and an employment specialist was recorded over a two-year period (Heslin et al., 2011). This is 3.2 times lower than the mean of 48 contacts over the two-year study period found by McGuire et al. (2011) in the USA. This USA IPS intervention achieved high levels of fidelity (72/75) and encouragingly, 75% of participants commenced competitive employment (Bond et al., 2007).
The quality of program delivery
The quality of program delivery refers to how well the different aspects of the program have been delivered to participants. This has been found to be positively associated with program outcomes (Berkel et al., 2011; Dane and Schneider, 1998; Dusenbury et al., 2005). In vocational rehabilitation, measures of the quality of program delivery include how well the employment specialist engages with participants about disclosure of participants’ mental health history, or how empathetic and responsive the employment specialist is to participants’ preferences (Areberg, Björkman, and Bejerholm, 2013; Johnson et al., 2009; Topor and Ljungberg, 2016). The quality of relationships between the employment specialist and participants, were found to be positively associated with primary program outcomes in a six-site international randomized controlled trial (Catty et al., 2011). Other authors have also acknowledged that the quality of program delivery is likely to be an important influence on IPS program performance (Drake et al., 2012, p. 44).
The quality of mental health treatment
Some forms of vocational rehabilitation for people with severe mental illnesses such as IPS involve close coordination with a complementary program, namely continuing mental health treatment and care. Complementary programs are considered a core dimension of implementation quality which also needs to be taken into account (Durlak and DuPre, 2008). Monitoring these is important because comparison services and complementary program conditions can influence primary program outcomes. This implies a need to measure the amount and type of complementary program services received by participants. In the case of evidence-based vocational rehabilitation for people with severe mental illnesses, this could involve measuring the type and extent of mental health services along with any other wrap-around health or social services such as: general health services, housing, income support and welfare benefits, financial counselling, disability support, and legal assistance.
The provision of wrap-around services including mental health treatment and medication management, in addition to employment services were an integral part of the IPS intervention in a USA multi-site randomized controlled trial (Drake et al., 2013). The intervention outperformed the control in terms of the proportion of participants commencing employment (60% versus 40%). In another IPS randomized controlled trial, participants in the IPS program were found not only to receive more vocational services, but also more case management and medication management services. They also had more contacts with mental health services than the control group (140 contacts versus 83 contacts) (Gold et al., 2006). It remains unclear whether the amount of mental health treatment confounded the results or were an unintended, yet beneficial, consequence of program integration. Further investigations are needed to understand how the quality of mental health treatment, as a complementary program, contributes to the performance of the primary program, namely evidence-based vocational rehabilitation.
Other dimensions of quality in the new framework
The conceptual framework identifies four other dimensions of quality in addition to the four discussed above. These are: program evaluation and feedback, technical assistance, removal of non-evidence-based practices, and participant responsiveness. Participant responsiveness is how the program holds the involvement and interest of program participants. These four dimensions were identified as direct influences on program performance in this review, because evidence from two or more studies converged to indicate a direct relationship. For example, program evaluation and feedback were found to distinguish high performing from low performing IPS sites (Gowdy et al., 2004). Technical assistance was provided across all the IPS sites in the large-scale USA trial (Drake et al., 2013), and continues to be a core feature of IPS implementation across the USA, (Drake and Bond, 2017) because it is believed by program leaders to support higher program fidelity and higher program outcomes.
The presence of pre-vocational training, an example of a non-evidence-based practice, was considered to have lowered the performance of a Swedish IPS program (Bejerholm, Areberg, Hofgren, Sandlund, and Rinaldi, 2015) and a Canadian IPS program (Latimer et al., 2006). Participant responsiveness may also influence program outcomes. For example, Latimer et al. (2006) found that the difference between the performance of the IPS program and the control group was no longer significant once participant responsiveness, measured as program attrition, was controlled.
Evidence for these other dimensions of quality may not be as strong in the vocational rehabilitation or general psychiatry literature as for the previously discussed four dimensions. However, when considered among the evidence from the broader implementation science literature (Coleman, Mattke, Perrault, and Wagner, 2009; Dorr et al., 2007; Fixsen et al., 2005; Kitson et al., 1998; Woltmann et al., 2012), these other dimensions appear relevant and applicable to vocational rehabilitation. For these reasons, they are also included as dimensions of implementation quality likely having a direct influence on program performance, in the conceptual framework for vocational rehabilitation (see Fig. 1).
Quality assurance and control
This review also noted the extensive business management literature on quality assurance and quality control, and particularly the concept of total quality management. This approach attempts to ensure that an organization delivers a consistent quality of goods of services (Schröder, Schmitt, and Schmitt, 2015). Recommended practices in quality assurance and quality management can and should underpin the delivery of all psychosocial programs. However, a focus on this alone is unlikely to advance our understanding of how to bring about higher program performance. This is because the focus of quality management is typically on compliance and the prevention of poor performance (Schröder et al., 2015), rather than generating high performance.
Limitations
It is possible that some relevant literature was missed because the wider implementation science literature is extensive and distributed across a large number of journals, and electronic databases. Mapping empirical evidence to dimensions of implementation quality and contextual factors revealed that the evidence appeared to become weaker when mapped against specific dimensions of quality and contextual factors. Perhaps this was because it was difficult to isolate the effects of individual dimensions or factors on program performance. There are also likely to be multiple dynamic relationships among the dimensions of quality and contextual factors, yet these are not depicted in the framework.
Conclusions
This literature review and synthesis generated a new conceptual framework to guide the implementation of evidence-based vocational rehabilitation program for people with severe mental illnesses. This framework extends current thinking by widening the definition of implementation quality beyond program fidelity, which considers only a predefined set of established principles and practices. This broader framework can be used to guide investigations of other candidates that may be impeding high fidelity implementations of evidence-based programs from achieving expected levels of program performance, as well as to further enhance the outcomes achieved by high fidelity, high performing programs. It may also have relevance to vocational rehabilitation for individuals other than those with severe mental illnesses. This conceptual framework is intended to assist program funders, researchers, practitioners and policymakers to consider a wider range of potential influences on program performance, in order to advance the science and practice of vocational rehabilitation for people with severe mental illness.
Conflict of interest
None to report.
Footnotes
Appendix 1
The empirical studies used to develop the conceptual framework for evidence-based practices in vocational rehabilitation (n = 27)1.
| Included paper | Brief description of paper | Factors found to be directly influencing program performance | Empirical findings which were mapped to the conceptual framework | Relevance rating2 |
| Areberg et al., 2013 | A qualitative study interviewing 17 IPS3 program participants, 15 of whom got a job through the program. | •Individual needs and wishes valued by ES4 | ES expertise | A |
| •Benefitting from the skills of ES and from being two people instead of one | The quality of program delivery Participant responsiveness | |||
| •Bringing the individual’s own qualifications to the process | ||||
| •Participants having a clear vision of the benefits for them through their participation | Clinicians’ beliefs | |||
| •Having others who also believe (including the clinical team). | ||||
| Bejerholm et al., 2015 | A randomized controlled trial in Sweden of IPS versus TVR5 with 120 participants. | •Welfare regulations required all participants to go through a period of internship and assessment prior to searching for competitive employment. | Removal of non-evidence-based practices Social security policy | A |
| Berkel et al., 2011 | A review of empirical papers which explored the association between dimensions of implementation and outcomes in prevention programs. | •Fidelity, quality of delivery, adaptation and participant responsiveness. | Quality of delivery Adaptation Participant responsiveness | B |
| Boycott et al., 2012 | A rapid evidence assessment of interventions designed to supplement IPS. It included six empirical papers which looked at enhancing IPS with either skills training or cognitive remediation. | •It was difficult to measure the contribution that the IPS adaptation was having on program performance. | Program adaptation | A |
| Catty et al., 2011 | Analysis of data from a six-site randomized controlled trial of IPS versus TVR across the UK and Europe to identify factors affecting outcomes. | •Patient and worker rated therapeutic relationship as measured by the Helping Alliance Scale •Patients’ previous work history. | The quality of program delivery | A |
| Coleman et al., 2009 | A review and meta-analysis of the chronic illness management literature, included 56 empirical articles. The aim was to examine the association between practice re-design and outcomes. | Studies which focused on practice re-design, such as re-organizing the way care was delivered and which used data and guidelines to engage patients as well as focusing on modifying patient behaviors, were more likely to have better patient outcomes. | Program evaluation and feedback | B |
| Corbière et al., 2014 | The behaviors, attitudes and competencies of 153 ESs were measured to explore the associations with employment outcomes. | The BAKES6 scale was developed and tested. The subscales of ‘relationships with employers and supervisors’ and ‘support and client centered approach’ were associated with employment commencements and six month job tenure. | ES expertise Quality of program delivery i.e. relationships between the ES and employers | A |
| Dane and Schneider, 1998 | A review of the prevention literature from 1980–1994, included 162 empirical articles. | Identified three dimensions to program fidelity which effect outcomes: 1. exposure/dosage; 2. fidelity (adherence); 3. quality of delivery (one study). | Program intensity Quality of program delivery | B |
| Dorr et al., 2007 | Review of chronic illness literature to examine the association between information systems, quality and outcomes. Included 109 empirical articles, 23% of which had a focus on patients with mental illness. | Factors correlated with positive outcomes were: electronic medical record, computerized prompts, reports and feedback, decision-support aids, electronic scheduling and personal health records. Those factors which were barriers to outcomes were: costs, data privacy, security concerns and a lack of focus on workflows. | Program evaluation and feedback Administration and information systems | B |
| Drake et al., 2013 | A randomized controlled trial testing the chronic care model of wrap-around services including supported employment, medication management and other mental health treatments against usual services, on employment outcomes. | The intervention group had a greater proportion of employment commencements (60% versus 40%) and had improvements in mental health and quality of life. Additional services offered to the intervention group were: training, experts, insurance cover and fidelity monitoring. | The quality of mental health treatment Technical assistance Legislation and regulation Program evaluation and feedback | A |
| Durlak and DuPre, 2008 | A review of the prevention and health promotion literature from 1976-2006 to examine the influence of implementation on outcomes (542 studies) and the factors affecting implementation quality (81 studies). | •Identified seven dimensions of implementation in addition to fidelity and twenty-three ecological factors. | Eight dimensions of implementation quality | B |
| Fixsen et al., 2005 | A systematic review covering multiple research fields including mental health. The findings from 22 empirical papers were reported separately. | •The role of experts in improving implementation, quality of care and patient outcomes |
Technical assistance ES expertise Quality of mental health treatment | B |
| Glover and Froundfelker, 2011 | An exploratory study to identify factors which differentiated successful from less successful employment specialists. | The more successful employment specialists worked efficiently, developed egalitarian relationships with program participants and collaborated effectively with partners. | ES expertise The quality of program delivery | A |
| Gold et al., 2006 | A randomized controlled trial comparing assertive community treatment combined with IPS versus TVR in a rural area, 143 participants. | It was difficult to recruit staff in a rural area. The intervention outperformed the control on proportion of employment commencements and IPS participants also had more mental health services than control. | Role of geography Quality of mental health treatment Program intensity | A |
| Gowdy et al., 2004 | A qualitative study exploring the differences between high-performing and low-performing supported employment programs. | Leaders emphasize the value of work and the belief that if people want to work they can work. High-performing sites used data to track and feedback outcomes. ESs recognized that participants are motivated to find employment. | ES expertise Administration and information systems Leadership and program champions Program evaluation and feedback | A |
| Guydish et al., 2014 | An examination of the relationship between treatment fidelity, competence and empathy and six outcomes (number of days of drug, employment, medical, social and psychological outcomes) using an addiction treatment (twelve step facilitation) with 151 patients in 871 therapy sessions. | All three dimensions of quality were associated with improved employment status at three month follow-up. Higher therapist competence was associated with lower self-report drug use. | Staff expertise The quality of delivery i.e. empathy | A |
| Heslin at al., 2011 | A randomized controlled trial comparing IPS to TVR in England with 219 participants followed for two years. | Fidelity was measured and found to be good, but outcomes were lower than other randomized controlled trials. The average number of contacts with the ES were low (15 over two-year period). | Program intensity | A |
| Johnson et al., 2009 | A qualitative study which interviewed 182 participants to understand their perceptions of the effective ingredients of supported employment. | Service participants highlighted the importance of the quality of support that was offered, particularly emotional support and practical assistance, combined with a person-centered approach. | The quality of delivery, particularly through interpersonal dynamics | A |
| Kitson et al., 1998 | Empirical testing of the PARIHS7 framework using four case studies in health care settings. | For successful implementation, the evidence for the intervention, the context and the method of facilitation should be equally weighted. For example, strength of evidence doesn’t warrant more attention than the process of implementation. In one case study, the facilitator’s role overcame poor contextual conditions. | Technical assistance | B |
| Latimer et al., 2006 | A randomized controlled trial of IPS vs. TVR in Canada, with 150 participants. | IPS program outcomes were lower than in the USA8 studies. This was attributed to aspects of social security policy. For example: a lack of financial incentives to earn more than Canadian $100 per week, free public transport passes only for those in TVR, and the availability of non-evidence based programs which paid minimum wage or above. In addition, when attrition was controlled for the difference between the outcomes in the IPS sites and the controls was no longer significant. | Social security policy Removal of non-evidence based practices Participant responsiveness i.e. attrition | A |
| Lockett et al., 2016 | A systematic review and meta-analysis of IPS implementation studies to examine the predictive validity of the IPS fidelity scales. | Fidelity, whilst an important predictor of outcomes in a range of countries and contexts, only explains 17% of the variance in employment commencements. | There are factors in addition to fidelity which influence the outcomes of IPS programs. | A |
| MacDonald-Wilson et al., 2003 | Surveyed 539 service users, 120 service providers and 174 family members to examine the effects of social security work incentives on return to work. | •The concern over loss of welfare and health benefits •Lack of awareness of financial incentives that were available. | The role of social security policy Service users and families’ attitudes and knowledge | A |
| McGuire et al., 2011 | Examined data from studies of 91 people who had received IPS to explore the relationship between service intensity and outcomes. | The level of service intensity predicted job tenure although where participants had high levels of cognitive symptoms this weakened the relationship. | Program intensity | A |
| Schell et al., 2013 | A literature review of 85 papers, combined with input from an expert panel, to identify the core constructs for achieving sustainability in public health programs. | Funding stability, political support, partnerships, organizational capacity, program adaptation, program evaluation, communications, public health impacts, strategic planning | Adaptation Program evaluation and feedback | B |
| Taylor and Bond, 2014 | A cross-sectional survey of 57 ESs and their supervisors in high fidelity IPS programs to examine predictors of employment outcomes. | The amount of contact with an ES and the supervisor rated ES job performance and efficacy were related to job commencements and job tenure. | Program intensity ES expertise | A |
| Topor and Ljungberg, 2016 | Interviews with nine participants in an IPS program in Sweden to examine the importance of the relationship with the ES. | Participants described the ‘helpful relationship’ with the ES where they were ‘treated like a human being’. | Quality of delivery | A |
| Woltmann et al., 2012 | A systematic review and meta-analysis of randomized controlled trials comparing collaborative chronic care models with other care models on patient outcomes: mental health symptoms and quality of life. Seventy-eight articles with 161 analyses from 57 trials. | Chronic care models improve mental health outcomes as well as physical health. | Administration and information systems Program evaluation and feedback Organizational and leadership support Patient support, including self-management tools, and coaching Networks and community links | B |
Notes: 1. n = number of included studies; 2. Each empirical paper was rated according to its relevance to vocational rehabilitation for people with severe mental illness. A = Papers from vocational rehabilitation or psychiatry, B = Papers from the health sector but outside psychiatry; 3. IPS = Individual Placement and Support; 4. ES = Employment specialist; 5. TVR = Traditional Vocational Rehabilitation; 6. BAKES scale = Behavior Attitudes Knowledge of Employment Specialists scale; 7. PARIHS = Promoting Action on Research Implementation in Health Services framework 8. USA = United States of America.