Promising and evidence-based practices in vocational rehabilitation: Results of a national Delphi study

Abstract

BACKGROUND:

Recently, Leahy and his colleagues (Anderson, Leahy, Valle, Sherman, & Tansey, 2014; Del Valle, Leahy, Sherman, Anderson, Tansey, & Schoen, 2014; Leahy, Leahy, Chan, Lui, Rosenthal, Tansey, Wehman, P, & Menz, 2014; Sherman, Leahy, Del Valle, Anderson, Tansey, & Lui, 2014; Tansey, Bezyak, Chan, Leahy, & Lui, 2014) completed a comprehensive qualitative case study of four state VR agencies (Maryland, Mississippi, Texas, and Utah) to identify promising or evidence-based practices that can be used to improve management and clinical rehabilitation counseling practices. Fourteen promising or evidence-based VR practices were reported by administrators, supervisors, and counselors in those four VR agencies as useful for improving psychosocial and employment outcomes of persons with disabilities receiving services from state rehabilitation agencies.

OBJECTIVE:

The purpose of this article is to describe the outcomes of a Delphi study of experts on evidence-based VR practices.

METHODS:

A Delphi study was conducted to obtain the consensus of 35 national experts in vocational rehabilitation (VR) on the relevance and levels of scientific evidence of 26 promising or evidence-based VR practices in state agency settings.

RESULTS:

Consensus was achieved through three rounds of the Delphi process. National experts rated the employment-based interventions as highly relevant to state VR service delivery practices, but rated their scientific evidence in the lower end of the hierarchy of evidence. Experts rated psychosocial and counseling interventions, except for motivational interviewing and working alliance, as less relevant to state VR, but as having high levels of scientific evidence.

CONCLUSION:

This study represents an important step towards identifying specific promising or evidence-based VR practices in state agency settings that could be used to improve psychosocial and employment outcomes for people with disabilities. The results can also be used to plan the in-service training agenda for state VR agencies in terms of professional development and in pre-service academic programs to ensure that rehabilitation counseling students are adequately trained in relation to these promising or evidence-based VR practices.

Keywords

Evidence-based practice Delphi study and vocational rehabilitation

1 Introduction

The state-federal vocational rehabilitation (VR) program spends more than three billion dollars annually to help over one million people with disabilities pursue their independent living and employment goals (Leahy et al., 2014; Martin, West-Evans, & Connelly, 2010; U.S. Government Accountability Office [GAO], 2005). However, only slightly over one-half of these consumers of VR services have found competitive employment in integrated settings after receiving services, and their average hourly earnings is only 52% of the general workforce (Dutta, Gervey, Chan, Chou, & Ditchman, 2008; Leahy et al., 2014). Employment, earnings, and the number of purchased services received while in the state-federal VR program also varied significantly by disability types and other demographic and contextual factors (GAO, 2005). For example, individuals with disabilities from racial and ethnic minority backgrounds received fewer high expenditure services (e.g., university and college training) compared to their white counterparts (Wilson, Turner, & Jackson, 2002). The U.S. Office of Management and Budget and Federal agencies (2006) gave the state-federal VR program only an “adequate” performance rating, indicating a strong need for the agency to (a) set more ambitious goals, (b) achieve better results, (c) improve accountability, and (d) strengthen its management practices. Similarly, the GAO (2005) placed the state-federal VR program among other federal disability programs on its list of “high-risk” programs for not keeping up with scientific advances and economic and social changes.

In this era of empowerment, accountability, and constrained budgets, state VR agencies must do more to improve independent living and employment outcomes and help people with disabilities work their way into the middle class (Chan et al., 2016; Leahy et al., 2014; Rubin, Chan, & Thomas, 2003). Rehabilitation counselors must provide the most effective psychosocial and vocational services and interventions to help people with disabilities find good jobs with benefits consistent with their abilities and career interests (Chan et al., 2016; Leahy, Thielsen, Millington, Austin, & Fleming, 2009). As a result, there has been a significant amount of interest and movement towards the use of evidenced-based practices (EBP) in VR to improve service outcomes (Chan et al., 2016; Leahy et al., 2014). For example, the National Institute on Disability, Independent Living and Rehabilitation Research (NIDILRR) has underscored the importance of research studies funded by the agency to meet standards for inclusion in systematic reviews and meta-analyses. The Rehabilitation Services Administration (RSA) also emphasizes the importance of developing and implementing VR services that integrate the best scientific evidence with clinical expertise and client perspectives to improve employment outcomes (Chan, Tarvydas, Blalock, Strauser, & Atkins, 2009). Both agencies also emphasize that efforts must be expended to translate rehabilitation and disability research into clinical rehabilitation counseling practice and policy. However, little is known about what types of empirically-supported psychosocial and vocational interventions are being used in VR service delivery practices.

Recently, Leahy and his colleagues (Anderson, Leahy, Valle, Sherman, & Tansey., 2014; Del Valle et al., 2014; Leahy et al., 2014; Sherman et al., 2014; Tansey, Bezyak, Chan, Leahy, & Lui, 2014) completed a comprehensive qualitative case study of four state VR agencies (Maryland, Mississippi, Texas, and Utah) to identify promising or evidence-based practices that can be used to improve management and clinical rehabilitation counseling practices. In their case study, they defined evidence-based practices as interventions that are supported by randomized controlled trials (the two highest levels of evidence in the evidence hierarchy), while promising practices were defined as any intervention, program/service, strategy, or policy that has strong quantitative and qualitative data showing positive outcomes, but does not yet have enough research or replication to support generalizable positive outcomes for consideration as an evidence-based practice (Leahy et al., 2013).

This multi-state case study of promising or evidence-based VR practices project spanned a three-year period and represented the perspectives of 29 leaders and administrators, 56 mid-level managers, and 469 counselors in state VR agencies. As part of the case study, Del Valle et al. (2014) identified 14 promising or evidence-based VR practices that were reported by administrators, supervisors, and counselors in those four VR agencies as useful for improving psychosocial and employment outcomes of persons with disabilities receiving services from state rehabilitation agencies. These promising or evidence-based VR practices included supported employment for people with intellectual and developmental disabilities, individual placement and support model of supported employment (IPS) for people with psychiatric disabilities, benefits counseling, secondary transition services for youth with disabilities, motivational interviewing (MI), demand-side employment strategies, and workplace socialization skills training.

1.1 Purpose of the study

Leahy et al.’s (2014) study represents an important step to identify promising and evidence-based VR practices that are used by state VR agencies to improve psychosocial and employment outcomes of people with disabilities. However, their findings were based on four state agencies and there may be other empirically supported interventions for improving the effectiveness of VR service delivery practices. Therefore, we convened a panel of VR experts using Delphi procedures to determine whether there is national consensus regarding the applicability of the promising or evidence-based VR practices identified by Leahy et al. (2014) for improving psychosocial and employment outcomes of people with disabilities receiving services from state VR agencies. In addition, the experts in our study were asked to identify additional empirically supported psychosocial and vocational interventions that were not included in Leahy et al.’s (2014) multi-state case study. The purpose of this article is to describe the outcomes of this Delphi study.

2 Method

2.1 Participants

The present study used the Delphi technique, a widely used and accepted method for gathering data from respondents within their domain of expertise (Vazques-Ramos, Leahy, & Hernandez, 2007), to obtain consensus from a panel of VR experts regarding promising or evidence-based VR practices that could be used to improve rehabilitation outcomes for people with disabilities receiving services from state VR agency settings. Concerted efforts were taken to identify the strongest panel possible for the Delphi Study, which consisted of exemplary rehabilitation educators and researchers, rehabilitation counselors/supervisors, and administrators of state VR agencies. These national experts were selected based on their knowledge and clinical experience with VR services relevant to the rehabilitation process, as well as their expertise and research scholarship in EBP related to VR services and interventions. Nominations were sought from all the researchers associated with the Research and Training Center on Evidence-Based Practice in Vocational Rehabilitation (RRTC-EBP VR), its advisory council members, and other stakeholders to generate a comprehensive list of possible subject matter experts (SMEs; N = 35) for the study. These experts were contacted via email and all (100%) agreed to participate. The panel comprised 16 (45.7%) women and 19 (54.3%) men, with reported years of experience in VR ranging from 5 years to 44 years of service (M = 30.14, SD = 10.17). Twenty-three SMEs (65.7%) reported a doctoral degree and 10 reported a master’s degree as their highest educational credential (n = 10, 28.6%), and most reported their academic training was within the closely aligned fields of rehabilitation psychology (n = 8, 22.9%), rehabilitation counseling (n = 6, 17.1%), or rehabilitation counselor education (n = 4, 11.4%). Other educational specialties included business, human resource management, education, social welfare, and special education. Panelists primarily reported working in either academic/university settings (n = 21, 60.0%) or state VR agencies (n = 10, 28.6%), with the remaining four reporting employment in non-profit rehabilitation settings as a disability policy consultant, as a contractor for the Social Security Ticket to Work, or as an administrator with a Tribal Vocational Rehabilitation program.

2.2 Measures

The standard Delphi technique consists of a series of questionnaires about an important question or problem that are usually mailed to a group of individuals who are considered experts in a particular subject matter area (Vazques-Ramos et al., 2007). It involves recruitment of a panel of experts in a particular field and repeated questioning of each group member using at least two sequential questionnaires, called rounds. The statistical summary prepared after each round is used for the next round of questions to provide the needed feedback for experts to consider revising their views through awareness of overall responses from other experts in the panel. For each item in the questionnaire, the statistical analysis includes some combination of the mean, median, and mode of the panelists’ responses and at least one measure of dispersion (e.g., standard deviation or inter-quartile range). The use of these iterative procedures is common in Delphi studies to achieve group consensus among experts, with consensus being reflected in the convergence of variance or standard deviations in subsequent iterations (Hsu & Standford, 2007).

The Round 1 questionnaire

for the present study comprises one open-ended question asking the SMEs in the expert panel to provide specific examples of promising or evidence-based VR practices that could be used to improve employment outcomes for individuals with disabilities in state agency settings. Promising practice was defined using the definition provided by Leahy et al.’s (2014) multi-state case study. The Round 2 questionnaire was developed based on (a) a comprehensive literature review of employment interventions for people with disabilities reported by Fleming, Del Valle, Kim, and Leahy (2013); (b) the 14 promising or evidence-based VR practices identified by Del Valle et al. (2014) in their multi-state case study; and (c) specific promising or evidence-based VR practices provided by the SMEs in the Round 1 questionnaire. The promising or evidence-based practices included rehabilitation interventions that lead to successful psychosocial and employment outcomes such as supported employment, generic counseling approaches such as MI, therapeutic conditions such as working alliance that are either services that can be purchased from community-based rehabilitation agencies or counseling practices that can be used by rehabilitation counselors to improve client-counselor relationship, outcome expectations, and VR service engagement. The Round 2 questionnaire includes 26 promising or evidence-based VR practices that were triangulated from the three sources of data. See Appendix A for brief descriptions of practices described in this study. Two five-point Likert type scales were developed for the experts to rate each promising or evidence-based VR practice. The first rating scale ranging from 1 (very low relevance) to 5 (very high relevance), was used to evaluate the relevance of the 26 promising or evidence-based VR practice to effective VR service delivery. A second five-point Likert-type scale ranging from 5 (the highest evidence) to 1 (low evidence), was utilized to rate the level of scientific evidence for each promising or evidence-based VR practice item based on the hierarchy of evidence that is a core principal of EBP (Chan et al., 2016). The Round 2 and 3 questionnaires provided the SMEs with definitions for both the relevance of the promising practice or EBP for VR service delivery and level of evidence. Table 1 presents the definitions and the rating system for the hierarchy of evidence. The Round 3 questionnaire comprises the same items as the Round 2 questionnaire; however, the SMEs are provided with feedback related to their own rating on each item along with the average group ratings and dispersion information on each item (means and standard deviations) based on the Round 2 data.

Table 1
Hierarchical levels of evidence

Level Evidence

1. Strong evidence from at least one systematic review of multiple well-designed randomized controlled trials.

2. Strong evidence from at least one properly designed randomized controlled trials of appropriate size.

3. Evidence from well-designed trials without randomization, single group pre-post, cohort, time series, or matched case-controlled studies.

4. Evidence from well-designed non-experimental studies from more than one center or research group.

5. Opinions of respected authorities, based on clinical evidence, descriptive studies, or reports of expert committees.

Level	Evidence
1.	Strong evidence from at least one systematic review of multiple well-designed randomized controlled trials.
2.	Strong evidence from at least one properly designed randomized controlled trials of appropriate size.
3.	Evidence from well-designed trials without randomization, single group pre-post, cohort, time series, or matched case-controlled studies.
4.	Evidence from well-designed non-experimental studies from more than one center or research group.
5.	Opinions of respected authorities, based on clinical evidence, descriptive studies, or reports of expert committees.

Note. In the present study, the 5-point evidence rating scale was reverse-scored so that high scores represented higher levels of evidence. Interventions are considered evidence-based practice if supported by randomized controlled trials (levels 1 and 2 evidence) and interventions with levels 3, 4, or 5 evidence were considered promising practice in this study.

2.3 Procedures

The Round 1 questionnaire was emailed to the 35 SMEs who had agreed to participate and a follow-up reminder email was sent to non-respondents two weeks after the initial contact. Data for the open-ended question were gathered using Qualtrics, an online survey software tool. As noted above, findings from the Round 1 study along with the comprehensive literature review on employment interventions for people with disabilities reported by Fleming et al. (2013) and the 14 promising VR practices identified by Del Valle et al. (2014) were then used to develop the 26-item version of the Round 2 EBP VR questionnaire.

The web link for completing the Round 2 questionnaire was sent to the 35 SMEs via an introductory email. Non-respondents were contacted two weeks after the first mailing via email to remind them to complete the Delphi questionnaires and again providing them with the web link. Based on the responses obtained in Round 2, means and standard deviations were calculated for each promising or evidence-based VR practice item according to its degree of relevance and the hierarchy of scientific evidence; the Round 3 questionnaire was then developed based on those data. The web link for completing the Round 3 questionnaire was sent via email to the SMEs who had completed Round 2; a reminder email was sent to non-respondents two weeks after. The Round 3 questionnaire provided the SMEs with their own ratings of each promising or evidence-based VR practice item from Round 2 in terms of relevance and level of evidence, along with the Round 2 group means and standard deviations for each EBP item. The SMEs were asked to review their own responses and compare their responses against the group average, reconsider their responses as they saw appropriate, and then complete the Round 3 questionnaire. Data for the Delphi study were then analyzed using descriptive and inferential statistics.

3 Results

For the Round 1 study, all the 35 SMEs provided demographic information; 30 experts provided examples of promising or evidence-based VR practices. Examples of promising or evidence-based VR practices provided by the VR experts were supported employment, IPS, MI, secondary transition services, benefits counseling, and internship/summer paid work experience for transition-age youth with disabilities. Of the 35 Round 2 mailings, 28 responses were obtained after two mailings (80% response rate). 26 of the 28 Round 2 SMEs responded to the Round 3 of the Delphi study, resulting in a response rate of 92% for that iteration and 74% for the entire Delphi study.

Of the 28 SMEs who provided data on Round 2, 26 also provided usable responses on the consensus round (Round 3). The mean standard deviations for the relevance scale was reduced from 0.81 in Round 2 to 0.72 in Round 3, t (25) = 3.14, p < 0.01. The mean standard deviations also dropped significantly for the level of evidence scale, from 1.38 in Round 2 to 1.02 in Round 3, t (25) = 11.10, p < 0.01. The ranks, means, and standard deviations related to relevance to VR and level of evidence for each promising or evidence-based VR practices computed for Round 3 is presented in Table 2. The average rating for the relevance of the 26 promising or evidence-based VR practices was 3.80, while the mean for the evidence level ratings was 2.51. Thirteen promising/evidence-based VR practices were ranked as having a high level of relevance; however, of these highly ranked services, only IPS, MI, and working alliance received mean ratings of between at least one randomized controlled trial and well-designed trials without randomization, single group, pre-posttest, cohort, time series, or matched controlled studies in terms of level of evidence. The other top tier VR services (e.g., secondary transition services, assistive technology, demand-side employment strategies, benefits counseling, customized employment, person centered planning, and soft skills training) were ranked high in relevance to VR but not as high in terms of levels of evidence.

Table 2
Relevance and level of evidence ratings for VR practices

Relevance Level of Evidence

VR Practices Rank M SD Rank M SD

Secondary Transition Services 1 4.77 0.43 12 2.50 1.06

Assistive Technology 2 4.54 0.65 14 2.46 0.93

IPS Supported Employment 3 4.46 0.71 3 3.39 1.27

On-the-Job Training 4 4.42 0.50 15.5 2.35 0.90

Demand-Side Employment Strategies 5 4.31 0.68 21 2.00 0.85

Benefits Counseling 6 4.23 0.77 19 2.14 0.97

Motivational Interviewing 7 4.19 0.69 4 3.17 1.19

Customized Employment 8 4.19 0.63 20 2.08 1.06

Community Based Work Program 9 4.19 0.63 18 2.30 0.97

Working Alliance 10 4.15 0.83 6 3.11 0.74

Person Centered Planning (PCP) 11 4.04 0.60 22.5 1.96 0.86

Soft Skills Training 12 4.00 0.76 22.5 1.96 0.81

Dual Customer Approach 13 4.00 0.57 26 1.45 0.74

Job Club 14 3.88 0.86 15.5 2.35 1.03

Family Involvement &Support 15 3.88 0.82 17 2.32 0.89

Social Skills Training 16 3.73 0.78 13 2.48 0.90

Cognitive Behavioral Therapy 17 3.58 0.64 1 3.54 1.22

Assertive Community Treatment (ACT) 18 3.54 0.65 2 3.50 1.01

Positive Psychology Interventions 19 3.42 0.64 8 2.86 1.13

Project Search 20 3.42 0.90 11 2.52 1.29

Solution-Focused Brief Therapy 21 3.19 0.75 5 3.15 1.04

Health Promotion Interventions 22 3.12 0.77 10 2.60 1.14

Person Centered Therapy 23 3.08 0.89 7 2.99 1.12

Online Community of Practice 24 3.04 0.82 25 1.57 1.16

Social Media 25 2.92 0.85 24 1.61 1.04

Tele-Health/Tele-Rehabilitation 26 2.62 0.80 9 2.83 1.34

Mean of VR Practice 3.80 0.72 2.51 1.02

	Relevance	Level of Evidence
Secondary Transition Services	1	4.77	0.43	12	2.50	1.06
Assistive Technology	2	4.54	0.65	14	2.46	0.93
IPS Supported Employment	3	4.46	0.71	3	3.39	1.27
On-the-Job Training	4	4.42	0.50	15.5	2.35	0.90
Demand-Side Employment Strategies	5	4.31	0.68	21	2.00	0.85
Benefits Counseling	6	4.23	0.77	19	2.14	0.97
Motivational Interviewing	7	4.19	0.69	4	3.17	1.19
Customized Employment	8	4.19	0.63	20	2.08	1.06
Community Based Work Program	9	4.19	0.63	18	2.30	0.97
Working Alliance	10	4.15	0.83	6	3.11	0.74
Person Centered Planning (PCP)	11	4.04	0.60	22.5	1.96	0.86
Soft Skills Training	12	4.00	0.76	22.5	1.96	0.81
Dual Customer Approach	13	4.00	0.57	26	1.45	0.74
Job Club	14	3.88	0.86	15.5	2.35	1.03
Family Involvement &Support	15	3.88	0.82	17	2.32	0.89
Social Skills Training	16	3.73	0.78	13	2.48	0.90
Cognitive Behavioral Therapy	17	3.58	0.64	1	3.54	1.22
Assertive Community Treatment (ACT)	18	3.54	0.65	2	3.50	1.01
Positive Psychology Interventions	19	3.42	0.64	8	2.86	1.13
Project Search	20	3.42	0.90	11	2.52	1.29
Solution-Focused Brief Therapy	21	3.19	0.75	5	3.15	1.04
Health Promotion Interventions	22	3.12	0.77	10	2.60	1.14
Person Centered Therapy	23	3.08	0.89	7	2.99	1.12
Online Community of Practice	24	3.04	0.82	25	1.57	1.16
Social Media	25	2.92	0.85	24	1.61	1.04
Tele-Health/Tele-Rehabilitation	26	2.62	0.80	9	2.83	1.34
Mean of VR Practice	3.80	0.72	2.51	1.02

Note. For the relevance scale, 1 = not relevant and 5 = highly relevant; for the level of evidence rating, 1 = lowest level of evidence, and 5 = highest level of evidence.

4 Discussion

The use of EBP in VR represents a wide array of services and interventions based on various levels of “evidence” and applications within the state-federal VR program (Chan et al., 2016; Leahy et al., 2014). The present study was conducted to gain a national consensus on current VR practices and interventions that a panel of experts considers promising or evidence-based in relation to effective state VR service delivery practices. This study represents an important first step in identifying specific promising and evidence-based practice in state VR agency settings that could lead to successful psychosocial and employment outcomes for the consumers served by state rehabilitation agencies.

Findings from our Delphi study demonstrated agreement among SMEs through the convergence of standard deviations in subsequent iterations. The study also indicated that the standard deviations for the level of evidence scale were larger than the relevance scale. This is because although SMEs were able to identify several promising practices that are important and relevant to VR, it was more difficult for the experts to achieve consensus on the evidence-level of the promising VR practices. In addition, of the 13 promising/evidence-based VR practices ranked as having a high level of relevance, only IPS, MI, and working alliance were rated as having a high level of evidence. As noted above, the remainder of the top tier VR services (e.g., secondary transition services, demand-side employment strategies, benefits counseling, customized employment, person centered planning and soft skills training) were ranked high in relevance to VR, but not as high in terms of levels of evidence. It seems that some of these employment services can be viewed as promising practices, while others can be considered emerging practices. Emerging practices are practices that have anecdotal evidence and are consistent with professional wisdom, but are not necessary based on research or theory and have not been studied through the collection of original data. Because many of the practices are emerging practices, they were also ranked relatively low in terms of levels of evidence.

The endorsement of employment-related services as highly relevant to VR service delivery practice can be partially attributed to the traditional emphasis of state VR agencies on employment outcomes as well as recent passage of the Workforce Innovation and Opportunity Act (WIOA) and the accompanying amendments of the Rehabilitation Act, which place a strong emphasis on local labor market analysis, employer engagement, customized training, supported employment, postsecondary education intervention, and secondary transition service (U.S. Department of Education, 2014). Interestingly, experts in our study appeared to be more cautious in their evaluation of the level of evidence for some of the employment-related interventions. For example, IPS is clearly an evidence-based mental health service (Mueser, Drake, & Bond, 2016). Campbell, Bond, and Drake (2011) conducted a meta-analysis to compare the effect of the IPS model of supported employment with traditional vocational interventions for people with severe mental illness. They found large effect sizes favoring the use of IPS in job acquisition (d = 0.90), total weeks worked (d = 0.79), and job tenure (d = 0.74) over traditional vocational interventions. Therefore, participants in the present study should rate IPS high in terms of levels of evidence (i.e., strong evidence from at least one systematic review of multiple well-designed randomized controlled trials). Similarly, ACT, an evidence-based mental health service, should also rate high by our experts. In addition, there is emerging evidence to support the effectiveness of benefits counseling, an important VR intervention, to improve employment outcomes of social security beneficiaries (Delin, Hartman, & Sell, 2012; Nazarov, 2013; Ruiz-Quintanilla, Weathers II, Melburg, Campbell, & Madi, 2006; Tremblay, Smith, Xie, & Drake, 2006). However, benefits counseling was rated low in terms of scientific evidence.

Conversely, several of the evidence-based counseling/psychotherapy approaches (e.g., cognitive behavior therapy and person-centered therapy) and psychosocial interventions (e.g., social skills training) that are highly emphasized in the rehabilitation counselor education curriculum (Chan, Berven, & Thomas, 2015) were rated the lowest in terms of relevance to VR practice. The low relevance ratings for these empirically supported psychosocial and counseling interventions were surprising as psychosocial and vocational adjustment are important requisite factors to successful employment and career advancement outcomes (Chan et al., 2015; Chan, Cardoso, & Chronister, 2009).

It was less surprising to find our experts rated MI and working alliance as highly relevant to effective VR service delivery practice since both techniques are considered evidence-based practice in professional psychology and counseling (Manthey, Brooks, Chan, Hedenblad, & Ditchman, 2015). Client motivation has been considered one of the essential ingredients needed to provide effective counseling in VR settings (Chan, Shaw, McMahon, Koch, & Strauser, 1997; Cook, 2004; Manthey et al., 2015; Wagner & McMahon, 2004). Vocational rehabilitation consumers often have competing interests, values, and conflicts related to rehabilitation plans, and it is invaluable for rehabilitation counselors to learn how to effectively help individuals to explore and resolve this ambivalence. Motivational interviewing is useful for changing counselors’ attitudes toward helping clients and provides a framework for counselors to use specific techniques to increase client motivation in order to achieve goals and successful rehabilitation outcomes (Manthey et al., 2015; Wagner & McMahon, 2004). Several VR agencies including Michigan, Wisconsin, Minnesota, and Washington have provided MI training to their rehabilitation counselors. Similarly, working alliance, defined as a high quality of partnership and mutual collaboration between therapist and client, was endorsed as an evidence-based VR practice by the experts in our study. Working alliance is one of the most researched common factors in counseling/psychotherapy. There is strong research evidence to support working alliance as an important aspect of psychotherapy (Wampold, 2001; Wampold & Imel, 2015). Horvath, Del Re, Flückiger, and Symonds (2011) analyzed 190 studies representing over 14,000 and found a “relatively robust” relationship between working alliance and positive counseling outcomes (r = 0.27), which is equivalent to a Cohen’s d of 0.57, surpassing the threshold for a medium sized effect. In VR, working alliance was found to be associated with self-determined work motivation, vocational outcome expectancy, VR engagement, and readiness for employment (Tansey, Bezyak, Iwanaga, Anderson, & Ditchman, 2016).

Some of the support services for rehabilitation counselors that were simultaneously ranked low in terms of relevance and levels of evidence are important knowledge translation practices that can be used to encourage counselors to implement EBP in their practices (e.g., online community of practice for rehabilitation counselors). Similarly, the social media applications that are considered the future of health and human services delivery (Moorhead et al., 2013), were not ranked by the SMEs as either relevant or having strong evidence to support their use in VR service delivery.

According to Chan et al. (2011), there is a general misperception among stakeholder groups in the disability and rehabilitation communities (e.g., policymakers, advocates, and consumers) that rehabilitation counselors in state VR agencies do not use empirically validated interventions to improve the psychosocial, health, and employment outcomes of people with disabilities. To the contrary, and supported by the Leahy et al. (2014) study and the present study findings, many services provided by rehabilitation counselors in state VR agencies are supported by strong scientific evidence, including best practices in supported employment/IPS, counseling/psychotherapy, MI, and skills training (Chan et al., 2016; Leahy et al., 2014).

4.1 Limitations

There are several limitations to consider when reviewing the results. Although great care was taken to select the SMEs, selection bias is still a possibility. Another group with different backgrounds, yet equal expertise might evaluate the relevance of the promising and evidence-based VR practices somewhat differently. To safeguard against selection bias, we requested input from the entire research staff of the RRTC-EBP VR and their respective advisory council. This broad input and the careful examination of the characteristics of each panel member nominated hopefully reduced the possibilities of selection bias of the panel members. In addition, although the list of the promising and evidence-based VR practices identified by the respondents is formidable, it should not be assumed that the services identified are exhaustive in nature. Examples may represent items with which participants were most familiar, and a different sample may have identified somewhat different, yet equally important services. The definition and description of promising and evidence based practices in VR service delivery may have been similarly impacted.

As researchers, our experience in the state VR agencies and community-based rehabilitation agencies may have conditioned us to examine specific responses reinforcing preconceived ideas of what defines EBP and what services are most important. Care was taken to limit this effect, but the potential remains that responses confirming researcher assumptions were somehow viewed as more important. The list of 26 promising and evidence-based practices represent an assortment of actual VR practices along with generic therapeutic modalities, and other auxiliary services may be too cluttered to be evaluated accurately by the experts. Finally, some of the experts in this study are not rehabilitation counselor educators or researchers and may not be familiar with the available evidence for some practices, which may have prevented them from accurately responding to the question about “level of evidence” of VR practices.

4.2 Implications for rehabilitation counseling

The use of scientific evidence to guide rehabilitation counseling practices and policymaking has the potential to improve psychosocial and employment outcomes for people with disabilities. In terms of practice, it has the benefit of helping VR counselors fulfill their ethical obligations to their clients by protecting them from harm (non-maleficence), improving effectiveness and efficiency in utilization of scarce resources (justice), and empowering consumers to exercise self-determination and informed choice (autonomy) (Chan et al., 2009). Findings of this study identified several practices that have great potential to improve consumer-counselor working relationship (e.g., MI), psychosocial outcomes (e.g., person-centered therapy and cognitive behavior therapy), motivation to work (e.g., MI and benefits counseling), and job placement and job retention (e.g., supported employment, IPS, and demand-side employment strategies). Rehabilitation counselors in state agencies should include these promising and evidence-based VR practice in their repertoire of treatments and implement the evidence-based VR practices that are appropriate for their clients.

From a training perspective, state VR agencies can use our findings to prioritize their in-service training and professional development agenda for rehabilitation counselors in their efforts to encourage the implementation of EBP in state VR agency settings. At the pre-service level, academic programs that train rehabilitation counselors at the graduate and undergraduate levels can use these data to inform their respective curricula in relation to effective employment strategies so that new graduates have specific training and knowledge about these evidence-based and promising practices and will be able to implement them with consumers within their practice setting upon graduation. Finally, at the doctoral level, students can study and learn about the current state of evidence-based practices in rehabilitation counseling and be prepared to not only teach these practices to future students, but also to contribute to our ongoing research efforts to identify what services or practices are most effective with specific populations under certain conditions.

4.3 Implication for rehabilitation research

Researchers can use the findings of this study to expend research efforts to further validate practices that have the highest potential to improve VR outcomes. In addition, researchers and program evaluators at the state level, in partnership with state agency VR personnel, can design and implement studies to provide quantitative and qualitative evidence regarding promising and emerging practices that are being designed and implemented within state agencies. Although the experts in the present study identified several evidence-based VR practices, most of the VR research conducted over the past several decades has been descriptive in nature with many efforts attempting to determine relationships among existing data, rather than taking a purposeful approach to define what types of intervention or services work best with specific populations, under specific conditions (Chan et al., 2009; Paul, 1967). The lack of randomized controlled trial (RCT) studies in VR research remains a major barrier to implementing EBP in VR and will need to be comprehensively addressed in future research initiatives (Chan et al., 2009; Leahy et al., 2014). For example, Chan and colleagues (2009) have advocated the use of the World Health Organization (WHO) International Classification of Functioning, Disability and Health (ICF) framework to formulate a systematic research agenda to validate evidence-based VR practices. Finally, as indicated by Saunders, Leahy, McGlynn, and Estrada-Hernandez (2006) and further confirmed by this Delphi study process, more intervention related studies (e.g., experimental, quasi experimental) and fewer ad hoc studies regarding employment and other related outcomes of the rehabilitation process are needed to advance the EBP movement in rehabilitation counseling. Data, such as effect size and confidence intervals, need to be routinely included in research articles to conduct high quality systematic reviews and meta-analysis. To meaningfully address knowledge translation and the development of EBP, there needs to be more replication and extensions of previous research in order to build coherent, rigorous lines of research that serve to specifically inform practice and policy in the future (Fleming et al., 2013).

5 Conclusion

This Delphi study utilized a panel of national experts in VR research, rehabilitation counseling, and administration to help identify promising or evidence-based VR practices that are both relevant to VR and supported by some degree of scientific rigor. Our findings support that state VR agencies are beginning to recognize the relevance of some of the evidence-based practices such as MI, working alliance, supported employment, IPS, and skills training in VR service delivery practices. Experts also identified several promising employment intervention practices as more relevant to VR than psychosocial and counseling interventions, perhaps because employment interventions are vital to the implementation of the mandates of the WIOA and the amendments to the Rehabilitation Act. However, the low regard for psychosocial and counseling interventions is disconcerting as mental health, physical health, and well-being are all important requisite factors to successful employment and career advancement outcomes.

Conflict of interest

None to report.

Footnotes

Appendix A

Brief Description of VR Services as Included in the Study

1. Assertive Community Treatment (ACT): Community team based approach specializing in customized services and case management for people with mental illness.

2. Assistive Technology: Application of rehabilitation engineering services and assistive technology services to meet the needs of, and address the barrier confronted by individuals with disabilities.

3. Benefits Counseling: Benefits counseling program for consumers who receive SSI or SSDI.

4. Brief Solution Focused Therapy: Goal directed, collaborative counseling approach focusing on problem solving and solutions leading to change.

5. Cognitive Behavioral Therapy: Short term therapeutic approach to problem solving and modification of thinking and behavior.

6. Community Based Work Program (Adults): On-site training program tailored to an employer’s specific hiring needs. The training takes place in the actual work area in the workplace, and may include a trained job coach located on-site to provide additional support in training.

7. Customized Employment: Flexible process designed to personalize the employment relationship between a job candidate and an employer in a way that meets the needs of both. It is based on an individualized match between the strengths, conditions, and interests of a job candidate and the identified business needs of an employer.

8. Demand Side Employment Strategies: Strategies focusing on the employer and work environment, emphasizing the preparation of people with disabilities for employment (e.g. Diversity Training, ADA & Job Accommodation Consultation).

9. Dual Customer Approach: Both employers and clients are considered to be customers with an approach of meeting business need through referral of qualified VR applicant.

10. Family Involvement & Support: Families, community groups, and individuals partnering to maximize resources and expertise in addressing client issues.

11. Health Promotion Interventions: Interventions to manage disability and promote quality of life.

12. Individual Placement & Support (IPS) Model of Supported Employment: Supported employment strategies to help people with mental illness obtain employment (Dartmouth Model).

13. Job Club: Formal behavioral support group for job seekers that assists participants with job-hunt activities and helps with job search support and advice (Azerin Model).

14. Motivational Interviewing (MI): Goal directed counseling technique, which seeks to help a client overcome ambivalence to change.

15. On the Job Training: Training in a specific job skill by a prospective employer, either paid or unpaid, to help an individual remain in the same or similar job upon successful completion of the training program.

16. Online Community of Practice for VR Counselors: Exchanges information collaboratively and contributes solutions to identified challenges aimed at improving employment outcomes for PWD.

17. Person Centered Planning (PCP): A process for planning and support for an individual with a disability receiving services that builds upon the individual’s capacity to engage in activities that promote community life, honoring preferences, choices, and abilities.

18. Person Centered Therapy: Non-directive form of therapy based on counselor congruence, unconditional positive regard and empathic understanding for the client.

19. Positive Psychology Interventions: Psychological interventions designed to facilitate positive emotions, wellbeing, growth, and happiness.

20. Project Search: Program that provides competitive employment opportunities for individuals with disabilities and places these individuals in internships at local businesses with the goal of securing non-traditional employment for persons with disabilities (Transition Service/Supported Employment for those with intellectual disabilities).

21. Social Media: Use of media sites, often available to the public (e.g. Linked-In, Facebook, etc.) where someone posts information and/or comments seeking employment opportunities and information.

22. Social Skills Training: Systemically teaching new interpersonal skills through social learning strategies.

23. Soft Skills Training: Training programs with a standardized curriculum designed to change behavior and improve employability through an interactive training approach

24. Tele-health/Tele-Rehabilitation: The use of electronics information and telecommunications technologies (using internet and mobile platforms) to support long distance clinical health care, patient and professional health- related education public health and health administration.

25. Transition Services: A coordinated set of activities for a student with a disability designed to be a results-oriented process facilitating movement from the school to post-secondary activities and/or employment (e.g. early intervention, interagency collaboration, summer work experiences).

26. Working Alliance: The affective relationship between the client and the counselor, which makes it possible for the client to work purposefully in the counseling relationship and for the counselor and client to agree on goals and tasks.

Funding/Acknowledgments

The contents of this article were developed with support from the Rehabilitation Research and Training Center on Evidence-Based Practice in Vocational Rehabilitation (RRTC-EBP VR) at the University of Wisconsin-Madison and the University of Wisconsin-Stout and with funding provided by the U.S. Department of Health and Human Services, National Institute on Disability, Independent Living and Rehabilitation Research (Grant H133B100034). The ideas, opinions, and conclusions expressed, however, are those of the authors and do not represent recommendations, endorsements, or policies of the U.S. Department of Health and Human Services.

References

Anderson

C. A.

, Leahy

M. J.

, Valle

R. D.

, Sherman

, & Tansey

T. N.

(2014). Methodological application of multiple case study design using modified consensual qualitative research (CQR) analysis to identify best practices and organizational factors in the public rehabilitation program, Journal of Vocational Rehabilitation, 41, 87–98.

Campbell

, Bond

G. R.

, & Drake

R. E.

(2011). Who benefits from supported employment: A meta-analytic study, Schizophrenia Bulletin, 37(2), 370–380.

Chan

, & Berven N. L. Thomas

K. R.

(Eds.). (2015), Counseling theories and techniques for rehabilitation and mental health professionals. New York, NY: Springer Publishing Company.

Chan

, Chronister

, & Cardoso

E. D. S.

(2009) An introduction to evidence-based practice approach to psychosocial interventions for people with chronic illness and disability. In

Chan ,

Da Silva Cardoso &

J. A.

Chronister (Eds.), Understanding psychosocial adjustment to chronic illness and disability: A handbook for evidence-based practitioners in rehabilitation; (pp. 3–19). New York, NY: Springer Publishing Company.

Chan

, Keegan

, Muller

, Kaya

, Flowers

, & Iwanaga

(2016) Evidence-based practice and research in rehabilitation counseling. In Marini & I.. Stebnicki

M.A

(Eds.), The professional counselor’s desk reference (2nd ed., pp. 605–610). New York, NY: Springer Publishing Company.

Chan

, Sung

, Muller

, Wang

C. C.

, Fujikawa

, & Anderson

C. A.

(2011) Evidence-based practice and research utilization. In Maki,&D. Tarvydas

V. M.

(Eds.), Professional practice of rehabilitation counseling (pp. 391–412). New York, NY: Springer Publishing Company.

Chan

, Shaw

, McMahon

B. T.

, Koch

, & Strauser

(1997). A model for enhancing consumer-counselor working relationships in rehabilitation, Rehabilitation Counseling Bulletin, 41, 122–137.

Chan

, Tarvydas

, Blalock

, Strauser

, & Atkins

B. J.

(2009). Unifying and elevating rehabilitation counseling through model-driven, diversity-sensitive evidence-based practice, Rehabilitation Counseling Bulletin, 52, 114–119.

Cook

D. W.

(2004) Counseling people with physical disabilities. In Chan

, Berven, &K. R. (Eds) Counseling theories and techniques for rehabilitation health professionals (pp. 328–341). New York, NY: Springer Publishing Company.

10.

Del Valle

R. J.

, Leahy

M. J.

, Sherman

, Anderson

C. A.

, Tansey

, & Schoen

(2014). Promising best practices that lead to employment in vocational rehabilitation: Findings from a four-state multiple case study, Journal of Vocational Rehabilitation, 41, 99–113.

11.

Delin

B. S.

, Hartman

E. C.

, & Sell

C. W.

(2012). The impact of work incentive benefits counseling on employment outcomes: Evidence form two return-to-work demonstrations, Journal of Vocational Rehabilitation, 36, 97–107.

12.

Dutta

, Gervey

, Chan

, Chou

C. C.

, & Ditchman

(2008). Vocational rehabilitation services and employment outcomes for people with disabilities: A United States study, Journal of Occupational Rehabilitation, 18(4), 326–334.

13.

Fleming

A. R.

, Del Valle

, Kim

, & Leahy

M. J.

(2013). Best practice models of effective vocational rehabilitation service delivery in the public rehabilitation program: A review and synthesis of the empirical literature, Rehabilitation Counseling Bulletin, 56(3), 146–159.

14.

Horvath

A. O.

, Del Re

A. C.

, Flückiger

, & Symonds

(2011). Alliance in individual psychotherapy, Psychotherapy, 48(1), 9–16.

15.

Hsu

C. C.

, & Standford

B. A.

(2007). The Delphi technique: Making sense of consensus, Practical Assessment, Research & Evaluation, 12(10), 1–8.

16.

Leahy

M. J.

, Chan

, Lui

, Rosenthal

, Tansey

T. N.

, Wehman

, & Menz

F. E.

(2014). An analysis of evidence-based best practices in the public vocational rehabilitation program: Gaps, future directions, and recommended steps to move forward, Journal of Vocational Rehabilitation, 41, 147–163.

17.

Leahy

M. J.

, Thielsen

V. A.

, Millington

M. J.

, Austin

, & Fleming

(2009). Quality assurance and program evaluation: Terms, models, and applications, Journal of Rehabilitation Administration, 33(2), 69–82.

18.

Manthey

, Brooks

, Chan

, Hedenblad

, & Ditchman

(2015) Motivational interviewing in rehabilitation health settings. In Chan

, Berven,&K. (Eds.), Counseling theories and techniques for rehabilitation and mental health professionals (2nd ed., pp. 247–278). New York, NY: Springer Publishing Company.

19.

Martin

, West-Evans

, & Connelly

(2010). Vocational rehabilitation: Celebrating 90 years of careers and independence, American Rehabilitation, 15–18.

20.

Moorhead

S. A.

, Hazlett

D. E.

, Harrison

, Carroll

J. K.

, Irwin

, & Hoving

(2013). A new dimension of health care: Systematic review of the uses, benefits, and limitations of social media for health communication, Journal of Medical Internet Research, 15(4), 118–133.

21.

Mueser

K. T.

, Drake

R. E.

, & Bond

G. R.

(2016). Recent advances in supported employment for people with serious mental illness, Current Opinion in Psychiatry, 29(3), 196–201.

22.

Nazarov

Z. E.

(2013), Can benefits and work incentives counseling be a path to future economic self-sufficiency for SSI/SSDI beneficiaries?Chestnut Hill, MA: Center for Retirement Research at Boston College.

23.

Paul

(1967). Strategy of outcome research in psychotherapy, Journal of Counseling Psychology, 31(2), 109–188.

24.

Rubin

S. E.

, Chan

, & Thomas

D. L.

(2003). Assessing changes in life skills and quality of life resulting from rehabilitation services, Journal of Rehabilitation, 69(3), 4–9.

25.

Ruiz-Quintanilla

S. A.

, Weathers II

R. R.

, Melburg

, Campbell

, & Madi

(2006). Participation in programs designed to improve employment outcomes for persons with psychiatric disabilities: Evidence from the New York WORKS demonstration project, Social Security Bulletin, 66(2), 49–79.

26.

Saunders

J. L.

, Leahy

M. J.

, McGlynn

, & Estrada-Hernandez

(2006). Predictors of employment outcomes for persons with disabilities: An integrative review of potential evidence-based factors, Journal of Applied Rehabilitation Counseling, 37(2), 3–20.

27.

Sherman

S. G.

, Leahy

M. J.

, Del Valle

, Anderson

C. A.

, Tansey

T. N.

, & Lui

(2014). Organizational and cultural factors that promote creative best practices in the public rehabilitation program: Findings from a four-state multiple case study, Journal of Vocational Rehabilitation, 41(2), 115–125.

28.

Tansey

T. N.

, Bezyak

, Chan

, Leahy

M. J.

, & Lui

(2014). Social-cognitive predictors of readiness to use evidence-based practice: A survey of state vocational rehabilitation counselors, Journal of Vocational Rehabilitation, 41(2), 127–136.

29.

Tansey

T. N.

, Bezyak

J. L.

, Iwanaga

, Anderson

, & Ditchman

(2016). Path analysis of a self-determination model of work motivation in vocational rehabilitation. A paper presented at the 2016 National Council on Rehabilitation Education Spring Conference, Newport Beach, CA.

30.

Tremblay

, Smith

, Xie

, & Drake

R. E.

(2006). Effect of benefits counseling services on employment outcomes for people with psychiatric disabilities, Psychiatric Services, 57(6), 816–821.

31.

U.S.Department of Education. (2014) The Workforce Innovation and Opportunity Act overview of Title IV: Amendments to the Rehabilitation Act of 1973Retrieved from https://www2.ed.gov/about/offices/list/osers/rsa/publications/wioa-changes-to-rehab-act.pdf

32.

U.S. Government Accountability Office. (2005), Vocational rehabilitation—Better measures and monitoring could improve the performance of the VR Program (GAO-05-865). Washington, DC: Author.

33.

U.S. Office of Management and Budget and Federal agencies. (2006) The PART process. [Electronic version].Retrieved from http://wwww.whitehouse.gov/omb/.expectmore/

34.

Vazques-Ramso

, Leahy

M. J.

, & Hernandez

(2007). The Delphi method in rehabilitation counseling research, Rehabilitation Counseling Bulletin, 50(2), 111–118.

35.

Wagner

C. C.

, & McMahon

B. T.

(2004). Motivational interviewing and rehabilitation counseling practice, Rehabilitation Counseling Bulletin, 47(3), 152–161.

36.

Wampold

B. E.

(2001) The great psychological debate. Mahwah, NJ: Lawrence Erlbaum Associates .

37.

Wampold

B. E.

, & Imel

Z. E.

(2015), The great psychotherapy debate: The research evidence for what works in psychotherapy (2nd ed.). New York, NY: Routledge.

38.

Wilson

K. B.

, Turner

, & Jackson

R. L.

(2002). Vocational rehabilitation services received after successful closure: A comparison by race, Journal of Applied Rehabilitation Counseling, 33(1), 26–34.

	Relevance			Level of Evidence
VR Practices	Rank	M	SD	Rank	M	SD
Secondary Transition Services	1	4.77	0.43	12	2.50	1.06
Assistive Technology	2	4.54	0.65	14	2.46	0.93
IPS Supported Employment	3	4.46	0.71	3	3.39	1.27
On-the-Job Training	4	4.42	0.50	15.5	2.35	0.90
Demand-Side Employment Strategies	5	4.31	0.68	21	2.00	0.85
Benefits Counseling	6	4.23	0.77	19	2.14	0.97
Motivational Interviewing	7	4.19	0.69	4	3.17	1.19
Customized Employment	8	4.19	0.63	20	2.08	1.06
Community Based Work Program	9	4.19	0.63	18	2.30	0.97
Working Alliance	10	4.15	0.83	6	3.11	0.74
Person Centered Planning (PCP)	11	4.04	0.60	22.5	1.96	0.86
Soft Skills Training	12	4.00	0.76	22.5	1.96	0.81
Dual Customer Approach	13	4.00	0.57	26	1.45	0.74
Job Club	14	3.88	0.86	15.5	2.35	1.03
Family Involvement &Support	15	3.88	0.82	17	2.32	0.89
Social Skills Training	16	3.73	0.78	13	2.48	0.90
Cognitive Behavioral Therapy	17	3.58	0.64	1	3.54	1.22
Assertive Community Treatment (ACT)	18	3.54	0.65	2	3.50	1.01
Positive Psychology Interventions	19	3.42	0.64	8	2.86	1.13
Project Search	20	3.42	0.90	11	2.52	1.29
Solution-Focused Brief Therapy	21	3.19	0.75	5	3.15	1.04
Health Promotion Interventions	22	3.12	0.77	10	2.60	1.14
Person Centered Therapy	23	3.08	0.89	7	2.99	1.12
Online Community of Practice	24	3.04	0.82	25	1.57	1.16
Social Media	25	2.92	0.85	24	1.61	1.04
Tele-Health/Tele-Rehabilitation	26	2.62	0.80	9	2.83	1.34
Mean of VR Practice	3.80	0.72	2.51	1.02