Promising and evidence-based practices in vocational rehabilitation: Results of a national Delphi study

1.1 Purpose of the study

Leahy et al.’s (2014) study represents an important step to identify promising and evidence-based VR practices that are used by state VR agencies to improve psychosocial and employment outcomes of people with disabilities. However, their findings were based on four state agencies and there may be other empirically supported interventions for improving the effectiveness of VR service delivery practices. Therefore, we convened a panel of VR experts using Delphi procedures to determine whether there is national consensus regarding the applicability of the promising or evidence-based VR practices identified by Leahy et al. (2014) for improving psychosocial and employment outcomes of people with disabilities receiving services from state VR agencies. In addition, the experts in our study were asked to identify additional empirically supported psychosocial and vocational interventions that were not included in Leahy et al.’s (2014) multi-state case study. The purpose of this article is to describe the outcomes of this Delphi study.

2.1 Participants

The present study used the Delphi technique, a widely used and accepted method for gathering data from respondents within their domain of expertise (Vazques-Ramos, Leahy, & Hernandez, 2007), to obtain consensus from a panel of VR experts regarding promising or evidence-based VR practices that could be used to improve rehabilitation outcomes for people with disabilities receiving services from state VR agency settings. Concerted efforts were taken to identify the strongest panel possible for the Delphi Study, which consisted of exemplary rehabilitation educators and researchers, rehabilitation counselors/supervisors, and administrators of state VR agencies. These national experts were selected based on their knowledge and clinical experience with VR services relevant to the rehabilitation process, as well as their expertise and research scholarship in EBP related to VR services and interventions. Nominations were sought from all the researchers associated with the Research and Training Center on Evidence-Based Practice in Vocational Rehabilitation (RRTC-EBP VR), its advisory council members, and other stakeholders to generate a comprehensive list of possible subject matter experts (SMEs; N = 35) for the study. These experts were contacted via email and all (100%) agreed to participate. The panel comprised 16 (45.7%) women and 19 (54.3%) men, with reported years of experience in VR ranging from 5 years to 44 years of service (M = 30.14, SD = 10.17). Twenty-three SMEs (65.7%) reported a doctoral degree and 10 reported a master’s degree as their highest educational credential (n = 10, 28.6%), and most reported their academic training was within the closely aligned fields of rehabilitation psychology (n = 8, 22.9%), rehabilitation counseling (n = 6, 17.1%), or rehabilitation counselor education (n = 4, 11.4%). Other educational specialties included business, human resource management, education, social welfare, and special education. Panelists primarily reported working in either academic/university settings (n = 21, 60.0%) or state VR agencies (n = 10, 28.6%), with the remaining four reporting employment in non-profit rehabilitation settings as a disability policy consultant, as a contractor for the Social Security Ticket to Work, or as an administrator with a Tribal Vocational Rehabilitation program.

2.2 Measures

The standard Delphi technique consists of a series of questionnaires about an important question or problem that are usually mailed to a group of individuals who are considered experts in a particular subject matter area (Vazques-Ramos et al., 2007). It involves recruitment of a panel of experts in a particular field and repeated questioning of each group member using at least two sequential questionnaires, called rounds. The statistical summary prepared after each round is used for the next round of questions to provide the needed feedback for experts to consider revising their views through awareness of overall responses from other experts in the panel. For each item in the questionnaire, the statistical analysis includes some combination of the mean, median, and mode of the panelists’ responses and at least one measure of dispersion (e.g., standard deviation or inter-quartile range). The use of these iterative procedures is common in Delphi studies to achieve group consensus among experts, with consensus being reflected in the convergence of variance or standard deviations in subsequent iterations (Hsu & Standford, 2007).

The Round 1 questionnaire

for the present study comprises one open-ended question asking the SMEs in the expert panel to provide specific examples of promising or evidence-based VR practices that could be used to improve employment outcomes for individuals with disabilities in state agency settings. Promising practice was defined using the definition provided by Leahy et al.’s (2014) multi-state case study. The Round 2 questionnaire was developed based on (a) a comprehensive literature review of employment interventions for people with disabilities reported by Fleming, Del Valle, Kim, and Leahy (2013); (b) the 14 promising or evidence-based VR practices identified by Del Valle et al. (2014) in their multi-state case study; and (c) specific promising or evidence-based VR practices provided by the SMEs in the Round 1 questionnaire. The promising or evidence-based practices included rehabilitation interventions that lead to successful psychosocial and employment outcomes such as supported employment, generic counseling approaches such as MI, therapeutic conditions such as working alliance that are either services that can be purchased from community-based rehabilitation agencies or counseling practices that can be used by rehabilitation counselors to improve client-counselor relationship, outcome expectations, and VR service engagement. The Round 2 questionnaire includes 26 promising or evidence-based VR practices that were triangulated from the three sources of data. See Appendix A for brief descriptions of practices described in this study. Two five-point Likert type scales were developed for the experts to rate each promising or evidence-based VR practice. The first rating scale ranging from 1 (very low relevance) to 5 (very high relevance), was used to evaluate the relevance of the 26 promising or evidence-based VR practice to effective VR service delivery. A second five-point Likert-type scale ranging from 5 (the highest evidence) to 1 (low evidence), was utilized to rate the level of scientific evidence for each promising or evidence-based VR practice item based on the hierarchy of evidence that is a core principal of EBP (Chan et al., 2016). The Round 2 and 3 questionnaires provided the SMEs with definitions for both the relevance of the promising practice or EBP for VR service delivery and level of evidence. Table 1 presents the definitions and the rating system for the hierarchy of evidence. The Round 3 questionnaire comprises the same items as the Round 2 questionnaire; however, the SMEs are provided with feedback related to their own rating on each item along with the average group ratings and dispersion information on each item (means and standard deviations) based on the Round 2 data.

The Round 1 questionnaire was emailed to the 35 SMEs who had agreed to participate and a follow-up reminder email was sent to non-respondents two weeks after the initial contact. Data for the open-ended question were gathered using Qualtrics, an online survey software tool. As noted above, findings from the Round 1 study along with the comprehensive literature review on employment interventions for people with disabilities reported by Fleming et al. (2013) and the 14 promising VR practices identified by Del Valle et al. (2014) were then used to develop the 26-item version of the Round 2 EBP VR questionnaire.

The web link for completing the Round 2 questionnaire was sent to the 35 SMEs via an introductory email. Non-respondents were contacted two weeks after the first mailing via email to remind them to complete the Delphi questionnaires and again providing them with the web link. Based on the responses obtained in Round 2, means and standard deviations were calculated for each promising or evidence-based VR practice item according to its degree of relevance and the hierarchy of scientific evidence; the Round 3 questionnaire was then developed based on those data. The web link for completing the Round 3 questionnaire was sent via email to the SMEs who had completed Round 2; a reminder email was sent to non-respondents two weeks after. The Round 3 questionnaire provided the SMEs with their own ratings of each promising or evidence-based VR practice item from Round 2 in terms of relevance and level of evidence, along with the Round 2 group means and standard deviations for each EBP item. The SMEs were asked to review their own responses and compare their responses against the group average, reconsider their responses as they saw appropriate, and then complete the Round 3 questionnaire. Data for the Delphi study were then analyzed using descriptive and inferential statistics.

4.1 Limitations

There are several limitations to consider when reviewing the results. Although great care was taken to select the SMEs, selection bias is still a possibility. Another group with different backgrounds, yet equal expertise might evaluate the relevance of the promising and evidence-based VR practices somewhat differently. To safeguard against selection bias, we requested input from the entire research staff of the RRTC-EBP VR and their respective advisory council. This broad input and the careful examination of the characteristics of each panel member nominated hopefully reduced the possibilities of selection bias of the panel members. In addition, although the list of the promising and evidence-based VR practices identified by the respondents is formidable, it should not be assumed that the services identified are exhaustive in nature. Examples may represent items with which participants were most familiar, and a different sample may have identified somewhat different, yet equally important services. The definition and description of promising and evidence based practices in VR service delivery may have been similarly impacted.

As researchers, our experience in the state VR agencies and community-based rehabilitation agencies may have conditioned us to examine specific responses reinforcing preconceived ideas of what defines EBP and what services are most important. Care was taken to limit this effect, but the potential remains that responses confirming researcher assumptions were somehow viewed as more important. The list of 26 promising and evidence-based practices represent an assortment of actual VR practices along with generic therapeutic modalities, and other auxiliary services may be too cluttered to be evaluated accurately by the experts. Finally, some of the experts in this study are not rehabilitation counselor educators or researchers and may not be familiar with the available evidence for some practices, which may have prevented them from accurately responding to the question about “level of evidence” of VR practices.

4.2 Implications for rehabilitation counseling

The use of scientific evidence to guide rehabilitation counseling practices and policymaking has the potential to improve psychosocial and employment outcomes for people with disabilities. In terms of practice, it has the benefit of helping VR counselors fulfill their ethical obligations to their clients by protecting them from harm (non-maleficence), improving effectiveness and efficiency in utilization of scarce resources (justice), and empowering consumers to exercise self-determination and informed choice (autonomy) (Chan et al., 2009). Findings of this study identified several practices that have great potential to improve consumer-counselor working relationship (e.g., MI), psychosocial outcomes (e.g., person-centered therapy and cognitive behavior therapy), motivation to work (e.g., MI and benefits counseling), and job placement and job retention (e.g., supported employment, IPS, and demand-side employment strategies). Rehabilitation counselors in state agencies should include these promising and evidence-based VR practice in their repertoire of treatments and implement the evidence-based VR practices that are appropriate for their clients.

From a training perspective, state VR agencies can use our findings to prioritize their in-service training and professional development agenda for rehabilitation counselors in their efforts to encourage the implementation of EBP in state VR agency settings. At the pre-service level, academic programs that train rehabilitation counselors at the graduate and undergraduate levels can use these data to inform their respective curricula in relation to effective employment strategies so that new graduates have specific training and knowledge about these evidence-based and promising practices and will be able to implement them with consumers within their practice setting upon graduation. Finally, at the doctoral level, students can study and learn about the current state of evidence-based practices in rehabilitation counseling and be prepared to not only teach these practices to future students, but also to contribute to our ongoing research efforts to identify what services or practices are most effective with specific populations under certain conditions.

4.3 Implication for rehabilitation research

Researchers can use the findings of this study to expend research efforts to further validate practices that have the highest potential to improve VR outcomes. In addition, researchers and program evaluators at the state level, in partnership with state agency VR personnel, can design and implement studies to provide quantitative and qualitative evidence regarding promising and emerging practices that are being designed and implemented within state agencies. Although the experts in the present study identified several evidence-based VR practices, most of the VR research conducted over the past several decades has been descriptive in nature with many efforts attempting to determine relationships among existing data, rather than taking a purposeful approach to define what types of intervention or services work best with specific populations, under specific conditions (Chan et al., 2009; Paul, 1967). The lack of randomized controlled trial (RCT) studies in VR research remains a major barrier to implementing EBP in VR and will need to be comprehensively addressed in future research initiatives (Chan et al., 2009; Leahy et al., 2014). For example, Chan and colleagues (2009) have advocated the use of the World Health Organization (WHO) International Classification of Functioning, Disability and Health (ICF) framework to formulate a systematic research agenda to validate evidence-based VR practices. Finally, as indicated by Saunders, Leahy, McGlynn, and Estrada-Hernandez (2006) and further confirmed by this Delphi study process, more intervention related studies (e.g., experimental, quasi experimental) and fewer ad hoc studies regarding employment and other related outcomes of the rehabilitation process are needed to advance the EBP movement in rehabilitation counseling. Data, such as effect size and confidence intervals, need to be routinely included in research articles to conduct high quality systematic reviews and meta-analysis. To meaningfully address knowledge translation and the development of EBP, there needs to be more replication and extensions of previous research in order to build coherent, rigorous lines of research that serve to specifically inform practice and policy in the future (Fleming et al., 2013).