Practice motivated research: Application of an evidence-informed prognostic model in vocational rehabilitation to increase the chance of employment at closure

Abstract

BACKGROUND:

We argue that relevant research must focus on a problem of practice. We demonstrate this approach by developing a new product for practice to help counselors make informed vocational rehabilitation decisions. With only 18.7% of persons with disabilities employed in 2017, an accurate and simple prognostic tool could improve the effectiveness of the Individualized Plan for Employment and thus assist persons with disabilities to live independent lives.

OBJECTIVE:

To demonstrate the validity and practical relevance of a counseling decision support tool that accurately predicts clients’ employment outcomes based on demographic characteristics and vocational rehabilitation factors.

METHODS:

Using a historical sample of 53,629 persons with disabilities who completed vocational rehabilitation in a state agency, we derived our prediction model using logistic regression with 90-day employment as the outcome.

RESULTS:

The final prognostic model was derived from 12 client demographics and 20 vocational rehabilitation factors. The model correctly classified the outcome for 72% of the clients and demonstrated strong calibration and discrimination. The resulting app is available here: www.ablescore.com.

CONCLUSIONS:

AbleScore accurately classifies client probability of employment at closure. The app therefore has immediate application in providing evidence-informed rehabilitation counseling to people with disabilities to improve the odds of employment at closure.

Keywords

Disability vocational rehabilitation employment prognostic tool research-practice gap AbleScore

1 Introduction

Despite the growing and spirited calls for the application of research evidence in the practice of vocational rehabilitation (VR), and the considerable resources devoted to evidence-based practice (EBP) and Knowledge Translation (KT) movements, a consistent conclusion in the literature and from the field is that available VR research evidence is rarely used by VR practitioners (Bennett et al., 2003; Chan et al., 2010; Dubouloz et al., 1999; Graham et al., 2013; Humphris et al., 2000; Leahy, Thielsen, Millington, Austin & Fleming, 2009). Even when research evidence makes it into practice, studies have shown no change in practitioners’ attitudes toward adoption and use of such evidence in practice (Graham et al., 2013; Menon et al., 2009). For example, an important study by Graham et al. (2013) shows that while 84% of practitioners (counselors) reported that they value research for practice, more than 40% found research evidence in academic papers impractical to implement or, frankly, they find research evidence irrelevant and somewhat detached from the real-world challenges they face in their local practices. Evidence from the history of the research-practice gap in healthcare suggests that adoption of research evidence in VR practice may not improve soon. For example, Westfall and colleagues (2007) found only 14 percent of research in healthcare is translated into practice and it often takes a long time to complete the research-to-practice cycle. Morris et al. (2011) have shown that in medicine, it has taken over 17 years for discoveries to be translated into practice.

Scholars and policymakers who have studied this issue have spent most of their energy debating the gap and attempting to explain why it exists. Scholars suggest that differences in cultural norms, rules, communication styles, and goals between researchers and practitioners make knowledge exchanges and interactions difficult (Graham et al., 2013; Salbach et al., 2007; Sherman et al., 2014; Winch, Henderson, & Creedy, 2005; O’Donnell, 2004). For example, Graham et al. (2013) pointed to a lack of state agency support and lack of funding for EBP and KT initiatives. Martin and Martin (1989) suggested that research evidence is difficult for counselors to comprehend and mostly irrelevant to their practice. O’Donnell (2004) cited counselors’ lack of time due to excessive caseloads, limited access to research and lack of expertise in evaluating and applying available evidence effectively in practice. Further, Salbach et al. (2007) argued that the stock of rigorous research evidence is limited because VR studies are more descriptive than empirical, making it difficult in determining best evidence. Others lay the blame on counselors’ negative attitudes toward the use of academic evidence in practice (Martin & Martin, 1989; Winch, Henderson, & Creedy, 2005).

Not surprising, practitioners have a different take. They point to several problem areas. First and foremost, is that research evidence is not relevant to problems they face in their situated practices (Armstrong et al., 2007), underscoring the lack of interest on the part of scholars to do research that is practice-oriented rather than theory-driven. Many practitioners find the language of research incomprehensible, and difficulties in interpreting technical/academic jargon is often cited (Armstrong et al., 2007; Martin & Martin, 1989). Another commonly mentioned issue is that even when relevant, the evidence is decontextualized, thus impractical and difficult to implement in a different setting.

Sadly, while this debate is raging, persons with disabilities are not getting the best VR interventions to help them find competitive employment. Regardless of the reasons cited by the debaters, this is unfortunate given the over $38 billion spent over the last decade in the state-federal vocational rehabilitation program (Department of Education: Rehabilitation Services and Disability Research budget allocation), and this despite $142 million specifically allocated in 2017 by the Administration for Community Living (ACL) to fund new research and translate existing research evidence into practice (ACL 2017 budget allocations). Notwithstanding this large investment, the result is the same: in 2008 the employment rate for persons with disabilities was about 39%, while today (ten years later) it is 36% (Erickson, et al., 2017; U.S. Bureau of Labor Statistics, 2017). While progress has been made legislatively (e.g., the Workforce Innovation and Opportunity Act [WIOA]), one might question whether persons with disabilities are reaping the full benefits of our investments in the Rehabilitation Services Administration (RSA) to help them find jobs and live independent lives.

To address the problem of making research more relevant to practice, the state-federal rehabilitation system has promoted knowledge translation with the hope to increase the uptake of research into practice and improve rehabilitation rates. The U.S. Department of Education, one of the champions of KT, defines KT as “the multidimensional, active process of ensuring that new knowledge gained through the course of research ultimately improves the lives of people with disabilities and furthers their participation in society” (U.S. Department of Education, 2006, p. 8195). The National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR), whose key mission is to generate new knowledge and promote its effective use to improve the abilities of people with disabilities to achieve long-term outcomes, emphasized that “the KT process is active, as it not only accumulates information, but it also filters the information for relevance and appropriateness, and recasts that information in language useful and accessible for the intended audience.” The Knowledge-to-Action (KTA) approach, a variant of KT, has also been suggested as an appealing solution to bridge the gap between research and practice in VR (Graham et al., 2006; Leahy et al., 2014; Sudsawad, 2007). Unfortunately, there’s limited evidence to support or dispute the efficacy of knowledge translation strategies in rehabilitation. However, examining KT literature through the lens of practitioner-scholarship elucidates certain obstacles, which we next discuss.

First, implied in the NIDILRR definition of KT is the assumption that research evidence can indeed be filtered, recast, translated and made relevant to problems faced by practitioners in their situated practice. But an important point that is commonly overlooked in the KT literature, yet bears significant implications for applying evidence in practice, is that if the evidence, at its creation, is irrelevant to problems faced by practitioners in the field, relevance cannot simply be reengineered and added later. Research evidence and practice relevance cannot be separated empirically and united practically by filtering, recasting and translating. This is because the knowledge to be translated, at its creation, represents a particular reality, a specific worldview, influenced by the social context in which it was created, which often is distinct from the practitioner’s reality and worldview, and the context in which practice is embedded. Trying to translate scientific knowledge, which sometimes lacks relevance in practice, into practice knowledge is akin to force-fitting a square peg in a round hole. It is difficult, costly, time consuming and often wasteful. We argue that research evidence and practice relevance are mutually constituted, and therefore must be designed and integrated at the onset of a research project, not after evidence has been created. Thus, rather than starting with a research question, this view starts with a problem of practice (an important real-world problem acknowledged by members of a practice community), and through recurrent interaction between researchers, practitioners and other stakeholders, the problem of practice (PoP) is reconstituted as a research problem and a research question to address the PoP using rigorous academic methods.

A second salient concern pointed out by Doane and Varcoe (2008) is the conception of KT elements as dichotomies separating knower (researcher) from doer (practitioner), knowing (evidence) from doing (practice), and research context from practice context, overlooking the critical interrelationship between these elements in the production and application of research evidence in situated practice. In a parallel concern, Poole (2008) points to the oversight of extant KT literature that implies scholars are ‘experts’ translating knowledge without consideration for the grounded experience of the practitioner. Kitson (2008) and Reimer-Kirkham et al. (2007) warned that the KT assumption that knowledge can be decontextualized, repackaged and translated as a neutral, discrete entity is a problematic underpinning of the knowledge-to-action movement. We argue that the researcher and practitioner are co-creators of evidence, grounded in the context of practice. This partnership is achieved through intentional alignment of goals, clarification of roles and responsibilities, collaborative interaction and mutual adjustments of expectations.

A third frequently identified problem with KT is the lack of clarity on the end product (deliverable). A weakness commonly cited by practitioners is that, even after research evidence is translated, many findings were not readily fit-for-use in practice (Groah et al., 2009). The reason is that the starting point of KT and KTA is knowledge creation, not the problem faced by practitioners in the field. So KT’s primary focus is translating evidence to knowledge, but not translating evidence to a product to solve a problem. As such, the translated knowledge needs another translation by another knowledge translator to make it into the practice domain. As pointed out by Dijkers (2016), what has emerged is a multimillion-dollar cottage industry debating what constitutes research evidence and which of the never-ending iterations of KT framework should be adopted. We argue that if the goal of KT (not of science in general) is to create knowledge fit for practice, then the end product must be some practicable artifact. In the case of VR, perhaps such an artifact would be a tool counselors can implement to solve a rehabilitation problem in their situated practice; rather than just filtered, recast or translated knowledge.

A fourth handicap is availability of rigorous VR evidence to translate. A primary assumption underlying adoption of KT in a context such as VR, is that there is robust inventory of validated and accepted VR evidence derived from rigorous scientific research. However, as pointed out by Salbach et al. (2007), Groah et al (2009) and Tsang, et al. (2000), the stock of rigorous research evidence in the VR discipline is limited because VR studies are more descriptive than empirical, making it difficult to determine the best evidence for translation. Even when evidence exists, interpreting and translating such evidence into practice is often impractical. Because researchers strive for objectivity and generalizability (Bacharach, 1989; Sutton & Staw, 1995), while practitioners focus on problem specificity and ease of implementation (Van de Ven & Johnson, 2006), how to achieve alignment around a shared goal is a crucial missing link in the chain of activities defined in KT. Because scholars (e.g., Chan, Wang, Muller, & Fitzgerald, 2011; GAO, 2005, 2007) have suggested variation in state VR agency outcomes that can be attributed to differences at the individual level, program execution, contextual and economic characteristics of the state, and differences in performance of the VR agencies themselves, interpreting and translating generalized evidence to solve specific practice problems has not yielded expected results. Therefore, we argue that right from the onset of a research project (at least one claiming implications for practice), the agenda must be research that is mutually beneficial to the practitioner’s goal of practice specificity and the researcher’s goal of evidence generalizability.

Finally, as stated by Sudsawad (2007), KT in all its variants and abstractions is overwhelmingly complex, and as a multidimensional concept it demands a comprehensive understanding of its frameworks, strategies, methods, as well as the human behavior and contextual factors influencing its design and implementation. This conceptual and methodological complexity contributes to the difficulty of implementation in the field (McCluskey & Lovarini, 2005). With its numerous constructs, sub-constructs and contingencies, and limited details provided for each, knowing which combination of strategies and methods work for a particular knowledge and practice context is bewilderingly difficult. Additionally, the KT framework does not provide any behavioral change management process, or stakeholder alignment tools and guidelines on how to customize and implement research evidence in different practice environments. We argue that change leadership, stakeholder alignment and use case evaluation must be central to any KT initiative.

Given these identified obstacles, it should be no surprise that the attempt to use KT to unite evidence and practice has yielded poor results (Baumbusch et al., 2008). Because the KT movement in VR is in its formative stage, there is opportunity to reconstruct it to have a greater positive impact on creating and applying research evidence to solve rehabilitation problems to help more individuals with disabilities obtain and retain employment.

In response to this troubled context, the primary objective of this study is to propose a novel approach for creating research evidence relevant to the practice of rehabilitation counseling in a state vocational rehabilitation agency. The new approach, which we call Practice Motivated Research (PMR), starts with a problem of practice and ends with an evidence-informed solution to the problem of practice. As such, we reversed the current order of research-to-practice to a new order of practice-to-research, where a problem of practice is the motivation for research, and research evidence is both a product of and a solution to a PoP. In this paper, we applied the new approach in a situated case study to demonstrate how practice-motivated research can meet the requirements for academic rigor, practice relevance, and clinical usefulness, demonstrating how scholars and practitioners can work together to develop and apply evidence to solve important problems and make academic contributions at the same time. Finally, to enable practitioners to easily apply the evidence, we developed a user-friendly product for practice (PfP)—AbleScore—which applies prognostic/predictive analytics to help practitioners personalize rehabilitation by designing and implementing an optimal Individualized Plan for Employment (IPE) to improve the odds of employment for clients. Our proposed approach sheds light on how research should be motivated by a problem of practice, designed and conducted by a collaborative team of researchers and practitioners to give power to both practice utility and scientific contribution.

2 Case study: Evidence-informed counseling decision support application (ablescore)

2.1 Purpose and background

In 2016, a state VR agency engaged in a transformation project to improve the employment rate for persons with disabilities. To this end, the agency’s executive leadership team (ELT) chartered and sponsored an integrated, cross-functional team of experienced counselors and managers and engaged the services of a consulting firm led by a practitioner-scholar to assist the agency in the transformation project. The first phase of the project involved a detailed situation analysis to identify and prioritize opportunities for improvement in agency mission, vision, strategy, organization, client satisfaction, processes, decision-making, and policy. Further, the analysis aimed to understand the cultural, social and political contexts in which rehabilitation practice was embedded, and to capture the voice of key stakeholders – clients, counselors, providers, and agency executives and staff. The outcome of the analysis phase included strengths of the agency, problems faced by counselors in their practice setting, organizational capacity to address the problems, and cultural and behavioral barriers that must be overcome. The scope of the analysis included both internal and external perspectives, mixing qualitative (interviews and focus groups) and quantitative (analysis of historical performance data and survey data) methods of analysis to yield a more holistic view of problems faced by counselors and clients (the problem of practice) and the agency’s capacity and readiness for behavior change. From the internal analysis, the assessment included a review of the vision, mission, strategy, performance objectives, culture, organization structure, information flows, policies, and core processes that affect employment rate, and how decisions were made and by whom. From the external analysis, the assessment captured the voices of clients and service providers, as well as the voices of researchers and policymakers by reviewing extant literature and policy briefs.

One of the strategically important issues that emerged from the analysis was the lack of decision support tool to help counselors make informed decisions when designing and implementing the IPE to maximize employment outcome at closure. In the next section, we discuss the development of an evidence-informed prognostic tool to help counselors make informed rehabilitation decisions.

2.2 The problem of practice

Decision-making in VR counseling is multifaceted and complex, often requiring careful consideration of multiple prognostic factors before arriving at a course of action suitable for employment for a person with a disability (client). The problem of individualizing vocational rehabilitation for employment is related to the heterogeneity in disabilities, services, service delivery factors, and demographic characteristics (Gonzalez et al., 2011; Lusk, 2018; Weweiorski & Fabian, 2004). Although there are VR services that have been associated with 90-day employment outcomes (Dutta et al., 2008; Evensen et al., 2017; Glynn & Schaller, 2017; Huang et al., 2013; Rumrill et al., 2017), the prognostic problem faced by counselors and clients when designing the Individualized Plan for Employment (IPE) is knowing which of the fifty-seven or more VR services are most effective for a particular person with one or more of the known 19 disability classes and a complex set of demographic characteristics such as age, gender, race/ethnicity, marital status, educational level, work status, criminal history, alcohol abuse, drug abuse, receipt of disability income, etc. Despite this broad variability in VR service effects and demographics on employment outcomes, prognostic tools to support decision-making during the design and implementation of the IPE are not well established in rehabilitation practice.

Even the most experienced and knowledgeable counselors can’t possibly commit to memory all the knowledge about VR therapies they need to select from for each client situation. Even if they did have access to the massive amounts of knowledge needed to compare and select the best therapies to achieve the right outcomes for all the disabilities they encounter, they would still need time and expertise to analyze that information and integrate it with the client’s own personal characteristics. With increasing demand for VR services and counselors overwhelmed with increasing caseloads, this kind of in-depth prognosis and decision-making is beyond the scope of a counselor’s work. There are no evidence-based decision support tools we know of to assist counselors and clients to design the most effective IPEs based on significant predictors of 90-day employment outcome. In medicine, decision support tools such as predictive analytics or prognostic models are commonly used in predicting infections, determining the probability of disease, assisting a physician with a diagnosis, as well as predicting future wellness (Castaneda et al., 2015).

For the rehabilitation counselor to be effective in helping clients achieve successful employment outcomes, systematic effort must be undertaken to provide evidence-based tools to help counselors make informed decisions when selecting therapies and service delivery factors to be included in the IPE. In the absence of such decision support tools, counselors are forced to make ad hoc decisions about which therapies and various service delivery factors work for each individual, risking delays and adverse employment outcomes for those who ultimately complete the program. The counselor can use estimates of prognosis as a guide for ordering additional diagnostic tests and selecting appropriate VR therapies to develop a new Individualized Plan for Employment (IPE) or change an existing one. Utilizing such effective tool to aid counseling decision-making, can result in a better-informed decision today about outcomes tomorrow. It also reduces the inherent bias in ad hoc decision making.

Unfortunately, this prognostic challenge is longstanding. It has been over 5 decades since Gordon Paul (1967) proposed the challenge as the central research question in vocational rehabilitation: What treatment, by whom, is most effective for this individual with that specific problem, and under which set of circumstances? With only 18.7 percent of persons with disabilities employed in 2017 (U.S. Bureau of Labor Statistics, 2017) compared to 65.5 percent for those without a disability, an accurate decision support tool should improve the odds of employment and allow persons with disabilities to live independent lives. Thus, we summarize the problem of practice—the real-world problem faced by counselors and clients in their context-specific practice setting as follows: What prognostic tool can counselors and clients use to predict employment outcomes for eligible persons with disabilities who receive VR services from a state agency? To the best of our knowledge, this is the first comprehensive investigation of the predictors of employment and the construction of a practical decision support tool for counselors and clients.

2.3 Proposed evidence-informed solution: Ablescore

The purpose of this case study was to develop and validate an evidence-informed prognostic application (app) to assist counselors and clients to quantify the probability of employment outcomes across the full spectrum of VR therapies, key service delivery factors and personal characteristics. To this end, we began with the problem of practice and then we reviewed extant literature on prognosis and prediction research in vocational rehabilitation to frame the research question (RQ) and identify known predictors of employment outcomes. Though informed by theory, the RQ is based on the experience of difficulties and opportunities as encountered by VR counselors and their clients in their situated practice setting. Then we built a multivariate logistic regression model to isolate independent factors that significantly predict successful 90-day employment at closure. Next, we validated the model on a randomized test dataset to evaluate its predictive performance in terms of overall prediction accuracy, calibration quality, discriminative ability and practice usefulness. Afterward, we used the validated predictive model to construct an evidence-informed and practice-oriented prognostic tool we call AbleScore. In the final section, we discussed the implications of this tool for rehabilitation decision-making, application with persons with disabilities in other state agencies and future directions of prognostic or prediction research in VR practice.

2.4 Prognosis and prediction research in vocational rehabilitation

Prediction research, which aims to predict future events or outcomes based on patterns within a set of variables, has become increasingly prevalent in vocational rehabilitation research (Bellini, 1995; Bolton, et al., 2000; Capella-McDonnall, 2005; Chan et al., 2006; Dutta et al., 2008; Evensen et al., 2017; Glynn & Schaller, 2017; Hayward & Schmidt-Davis, 2003; Huang et al., 2013; Lusk, 2018; Rosenthal et al., 2003; Rumrill et al., 2017). These studies have related individual characteristics and certain rehabilitation services to employment outcomes using aggregate data from the RSA-911 database. However, these studies present three challenges, which tend to weaken their utility in rehabilitation counseling practice.

First, they lack evidence of validation and predictive performance, such as measures of calibration, discriminative ability, and predictive accuracy. Validation refers to the performance of the model when tested with clients from a similar population of a randomized sample (internal validation) and samples originated from a new and different population from the development model (external validation). Calibration describes how closely the predicted probabilities agree numerically with the observed outcomes. Discrimination refers to the ability of a model to correctly distinguish between 2 classes of outcomes—employed or not employed. Predictive accuracy refers to the model’s usefulness in counseling practice. These conceptual, methodological and performance problems continue to impede replication and comparison between studies.

Second, most of these studies, from which general models are derived, used cross-sectional designs and aggregate RSA-911 data. Because there is evidence (Chan, Wang, Muller, & Fitzgerald, 2011) suggesting significant variation in state VR agency outcomes that can be attributed to differences at the individual level, program execution, contextual and economic characteristics of the state, and differences in performance of the VR agencies themselves, studies using aggregate RSA-911 population data may not adequately account for these differences and thus make the outcome of these studies less useful at the state agency level to counsel individuals. These models are, therefore, susceptible to the “ecological fallacy” biases that may occur when an observed relationship between aggregated variables differ from the association at an individual level. For these reasons, the majority of models derived from aggregate national data are less useful in making individual client-centered counseling decisions.

Third, a majority of the studies were specific to a particular type of disability group. For example, individuals with substance abuse disorder (Lusk, 2018), schizophrenia (Evensen et al., 2017), learning disabilities (Rumrill et al., 2017), attention-deficit/hyperactivity disorder (Glynn & Schaller, 2017), orthopedic disabilities (Fong et al., 2006), the blind and visually impaired (Capella-McDonnall, 2005), and adults with cerebral palsy (Huang et al., 2013). Only Dutta and colleagues (2008) presented the most comprehensive analysis and summary of factors and characteristics associated with employment outcome, but in three discrete disability groups. None of these studies has examined comprehensively the effects of predictors on employment across all disability classes in a single state VR agency, where evidence can readily be interpreted and easily applied in practice without the aggregation problem.

Recognizing the dearth of evidence-based decision support tools in vocational rehabilitation counseling, this study addressed five interrelated and practice-oriented research questions to solve the problem of practice:

What individual characteristics, VR service provisions, and service delivery factors predict an individual’s employment outcome at closure?

How well do the predictors explain employment outcome at closure?

How accurate is the prediction?

What is the usefulness of the prediction tool in counseling practice?

How can counselors use the prediction tool in making decisions in their practice setting?

3 Development and validation of the prognostic models

3.1 Data and statistical analyses

Data for this study were extracted from a large archival database of client case records in a state vocational rehabilitation agency. The data was recorded by counselors at various stages in the rehabilitation process and contains demographic characteristics including racial/ethnic identity, gender, age at entry, marital status, type and severity of disability, receipt of public benefits such as disability income, educational attainment, types of VR services received, service delivery attributes, other pertinent personal history such as drug and alcohol abuse, criminal record, and the 90-day employment outcome information during a 20-year period from fiscal 1996 to 2016.

The state agency data we used grouped people with disabilities broadly into the three major disability groups established by the RSA (Reporting Manual for the Case Service Report (RSA-911)) for analysis and reporting purposes: sensory/communicative (e.g., visual impairment/blindness and hearing impairment/deafness), physical (e.g., arthritis, spinal cord injury), and mental impairments (e.g., depression, schizophrenia, and learning disabilities). This standard grouping has been used in several influential, refereed VR publications (e.g., Dutta et al., 2008) in the Journal of Occupational Rehabilitation. Table 1 summarizes the VR program services we identified as potential predictors of 90-day employment outcome at closure. VR services were carefully defined using RSA and state agency published criteria.

Table 1
Summary of VR program services

VR Program Service Description

Assessment Assessment services and activities performed to determine an individual’s eligibility for VR services, to assign an individual to a priority category of a state VR agency that operates under an order of selection, and/or to determine the nature and scope of VR services to be included in the Individual Plan for Employment (IPE); included in this category were trial work experiences and extended evaluation.

Diagnosis and treatment of impairments Surgery, prosthetics and orthotics, nursing services, dentistry, occupational therapy, physical therapy, speech therapy, and drugs and supplies; this category included diagnosis and treatment of mental and emotional disorders.

Vocational rehabilitation counseling and guidance Discrete therapeutic counseling and guidance services necessary for an individual to achieve an employment outcome, including personal adjustment counseling; counseling that addresses medical, family, or social issues; vocational counseling; and any other form of counseling and guidance necessary for an individual with a disability to achieve an employment outcome; this service is distinct from the general counseling and guidance relationship that exists between the counselor and the individual during the entire rehabilitation process.

College or university training Full-time or part-time academic training above the high school level that leads to a degree (associate, baccalaureate, graduate, or professional), a certificate, or other recognized educational credential; such training may be provided by a 4-year college or university, community college, junior college, or technical college.

•  Graduate university training

•  Four-year college training

•  Junior community college training

Occupational/vocational training Occupational, vocational, or job skill training provided by a community college and/or a business, vocational/trade, or technical school to prepare students for gainful employment in a recognized occupation; this training does not lead to an academic degree or certification.

On-the-job training Training in specific job skills by a prospective employer; generally, the individual is paid during this training and will remain in the same or a similar job upon successful completion; this category also includes apprenticeship training programs conducted or sponsored by an employer, a group of employers, or a joint apprenticeship committee representing both employers and a union.

Community based work adjustment training (CWAT) Community-based training to assist the individual in developing work skills, work habits, and job retention skills required to obtain and maintain employment. Work adjustment training includes activities to improve and increase productivity, attendance, punctuality, ability to work with others, ability to work under supervision, and work tolerance.

Facility based work adjustment training (FWAT) Facility-based training to provide a variety of hands-on work experience in areas such as packaging, assembly, inspection, quality control, sorting, and materials handling.

Basic academic remedial or literacy training Literacy training or training provided to remediate basic academic skills needed to function on the job in the competitive labor market.

Job readiness training Training to prepare an individual for the world of work (e.g., appropriate work behaviors, methods for getting to work on time, appropriate dress and grooming, methods for increasing productivity).

Disability-related, augmentative skills training Includes, but is not limited to, orientation and mobility, rehabilitation teaching, training in the use of low vision aids, Braille, speech reading, sign language, and cognitive training/retraining.

Miscellaneous training Any training not recorded in one of the other categories listed, including GED or high school training leading to a diploma.

Job search assistance Job search activities that support and assist a consumer in searching for an appropriate job; may include help in preparing resumes, identifying appropriate job opportunities, developing interview skills, and making contacts with companies on behalf of the consumer.

Job placement assistance A referral to a specific job resulting in an interview, whether or not the individual obtained the job.

Job supports Support services provided to an individual who has been placed in employment in order to stabilize the placement and enhance job retention; such services include job coaching, job follow along, job follow up and other job retention services.

Transportation services Travel and related expenses necessary to enable an applicant or eligible individual to participate in a VR service; includes adequate training in the use of public transportation vehicles and systems.

Maintenance Monetary support provided for expenses such as food, shelter, and clothing that are in excess of the normal expenses of the individual and that are necessitated by the individual’s participation in an assessment for determining eligibility for VR needs or that are incurred while an individual receives services under an IPE.

Rehabilitation technology The systematic application of technologies, engineering methodologies, or scientific principles to meet the needs of, and address the barriers confronted by, individuals with disabilities in areas that include education, rehabilitation, employment, transportation, independent living, and recreation; includes rehabilitation engineering services, assistive technology devices, and assistive technology services.

Reader services Services for individuals who cannot read print because of blindness or other disability; includes reading aloud and transcribing printed information into Braille or sound recordings if requested by the individual; generally, are offered to individuals who are blind or deaf-blind but may also be offered to individuals unable to read because of serious neurological disorders, specific learning disabilities, or other physical or mental impairments.

Interpreter services Sign language or oral interpretation services performed by specially trained persons for individuals who are deaf or hard of hearing, and tactile interpretation services for individuals who are deaf-blind; includes real-time captioning services; does not include language interpretation.

Personal attendant services Those personal services that an attendant performs for an individual with a disability, such as bathing, feeding, dressing, providing mobility and transportation, and so on.

Technical assistance services Technical assistance and other consultation services provided to conduct market analyses, to develop business plans, and to provide re- sources to individuals in the pursuit of self-employment, telecommuting, and small business operation outcomes.

Information and referral services Services provided to individuals who need assistance from other agencies (through cooperative agreements) not available through the VR program.

Other services All other VR services that cannot be recorded elsewhere; included here are occupational li- censes, tools and equipment, initial stocks and supplies, and medical care for acute conditions arising during re- habilitation and constituting a barrier to the achievement of an employment outcome.

Assistive technology evaluation service Service to evaluate, recommend and provide assistive technology to clients.

VR Program Service	Description
Assessment	Assessment services and activities performed to determine an individual’s eligibility for VR services, to assign an individual to a priority category of a state VR agency that operates under an order of selection, and/or to determine the nature and scope of VR services to be included in the Individual Plan for Employment (IPE); included in this category were trial work experiences and extended evaluation.
Diagnosis and treatment of impairments	Surgery, prosthetics and orthotics, nursing services, dentistry, occupational therapy, physical therapy, speech therapy, and drugs and supplies; this category included diagnosis and treatment of mental and emotional disorders.
Vocational rehabilitation counseling and guidance	Discrete therapeutic counseling and guidance services necessary for an individual to achieve an employment outcome, including personal adjustment counseling; counseling that addresses medical, family, or social issues; vocational counseling; and any other form of counseling and guidance necessary for an individual with a disability to achieve an employment outcome; this service is distinct from the general counseling and guidance relationship that exists between the counselor and the individual during the entire rehabilitation process.
College or university training	Full-time or part-time academic training above the high school level that leads to a degree (associate, baccalaureate, graduate, or professional), a certificate, or other recognized educational credential; such training may be provided by a 4-year college or university, community college, junior college, or technical college.
• Graduate university training
• Four-year college training
• Junior community college training
Occupational/vocational training	Occupational, vocational, or job skill training provided by a community college and/or a business, vocational/trade, or technical school to prepare students for gainful employment in a recognized occupation; this training does not lead to an academic degree or certification.
On-the-job training	Training in specific job skills by a prospective employer; generally, the individual is paid during this training and will remain in the same or a similar job upon successful completion; this category also includes apprenticeship training programs conducted or sponsored by an employer, a group of employers, or a joint apprenticeship committee representing both employers and a union.
Community based work adjustment training (CWAT)	Community-based training to assist the individual in developing work skills, work habits, and job retention skills required to obtain and maintain employment. Work adjustment training includes activities to improve and increase productivity, attendance, punctuality, ability to work with others, ability to work under supervision, and work tolerance.
Facility based work adjustment training (FWAT)	Facility-based training to provide a variety of hands-on work experience in areas such as packaging, assembly, inspection, quality control, sorting, and materials handling.
Basic academic remedial or literacy training	Literacy training or training provided to remediate basic academic skills needed to function on the job in the competitive labor market.
Job readiness training	Training to prepare an individual for the world of work (e.g., appropriate work behaviors, methods for getting to work on time, appropriate dress and grooming, methods for increasing productivity).
Disability-related, augmentative skills training	Includes, but is not limited to, orientation and mobility, rehabilitation teaching, training in the use of low vision aids, Braille, speech reading, sign language, and cognitive training/retraining.
Miscellaneous training	Any training not recorded in one of the other categories listed, including GED or high school training leading to a diploma.
Job search assistance	Job search activities that support and assist a consumer in searching for an appropriate job; may include help in preparing resumes, identifying appropriate job opportunities, developing interview skills, and making contacts with companies on behalf of the consumer.
Job placement assistance	A referral to a specific job resulting in an interview, whether or not the individual obtained the job.
Job supports	Support services provided to an individual who has been placed in employment in order to stabilize the placement and enhance job retention; such services include job coaching, job follow along, job follow up and other job retention services.
Transportation services	Travel and related expenses necessary to enable an applicant or eligible individual to participate in a VR service; includes adequate training in the use of public transportation vehicles and systems.
Maintenance	Monetary support provided for expenses such as food, shelter, and clothing that are in excess of the normal expenses of the individual and that are necessitated by the individual’s participation in an assessment for determining eligibility for VR needs or that are incurred while an individual receives services under an IPE.
Rehabilitation technology	The systematic application of technologies, engineering methodologies, or scientific principles to meet the needs of, and address the barriers confronted by, individuals with disabilities in areas that include education, rehabilitation, employment, transportation, independent living, and recreation; includes rehabilitation engineering services, assistive technology devices, and assistive technology services.
Reader services	Services for individuals who cannot read print because of blindness or other disability; includes reading aloud and transcribing printed information into Braille or sound recordings if requested by the individual; generally, are offered to individuals who are blind or deaf-blind but may also be offered to individuals unable to read because of serious neurological disorders, specific learning disabilities, or other physical or mental impairments.
Interpreter services	Sign language or oral interpretation services performed by specially trained persons for individuals who are deaf or hard of hearing, and tactile interpretation services for individuals who are deaf-blind; includes real-time captioning services; does not include language interpretation.
Personal attendant services	Those personal services that an attendant performs for an individual with a disability, such as bathing, feeding, dressing, providing mobility and transportation, and so on.
Technical assistance services	Technical assistance and other consultation services provided to conduct market analyses, to develop business plans, and to provide re- sources to individuals in the pursuit of self-employment, telecommuting, and small business operation outcomes.
Information and referral services	Services provided to individuals who need assistance from other agencies (through cooperative agreements) not available through the VR program.
Other services	All other VR services that cannot be recorded elsewhere; included here are occupational li- censes, tools and equipment, initial stocks and supplies, and medical care for acute conditions arising during re- habilitation and constituting a barrier to the achievement of an employment outcome.
Assistive technology evaluation service	Service to evaluate, recommend and provide assistive technology to clients.

Data was entered into the state agency database by certified rehabilitation counselors and was checked for errors by two data analysts and counselors who were knowledgeable about the database. Elements of this large dataset are furnished annually to the RSA by the state agency.

The sample frame included 53,629 clients for all disability groups whose cases were closed as either employed or not. Clients included in the model met the following criteria: (1) were eligible for vocational rehabilitation, (2) received at least one VR service, and (3) closed with or without employment. A major strength of this study is the quality, size and the representativeness of the sample and the comprehensiveness of the VR service provision. Table 2 summarizes the variables.

Table 2

Description of variables

Variable	Scale	Frequency	Percentage
Outcome
Employment at closure	no	24637	45.9%
	yes	28993	54.1%
Client variable
Type of Disability¹	Sensory &Communicative	7616	14.2%
	Physical	8899	16.6%
	Mental	37114	69.2%
Severity of Disability	Disability	5802	10.8%
	Significant Disability	9731	18.1%
	Most Significant Disability	38096	71.0%
Gender	Female	21299	39.7%
	Male	32330	60.3%
Marital status²	Single or never married	37875	70.6%
	Married or divorced or separated or widowed	15754	29.4%
Drug Abuse (self-reported)	no	50407	94.0%
	yes	3222	6.0%
Alcohol abuse (self-reported)	no	51729	96.5%
	yes	1900	3.5%
Criminal history	no	51905	96.8%
	yes	1724	3.2%
Race	White	25448	47.5%
	Black	26255	49%
	Others (Hispanic or Indian/Alaska or Asian or Pacific Islander or Mixed)	1926	1.8%
Educational level at Application	No education or grades 1–8 or grades 9–12	19054	35.5%
	Special education certificate or high school diploma or equivalent certificate	23849	44.5%
	Post-secondary no degree or associate/vocational/technical certificate or bachelor’s degree or masters and above	10726	20.0%
Disability income	no	39527	73.7%
	yes	20127	37.5%
Age	median age in years at entry (continuous)	27
Working at entry	no	46226	86.2%
	yes	7403	13.8%
VR Services - assessment/diagnostic evaluation and treatment services
Assessment	no	39038	72.8%
	yes	14591	27.2%
Diagnosis	no	19708	36.7%
	yes	33921	66.3%
Counseling &Guidance	no	49550	92.4%
	yes	4079	7.6%
VR services variable - employment development
On-the-job-training (OTJT)	no	53425	99.6%
	yes	194	0.4%
Job Placement	no	48099	89.7%
	yes	5530	10.3%
Community Work Adjustment Training	no	46651	87.0%
	yes	6978	13.0%
Facility Based Work Adjustment Training	no	50562	94.5%
	yes	3067	5.7%
Job Supports	no	48095	89.7%
	yes	5534	10.3%
Job Search	no	52758	98.4
	yes	871	1.6%
Job Readiness Training	no	41217	76.9%
	yes	12412	23.1%
VR services variable - education services & skills training
Occupational Vocational Training	no	49639	92.6%
	yes	3990	7.4%
Graduate University Training
	no	53405	99.6%
	yes	224	0.4%
Four Year College Training	no	48202	89.9%
	yes	5427	10.1%
Junior Community Training	no	52491	97.9%
	yes	1138	2.1%
Miscellaneous Training	no	49043	91.4%
	yes	4586	8.6%
Disability Related Skills Training	no	52993	98.8%
	yes	636	1.2%
Maintenance	no	38674	72.1%
	yes	14955	27.9%
Other Services	no	48579	96.6%
	yes	5050	9.4%
Interpreter services	no	52541	98.0%
	yes	1087	2.0%
VR services variable - enabling services and technology
Rehabilitation Technology	no	46184	86.1%
	yes	7445	13.9%
Assistive Technology Evaluation	no	52699	98.3%
	yes	930	1.7%
Transportation	no	37942	70.7%
	yes	15687	29.3%
Vehicle modification	no	53373	99.5%
	yes	256	0.5%
VR services delivery attributes
Length of stay in program	Time from entry to closure (range)	0.5 to 17 years
Counselor count	Number of counselors assigned to client in program (range)	1 to 20 counselors
VR services variable - removed services
Home modification	no	53555	99.9%
	yes	74	0.1%
Custom fabrication	no	53629	100%
	yes	0	0.0%
Technology training	no	53595	99.9%
	yes	34	0.1%
Job appropriateness evaluation	no	53574	99.9%
	yes	55	0.1%
Basic academic remedial training	no	53629	100%
	yes	0	0.0%
Benefits counseling	no	53629	100%
	yes	0	0.0%
Customized employment services	no	53629	100%
	yes	0	0.0%
Reader services	no	53629	100%
	yes	0	0.0%
Technical assistance services	no	53629	100%
	yes	0	0.0%
Information referral services	no	53629	0.0%
	yes	0	0.0%
Apprenticeship training	no	53629	100%
	yes	0	0.0%

1. We note that the state agency data we used grouped people with disabilities broadly into the three major disability groups established by the RSA (Reporting Manual for the Case Service Report (RSA-911)) for analysis and reporting purposes: sensory/communicative (e.g., visual impairment/blindness and hearing impairment/deafness), physical (e.g., arthritis, spinal cord injury), and mental impairments (e.g., depression, schizophrenia, and learning disabilities). This standard grouping has been used in several influential, refereed VR publications (e.g., Dutta et al., 2008) in the Journal of Occupational Rehabilitation). 2. We dichotomized some variables (e.g., marital status) since including all separate values for each category did not add value to the predictiveness of the model. Dichotomizing it instead resulted in a more parsimonious model with just as much explanatory value.

The data were randomly divided into two equal samples: a development sample (n = 26,814) used in model development, and a validation sample (n = 26,815) used in model validation. These two samples are nearly identical across all variables. Such an outcome is to be expected with such large sample sizes.

4 Method

4.1 Model development

We derived our prediction model with competitive employment as the outcome. Competitive employment defined in the RSA-911 manual represents employment for at least 90 days in an integrated setting, self-employment, or employment in a state-managed Business Enterprise Program (BEP) for which a person is compensated at or above the minimum wage. The employment outcome was coded “1” for successful employment and “0” for clients who were not working after completing their planned vocational rehabilitation program. Candidate predictors included those that had been reported to be prognostic and predictive, including client demographic characteristics, type and severity of disability, receipt of disability benefits, VR service provision history, and service delivery attributes. The initial set of 37 predictor variables included 12 client characteristics, 23 rehabilitation program services, and 2 service delivery variables. Thirty predictor variables were binary and four were multinomial (education, race, severity and type of disability). However, there were three continuous predictors including age, length of stay in program (time from entry to closure) and the number of counselors assigned to the client during rehabilitation. The model was developed using multivariable logistic regression1

¹
Multivariable logistic regression predicts probabilities not causation. We chose to assume linear relationships (implied by logistic regression’s generalized linear model) for the ease of creating a prediction calculator and for ease of interpretation. As one of the key points of this paper was to provide a product for practice to solve a problem of practice, a more precise but more complicated model would likely not be as useful as a simpler although modestly less precise tool. Of note, Euroscore (http://www.euroscore.org/calc.html) and STS Score (http://riskcalc.sts.org/stswebriskcalc/#/calculate) are well-known and highly regarded calculators for cardiology and thoracic surgery which use a similar linear approach and analytic model.

from the SPSS 25.0 statistical package. Variables entered in the development model were selected using stepwise backward-elimination approach, starting with all previously reported significant predictors found theoretically and practically to be associated with 90-day employment outcome. All variables significant at the p < 0.05 level were included in the model.

4.2 Evaluation of prognostic performance measures

To examine the performance and goodness of fit of the model, we evaluated measures of overall performance, calibration and discrimination. Overall performance was evaluated using predictive accuracy, Nagelkerke R² and Brier score statistics. Predictive accuracy assessed how well the model predicted the likelihood of an outcome for an individual client. The Nagelkerke R² quantified the percentage of the outcome variable (90-day employment) explained by predictors in the model. The Brier score quantified differences between actual outcomes and their predicted probabilities, that is, the mean square error (Steyerberg et al., 2010). The Brier score ranges from 0 to 0.25, values close to 0 indicate a useful model and values close to 0.25 a non-informative or worthless model (Steyerberg et al., 2010).

Calibration (goodness of fit) refers to the agreement between observed outcomes and prediction (Steyerberg et al., 2010). As recommended by Steyerberg et al., we used the calibration plot (Cox, 1958) to graphically assess model goodness-of-fit. The calibration plot is characterized by an intercept α, which indicates the extent that predictions are consistently too low or too high (‘calibration-in-the-large’), and a calibration slope β, which should be 1 and intercept α which should be 0 (Steyerberg et al., 2014; Cox, 1958), indicating good calibration and thus, model goodness of fit. The commonly used Hosmer–Lemeshow test produced statistically significant lack of fit due to the large sample size in our study. The Hosmer–Lemeshow test tends to fail even for good models when sample size is greater than 25,000 (Yu et al., 2017). For these reasons, we did not rely on the Hosmer–Lemeshow test of goodness of fit.

Discrimination refers to the ability of the model to discriminate between employed and not employed clients at closure and was determined from the area under the curve (AUC) of the Receiver Operator Characteristic (Royston et al., 2009). The ROC curve is a plot of the true positive rate (sensitivity) versus the false positive rate (1-specificity) evaluated at an optimal cutoff point for the predicted probability. A useless predictive model, such as a coin flip, would generate an AUC of 0.5. When the AUC is 1.0, the model discriminates outcomes perfectly. Therefore, a good AUC statistic is closer to 1.0.

4.3 Evaluation of model’s practice usefulness

Statistics such as sensitivity, specificity and area under the curve, while useful in assessing model performance, do not tell us whether the model would do more good than harm if used in counseling decision making. To address the question about the value of the model in terms of counseling decision making, we used a decision-analytic approach proposed by Vickers and Elkin (2006) to quantify its clinical usefulness based on the following premises: (1) prognostic tool is only valuable to the extent that it improves clinical decision making, and (2) the value of clinical decision making depends on the benefits and harms of these decisions.

A simple calculation for evaluating the usefulness of prognostic tools in clinical practice is the Net Benefit analysis. We used net benefit analysis to evaluate the potential clinical consequences of using our model in counseling decision making. Net benefit attempts to quantify potential harms and benefits of classification error – false positive and false negative classifications. The performance of a prediction model was assessed by Steyerberg et al. (2010) following Peirce (1884). In calculating error rates, we classified clients as positive when their predicted probability of the employment exceeds 0.515 (the optimal cut-off – the value that maximized sensitivity and specificity on the ROC curve) (Lalkhen & McCluskey, 2008) and as negative otherwise. This implies approximately an equal weighting of false-positive and false-negative classifications. The basic interpretation of a decision curve is that the model with the highest net benefit at a particular threshold probability has the highest clinical value. As reported later in the article, we also conducted a Use Case Evaluation involving experienced practitioners to ascertain the potential clinical benefit and harm of the practice product derived from the model.

5 Results

The estimates of beta coefficients (β), odds ratios (OR)2

²
The odds ratios and beta coefficients both estimate the effect size of the predictor on the dichotomous outcome variable; the latter one being the natural logarithm of the former one (Ialongo, C., 2016).

and confidence intervals (CI) for 33 predictors in the full model found to be of independent prognostic significance (p < 0.05) are summarized in Table 3. Twenty VR service categories and two service delivery factors were significant predictors of 90-day employment, indicating that receiving these VR services can help predict3

Multivariable logistic regression allows us to study the simultaneous effect of multiple predictor variables on a dichotomous outcome such as whether a client will be employed or not at closure.

whether a client is employed at closure. Several demographic predictors were found to be significantly related to employment outcome. For age, older clients were less likely to be employed than younger clients (β= –0.35, OR = 0.70). Compared to female clients, males were 1.2 times (OR = 1.2; 95% CI = 1.15–1.25) more likely to be employed. Having a criminal record was negatively associated with employment. A person with criminal record at entry was 56% (OR = 0.44; 95% CI = 0.39–0.50) less likely to be employed at closure than one without. Although drug and alcohol abuse were not independent predictors (p > 0.05), when co-occurring, that is, when a client abused both drug and alcohol, the chances of employment were reduced by 35% compared to a client without these issues (OR = 0.65; CI = 0.52–0.82). A client’s race/ethnicity also has an effect. Blacks were 22% less likely to be employed at close than white clients (OR = 0.78; CI = 0.75–0.81).

Type and severity of disability impacted the chance of employment at closure. For example, from our archival dataset of 53,629 clients, a person with mental disability was 27% less likely to be employed than a person with sensory-communicative disabilities. Similarly, a person with physical disability was 30% less likely to be employed than a person with sensory-communicative disabilities. A person with the most severe disability was 35% less likely to be employed than one with a less significant disability. The employment status of client at the time of application was significantly associated with employment at closure. Clients who had prior work experience before entry were 2.3 times more likely to be employed at closure than those who were not (OR = 2.3, CI = 2.19–2.48).

Employment-related training and services had the largest impact on employment. This result is consistent with the rehabilitation literature that shows that employment-related services have a strong impact on employment outcomes. For example, clients who received occupational vocational training were 2 times more likely to be employed at closure than those without (OR = 2.0; 95% CI = 1.85–2.16), 3.7 times for on-the-job training (OR = 3.69; 95% CI = 2.54–5.36), 3.5 times for miscellaneous training (OR = 3.5; 95% CI = 3.24–3.81), 2.4 times for job placement service (OR = 2.4; 95% CI = 2.22–2.59), 3 times for job supports (OR = 3; 95% CI = 2.79–3.30), 1.5 times for community work adjustment training (OR = 1.5; 95% CI = 1.43–1.61), and 1.3 times for disability-related skills training (OR = 1.3; 95% CI = 1.00–1.53). Unfortunately, very few clients received these employment services that could have improved their chances of employment. For example, as pointed out earlier, only 0.4% of clients received on-the-job training, 1.2% received disability-related skills training, and 10.3% received job supports and job placement.

Higher education services were positive contributors to 90-day employment. Several enabling technologies and services such as rehabilitation technology, vehicle modification, transportation and other services increased the chance of employment for those who received them. Interestingly, we found that several VR services that are commonly provided to clients were associated with significantly reduced chances of employment at closure. Assessment and diagnosis & treatment were negatively related to employment at closure. Clients who received assessment were 13% less likely to be employed (OR = 0.87; CI = 0.83–0.91), and 17% for those receiving diagnostic and treatment (OR = 0.83; CI = 0.79–0.86). Further and importantly, receipt of disability benefits reduced the chance of finding employment. Clients who received benefits were 50% less likely to be employed than those who did not (OR = 0.50; CI = 0.48–0.53). Some commonly utilized services were also found to be non-significant (p > 0.05) predictors of employment at closure: facility-based work adjustment training, job readiness training, job search and interpreter services.

Table 3

Full Model: Variables showing prognostic significance

Predictors	β coefficients	Standard error	p-value¹	Odds ratio	95% C.I. for Odds ratio
					Lower	Upper
Type of Disability (reference is sensory-communicative disability)			0.000
Type of Disability (physical)	–0.358	0.041	0.000	0.699	0.645	0.757
Type of Disability (mental)	–0.316	0.038	0.000	0.729	0.677	0.785
Severity of Disability (reference is disability)	0.000
Severity of Disability (significant)	–0.248	0.039	0.000	0.780	0.723	0.842
Severity of Disability (most significant)	–0.424	0.035	0.000	0.654	0.611	0.701
Assessment	–0.139	0.024	0.000	0.870	0.830	0.913
Diagnosis and Treatment	–0.192	0.022	0.000	0.826	0.790	0.862
Occupational Voc. Training	0.694	0.040	0.000	2.002	1.85	2.157
On the Job Training	1.300	0.190	0.000	3.664	2.542	5.355
Miscellaneous Training	0.948	0.045	0.000	2.580	2.363	2.816
Job Placement	0.873	0.04	0.000	2.395	2.215	2.589
Transportation	0.070	0.024	0.004	1.073	1.024	1.124
Maintenance	1.065	0.024	0.000	2.901	2.765	3.042
Rehabilitation Technology	0.793	0.037	0.000	2.209	2.056	2.374
Vehicle Modification	1.489	0.157	0.000	4.431	3.258	6.026
Other Services	0.479	0.038	0.000	1.614	1.499	1.737
Graduate University Training	1.261	0.205	0.000	3.528	2.362	5.271
Four-Year University	0.573	0.043	0.000	1.773	1.631	1.928
Junior Community College	0.385	0.078	0.000	1.469	1.260	1.713
Job Supports	1.110	0.043	0.000	3.033	2.790	3.298
Disability-Related Skills	0.231	0.098	0.018	1.255	1.035	1.520
Community Work Adjustment Training	0.416	0.031	0.000	1.516	1.427	1.612
Counselling Guidance	0.261	0.047	0.000	1.298	1.184	1.423
Receipt of Disability Income	–0.681	0.025	0.000	0.506	0.482	0.532
Race/ethnicity (reference is White)			0.000
Race/ethnicity (Black)	–0.246	0.021	0.000	0.782	0.75	0.815
Race/ethnicity (Others)	–0.195	0.054	0.000	0.823	0.74	0.915
Age	–0.353	0.082	0.000	0.703	0.599	0.825
Gender	0.183	0.021	0.000	1.203	1.154	1.254
Educational attainment (reference: less than High School)			0.000
Educational attainment (High school diploma and post-secondary on degree)	0.023	0.023	0.269	1.023	0.978	1.073
Educational attainment (post-secondary with degree, bachelors and higher)	0.145	0.031	0.000	1.156	1.088	1.229
Length of stay in program	–0.214	0.007	0.000	0.808	0.797	0.819
Marital Status	0.085	0.029	0.003	1.088	1.029	1.151
Alcohol Abuse	0.174	0.068	0.012	1.185	1.038	1.354
Drug Abuse	–0.010	0.048	0.827	0.996	0.901	1.087
Work experience at entry	0.843	0.032	0.000	2.323	2.180	2.475
Criminal Record at entry	–0.811	0.063	0.000	0.444	0.393	0.502
Counselor Count	–0.084	0.007	0.000	0.919	0.906	0.933
Alcohol Abuse by Drug Abuse	–0.428	0.115	0.000	0.652	0.521	0.817
Constant	1.283	0.129	0.000	3.606

¹The high p-values are expected given the large sample size used in the development model.

5.1 Performance evaluation result

The predictive accuracy, discriminative ability, and calibration quality of the development model was assessed prior to testing in the validation sample. The overall predictive accuracy of the model was good. The development model correctly classified employment outcome for 72% of clients compared to 54% in the null model. The Nagelkerke R² indicated that predictors in the development model explained 29.3% of the variance in 90-day employment outcome, which indicates strong effect (Cohen, 1988). The discriminative ability of the model was evaluated by the ROC curve, showing an AUC of 0.78 (SE = 0.003), indicating a strong effect size (Rice & Harris, 2005). The model demonstrated good calibration (calibration slope = 1; intercept = 0.00). We summarized the performance of the development model in Table 4.

Table 4
Development model performance measure

Overall Performance Measure Discrimination Calibration Clinical usefulness

R² Brier Score Predictive Accuracy AUC/ROC Calibration slope Calibration intercept Net benefit

29.3% 0.190 72% 0.78 1.00 0.000 0.252

Overall Performance Measure	Discrimination	Calibration	Clinical usefulness
29.3%	0.190	72%	0.78	1.00	0.000	0.252

5.2 Model validation and performance comparison

The development model was validated in the randomized validation dataset (n = 26,815) using the bootstrap technique. The results confirmed that the model performed well in its predictive accuracy, calibration and discrimination. Figure 1 represent the ROC curves for the development, validation and full samples.

Fig.1

AUC/ROC Curve for (a) Development, (b) Validation, (c) Full models.

Further, the models demonstrated good calibration, indicating good goodness-of-fit and good agreement between observed and predicted employment outcomes. The calibration plots are presented in Fig. 2.

Fig.2

Calibration plots: (a) Development, (b) Validation and (c) Full models.

The highest net benefit at a particular threshold probability has the highest clinical value (Vickers, 2008). As shown in Table 5, the net benefit value of the models were nearly identical: 0.26 for the development, 0.25 for the validation and 0.26 for the full model.

Table 5

Model validation and performance comparison

Performance Measure	Development model	Validation model - bootstrap resampling
	50% random sample	50% random sample	100% full dataset
Overall
R²	0.293	0.291	0.292
Brier score	0.190	0.191	0.190
Predictive accuracy	72%	71%	72%
Discrimination
AUC/ROC	0.78	0.78	0.78
Calibration
Calibration slope	1.00	1.00	1.00
Calibration intercept	0.00	0.00	0.00
Clinical usefulness (cutoff = 0.5)
True positive rate (sensitivity)	72%	72%	72%
True negative rate (specificity)	71%	71%	71%
False positive rate	29%	29%	29%
False negative rate	28%	28%	28%
Net benefit	0.252	0.250	0.250

Employed outcome: cutoff > = 0.515. Unemployed outcome: cutoff <0.515. Net Benefit cut-off = 0.515.

6 Product development

6.1 Productizing – development of product for practice

Product development and use case evaluation were the final steps in applying the research evidence to build a user-friendly prognostic tool to assist counselor in making informed rehabilitation decisions. We used the significant β-coefficients (p < 0.05) estimated in the full population (n = 53,629) to develop a prognostic calculator we call AbleScore. For a given client, AbleScore predicts a probability of employment according to the logistic regression equation with the following formula:

Predicted probability (AbleScore) = 1/(1+e^–α), where α=β₀+β₁x₁+β₂x₂+…+β_nx_n

In this formula, X₁ to X_n are the values of the independent predictors included in the final prognostic model, β₁ to β_n are their corresponding regression coefficients and b₀ is a constant provided in Table 3. For binary predictors in the model, X_i = 1 if variable is present and 0 if it is absent.

We conducted product design workshops to specify key design elements and online user interface criteria. We then incorporated the design criteria to build the prognostic tool. Finally, as reported in the Use Case Evaluation section, the product was evaluated in practice by expert counselors with many years in vocational rehabilitation. The evaluators (counselors) found the tool useful in augmenting their judgment and an effective device to motivate and engage clients in the rehabilitation process, thus indicating practical validity and potential counseling usefulness. The tool has two versions—a mobile device application and a browser version for use on non-mobile devices. A screenshot of AbleScore is shown in Fig. 3.

Fig.3

AbleScore Display: www.ablescore.com.

6.2 Application Use Case Evaluation

The Use Case Evaluation (UCE) method was employed as a valuable means for usability (usefulness and ease of use) evaluation. The UCE method was deployed in a facilitated full-day workshop and involved several key steps: (1) purposive selection of 10 evaluators – best-in-class counselors who were selected by the agency’s executive leadership team as subject matter experts; (2) development of 5 randomized and anonymized use cases derived from recently closed cases not included in the application development and validation samples; (3) explanation of use cases – evaluators received as much information as possible about each case; (4) description of the purpose and demonstration of the functionality and features of the application; (5) counselor self-scoring of the use cases to establish a baseline for evaluation – before evaluating the tool, each counselor, based on experience, predicted the probability of employment for each use case; (6) participant evaluation of the application’s features, functionality and ease of use. The facilitator encouraged the counselors to continuously think out loud while using the application – that is, simply verbalizing their thoughts, concerns and benefits as they move through the application; (7) application-scoring of the use cases; (8) comparative evaluation – the baseline expert self-scores and the application scores were compared to the actual outcome of each use case; and (9) finally, documentation of results and participant feedback – accuracy, usefulness, ease of use, and opportunities for improvement.

The results and feedback comments revealed key benefits and concerns of the application:

The application was judged as novel and useful as a supplement to counselor’s expertise in new IPE design and in modifying existing ones. For example, the expert counselors commented (in writing):

“I see how this can help to streamline the IPE and make it more effective...but to me, the real value is using the tool to engage my clients in building their IPE and motivate them to be accountable for their own rehabilitation...the motivation will come from knowing the potential outcome of the VR program before receiving services...this tool has potential to be a game changer.”

“The fact that it correctly predicted employment outcome in 4 out of 5 cases is truly impressive...it did a better job than us [evaluators] with only 2 of 5 cases predicted correctly.”

“The tool focuses action on evidence and away from rule of thumb, folklore and traditional VR approaches that fail to help clients find the jobs they need to live independent lives.”

The app has potential to “significantly reduce rehabilitation cost-per-client-served by increasing IPE quality, client ownership, service delivery quality and reducing the time it takes to rehabilitate clients.”

The app will “significantly reduce dropout cost (dropout cost is currently $3 million per year) by 35% – $1.05 million in potential savings which can be redeployed to serve more [clients] with disabilities.”

All the evaluators commented positively on the ease of use of the application – the app is driven by information that is readily available in the counselor’s office and the click-and-select functionality and portability as a mobile app were highly valued. For example:

One counselor said: “I find the tool very easy to use… there isn’t a lot of data entry and the data I need are easily available in my case files.”

Another counselor commented: “What I like the most is that it generates a single number which one can easily interpret as a “yes” or “no” employment at closure.”

However, the evaluators were concerned that the app could be used by inexperienced counselors as the sole tool for selecting VR services. Some of the counselors’ comments include:

“I’m concerned that in the hands of inexperienced counselors, the tool may become a panacea...and that may have some negative unintended consequences.”

“I hope that the tool will not be used to discriminate clients based on who can and cannot get a job.”

“I’m concerned that the tool may replace counselors in the future and I’m concerned about their own careers.”

In response to the above concerns, other counselors pointed out:

“The tool is not a discriminatory tool. [Counselors] are accountable to our clients to provide the best level of service possible ... In no way does this remove the need for counselors.”

“The app should not be used as the sole source of information to determine a clients’ employability.”

“This is just one more tool in the counselors’ toolkit and never intended to replace clinical experience and judgement.”

“The key to avoiding these issues is training… counselors will be trained on the objective of the app as a support tool, how it was developed, how to use it to select services and how to interpret the result correctly.”

These issues were further discussed by the product development team and used to improve the application and guide the development of a training document.

7 Discussion

In this paper, we responded to the growing call (Leahy et al., 2014) for innovative approaches to supplement existing models to make research more relevant to problems faced by vocational rehabilitation practitioners and their clients (persons with disabilities). We proposed a novel approach that addressed the weaknesses we identified in the KT framework, including the dichotomies between knowing and doing, knower and doer and knowledge and practice tool. We note that unlike KT, where the knowledge to be translated is often irrelevant to the real-world problems faced by counselors and clients, the new approach starts with a situated problem of practice and ends with the construction of an evidence-informed product for practice (PfP) to solve the problem of practice. We achieved this goal in the case study by (1) intentionally aligning the interests and goals of the counselors, clients and practitioner-scholars, (2) balancing practice relevance and research rigor, and (3) applying rigorous research methods to derive the evidence used in productizing and producing the PfP. Importantly, the new research approach reverses the current order of research-to-practice to a new order of practice-to-research, where a problem of practice (PoP) is the motivation for research, and research evidence is both a product of and a solution to a problem of practice.

We applied the new approach, which we call Practice Motivated Research (PMR) in a case study in a state vocational rehabilitation agency to demonstrate how practice-informed research can meet the requirements for academic rigor and practice relevance, revealing the recursive interdependence between practitioners and researchers in the production of clinically useful prognostic tool we call AbleScore. The practice output generated by using AbleScore is intuitive, empirically derived, practically meaningful, and easily utilized by counselors in their context-specific practice setting. To use the prognostic app, counselors need only collect demographic data for the client and potential VR services and then enter this data into the app on their mobile device or laptop. The computations are automatic, and the outputs are easy to interpret. Used as intended, AbleScore has the potential to augment counseling decisions, improve the effectiveness of IPEs and increase the chances of successful employment outcomes for persons with disabilities. To our knowledge, this is the first study that attempts to develop a prognostic tool in a state vocational rehabilitation agency. As such, the paper should foster additional discussion.

Results from this study support and further inform existing literature regarding the association between employment outcomes among people with disabilities and their demographic characteristics, vocational rehabilitation services and service delivery attributes. Accordingly, most of the factors used in our prognostic tool have been previously identified by other studies as prognostic factors associated with 90-day employment outcome. We examined comprehensively the effects of 38 predictors on employment across all disability classes in a single state VR agency, where evidence can readily be interpreted and easily applied in practice. The evidence used in building the app demonstrated good discrimination, calibration, accuracy and goodness of fit.

The prognostic app has several distinctive strengths compared with prior predictive research in VR. First, it consists of clearly defined, routinely available predictors and does not require any additional data not routinely available in the design of an IPE. Second, the accuracy and applicability of the tool are supported by its derivation and validation using historic data from 53,629 independent clients from the same state population. Third, our study samples are comprehensive and represent all disability groups and several demographic characteristics and service delivery attributes never before included in other studies. Fourth, because we used data from a single state agency, we mitigated the aggregation problem associated with using multi-state, aggregated data from the RSA, which presents a major problem of interpretation at the individual level in state agencies. Finally, AbleScore is a practice-ready tool, not just another intellectual contribution to knowledge that needs translation before it can be applied in counseling practice. AbleScore can be freely accessed at http://ablescore.com/.

7.1 Implications

We found that several predictors differed in their strength of association with employment outcome according to vocational rehabilitation services received, suggesting the importance of optimal IPE design. We have developed a methodologically valid, simple, and accurate model that has been shown4

⁴
See Application Use Case Evaluation sub-section of the Product Development section.

to help decisions about IPE design for individual clients. It is important to emphasize, however, that while our prognostic tool may complement counseling decision making, it cannot and should not replace counseling experience and judgment. This is particularly important in the context of judgments about service provisioning and when auditing existing IPEs. Thus, a more qualitative interpretation of our numerical results is advisable.

Perhaps of particular note are the strongest positive and negative predictors of successful rehabilitation: Criminal Record, Receipt of Disability Income, Co-occurrence of Alcohol and Drug abuse, Most Severe Disability ... Vehicle Modification, On-The-Job Training, Graduate University Training, Job Supports, Maintenance, Job Placement, Work Experience at Entry, Rehabilitation Technology and Occupational Vocational Training.

The state-federal vocational rehabilitation program emphasizes working with individuals with the most severe disabilities; i.e., those disabilities that significantly limit one or more life functions (Rehabilitation Services Administration, 1995). However, we provide evidence for careful consideration of other potentially co-occurring client factors, such as criminal record, substance abuse and receipt of disability income. VR practitioners may benefit from emphasizing the positive predictors, and at least being aware of the negative predictors, in order to better serve their clients. Again, careful judgement must be used; predictor variables in logistic models predict probabilities, not correlations. Thus, variables with positive or negative effects from our model should not be interpreted as causal, and should therefore be applied with caution and with an open mind to other possibly unmeasured variables.

As an illustration of direct application, below we apply the AbleScore app to three different client scenarios: difficult, average, and favorable. We then show how to apply some of our research findings to help guide these cases.

Client 1. high-school dropout, with moderate cognitive disability, with a criminal record, currently unemployed and receiving disability income. AbleScore: 18%

Client 2. high-school diploma, with a moderate mobility disability, with a clean record, currently unemployed and receiving disability income. AbleScore: 37%

Client 3. graduate degree, with mild deafness disability, with a clean record, employed at entry and not receiving disability income. AbleScore: 88%

In these three scenarios, if we add on-the-job training to the client’s IPE, the AbleScores jump significantly to 45%, 65%, and 94% probability of employment at closure. What this means is that others (within our population of 53,629) with IPEs like these scenarios, but with the addition of just on-the-job training, have had much higher success rates of gaining employment. These illustrative examples show us that, for those in less than favorable scenarios, including employment-enabling training like on-the-job training in the client’s IPE has the potential to make a notable difference in their rehabilitation outcome.

Clearly, this tool should not be over-relied upon, or used as the sole determinant of guidance decisions and counseling. This application should be only one tool among many to help the counselor make the best decisions for her clients. Assuming that the predictions of the application will be accurate for any specific case is naïve at best, but potentially harmful. The value of the application is to indicate the potential difficulty of a particular case, as well as highlight the best possible uses of resources to lead to positive employment outcomes.

7.2 Future research

Because our model was developed with data from a single state agency, and validated only with clients from the same population, further prospective validation in independent client cohorts is needed to strengthen the generalizability of the model. To make the findings more generalizable, a sample of clients and counselors from multiple states (ideally all states) and different offices should be used. Additionally, the predictive relevance of the model is limited to the current sample, and should not be generalized to future samples in other states. Nevertheless, a comprehensive dataset such as the one we used for this study, which included all clients for the past 20 years from an entire state, does provide strong evidence of patterns and trends at least for that state.

Future research could also evaluate different ways, or formats, for presenting the model to counselors; their use in counseling practice; and whether ultimately, they have any impact on the management and outcomes of clients in other states. With prediction in mind, agencies may be inclined to collect more data on clients in the future. A more predictive model could be created if more variables were collected. Particularly missing from the current set of variables are local labor market factors and social factors, such as social support from friends and family, regular communication with the VR agent, etc. For our study, these variables, and many others, were not available because they had not been recorded by the state agencies who archived this information.

7.3 Limitations

Several limitations apply to our study. Foremost, the analysis with a single state VR agency data represents essentially a case study (albeit with a very large sample size). Although the structure of the dataset may be representative of other state agency datasets, the prognostic tool may not apply in a different population where the client characteristics and service protocols are different from those of the development environment. Therefore, this model should not be implemented outside of this state before its validity has been tested in the local client population.

Next, heterogeneity of client populations between practice locations or counties or rural locations versus urban locations was not taken into account in our analysis. We also did not consider differences in counselors or in offices. However, this may be an important consideration in situated counseling practice. This oversight may also inadvertently overlook critical contextual variables that may alter the results we found with the set of variables we did have available to work with. Nevertheless, extant research indicates there may be no substantial differences between these contextual levels in VR practice (Hayward and Schmidt-Davis 2005).

Also, we note that the state agency data we used grouped people with disabilities broadly into the three major disability groups established by the RSA (Reporting Manual for the Case Service Report (RSA-911)) for analysis and reporting purposes: sensory/communicative (e.g., visual impairment/blindness and hearing impairment/deafness), physical (e.g., arthritis, spinal cord injury), and mental impairments (e.g., depression, schizophrenia, and learning disabilities). There may be potential loss of information due to the aggregation.

Further, while we used logistic regression which is common in the healthcare prediction studies, other analytic approaches such as those leveraging machine learning, may provide better predictions, but more difficult to interpret. We encourage comparative studies of other methods that look promising from our analysis. Although we eliminated non-significant factors, performed internal validation and sensitivity analysis to avoid overfitting, testing the performance of the model in an independent population sample will be necessary in the future. We encourage future studies to do so. Of note, using the prognostic model generated with historical data to predict future events can be problematic. Though the model may fit historical data well, future circumstances may not be sufficiently similar to allow for reliable predictions from the model. Even when the internal validation bootstrap resampling technique was applied to correct for overfitting and optimism, the accuracy of our prognostic tool can be substantially lower (or higher) in new clients compared to the accuracy found in the clients in the development population. The differences may arise due to client mix. For example, the development sample may have more or fewer clients of a particular age group or with a particular disability and severity group than future clients. Vergouwe et al. (2002) described the effect as “case mix” differences. Case mix disparity can affect both discrimination and calibration of the predictive model. As such, the calculator must be calibrated regularly with new dataset as client population changes in the future.

We also note that the net-benefits analysis is still evaluated at the aggregate level however. Thus, implementation of the application in individual cases may vary widely. The usefulness of a model such as this one, which is trained on the individual circumstances of 53,629 client cases, is that it shows the trends and patterns that we can expect to be useful and informative for the majority of cases. This does not mean that every aspect of the model will be useful for every counseling case – and that is not its intent. The intent of the application is to provide guidance. For example, if a counselor enters her client’s information into the application and the results show that cases like this client’s have resulted in employment only 53% of the time, she may need to provide more guidance and energy on this case than for one who is more similar to cases resulting in employment 94% of the time. Additionally, this application is only one tool among many that the counselor will use to help her make good guidance decisions for any particular case.

Finally, our prognostic tool, like many in medicine (e.g., Euroscore (http://www.euroscore.org/calc.html) and STS Score (http://riskcalc.sts.org/stswebriskcalc/#/calculate), which are well-known and highly regarded calculators for cardiology and thoracic surgery), are nomothetically-derived but idiographically-applied by practitioners who understand the particular world and characteristics of the individual. In the present research, we started with cohorts of individuals from a large and defined population and used logistic regression to identify the presence or absence of an employment outcome given a set of predictors in the data – a nomothetic perspective. Then the tool will be applied idiographically by counselors to validate the nomothetic principles at the individual level. However, nomothetic models are probabilistic and usually incomplete when applied to an individual and must therefore be validated in the field before general application.

Despite these limitations, the application promises to inform clients and counselors of the probability of successful employment outcomes for a group of clients with similar profiles and proposed IPEs. This information is useful and should form part of the basis for informed choice through which the client and counselor can decide on an appropriate rehabilitation path.

Conflict of interest

The authors declare that they have no conflict of interest.

Footnotes

Acknowledgments

This study was a part of a consulting engagement involving several counselors, clients, providers and agency leaders. We acknowledge all the 46 individuals who participated in interviews, workshops and focus groups to provide information, validate models and findings and test the prognostic tool. The involvement of the agency’s executive leadership team (ELT) in providing direction, sponsorship, project resources, and role modeling behavior change was invaluable. Without the ELT this project would not have been possible.

References

Armstrong,

, Waters,

, Crockett,

, & Keleher,

(2007). The nature of evidence resources and knowledge translation for health promotion practitioners. Health Promotion International, 22(3), 254–260.

Bacharach,

S. B.

(1989). Organizational theories: Some criteria for evaluation. Academy of Management Review, 14, 496–515.

Baumbusch,

J. L.

, Kirkham,

S. R.

, Khan,

K. B.

, McDonald,

, Semeniuk,

, Tan,

, & Anderson,

J. M.

(2008). Pursuing common agendas: A collaborative model for knowledge translation between research and practice in clinical settings. Research in Nursing and Health, 31, 130–140.

Bellini,

(1995). A comparison of linear multiple regression and a simplified approach in the prediction of rehabilitation outcomes. Rehabilitation Counseling Bulletin, 39(2), 151–160.

Bellini,

, Neath,

Bolton,

B. (1995). A comparison of linear multiple regression and a simplified approach in the prediction of rehabilitation outcomes. Rehabilitation Counseling Bulletin, 39(2), 151–160.

Bennett,

, Tooth,

, McKenna,

, Rodger,

, Strong,

, Ziviani,

, ... Gibson,

(2003). Perceptions of evidence-based practice: A survey of occupational therapists. Australian Occupational Therapy Journal, 50, 13–22.

Bolton,

, Bellini,

J. L.

, & Brookings,

J. B.

(2000). Predicting client employment outcomes from Personal history, functional limitations, and rehabilitation services. Rehabilitation Counseling Bulletin, 44(1), 10–20.

Capella-McDonnall,

M. E.

(2005). Predictors of competitive employment for blind and visually impaired consumers of vocational rehabilitation services. Journal of Visual Impairment Blindness, 99(5), 303–315.

Castaneda,

, Nalley,

, Mannion,

, Bhattacharyya,

, Blake,

, Pecora,

, ... Suh,

K. S.

(2015). Clinical decision support systems for improving diagnostic accuracy and achieving precision medicine. Journal of Clinical Bioinformatics, 5(1), 4.

10.

Chan,

, Bezyak,

, Ramirez,

M. R.

, Chiu,

C. Y.

, Sung,

, & Fujikawa,

(2010). Concepts, challenges, barriers, and opportunities related to evidence-based practice in rehabilitation counseling. Rehabilitation Education, 24, 179–190.

11.

Chan,

, Cheing,

, Chan,

C. Y. J.

, Rosenthal

A. D.

, & Chronister

(2006). Predicting employment outcomes of rehabilitation clients with orthopedic disabilities: A CHAID analysis. Disability and Rehabilitation, 28(5), 257–270.

12.

Chan,

, Wang,

C. C.

, Muller,

, & Fitzgerald,

(2011). Vocational rehabilitation outcomes: A multi-level analysis of economic indicators, VR agency characteristics, and RSA-911 data (Phase research: A practical resource for investigating social science phenomena (pp. 285–303). Washington, DC: American Psychological Association.

13.

Cohen,

(1988). Statistical power analysis for the behavioral sciences (2nd ed.), Hillsdale, NJ: Lawrence Erlbaum Associates.

14.

Cox,

D. R.

(1958). Two further applications of a model for binary regression. Biometrika, 45, 562–565.

15.

Dijkers,

(2016). Four types of evidence abuse. KT Update, 4(3). Center on Knowledge Translation for Disability and Rehabilitation Research.

16.

Doane,

G. H.

, & Varcoe,

(2008). Knowledge translation in everyday nursing: From evidence-based to inquiry-based practice. Advances in Nursing Science, 31(4), 283–295.

17.

Dubouloz,

, Egan,

, Vallerand,

, & von Zweck,

(1999). Occupational therapists' perceptions of evidence-based practice. American Journal of Occupational Therapy, 53, 445–453.

18.

Dutta,

, Gervey,

, Chan,

, Chou,

C. C.

, & Ditchman,

(2008). Vocational rehabilitation services and employment outcomes of people with disabilities: A United States study. Journal of Occupational Rehabilitation, 18, 326–334.

19.

Erickson,

, Lee,

, & von

Schrader, S.

(2017). Disability Statistics from the American Community Survey (ACS)Ithaca, NY: Cornell University Yang-Tan Institute (YTI). Retrieved from www.disabilitystatistics.org..

20.

Evensen,

, Ueland,

, Lystad,

J. U.

, Bull,

, Klungsøyr,

, Martinsen,

E. W.

, & Falkum,

(2017). Employment outcome and predictors of competitive employment at 2-year follow-up of a vocational rehabilitation programme for individuals with schizophrenia in a high-income welfare society. Nordic Journal of Psychiatry, 71(3), 180–187.

21.

Glynn,

, & Schaller,

(2017). Predictors of employment outcomes for transition-age state-federal vocational rehabilitation consumers with attention-deficit/hyperactivity disorder. Journal of Vocational Rehabilitation, 47(2), 159–174.

22.

Gonzalez,

, David

A. R.

, & Jeong,

H. K.

(2011). Predicting vocational rehabilitation putcomes of young adults with specific learning disabilities: Transitioning from school to work. Journal of Vocational Rehabilitation, 34, 163–172.

23.

Graham,

, Inge,

, Wehman,

, Murphy,

, Revell,

W. G.

, & West,

(2013). Moving employment research into practice: Knowledge and application of evidence-based practices by state vocational rehabilitation agency staff. Journal of Vocational Rehabilitation, 39(1), 75–81.

24.

Groah,

S. L.

, Libin,

, Lauderdale,

, Kroll,

, DeJong,

, & Hsieh,

(2009). Beyond the evidence-based practice paradigm to achieve best practice in rehabilitation medicine: A clinical review. Physical Medicine and Rehabilitation, 1, 941–950.

25.

Harrell,

, Lee,

K. L.

, & Mark,

D. B.

(1996). Tutorial in biostatistics multivariable prognostic models: Issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in Medicine, 15, 361–387.

26.

Hayward,

B. J.

, & Schmidt-Davis,

(2003). Longitudinal study of the vocational rehabilitation services program final report: VR services and outcomes. Report to the Rehabilitation Services Administration under ED Contract No. HR92022001.

27.

Hayward,

B. J.

, & Schmidt-Davis,

(2005). Longitudinal study of the vocational rehabilitation (VR) services program, third final report: The context of VR services. US Department of Education, Office of Special Education and Rehabilitative Services, Rehabilitation Services Administration.

28.

Huang,

I. C.

, Wang,

Y. T.

, & Chan,

(2013). Employment outcomes of adults with cerebral palsy in Taiwan. Disability Rehabilitation, 35, 228–235.

29.

Humphris,

, Littlejohns,

, Victor,

, O'Halloran,

, & Peacock,

(2000). Implementing evidence-based practice: Factors that influence the use of research evidence by occupational therapists. British Journal of Occupational Therapy, 3, 516–522.

30.

Ialongo,

(2016). Understanding the effect size and its measures. Biochemia Medica, 26(2), 150–163.

31.

Kitson,

(2008). The uncertainty and incongruity of evidence-based healthcare. International Journal of Evidence-Based Healthcare, 6(1), 1–2.

32.

Lalkhen,

A. G.

, & McCluskey,

(2008). Clinical tests: Sensitivity and specificity. Continuing Education in Anesthesia Critical Care Pain, 8(6), 221–223.

33.

Leahy,

M. J.

, Thielsen,

V. T.

, Millington,

M. J.

, Austin,

, & Fleming,

(2009). Quality assurance and program evaluation: Terms, models, and applications in rehabilitation administration. Journal of Rehabilitation Administration, 33(2), 69–82.

34.

Leahy,

M. J.

, Chan,

, Lui,

, Rosenthal,

, Tansey,

, Wehman,

, ... Sherman,

(2014). An analysis of evidence-based best practices in the public vocational rehabilitation program: Gaps, future directions, and recommended steps to move forward. Journal of Vocational Rehabilitation, 41(2), 147–163.

35.

Lusk,

S. L.

(2018). Predictors of successful vocational rehabilitation closure among individuals with substance and alcohol use disorders: An analysis of Rehabilitation Services Administration data 2010–2014. Alcoholism Treatment Quarterly, 36(2), 224–237.

36.

Martin,

, & Martin,

(1989). Current trends: Bridging the gap between research and practice. Journal of Counseling Development, 87, 108–116.

37.

McCluskey,

, & Lovarini,

(2005). Providing education on evidence-based practice improved knowledge but did not change behaviour: A before and after study. BMC Medical Education, 5, 40.

38.

Menon,

, Korner-Bitensky,

, Kastner,

, McKibbon,

, & Straus,

(2009). Strategies for rehabilitation professionals to move evidence-based knowledge into practice: A systematic review. Journal of Rehabilitation Medicine, 41(13), 1024–1032.

39.

Morris,

Z. S.

, Wooding,

, & Grant,

(2011). The answer is 17 years, what is the question: Understanding time lags in translational research. Journal of the Royal Society of Medicine, 104, 510–520.

40.

O'Donnell,

C. A.

(2004). Attitudes and knowledge of primary care professionals towards evidence-based practice: A postal survey. Journal of Evaluation in Clinical Practice, 10, 197–205.

41.

Paul,

G. L.

(1967), Strategy of outcome research in psychotherapy. Journal of Counseling Psychology, 31, 109–118.

42.

Peirce,

C. S.

(1884). The numerical measure of success of predictions. Science, 4(93), 453–454.

43.

Poole,

(2008). Using consciousness-raising principles to inform modern knowledge translation practices in women's health. Canadian Journal of Nursing Research, 40(2), 76–93.

44.

Reilly,

B. M.

, & Evans,

A. T.

(2006). Translating clinical research into clinical practice: Impact of using prediction rules to make decisions. Annals of Internal Medicine, 144, 201e9.

45.

Reimer-Kirkham,

, Baumbusch,

, Schultz,

A. S. H.

, & Anderson,

J. M.

(2007). Knowledge development and evidence-based practice. Insights and opportunities from a postcolonial feminist perspective for transformative nursing practice. Advances in Nursing Science, 30(1), 26–40.

46.

Rice,

M. E.

, & Harris,

G. T.

(2005). Comparing effect sizes in follow-up studies: ROC Area, Cohen's d, and r. Law and Human Behavior, 29, 615–620.

47.

Rumrill,

P. D.

, Merchant,

, Kaya,

, Chan,

, Hartman,

, & Tansey,

(2017). Demographic and service-related correlates of competitive employment outcomes among state-federal vocational rehabilitation clients with learning disabilities: A purposeful selection logistic regression analysis. Journal of Vocational Rehabilitation, 47(2), 123–134.

48.

Royston,

, Moons,

K. G. M.

, Altman,

D. G.

, & Vergouwe,

(2009). Prognosis and prognostic research: Developing a prognostic model. BMJ, 338, b604.

49.

RSA: Reporting Manual for the Case Service Report (RSA-911). (2016). https://www2.ed.gov/programs/rsabvrs/rsa-911-case-service-report-manual-2016.pdf.

50.

Salbach,

N. M.

, Jaglal

S. B.

, Korner-Bitensky,

, Rappolt,

, & Davis,

(2007). Practitioner and organizational barriers to evidence-based practice of physical therapists for people with stroke. Physical Therapy, 87, 1284–1303.

51.

Sherman,

, Leahy,

M. J.

, Del Valle,

, Anderson,

, Tansey,

T. N.

, & Lui,

(2014). Organizational and cultural factors that promote creative best practices in the public rehabilitation program: Findings from a four-state multiple case study. Journal of Vocational Rehabilitation, 41(2), 115–125.

52.

Steyerberg,

E. W.

, & Vergouwe,

(2014). Towards better clinical prediction models: Seven steps for development and an ABCD for validation. European Heart Journal, 35(29), 1925–1931. 10.1093/eurheartj/ehu207

53.

Steyerberg,

E. W

, Harrell,F. E., Jr., Borsboom,

G. J.

, Eijkemans,

M. J.

, Vergouwe,

, & Habbema,

J. D.

(2001). Internal validation of predictive models: Efficiency of some procedures for logistic regression analysis. Journal of Clinical Epidemiology, 54(8), 774–781.

54.

Steyerberg,

E. W.

, Vickers,

A. J.

, Cook,

N. R.

, Gerds,

, Gonen,

, Obuchowski,

, ... Kattan,

M. W.

(2010). Assessing the performance of prediction models: A framework for some traditional and novel measures. Epidemiology, 21(1), 128–138.

55.

Sudsawad,

(2007). Knowledge translation: Introduction to models, strategies, and measures.Austin, TX: Southwest Educational Development Laboratory, National Center for the Dissemination of Disability Research.

56.

Sutton,

R. I.

, & Staw,

B. M.

(1995). What theory is not. Administrative Science Quarterly, 40(3), 371–384.

57.

Tsang,

, Lam,

, Bacon,

, & Leung,

(2000). Predictors of employment outcome for people with psychiatric disabilities: A review of the literature since the mid 80's. Journal of Rehabilitation, 66, 19–31.

58.

U.S. Bureau of Labor Statistics. (2017). Persons with a disability: Labor force characteristics summary. USDL-17-0857.

59.

U.S. Department of Education Office. (2006). The national institute on disability and rehabilitation research-notice of final long-range plan for fiscal years 2005-2009. Fed Reg, 31, 8165–8200.

60.

Van de Ven

A. H.,

, & Johnson,

P. E.

(2006). Knowledge for theory and practice. Academy of Management Review, 31(4), 802–821.

61.

van Houwelingen,

J. C.

, & le Cessie

(1990). Predictive value of statistical models. Statistics in Medicine, 8, 1303–1325.

62.

Vergouwe,

, Steyerberg,

E. W.

, Eijkemans,

M. J.

, & Habbema,

J. D.

(2002). Validity of prognostic models: When is a model clinically useful?Urologic Oncology: Seminars and Original Investigations, 20, 96e107.

63.

Vickers,

A. J.

, & Elkin,

E. B.

(2006). Decision curve analysis: A novel method for evaluating prediction models. Medical Decision Making: An International Journal of the Society for Medical Decision Making, 26(6), 565–574.

64.

Vickers,

A. J.

(2008). Decision analysis for the evaluation of diagnostic tests, prediction models and molecular markers. The American Statistician, 62, 314–320.

65.

Westfall,

J. M.

, Mold,

, & Fagnan,

(2007). Practice-based research — “blue highways” on the NIH roadmap. JAMA, 297(4), 403–406.

66.

Weweiorski,

N. J.

, & Fabian,

E. S.

(2004). Association between demographic and diagnostic factors and employment outcomes for people with psychiatric disabilities: A synthesis of recent re- search. Mental Health Services Research, 6, 9–21.

67.

Winch,

, Henderson,

, & Creedy,

(2005). Read, think, do!: A method for fitting research evidence into practice. Journal of Advanced Nursing Practice, 50, 20–26.

68.

Yu,

, Xu,

, & Zhu,

(2017). A modified Hosmer–Lemeshow test for large data sets. Communications in Statistics - Theory and Methods, 46(23), 11813–11825.

Practice motivated research: Application of an evidence-informed prognostic model in vocational rehabilitation to increase the chance of employment at closure

Abstract

BACKGROUND:

OBJECTIVE:

METHODS:

RESULTS:

CONCLUSIONS:

Keywords

1 Introduction

2 Case study: Evidence-informed counseling decision support application (ablescore)

2.1 Purpose and background

2.2 The problem of practice

2.3 Proposed evidence-informed solution: Ablescore

2.4 Prognosis and prediction research in vocational rehabilitation

3 Development and validation of the prognostic models

3.1 Data and statistical analyses

4.1 Model development

4.3 Evaluation of model’s practice usefulness

5 Results

2 The odds ratios and beta coefficients both estimate the effect size of the predictor on the dichotomous outcome variable; the latter one being the natural logarithm of the former one (Ialongo, C., 2016).

Table 4 Development model performance measure Overall Performance Measure Discrimination Calibration Clinical usefulness R2 Brier Score Predictive Accuracy AUC/ROC Calibration slope Calibration intercept Net benefit 29.3% 0.190 72% 0.78 1.00 0.000 0.252

6.1 Productizing – development of product for practice

7 Discussion

7.1 Implications

4 See Application Use Case Evaluation sub-section of the Product Development section.

7.3 Limitations

Conflict of interest

Footnotes

Acknowledgments

References

²
The odds ratios and beta coefficients both estimate the effect size of the predictor on the dichotomous outcome variable; the latter one being the natural logarithm of the former one (Ialongo, C., 2016).

Table 4
Development model performance measure

Overall Performance Measure Discrimination Calibration Clinical usefulness

R² Brier Score Predictive Accuracy AUC/ROC Calibration slope Calibration intercept Net benefit

29.3% 0.190 72% 0.78 1.00 0.000 0.252

⁴
See Application Use Case Evaluation sub-section of the Product Development section.