Abstract
BACKGROUND:
Youth with disabilities desire competitive employment, but most are placed in sheltered workshops. Illinois launched a pilot project to promote competitive employment for youth with disabilities, using Individual Placement and Support (IPS), an evidence-based vocational model for people with serious mental illness.
OBJECTIVE:
To evaluate the potential of IPS for youth with developmental and/or psychiatric disabilities.
METHODS:
We assessed fidelity to the IPS model, competitive employment outcomes, and implementation barriers over one year in 10 agencies in Illinois implementing new IPS programs for youth with developmental and/or psychiatric disabilities. Assessments included fidelity reviews, employment outcomes, and semi-structured interviews. Project outcomes were compared against national benchmarks for fidelity and employment outcomes.
RESULTS:
All 10 programs successfully implemented IPS with four achieving a fidelity score above 100, the benchmark for good fidelity. Over a 12-month follow-up period, most programs increased quarterly employment rates, reaching a mean employment rate of 36% (SD = 14%) by the fourth quarter, approaching the national benchmark for good employment outcome. A lack of collaboration between systems, competing expectations, and stigma were the main implementation barriers.
CONCLUSION:
The overall good employment rate and fidelity scores suggest that IPS is a promising approach for youth with disabilities.
Keywords
Introduction
Employment during youth’s transition age—the period between ages 16 and 24 years when youth are progressing out of the school system into adulthood—is critical for adult employment. As adults, youth who have no work experiences before reaching adulthood are more likely to be unemployed and in poorer health, and for those who are employed as adults, accrue lower earnings (Gregg & Tominey, 2005; Reine, Novo, & Hammarström, 2004). In the general population, the employment rate for transition age youth with disabilities is much lower than for youth without disabilities ages 16 to 19 years (15% vs. 29%) and for youth ages 20 to 24 years (35% vs. 65%). (U.S. Bureau of Labor, 2015a). The overwhelming majority of transition age youth with disabilities leave high school with little or no vocational experience (Carter, Austin, & Trainor, 2012) and are then directed to sheltered workshops or segregated day programs rather than to competitive employment, leading to demoralization and sharp increases in problem behaviors (Foley et al., 2014; Hasazi et al., 1985; Rusch & Braddock, 2004).
The U.S. Workforce Innovation and Opportunity Act of 2014 identified transition age youth with disabilities as a priority group for supported employment services and required state vocational rehabilitation agencies to provide employment transition services to transition age youth with disabilities. IPS is an evidenced-based supported employment model developed to help adults with severe mental illness to gain competitive employment (Becker & Drake, 2003; Drake, Bond, & Becker, 2012). IPS has consistently demonstrated better competitive employment outcomes compared to other employment models (Marshall et al., 2014). Although IPS was originally developed for adults aged 18 and older, recent pilot studies suggest that it may be suitable for transition age youth with psychiatric disabilities (Ellison et al., 2015; Ferguson, Xie, & Glynn, 2012); however, no research to date has examined whether it may be suitable for people with developmental disabilities.
The current study evaluated IPS for youth with developmental disabilities and/or mental illness. The evaluation assessed fidelity to the principles of IPS, competitive employment outcomes, and implementation barriers.
Methods
The U.S. federal Centers for Medicare and Medicaid awarded the state of Illinois $90 million through the Balancing Incentives Program (BIP) to increase access to non-institutional, long-term services and supports. In response, the Illinois Department of Human Services Division of Rehabilitation Services funded 20 community-based BIP teams to pilot vocational programs during a 20-month period starting July 2014. The project aimed primarily to serve transition age youth with developmental and/or psychiatric disabilities (other people served included adults with disabilities). The project design called for BIP programs to implement one of two vocational models: either Individual Placement and Support (IPS) or customized employment. The current study’s evaluators were not involved in the implementation of these programs.
Study sites
We examined the 10 of the 20 Illinois BIP programs that implemented IPS services for youth with developmental disabilities and/or mental illness initiated between July 2014 and October 2014. (The remaining 10 sites had later start dates, did not serve youth, did not provide information to identify which population they served, or did not implement IPS.) The 10 programs were located in community rehabilitation centers: five in rural, two in suburban, and three in urban regions (See Table 1 for site characteristics). Five agencies (50%) had previously established IPS programs serving adults with mental illness.
IPS program characteristics and fidelity scores
IPS program characteristics and fidelity scores
Four of the 10 programs served exclusively youth with mental illnesses, including depression, post-traumatic stress disorder, attention deficit hyperactivity disorder, and bipolar disorder. Four of the 10 programs served exclusively youth with developmental disorders, including youth with mild developmental disorders, autism, and Down syndrome. The remaining two programs served youth with developmental disabilities and youth with mental illness, many with co-occurring developmental disabilities and mental illness.
The institutional review board of Dartmouth College, which follows the principles outlined in the Declaration of Helsinki, reviewed the project and deemed it program evaluation, not research.
Between June 2015 and October 2015, we collected qualitative and demographic data through phone interviews with the IPS teams and in-person interviews during site visits. We manually recorded the detailed content of the responses from the one-hour interviews, followed by member-checking procedures (Guba & Lincoln, 1989), in which we verbally presented our manual recordings to the IPS teams to confirm that we captured the full content of responses, checking for accuracy, corrections, and/or additional information.
Measures
IPS fidelity
The IPS-25 is a widely-used and well-validated scale to assess adherence to the IPS model (Becker, Swanson, Reese, Bond, & McLeman, 2015). The 25 items are rated on behaviorally-anchored 5-point scales ranging from 1 indicating lack of adherence to 5 indicating close adherence to the IPS model. Total scores range from 25 to 125. Scores between 115–125 denote exemplary fidelity, scores between 100–114 denote good fidelity, scores between 74–99 denote fair fidelity, and scores of 73 and below indicate the program is not implementing IPS. IPS programs scoring with higher fidelity generally have better competitive employment rates (Bond, Peterson, Becker, & Drake, 2012; Kim, Bond, Becker, Swanson, & Reese, 2015).
At the time of the study, two Illinois state agencies, the Division of Mental Health and the Division of Rehabilitation Services, were monitoring fidelity in 72 IPS programs statewide. Three trained fidelity reviewers from these departments conducted 1.5-day fidelity reviews at the 10 study sites, following assessment procedures outlined in the fidelity manual (www.ipsworks.org). These procedures include interviews with the IPS and clinical teams, agency leaders, families, and clients; observations of team meetings and community job development; and chart reviews. Fidelity reviewers prepare a report including ratings for each of the 25 items and recommendations to improve fidelity and for quality assurance.
Quarterly competitive employment rate
Competitive employment means regular employment in an integrated community setting—a job that is available to any person qualified to perform the duties associated with the position (regardless of disability), paying at least minimum wage, in which the person earns the same wage as their co-workers who are doing the same work, and with supervision by personnel employed by the business. (An integrated work setting is one where employees are not exclusively or mostly people with disabilities.) The quarterly competitive employment rate is the number of clients employed for at least one day during a specified three-month period (i.e., a calendar quarter) divided by the total number of clients on the active caseload during that time. The 10 IPS teams provided four quarters of employment data over the course of one year. The four quarters started when the site’s BIP program began enrolling clients. Six sites provided employment data from July 2014 to June 2015 and four sites from October 2014 to September 2015.
Analysis of qualitative data
For each interview, we separated full responses from each question into discrete responses based on similarity of content and entered the detailed content into an electronic spreadsheet. The IPS teams reported between two and seven discrete barriers to implementation. Five researchers independently reviewed the content regarding barriers using the inductive orientation of thematic analysis (Braun & Clarke, 2006). Following the process of independent review, the team met to identify, discuss, and reach consensus on salient and prominent ideas and thematic concepts across the dataset (Braun & Clarke, 2006). This process yielded the common implementation barriers reported herein.
Results
Fidelity
Fidelity scores ranged from 86 to 120 (M = 99.8, SD = 11.8) (See Table 1). Two sites (20%) achieved exemplary fidelity, two (20%) had good fidelity, six (60%) had fair fidelity, and none had poor fidelity.
The average fidelity score across all items was 3.9 (SD = 1.0). On average, sites serving primarily youth with developmental disabilities and sites serving both populations had a score <3 on integration of rehabilitation with mental health treatment through team assignment, integration of rehabilitation with mental health treatment through frequent team member contact, diversity of job types, and diversity of employers. Sites serving primarily youth with developmental disabilities also had scores <3 on competitive jobs. Sites serving both populations had scores <3 on executive team support and assertive engagement and outreach. All sites serving primarily youth with mental illness scored >3 on all items.
Quarterly employment rate
Table 2 shows that the employment rate increased in all sites over the course of four quarters from program start-up until the final follow-up. The fourth quarter employment rate ranged from 14% to 58% (M = 36%, SD = 14%). On average across the 10 sites, the percentage of clients employed increased three-fold from the first quarter (12%) to the fourth quarter (36%). Overall, 50% of sites (n = 5) achieved an employment rate of 41% or more in at least one quarter.
Number of youth served and working during each quarter (3 months) in the first year of implementation
Number of youth served and working during each quarter (3 months) in the first year of implementation
Interviews with the IPS teams revealed three common barriers to implementation: lack of collaboration between systems, competing expectations, and stigma.
Lack of collaboration between systems
Several IPS teams reported difficulties entering the school system to access youth who would benefit from the BIP program. The high schools in Illinois offered a Secondary Transitional Experience Program (STEP), which provided skills training, career counseling, career exploration, and work experience to youth with disabilities. School administrators often believed that youth enrolled in STEP could not simultaneously engage in IPS. They were concerned about losing credit for the employment outcome if the youth enrolled in STEP gained employment while simultaneously enrolled in IPS.
The IPS programs within agencies that did not provide mental health services experienced difficulties integrating mental health and vocational services. For example, one agency, which served people with developmental disabilities, did not provide mental health services for their clients, even though 40% had co-occurring mental illness. In several instances, youth recruited through schools to the IPS program did not have a therapist. Finding mental health providers (e.g., bilingual providers) and establishing collaboration was a lengthy process.
Benefits planners poorly communicated information on the effect of employment on receipt of benefits to families. Some families discouraged their youth from gaining employment for fear of losing the benefits that supported the family financially. Five of the 10 programs reported difficult collaborations with local Vocational Rehabilitation offices. Of these five sites, three had below standard fidelity.
Competing expectations
Parents sometimes undermined the employment efforts of the IPS teams. Some youth from lower socioeconomic status backgrounds came from generations of unemployment. Within this context, some parents did not view employment as a priority. The lack of family engagement led youth, who may have initially expressed interest in employment, to disengage from the employment process. Parents made excuses for youth who missed appointments and sometimes did not encourage participation in the employment process.
The balance between education and employment was another challenge. For example, one team reported an instance where parents did not allow their children to pursue employment if their grades were below a certain level. In these instances, the teams were uncertain whether to support the parents by encouraging education, act as an advocate for the youth to explore their employment interests, or act as a mediator to develop a resolution.
Many providers, clients, and family members had accepted for years that sheltered workshops met the needs and capabilities of people with developmental disabilities. The U.S. Workforce Innovation and Opportunity Act and the BIP program challenged these beliefs, but the IPS teams encountered resistance when they began placing youth with developmental disabilities in competitive employment. Among parents, one common fear was bullying at work. Some Vocational Rehabilitation counselors viewed people with developmental disabilities as incapable of meeting the demands of competitive employment. Some of the clients themselves were reluctant to leave the workshops entirely for fear of losing social networks.
Stigma
The IPS teams noted a resistance to zero exclusion for people with developmental disabilities from Vocational Rehabilitation counselors and from clinical teams referring youth with mental illness (in sites that were new to the IPS model) as well as counselors. These staff were limiting referrals to youth who they felt would benefit from employment, were managing their symptoms, or had the capacity for competitive employment; some parents expressed similar beliefs.
Discussion
The average fidelity score across the 10 IPS programs met the benchmark for good fidelity, but the scores did show a wide distribution ranging from fair fidelity to exemplary fidelity. The average fourth quarter employment rate of 36% approached the benchmark for good employment outcomes (41%), previously defined in IPS programs serving adults with mental illness (Becker, Drake, & Bond, 2014). In the current study, half of the sites exceeded this benchmark during at least one quarter. All sites displayed a pattern of steady increase in employment rate over the first year. Nevertheless, we identified lack of collaboration between systems, competing expectations, and stigma as common barriers to implementation.
Fidelity scores were low on the integration with mental health items because youth with developmental disabilities typically did not receive mental health treatment services. One IPS team coordinated with case managers to ensure their youth were meeting their life goals and to promote competitive employment for those youth not yet enrolled in the program. Integrating the youth’s support services may address this issue. Support services include case managers and other professionals within the special education system who have similar responsibilities as treatment teams in the mental health system. A second factor impacting fidelity may be that during the start-up phase of any program, team members are not only delivering an intervention but are developing and streamlining the process of incorporating the new program within their agency (e.g., educating staff members outside the program, coordinating services), developing relationships with vocational rehabilitation counselors, establishing relationships with employers, and establishing methods of recruitment. Sustained programs have addressed these factors and can focus solely on providing IPS services to clients.
Becker et al. (2014) set the 50th percentile of the average quarterly employment rate of IPS programs in the U.S. national IPS learning community as the benchmark for good employment outcome (41%). Given five sites exceeded this benchmark in at least one quarter, IPS may be as effective for youth as it is for adults in enabling clients to gain employment. We may still consider establishing a benchmark for youth with disabilities such as the rate of employment of youth post high school without disabilities (64%) or the summer employment rate among transition age youth (53%) (U.S. Bureau of Labor, 2015b).
The IPS teams faced several implementation barriers, which may have reduced employment outcomes. State officials could perhaps overcome the lack of system collaboration by implementing state-level and local steering committees. A state-level committee could include VR leadership, leadership from the school system, IPS team leadership, and family members. A local steering committee could consist of VR leadership, local VR counselors, school system representatives, family members, IPS staff, agency leadership, and benefits planners. VR leadership could liaise between the two committees and establish a network of support systems to improve competitive employment outcomes for youth with disabilities.
A similar network among IPS programs serving youth with disabilities could facilitate sharing of strategies to improve access, employment outcomes, and education outcomes. Engaging in a learning community could contribute to improvements in the quality and sustainability of services (Bond, Drake, Becker, & Noel, 2016; Van Duin et al., 2013). Furthermore, this learning community could track employment and education outcomes and develop needed benchmarks for success.
Competing expectations between families, youth, and systems slowed the job development process. Several approaches may overcome this barrier. Motivational interviewing with parents may be effective in identifying reasons for resistance to change or poor engagement in the job development process, and eliciting behaviors that support their youth’s employment and education goals. Targeting benefits counseling to both youth and the parents who are financially dependent on benefits may overcome parents’ discouragement of their youth from employment, which ultimately reduces the risk of youth’s disengagement from the program. Historically, community rehabilitation programs have directed youth and adults with developmental disabilities to sheltered workshops and rarely to integrated competitive employment (Hasazi et al., 1985; Rusch & Braddock, 2004). According to a U.S. national study, only 14% of youth with developmental disabilities have sheltered workshops as an employment goal after high school (Wagner, Newman, Cameto, Garza, & Levine, 2005). Despite their interest in competitive employment, these youth continue to lack employment experience while in high school and the majority are unemployed after completing high school (Carter et al., 2010; Wagner et al., 2005). Promoting outcomes may overcome both systemic beliefs and skepticism of youth’s capacity to work in competitive jobs. Social clubs could replace sheltered workshops to maintain connections within the community while people are working in competitive jobs.
Limitations
The IPS programs did not systematically collect information about referral rates or reasons for lack of referrals across organizations (e.g., schools, local VR, and clinical teams). We also do not know how many of those referred actually received IPS services. The program evaluation did not track supported education outcomes; employment statistics could have been reduced if a significant proportion of youth were prioritizing education. The IPS teams collected the data on quarterly employment rates from their reports, and the accuracy of the data is uncertain. We did not collect data on the rate of cases closed during the evaluation period to examine its effect on the quarterly employment rate.
Conclusion
To our knowledge, this study presents the first evaluation of IPS in programs serving youth with developmental disabilities. IPS appears to be a promising model for this population as well as for youth with mental illness. All programs were able to implement IPS and several achieved good employment outcomes, but collaboration across organizations and entities was difficult. Researchers need to learn more about implementing IPS to fidelity in both populations. It may be necessary to modify fidelity items (e.g. mental health integration items) and/or modify the training given to IPS supervisors and employment specialists who serve these populations. The development of a learning community could contribute to implementation and sustainment. Controlled trials comparing IPS to other vocational services for transition age youth with disabilities should be one next step for assessing the effectiveness of IPS in serving this population.
Conflict of interest
None to report.
Footnotes
Acknowledgments
The authors thank the State of Illinois and the Individual Placement and Support team members at the community rehabilitation centers for their contribution to this report and the services that they provide to the youth described in this evaluation, and Mr. Doug Morton, Ms. Deborah Becker, and the fidelity reviewers for their expertise in employment services.
This study was funded by the Illinois Department of Human Services Division of Rehabilitation Services and SourceAmerica. The authors thank SourceAmerica for their support.