Vocational rehabilitation experiences of transition-age youth with autism spectrum disorder across states: Prioritizing modifiable factors for research

Abstract

BACKGROUND:

Tremendous state-level variation exists in the Vocational Rehabilitation (VR) service experiences of transition-age youth with autism spectrum disorder (TAY-ASD). Individual-level factors alone have not adequately explained differences in VR service utilization and employment outcomes for these youth, and the socio-demographic and economic state-level factors examined to date are largely non-modifiable.

OBJECTIVE:

This study aimed to identify and prioritize modifiable state-level policies and practices that may affect VR service utilization and outcomes of TAY-ASD.

METHODS:

We used a modified, two-round Delphi approach with 12 subject matter experts within this mixed methods study.

RESULTS:

Twenty-three factors of VR service use and outcomes were rated in the top half of importance scores – most of which were also in the top half of modifiability scores. These factors were organized into five themes regarding capacity, efficient and effective VR processes, innovation, inter-agency efforts, and staff training and competency. All participants agreed that these themes captured the main types of factors that might be driving state-level variability in VR outcomes.

CONCLUSIONS:

This study identified key themes and modifiable factors for investigation within next-step research. Studying these themes and factors may enable researchers to learn from what is working in some states to enhance employment outcomes for TAY-ASD.

Keywords

Vocational rehabilitation (VR)transition-age youth (TAY)autism spectrum disorder (ASD)state systems transition

1. Background

About half of transition-age youth with autism spectrum disorder (TAY-ASD) in the U.S., 53–58% across population-based studies, will work between high school and their early 20 s – a rate of employment that is much lower than that of their peers with other types of disabilities (Roux et al., 2013; Roux, Shattuck, Rast, Rava, & Anderson, 2015; Shattuck et al., 2012). Growing numbers of transition-age youth (TAY) and adults with autism are found eligible for help from state-administered Vocational Rehabilitation (VR) agencies to find and maintain employment (Burgess & Cimera, 2014; Smith & Lugas, 2010). The number of closed VR cases for adults with autism (ages 18–64 years) more than doubled from 7,428 in federal fiscal year (FFY) 2009 to 17,753 in FFY 2014 (Roux, Rast, Anderson, & Shattuck, 2016), and approximately 60% of those with autism who exited VR services in FFY 2014 attained employment. We know less about how TAY-ASD utilize and benefit from VR services.

VR service utilization and subsequent employment outcomes varies greatly by state for people with autism (Roux et al., 2016). For example, 39% of VR participants with autism in North Dakota actually received services before their case closed, compared to 88% in Pennsylvania. Only 29% of VR service users with autism in New Mexico exited with employment, compared to 79% in Alabama. The percentage of VR consumers with autism who received VR services during secondary school varied by 70 points across states. State-level variation has also been found in longer-term trend studies of VR outcomes for TAY-ASD (Burgess & Cimera, 2014; Migliore, Butterworth, & Zalewska, 2014), and in studies of TAY with other disabilities (Honeycutt, Thompkins, Bardos, & Stern, 2015).

Across states, a host of factors related to the organization, delivery and coordination of VR services may influence the early employment experiences of TAY-ASD. Some states opt to prioritize VR services for people who have an intellectual disability (ID), making access to vocational services difficult for people with autism without co-occurring ID (Nicholas, Hodgetts, et al., 2017; Taylor & Seltzer, 2011), even though these individuals are likely to face employment challenges whether they have accompanying intellectual impairment or not (Baldwin, Costley, & Warren, 2014; Nord, Stancliffe, Nye-Lengerman, & Hewitt, 2016; Taylor & Seltzer, 2011). Conversely, some people with autism are deemed too severely impaired to benefit from services (Lawer, Brusilovskiy, Salzer, & Mandell, 2009). Even among those who do meet eligibility criteria for services, consumers and families may have difficulty navigating the complex and fragmented service delivery system (Anderson, Roux, Kuo, & Shattuck, 2018; Honeycutt, Bardos, & McLeod, 2015; Pillay & Brownlow, 2016).

The emerging knowledge base about the challenges and strengths of individuals with autism has also been slow to translate into effective interventions and services to aid TAY-ASD in the transition from school to work (Nicholas, Zwaigenbaum, et al., 2017; Taylor et al., 2012; Wehman, Schall, et al., 2014). People with autism have unique vocational challenges related to their social-cognitive, communication and behavioral characteristics (Hendricks, 2010; Müller, Schuler, Burton, & Yates, 2003; Richards, 2012), including difficulty completing applications and interviews, coping with sensitivity to environmental lights and sounds, understanding workplace politics, and following complex directions (Hurlbutt & Chalmers, 2004; Müller et al., 2003; Richards, 2012). Adults with autism may also bring strengths to their jobs such as strong memory, intense focus, honesty, and knowledge about special interest areas (Baldwin et al., 2014; Gotham et al., 2015).

Most conceptual frameworks of human development and the experience of obtaining employment are anchored in an ecological approach that notes the important influence of individual, family, community, policy and cultural factors (Anderson et al., 2018). Individual-and family-level correlates of employment for people with developmental disabilities are well-established. Studies using longitudinal datasets have found better employment outcomes for adults with ASD who had higher intellectual ability (Chan, et al., 2017; Taylor & Seltzer, 2011), independence in daily living skills (Chan, et al., 2017; Taylor & Mailick, 2014), higher communication skills (Roux et al., 2013), better social skills (Chiang, Cheung, Li, & Tsai, 2013), initiative and motivation to work (Holwerda, van der Klink, Groothoff, & Brouwer, 2012), and personal hygiene care (Chan, et al., 2017); as well as exposure to more services (Taylor & Mailick, 2014), high maternal positivity (Howlin & Magiati, 2017), higher household income (Roux et al., 2013; Shattuck et al., 2012), higher levels of parental education (Chiang et al., 2013), and parental involvement in job finding (Holwerda et al., 2012). Although not specific to TAY-ASD, youth who received special education services were more likely to be employed after high school if they were employed during high school and if parents had higher expectations for postsecondary employment (Wehman et al., 2015).

Studies of TAY-ASD who received VR services have also focused primarily on identifying individual-level correlates of successful employment. These include attaining additional postsecondary education between the time of VR application and case closure (Migliore, Timmons, Butterworth, & Lugas, 2012); less time in the VR program (Migliore et al., 2012); and receipt of more VR services, higher levels of education, and not receiving Supplemental Security Income (SSI) benefits (Kaya et al., 2016). TAY-ASD were three to four times more likely to attain competitive employment following VR services if they received on-the-job support and job placement services (Kaya et al., 2016; Migliore et al., 2012). Receipt of supportive employment services was associated with a 20% higher employment rate among high school graduates with ASD who received SSI and/or Social Security Disability Insurance (SSDI) benefits – a group particularly at risk for poor employment outcomes (Wehman, Chan, Ditchman, & Kang, 2014). Others found cost of services to be predictive of positive employment outcomes for young VR users with autism (Butterworth, Migliore, & Timmons, 2010). However, individual and family-level demographic and service receipt factors together only explained 32% of variance in employment outcomes for TAY-ASD following VR services in one study (Kaya et al., 2016), and only 14% of variance in earnings and 11% of variance in work hours in another study (Migliore et al., 2012).

Very little research has explored the institutional, community or policy level factors that may be associated with the wide variation among states in VR service utilization and outcomes for TAY and adults with ASD. Many VR studies rely on data readily available in the Rehabilitation Services Administration (RSA) datasets that contain records for all VR applicants and service users. The RSA dataset does not include variables related to more distal levels of ecological influence on employment outcomes; thus, research that relies entirely on the RSA dataset has been unable to detect the influence of factors beyond the level of individual characteristics and types of services received. Little guidance is available to help prioritize which additional state-level factors might be most beneficial to study. Preliminary evidence suggests that TAY-ASD from states who receive transition services beginning at age 14 (versus 16) are more likely to be employed four years after high school (Cimera, Burgess, & Wiley, 2013). Most investigation of state-level factors, however, has focused on VR service users across disability categories and has not produced autism-specific findings.

Several findings regarding VR service users across disability groups provide clues about state-level factors that could also impact service utilization and employment outcomes for TAY-ASD. Fixed factors, such as state unemployment rate and state per capita income, are known to modify individual-level predictors of VR outcomes for both TAY and adults with disabilities (Alsaman & Lee, 2016; Chan, et al., 2016; Chan, et al., 2013; Giesen & Cavenaugh, 2013). Increased amounts of state VR grant funding per consumer were correlated with a higher proportion of TAY who applied for VR services (Honeycutt, Thompkins, et al., 2015); although application rates fell as state unemployment rates increased. Further, increases in youth labor force participation in a state were positively correlated with both the ratio of TAY who applied for VR services and the ratio that became employed. An inverse correlation was found between the cost of VR services per youth and the proportion of youth applicants who received services (Honeycutt, Thompkins, et al., 2015). Yet, none of these studies yielded easily-modifiable factors that could be feasibly addressed through changes to policy and practice.

Other cross-disability employment studies point to institutional and policy factors that may be more modifiable. A case study of five states highlighted the need for alignment of employment policy, funding structures, and data collection to achieve better integrated employment rates (Hall, Freeze, Butterworth, & Hoff, 2011). Others have called for collaboration of public schools and adult-serving agencies to achieve a more seamless transition between school and work via (Certo et al., 2003; Fabian & Luecking, 2012; Luecking & Luecking, 2015), paired with post-school follow-up to ensure students are continuing to progress toward competitive employment (Rusch & Braddock, 2004). The role of public schools was also highlighted in a study finding that students with severe disabilities were 15 times more likely to have a paid, summer job if their teacher expected they would work (Carter et al., 2010), with teachers also being the most frequent source of help in finding competitive, summer employment. Less is known about whether these factors are correlates of successful employment for TAY-ASD.

1.1. Purpose of this study

Current national-level efforts to improve employment outcomes for persons with disabilities include the implementation of the federal Workforce Innovation and Opportunity Act of 2014 (WIOA), which provides regulations governing VR services. One of WIOA’s key aims is to reach youth with disabilities earlier with employment interventions, and each state currently has an approved plan for the provision of pre-employment transition services (access at rsa.ed.gov), which will be updated every few years. Knowing which aspects of VR-related policy and practice might exert the most influence on VR outcomes for TAY-ASD is critical for guiding and refining state disability employment policy. Yet, the current state of knowledge provides insufficient evidence for informing system change efforts specific to those with ASD.

We sought to identify and prioritize modifiable aspects of policy and practice that might affect VR service utilization and outcomes of TAY-ASD through a modified Delphi approach with a panel of subject matter experts. This approach is responsive to calls for collaborative, stakeholder-informed research in the quest to understand resource capacity, innovation, and policy that support employment for people with autism (Nicholas, Hodgetts, et al., 2017).

2. Methodology

We used a modified Delphi approach with subject matter expert stakeholders to identify and prioritize modifiable state-level policies and practices that may affect VR service utilization and outcomes of TAY-ASD. Literature review and stakeholder interviews were used to elicit ideas about factors that may affect service use and outcomes, followed by two rounds of online surveys. This study was deemed exempt by the Drexel University Institutional Review Board.

2.1. Delphi approach

Delphi methods provide a structured approach to group communication among a panel of experts on a specific issue or topic that is in need of informed judgment to guide decision-making (Adler & Ziglio, 1996; Pare, Cameron, Poba-Nzaou, & Templier, 2013). While some Delphi studies aim to reach consensus on a topic, this is not always the purpose. Delphi methods are also used for generating ideas and gathering insights when there is a lack of supporting evidence to guide research (Adler & Ziglio, 1996; Powell, 2003). For example, researchers have used variations of Delphi methods to develop possible ideas or alternatives (Adler & Ziglio, 1996; Hsu & Sandford, 2007), to set research priorities (Sawford, Dhand, Toribio, & Taylor, 2014), or to identify and prioritize factors affecting an issue (Schmidt, Lyytinen, & Mark Keil, 2001).

A number of modifications to traditional Delphi techniques have evolved since its origination in the 1950’s (Okoli & Pawlowski, 2004). Traditionally, Delphi methods used in-person meetings with experts. As technology evolved, changes to the Delphi resulted in a collection of techniques referred to as modified Delphi approach or process. We used online survey methods and a specific sequence of steps, suggested by Mead and Moseley (2001), in which the panel is selected, researchers formulate questions, input is gathered from the panel, responses are reduced and categorized by the researchers, the panel rates the resultant statements, and the researchers determine areas of agreement.

The traditional Delphi approach typically uses several rounds of questionnaires, often beginning with a round of open-ended questions to elicit an initial pool of ideas (Hasson, Keeney, & McKenna, 2000). Because this technique can generate an excessive number of items for the first survey round, we employed a modified Delphi technique in which a pre-generated list of items for ranking is used in the first round (Hsu & Sandford, 2007; Keeney, Hasson, & McKenna, 2001).

As there is no set of rules for determining consensus when using a Delphi process (Hsu & Sandford, 2007), it is imperative that the researchers establish rules for decisions including how items will be eliminated, categorized, or restated (Mead & Moseley, 2001). We reviewed other published studies that established and prioritized items based upon their relevance and feasibility of measurement, which have judged consensus using agreement on Likert score ratings (Lindsay, Schull, Bronskill, & Anderson, 2002). We established agreement within Round 1 as having a mean Likert rating that fell above the median rating for all items. However, we also chose to consider and report both convergent and divergent qualitative comments, which is a published modification of Delphi studies that do not focus exclusively on convergence (Kendall, Kendall, Smithson, & Angell, 1992).

2.2. Study participants

The quality of collective input gained through a Delphi process is thought to be related to the qualifications of participants, and their willingness and capacity to participate (Adler & Ziglio, 1996; Powell, 2003). Heterogeneous panels composed of 5–20 subject matter experts from a variety of backgrounds and perspectives are useful when areas of uncertainty are being explored and when gathering input about current problems (versus forecasting future trends) (Murphy et al., 1998; Rowe & Wright, 2001). Our modified Delphi approach used the experience and knowledge of participants from across a diverse geographic area, and eliminated power dynamics commonly found in face-to-face group processes by using structured, anonymous methods of communication and feedback.

We recruited stakeholder participants who had topical expertise and/or practical experience in employment for people with autism or other developmental disabilities (DDs), with a focus on VR services. We targeted representation across different types of professional roles, primary disability orientation (autism, intellectual and developmental disabilities, or both), and whether the individual’s work was focused at the national versus state level. Stakeholder panelists were initially identified from within a network of professional contacts considered to be national-level experts on disability and employment. We then used snowball sampling techniques – asking contacts to refer us to additional national experts in employment for people with autism and other DDs.

Twelve experts agreed to participate in the study, which required a total of two to three hours of time per participant. Their professional roles included administrators of state VR or Pre-Employment Transition Services programs, directors of employment services and/or transition programs, university faculty members and/or researchers in special education and/or autism services, public policy specialists for adult autism services or employment for people with disabilities, and representatives of adult autism self-advocacy organizations. Many (67%) had interacted with, or been employed by, a VR agency for greater than 5 years and four (33%) had over 20 years of experience working with VR agencies. Nearly all (n = 11) had been involved in work concerning employment for people with disabilities for more than 5 years and 42% had more than 20 years of experience. All reported having at least some experience with employment issues for people with autism, and 50% had more than 10 years of experience regarding this topic. Eight of the 12 panelists were involved in work that required in-depth understanding of employment services across multiple states or the nation; the remaining four were state focused.

2.3. Delphi survey construction

In preparation for construction of the Delphi survey, we first conducted a literature review to devise an a priori list of factors operating in state-level environments that might influence VR service utilization and employment outcomes. This list was derived using rapid review techniques, which mapped key concepts from existing publications to enhance the scientific foundation of the Delphi discussion (Kelly, Moher, & Clifford, 2016). Rapid reviews provide sufficient detail to guide discussion and inform panelists of prior findings, but do not aim to be exhaustive or definitive.

The literature review was limited to studies, literature, or documents published between the years 2000 and 2017. Our search strategy included online databases (e.g., Science Direct, Google Scholar, PubMed), forward and backward citation searches of reference lists from found articles, internet searches of grey literature, and searches of the RSA website and websites of organizations which conduct research regarding VR. Specific sources included journal publications, government reports, VR state plans and VR statewide needs assessments. We conducted searches using combinations of the following terms: vocational rehabilitation OR employment; autism OR disability; state OR state-level; transition-age youth; and factors, outcomes, determinants, OR predictors. Given the limited amount of publications concerning VR service users with autism, the search was inclusive of articles concerning VR users with other types of disabilities.

We used a spreadsheet to record factors that were known to, or hypothesized to, impact VR service use or outcomes. Our literature search yielded a list of 95 factors. We then narrowed this list by excluding factors not related to state-level or systems-level policy or practice. For example, we excluded individual-level and family-level factors (e.g., attitudes of TAY about employment, parental expectations about whether youth would work). We then excluded factors that were not feasibly modifiable through changes to policy and practice. However, we included factors related to the population of people served by a state’s VR caseload. We combined any factors which were topically related. Following this winnowing process, 36 factors remained, which we organized into four categories to construct our a priori list of factors:

State VR agency policy and administration. This category included policies and practices a state VR agency could use that might affect services and related outcomes for TAY. Subcategories included VR administration and processes (e.g., number of people on the waitlist for state VR services), financing (e.g., how much a state spends on VR services), and special initiatives (e.g., specialized training to help VR counselors support school-to-work transition).

Autism-specific state policies and programs. This category included special policies or programs used in a state to improve VR services for people with ASD. Examples included use of specialized ASD training and technical assistance for VR counselors, and use of policy tools like an autism state plan to address the needs of TAY and adults.

Other state-level factors. This category included state policies or practices that are not autism specific but nonetheless might affect employment outcomes or job opportunities. Examples included the number of community rehabilitation agencies in the state, whether the state allows employment in sheltered workshop settings, availability of technical assistance to support integrated employment, and whether funding for employment services in the state is aligned with the state’s Employment First philosophy.

Characteristics of the state’s served population or caseload. This category included aspects of a state’s served population that are influenced by policies, finances and procedures for targeting services to specific subpopulations. Examples included the size of a state’s VR program, and the proportion of the VR caseload who were transition-age when they applied for VR services.

We emailed the a priori list of factors to our expert panelists for review, and followed up with a 30–60 minute phone interview asking participants to comment on the importance of the listed factors and to nominate additional factors, a technique used in previous Delphi studies (Nambisan, Agarwal, & Tanniru, 1999). Delphi approaches often begin with an open-ended step of idea generation, such as ours, using questions to elicit opinions from experts about what they feel is important (Powell, 2003; Rowe & Wright, 2001). We recorded the interviews but did not transcribe them, instead taking detailed field notes annotated with key words. The lead author reviewed the recordings and notes, and used checklists and matrices to record participant perspectives on a priori factors and additional nominated factors (Miles & Huberman, 1984). Techniques were borrowed from Rapid Identification of Themes from Audio Recordings (RITA) - a type of Rapid Evaluation and Assessment Method beneficial for efficient identification of participant perspectives within a research study (McNall & Foster-Fishman, 2007; Neal, Neal, VanDyke, & Kornbluh, 2015).

We compiled our interview data into an expanded list of 108 factors. Three members of the research team reviewed these factors. We eliminated items if they were either unrelated to state-level VR outcomes of individuals, low in perceived modifiability, or low in perceived impact. For example, if a factor only affected VR service users at the local level, it was considered low in perceived impact. We then combined items that were closely related. A final list of 40 factors was used to formulate the questions for Delphi Round 1.

2.4. Delphi survey Round 1

The purpose of Round 1 was to prioritize potential factors based on their importance and perceived modifiability. We conducted the survey online, which took respondents took an average of 30–45 minutes to complete. An email invitation and follow-up reminders were sent until all stakeholders had at least partially completed the survey. In Round 1, we asked participants to rate two statements for each of the 40 factors: 1) “This factor is related to VR service utilization and/or employment outcomes,” and 2) “This factor would be possible to modify through changes to policy or practice.” A 6-point Likert scale was presented along with a “do not know” option. Points were assigned to ratings (1 = strongly disagree, 2 = disagree, 3 = somewhat disagree, 4 = somewhat agree, 5 = agree, 6 = strongly agree), and “do not know” responses were tallied but not included in the analysis. Participants were also given an opportunity to write clarifying comments and to nominate any additional items for consideration.

We calculated mean scores and standard deviations per question. A median score was used to divide items into those with higher versus lower ratings of importance. All items are presented in Table 1; however, we only retained items with higher importance ratings for use in Round 2, a technique used in prior modified Delphi factor research (Schmidt et al., 2001). The lead author grouped the items with high importance into expanded thematic categories, assigning a conceptual label and a definition for each theme. Qualitative comments were also grouped by theme and used to justify placement of factors into categories. Two co-authors reviewed the groupings, conceptual labels, and definitions and suggested revisions for clarity.

Table 1
Round 1 Delphi survey ratings of factors that might influence Vocational Rehabilitation services and outcomes

Key Factors Importance rating (mean, SD) Modifiability rating (mean, SD)

1. VR policy changes or new programs are evaluated to determine association with outcomes for certain populations (e.g., ASD) 5.7 (0.7) 5.5 (0.9)

2. Capacity to provide a wide range of jobs to meet a variety of skills and interests 5.7 (0.9) 5.5 (1.2)

3. State has adequate number certificate or degree-granting programs for training rehabilitation counselors (or allows related degrees) 5.6 (0.5) 5.4 (0.7)

4. Capacity of transition support providers to meet statewide needs 5.6 (0.5) 5.6 (0.5)

5. Degree to which a state uses Medicaid waivers to fund employment supports for people with autism 5.6 (0.5) 5.6 (0.7)

6. Use of automated systems for data tracking, so real time counts of consumers with autism are available for planning 5.6 (0.7) 5.6 (0.7)

7. Ease of applying for VR services 5.5 (1.0) 5.6 (0.7)

8. Autism training and technical assistance for VR counselors 5.5 (0.7) 5.5 (0.9)

9. State uses special programs, demonstration projects, or innovation and expansion grant funds to address employment outcomes 5.5 (0.7) 5.6 (0.7)

10. Connectedness of the state VR agency with the labor market 5.5 (1.0) 5.4 (1.2)

11. State education system fosters high or low expectations for students with autism 5.5 (1.3) 5.6 (1.3)

12. State creates unique supports to meet needs of VR service users with autism 5.4 (0.8) 5.6 (1.0)

13. State uses VR assessment tools and assessment methods tailored to needs of people with autism 5.4 (1.0) 5.3 (1.1)

14. Difficulty obtaining services for those who have ASD without intellectual disability 5.4 (1.6) 5.3 (1.4) *

15. Competency of community rehabilitation providers to meet the needs of VR service users with autism. 5.4 (0.8) 5.5 (0.7)

16. State Department of Education encourages use of special education programming for 18–21 year olds to provide vocational training 5.4 (0.9) 5.6 (0.7)

17. Length of VR application process 5.3 (1.3) 4.6 (1.5) *

18. Aligning funding for VR services with Employment First philosophy 5.3 (0.9) 5.4 (1.0)

19. State VR agency uses specialty caseloads (e.g., autism specialists, transition specialists) 5.3 (0.8) 5.3 (0.8)

20. Technical assistance to support integrated employment 5.3 (0.8) 5.2 (1.2) *

21. Consultants are available throughout the state to help people understand how to keep public benefits while working. 5.3 (0.9) 5.3 (0.8) *

22. VR counselors consistently attend transition planning meetings for special education students when appropriate 5.3 (1.0) 5.1 (0.9) *

23. State has champions for autism who help advance needed changes 5.2 (0.8) 4.8 (1.6)

24. Consistent implementation of VR policies across a state 5.2 (1.2) 5.0 (1.2)

25. Manageability of VR caseloads 5.2 (1.2) 5.3 (0.9)

26. State VR needs assessment and/or state VR plan calls out specific needs of youth and adults with autism 5.2 (1.1) 5.3 (0.9)

27. State VR agency encourages use of applied behavior analysis (ABA) techniques when providing supports for people with autism 5.1 (1.3) 5.4 (1.3)

28. State VR administrators regularly communicate with and listen to stakeholders 5.1 (1.2) 5.0 (1.5)

29. Types of VR fee structures (e.g., flat rate vs. fee-for-service) 5.0 (1.3) 5.5 (0.5)

30. Consistent reimbursement rates for employment services across VR and Medicaid 5.0 (0.8) 5.3 (0.7) **

31. Sharing of data across state agencies (for planning purposes) 5.0 (0.9) 5.2 (1.0)

32. Extent to which state uses discovery and customized employment 4.9 (1.5) 4.9 (1.5)

33. Extent to which state uses contracted community rehabilitation providers for service delivery 4.9 (1.7) 5.3 (1.3)

34. Which state department VR is organized within (e.g., Education, Labor) 4.7 (1.0) 5.0 (1.0)

35. State allows employment in sheltered workshops 4.7 (1.5) 4.9 (1.2)

36. Avoiding conflicts of interest (e.g., provider/agency who assesses employability is the same provider/agency who implements services) 4.6 (1.4) 5.0 (1.2)

37. State agencies actively interact and education each other about what they do to advance employment outcomes 4.4 (1.6) 5.0 (1.2)

38. Differential reimbursement rates to incentive community rehabilitation providers to take on more intensive cases 4.3 (1.0) 5.3 (1.1)

39. Waitlists for VR services 4.2 (1.8) 4.1 (1.7)

40. Tailoring policies, programs, and practices to different regions of a state 3.9 (1.7) 4.1 (1.5)

Key Factors	Importance rating (mean, SD)	Modifiability rating (mean, SD)
1. VR policy changes or new programs are evaluated to determine association with outcomes for certain populations (e.g., ASD)	5.7 (0.7)	5.5 (0.9)
2. Capacity to provide a wide range of jobs to meet a variety of skills and interests	5.7 (0.9)	5.5 (1.2)
3. State has adequate number certificate or degree-granting programs for training rehabilitation counselors (or allows related degrees)	5.6 (0.5)	5.4 (0.7)
4. Capacity of transition support providers to meet statewide needs	5.6 (0.5)	5.6 (0.5)
5. Degree to which a state uses Medicaid waivers to fund employment supports for people with autism	5.6 (0.5)	5.6 (0.7)
6. Use of automated systems for data tracking, so real time counts of consumers with autism are available for planning	5.6 (0.7)	5.6 (0.7)
7. Ease of applying for VR services	5.5 (1.0)	5.6 (0.7)
8. Autism training and technical assistance for VR counselors	5.5 (0.7)	5.5 (0.9)
9. State uses special programs, demonstration projects, or innovation and expansion grant funds to address employment outcomes	5.5 (0.7)	5.6 (0.7)
10. Connectedness of the state VR agency with the labor market	5.5 (1.0)	5.4 (1.2)
11. State education system fosters high or low expectations for students with autism	5.5 (1.3)	5.6 (1.3)
12. State creates unique supports to meet needs of VR service users with autism	5.4 (0.8)	5.6 (1.0)
13. State uses VR assessment tools and assessment methods tailored to needs of people with autism	5.4 (1.0)	5.3 (1.1)
14. Difficulty obtaining services for those who have ASD without intellectual disability	5.4 (1.6)	5.3 (1.4) *
15. Competency of community rehabilitation providers to meet the needs of VR service users with autism.	5.4 (0.8)	5.5 (0.7)
16. State Department of Education encourages use of special education programming for 18–21 year olds to provide vocational training	5.4 (0.9)	5.6 (0.7)
17. Length of VR application process	5.3 (1.3)	4.6 (1.5) *
18. Aligning funding for VR services with Employment First philosophy	5.3 (0.9)	5.4 (1.0)
19. State VR agency uses specialty caseloads (e.g., autism specialists, transition specialists)	5.3 (0.8)	5.3 (0.8)
20. Technical assistance to support integrated employment	5.3 (0.8)	5.2 (1.2) *
21. Consultants are available throughout the state to help people understand how to keep public benefits while working.	5.3 (0.9)	5.3 (0.8) *
22. VR counselors consistently attend transition planning meetings for special education students when appropriate	5.3 (1.0)	5.1 (0.9) *
23. State has champions for autism who help advance needed changes	5.2 (0.8)	4.8 (1.6)
24. Consistent implementation of VR policies across a state	5.2 (1.2)	5.0 (1.2)
25. Manageability of VR caseloads	5.2 (1.2)	5.3 (0.9)
26. State VR needs assessment and/or state VR plan calls out specific needs of youth and adults with autism	5.2 (1.1)	5.3 (0.9)
27. State VR agency encourages use of applied behavior analysis (ABA) techniques when providing supports for people with autism	5.1 (1.3)	5.4 (1.3) **
28. State VR administrators regularly communicate with and listen to stakeholders	5.1 (1.2)	5.0 (1.5)
29. Types of VR fee structures (e.g., flat rate vs. fee-for-service)	5.0 (1.3)	5.5 (0.5) **
30. Consistent reimbursement rates for employment services across VR and Medicaid	5.0 (0.8)	5.3 (0.7) **
31. Sharing of data across state agencies (for planning purposes)	5.0 (0.9)	5.2 (1.0)
32. Extent to which state uses discovery and customized employment	4.9 (1.5)	4.9 (1.5)
33. Extent to which state uses contracted community rehabilitation providers for service delivery	4.9 (1.7)	5.3 (1.3)
34. Which state department VR is organized within (e.g., Education, Labor)	4.7 (1.0)	5.0 (1.0)
35. State allows employment in sheltered workshops	4.7 (1.5)	4.9 (1.2)
36. Avoiding conflicts of interest (e.g., provider/agency who assesses employability is the same provider/agency who implements services)	4.6 (1.4)	5.0 (1.2)
37. State agencies actively interact and education each other about what they do to advance employment outcomes	4.4 (1.6)	5.0 (1.2)
38. Differential reimbursement rates to incentive community rehabilitation providers to take on more intensive cases	4.3 (1.0)	5.3 (1.1)
39. Waitlists for VR services	4.2 (1.8)	4.1 (1.7)
40. Tailoring policies, programs, and practices to different regions of a state	3.9 (1.7)	4.1 (1.5)

^(*) Items ranked in top half of importance but lower half of modifiability,

^(**)Items ranked in top half of modifiability but in the bottom half of importance.

Bolded numbers denote items rated at or above the median mean importance rating.

2.5. Delphi survey Round 2

The purpose of Round 2 was to validate the accuracy and importance of new categorical themes that emerged as an underlying structure for important factors of VR service utilization and employment outcomes for TAY-ASD. The results from Round 1 were fed back to participants for rating in Round 2. The survey took 15–20 minutes to complete and included only the items that were ranked at or above the median score for importance in Round 1, categorized by themes. We provided participants with a working definition for each thematic category and the corresponding factors that were grouped into that theme. We then asked participants: 1) whether the five major themes adequately captured the main types of factors that might be driving state-level variability in VR outcomes, 2) whether individual items were correctly placed into themes, and 3) whether the identification of factors and thematic groupings was useful for thinking about strategies for improving policy and practice. Participants rated the importance of each theme for improving the outcomes of state VR services for TAY-ASD, using Likert scales identical to those in the previous round; however, the “don’t know’ option was not provided. Participants could suggest revisions to thematic groupings and their conceptual labels.

Round 2 survey participants also rated the perceived importance and modifiability of any additional factors nominated during Round 1, using the same scale from Round 1. We also presented a final opportunity to nominate any additional factors. A summary of the Delphi process is outlined in Fig. 1.

Fig. 1.

Research Design.

3. Results

3.1. Delphi results for Round 1 survey

Ten of the 12 study participants (83%) fully completed the Round 1 survey. Factors and rating scores are listed in Table 1. Mean scores of importance ranged from 3.9 to 5.7. The median mean importance score was 5.3. The mean modifiability scores ranged from 4.1 to 5.6 across all items with a median of 5.3. There were only two factors for which more than 20% of experts indicated a “don’t know” response regarding importance: which department VR is housed within, and fee structures. Items ranked high in one domain were not necessarily ranked high in the other. Five items were ranked in the top half of importance but lower half of modifiability (*), and three items ranked in the top half of modifiability but in the bottom half of importance (**). One additional item was nominated during Round 1 - proactive follow-up with youth following exit from VR services. This item was added to the Round 2 survey.

Nine participants provided a total of 135 qualitative comments to elaborate on their ratings. We identified five general themes within these comments.

3.1.1. Capacity to provide VR services and job opportunities

Participants underscored the need for diverse job opportunities to meet a wide range of VR service user goals and skills. One said, “The VR system needs to be more cued up for providing multiple opportunities for work for youth particularly. Very rarely is the first job the final job. So the system needs to go beyond one and done.” Commenters pointed to a public perception that VR ‘funnels’ people with ASD into job types that are narrow in scope, and one suggested, “Large caseloads may contribute to use of cookie-cutter approaches to job finding.” Three commented that job opportunities could be increased by building the capacity of other job service agencies, such as One Stop/Career Link centers (now called American Job Centers), to serve more people with autism. However, others cautioned that too much focus on meeting the needs of the labor market might “result in ‘funneling’ people toward jobs that are seen as ‘hot,’ without regard to the individual’s interests, needs, or skills.” Several participants commented on the role of sheltered workshops, such as, “When sheltered workshops exist, they are a constant ‘temptation’ for vocational counselors looking for an easy way to close a case… It becomes much harder for someone with significant needs to achieve a competitive integrated outcome.”

Participants stated that staffing capacity is subject to the influence of funding, fiscal conservativeness of the state legislature, and administrative decisions about distribution of staff across the state. Several were concerned about availability of qualified staff to provide benefits counseling, with one stating, “[VR]… just does not have capacity to do this at the level it really needs to be done.” In regard to capacity of pre-service training programs for rehabilitation counselors, two participants suggested that research examine states that allow VR to hire providers with related degrees or certification, such as board certified behavior analysts.

3.1.2. Efficient and effective VR processes

Several participants mentioned the critical importance of evaluation with comments such as, “Every VR program or policy should be assessed to ensure it is actually leading to positive outcomes.” They also suggested use of purposeful, automated data tracking to better meet service and training needs. One participant described use of real-time data to target autism training toward VR counselors whose caseloads have a higher ratio of service users with ASD. Another said, “Data tracking may be more useful if it is used to help VR counselors share their experiences with each other on what is and isn’t working, or to evaluate counselors’ performance.”

Several suggested that alternate means of applying for VR services should be studied, such as ability to schedule appointments and complete applications online. Others explained the importance of active follow-up of youth once they exit VR services.

Youth today change jobs often. People need to be able to come back to VR or obtain further job counseling services from some other source (e.g. Medicaid) even after they’ve been ‘placed’ in their first job. An 18-year-old who spends the next 5–10 years stuck in their first job is not necessarily a good outcome.

Three participants mentioned the importance of having a solid foundation of statewide policies, as opposed to policies and practices that vary locally. One said that tailoring policies and practices at the individual level was more important for effectiveness than tailoring to regional characteristics and needs. In regard to whether VR state plans should specifically include policies and programs tailored to VR service users with ASD, commenters clarified, “Autism should be mentioned, but should be mentioned along with other populations that may have shared needs (such as learning disabilities, ADHD, and other developmental disabilities),” and “Calling out one group over another is not right.”

3.1.3. Innovative state-level VR practices and policies

Overall, participants favored development of services that might better meet the needs of VR service users with autism, particularly those who do not have intellectual disability, and mentioned the benefits of demonstration projects for testing new services. Five commenters mentioned specific assessment tools and methods that some states are using to evaluate the needs of VR consumers with autism, and one commented on the importance of assessing effects of sensory input when making job placement decisions. However, others indicated hesitation about gearing VR practices and policies to people with autism, stating, “People may focus on stereotypical challenges (e.g., interacting with others) and not be sufficiently attuned to individual variation,” and that “… use of unique ‘autism codes’ can make it difficult for undiagnosed autistics, especially people of color and women, to access the same supports if they need them.”

While the importance of studying strategies like use of specialty caseloads was strongly endorsed overall, participant comments again reflected a range of opinions. Some saw specialty caseloads as an effective use of staff resources when serving consumers with more intense needs. However, two commented that use of specialists may result in “worse services” if specialists do not have the skills required to meet a wide range of needs. Others feared that use of specialists could result in decreased competence of non-specialist VR providers to serve TAY and adults with autism.

3.1.4. Interagency efforts to improve employment outcomes

Commenters said agencies “… still operate in silos.” Commenters discussed the need for both consistent reimbursement rates and frequent communication across agencies to facilitate partnership; although, one cautioned, “I think it’s more important that LOCAL agencies interact and educate [each other]. State agency interaction won’t always trickle down to the local level.” Four participants commented that VR and Medicaid-funded employment supports (administered through Intellectual/ Developmental Disability (I/DD) agencies), should be used together to complement each other with a “no wrong door policy,” and that use of one agency’s services should not be required as a prerequisite to access the other. While sharing of data across state agencies was not rated as highly important for research, one participant mentioned this is “not happening in our state, but it would be tremendous.” Another commenter questioned violation of confidentiality in data sharing and said, “More important is partnerships.”

In regard to interagency collaboration to support school-to-work transition, commenters emphasized the role of state education departments in providing vocational programming instead of deferring the development of employment skills to VR. Most agreed that low expectations in school settings could limit student, teacher, and family expectations about student capabilities for employment.

For example, “If people are told at school that they can’t expect more than a sheltered workshop or low skill job, they may assume that VR would view them similarly and not realize that VR can help them achieve their actual goals.”

3.1.5. VR staff training and competency

Participants viewed training and development of competency as highly important, saying, “This [training] can be critical, especially for VR counselors who were previously assuming that people on the autism spectrum are best served in autism-specific settings or sheltered workshops.” Enthusiasm was paired with cautions about quality of training, the mechanisms for training delivery, and who provides the training. Some viewed direct, hands-on training as more effective than communities of practice or remote webinars. Others suggested that trainings and supports also need to incorporate employers and community businesses, self-advocates, and former VR service users. One returned to the theme of the need to broaden job opportunities, “It’s important that training and programs maintain the focus on a range of jobs and supports” versus giving “the idea that certain jobs are universally ‘good’ or ‘bad’ for autistic individuals.”

Five participants commented that competency of contracted community rehabilitation providers (CRPs) is an important factor of outcomes, with one clarifying, “Many CRPs have NO competency to serve folks with ASD.” Another pointed out that while some CRPs have limited competency to serve consumers with autism, others have strong credentials and serve as key specialists, such as BCBAs who are contracted to provide behavioral and social intervention to support a person with autism in the workplace.

3.2. Delphi results for Round 2

Ten of the 12 study participants (83%) fully completed the Round 2 survey. All indicated “agree” or “strongly agree” that the five identified themes adequately captured the main types of factors that might be driving state-level variability in VR outcomes. Most (90%) indicated “agree” or “strongly agree” with which factors were organized into each theme. Most (90%) also indicated “agree” or “strongly agree” that the identified items and thematic groupings are useful for identifying strategies for improving policy and practice. A complete list of themes, their working definitions, and corresponding factors is provided in Table 2.

Table 2
Thematic grouping of factors that might influence Vocational Rehabilitation services and outcomes

1. CAPACITY TO PROVIDE VR SERVICES AND JOB OPPORTUNITIES

Staffing levels and range of job opportunities enable VR to meet the needs of a wide range of VR users statewide.

•Capacity to provide a wide range of jobs to fit a variety of skills and interests

•Adequate number of certificate/degree programs in the state for training rehabilitation counselors

•Capacity of transition support providers to meet statewide needs

•Consultants are available to help people keep needed public benefits if they become employed *

2. EFFICIENT AND EFFECTIVE VR PROCESSES

VR services are easily accessible and timely, meet the needs of consumers with a wide range of job impediments, and are evaluated for their effectiveness.

•VR policy changes and programs are evaluated for association with outcomes

•Ease of application process for VR services

•VR applicants who have autism without intellectual disability are able to obtain VR services *

•Length of the VR application process *

•VR counselors consistently attend transition planning meetings for special education students *

•State uses automated systems for data tracking to plan for employment services needs *

•Proactive follow-up with youth following exit from VR services a

3. INNOVATIVE STATE-LEVEL VR PRACTICES AND POLICIES

The state VR agency develops and uses innovative practices and policies to meet unique assessment and job support needs of VR services users with autism.

•State VR agency uses demonstration projects and innovative programs to improve outcomes

•State VR agency creates unique supports to meet the needs of service users with autism

•State VR agency uses specialty caseloads (eg, transition specialists, autism specialists)

•State VR agency uses assessment tools/methods tailored to challenges of people with autism *

4. INTERAGENCY EFFORTS TO ACHIEVE EMPLOYMENT OUTCOMES

Agencies and organizations in the state (e.g., VR, Special Education, Medicaid/Developmental Disability Services, Labor) collaborate to achieve positive employment outcomes for people with autism.

•State VR agency collaborates with business organizations to best meet labor market needs

•State special education system actively uses 18–21 year old programs to provide vocational

training

•State special education system fosters high or low expectations for students with autism

•Funding for state VR services is aligned with Employment First philosophy

•State uses Medicaid waivers to support employment of people with autism in the state *

5. VR STAFF TRAINING AND COMPETENCY

VR counselors and contracted community rehabilitation providers are trained in basic competencies needed to meet the needs of VR consumers with autism spectrum disorder.

•Autism training and technical assistance is available for all VR counselors

•Competency of community rehabilitation providers to serve people with autism

•Technical assistance is available in the state to support integrated employment *

^*Items ranked in top half of importance but lower half of modifiability.

^a Item nominated during Round 1 and scored in Round 2.

Participants rated the importance of each of the identified themes highly. Mean importance ratings ranged from 5.7 to 5.9, falling between “agree” (5) and “strongly agree” (6). The additional factor nominated during Round 1 (proactive follow-up with youth following exit from VR services) received a mean importance score of 5.6 and a mean modifiability score of 5.5. This item was subsequently added to Table 2, as ratings were above the median mean importance score. No novel, modifiable factors were identified during Round 2. We used participant comments and explanations about importance and modifiability of factors (summarized in section 3.3) to refine and finalize identified themes and categorization of factors.

4. Discussion

This study used the knowledge and practical experience of subject matter experts to identify and prioritize a field of modifiable factors that may contribute to differences in individual-level VR service use and employment outcomes across states for TAY-ASD. Participants ranked 23 factors as highly important, and we identified five major themes within survey comments: Capacity to provide VR services and job opportunities, Efficient and effective VR processes, Innovative state-level VR practices and policies, Inter-agency efforts to improve employment outcomes, and VR staff training and competency. These themes exemplified the breadth and complexity of issues which may contribute to VR service utilization and employment outcomes of TAY-ASD in a state.

The theme of building capacity to meet a wide range of employment needs and job interests across the autism spectrum was consistent with concerns that emerged in a previous convening of expert stakeholders and self-advocates (Nicholas, Zwaigenbaum, et al., 2017). Frequent references to a narrowed scope of job opportunities, and a perceived “funneling” of people with ASD into stereotypical job types, also confirmed previous research that indicated a decreased variety of job types for young adults with autism compared to those with other disability types (Roux et al., 2013). Recent focus groups with adult autistic VR service users have pointed to an over-placement of people with autism in technical jobs (Knowledge Translation for Employment Research Center, 2017). Researchers in the UK have noted that employment specialists may have a constricted understanding of jobs that people on the spectrum might be suited for (Richards, 2012), similar to concerns expressed by participants in this study.

Factors related to efficient and effective VR processes were rated as highly important across all stages of the VR experience – from ease of application, to evaluation of programs. Improving ease of access to and use of VR services may be particularly relevant for individuals with ASD who do not have co-occurring ID, since these individuals have fewer formal supports as they attempt to navigate the VR service system (Taylor et al., 2012). The high importance rating for proactive follow-up of TAY-ASD after job placement was also consistent with previous calls for long-term employment supports (Nicholas, Hodgetts, et al., 2017), and highlights a need for flexible supports. TAY-ASD tend to cycle in and out of employment (Baldwin et al., 2014; Müller et al., 2003; Wei, Wagner, Hudson, Jennifer, & Shattuck, 2014), and exhibit long-term vocational needs (Hendricks, 2010). Nicholas and colleagues found that once an individual with autism obtains a job, loss of employment supports can create vulnerabilities, particularly if the skills required to perform a job change as individuals advance in a career track (Nicholas, Zwaigenbaum, et al., 2017).

The need to address the unique vocational needs of TAY-ASD through innovative services and practices was a repeated theme; although there was a lack of consensus regarding the appropriateness of tailoring VR programming to a specific disability group. Published research has long recognized the need for autism-specific interventions that address interactional skills and incorporate principles of behavioral learning (Müller et al., 2003; Nicholas, Zwaigenbaum, et al., 2017; Wehman et al., 2014). Muller (2003) and colleagues outlined examples of autism-related vocational needs including: ongoing support to analyze and navigate social nuances and communication in a work environment (e.g., acceptability of asking questions during meetings, interpretation of indirect feedback); need for individualized instruction and break-down of tasks into smaller components; and flexible intensity of job coaching with increased support during skill acquisition and follow-along supports later. Future efforts to test the effects of innovative services and practices that address these identified autism-specific needs would be beneficial.

Participants agreed upon the importance of interagency collaboration, which has been identified as one of seven key transition elements for students with autism (Lee & Carter, 2012). Interagency collaboration is also a vital component of addressing the needs of TAY with disabilities in The 2020 Federal Youth Transition Plan: A Federal Interagency Strategy (accessed at www.dol.gov/odep). Other researchers have noted the need for increased knowledge sharing across entities during the transition between special education and community-based vocational services (Cimera & Cowan, 2009). Historically, involvement of VR during the special education transition process has been infrequent (Cameto, Levine, & Wagner, 2004). However, preliminary data from a model transition program in Maryland indicated positive employment outcomes for students with disabilities (including autism) who participated in a program that facilitated system linkages and collaboration as one of its primary intervention components, in addition to information sharing between the school and community service providers and early paid work experiences (R. G. Luecking, Fabian, Contreary, Honeycutt, & Luecking, 2017). TAY who received this seamless transition intervention were more likely to receive job search and on-the-job supports services than youth who received usual VR services, and were more likely to exit VR with employment. Interagency partners in this seamless transition model included school staff, VR counselors and CRPs, mental health service providers, and I/DD agency representation (D. M. Luecking & Luecking, 2015).

Training, education, and qualification of VR counselors and contracted CRPs was another major theme in this study, consistent with previous studies (Migliore et al., 2012). In recent years, state efforts have focused on increasing the competency of VR providers to serve individuals with ASD - sometimes by assigning autism specialists to consult to VR service providers (Del Valle et al., 2014). A national survey completed by nearly 800 VR counselors revealed a high level of interest in autism trainings (Murza, 2016). One example of an innovative training program is the online Autism Specialization Endorsement for Texas Vocational Rehabilitation Certificate (www.twc.state.tx.us). Previously, the federal government supported autism training for VR agency staff via the Technical Assistance and Continuing Education Center (TACE) programs, but funding for this program has ended. Ability to provide training and technical assistance is contingent upon having evidence-based vocational interventions that can be tailored to the needs of youth and adults with autism. Recent reviews have found a continued, significant lack of such interventions (Murza, 2016; Nicholas, Zwaigenbaum, et al., 2017).

The five main themes and many of the specific factors identified in this study align with Employment First principles and practices that aim to advance systems changes to build capacity, transform providers, enhance school-to-work transition, increase employer engagement, and align policy and funding (Nord et al., 2016). WIOA provides an additional road to implementation for the types of factors identified in this study as important for improving VR outcomes. WIOA regulations require state agencies to formally collaborate to address school-to-work transition, and instruct VR to coordinate with the state’s I/DD agency and Medicaid administrators, in addition to partnering with employment support programs in the community such as American Job Centers. New WIOA performance measurement also includes follow-up measures in the second and fourth quarters following exit from VR services.

Our research findings provide suggestions for correlates and themes to consider when researching the impact of WIOA-related interventions on TAY-ASD. These include: the capacity of pre-employment transition services and traditional VR services to support a broad range of TAY-ASD who seek to enter the job force, development of service models tailored to meet vocational needs of TAY-ASD, characterization of effective models for developing competency among VR counselors and CRPs who serve TAY-ASD, and identification of which aspects of interagency collaboration are most critical for enhancing seamless transitions from school to work.

Some of the factors identified in this study are already written into regulations; although, they might not be consistently implemented in practice. Some factors are heavily influenced by funding; and other factors – like training and technical assistance – are dependent on high quality of implementation. Yet, as one participant stated, “…it is actually a very small financial investment to put in place these five identified themes when they look at the significant return,” raising the possibility that meeting the needs of TAY-ASD may be less costly for states if improving employment results in decreased dependency on public benefits and Medicaid waivers. Others expressed that improvement in outcomes is dependent upon “buy-in” from VR administrators and willingness to address the type of factors identified in this study.

It is possible that federal policies emphasizing community-based, integrated jobs have created further strain on capacity. Employment support needs of workers with autism may be more prominent within this context, as social stressors are more likely to occur within integrated and non-autism specific work settings (Lorenc et al., 2017; Lorenz, Frischling, Cuadros, & Heinitz, 2016). The relevance of these challenges may also be fueled by increases in the proportion of cognitively-able TAY-ASD (Christensen et al., 2016), who may be more likely to have a stated goal of competitive, integrated employment.

Several modifiable factors identified in this study have received little attention in vocational literature regarding TAY-ASD, including: geographic reach and coverage of vocational transition providers and of benefits counselors, accessibility and ease of the VR application process and its integration with determination of eligibility for other adult supports, and innovative services and strategies states are developing to better meet the employment needs of TAY-ASD. The need for further study of these factors is intertwined with an apparent need to better delineate justification for providing autism-specific services and strategies – for whom and when are they appropriate and most effective. Are innovations such as autism-specific VR services and autism-focused staff training associated with improved employment outcomes for TAY-ASD and reduced costs of serving this population? Are autism-specific interventions more effective and efficient when delivered to students via tailored Pre-ETS programs, or upon entry into traditional VR services? Research questions like these point to a need to probe beyond data routinely collected by VR (e.g., RSA-911 data) to better understand the landscape of what states are doing to improve employment outcomes for TAY-ASD as their use of VR services increases.

4.1. Study limitations

This study had several limitations. Use of an online, modified Delphi process enhanced the efficiency of the study and equalized opportunities for participation. However, online methods reduced discussion amongst participants that could have differently shaped collective ratings (Khodyakov et al., 2016).

We harnessed the knowledge of subject matter experts, including several self-advocates, for this study but did not include the voices of VR service users, their families, or their employers. This approach was necessary given the specificity of knowledge required to reflect on explicit VR program and policy issues and other vocational programs in states, as the underlying purpose of this study was to hone the field of state-level factors for consideration in future research. However, the direct experiences of those who use the VR system for employment supports may have differed from the opinions expressed in this study and should be a central focus of next-step research questions. Further, the experts who contributed to this study did not represent all states; although many had an understanding of VR programs across states. State VR programs are highly variable in their administration and in who they serve, as well as the state-level contexts in which they exist. Obtaining representative input from all states might have broadened the scope of our findings or reinforced central themes, and should be a key component of future state-level data collection.

Finally, this study identified programmatic, procedural, and policy relevant ideas that could positively impact the employment outcomes of TAY-ASD and other youth who use VR services; however, additional research is needed to investigate their implementation and results.

5. Conclusion

Poor employment outcomes for adults with ASD have remained relatively persistent over time – both nationally and globally (Burgess & Cimera, 2014; Howlin & Magiati, 2017; Howlin & Moss, 2012; Steinhausen, Mohr Jensen, & Lauritsen, 2016), and adults with ASD and their families have rated employment services and supports among the most important research priorities (Gotham et al., 2015; Pellicano, Dinsmore, & Charman, 2014). The current state of knowledge provides insufficient evidence for informing system change efforts needed to improve employment outcomes for TAY-ASD. Research that investigates the factors and themes identified in this study may yield valuable insights about the correlates of VR service utilization and successful employment outcomes for TAY-ASD. Results of this study are relevant for both researchers and program administrators as they consider the types of data that should be collected when studying state-level variation and possible targets for state-level interventions. As states are increasingly serving as incubators for development of more effective VR services, mechanisms for measuring implementation, outcomes, and correlates of utilization of these services are timely and crucial.

Conflict of interest

The authors declare no conflicts of interest.

Footnotes

Acknowledgments

We would like to acknowledge the expert stakeholders who voluntarily contributed valuable insights to this research.

This project was supported by the Health Resources and Services Administration (HRSA) of the U.S. Department of Health and Human Services (HHS) under UJ2MC31073: Maternal and Child Health-Autism Transitions Research Project. This information or content and conclusions are those of the author and should not be construed as the official position or policy of, nor should any endorsements be inferred by HRSA, HHS or the U.S. Government.

This project was also supported by funding from the Organization for Autism Research (OAR), Inc. Applied Research Grant, under the title “Association of state-level factors with vocational outcomes for transition-age youth with autism.”

References

Adler, M., & Ziglio, E. (1996). Gazing into the oracle: The Delphi method and its application to social policy and public health. Philadelphia, PA:Jessica Kingsley Publishers.

Alsaman, M. A., & Lee, C.-L. (2016). Employment outcomes of youth with disabilities in Vocational Rehabilitation: A multilevel analysis of RSA-911 data. Rehabilitation Counseling Bulletin, 0034355216632363.

Anderson,

K. A.

, Roux,

A.M.

, Kuo,

, & Shattuck,

P. T.

(2018). Social-ecological correlates in adult autism outcome studies: A scoping review. Pediatrics, 141(Supplement 4), S306–S317.

Baldwin, S., Costley, D., & Warren, A. (2014). Employment activities and experiences of adults with high-functioning autism and Asperger's disorder. Journal of Autism and Developmental Disorders, 44(10), 2440–2449.

Burgess, S., & Cimera, R. E. (2014). Employment outcomes of transition-aged adults with autism spectrum disorders: A state of the states report. American Journal on Intellectual and Developmental Disabilities, 119(1), 64–83.

Butterworth, J., Migliore, A., & Timmons, J. (2010). Services and outcomes for transition age youth with autism spectrum disorders: Secondary analysis of the NLTS2 and RSA911. Boston, MA:Institute for Community Inclusion.

Cameto, R., Levine, P., & Wagner, M. (2004). Transition planning for students with disabilities: A special topic report of findings from the National Longitudinal Transition Study-2 (NLTS2). Washington, DC:National Center for Special Education Research.

Carter, E. W., Ditchman, N., Sun, Y., Trainor, A. A., Swedeen, B., & Owens, L. (2010). Summer employment and community experiences of transition-age youth with severe disabilities. Exceptional Children, 76, 194+.

Certo, N. J., Mautz, D., Pumpian, I., Sax, C., Smalley, K., Wade, H. A.,… & Batterman, N. (2003). Review and discussion of a model for seamless transition to adulthood. Education and Training in Developmental Disabilities, 3–17.

10.

Chan, F., Wang, C.-C., Fitzgerald, S., Muller, V., Ditchman, N., & Menz, F. (2016). Personal, environmental, and service-delivery determinants of employment quality for state vocational rehabilitation consumers: A multilevel analysis. Journal of Vocational Rehabilitation, 45(1), 5–18.

11.

Chan, J. Y., Wang, C.-C., Ditchman, N., Kim, J. H., Pete, J., Chan, F., & Dreis, B. (2013). State unemployment rates and vocational rehabilitation outcomes: A multilevel analysis. Rehabilitation Counseling Bulletin, 0034355213509855.

12.

Chan, W., Smith, L. E., Hong, J., Greenberg, J. S., Lounds Taylor, J., & Mailick, M. R. (2017). Factors associated with sustained community employment among adults with autism and co-occurring intellectual disability. Autism, 1362361317703760.

13.

Chiang, H.-M., Cheung, Y. K., Li, H., & Tsai, L. Y. (2013). Factors associated with participation in employment for high school leavers with autism. Journal of Autism and Developmental Disorders, 43(8), 1832–1842.

14.

Christensen, D. L., Bilder, D. A., Zahorodny, W., Pettygrove, OS., Durkin, M. S., Fitzgerald, R. T.,… & Yeargin-Allsopp, M. (2016). Prevalence and characteristics of autism spectrum disorder among 4-year-old children in the autism and developmental disabilities monitoring network. Journal of Developmental & Behavioral Pediatrics, 37(1), 1–8.

15.

Cimera, R. E., Burgess, S., & Wiley, A. (2013). Does providing transition services early enable students with ASD to achieve better vocational outcomes as adults?Research and Practice for Persons with Severe Disabilities, 38(2), 88–93.

16.

Cimera, R. E., & Cowan, R. J. (2009). The costs of services and employment outcomes achieved by adults with autism in the US. Autism, 13(3), 285–302.

17.

Del Valle, R., Leahy, M. J., Sherman, S., Anderson, C. A., Tansey, T., & Schoen, B. (2014). Promising best practices that lead to employment in vocational rehabilitation: Findings from a four-state multiple case study. Journal of Vocational Rehabilitation, 41(2), 99–113.

18.

Fabian, E., & Luecking, D. M. (2012). Does service collaboration enhance seamless transition outcomes? (Issue Brief #1105). Center on Transition to Employment for Youth with Disabilities. Retrieved fromhttp://www.transitiontoemployment.org

19.

Giesen, J. M., & Cavenaugh, B. S. (2013). Disability insurance beneficiaries with visual impairments in vocational rehabilitation: Socio-demographic influences on employment. Journal of Visual Impairment & Blindness (Online), 107(6), 453.

20.

Gotham, K., Marvin, A. R., Taylor, J. L., Warren, Z., Anderson, C. M., Law, P. A.,… & Lipkin, P. H. (2015). Characterizing the daily life, needs, and priorities of adults with autism spectrum disorder from Interactive Autism Network data. Autism, 19(7), 794–804.

21.

Hall, A. C., Freeze, S., Butterworth, J., & Hoff, D. (2011). Employment funding for intellectual/developmental disability systems. Journal of Vocational Rehabilitation, 34(1), 1–15.

22.

Hasson, F., Keeney, S., & McKenna, H. (2000). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing, 32(4), 1008–1015.

23.

Hendricks, D. (2010). Employment and adults with autism spectrum disorders: Challenges and strategies for success. Journal of Vocational Rehabilitation, 32(2), 125–134.

24.

Holwerda, A., van der Klink, J. J., Groothoff, J. W., & Brouwer, S. (2012). Predictors for work participation in individuals with an autism spectrum disorder: A systematic review. Journal of Occupational Rehabilitation, 22(3), 333–352.

25.

Honeycutt, T., Bardos, M., & McLeod, S. (2015). Bridging the gap: A comparative assessment of vocational rehabilitation agency practices with transition-age youth. Journal of Vocational Rehabilitation, 43(3), 229–247.

26.

Honeycutt, T., Thompkins, A., Bardos, M., & Stern, S. (2015). State differences in the vocational rehabilitation experiences of transition-age youth with disabilities. Journal of Vocational Rehabilitation, 42(1), 17–30.

27.

Howlin, P., & Magiati, I. (2017). Autism spectrum disorder: Outcomes in adulthood. Current Opinion in Psychiatry, 30(2), 69–76.

28.

Howlin, P., & Moss, P. (2012). Adults with autism spectrum disorders. The Canadian Journal of Psychiatry, 57(5), 275–283.

29.

Hsu, C.-C., & Sandford, B. A. (2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research & Evaluation, 12(10), 1–8.

30.

Hurlbutt, K., & Chalmers, L. (2004). Employment and adults with Asperger syndrome. Focus on Autism and Other Developmental Disabilities, 19(4), 215–222.

31.

Kaya, C., Chan, F., Rumrill, P., Hartman, E., Wehman, P., Iwanaga, K.,… & Avellone, L. (2016). Vocational rehabilitation services and competitive employment for transition-age youth with autism spectrum disorders. Journal of Vocational Rehabilitation, 45(1), 73–83.

32.

Keeney, S., Hasson, F., & McKenna, H. P. (2001). A critical review of the Delphi technique as a research methodology for nursing. International Journal of Nursing Studies, 38(2), 195–200.

33.

Kelly, S. E., Moher, D., & Clifford, T. J. (2016). Defining rapid reviews: A modified Delphi consensus approach. International Journal of Technology Assessment in Health Care, 32(4), 265–275.

34.

Kendall, J. E., Kendall, K. E., Smithson, S., & Angell, I. O. (1992). SEER: A divergent methodology applied to forecasting the future roles of the systems analyst. Human Systems Management, 11(3), 123–135.

35.

Khodyakov, D., Grant, S., Barber, C. E., Marshall, D. A., Esdaile, J. M., & Lacaille, D. (2016). Acceptability of an online modified Delphi panel approach for developing health services performance measures: Results from 3 panels on arthritis research. Journal ofEvaluation in Clinical Practice.

36.

Knowledge Translation for Employment Research Center. (2017). Focus group findings from vocational rehabilitation (VR) staff and adults with autism spectrum disorder (ASD): Improving VR counseling services for adults with ASD. Retrieved fromwww.kter.org

37.

Lawer, L., Brusilovskiy, E., Salzer, M. S., & Mandell, D. S. (2009). Use of vocational rehabilitative services among adults with autism. Journal ofAutism and Developmental Disorders, 39(3), 487–494.

38.

Lee, G. K., & Carter, E. W. (2012). Preparing transition-age students with high-functioning autism spectrum disorders for meaningful work. Psychology in the Schools, 49(10), 988–1000.

39.

Lindsay, P., Schull, M., Bronskill, S., & Anderson, G. (2002). The development of indicators to measure the quality of clinical care in emergency departments following a modified-Delphi approach. Academic Emergency Medicine, 9(11), 1131–1139.

40.

Lorenc, T., Rodgers, M., Marshall, D., Melton, H., Rees, R., Wright, K., & Sowden, A. (2017). Support for adults with autism spectrum disorder without intellectual impairment: Systematic review. Autism, 1362361317698939.

41.

Lorenz, T., Frischling, C., Cuadros, R., & Heinitz, K. (2016). Autism and overcoming job barriers: Comparing job-related barriers and possible solutions in and outside of autism-specific employment. PLoS One, 11(1), e0147040.

42.

Luecking, D. M., & Luecking, R. G. (2015). Translating research into a seamless transition model. Career Development and Transition for Exceptional Individuals, 38(1), 4–13.

43.

Luecking, R. G., Fabian, E. S., Contreary, K., Honeycutt, T. C., & Luecking, D. M. (2017). Vocational rehabilitation outcomes for students participating in a model transition program. Rehabilitation Counseling Bulletin, 0034355217713167.

44.

McNall, M., & Foster-Fishman, P. G. (2007). Methods of rapid evaluation, assessment, and appraisal. American Journal of Evaluation, 28(2), 151–168.

45.

Mead, D., &Moseley, L. (2001). The use of the Delphi as aresearch approach. Nurse Researcher, 8(4), 4–23.

46.

Migliore, A., Butterworth, J., & Zalewska, A. (2014). Trends in vocational rehabilitation services and outcomes of youth with autism: 2006-2010. Rehabilitation Counseling Bulletin, 57(2), 80–89.

47.

Migliore, A., Timmons, J., Butterworth, J., &Lugas, J. (2012). Predictors of employment and postsecondary education of youth with autism. Rehabilitation Counseling Bulletin, 55(3), 176–184.

48.

Miles, M. B., & Huberman, A. M. (1984). Drawing valid meaning from qualitative data: Toward a shared craft. Educational Researcher, 13(5), 20–30.

49.

Muller, E., Schuler, A., Burton, B. A., & Yates, G. B. (2003). Meeting the vocational support needs of individuals with Asperger syndrome and other autism spectrum disabilities. Journal of Vocational Rehabilitation, 18(3), 163–175.

50.

Murphy, M., Black, N., Lamping, D., McKee, C., Sanderson, C., Askham, J., & Marteau, T. (1998). Consensus development methods, and their use in clinical guideline development. Health Technology Assessment, 2(3), i.

51.

Murza, K. A. (2016). Vocational rehabilitation counselors' experiences with clients diagnosed with autism spectrum disorder: Results of a national survey. Journal of Vocational Rehabilitation, 45(3), 301–313.

52.

Nambisan, S., Agarwal, R., & Tanniru, M. (1999). Organizational mechanisms for enhancing user innovation in information technology. MIS quarterly, 365–395.

53.

Neal, J. W., Neal, Z. P., VanDyke, E., & Kornbluh, M. (2015). Expediting the analysis of qualitative data in evaluation: A procedure for the rapid identification of themes from audio recordings (RITA). American Journal of Evaluation, 36(1), 118–132.

54.

Nicholas, D. B., Hodgetts, S., Zwaigenbaum, L., Smith, L. E., Shat-tuck, P., Parr, J. R.,… & Sacrey, L. (2017). Research needs and priorities for transition and employment in autism: Considerations reflected in a "Special Interest Group" at the International Meeting for Autism Research. Autism Research, 10(1), 15–24.

55.

Nicholas, D. B., Zwaigenbaum, L., Zwicker, J., Clarke, M. E., Lamsal, R., Stoddart, K. P.,… &Lowe, K. (2017). Evaluation of employment-support services for adults with autism spectrum disorder. Autism, 1362361317702507.

56.

Nord, D., Butterworth, J., Carlson, D., Grossi, T., Hall, A., & Nye-Lengerman, K. (2016). Where we are and where we're headed: Employment for people with IDD. In Critical issues in intellectual and developmental disabilities: Contemporary research, practice, and policy. Washington, DC:American Association on Intellectual and Developmental Disabilities.

57.

Nord, D. K., Stancliffe, R. J., Nye-Lengerman, K., & Hewitt, A. S. (2016). Employment in the community for people with and without autism: A comparative analysis. Research in Autism Spectrum Disorders, 24, 11–16.

58.

Okoli, C., & Pawlowski, S. D. (2004). The Delphi method as a research tool: An example, design considerations and applications. Information & Management, 42(1), 15–29.

59.

Pare, G., Cameron, A.-F., Poba-Nzaou, P., & Templier, M. (2013). A systematic assessment of rigor in information systems ranking-type Delphi studies. Information & Management, 50(5), 207–217.

60.

Pellicano, E., Dinsmore, A., & Charman, T. (2014). What should autism research focus upon? Community views and priorities from the United Kingdom. Autism, 18(7), 756–770.

61.

Pillay, Y., & Brownlow, C. (2016). Predictors of successful employment outcomes for adolescents with autism spectrum disorders: A systematic literature review. Review Journal of Autism and Developmental Disorders, 1–11.

62.

Powell, C. (2003). The Delphi technique: Myths and realities. Journal of Advanced Nursing, 41(4), 376–382.

63.

Richards, J. (2012). Examining the exclusion of employees with Asperger syndrome from the workplace. Personnel Review, 41(5), 630–646.

64.

Roux, A. M., Rast, J. E., Anderson, K. A., & Shattuck, P. T. (2016). National autism indicators report: Vocational rehabilitation. Life Course Outcomes Research Program, A.J. Drexel Autism Institute. Retrieved fromhttp://www.drexel.edu/AutismOutcomes.

65.

Roux, A. M., Shattuck, P. T., Cooper, B. P., Anderson, K. A., Wagner, M., & Narendorf, S. C. (2013). Postsecondary employment experiences among young adults with an autism spectrum disorder. Journal of the American Academy of Child and Adolescent Psychiatry, 52(9), 931–939.

66.

Roux, A. M., Shattuck, P. T., Rast, J. E., Rava, J. A., & Anderson, K. A. (2015). National autism indicators report: Transition into young adulthood. Life Course Outcomes Research Program, A.J. Drexel Autism Institute. Retrieved fromhttp://www.drexel.edu/AutismOutcomes.

67.

Rowe, G., & Wright, G. (2001). Expert opinions in forecasting: The role of the Delphi technique. In Armstrong, J.S. (Eds.), Principles of forecasting. International Series in Operations Research & Management Science, 30,125–144. Boston, MA:Springer.

68.

Rusch, F. R., & Braddock, D. (2004). Adult day programs versus supported employment (1988-2002): Spending and service practices of mental retardation and developmental disabilities state agencies. Research and Practice for Persons with Severe Disabilities, 29(4), 237–242.

69.

Sawford, K., Dhand, N. K., Toribio, J.-A. L., & Taylor, M. R. (2014). The use of a modified Delphi approach to engage stakeholders in zoonotic disease research priority setting. BMC Public Health, 14(1), 182.

70.

Schmidt, R., Lyytinen, K., & Mark Keil, P. C. (2001). Identifying software project risks: An international Delphi study. Journal of Management Information Systems, 17(4), 5–36.

71.

Shattuck, P. T., Narendorf, S. C., Cooper, B., Sterzing, P. R., Wagner, M., & Taylor, J. L. (2012). Postsecondary education and employment among youth with an autism spectrum disorder. Pediatrics, 129(6), 1042–1049. 10.1542/peds.2011-2864

72.

Smith, F. A., & Lugas, J. (2010). Vocational Rehabilitation (VR) employment outcomes for transition-age youth with autism, and other disabilities. Retrieved from Boston, MA:http://www.communityinclusion.org

73.

Steinhausen, H. C., Mohr Jensen, C., & Lauritsen, M. (2016). A systematic review and meta-analysis of the long-term overall outcome of autism spectrum disorders in adolescence and adulthood. Acta Psychiatrica Scandinavica, 133(6), 445–452.

74.

Taylor, J. L., & Mailick, M. R. (2014). A longitudinal examination of 10-year change in vocational and educational activities for adults with autism spectrum disorders. Developmental Psychology, 50(3), 699.

75.

Taylor, J. L., McPheeters, M. L., Sathe, N. A., Dove, D., Veenstra-VanderWeele, J., & Warren, Z. (2012). A systematic review of vocational interventions for young adults with autism spectrum disorders. Pediatrics, 130(3), 531–538.

76.

Taylor, J. L., & Seltzer, M. M. (2011). Employment and post-secondary educational activities for young adults with autism spectrum disorders during the transition to adulthood. Journal of Autism and Developmental Disorders, 41(5), 566–574.

77.

Wehman, P., Chan, F., Ditchman, N., & Kang, H.-J. (2014). Effect of supported employment on vocational rehabilitation outcomes of transition-age youth with intellectual and developmental disabilities: A case control study. Intellectual and Developmental Disabilities, 52(4), 296–310.

78.

Wehman, P., Schall, C., Carr, S., Targett, P., West, M., & Cifu, G. (2014). Transition from school to adulthood for youth with autism spectrum disorder: What we know and what we need to know. Journal of Disability Policy Studies, 25(1), 30–40.

79.

Wehman, P., Sima, A. P., Ketchum, J., West, M. D., Chan, F., & Luecking, R. (2015). Predictors of successful transition from school to employment for youth with disabilities. Journal of Occupational Rehabilitation, 25(2), 323–334.

80.

Wehman, P. H., Schall, C. M., McDonough, J., Kregel, J., Brooke, V., Molinelli, A.,… & Collins, H. T. (2014). Competitive employment for youth with autism spectrum disorders: Early results from a randomized clinical trial. Journal of Autism and Developmental Disorders, 44(3), 487–500.

81.

Wei, X., Wagner, M., Hudson, L., Jennifer, W. Y., & Shattuck, P. (2014). Transition to adulthood employment, education, and disengagement in individuals with autism spectrum disorders. Emerging Adulthood, 2167696814534417.