Customized employment discovery fidelity: Developing consensus among experts

Abstract

BACKGROUND:

Despite promising economic and job growth over the last decade, achieving quality employment outcomes for people with significant disabilities remains a challenge in the United States. Recent legislation responded to this challenge by, among other things, expanding the definition of supported employment to include customized employment. Customized employment uses an individualized process to first examine a person’s skills, interests, and support needs and then to create integrated employment by matching these with business needs. As states integrate customized employment into their service delivery systems, they will need guidelines about what is and what is not considered best practice. Because discovery is the foundation of the customized employment process, measures to determine adherence to critical elements of the discovery process must be developed.

OBJECTIVE:

The purpose of this study was to generate consensus about what experts believe are acceptable and unacceptable tenets of customized employment discovery systems and services.

METHODS:

A 3-step, modified online Delphi method was used to obtain consensus from a panel of experts in customized employment discovery. The Delphi panel rated items on a discovery systems and discovery services fidelity scale over three iterations.

RESULTS:

A total of 60 discovery systems and services tenets were reviewed by the panel. Of the 20 systems discovery tenets, 14 of the acceptable and three of the unacceptable systems tenets met criteria for consensus. Of the 40 discovery services tenets, 20 of the acceptable and 10 of the unacceptable services tenets met criteria for consensus.

CONCLUSIONS:

While there appears to be subtle differences in what experts believe are acceptable and unacceptable tenets of discovery, a significant majority of the acceptable and not acceptable discovery systems and services tenets achieved consensus after three iterative rounds. As more is learned about the critical components of the discovery process, a fidelity scale can ensure consistent implementation of discovery.

Keywords

Customized employment discovery disability fidelity delphi study

1. Introduction

Despite promising economic and job growth over the last decade, achieving quality employment outcomes for people with disabilities remains a challenge in the United States. In fact, the American Community Survey (ACS) reported that only 33.7% of individuals with disabilities between ages 16–64 are employed (Erickson, Lee, & von Schrader, 2015). Employment outcomes for individuals with more significant disabilities are even poorer. Winsor, et al., (2017) reported that 18.6% of adults with intellectual disabilities are employed in integrated employment while the National Core Indicators project indicate only 14.8% of working aged adults who are supported by state intellectual and development disabilities (IDD) agencies are employed in paid community jobs (Hiersteiner, Butterworth, Bershadsky, & Bonardi, 2016).

Because poor integrated employment outcomes of individuals with more significant disabilities remain a pervasive problem in the United States, researchers and policymakers continue to look for strategies to improve these outcomes. Customized employment (CE) is one such strategy that has gained prominence over the last decade. CE first emerged in 2001 when the U.S. Department of Labor, Office of Disability Employment (ODEP) examined methods to improve employment outcomes for people with disabilities using the One Stop Career centers (Wehman, Inge, Revel, & Brook, 2007). From its inception, the definition of CE highlighted the importance of individualizing the relationship between employees and employers in ways that met the needs of both. The definition also stated that CE is “based on individual determination of the strengths, needs, and interests of the person with a disability, and is designed to meet the specific needs of the employer” (Federal Register, 2002, p. 43154-43149).

More recently, the Rehabilitation Act, as amended in the Workforce Innovation and Opportunity Act (WIOA, 2014), modified the definition of supported employment to include customized employment. The statute defines customized employment as “competitive integrated employment, for an individual with a significant disability that is based on an individualized determination of the strengths, needs, and interests of the individual with a significant disability, and is designed to meet the specific abilities of the individual with a significant disability and the business needs of the employer” (p. 1634). The statute also outlines specific strategies for implementing CE including (a) exploring jobs with the individual; (b) working with employers to facilitate placement including customizing a job description based on current employer needs or on previously unidentified and unmet employer needs; (c) developing a set of job duties, a work schedule, and job arrangement, along with specifics of supervision (including performance evaluation review), and determining a job location; (d) representing a professional chosen by the individual, or self-representation of the individual in working with an employer to facilitate placement; and (e) providing services and supports at the job placement (29 U.S.C §705 et seq.).

CE, therefore, is designed to support individuals with disabilities in finding integrated employment through a process that examines the strengths, interests, and needs of a person with a disability and matches those needs to a business. The person with a disability directs the CE process, and mutually beneficial job tasks are individually identified and negotiated with employers. The Workforce Innovation Technical Assistance Center (WINTAC), in collaboration with CE experts, developed an essential elements of customized employment document (WINTAC, 2017). The aim of the document was to provide information about universal application of components of CE including discovery. WINTAC listed a number of elements of discovery that should be implemented during a 35 hour or seven-week period. For example, WINTAC stated that discovery is a qualitative process used to determine an individual’s assets, interests, skills and support needs to obtain and maintain gainful employment. This process includes interviews, observation, records review, and interactions with the job seeker. Rather than using a traditional evaluation approach to learn about an individual, the discovery process is utilized. Discovery includes observations of an individual engaged in life activities and interviews with the job seeker, family members, and other stakeholders. WINTAC recommends that discovery concludes with a records review to fill in any gaps regarding the individual’s strengths, interests, and support needs.

While CE is considered a viable strategy that leads to competitive integrated employment for people with significant disabilities, there remain gaps in the current research establishing the empirical support for the CE process. These gaps were highlighted in a review of 25 peer-reviewed articles on customized employment published between 2006 and 2015 (Riesen, Morgan, & Griffin, 2015). The authors noted that despite the value of the current literature on CE, there remain questions about the operationalized evidence for effective and consistent implementation of CE at both the systems and services levels. The authors recommended that a fidelity scale be developed to determine the degree to which processes, such as discovery, adhere to best practice standards that are operationalized, reliable and valid, and produce positive employment outcomes. Fidelity scales have been developed for other employment support models such as the individual placement and support model (IPS) for people with severe mental illness (Bond, Becker, Drake, & Vogler.1997). The use of an IPS fidelity scale promoted standardized service implementation and provided guidelines for quality improvement for supported employment programs (Bond, Becker, & Drake, 2011; Knaeps, DeSmet, Van Audenhove, 2012). In addition, researchers found that employment programs that scored highest in fidelity also had higher employment outcomes compared to their counterparts (Bond, et al., 2011) and fidelity scales permitted sustainability and guided the planning process for new programs (Kim, Bond, Becker, Swanson, & Langfitt-Reese, 2015).

The development and validation of a CE discovery systems and services fidelity scale is necessary to capture and delineate the essential practices and procedures of discovery. A validated discovery fidelity scale could be used as an objective measure of performance to assist funding agencies determine which community rehabilitation agencies provide discovery that aligns with best practice. The purpose of this study was to generate consensus about what experts believe are acceptable and not acceptable tenets of a discovery fidelity scale. The following research questions guided this study:

RQ1: How do experts in CE rate the importance of tenets on a discovery systems and discovery services fidelity scale?

RQ2: What discovery systems and discovery services tenets are considered acceptable and unacceptable?

2. Method and procedures

2.1. Participants

We used purposive sampling (Fraenkel & Wallen, 2003) to select a Delphi expert panel with in-depth knowledge of the CE discovery process. The experts were recruited from various geographic regions of the United States and met the following inclusion criteria: (a) had a minimum of three years of experience implementing CE service delivery, (b) had working knowledge of systems-level implementation of discovery, and (c) had working knowledge of discovery services level implementation of discovery. We sent email invitations to participate in the Delphi study to 35 experts based on these inclusion criteria.

2.2. Design

We used a three-step, modified online Delphi method to gather consensus from a panel of experts in CE discovery (Custer, Scarcella, & Stewart, 1999; Keeney, Hasson, & McKenna, 2013). We elected to use a modified Delphi technique because we asked the expert panel to rate a large set of existing items on a fidelity scale. Custer, et al. suggested that a modified Delphi technique provides the opportunity for the panel to maintain concentration when asked to rate larger data sets and the modified process reduces panel attrition. Panel members participated in three rounds or iterations where each expert examined, rated, and commented on discovery systems and services tenets. The responses from each round were used to build the subsequent Delphi rounds. All data were collected and recorded via Qualtrics^® research Suite (2017).

2.3. Instrumentation

The Delphi survey instrument was comprised of discovery systems and discovery services fidelity tenets developed by Hall, Keeton, Cassidy, Iovannone, & Griffin, (2016). Specifically, Hall, et al. created the Discovery Fidelity Scale (DFS) using an iterative process that included a literature review, interviews, focus groups, case studies, and surveys with education and adult service professionals with demonstrated knowledge and experience in discovery implementation. During the first phase, Hall et al. interviewed 15 individuals with significant discovery experience, including representatives from two leading national authorities on discovery services (e.g. Marc Gold & Associates and Griffin-Hammis Associates). Subsequently, focus groups were held in five different Florida locations with 60 people who represented a larger group of Florida education and adult services professionals, government officials, educators, providers, and vocational rehabilitation personnel. All participants in either the interview or focus groups responded to a series of semi-structured questions about the discovery process and ways in which it could be improved. Hall et al. also conducted case studies with five individuals who previously completed discovery services. These case studies documented the experiences, insights, and perspectives of discovery service recipients and their families. The data collected from the interviews, focus groups, and case studies was analyzed and used to design and create a ten-question Likert-type scale survey to further assess the strength of varied discovery practices and opinions in a representative sample of 30 Florida focus group participants. Responses from the survey were used to refine the DFS tenets and to encourage a common set of discovery practices.

The data compiled from the iterative process became the foundation for the DFS. The completed scale consisted of two sections, discovery systems fidelity and discovery services fidelity, and incorporated key practices utilized by national training experts (e.g., Griffin-Hammis Associates and Marc Gold and Associates). The discovery systems fidelity items contained tenets related to authorization and access, finance, and quality requirements for discovery providers. The discovery services fidelity items included tenets related to home and neighborhood visits, discovery activities, informational interviews, vocational profiles, and employment plans. The DFS developed by Hall et al. used a rating scale, ranging from unacceptable, or “not discovery”, to exemplary discovery services and practices. The scale incorporated examples of unacceptable discovery that represent misconceptions or pitfalls identified by research participants and national discovery implementation experts. Unacceptable discovery tenets were included in this study so experts could make discriminations between a nuanced range of acceptable discovery practices and those practices that should not be a part of discovery. The rational to include unacceptable tenets was based on the hypothesis that determining what practices are and are not consistent with best practice discovery is a critical element of evaluating the veracity and quality of discovery practices.

2.4. Procedures

2.4.1. Round one

We sent an invitation email to 35 individuals identified as experts in discovery. The email provided information about the Delphi study and a uniform resource locater (URL) to the modified online Delphi. When the experts clicked the link to the modified Delphi, they were directed to an IRB letter of information page that described (a) the reason for the study and the Delphi process, (b) the process to withdraw from the study, (c) the contact information for the principle investigator, and (d) the procedures to ensure confidentiality. After reading the letter of information, the experts verified they were over 18-years-old by clicking on a verification link. Experts were then directed to a number of demographic questions such as position, years employed in the disability field, years providing CE services, geographic location, and education level. After completing the demographic section, experts were asked to examine and rate specific discovery systems tenets and discovery services tenets.

The discovery systems section included 20 items that represented acceptable and unacceptable systems tenets. Respondents were asked to review and rate each tenet using a five-point Likert-type scale with one indicating (never), two (rarely), three (sometimes), four (often), and five (always). The discovery services tenet section included 40 items that represent acceptable and unacceptable services tenets using a five-point Likert-type scale with one indicating (not discovery), two (poor discovery), three (acceptable discovery), four (good discovery), and five (exemplary discovery). Respondents were also provided the opportunity to write comments at the conclusion of the round one survey. The survey remained open for three weeks. One reminder email to complete the first round was sent at 1.5 weeks after the initial invitation email and on the day the survey was scheduled to close. Twenty eight of the 35 experts who were sent an invitation email started the survey and 26 (74.3%) of these experts completed the first round.

After the data collection period ended, we calculated measures of central tendency including the mean, median, and mode and measures of dispersion including standard deviation and interquartile range (IQR) for each discovery systems and discovery services tenets. We used these statistics to determine which systems and service items needed revisions for content and clarity to increase consensus. Specifically, tenets that we predicted would have high scores (acceptable tenets) were revised for clarity if the median score was three or lower or the standard deviation was 0.75 or higher or IQR was one or higher. Tenets that we predicted to have low scores (unacceptable tenets) were revised for clarity if the median score was two or lower and the standard deviation was 0.75 or higher or IQR was one or higher. Based on these criteria, 14 discovery systems tenets and 19 discovery services tenets were revised. For example, the systems tenet “Discovery is delivered by the same person who delivers other components of customized employment or supported employment, such as job development and job site training” was revised to “Discovery should be delivered by the same person who delivers the job development and job site training services.” The discovery services tenet “The vocational profile is written only in a positive, affirming, and narrative manner” was revised to “The vocational profile should be a narrative and descriptive document written in a positive and affirming manner”.

2.4.2. Round two

We sent the 26 completers of round one a URL link to participate in the second Delphi round. During this round, the panel members were provided with the tenets that did not meet the established statistical requirements for revision as well as each revised tenet. We also provided the original tenet and the revised tenet for the experts to review. We provided the mean, median, mode, and dispersion measure for each of the tenets. Panel members were asked to consider the statistical information and revisions and rerate each revised tenet using the same Likert-type scale. The panel members were also provided a text box to comment on each tenet. The survey remained open for 3 weeks. One reminder email to complete the second round was sent at 1.5 weeks after the initial invitation email and on the day the survey was scheduled to close. Nineteen of the 26 experts who were sent an invitation email started the survey and 18 (69.2%) of these experts completed the second round.

2.4.3. Round three

We sent the 18 completers of round two a URL link to participate in the third round. During this round, panel members were provided with the tenets that did not achieve consensus. Nine revised discovery systems and 10 revised discovery services tenets were sent to panel members to review and rate. If any minor revisions were made to a tenet for clarity, the original tenet and the revised version were presented for review. Similar to round two, we provided the corresponding statistical information for respondents to review. Because we wanted to increase expert understanding that some tenets were considered acceptable and others unacceptable components of discovery, we provided clarifying statements for each remaining discovery systems and services tenet. These statements indicated whether the tenet was presumed to be acceptable or unacceptable discovery. Respondents were also provided with comment box for each of the tenets.

2.5. Data analysis

We analyzed the second and third round responses for consensus among panel members. Criteria for consensus for tenets presumed to be acceptable was a median score four or higher and the IQR equal to or less than one. For tenets presumed unacceptable, consensus was achieved when the median score was two or lower and the IQR was equal to or less than one. We ranked acceptable and unacceptable systems and services tenets by median and IQR. We also examined comments from round two and three to determine any primary concerns panel members expressed on items that did not achieve consensus.

3. Results

Table 1 presents the demographic information for the expert panel. Half of the experts (50.0%) were consultants, 19.2% were community rehabilitation providers, 15.4% listed other (TA center project director, research grant director, executive director of a nonprofit, research associate), 11.5% worked for state government, and 3.9% worked for Federal Government. The majority of the experts (69.3%) had been employed in the disability field for over 20 years. The experts had considerable experience providing CE; 45.5% indicated they had over 15 years of experience, 50.0% indicated they had between five and 14 years of experience, and 4.5% indicated three years of experience. Over half (56%) were located either in Region IV (Alabama, Florida, Georgia, Kentucky, Mississippi, North Carolina, South Carolina, Tennessee) or Region VIII (Colorado, Montana, North Dakota, South Dakota, Utah, Wyoming). The majority of experts had a graduate degree (80.8%).

Table 1
Demographics of 1st round Respondents (n = 26)

Demographic n % of total

Position

Community Rehabilitation Provider 5 19.2

State Government Employee 3 11.5

Federal Government Employee 1 3.9

Consultant 13 50.0

Other 4 15.4

Years employed in disability field*

≥ 40 6 23.1

35–39 6 23.1

30–34 3 11.5

25–29 1 3.9

20–24 2 7.7

15–19 5 19.2

10–14 3 11.5

Years providing customized employment*

25–30 4 18.2

20–24 2 9.1

15–19 4 18.2

10–14 5 22.7

5–9 6 27.3

0–4 1 4.5

*Geographic Location**

Region I (Connecticut, Maine, Massachusetts, New Hampshire, Rhode Island, Vermont) 1 4.0

Region II (New Jersey, New York, Puerto Rico, US Virgin Islands 0 0.0

Region III (Delaware, District of Columbia, Maryland, Pennsylvania, Virginia, West Virginia) 4 16.0

Region IV (Alabama, Florida, Georgia, Kentucky, Mississippi, North Carolina, South Carolina, Tennessee) 9 36.0

Region V (Illinois, Indiana, Michigan, Minnesota, Ohio, Wisconsin) 2 8.0

Region VI (Arkansas, Louisiana, New Mexico, Oklahoma, Texas) 0 0.0

Region VII (Iowa, Kanas, Missouri, Nebraska) 1 4.0

Region VIII (Colorado, Montana, North Dakota, South Dakota, Utah, Wyoming) 5 20.0

Region IX (Arizona, California, Hawaii, Nevada, American Samoa, Guam, Northern Mariana Islands, Trust Territory of the Pacific Islands) 2 8.0

Region X (Alaska, Idaho, Oregon, Washington) 1 4.0

Education Level

Some College but no degree 1 4.0

Bachelor’s Degree 4 15.4

Master’s degree 16 61.5

Doctoral degree 5 19.3

Demographic	n	% of total
Position
Community Rehabilitation Provider	5	19.2
State Government Employee	3	11.5
Federal Government Employee	1	3.9
Consultant	13	50.0
*Other	4	15.4
Years employed in disability field
≥ 40	6	23.1
35–39	6	23.1
30–34	3	11.5
25–29	1	3.9
20–24	2	7.7
15–19	5	19.2
10–14	3	11.5
Years providing customized employment*
25–30	4	18.2
20–24	2	9.1
15–19	4	18.2
10–14	5	22.7
5–9	6	27.3
0–4	1	4.5
Geographic Location*
Region I (Connecticut, Maine, Massachusetts, New Hampshire, Rhode Island, Vermont)	1	4.0
Region II (New Jersey, New York, Puerto Rico, US Virgin Islands	0	0.0
Region III (Delaware, District of Columbia, Maryland, Pennsylvania, Virginia, West Virginia)	4	16.0
Region IV (Alabama, Florida, Georgia, Kentucky, Mississippi, North Carolina, South Carolina, Tennessee)	9	36.0
Region V (Illinois, Indiana, Michigan, Minnesota, Ohio, Wisconsin)	2	8.0
Region VI (Arkansas, Louisiana, New Mexico, Oklahoma, Texas)	0	0.0
Region VII (Iowa, Kanas, Missouri, Nebraska)	1	4.0
Region VIII (Colorado, Montana, North Dakota, South Dakota, Utah, Wyoming)	5	20.0
Region IX (Arizona, California, Hawaii, Nevada, American Samoa, Guam, Northern Mariana Islands, Trust Territory of the Pacific Islands)	2	8.0
Region X (Alaska, Idaho, Oregon, Washington)	1	4.0
Education Level
Some College but no degree	1	4.0
Bachelor’s Degree	4	15.4
Master’s degree	16	61.5
Doctoral degree	5	19.3

*Other TA center project director, Research grant director, ED of nonprofit, Research Associate. *Only 25 responses for geographic location. *23 responses for years in CE, 4 indicated they were involved with research or capacity building but not direct service delivery.

3.1. Discovery systems tenets

Table 2 presents the 15 discovery systems tenets considered acceptable ranked from high to low by median and IQR. Nine of these tenets met criteria for consensus after round two and five of these tenets met criteria for consensus after round three. Tenets one through five had a median score of five (always) and an IQR equal to zero. Tenets six through nine had a median score of five and an IQR equal to one. Tenet 10 had a median score of four (often) and an IQR equal to zero, tenets 11–14 had a median score of four and an IQR equal to one. Tenet 15 had a median score of four and an IQR of two; this tenet did not meet criteria for consensus.

Table 2
Discovery Systems Tenets Considered Acceptable Ranked High to Low Median and IQR

Discovery Systems Tenets Considered Acceptable Round 1 n = 26 Round 2 n = 18 Round 3 n = 12 Consensus

$\bar{x}$ Mdn SD IQR $\bar{x}$ Mdn SD IQR $\bar{x}$ Mdn SD IQR

1. Discovery should be authorized for anyone eligible for supported employment or customized employment. 4.57 5.00 0.662 1 4.77 5.00 0.427 0 Yes

2. Discovery should be authorized for working-age students who have significant disabilities. 4.26 5.00 1.01 1 4.82 5.00 0.528 0 Yes

3. The discovery payment rate should be adequate to ensure discovery services are delivered by a trained, skilled, and qualified employment specialists. 5.00 5.00 0 0 4.94 5.00 0.242 0 Yes

4. The discovery payment rate should be sufficient to ensure providers can deliver discovery and cover their costs. 4.16 4.00 1.02 1 3.77 4.00 1.23 2 4.91 5.00 0.288 0 Yes

5. The discovery payment rate should be sufficient to ensure a choice among discovery providers in every area of the state. 4.79 5.00 0.509 0 4.88 5.00 0.480 0 Yes

6. Discovery should be authorized as an alternative to traditional assessments or evaluations. 4.76 5.00 0.831 0 4.55 5.00 0.615 1 Yes

7. Discovery should always be authorized as a part of a customized employment or supported employment referral for services. 4.52 5.00 0.653 1 4.64 5.00 0.491 1 Yes

8. Discovery referrals should be issued to the specific agency that will also deliver the person’s customized employment or supported employment services 4.60 5.00 0.577 1 4.64 5.00 0.606 1 Yes

9. The discovery payment should allow for increased funding when discovery requires more hours than originally authorized. 4.28 5.00 0.891 2 4.47 5.00 0.799 1 Yes

10. Discovery should lead to high-quality outcomes that result in most persons working 20–40 hours each week in a community integrated job. 4.04 4.00 0.611 1 4.11 4.00 0.600 0 Yes

11. The employment specialist who provides discovery services should also provide the supported employment or customized employment services. 4.24 4.00 0.879 2 4.12 4.00 0.806 2 4.30 4.00 0.651 1 Yes

12. Discovery services should be delivered by personnel solely dedicated to providing employment services. 4.08 4.00 0.759 2 4.05 4.00 0.774 2 4.25 4.00 0.452 1 Yes

13. Discovery services should be delivered by the same persons who deliver other components of customized or supported employment. 4.12 4.00 0.833 2 4.00 4.00 0.935 2 4.16 4.00 0.717 1 Yes

14. Discovery services should be delivered by the same persons who deliver other components of customized or supported employment, including long-term support and follow-along services. 3.88 4.00 0.927 2 3.47 4.00 1.00 2 3.58 4.00 0.900 1 Yes

15. Each provider should be paid a reasonable individualized rate for payment of their costs to deliver discovery. 3.08 3.00 1.41 3 3.35 3.00 1.27 2 4.08 4.00 0.996 2 No

Discovery Systems Tenets Considered Acceptable	Round 1 n = 26	Round 2 n = 18	Round 3 n = 12	Consensus
1. Discovery should be authorized for anyone eligible for supported employment or customized employment.	4.57	5.00	0.662	1	4.77	5.00	0.427	0					Yes
2. Discovery should be authorized for working-age students who have significant disabilities.	4.26	5.00	1.01	1	4.82	5.00	0.528	0					Yes
3. The discovery payment rate should be adequate to ensure discovery services are delivered by a trained, skilled, and qualified employment specialists.	5.00	5.00	0	0	4.94	5.00	0.242	0					Yes
4. The discovery payment rate should be sufficient to ensure providers can deliver discovery and cover their costs.	4.16	4.00	1.02	1	3.77	4.00	1.23	2	4.91	5.00	0.288	0	Yes
5. The discovery payment rate should be sufficient to ensure a choice among discovery providers in every area of the state.	4.79	5.00	0.509	0	4.88	5.00	0.480	0					Yes
6. Discovery should be authorized as an alternative to traditional assessments or evaluations.	4.76	5.00	0.831	0	4.55	5.00	0.615	1					Yes
7. Discovery should always be authorized as a part of a customized employment or supported employment referral for services.	4.52	5.00	0.653	1	4.64	5.00	0.491	1					Yes
8. Discovery referrals should be issued to the specific agency that will also deliver the person’s customized employment or supported employment services	4.60	5.00	0.577	1	4.64	5.00	0.606	1					Yes
9. The discovery payment should allow for increased funding when discovery requires more hours than originally authorized.	4.28	5.00	0.891	2	4.47	5.00	0.799	1					Yes
10. Discovery should lead to high-quality outcomes that result in most persons working 20–40 hours each week in a community integrated job.	4.04	4.00	0.611	1	4.11	4.00	0.600	0					Yes
11. The employment specialist who provides discovery services should also provide the supported employment or customized employment services.	4.24	4.00	0.879	2	4.12	4.00	0.806	2	4.30	4.00	0.651	1	Yes
12. Discovery services should be delivered by personnel solely dedicated to providing employment services.	4.08	4.00	0.759	2	4.05	4.00	0.774	2	4.25	4.00	0.452	1	Yes
13. Discovery services should be delivered by the same persons who deliver other components of customized or supported employment.	4.12	4.00	0.833	2	4.00	4.00	0.935	2	4.16	4.00	0.717	1	Yes
14. Discovery services should be delivered by the same persons who deliver other components of customized or supported employment, including long-term support and follow-along services.	3.88	4.00	0.927	2	3.47	4.00	1.00	2	3.58	4.00	0.900	1	Yes
15. Each provider should be paid a reasonable individualized rate for payment of their costs to deliver discovery.	3.08	3.00	1.41	3	3.35	3.00	1.27	2	4.08	4.00	0.996	2	No

A closer examination of the systems tenets results revealed that there were a number of tenets that had strong agreement (median 5 for acceptable) and little variability (IQR≤1) among the experts. Specifically, these tenets (1–9) are related to authorization, referral, and funding. The expert panel strongly agreed that discovery should be authorized for anyone eligible for supported employment or customized employment, including working-aged students with disabilities. The expert panel also strongly agreed that payment rates should be sufficient to ensure that discovery be delivered by trained employment specialists, cover provider costs, and ensure choice among providers. In addition, experts strongly agreed that discovery should be authorized as an alternative to traditional assessments or evaluations. Tenet four, which states “The discovery payment rate should be sufficient to ensure providers can deliver discovery and cover their costs” required two revisions for clarity before it met criteria for consensus. Revisions to this tenet were based on second round comments from panel members. Specifically, some panel members indicated the initial wording was confusing and one panel member stated “While agencies must remain solvent, they must also accept responsibility for securing alternative funding when payment rates do not cover their costs. Alternative funding might include inter-agency collaboration and exchange of service/hours/personnel.” Another expert stated “As a private entity, providers should have the opportunity to make determinations about the referrals they accept.” The final revision for this tenet attempted to reflect the reality that while some discovery providers want to implement discovery, they may not have that capacity to do so without an acceptable payment rate.

Systems tenets 10–14 achieved consensus and had strong agreement (median score of 4) and little variability (IQR≤1). Tenets 11–14 required revisions for clarity before they met criteria for consensus. These tenets addressed information about who should be providing discovery services. Panel members expressed some concerns about the feasibility of an employment specialist providing both discovery services and other CE services. Specifically, in regard to tenet 11, one panel member stated: “I question whether this should be a tenet. The ES [employment specialist] providing SE or CE should be mutually agreed upon by the agency, the job seeker and perhaps even the employer. Making this a tenet might be tricky and send an inadvertent message to practitioners that minimizing the use and development of natural supports is acceptable.” Another panel member was concerned that turnover and agency capacity may be a barrier to the same person providing discovery and other phases of CE services. For tenet 12, panel members commented that there are times when non-paid persons such as family and friends may participate in discovery activities. For tenets 13 and 14, panel members expressed some concerns about the qualifications of personnel to support an individual with a disability through all phases of CE. Systems tenet 15, which states that each provider should be paid a reasonable individualized rate for discovery, did not meet criteria for consensus after three rounds because of variability in responses. Based on reviews of comments for this tenet, some of the panel members stated that a “flat rate” for discovery may be more reasonable and another expert stated that “Individualized rates based upon unique agency costs might inadvertently support and encourage ill-managed fiscal practices.”

Table 3 presents the five discovery systems tenets that were by design unacceptable discovery. Three of these tenets met criteria of consensus; two tenets met criteria after two rounds and one item met criteria after three rounds. Tenet number one had a median score of one (never) and an IQR equal to zero, tenets two and three had a median score of one and IQR equal to one. Tenet four and five did not meet criteria for consensus. Unacceptable tenet four had a median score of two (rarely) and IQR of two; tenet five had a median score of three (sometimes) and an IQR equal to two. In regard to tenet four, which states that discovery should be authorized only for people who have the most significant challenges to community employment, one panel member stated that this tenet was difficult to rate because the Likert-type scale used to rate this tenet was not appropriate. A number of panel members stated that they did not like the word “only” used for this unacceptable tenet. For example, one panel member stated “Everyone can benefit from a process of discovery, regardless of perceived or real challenges to securing community employment. To state discovery should ONLY be authorized for people with MS [the most significant disabilities] dismisses the value of discovery for all people. The word ‘only’ heavily influences my response” while another panel member stated, “I think the discovery process, and the creation of a discovery profile, can be beneficial for various aspects of individuals’ lives, and I wouldn’t want it to be so restricted (it is the “only” statement that I am stuck on).” There were a number of comments for unacceptable tenet five, which states that only one statewide provider payment rate should be used to fund discovery services. Specifically, one member stated “To my mind, the higher the fidelity and/or the more intensive a time crunch a case is, could warrant a sliding scale. Higher fidelity practice should be rewarded until we get to a point where a strict line in the sand can be drawn as a result of high standards being understood and practiced across the board. Blending/braiding of resources is essential to expanding access to discovery. However, current federal policies around “payer of last resort” greatly impede this.” Another panel member stated “While ideally it would be great to have some variation in terms of number of hours, etc. I worry about the difficulty of funders making decisions about who should get what rate.”

Table 3

Discovery Systems Tenets Considered Unacceptable Ranked by Median and IQR

Discovery Systems Tenet Considered Unacceptable	Round 1 n = 26				Round 2 n = 18				Round 3 n = 1				Consensus
	$\bar{x}$	Mdn	SD	IQR	$\bar{x}$	Mdn	SD	IQR	$\bar{x}$	Mdn	SD	IQR
1. Discovery should be used to determine whether someone can work.	2.72	3	1.51	3	1.35	1.00	0.780	0					Yes
2. It is acceptable for providers to make changes to the discovery process to reduce costs if the payment rate is insufficient.	2.48	2.00	1.38	3	2.82	3.00	1.13	1	1.41	1.00	0.688	1	Yes
3. Discovery services should be delivered by different personnel than those who provide the customized employment or supported employment services.	2.48	3.00	0.918	1	2.47	2.00	0.874	1					Yes
4. Discovery should be authorized only for people who have the most significant challenges to community employment.	2.92	3.00	1.32	3	2.83	3.00	1.46	3	2.50	2.00	1.50	2	No
5. Only one statewide provider payment rate should be used to fund discovery Services.	3.36	3.00	1.31	3	3.05	3.00	1.29	2	2.63	3.00	1.28	2	No

3.2. Discovery services tenets

Table 4 presents the 30 discovery services tenets considered acceptable, ranked high to low by median and IQR. The table provides descriptive statistics for each round of the Delphi process. Twenty-nine of these 30 service tenets met criteria for consensus; 25 tenets met criteria after the second round. The remaining four tenets met criteria for consensus after three rounds. Tenets one through 14 had strong agreement (median 5 for acceptable) and little variability (IQR = 0) among the experts. Six of these tenets explored discovery elements related to the home visits, three tenets provided information about neighborhood explorations, and one tenet provided information about specific observations of the employment seeker. The four-remaining high rated tenets provided information about elements of the vocational profile. Tenets 15–29 had a median score of five and little variability (IQR = 1). Three of these tenets provided information about the home visit, four tenets provided information about observations and compiling information from these observations, three tenets provided information about engaging with businesses (informational interviews), two tenets provided information about the vocational profile, and three tenets provided information about planning.

Table 4
Discovery Services Tenets Considered Acceptable Ranked High to Low by Median and IQR

Discovery Services Tenets Considered Acceptable Round 1 n = 26 Round 2 n = 18 Round 3 n = 12 Consensus

$\bar{x}$ Mdn SD IQR $\bar{x}$ Mdn SD IQR $\bar{x}$ Mdn SD IQR

1. During the home visit, the employment specialist should use natural conversation to learn new information. 4.65 5.00 0.573 1 4.70 5.00 0.578 0 Yes

2. During the home visit the employment specialist should gather information through natural interactions with the employment seeker. 4.61 5.00 0.583 1 4.76 5.00 0.437 0 Yes

3. During the home visit the employment specialist should learn about the employment seeker’s personal spaces, his or her belongings, and how they are used. 4.52 5.00 0.730 1 4.74 5.00 0.437 0 Yes

4. During the home visit, the employment specialist should learn the history and background behind the employment seeker’s valued possessions and expressed interests. 4.64 5.00 0.581 1 4.88 5.00 0.332 0 Yes

5. During the home visit, the employment specialist should gather information about the employment seeker’s chores and responsibilities. 4.52 5.00 0.593 0 4.68 5.00 0.662 0 Yes

6. During the home visit, the employment specialist should learn about the employment seeker’s interests and preferred activities or hobbies, including how he or she spends time. 4.61 5.00 0.583 1 4.82 5.00 0.392 0 Yes

7. The employment specialist should explore the employment seeker’s neighborhood by talking with key people such as, neighbors, business owners, and members of community institutions. 4.13 5.00 1.10 2 4.81 5.00 0.543 0 Yes

8. The employment specialist should conduct neighborhood visits at times they can meet with key people, which may include evenings or weekends. 4.39 5.00 0.941 1 4.93 5.00 0.258 0 Yes

9. The employment specialist should include others in the neighborhood explorations to gather different perspectives. 4.43 5.00 1.03 1 4.70 5.00 0.469 0 Yes

10. The employment specialist should observe the employment seeker completing more than three familiar activities that demonstrate his or her unique skills, interests, and support needs. 4.48 5.00 0.790 1 4.82 5.00 0.392 0 Yes

11. The vocational profile should identify ideal conditions for employment and vocational themes that are supported by evidence gathered during discovery. 4.57 5.00 0.662 1 4.82 5.00 0.392 0 Yes

12. The employment specialist should make changes or necessary revisions to the vocational profile based on information learned during the latter stages of the discovery process. 4.65 5.00 0.647 1 4.82 5.00 0.491 0 Yes

13. The vocational profile should be a narrative and descriptive document written in a positive and affirming manner. 4.22 4.00 1.04 1 4.82 5.00 0.392 0 Yes

14. The vocational profile should document the employment seeker’s skills, tasks, and accomplishments supported by objective descriptions, videos, and/or pictures. 4.68 5.00 0.646 1 4.94 5.00 0.024 0 Yes

15. During the home visit the employment specialist should gather information about the employment seeker’s daily life. 4.52 5.00 0.593 1 4.70 5.00 0.469 1 Yes

16. During the home visit the employment specialist should gather information about the employment seeker’s personal connections. 4.57 5.00 0.662 1 4.70 5.00 0.469 1 Yes

17. During the home visit the employment specialist should capture quotes from the employment seeker and his or her family and friends. 4.26 5.00 1.01 1 4.41 5.00 0.718 1 Yes

18. The employment specialist should observe the employment seeker actively participating in two or three familiar activities in his or her home or community that demonstrate multiple tasks or skills and reveals potential support needs. 4.43 5.00 0.662 1 4.70 5.00 0.469 1 Yes

19. The employment specialist should analyze the information gathered during home and community observations of the employment seeker completing familiar activities to identify emerging vocational patterns and preferences. 4.43 5.00 0.662 1 4.52 5.00 0.717 1 Yes

20. The employment specialists should use information gathered during home and community observations to identify skills, capacities, and potential vocational themes. 3.70 4.00 1.14 2 4.52 5.00 0.634 1 Yes

21. The employment specialist should gather information related to the employment seeker’s skills and support needs by observing him or her participating in activities in novel community locations. 4.04 5.00 1.33 2 4.64 5.00 0.701 1 Yes

22. The employment specialist should create opportunities at local businesses for the employment seeker to participate in activities related to his or her skills or emerging vocational themes. 4.13 5.00 1.18 2 4.35 5.00 1.14 1 Yes

23. The employment specialist and the employment seeker should conduct three to five informational interviews in businesses without apparent job openings that match the employment seeker’s skills and vocational themes. 4.13 5.00 1.21 2 4.05 5.00 1.47 2 4.00 5.00 1.54 1 Yes

24. The employment specialist and the employment seeker should use the recommendations and feedback gathered from business representatives during informational interviews to guide future discovery activities. 3.39 4.00 1.53 3 3.88 4.00 1.57 3 4.09 5.00 1.57 1 Yes

25. The vocational profile should include an analysis of what was learned during discovery. 3.96 4.00 1.14 2 4.52 5.00 0.624 1 Yes

26. The vocational profile should include a next-steps list for subsequent job development. 3.74 4.00 1.06 3 4.11 5.00 1.36 2 4.36 5.00 1.28 1 Yes

27. Discovery should conclude with a job development plan that includes contacting specific businesses related to the employment seeker’s identified skills and vocational themes. 4.00 4.00 1.12 1 4.68 5.00 0.602 1 Yes

28. The job development plan should identify dozens of local businesses that relate to the employment seeker’s vocational themes. 4.29 4.00 0.644 1 4.60 5.00 0.507 1 Yes

29. Discovery team members should meet and work with the employment seeker to create and assist with the initial implementation of the job development plan. 4.45 5.00 0.962 1 4.60 5.00 1.05 0 4.36 5.00 1.28 1 Yes

30. The employment specialist should ensure the employment seeker has the opportunity to try new tasks in businesses that match his or her skills or vocational themes. 4.04 5.00 1.39 2 4.05 4.00 1.34 2 4.08 5.00 1.56 2 No

Discovery Services Tenets Considered Acceptable	Round 1 n = 26	Round 2 n = 18	Round 3 n = 12	Consensus
1. During the home visit, the employment specialist should use natural conversation to learn new information.	4.65	5.00	0.573	1	4.70	5.00	0.578	0					Yes
2. During the home visit the employment specialist should gather information through natural interactions with the employment seeker.	4.61	5.00	0.583	1	4.76	5.00	0.437	0					Yes
3. During the home visit the employment specialist should learn about the employment seeker’s personal spaces, his or her belongings, and how they are used.	4.52	5.00	0.730	1	4.74	5.00	0.437	0					Yes
4. During the home visit, the employment specialist should learn the history and background behind the employment seeker’s valued possessions and expressed interests.	4.64	5.00	0.581	1	4.88	5.00	0.332	0					Yes
5. During the home visit, the employment specialist should gather information about the employment seeker’s chores and responsibilities.	4.52	5.00	0.593	0	4.68	5.00	0.662	0					Yes
6. During the home visit, the employment specialist should learn about the employment seeker’s interests and preferred activities or hobbies, including how he or she spends time.	4.61	5.00	0.583	1	4.82	5.00	0.392	0					Yes
7. The employment specialist should explore the employment seeker’s neighborhood by talking with key people such as, neighbors, business owners, and members of community institutions.	4.13	5.00	1.10	2	4.81	5.00	0.543	0					Yes
8. The employment specialist should conduct neighborhood visits at times they can meet with key people, which may include evenings or weekends.	4.39	5.00	0.941	1	4.93	5.00	0.258	0					Yes
9. The employment specialist should include others in the neighborhood explorations to gather different perspectives.	4.43	5.00	1.03	1	4.70	5.00	0.469	0					Yes
10. The employment specialist should observe the employment seeker completing more than three familiar activities that demonstrate his or her unique skills, interests, and support needs.	4.48	5.00	0.790	1	4.82	5.00	0.392	0					Yes
11. The vocational profile should identify ideal conditions for employment and vocational themes that are supported by evidence gathered during discovery.	4.57	5.00	0.662	1	4.82	5.00	0.392	0					Yes
12. The employment specialist should make changes or necessary revisions to the vocational profile based on information learned during the latter stages of the discovery process.	4.65	5.00	0.647	1	4.82	5.00	0.491	0					Yes
13. The vocational profile should be a narrative and descriptive document written in a positive and affirming manner.	4.22	4.00	1.04	1	4.82	5.00	0.392	0					Yes
14. The vocational profile should document the employment seeker’s skills, tasks, and accomplishments supported by objective descriptions, videos, and/or pictures.	4.68	5.00	0.646	1	4.94	5.00	0.024	0					Yes
15. During the home visit the employment specialist should gather information about the employment seeker’s daily life.	4.52	5.00	0.593	1	4.70	5.00	0.469	1					Yes
16. During the home visit the employment specialist should gather information about the employment seeker’s personal connections.	4.57	5.00	0.662	1	4.70	5.00	0.469	1					Yes
17. During the home visit the employment specialist should capture quotes from the employment seeker and his or her family and friends.	4.26	5.00	1.01	1	4.41	5.00	0.718	1					Yes
18. The employment specialist should observe the employment seeker actively participating in two or three familiar activities in his or her home or community that demonstrate multiple tasks or skills and reveals potential support needs.	4.43	5.00	0.662	1	4.70	5.00	0.469	1					Yes
19. The employment specialist should analyze the information gathered during home and community observations of the employment seeker completing familiar activities to identify emerging vocational patterns and preferences.	4.43	5.00	0.662	1	4.52	5.00	0.717	1					Yes
20. The employment specialists should use information gathered during home and community observations to identify skills, capacities, and potential vocational themes.	3.70	4.00	1.14	2	4.52	5.00	0.634	1					Yes
21. The employment specialist should gather information related to the employment seeker’s skills and support needs by observing him or her participating in activities in novel community locations.	4.04	5.00	1.33	2	4.64	5.00	0.701	1					Yes
22. The employment specialist should create opportunities at local businesses for the employment seeker to participate in activities related to his or her skills or emerging vocational themes.	4.13	5.00	1.18	2	4.35	5.00	1.14	1					Yes
23. The employment specialist and the employment seeker should conduct three to five informational interviews in businesses without apparent job openings that match the employment seeker’s skills and vocational themes.	4.13	5.00	1.21	2	4.05	5.00	1.47	2	4.00	5.00	1.54	1	Yes
24. The employment specialist and the employment seeker should use the recommendations and feedback gathered from business representatives during informational interviews to guide future discovery activities.	3.39	4.00	1.53	3	3.88	4.00	1.57	3	4.09	5.00	1.57	1	Yes
25. The vocational profile should include an analysis of what was learned during discovery.	3.96	4.00	1.14	2	4.52	5.00	0.624	1					Yes
26. The vocational profile should include a next-steps list for subsequent job development.	3.74	4.00	1.06	3	4.11	5.00	1.36	2	4.36	5.00	1.28	1	Yes
27. Discovery should conclude with a job development plan that includes contacting specific businesses related to the employment seeker’s identified skills and vocational themes.	4.00	4.00	1.12	1	4.68	5.00	0.602	1					Yes
28. The job development plan should identify dozens of local businesses that relate to the employment seeker’s vocational themes.	4.29	4.00	0.644	1	4.60	5.00	0.507	1					Yes
29. Discovery team members should meet and work with the employment seeker to create and assist with the initial implementation of the job development plan.	4.45	5.00	0.962	1	4.60	5.00	1.05	0	4.36	5.00	1.28	1	Yes
30. The employment specialist should ensure the employment seeker has the opportunity to try new tasks in businesses that match his or her skills or vocational themes.	4.04	5.00	1.39	2	4.05	4.00	1.34	2	4.08	5.00	1.56	2	No

Tenets 23, 24, 26, and 29 required three rating iterations before they met criteria for consensus. In regard to tenet 23 and 24, which addresses conducting and gathering feedback through informational interviews, some experts stated that informational interviews are a part of job development and not discovery. One panel member stated “I know the Griffin-Hammis model does include this activity, and I see that most of the other people who responded are in total agreement; however, in other trainings I have participated in regarding the discovery process, the informational interviews with businesses are part of job development, not the profile development process.” For tenet 29, panel members commented “My experience is that having a meeting (another meeting!) is not critical to creating or implementing a job development plan. There are many ways to include all team members in the process through good communications that occur throughout the discovery process and to include all team members. There are other strategies that can achieve the same result of having team members help with connections, ideas, and supports to implement job development.”

Tenet item 30, which states that the employment seeker should have the opportunity to try new tasks in businesses that match his or her skills or vocational themes, did not meet criteria for consensus and had a median score of five and an IQR equal to two. A review of comments for this tenet indicated that some experts believed that this tenet was too narrow or believed it represented an expanded form of discovery. For example, one panel member stated “the opportunity to demonstrate tasks and skills that are new or novel to the job seeker do not have to occur in a business; this opportunity could be at home, or other community setting too” while another stated “I am not trying to be a contrarian but these strategies would need to be labeled as a form of “expanded discovery” rather than the discovery concept defined here.”

Table 5 presents the 10 discovery service tenets that were purposely designed to be unacceptable, ranked from low to high by median and IQR. Nine of these 10 unacceptable tenets met criteria for consensus; five of these unacceptable tenets met criteria for consensus after the second round, four unacceptable tenets met criteria for consensus after three rounds, and one tenet did not meet criteria for consensus after three rounds. Unacceptable tenets one and two had a median score of one (not discovery) and IQR equal to zero. Tenets three through six had a median of one and an IQR of one. Tenet seven had a median score of two (poor discovery) and an IQR equal to zero. Tenets eight and nine had a median score of two and an IQR equal to one. Unacceptable tenet 10, which states that the employment specialist should assess the employment seeker performing activities of daily living, had a median score of two and an IQR equal to two and did not achieve panel consensus. Comments from the panel for this tenet indicated that some experts were uncomfortable with the word “assessment” being used. For example, one panel member said “The word ‘assessment’ throws me off because I think it implies we are comparing or measuring a person’s skills” while another panel member said “Discovery is not an assessment of any sort, in my opinion.” Another expert stated “I do not like the word assessment. This implies the use of a checklist, which may be intrusive.” Finally, another expert stated “Discovery uses descriptive observations, not assessment, to understand the individual.”

Table 5

Discovery Services Tenets Considered Unacceptable Ranked Low to High by Median and IQR

Discovery Services Tenet Considered Unacceptable	Round 1n = 26				Round 2 n = 18				Round 3 n = 12				Consensus
	$\bar{x}$	Mdn	SD	IQR	$\bar{x}$	Mdn	SD	IQR	$\bar{x}$	Mdn	SD	IQR
1. During the home visit, the employment specialist should meet with the employment seeker and his or her family to fill out necessary forms.	1.61	1.00	0.891	2	1.58	1.00	0.870	2	1.08	1.00	0.228	0	Yes
2. The employment specialist should begin job development by visiting businesses to ask about available jobs and to inquire about the possibility of the employment seeker working there.	1.00	1.00	0	0	1.17	1.00	0.528	0					Yes
3. During the home visit, the employment specialist should assess the general organization and cleanliness of the home.	2.65	3.00	1.30	3	2.52	3.00	1.00	1	1.54	1.00	0.820	1	Yes
4. The employment specialist should use pre-established job sampling sites to assess the employments seeker’s skills.	1.83	1.00	1.37	2	1.29	1.00	0.587	1					Yes
5. To complete the vocational profile, the employment specialist should complete forms by asking a series of questions and recording the employment seeker’s and family member’s responses.	2.35	2.00	1.4	3	2.53	2.00	1.23	1	1.25	1.00	0.452	1	Yes
6. The vocational profile should document the results of how the employment seeker performed on assessments to determine possible jobs for future employment.	1.35	1.00	0.714	1	1.29	1.00	0.587	1					Yes
7. The discovery documentation should not include detailed information on specific businesses for the job developer to contact during job development.	2.09	2.00	1.06	2	1.93	2.00	0.444	0					Yes
8. If a home visit or a visit to a mutually acceptable place in the community cannot be arranged, a meeting with the family may be held at the provider agency.	1.83	2.00	0.650	1	1.94	2.00	0.740	1					Yes
9. The employment specialist should conduct internet research as the primary way to learn about businesses or landmarks near the employment seeker’s home.	2.83	3.00	1.40	3	2.94	2.00	1.04	1	1.75	2.00	0.621	1	Yes
10. The employment specialist should assess the employment seeker performing activities of daily living such as, self-help, personal hygiene, and eating.	2.87	3.00	1.32	3	2.58	2.00	1.46	3	1.91	2.00	0.900	2	No

4. Discussion

Experts in CE discovery participated in a three round, modified Delphi study to identify acceptable and unacceptable discovery systems and services tenets outlined in the DFS. The information gleaned from experts in CE discovery is the first step in validating both systems and services tenets used during the discovery process. This validation process is important because there appears to be only a small number of studies on the implementation and efficacy of CE (Riesen, et al., 2015) and there is a renewed priority to ensure rehabilitation professionals are implementing evidence-based practices (Huber, Oswald, Chan, Shaw, & Wilson, 2017).

The DFS included tenets designed to discriminate between practices that meet discovery standards and other practices or activities that may be inconsistent or unacceptable with discovery standards. After three Delphi rounds, experts agreed on a number of unacceptable discovery practices. Unacceptable discovery practices included filling out forms during home visits, asking employers about available jobs, using pre-established job sampling sites, using assessments or evaluations to determine possible jobs, meeting at provider agencies, etc. These unacceptable discovery practices align with other research and reports about CE (Riesen & Morgan, 2018; WINTAC, 2017) and can be used to help practitioners understand how discovery is different than traditional comparative assessments or evaluations.

Even though consensus was achieved for the majority of discovery systems and services tenets, there appeared to be some nuanced difference about what panel members considered discovery and what tenets should be included in a fidelity scale. For example, systems tenet 11 states that the persons completing discovery should be the same person implementing other components of customized or supported employment. One expert commented “While this might seem ideal … a fidelity component should not dictate personnel decisions.” The one acceptable systems level tenet that did not achieve consensus (tenet 15) posits that there should be a custom reimbursement rate for every provider based on their costs. The lack of consensus on this tenet may reflect unfamiliarity with individually determined provider rates based on actual provider costs for integrated employment components which have been available in some states since 1985. The lack of consensus on this tenet may also be attributed to concerns about fiscal management as one expert stated that a custom rate might “inadvertently support and encourage ill-managed fiscal practice.” Despite not meeting criteria for consensus, a review of comments for tenet 15 seemed to suggest that experts generally agreed rates should be reasonable, but they are unsure about how to structure these reimbursement rates. For example, one expert stated “Again, I am in favor of reasonable, established rates at the state level. All rates should be reasonable.” Similarly, unacceptable systems tenet five, which states that only one statewide provider payment rate should be utilized for discovery services had different expert opinions. Experts rated this unacceptable tenet within the acceptable range (median 3, acceptable discovery), despite the fact that it seems reasonable to provide financing for persons who may take more time and effort. A review of comments for this tenet indicated that most believed some rate variations would be beneficial but may not be feasible. One stated concerns over the possibility that providers would take advantage of such a system by establishing “sweetheart deals” and another expressed “… worry about the difficulty of funders making decisions about who would get what rate.” Another participant commented, “I think one provider payment rate would be helpful, provided there is a way to request additional funding if the process takes longer than what is typically expected.”

Experts also expressed differences in approaches for some of the acceptable discovery services tenets. For example, acceptable discovery services tenets 23 and 24 addressed discovery activities and informational interviews. Reviewer comments indicated differences of opinions over whether these practices should be included as a part of discovery or as a part of job development. For example, one expert stated “I consider this part of customized employment, not discovery and gathering information to complete the profile. I am confused, and I apologize that this response is not in alignment with the others—but this tenet is referencing “career exploration” not discovery. The strategies in this component might guide job development but if one found job features during informational interviews, those features do not define the individual, they help in planning and job development.” Similarly, for acceptable services tenet 26, some expert comments emphasized the importance of a next steps list guiding subsequent activities of the job developer, while others agreed it is critical to have an employment plan but this work is a component of job development and not discovery: “A distinct planning process should include the next steps for job development, not the profile.” Finally, differences in approaches were evident in comments about acceptable services tenet 29. Some comments for this tenet indicated that the facilitator of discovery should “facilitate” the plan but not “create” the job development plan.

5. Limitations

The utility of Delphi research method to gather information and build consensus from a group of experts is well documented. However, there are some limitations to this type of research methodology. While we attempted to identify and select individuals who are knowledgeable about CE discovery, their response must be interpreted with some caution because their responses are professional opinion rather than fact. The small sample size also limits generalizability of the results as the individual responses for each identified tenet may be different in a larger sample or with different expert panel members. Finally, while attempts were made to incorporate universal discovery practices in the DFS, the tenets outlined in this study represent one approach to discovery systems and discovery services implementation.

6. Conclusions and future research

While there appear to be some differences in what experts believe are essential tenets of discovery, almost all (55 of 60) of both the acceptable and not acceptable discovery systems and services tenets met criteria for consensus after three iterative rounds. Based on the results, experts unequivocally support the contention that discovery is an alternative to traditional employment assessments. This distinction aligns with information about discovery outlined in the literature (e.g. Callahan, Shumpert, & Condon, n.d.; Griffin, Hammis, Geary, 2007: Phillips, et al. 2009; WINTAC, 2017). In addition, experts also strongly agree that discovery should be authorized for individuals eligible for supported employment or customized employment and that payment rates should be sufficient to ensure discovery is implemented by well-trained employment specialists. Ensuring that employment specialists are well-trained to support people with disabilities in obtaining and maintaining meaningful work is certainly a priority (Harker, & Desenberg-Wines, 2017; Jorgensen, Dillahunt-Aspillaga, & Kenney, 2015), and a validated fidelity scale can ensure consistent implementation of discovery. Without operationalized discovery fidelity tenets, we run the risk of poorly informed employment specialists implementing strategies that do not comport to principles of discovery. This Delphi process provided valuable insights about what experts believe are best discovery practices which is a first step in creating a validated fidelity scale. To further validate the DFS, we recommend that future research examine the predictive validity of the DFS to determine if tenets are correlated to positive employment outcomes. Research should examine ways in which aligning discovery systems implementation practices with the DFS in the areas of authorization and access, finance, and quality requirements for discovery providers improves employment outcomes. Similarly, implementation research should examine ways in which reliable employment outcomes occur by aligning discovery services practices with discovery services tenets related to home and neighborhood visits, discovery activities, informational interviews, vocational profiles, and employment plans.

Conflict of interest

None to report.

References

Bond, G. R., Becker, D. R., & Drake, R. E. (2011). Measurement of fidelity of implementation of evidence-based practices: Case example of the IPS fidelity scale. Clinical Psychology Science and Practice, 18, 126–141.

Bond, G. R., Becker, D. R., & Drake, R.E., & Volgler, K. M. (1997). A fidelity scale for the individual placement and support model of supported employment. Rehabilitation Counseling Bulletin,40(4), 265–284.

Callahan, M., Shumpert, N., & Condon, E.(n.d.).A strategy to provide the foundation for customized employment for persons with complex lives. Marc Gold & Associates.

Custer, R. L.

Scarcella, J. A., & Stewart, B. R. (1999). The modified Delphi technique: Arotational modification. Journal of Career and Technical Education, 15(2), 1–10.

Fraenkel, J. R. & Wallen, N. E. (2003). How to design and evaluate research in education. New York, NY: MeGraw-Hill.

Griffin, C., Hammis, D. and Geary, M. (2007). The job developer's handbook: Practical tactics for customized employment. Baltimore, MD: Brookes

Hall, S., Keeton, B., Cassidy, P., Iovannone, R., & Griffin, C. (2016). Discovery fidelity scale. Atlanta, GA: Center for Social Capital.

Harker, B., & Desenberg-Wines, A. (2017). Iowa coalition for integrated employment. Journal ofVocational Rehabilitation, 47(3), 317–326.

Hiersteiner, D., Butterworth, J.Bershadsky, J. & Bonardi, A. (2016). Working in the community: The status and outcomes of people with intellectual and developmental disabilities in integrated employment—Update 2. NCI Data Brief,. Cambridge, MA: Human Services Research Institute.

10.

Huber, M. J., Oswald, G., Chan, F., Shaw, L.R., & Wilson, J., (2017). A call for a national evidence-based programs and practices register in vocational rehabilitation. Journal ofVocational Rehabilitation, 46, 11–18.

11.

Kim, S. J., Bond, G. R., Becker, D. R., Swanson, S. J., & Langfitt-Reese, S. (2015). Predictive validity of the Individual Placement and Support fidelity scale (IPS-25): A replication study. Journal of Vocational Rehabilitation, 43, 209–216.

12.

Jorgensen Smith, T., Dillahunt-Aspillaga, C., & Kenney, C. (2015). Integrating customized employment practices within the vocational rehabilitation system. Journal of Vocational Rehabilitation, 42, 201–208.

13.

Keeney, S., Hasson, F. & McKenna, H., (2011). The Delphi technique in nursing and health research.John Wiley & Sons.

14.

Knaeps, J., DeSmet, A., & Van Audenhove, C. (2012). The IPS fidelity scale as a guideline to implement supported employment. Journal of Vocational Rehabilitation, 37, 13–23.

15.

Phillips, W. L., Callahan, M., Shumpert, N., Puckett, K., Petrey, R., Summers, K., & Phillips, L. (2009). Customized transitions: Discovering the best in us. Journal of Vocational Rehabilitation, 30, 49–55.

16.

Riesen, T., & Morgan, R. L. (2018). Employer’s perspective of customized employment: A focus group analysis. Journal of Vocational Rehabilitation, 49(1), 33–44.

17.

Riesen, T., Morgan, R. L., & Griffin, C. (2015). Customized employment: A review of the literature. Journal of Vocational Rehabilitation, 43183–193.

18.

Wehman, P., Inge, K., Revell, G. W., & Brooke, V. A. (2007). Real work for real pay: Inclusive employment for people with disabilities. Baltimore, MD: Brookes.

19.

Winsor, J., Timmons, J., Butterworth, J., Shepard, J., Landa, C., Smith, F., Domin, D., Migliore, A., Bose, J., & Landim, L. (2017). StateData: The national report on employment services and outcomes.Boston, MA: University ofMassachusetts Boston, Institute for Community Inclusion.

20.

WINTAC, (2017). The essential elements of customized employment for universal application. Workforce Innovation Technical Assistance Center. Retrieved from: www.wintac.org/topic-areas/resources-and-strategies-competitive-integrated-employment/resources.

21.

Workforce Innovation and Opportunity Act 29 U.S.C §705 et seq. (2014).