The Diversity Partners Project: Multi-systemic knowledge translation and business engagement strategies to improve employment of people with disabilities

Abstract

BACKGROUND: Fragmented employment service systems, a lack of information about disability across sectors, and a wide disconnect between the efforts to employ people with disabilities and the needs of the business community, have presented barriers to the adoption of best practices and effective implementation of new policies designed to improve the employment rate of people with disabilities.

OBJECTIVE: The goal of the Diversity Partners Project is to develop, test, and launch an innovative learning intervention to improve the relationships between employment service professionals (including disability services, workforce development and staffing organizations) and employers seeking to hire individuals with disabilities. This paper illustrates how a systematic approach to knowledge translation (KT) was used in an iterative intervention development process that engaged key stakeholders at every phase.

CONCLUSION: In the disability arena, KT requires a willingness to challenge strongly held assumptions on the part of the project team, to move swiftly and repeatedly between inquiry and development, and to honestly engage with potential stakeholders who have a vested interest in the development efforts being undertaken.

Keywords

Disability employer partnerships knowledge translation qualitative inquiry capacity building workforce development

1 Introduction

Despite the efforts of lawmakers and employment service professionals, the employment rate of individuals with disabilities has not been profoundly improved in the 21st century (Erickson & von Schrader, 2016). Factors that contribute to the problem of unemployment among people with disabilities, such as the growth in funding of segregated supports and services over the last several decades (Butterworth et al., 2015; Inge et al., 2009; Nazarov, Golden, & von Schrader, 2012; Rusch & Braddock, 2004), and the receipt of public benefits, which may have the unintended consequence of a reduction in work activity among a broad population of people with disabilities (Golden, Zeitzer, & Bruyère, 2014; Nord & Nye-Lengerman, 2015), are compounded by poorly compensated direct support personnel with a high rate of job turnover in non-profit agencies that provide employment services (Bogenshutz, Hewitt, Nord, & Hepperlen, 2014). Further, our observations from the field suggest that fragmented employment service systems, a lack of information about disability across sectors, and a wide disconnect between the efforts to employ people with disabilities and the needs of the business community have presented barriers to the adoption of best practices and effective implementation of new policies designed to improve the employment rate of people with disabilities.

The goal of the Diversity Partners Project is to develop, test, and launch an innovative learning intervention that will improve the employment outcomes of people with disabilities by enhancing the capabilities of key arbiters around building relationships with employers. Key arbiters, according to Diversity Partners, are employment service professionals (ESPs) in different parts of the employment service system: workforce development staff who are skilled at using labor market data and building relationships with employers, community-based disability service professionals who have a deep understanding of the impact of disability at work, and staffing firm staff who specialize in locating human resources in response to the demands of employers.

These arbiters are the target audience for the Diversity Partners intervention, and they all face barriers in serving as an effective conduit to employment for job seekers with disabilities. Some of these barriers are distinct to each group, whereas others are shared experiences. Each group also has distinct strengths that, if shared, could improve the capacity of others to better support the employment of people with disabilities for the long term. Using information gathered from this target audience, as well as from employers and from job seekers with disabilities, Diversity Partners aims to facilitate knowledge translation among stakeholders who play a role in improving employment opportunities for people with disabilities. The intervention consists of one website that contains online training modules for organizational leadership and frontline personnel. The self-paced modules are reinforced through on-demand technical assistance and training events offered by subject matter experts to support both business partnerships and overall excellence in serving job seekers with disabilities.

1.1 Challenges faced by employment service systems

Recent policy changes, such as rules under Section 503 of the Rehabilitation Act and changes initiated by Title I and Title IV of the Workforce Innovation and Opportunities Act (WIOA), have been designed to reduce fragmentation and promote cross-sector partnerships to increase opportunities for individuals with disabilities. The effectiveness of these initiatives may rest largely in the ability of employment service professionals to create a bridge between job seekers with disabilities and employers – one that is built upon mutually beneficial and sustained partnerships.

Historically, services within the workforce development system have not been widely available to individuals with disabilities (Hall & Parker, 2005; Luecking, Cuozzo, Leedy, & Seleznow, 2008). The new WIOA rules required increased access for job seekers with disabilities and made these systems accountable for serving them. While the workforce development system is by design connected to the business community, our previous experience with this system raised questions about its readiness to effectively serve job seekers with disabilities. The Diversity Partners inquiry has supported the assumption that industry-driven strategies are not being leveraged to the advantage of individuals with disabilities.

Research suggests that there are gaps between the support needs of employers and what they are offered by community-based disability services (Simonsen, Fabian, Buchanan, & Luecking, 2011; Waterhouse, Kimberley, Jonas, & Glover, 2010). The new rules under Section 503 of the Rehabilitation Act and the ADA Amendments Act of 2008 have impacted the demand for disability related employment services. Federal contractors are seeking additional assistance from these professionals in the form of training, on-site support, and development of a pipeline of qualified candidates with disabilities. Our inquiry and pilot interventions for Diversity Partners have demonstrated that disability service professionals are largely unaware of this change in demand for their services.

Staffing firms lack a similar mandate to improve services to job seekers with disabilities; however, these businesses (particularly contingent labor, or “temp” firms) do interact with people with multiple barriers to employment, including disability. Our inquiry of this group was rooted in the assumption that staffing firms are highly invested in meeting the workforce needs of the employers who are their clients, but are not aware of the reasons why specific employers may be incentivized to recruit qualified candidates with disabilities.

1.2 Stakeholder engagement to support knowledge translation

Diversity Partners was designed as a knowledge translation project aimed at the development of an intervention that would improve the ability of ESPs to form long term partnerships with employers. As such, it seeks to contextualize knowledge in daily practice of the people who will use the emerging intervention. Knowledge translation (KT) “promotes the use of research-based knowledge to support the ability of individuals to live successfully in society” and it requires that researchers “involve relevant stakeholders in the design and conduct of research activities to optimize the relevance and use of proposed outputs, use tools such as systematic reviews and research synthesis to assess and disseminate the information generated through research, and translate research findings into information that is usable by individuals with disabilities and their families, practitioners, and policymakers” (U.S. Department of Education, 2013). According to Sudsawad (2007), one of the key characteristics of KT is that it “encompasses all steps between the creation of new knowledge and its application to yield beneficial outcomes for society.” Meaningful interaction with people who will apply the findings in their everyday lives is critical to this process. Though KT is frequently called for in the disability community, it is rarely applied effectively. All too often, disability researchers see KT as an add-on rather than an integral part of the process. In contrast, the Diversity Partners Project has aimed to collect and synthesize the knowledge and experiences of its intended audience and other stakeholders in a number of ways throughout the development process to ensure that: 1. The project’s assumptions about barriers to effective practices and business relationships were correct; 2. the content produced for the intervention was rooted in the everyday work of ESPs across public, private, and non-profit organizations; and 3. the recommendations produced as part of the process were consistent with both the needs and constraints of the business community.

As mentioned above, the stakeholders in the Diversity Partners intervention are employment service professionals, employers, and job seekers with disabilities. These three stakeholder groups were involved in the KT and development process in different ways, based on their intended relationship with the finalized intervention. Employment service professionals (disability services, workforce development, and staffing/contingent labor) are the target audience: i.e. those who will utilize and learn from the intervention. Employers are a secondary audience: they will be affected by the target audience’s changed actions as a result of the intervention but will not themselves directly utilize it. Job seekers with disabilities, finally, are defined as the beneficiaries of the intervention: they will reap the benefits from the changed actions of the target audience and secondary audience but will not utilize the intervention (see also Table 1).

Table 1
The Diversity Partners stakeholders and their role in the intervention

Stakeholder Group Intervention Role Intended Use

Employment service professionals (disability services, workforce development, and staffing/ contingent labor) Target audience Will utilize and learn from the intervention.

Employers Secondary audience Will be affected by the target audience’s changed actions as a result of the intervention but will not directly utilize it.

Job seekers with disabilities Beneficiaries Will benefit from the changed actions of the target and secondary audiences but will not utilize the intervention.

Stakeholder Group	Intervention Role	Intended Use
Employment service professionals (disability services, workforce development, and staffing/ contingent labor)	Target audience	Will utilize and learn from the intervention.
Employers	Secondary audience	Will be affected by the target audience’s changed actions as a result of the intervention but will not directly utilize it.
Job seekers with disabilities	Beneficiaries	Will benefit from the changed actions of the target and secondary audiences but will not utilize the intervention.

Knowledge translation requires a method of inquiry that ensures that the exchange of knowledge occurs not only by means of the finalized product, but also as a result of the development process itself. Existing KT frameworks are mostly derived from the field of healthcare, and rely largely on one-directional communication aimed at transferring knowledge from the researcher to the practitioner (Rudstam, Gower, & Van Looy, 2016). The Diversity Partners inquiry, by contrast, was informed by continuous stakeholder engagement and feedback, taking into account the differing contexts, knowledge needs, and power structures among the actors who will ultimately implement the recommendations created by the intervention itself. Thus, Diversity Partners is a true knowledge exchange effort; involving stakeholders in both creation and action which include identifying high need content areas, connecting actors within cross-systems advisory groups, and providing a mechanism for a continuous development-feedback loop (Graham et al., 2006) to inform the next step(s) of the development process and to reflect on and refine decisions already made.

2 Methodology

As with many development efforts, the Diversity Partners Project began with an existing base of knowledge and research, and a set of informed assumptions. The assumptions related to the nature of the issue we had set out to resolve (the repairing of the bridge between labor supply and demand), the characteristics of project stakeholders (their contexts, their knowledge base, their needs, and their relationships with each other), and the specific strategies and modalities proposed as solutions (online learning tools). Surfacing the assumptions and examining their merit, while building our knowledge base in collaboration with relevant stakeholders would be key to project success. Figure 1 depicts the conceptual model of the project’s development from its initial foundation of research-based content knowledge and informed assumptions to solidified content knowledge and understanding of design imperatives through field-based inquiry, to the development of a learning intervention grounded in the lived experiences of its diverse stakeholders.

Fig.1

Conceptual model of the project’s development pathway: Existing content knowledge and assumptions were turned into solidified knowledge and understanding of design imperatives through field-based inquiry, which was incorporated into a well-tested learning intervention through thorough development efforts and further inquiry.

Simplified, the development process can be described as unfolding in four sequential phases: 1. The Inquiry Phase , in which we engaged in internal exploration of foundational project assumptions, identified overarching research questions, engaged in our first knowledge exchange efforts with select stakeholders (stakeholder exploration), and situated the emerging intervention in relevant literature (literature review); 2. The Development Phase , in which we focused on content and web development based on lessons learned in the previous phase, while continuing to add to our knowledge base through purposeful content and stakeholder exploration, leading to constant refinement of both content and web structure; 3. The Pilot Phase , currently ongoing, in which we test the intervention with select pilot sites while continuing the development as informed by evaluative feedback; and 4. The Evaluation and Refinement Phase , in which we will launch the online learning intervention to a national audience. Content refinement and reorganization will continue throughout this phase based on information gathered through integrated evaluative processes.

The four phases and the processes involved are depicted in Fig. 2, and also outlined in some detail in Table 2. As shown, in the transition between each phase there is a “decision point.” Each point signifies a specific moment in time where the project team has dedicated itself to collaborative sense-making and reflection, and where findings from the preceding processes have been synthesized, reviewed, and used to inform next steps.

Fig.2

The phases, processes, and decision points of the Diversity Partners intervention development.

Table 2

The processes and decision points involved in the Diversity Partners knowledge translation and intervention development

Processes		Decision Point Focus	Status
Phase 1: Inquiry
1.	Internal exploration	Content, structure, and further knowledge	Complete
2.	Stakeholder exploration: Advisory Panel	needs: What content do we need to present
3.	Target audience exploration: understanding mental models through qualitative inquiry	to whom and how?
4.	Literature exploration
Phase 2: Development
1.	Web development	Planning for the pilot: How do we most	Mostly complete
2.	Content development	effectively test the intervention and gather	(content development and
3.	Content exploration	sufficient data to refine and further	surveys still in process)
4.	Target audience exploration: web site user feedback	develop necessary content and structures?
5.	Stakeholder exploration: ESP and employer surveys
Phase 3: Pilot
1.	Target audience exploration: piloting the intervention	Readiness to launch: Has the intervention	Ongoing
2.	Secondary audience exploration: ensuring relevance and appropriateness	been adequately vetted and refined to launch it at a National scale?
3.	Beneficiary exploration: ensuring relevance and appropriateness
4.	Content and web development
Phase 4: Evaluation and Refinement
Specifics to be determined		N/A	Not yet started

Throughout the development process, emerging findings have been used to iteratively develop the content, structural design, and communication strategy aspects. These efforts should be seen as parts of an ongoing evaluation study for the purpose of program planning, design, and development. All decisions on methodology have been utilization-focused, with keen attention to validity and rigor. No data have been collected that did not have a clearly identified development purpose.

2.1 Phase 1: Inquiry

2.1.1 Internal exploration

The internal exploration process resulted in five foundational project assumptions:

Employment service professionals are supposed to be the bridge between job seekers and employers;

The bridge is in need of repair, especially as it pertains to job seekers with disabilities – change toward more effective partnerships between employment service professionals and employers is necessary and will lead to better employment opportunities for individuals with disabilities;

For change to happen, we must understand that the three stakeholder groups – job seekers with disabilities, employers, and employment service professionals – act within different cultural contexts with distinct perspectives related to Language, Aim, and Discourse (Table 3: adapted from Rudstam et al., 2016);

Opportunities emerge from finding and acting upon the shared spaces between the three perspectives; and

Change is possible.

Table 3
The cultural contexts and distinct perspectives of the Diversity Partners stakeholders

Have a Language Aim to Act within a

Framework of Discourse of

Employment service professionals …program building …make placements/secure employment opportunities …service

Employers …value proposition …make a profit and/or meet organizational goals …performance

Job seekers with disabilities …equal treatment …make a living …justice

	Have a Language	Aim to	Act within a
Employment service professionals	…program building	…make placements/secure employment opportunities	…service
Employers	…value proposition	…make a profit and/or meet organizational goals	…performance
Job seekers with disabilities	…equal treatment	…make a living	…justice

In order to develop a powerful intervention that would both leverage and increase existing commonalities and opportunities, the project team needed to engage each stakeholder group in ways that would allow us to identify the “shared spaces” between them and also understand areas of discrepancy and/or flawed assumptions. We established the following research questions as an overarching framework for our exploration and development efforts:

How do we, as an external agency, create ongoing engagement in local ESP organizations around our project activities to spur and support organizational change?

What competencies (knowledge, skills, and behaviors) will lead to ESPs being successful in creating sustainable and mutually beneficial relationships with employers?

What are the situations, scenarios, and dilemmas that ESPs face when trying to build these relationships, and what language do they use to describe them?

While Question 2 captures the nature of the situated knowledge that we as a team needed to solidify through our stakeholder explorations, Questions 1 and 3 get at the need to effectively communicate that knowledge to our target audience (i.e. employment service professionals). Effective communication, Morgan, Fischhoff, Bostrom, and Atman (2002) argue, “must focus on the things that people need to know but do not already” (p.19). Accordingly, to build an effective intervention, we needed to develop a good understanding of both perceived and actual knowledge in the field in order to determine the gaps that needed to be filled. We also needed to learn the specific vocabulary that would resonate with our diverse audience, and how to frame our message in the most appealing way. The best way to do this, we decided, was to engage in a parallel process of stakeholder exploration and literature review.

2.1.2 Stakeholder exploration: Advisory Panel

In the first stakeholder exploration activity we engaged a number of representatives from each of our stakeholder groups in dialogue to further explore assumptions and learning needs based on the results from our internal exploration process. This was done by convening an Advisory Panel (AP) with national representation from employment services organizations (disability, workforce development, staffing), employers, and self-advocacy groups, i.e. a combination of target audience, secondary audience, and beneficiary representation. The overarching role of the Panel, which will meet on a quarterly basis throughout the five-year grant period, is to review progress, provide critical feedback, and help inform decisions – and to ensure that the knowledge exchange is situated within a framework of continuity and dialogue.

In the first AP meeting – an intensive and purposefully facilitated two-day in-person work session – we engaged members in prompted self-reflection around specified topics, followed by small group discussion and large group conversation. This allowed us to surface and address assumptions through spirited dialogue in a safe environment around topics determined to be critical to informing next steps: Barriers and Carriers; Assumptions about “the Other”; Relationships; Policies: Incentives and Implications; and Culture Change. The AP discussions affirmed the topics and issues that we had identified as important, but also revealed “tension areas” of which we had not been aware. As a result of the meeting we were able to develop protocols for further stakeholder exploration that both pinpointed the necessary areas of inquiry and made sense in the context of the audience.

2.1.3 Target audience exploration: Understanding mental models through qualitative inquiry

The results of the initial exploration processes allowed us to specify a framework for additional knowledge we needed in order to begin intervention development in earnest. This framework, depicted in some detail in Fig. 3, laid the foundation for the next step in our stakeholder exploration: mental model interviews as part of a qualitative inquiry with representatives from our target audience.

Fig.3

Knowledge-need framework for the qualitative inquiry: “How do people in the employment service professional field learn, how do they work with job seekers with disabilities, and how do they work with employers?”

A mental model is “a set of thoughts and beliefs that a person holds about how something works–or, more expansively, about how something is. Even more basically, mental models are representations in the mind of real or imagined conditions or situations—our model reality” (Cone, 2011, p. 3, italics added). By engaging employment service professionals in “mental model interviews,” we aimed to understand the field-based context of their work, their perceptions and belief structures around certain situations and phenomena, and the language they used to describe them. Knowledge of these elements enabled us to situate the content in the lived experiences of our target audience and understand how to effectively communicate that content. The diversity in our target audience’s work scope (disability services, workforce development, and staffing/contingent labor) translated into a diversity of knowledge, behaviors, attitudes, and language. A total of 30 in-person interviews were conducted with a mix of organizational leaders and frontline staff. All participants were interviewed by the same interviewer. Participants were offered a $50 gift card in compensation for time and effort.

The study design drew upon the mental model approaches used by Morgan et al. (2002) and Cone (2011). As such, the semi-structured mental model interview followed a funnel design, starting with a broad, non-directional question and then narrowing down to more specific questions. The interview protocol was developed to elicit information related to the three research questions and, specifically, the construct shown in Fig. 3. As a final activity, the interviewee was invited to participate in what we referred to as a “word association game.” This activity was heavily directed and consisted of a list of 20 words directly related to the practices and policies involved in the employment of people with disabilities. The words were read out loud by the interviewer, and the participant was prompted to respond with the first word/words/sentences that came to mind (if the word triggered no response, the participant was prompted to respond “nothing”). One of the hurdles to getting accurate information from the field, is to overcome social desirability bias in participant responses (van de Mortel, 2008). The word association exercise, which took place after an approximately hour-long conversation during which the interviewer had built trust with the interviewee, allowed us to gauge awareness and knowledge in a non-threatening way through an approach that did not lend itself to obfuscating or hiding knowledge gaps.

Initial analysis of the interviews followed the first five steps in Mayring’s (2000) six-step process of inductive qualitative content analysis as described in Cho & Lee (2014). Four overarching categories were identified: Relationships with Employers and Businesses; Services to Job Seekers, Job Holders, and Employers; Professional Development and Organizational Change; and Personal Stories/Narratives. Each category was further broken down according to sub-themes, which allowed us to surface both distinctions and commonalities between the different audience segments related to current knowledge and practice, learning needs, perceived barriers, and language on a detailed level. The sixth and final step of the analysis process, interpretation of results, was conducted as a participatory team activity.

The word association game was coded separately and with the pre-defined purpose of answering three key questions: Was the interviewee able to make any association to the word; what was the interviewee’s association in line with the definitions used by the project team; and what was the association? By gauging the reactions and responses triggered by each game-word, we were able to tease out areas of confusion and, as a result, could make field-informed decisions related to key language and learning needs.

2.1.4 Literature review

Parallel with the stakeholder exploration efforts described above was the process of literature exploration for the purpose of situating the emerging intervention in current research and practice, and providing evidence in support of best practices in cultivating relationships between employment service professionals and employers. The exploration encompassed both peer reviewed journals and “grey” literature, as well as promising practices from the field, and included academic journals from all of the identified employment service professional fields, NARIC archives, and research institute publications. It was done through a multi-database, multi parameter search. Utilized databases included Google Scholar, ERIC, EBSCO, and ProQuest. Although little empirical evidence to support the effectiveness of specific relationship building practices was discovered, Corbiere, Brouwers, Lanctot, and van Weeghel (2013) found that practitioners’ competencies around developing and maintaining “relationships with employers and supervisors” were most positively associated with positive vocational outcomes for those practitioners’ clients. Further, most of the literature presents a strong descriptive case for tying job development activities to regional economic initiatives and industry clusters (King, 2011); being knowledgeable of and responsive to the everyday reality and organizational culture of employers (Gustafsson, Peralta, & Danermark, 2013; Henry, Petkauskos, Stanislawzyk, & Vogt 2014; Stensrud, 2008); and implementing of demand-driven strategies to create sustained and productive relationships with employers (Unger, 2007). These findings would be important factors in determining how best to structure the intervention.

2.1.5 Decision point: Content, structure, and further knowledge needs

At the end of the Exploration Phase, we needed to make decisions related to both the content and the structure of the intervention. The basic (but not simple) question we needed to answer was: What content do we need to present, to whom, and how?

To grasp the full scope of the collected data, we engaged in a process of synergetic interpretation and meaning-making. First, each team member carefully reviewed the AP meeting transcripts, the coded target audience interviews, and the results of the literature review. This was followed by a procedure of team-based affinity mapping, in which we made sense of the data and explored our research questions by collaboratively identifying key findings, grouping them into categories, and developing themes and sub-themes.

Through this meaning-making process we identified stratified learning needs across our diverse target audience and made decisions on tentative structures for the online intervention. We decided on a topical organization of content presented to two user groups: organizational leaders and frontline staff. The word association findings and the AP meeting conversations allowed us to make informed decisions about appropriate language use. We ensured that key decisions resonated with our stakeholder groups by reviewing them with the Advisory Panel during our quarterly meetings.

2.2 Phase 2: Development

The Inquiry Phase added sufficient knowledge for us to initiate the development work. However, it also surfaced areas in need of further investigation. Hence, while Phase 2 entailed an intense period of content and web development, we also continued to engage in exploration efforts as required by our commitment to KT.

2.2.1 Web development

Based on the exploration findings around learning needs and preferred modalities of our target audience, we began by developing a “Toolbox for Frontline Staff.” We created an effective structural template for what would be the core content of the Toolbox; a number of learning modules organized into topical areas, where each module contains the same basic elements: learning objectives, a diagnostic test, plain language learning content, case scenarios, tools and resources, glossary, and an end-of-module quiz.

2.2.2 Content development

The content used to populate the modules was extensively informed by findings from the qualitative exploration processes, and the case scenarios are built off of real-life stories from the field as gathered through the mental model interviews.

2.2.3 Content exploration

Throughout the content development, we continued to explore alternate information sources and emerging research in the field (web resources, journal articles, etc.) to ensure that the intervention is built on a continuously relevant and evidence-based foundation.

2.2.4 Target audience exploration: Web site user feedback

In June 2016, we performed user testing with a beta version of the Toolbox for Frontline Staff with a limited number of self-selected employment service professionals at the National APSE conference. By inviting critical feedback from a key subset of our target audience at a relatively early stage of development, we were able to make informed decisions about revisions of some key aspects of the website. These changes have greatly benefited our work and increased the accessibility and usability of the site. In addition to providing us with specifically design-related input, the APSE testers gave us invaluable input on resource and learning needs.

2.2.5 Target audience and secondary audience exploration: ESP and employer surveys

To confirm findings from the mental model interviews and to fill in persisting knowledge gaps, we continued the qualitative inquiry study in Phase 2 by administering a brief online survey to ESPs nationally. Survey participation was solicited through trade organizations. To round out the inquiry, a second survey was then administered to employers. This employer survey, which is currently in distribution, is a “sister survey” to the ESP survey in that it contains some of the same elements. A goal of the two surveys is to make possible comparison between some of the results from these two stakeholder groups. Table 4 shows a modified sample of survey questions that will allow for such comparisons.

Table 4
Questions in the ESP and employer surveys that will allow for interesting comparison

Stakeholder Group Questions (multiple-choice)

Employment service professionals •Which services do you currently provide to employers?

•How confident are you in your ability to provide these services?

Employers •Which services do you currently receive from employment service professionals?

•How interested are you in receiving these services from an employment service professional?

Stakeholder Group	Questions (multiple-choice)
Employment service professionals	•Which services do you currently provide to employers?
	•How confident are you in your ability to provide these services?
Employers	•Which services do you currently receive from employment service professionals?
	•How interested are you in receiving these services from an employment service professional?

Once fully analyzed, survey results will allow the team to refine the Toolbox content based on additional learning, speak about findings in quantifiable terms, and infuse “calls to action” into the modules.

2.2.6 Decision point: Planning for the pilot

At the end of Phase 1, we decided on the presentation of content to two distinct user groups (frontline staff and organizational leaders), and began development of the Toolbox for Frontline Staff. At that time, we also outlined a tentative framework for needed leadership resources. The purpose of the leadership resources is to support implementation of the information and strategies in the Toolbox for Frontline Staff, and help build the needed capacity for ESP organizations to engender the change necessary to create sustainable business relationships that are inclusive of people with disabilities. At the end of Phase 2, we were able, based on our work with the Toolbox for Frontline Staff, to solidify the leadership resource framework as well as build a structure for the pilot process.

2.3 Phase 3: Pilot

The Pilot Phase is currently ongoing. Once completed, it will have involved six ESP organizations that are representative of disability services, workforce development, and staffing companies. The purpose of the pilot process is two-fold: 1. To develop a structure for leadership resources that can be transitioned into to an online Toolbox for Leadership, and 2. To refine the Toolbox for Frontline Staff into a launch-ready product through feedback from pilot site participants.

2.3.1 Target audience exploration: Piloting the intervention

With each participating ESP, the process consists of four stages: 1. A needs assessment with select leadership representatives from the client organization (conducted over phone or via WebEx), during which expectations are clarified and current needs with regard to business relationships are identified; 2. A two-day face-to-face “leadership session,” during which the client is introduced to the Toolbox for Frontline Staff and engages in facilitated learning and action planning to address the needs identified in the needs assessment; 3. A period of toolbox testing, during which select leaders and frontline staff access and provide feedback on the Toolbox for Frontline Staff, and during which all leaders and staff are encouraged to take advantage of the on-demand technical assistance services provided by Diversity Partners staff; and 4. An evaluation period, during which specifically invited leaders and frontline staff have agreed to participate in brief phone interviews to share their experiences with the Diversity Partners intervention.

2.3.2 Secondary audience exploration: Ensuring relevance and appropriateness

Continued refinement of the different aspects of the intervention is also ensured through feedback from our secondary audience (i.e. employers) via engagement of an Employer Review Panel (ERP). The membership of this panel consists of national representatives of employers of various sizes and reach. The primary role of the ERP is to review the content of the Toolbox for Frontline Staff and make certain that, where appropriate and necessary, it is representative of employer contexts and needs. ERP feedback will be collected through use of a specifically developed protocol and will focus on factual content as well as general tone and language. The ERP also provided critical input to the development of the employer survey (Phase 2).

2.3.3 Beneficiary exploration: Ensuring relevance and appropriateness

Another critical process during the Pilot Phase will be to gather input from the learning intervention’s beneficiaries (i.e. job seekers with disabilities). As with the secondary audience, the focus will be on content as well as general tone and language. It is critical that the Diversity Partners intervention accurately reflects the experience, contexts, and needs of job seekers with disabilities. The specific process for this exploration is still in development with guidance from the Advisory Panel.

2.3.4 Content and web development

As discussed, the information gathered in the various stakeholder exploration processes of the Pilot Phase has and will continue to inform the content and web development through an ongoing feedback loop. Specifically, stakeholder input will be used to refine the Toolbox for Frontline Staff, refine the content for the Toolbox for Leadership, and develop a functional web structure for the Toolbox for Leadership.

2.3.5 Decision point: Readiness to launch

Upon completion of the six pilots, we will determine the needs for additional testing and refinement of the intervention to ensure that it has been adequately vetted before we launch it as a no-cost learning resource to the employment services community.

2.4 Phase 4: Evaluation and Refinement

From a development perspective, once the Diversity Partners intervention is launched, it will enter the evaluation and refinement phase. Importantly, the degree to which it is successfully facilitating knowledge translation among its intended users will be a key evaluation priority. While specific evaluation activities have yet to be developed, they are expected to include, but not be limited to, the collection of user feedback through embedded surveys and online focus groups. Evaluation data will inform the refinement of the different parts of the intervention through an ongoing feedback loop.

3 Implications and conclusions

The challenges associated with KT have been evident throughout the Diversity Partners development process and the project team’s extensive stakeholder engagement. KT is difficult to achieve in the differing contexts of segmented audiences, each with the potential to improve opportunities for job seekers with disabilities within their own unique (and often ‘siloed’) systems. When it comes to employment for people with disabilities, KT results may vary according to the skills and experience of the people using the information, differing priorities and mandates among jobseeker-serving entities, and the labor market opportunities within each community.

KT requires that the project team be willing to challenge their own strongly held assumptions, to move swiftly and repeatedly between inquiry and development, and to openly engage with potential stakeholders who will ultimately use (or choose not to use) the resulting intervention. Inconvenient truths cannot be brushed aside, they must be heard and adapted to in order to ensure the relevance of the product developed. Engagement with stakeholders spurred evolution of the Diversity Partners project from knowledge translation to knowledge exchange. Each collaboration with stakeholders throughout our investigation and development efforts added value to both the project team and the stakeholder involved, as we actively engaged in both teaching and learning from each other.

The ongoing knowledge translation/exchange effort has entailed rapid flux between knowledge creation and use, requiring a strong team involved in the process of speeding research/evaluation into actionable information for the field. The Diversity Partners team represents diverse backgrounds and possesses expertise (academic as well as practical) in the fields of employment and disability, project evaluation, web-based instructional design, and technical assistance on complex issues involving multiple systems, policies, and ever-shifting policy mandates. The project Advisory Panel augments the expertise of project staff and faculty, with additional professional knowledge and perspectives to enrich the content and context of the intervention. Despite the proficiency of those involved in the development, the iterative and utilization-focused nature of the effort required constant vigilance to ovoid over-saturation of input from stakeholders. We could only afford to collect information that we knew we would act upon, given the time and resource constraints of content development. It was especially important to be cognizant of this limitation as we entered the pilot phase, during which the collection and incorporation of user feedback and the testing of the intervention became an intertwined and demanding process, yielding an overwhelming amount of direct information from the field.

The pilot process has surfaced many tacit barriers to effective practice regarding business engagement and supports to job seekers with disabilities, as well as some promising practices. These gaps and strengths vary by user context, but have so far fallen into categories that were largely predictable at the outset: workforce development and staffing professionals have little knowledge of disability inclusion, effective practices and civil rights protections; whereas disability service providers are not actively engaging in job-driven, industry-driven partnerships and coalitions. Not surprisingly, lack of time and uncertainty around change initiatives ranked high among barriers to improving practices in these areas. Although the long term outcomes and utility of the Diversity Partners intervention are still unclear, there are some early indications of progress within the pilot sites. For example, a disability service pilot site has created a business-focused inter-agency collaboration group as a result of what they learned in the two-day leadership session, and a workforce development region created an action plan (including timelines and assigned personnel) for fully implementing the non-discrimination provisions of WIOA Section 188, during their session. These and other actions taken by the pilot sites to date indicate a willingness to improve business relationships and disability-related practices when given the right information, delivered in the right format, and with access to the right resources.

Although outlined in linear fashion for the purposes of this paper, the phases of development described have not existed as clearly separate from one another. Instead, the exploration, engagement and refinement has been iterative, continuous and ongoing. The modules in the Toolbox for Frontline Staff are being populated with content at the same time as the pilot process is underway, with new content added as knowledge needs are revealed. The Toolbox for Leadership is developing directly from the dialogue and action planning efforts of the pilot sites. Although KT (as it happens in the reality of low-resource, high-reaching, quick turn-around grant-funded projects) is never easy, the result will be a product that is squarely situated in the world of its users, increasing its relevance. Critically, in the Evaluation and Refinement Phase, we will investigate whether the content developed through this KT process leads to enhanced application of the information presented. Our hope is that the intervention will lead to changes in the way that ESPs approach employment of people with disabilities, and encourage a focus on organizational strategy that creates space for new learning and implementation.

It is still too early to come to any real conclusions about the effectiveness of the Diversity Partners KT efforts. While emerging data from the pilot intervention indicate the potential of the intervention to encourage the use of promising practices, additional empirical evidence is necessary to determine the impact of specific relationship-building activities on behalf of job seekers with disabilities. It will also be necessary, especially in light of changing regulations and increasing access to the public workforce development system, to measure the impact of improved disability expertise within staffing firms and public workforce system personnel on outcomes for job seekers within these systems. Without the benefit of collecting outcome data at the individual level among users of this intervention, it will not be possible to isolate specific practices for the purposes of creating evidence. However, both the existing literature, our own qualitative inquiry, and active field engagement suggest that the content and format of the Diversity Partners Project is a step in the right direction.

Conflict of interest

The authors have no conflict of interest to report.

References

Bates-Harris

(2012). Segregated and exploited: The failure of the disability service system to provide quality work. Journal of Vocational Rehabilitation, 36, 39–64. doi: 10.3233/JVR-2012-0581

Bogenschutz

M. D.

, Hewitt

, Nord

, & Hepperlen

(2014). Direct support workforce supporting individuals with IDD: Current wages, benefits, and stability. Developmental Disabilities, 52(5), 317–329.

Butterworth

, Winsor

, Smith

F. A.

, Migliore

, Domin

, Ciulla Timmons

, & Hall

A. C.

(2015). State Data: The national report on employment services and outcomes. Boston, MA: University of Massachusetts Boston, Institute for Community Inclusion.

Erickson

, Lee

, & von Schrader

(2016). Disability Statistics from the 2014 American Community Survey (ACS). Ithaca, NY: Cornell University Yang Tan Institute (YTI). Retrieved Oct 19, 2016 from www.disabilitystatistics.org

Cho

J. Y.

, & Lee

(2014). Reducing confusion about grounded theory and qualitative content analysis: Similarities and differences. The Qualitative Report, 19(32), 1.

Cone

, & Winters

(2011). Mental models interviewing for more-effective communication. A Primer. Oregon State University. http://seagrant.oregonstate.edu/sgpubs

Citron

, Brooks-Lane

, Crandell

, Brady

, Cooper

, & Revell

(2008). A revolution in the employment process of individuals with disabilities: Customized employment as the catalyst for system change. Journal of Vocational Rehabilitation, 28(3), 169–179.

Corbière

, Brouwers

, Lanctôt

, & van Weeghel

(2014). Employment specialist competencies for supported employment programs. Journal of Occupational Rehabilitation, 24(3), 484–497.

Golden

T. P.

, Zeitzer

, & Bruyère

S. M.

(2014). New approaches to disability in social policy: The case of the United States. In Labor and Employment Relations in a Globalized World (pp. 73–114). Springer International Publishing.

10.

Graham

I. D.

, Logan

, Harrison

M. B.

, Straus

S. E.

, Tetroe

, Caswell

, & Robinson

(2006). Lost in knowledge translation: Time for a map?Journal of Continuing Education in the Health Professions, 26(1), 13–24.

11.

Gustafsson

, Peralta

J. P.

, & Danermark

(2013). The employer’s perspective on Supported employment for people with disabilities: Successful approaches of Supported employment organizations. Journal of Vocational Rehabilitation, 38(2), 99–111.

12.

Hall

J. P.

, & Parker

(2005). One-stop career centers and job seekers with disabilities: Insights from Kansas. The Journal of Rehabilitation, 71(4), 38.

13.

Henry

, Petkauskos

, Stanislawzyk

, & Vogt

(2014). Employer-recommended strategies to increase opportunities for people with disabilities. Journal of Vocational Rehabilitation, 41(3), 237–248.

14.

Inge

K. J.

, Wehman

, Revell

, Erickson

, Butterworth

, & Gilmore

(2009). Survey results from a national survey of community rehabilitation providers holding special wage certificates. Journal of Vocational Rehabilitation, 30(2), 67–85. Retrieved from http://iospress.metapress.com/content/n659q2452146689j/

15.

King

C. T.

(2011). Integrating job opportunities for People with disabilities into State and regional economic and workforce development strategies. InBrief. The NTAR Leadership Center. New Brunswick, NJ

16.

Luecking

R. G.

, Cuozzo

, Leedy

M. J.

, & Seleznow

(2008). Universal one-stop access: Pipedream or possibility?Journal of Vocational Rehabilitation, 28(3), 181–189.

17.

Morgan

M. G.

, Fischhoff

, Bostrom

, & Atman

C. J.

(2002). Risk communication: A Mental models approach. New York: Cambridge University Press.

18.

Nazarov

Z. E.

, Golden

T. P.

, & von Schrader

(2012). Prevocational services and supported employment wages. Journal of Vocational Rehabilitation, 37(2), 119–129.

19.

Nord

, & Nye-Lengerman

(2015). The negative effects of public benefits on individual employment: A multilevel analysis of work hours. Intellectual and Developmental Disabilities, 53(4), 308–318.

20.

Riesen

, Morgan

R. L.

, & Griffin

(2015). Customized employment: A review of the literature. Journal of Vocational Rehabilitation, 43(3), 183–193.

21.

Rudstam

, Golden

T. P.

, Gower

W. S.

, Switzer

, Bruyere

, & Van Looy

(2014). Leveraging new rules to advance new opportunities: Implications of the Rehabilitation Act Section 503 new rules for employment service providers. Journal of Vocational Rehabilitation, 41(3), 193–208.

22.

Rudstam

, Gower

W. S.

, & Van Looy

(2016). A muddy river alive with shifting currents: Knowledge translation, disability, employment and organizational change. Submitted for publication.

23.

Rusch

F. R.

, & Braddock

(2004). Adult day programs versus supported employment (1988-2002): Spending and service practices of mental retardation and developmental disabilities state agencies. Research and Practice for Persons with Severe Disabilities, 29(4), 237–242.

24.

Simonsen

, Fabian

E. S.

, Buchanan

, & Luecking

R. G.

(2011). Strategies used by employment service providers in the job development process: Are they consistent with want employers want?Retrieved from US DOL, Office of Disability Employment Policy website: http://www.dol.gov/odep/ietoolkit/publications/500.pdf

25.

Sudsawad

(2007). Poster: Rehabilitation researchers’ receptivity and perceived barriers to designing intervention studies to facilitate knowledge translation. Archives of Physical Medicine and Rehabilitation, 88(10), e23–e23.

26.

Unger

(2007). Addressing employer personnel needs and improving employment training, job placement and retention for individuals with disabilities through public-private partnerships. Journal of Vocational Rehabilitation, 26(1), 39.

27.

U.S. Department of Education (2013). National Institute on Disability and Rehabilitation Research: Long-range plan for fiscal years 2013-2017. Retrieved from https://www.federalregister.gov/documents/2013/04/04/2013-07879/national-institute-on-disability-and-rehabilitation-research-long-range-plan-for-fiscal-years

28.

van de Mortel (2008). Faking it: Social desirability response bias in self-report research. Australian Journal of Advanced Nursing, 25(4), 40–48. Retrieved from http://www.ajan.com.au/ajan_25.4.html

29.

Waterhouse

, Kimberley

, Jonas

, & Glover

(2010). What would it take? Employer perspectives on employing people with a disability. Retrieved from the National Centre for Vocational Education Research website: http://www.ncver.edu.au/publications/2219.html