Using a literacy-based behavioral intervention to teach job interviewing skills to adults with intellectual disability

2.1 Participants

The participants in this study were three college students (aged 20 to 22) enrolled in an inclusive postsecondary comprehensive transition program at a university in the southeastern United States. All students were diagnosed with ID based on the Wechsler Adult Intelligence Scale – Fourth Edition (WAIS-IV) (Weschler, 2008). All students also were assessed with the Job Observation Behavior Scale (JOBS; Rosenberg & Brady, 2002) to establish their (a) Qua-lity of Performance (QoP) and (b) Type of Support (ToS) measures on the three JOBS subscales: Work Required Daily Living Activity (DLA), Work Req-uired Behavior (BEH), and Work Required Job Duties (JD). One student had a secondary diagnosis of ASD. All students provided written and verbal consent to participate in the study. The investigators received formal approval from the university’s institutional review board prior to initiating the study.

2.2 Setting

Initially, all intervention and observation sessions were conducted in-person in the College of Education building of a public university in the southeastern United States. Baseline and intervention sessions were conducted in a meeting room consisting of a small table with chairs on opposite sides. There was a one-way mirror on one wall, adjacent to an observation room. During each baseline interview, the student and the investigator designated as “employer” were present in the meeting room. The student sat across the table facing the employer, and a data collector was positioned in the observation room. During the intervention sessions the student sat next to a different investigator designated as “interventionist.”

Mid-way through the study, the 2020 COVID-19 global pandemic occurred, resulting in colleges and universities in the state transitioning to fully remote, distance learning. Because in-person contact among investigators and students was no longer safe nor av-ailable, the research team implemented a series of methodological modifications that would enable a continuation of the study. Specifically, the investi-gators modified the study to continue virtually, with future baseline and intervention sessions delivered remotely via the smartphone videoconferencing app-lication WhatsApp. During this modification, the setting for the study was individualized for each participant. Investigators implemented the study protocol from their individual home offices, and students participated from a workspace in their individual homes. During the remote portion of the study, each baseline, follow-up, and generalization session was conducted as a group video call with three people: the investigator designated as “employer”, the student, and the data collector. The intervention sessions also were conducted as a group video call with three people: the investigator designated as “interventionist”, the student, and the data collector. Thus, the setting for every session included three individuals, and was delivered remotely to the individual students at their own homes.

To minimize setting distractions (at the students’ homes and investigators’ offices), a remote interaction protocol was developed. The investigator serving as “host” of the software application set the parameters of the software so that a student could only see and hear the investigator designated as employer or interventionist when the data collector’s camera and microphone were blind to the student. This reduced the probability of the data collector serving as an unintended reactive influence on the student’s performance.

2.3 Behavioral measures, data collection, and interobserver agreement

The dependent measure for this study was the percentage of steps performed correctly and independently in a task analysis of job interviewing skills (see Table 1). Interviewing was selected because it is a complex, anxiety-producing skill and many people with ID lack key interviewing skills such as communication, self-advocacy, and articulating details about themselves (Winsor et al., 2017). Target behaviors were divided into three categories: greeting, answering four interview questions, and closing. The interview questions used in this study were selected utilizing a curriculum to teach workforce readiness skills to youth with disabilities, “Skills to Pay the Bills: Mastering Soft Skills for Workplace Success” (U.S. Department of Labor, Office of Disability Employment Policy, 2012) and the Career One Stop website (U.S. Department of Labor Employment and Training Administration, 2019).

The task analysis for the in-person, face-to-face portion of the study was comprised of 17 steps. The task analysis for virtual job interviewing, was modified to remove 5 steps that required face-to-face interaction. Specifically, steps including ‘follow the interviewer’, ‘sit where directed’, and ‘shake interviewer’s hand’ were removed. The remaining 12 steps were kept unchanged (see Table 1). Because data were analyzed as percentage of steps, this allowed us to evaluate students’ performance with this modification in the task analysis. Each step in the task analysis had agreed upon acceptable responses from the students. For example, a silent smile for the greeting was not acceptable, the students were expected to verbally greet the interviewer in a professional manner, such as “hello” or “good afternoon” (rather than with an unprofessional “what’s crackin”).

To collect data, each student was individually ob-served while performing the job interviewing skills in a simulated interview. Steps had to be performed in a strict sequential order. For example, students were required to greet the interviewer at the beginning of the interview. However, the order of the answers to a particular question did not have to follow the same order provided in the task analysis. For example, in response to the question “Tell me about yourself”, the student could give the three responses in any order. Each step in the task analysis was scored by a data collector as (a) correct or (b) incorrect. A step was recorded as correct if the student performed it as identified in the task analysis within 30 seconds. An incorrect response was recorded if student did not correctly perform a step or omitted a step in the task analysis. Only steps that were both correct and provided without assistance were included in the graphed results and used for instructional decisions.

During the in-person, face-to-face portion of the study, one data collector was stationed in the observation room. Data were collected live, and data col-lectors were trained to use the data sheets prior to the study. During the virtual sessions, the data collector was part of the video call, however, the observer’s camera and microphone were blind to the student. The data collectors were experienced researchers including a faculty member and two graduate students enrolled in doctoral programs in special education and counselor education.

To establish interobserver agreement (IOA), two data collectors recorded data independently and at the same time during 81% of all sessions. Agreement was determined by counting the steps of the task analysis scored the same by both data collectors, then dividing that number by the total number of steps observed, then multiplying by 100. These agreement sessions were conducted during 71% of baseline, 83% of intervention, and 50% of follow-up sessions for David (76% overall); 63% of baseline, 71% of intervention, and 50% of follow-up sessions for Judith (67% overall); and 57% of baseline, 100% of intervention, and 50% of follow up sessions for Sharon (79% overall). The results of the IOA assessment showed substantial agreement between the data collectors. Agreement across all students and sessions (combined) was 97%. For individual students, agreement was 98% for David, 96% for Judith, and 98% for Sharon.

2.5 Experimental design and procedures

Due to the presence of the 2020 COVID-19 pandemic during this study, we were forced to examine experimental design features that would allow us to address our research questions, while respecting the principles and traditions of behavior analytic experimentation. In addressing the inherent flexibility of single subject research, researchers from Sidman (1960) to Baer et al., (1987) to Kennedy (2005) have addressed the sine qua non of experimental design. To understand experimental effects, a study must (a) have convincing demonstrations of experimental control, (b) establish a functional relation between independent and dependent variables, (c) demonstrate an adequate number of replications, and (d) show predictability in the data patterns. Ken-nedy summarized numerous studies in which design alterations have been made to seize opportunities to investigate phenomena brought on by serendipitous and unplanned opportunities, including stepping away from “rote and inflexible” “cookbook” design features and to construct “a good design ...   that answers the question convincingly” (Baer et al., 1987; p. 319).

For this study, we implemented a combined design approach (Kennedy, 2005) to capture the opportunity to study the efficacy of the LBBI under rapidly changing conditions. A multiple probe design across participants was used to determine the effects of the LBBI on the acquisition of job interviewing skills. The design used multiple probes during baseline to prevent repeated exposure to inaccurate practice of the skills before the intervention was introduced. When the COVID-19 forced the closure of on-campus interventions, we delayed the experimental procedures for the third participant, thus implementing a non-concurrent multiple baseline design for this participant. In addition, for David and Judith, we were able to return to baseline following their initial, in-person intervention. Although neither were able to demonstrate mastery after exposure to the in-person LBBI due to the abbreviated time before the campus closure, both showed clear effects when examining the level and trend of their data. For both, we followed the in-person intervention with a return to baseline, but this time the baseline and second intervention were modified to the remote application of the LBBI, thus consisting of an A¹-B¹-A²-B² design for these participants.

Finally, we added two design features to assess maintenance and generalization. Follow-up observations were held after the intervention was removed with each student to assess whether the experimental effects and learning would maintain when the students were no longer coached to use the LBBI. Additionally, we conducted a generalization probe for each student during intervention to determine if the students would use their new interview skills with a different investigator who played the role of a novel employer.

2.6 Treatment fidelity

To evaluate the degree to which the intervention was implemented as intended, data were collected on treatment fidelity for each student. Prior to the study, a training session was conducted to teach the data collector to collect fidelity data. The training required the data collector to watch the interventionist perform the actions on each page of the LBBI (without any student present). The data collector then practiced coding the treatment fidelity data sheets, first with the investigator and then independently. The data collector and investigator then discussed the scores until no disagreements were noted.

To collect the fidelity data, the data collector used a four-item checklist during baseline and a seven-item checklist during intervention. The baseline fidelity checklist was used to ensure the LBBI was not implemented but that the cue to begin the interview was given, and the interview was terminated as planned. The intervention fidelity checklist ensured that the interventionist used the LBBI accurately and foll-owed the pause, point, practice, and praise procedure. During the face-to-face sessions, the data collector watched the live sessions from the observation room. For the virtual sessions, the data collector was part of the video call with the student and interventionist. Throughout the study, fidelity data were collected during 83% of the sessions. A review of the fidelity results indicate that the protocol was followed with 100% fidelity during each observation.

2.7 Data analysis

First, data were analyzed by the investigators using traditional visual analysis procedures. This included calculating measures central tendency and deter-mining ranges for each participant during baseline, intervention, and follow-up conditions. These measures helped us establish direction, trend, and level of data within each condition, and were used to justify condition changes (Kratochwill et al., 2013). Next, the investigators conducted a post-hoc analysis of graphed data to establish the size of the effect of data changes. A Tau-U coefficient was calculated to determine true effect size. Tau-U provides an aggregate effect size based on the weighted average of each students’ baseline and intervention changes. To determine the Tau-U coefficient, the authors used the Tau-U web-based calculator (Vannest et al., 2016). This calculation includes only baseline to intervention effect of Sharon and second baseline to second intervention effect for David and Judith.

David originally scored relatively high during the initial baseline (56% accuracy). This score was influenced by David asking the interviewer questions about the position. After the second baseline session he no longer asked the questions and stabilized at 34% correct and independent. His in-person baseline scores ranged from 34% – 56%, averaging 41% correct and independent. Once the in-person intervention was introduced, David displayed little growth until his fifth intervention session, when he had a noticeable increase from 43% to 65% accuracy. Following this, David increased dramatically to 100% correct and independent during the seventh intervention session. Before an eighth intervention session could be delivered, the COVID-19 pandemic resulted in the state and campus closure. Three weeks passed before investigators were able to re-start the study. Because there was no opportunity for in-person, on-campus contact, we shifted David’s experimental protocol to the modified LBBI delivered remotely. During David’s remote baseline, the accuracy of his interviewing regressed, but was stable at 67% correct and independent. When the intervention was re-introduced virtually, David immediately increased to 100% accuracy, and maintained this performance for five consecutive sessions, plus one generalization probe. A follow-up probe occurred 20 days after the intervention was removed and he maintained his 100% correct and independent responding.

3.2 Judith

During Judith’s initial in-person baseline, her data were stable, with a range of 21% – 26% correct and independent, and an average of 25% accuracy across the five baseline sessions. During her first three in-person intervention sessions, Judith had a slow, increasing trend. By the fourth intervention Judith increased from 39% to 60% accuracy. By her sixth intervention session she had reached 95% accuracy, at which point data collection stopped due to COVID-19. Judith’s remote baseline condition was introduced three weeks after her last face-to-face intervention. Judith’s remote baseline condition was stable at 67% for three sessions. When her remote intervention was introduced, Judith showed a modest increase in accuracy for first three sessions, followed by a substantial increase to 100% accuracy. She maintained 100% accuracy for five consecutive sessions, plus one generalization probe. A follow-up probe was conducted 21 days later, and Judith maintained 100% correct and independent responding.

3.3 Sharon

Sharon’s first baseline session did not occur until virtual baselines were under way for David and Judith. Sharon’s first virtual baseline resulted in 33% accuracy, followed by an increase in accuracy during the second and third sessions. Baseline data stabilized at 58% correct and independent for four consecutive sessions before introducing the virtual LBBI. During Sharon’s first intervention sessions, she showed an increasing trend, then decreased from 83% to 58% accuracy during her fourth and fifth intervention session. During her sixth intervention session, Sharon’s accuracy increased to 100% and remained there for five consecutive sessions, plus one generalization session. A follow-up probe 15 days after the intervention was removed, showed that Sharon responded with 100% accuracy.

3.4 Post-hoc analysis and effect size

The Tau-U post-hoc analysis showed an overall effect size of .878, suggesting a robust effect size across students (Parker et al., 2011). It should be noted that this coefficient only reports the weighted average of David and Judith’s second baseline to their second intervention, and Sharon’s baseline to intervention.

4.1 Limitations and future research

Although this study supports the use of an LBBI to teach interview skills to college students with ID, some limitations should be noted. Replication with other adult populations is necessary; our study only included adults in the same inclusive postsecondary education program, and all had similar educational backgrounds and experiences. Additionally, all three students share similar characteristics (diagnosed with ID, similar ages, and had similar requisite skills). Because most adults with ID do not have access to inclusive college programs, we recognize that students in this study are not representative of the general population of people with ID.

Second, while social responses to interview questions are a critical skill for individuals with ID, other valuable interviewing skills also affect applicants’ access to employment, and we did not explore these variables. For example, we did not address issues of proper dress, personal grooming, hygiene, or punctuality. LBBIs have shown their value in improving independent daily living skills and routines in adolescents and adults with disabilities (Brady, Hall et al., 2016; Brady, Honsberger et al., 2016). We believe LBBIs might be valuable as a strategy for enhancing these and related variables that contribute to interview efficacy.

Finally, the disruption caused by the 2020 COVID-19 pandemic had obvious effects on this research. Because of the mandated campus closure and the switch to remote learning, the LBBIs were delivered in both face-to-face and virtual formats. The speed of the switch required investigators to implement rapid changes to our research protocols. Fortunately, we were able to implement experimental procedures to address our research questions. For example, by adopting combined experimental designs (Kennedy, 2005) we were able to use the flexibility inherent in single subject research to create designs to understand experimental effects while meeting the robust expectations for behavior analytic experiments. However, there were limitations. We were unable to implement the multiple probe design as typically described, and ideally, we would have extended the in-person LBBI a few more days to assess whether David and Judith would achieve mastery of the new skill. Yet, the combined designs provide strong evidence of experimental control. By comparing baseline and intervention performance among the participants, the LBBI demonstrated a functional relation with the acquisition of the interviewing skills. Across the study, there were 8 replications of this phenomenon. Thus, the data pattern for each student demonstrated predictability of the experimental effects.

An additional limitation of the study was our in-ability to program for generalization across conditions. In this study we did not include generalization probes during baseline, upon initiation of the intervention, nor follow-up. This precludes any comparison of the students across conditions. Our observation of spontaneous use of the interviews with a novel interviewer at the end of the study is encouraging and suggests that the LBBI might be effective for promoting such generalization. This should be explored in future research and the design implemented to specifically probe for such evidence of generalization. We also anticipate that future studies will examine generalization questions involving different types of interviews, different types of employment settings, and different types of job tasks. We also anticipate the need to examine other forms of generalization such as transitioning from in-person to virtual employment scenarios.

Replications of this study should be conducted with different populations, including participants wi-th other disability characteristics, ages, race and ethnic identities, and educational experiences. Given the recent importance of remote learning, researchers need to study the impact of remote coaching and instruction on these adults, including other college students with disabilities. Finally, additional research is needed on various delivery methods of the LBBI instructional package. This includes other methods of virtual delivery, and other skills sets needed for competitive employment.