Abstract
The present study isolated the effects of frequently used staff training intervention components to increase communication between direct care staff and clinicians working on an inpatient behavioral unit. Written “protocol review” quizzes developed by clinicians were designed to assess knowledge about a patient’s behavioral protocols. Direct care staff completed these at the beginning of each day and evening shift. Clinicians were required to score and discuss these protocol reviews with direct care staff for at least 75% of shifts over a 2-week period. During baseline, only 21% of clinicians met this requirement. Completing and scoring of protocol reviews did not improve following additional in-service training (M = 15%) or following an intervention aimed at decreasing response effort combined with prompting (M = 28%). After implementing an intervention involving specified performance criterion and performance feedback, 86% of clinicians reached the established goal. Results of a component analysis suggested that the presentation of both the specified performance criterion and supporting contingencies was necessary to maintain acceptable levels of performance.
Keywords
Residential and long-term hospital programs that provide behavioral services to children and adults with intellectual disabilities face many challenges with regard to staff competency and performance. To exacerbate these challenges, entry-level staff is often expected to perform a multitude of job duties simultaneously (e.g., protocol implementation, data collection, physical management of clients) with little to no previous experience or training in applied behavior analytic principles. Increased knowledge and skills on behalf of direct care staff can positively impact client behaviors (i.e., decrease aggressive behavior; Dean, Duke, George, & Scott, 2007) and quality of life outcomes, as well as mitigate problems associated with high staff turnover (Larson, Hewitt, & Anderson, 1999). Staff training is instrumental in ensuring employees gain job-specific skills, which ultimately translates to quality patient care.
Successful staff training programs are often comprehensive and multicomponent in nature. Frequently utilized components include in-services (i.e., classroom-based training), role-playing, goal setting, incentives, monitoring, and performance feedback (Arco & duToit, 2006). While these staff training programs have been found to be successful in training or improving the performance of staff in many human service settings (e.g., Cook & Dixon, 2005; Huberman & O’Brien, 1999; Luiselli & St. Amand, 2005; Noell et al., 2005; Schepis, Reid, Ownbey, & Parsons, 2001; Shore, Iwata, Vollmer, Lerman, & Zarcone, 1995) when several components are introduced simultaneously, it is unclear which components are responsible for any observed changes in behavior. A component analysis is often essential to determine the most effective as well as the most parsimonious intervention, but few studies have focused on isolating key variables responsible for behavior change (Cook & Dixon, 2005). In addition, failure to identify a cost-effective and least intrusive intervention can result in added response effort, increased costs, lengthy training sessions, and more opportunities to experience treatment integrity problems. The purpose of the current study was to extend the research on staff training by evaluating the individual effects of commonly used intervention components in a study aimed at increasing communication among pertinent staff working in a hospital unit.
Method
Participants and Setting
A total of 34 behavioral clinicians working on an inpatient behavioral unit took part throughout the course of the study. Clinicians had either a bachelor’s or master’s degree and were predominately graduate students in applied behavior analysis, aged 22 to 31. Each clinician was assigned to one patient and worked under the direction of a university faculty-level Board Certified Behavior Analyst. Clinicians provided 3.5 hr of behavioral therapy per day during the hours of 9 a.m. to 4 p.m., Monday through Friday. The primary responsibilities of clinicians included conducting behavioral sessions (i.e., functional analyses and treatment evaluations), training direct care staff in the delivery of behavioral services, and managing behavioral data.
The inpatient unit where the study was conducted provided services 24 hr a day to 16 children and adolescents diagnosed with autism and/or intellectual disabilities, who also exhibited severe problem behavior (e.g., aggression, self-injurious behavior, property destruction). Patients typically resided on the unit for 4 to 6 months, during which time problematic behaviors were addressed through the development of function-based individualized behavior plans, and in some cases, pharmacological interventions. A staffing ratio of at least one-to-one (direct care staff-to-patient) was necessary for all patients. To meet this staffing need, over 100 direct care workers were employed. Direct care staff was assigned (using a computer program that randomly assigns staff to a subset of 8 of the 16 patients) to work with one, and in some cases, two patients across one of three shifts. At the time of the study, direct care staff included full-time (32-40 hr per week), part-time (16-32 hr per week), and intermittent employees (8 hr per week). Direct care staff was responsible for data collection, behavioral protocol implementation, patient care, and patient safety.
On the inpatient unit, individualized behavioral protocols and data collection procedures were complex and changed frequently, often multiple times per week. Therefore, at the start of each shift, a shift change meeting was held, during which time supervisors reviewed major unit- or patient-related issues and gave direct care staff an opportunity to read patient-specific protocols contained in individualized patient protocol booklets. In addition, each direct care staff person was required to complete a “protocol review” quiz at each shift change meeting. A protocol review was a short quiz created by each patient’s clinician to guide direct care staff to important information and point out significant changes to protocols (see Appendix A). Protocol reviews were intended to assess direct care staff knowledge and function as stimuli to occasion interactions between clinicians and direct care staff that would ensure correct protocol implementation, aid in understanding the rationale of each protocol, and facilitate training opportunities. Following shift change, the behavioral clinician approached the direct care staff member assigned to his or her patient and scored the review with the direct care staff member present. After clinicians scored the review, they were expected to explain any incorrect answers, disseminate any additional pertinent information (e.g., modifications to protocols or data collection, or disturbances in mood), sign the form (along with the direct care staff member), and return it to a folder on the unit.
Dependent Variable
For the current study, scored and signed protocol reviews constituted a permanent product, which was used to determine clinician compliance with the instruction to score and review the protocol review with the direct care staff at the start of each shift. Supervisors recorded which clinicians had turned in a signed and scored protocol review for their patients. These data were summarized once every 2 weeks. Individual percentages were calculated for each clinician based on the total number of reviews completed during a 2-week period divided by the total number of opportunities to complete a review. During a typical 2-week period, clinicians were expected to complete 20 reviews (i.e., 2 per day, 5 days a week). The dependent variable of interest was the percentage of clinicians who completed the established criterion (75%) of protocol reviews across day and evening shifts. This criterion was set forth as the minimum level of acceptable completion by the program director. The percentage of clinicians who met the 75% review criterion was calculated by dividing total number of clinicians who achieved the 75% criterion for that 2-week interval by the total number of clinicians working during that time. On average, 15 clinicians (range 14-17) participated in the study each day and not all participants were necessarily exposed to each condition. This was due to the extended length of the study (over 100 weeks), staff turnover, and the time frame for which a clinician was assigned a patient.
Interobserver agreement
For each shift, 1 of 12 staff supervisors served as the primary data collector. A second observer (primary author) collected data on 28% of the total completed protocol reviews to assess interobserver agreement (IOA). To obtain IOA, the number of clinicians who completed the review was tallied for each 2-week period by each observer independently. The smaller number was then divided by the larger number for each 2-week observation period and the quotient was multiplied by 100. The average of the bi-monthly intervals was calculated for each phase. Agreement coefficients ranged from 67% to 100% and averaged 96%. During some phases, such a small number of therapists completed reviews (e.g., 1-2 therapists) that minor discrepancies in IOA led to low agreement coefficients.
Social validity
A survey was administered to assess clinicians’ perceptions about performing the protocol review task with direct care staff (see Appendix B). This was an open-ended seven-item questionnaire written by the first author and administrators on the inpatient unit. All responses were anonymous and clinicians were asked to comment on several facets of the task including the acceptability, satisfaction, and effort involved to complete it.
Design and Conditions
The current study consisted of the following phases: regular staff training (A), additional in-service training (B), modified format of protocol review checklist (C), specified performance criterion and feedback (D), specified performance criterion only (E), and performance feedback only (F). A reversal design (ABCDEFD) was used to evaluate the effects of the interventions and at the same time provided the opportunity to evaluate some of the separate components of the most effective intervention.
Regular staff training (A)
Through regular training and orientation upon hire, direct care staff and clinicians were instructed to do the following for each shift: Direct care staff was expected to complete a protocol review for their assigned patient at shift change (described above), and place the completed review in a folder on the unit; and clinicians were expected to retrieve their patient’s protocol review from the folder, and discuss the answers with the direct care staff member assigned to their patient within 2 hr of shift change.
Additional in-service training (B)
As a result of a low rate of completed protocol reviews by clinicians during baseline, at the start of Week 13 a senior administrator conducted a 15-min in-service training session with the clinicians to explain the expectations and rationale for the completion of protocol reviews. The in-service training included a PowerPoint presentation outlining protocol review policies and procedures. During the training session, clinicians made suggestions to improve the current protocol review process.
Modified protocol review format (C)
Based on the feedback received during the in-service training session, the format of the protocol review was modified. Protocol reviews were decreased in length to maximize efficiency and a checklist was added to the protocol review prompting clinicians to address key topics while communicating with direct care staff. Key topics included patient safety, medical issues, protocol changes, problems or difficulties with implementation, helpful strategies, family concerns, assessment and treatment results, and overall patient progress. This modification was designed to decrease response effort associated with completing the protocol reviews while highlighting the most important information. During this phase, no rules or performance feedback were provided to participants; hence, this phase functioned as a baseline control condition to evaluate subsequent intervention components.
Specified performance criterion and feedback (D)
At the start of Week 21, the director of the program sent a detailed e-mail message to all clinicians. In this e-mail, the director (a) outlined clinicians’ past performance following the in-service and change in the protocol review format (Phases B and C), (b) established a clear criterion that each clinician was expected to meet a minimum of 75% completed protocols per 2-week period (e.g., a minimum of 15 out of 20), and (c) set forth clear consequences for failure to complete the expected rate of reviews. Failure to complete the expected rate of reviews resulted in individual performance feedback from the director, and additional disciplinary action if necessary. It was assumed that such feedback would function as a powerful consequence. Following the group e-mail described above, the director then delivered individual performance feedback via e-mail to those who had failed to meet the established criterion over a 2-week period. At Week 30, to minimize the extra work required of the director, this responsibility was transferred to a senior administrator in the same department (who worked closely with the director), and she managed all subsequent e-mails. By clearly describing the performance criterion (i.e., response requirements) and consequences for compliance and noncompliance with the established criterion, the goal was to establish the policy as a formal rule.
Specified performance criterion only (E)
The performance criterion was restated via an e-mail communication to clinicians, but no performance feedback was delivered. This condition was conducted to examine whether the history of consequences supporting the instructions would result in rule following in the absence of supporting contingencies.
Performance feedback only (F)
When performance feedback was re-introduced, there was no re-statement of the instructions. That is, the director did not send a group or individual e-mails detailing the expectations or contingencies associated with protocol review completion.
Specified performance criterion and feedback (D)
The director sent a group e-mail providing both instructions and performance feedback as described above. All subsequent e-mails on staff performance were delivered by the previously identified senior administrator.
Results
Figure 1 depicts the percentage of clinicians meeting the criterion per week. During baseline, on average, 21% (range 6%-47%) of clinicians met the 75% established criterion. Neither the additional in-service nor the modified format led to an increase in completed protocol reviews (M = 15%, range 0%-27%; and M = 28%, range 21%-33%, respectively). The highest proportion of clinicians achieved the criterion when instructions were provided and performance feedback was delivered by the director or senior administrator (M = 86%, range 63%-100%). Intervention gains were sustained for many weeks and minimal individually based feedback needed to be delivered to maintain these gains (M = 2 clinicians per 2-week period needed feedback, range 0-2).
Percentage of clinicians at or above 75% criterion for protocol review completion.
Following the removal of performance feedback (during the instructions only condition, E), a clear downward trend in the number of clinicians meeting criteria was observed (M = 78%, range 63%-94%). Clinician compliance continued to be variable (M = 76%, range 58%-94%) in the feedback-only condition, F. During the final phase when both instructions and performance feedback were re-introduced (return to D), the percentage of clinicians meeting the 75% criterion for completed protocol reviews improved and remained stable for the next 12 weeks (M = 84%, range 81%-94%).
Results from the social validity survey indicated that 50% of the clinicians thought that the reviews were time-consuming. However, 86% of the clinicians found completion of the reviews with direct care staff to be a good use of their time.
Discussion
The findings of this study have many implications for organizational settings. First, a frequent and sometimes false assumption in organizational settings is that in-services, memos, or verbal instructions are an easy, effective, and cost-efficient way to change staff behavior. This study demonstrates that when these methods were used in isolation, they did not lead to any changes in staff performance. As a group, clinicians were only able to consistently achieve the established criterion when both the instructions and performance feedback were implemented simultaneously. It is important to note that while clinicians achieved the criterion in the feedback-only and instruction-only conditions, performance varied greatly from one observation period to the next. Future studies should continue to investigate the isolated effects of instructions and feedback on behavior for optimal training recommendations.
These findings support previous studies demonstrating that a specified performance criterion and feedback (i.e., consequences) was found to be a necessary component to ensure behavior change maintained over time (e.g., Alvero, Bucklin, & Austin, 2001; Chhokar & Wallins, 1984; Pampino, MacDonald, Mullin, & Wilder, 2003). Notably, 71% (24 out of 34) of all clinicians received performance feedback at least once throughout the course of this study (Table 1). That is, for most clinicians, neither providing specific performance criterion nor feedback alone was sufficient to change staff performance. It was necessary for the director to provide both the criterion for performance and feedback to achieve the targeted goal. Perhaps one explanation for the importance of both the criterion and feedback can be attributed to the longevity (over 2 years in duration) of the study and large participant pool. In human services settings faced with high staff turnover, it may be even more crucial that management provide ongoing instructions and expectations of staff behavior to ensure important tasks are completed, as such information may be lost with staff attrition.
Individual Clinician Data: Total Number of Patients That Clinicians Worked With Throughout the Course of the Study and Number of Times Performance Feedback Had to Be Delivered.
These findings are unique in that clinicians only received the performance feedback for failure to meet the criterion, which demonstrates the role that negative reinforcement (i.e., avoidance) may play in behavior change within organizational settings. Feedback from upper management may have functioned to exert aversive control over staff behavior, or stated differently, the clinicians were able to avoid potentially aversive feedback from the program director by submitting completed protocol reviews (Malott, 1992, 2001). Although completion of protocol reviews was a primary job responsibility for clinicians, most did not do it with consistency prior to being given specific performance criterion and feedback. This could be due to multiple factors, including insufficient reinforcement to consistently complete protocol reviews, as well as competing job duties and organizational contingencies that were more powerful or more immediately related to other job functions.
Although this study did not directly compare different modalities of performance feedback (e.g., face-to-face, graphic, or electronic), results do suggest that electronic feedback can be effective. In this case, it proved to be an efficient way to deliver the performance feedback. Future studies could directly compare different methods of delivering performance feedback to determine the optimal (i.e., least effortful and maximally effective) modality. Lastly, this study provides evidence for the effectiveness of interventions that utilize techniques to establish institutionalization for maintenance of intervention gains (McSween & Matthews, 2001; Sigurdsson & Austin, 2006). Unit supervisors were involved with data collection and program directors were responsible for the development of the intervention and maintaining the delivery of consequences for over 2 years. This intervention was successful at increasing the completion of reviews and organizational system changes were introduced to ensure that improvements were sustained.
There are a number of limitations to this study. First, data were not collected on the collateral effects of protocol review completion, such as improvements in treatment integrity of protocol implementation, accurate data collection, and patient engagement. Future studies should continue to examine how improvements in protocol review completion can impact performance of direct care staff in human service settings. For instance, Huberman and O’Brien (1999) demonstrated that by targeting staff performance, patient outcomes were greatly improved. In addition, data were not collected on other forms of communication or training that may have taken place between clinicians and direct care staff. Although protocol reviews were created as a mechanism to increase communication between direct care staff and clinicians, it is likely that communication between these individuals occurred at other times. However, anecdotal evidence did suggest that following the increased completion of reviews, direct care staff knew more of the clinicians by name and reported feeling more comfortable asking protocol-related questions. Another limitation of the study is that a functional assessment was not conducted (e.g., Performance Diagnostic Checklist, or PDC; Austin, 2000), which could have yielded findings that may have better guided the development of the intervention. Instead, a component analysis was used whereby the least invasive components were introduced first, and more restrictive and time-consuming components were introduced when clinicians’ behavior failed to change.
Footnotes
Appendix A
Appendix B
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.