Abstract
Background: Predictive Evaluation (PE) uses a four-step process to predict results then designs and evaluates a training intervention accordingly. In 2012, the Sustainable Management Development Program (SMDP) at the Centers for Disease Control and Prevention used PE to train Stop Transmission of Polio (STOP) program volunteers. Methods: Stakeholders defined specific beliefs and practices that volunteers should demonstrate. These predictions and adult learning practices were used to design a curriculum to train four cohorts. At the end of each workshop, volunteers completed a beliefs survey and wrote goals for intended actions. The goals were analyzed for acceptability based on four PE criteria. The percentage of acceptable goals and the beliefs survey results were used to define the quality of the workshop. A postassignment adoption evaluation was conducted for two cohorts, using an online survey and telephone or in-person structured interviews. The results were compared with the end of workshop findings. Results: The percentage of acceptable goals across the four cohorts ranged from 49% to 85%. In the adoption evaluation of two cohorts, 88% and 94% of respondents reported achieving or making significant progress toward their goal. A comparison of beliefs survey responses across the four cohorts indicated consistencies in beliefs that aligned with stakeholders’ predictions. Conclusions: Goal statements that participants write at the end of a workshop provide data to evaluate training quality. Beliefs surveys surface attitudes that could help or hinder workplace performance. The PE approach provides an innovative framework for health worker training and evaluation that emphasizes performance.
Introduction
Since 1958, the four-step model for evaluating training devised by Donald Kirkpatrick—(1) satisfaction, (2) knowledge transfer, (3) application, and (4) impact—has been the most widely used approach to training evaluation (Chief Learning Officer [CLO], 2009). Despite its position as the “gold standard” for training evaluation, all four steps are rarely implemented. In a 2007 survey, 92% of respondents said they measure at least Step 1 of the Kirkpatrick model, satisfaction. Measurement drops off dramatically with each subsequent step (American Society for Training and Development [ASTD], 2009). Furthermore, research has shown challenges with correlating Steps 1 and 2 to workplace performance (Alliger, Tannenbaum, Bennett, Traver, & Shotland, 1997). Decision makers thus obtain little or no information about the impact of the training on the problem it was supposed to solve. In 2012, the Sustainable Management Development Program (SMDP), at the Centers for Disease Control and Prevention (CDC), piloted Predictive Evaluation (PE) to develop and implement management training for international polio eradication workers.
The Predictive Evaluation Model
The PE model holds the training function to the same standards of accountability as any function in an enterprise for achieving organizational goals (Basarab, 2011). It uses a form of business forecasting to estimate what training will deliver in terms of increased organizational results. Each of the four key PE steps—prediction, intention, adoption, and impact—is summarized below and illustrated in Figure 1.
Predictive Evaluation Model.
Prediction. The training developer works with stakeholders to forecast the key beliefs and behaviors participants should adopt in order to improve performance. Beliefs are those attitudes that learners should hold to support the desired behavior. For example, the belief “timely project completion benefits a population’s health” could support the behavior of developing a detailed project schedule. A value can be assigned to the predicted behavior, such as a dollar value or a quantifiable outcome. The training developer uses the performance gap and predicted beliefs and behaviors to design and develop a training program (Basarab, 2011).
Intention. The Intention step is administered at the completion of the training event. Participants develop goals for intended actions based on what they have learned and connect their goals to the course topics. These goals are considered to be acceptable if they meet four criteria: specific, observable, directly related to the training, and significant (i.e., they should express a real difference in the work that is being done). Participants also describe the impact they expect this goal to achieve. Continuing with the project scheduling example, a participant may write a goal statement such as “I will develop a detailed Gantt chart that lists all project tasks and their dependencies. The impact I expect is a realistic schedule.” By considering one or two specific goals for intended actions in a thoughtful way, participants are envisioning the immediate application of new knowledge in their own work environment. The percentage of acceptable goals and the topics to which they refer can be used as indicators to evaluate the training.
The intention step also includes a survey of the beliefs or attitudes which the stakeholders had predicted the training to change. A workshop’s quality is defined by comparing participants’ goal statements and their responses to the beliefs survey to the stakeholders’ predictions (Basarab, 2011).
Adoption. After enough time has elapsed to enable participants to apply their new skills, changes in beliefs and on-the-job behaviors can be analyzed and compared with the goals that had been written during the intention step. The trainer can determine the percentage of goals that were completed and compare the topics that were deemed to be most useful during the intention evaluation with those that were actually used on the job (Basarab, 2011).
Impact. In this step, the stakeholders use the desired outcome that was identified during the prediction step to calculate the value of the training. The cost of training can be compared with the value received to calculate return on investment (Basarab, 2011).
Training the Polio Eradication Workforce
Poliovirus is a highly infectious virus spread primarily through the fecal-oral route. The virus attacks the nervous system and, in approximately 1 in 200 cases, causes irreversible paralysis (poliomyelitis) in a matter of hours (World Health Organization [WHO], 2017). Polio can strike at any age, but it mainly affects children younger than 5 years. There is no cure for polio, but there are safe and effective vaccines available to protect children and stop transmission (WHO, n.d.).
The global strategy to eradicate polio is to fully immunize every child until transmission stops, and the world is polio-free (WHO, 2017). Through long-term intensive efforts by a coalition of governments and partners, particularly the WHO, UNICEF, Rotary International, CDC, and the Bill & Melinda Gates Foundation, by 2016 polio was eliminated from all but three countries—Afghanistan, Nigeria, and Pakistan (The Global Polio Eradication Initiative [GPEI], 2017). Because poliovirus is highly infectious and risks of outbreaks continue, nonendemic countries continue to conduct disease surveillance, vaccination campaigns, and routine childhood immunizations to be sure poliovirus is not reintroduced.
Since 1988, CDC has supported the global effort to eradicate polio by providing technical assistance, funding, laboratory resources, and conducting research. CDC also trains volunteers participating in the Stop Transmission of Polio (STOP) program (CDC, 2013). Since 1999, STOP volunteers receive 2 weeks of technical training and are then deployed to endemic or at-risk countries where they provide support to the local staff for 3 to 6 months.
In 2012, the World Health Assembly declared achievement of polio eradication a global health emergency because eradication efforts were foundering (GPEI, 2012). The Independent Monitoring Board (IMB), which provides oversight to the Global Polio Eradication Initiative (GPEI, 2011), released a report in October 2011 stating pervasive poor management threatened the goal of polio eradication. In response to the IMB report, CDC designed and provided supplemental training in management to a subset of STOP volunteers who were destined for the most challenging countries. These volunteers are likely to find themselves in environments filled with operational challenges, such as planning and carrying out vaccination campaigns and leading teams of vaccinators.
Method
Use of the Prediction Step for Workshop and Evaluation Design
We used the PE model to develop STOP volunteer training because this approach supports the recommendation that training evaluation use precise behavioral measures reflecting the intended learning outcomes (Salas, Tannenbaum, Kraiger, & Smith-Jentsch, 2012). During the Prediction step, the STOP program stakeholders anticipated that the locations with a STOP volunteer would see improved surveillance and higher vaccination coverage achieved during immunization campaigns. The stakeholders also expected that STOP volunteers would become role models to local health management staff by demonstrating beliefs and behaviors that supported the polio vaccination teams. Based on the stakeholders’ predictions and the STOP volunteers’ terms of reference, the SMDP team designed a 4-day workshop called Management and Accountability Training for STOP, or MSTOP.
Implementation and Evaluation of the Intention Step
We launched MSTOP in June 2012, in conjunction with the training of the 40th STOP volunteer cohort. As part of the training materials, we designed a worksheet for the goal statements and a beliefs survey (Figure 2).
Sample goal/belief worksheet (portion).
We introduced the concept of developing goals for intended actions on the first day of the workshop. On the final day, the volunteers wrote one or two goals related to the operational aspects of their STOP assignment and indicated which workshop topics related to the goal(s) they had written. The volunteers also completed the beliefs survey.
We evaluated the Intention step with four MSTOP volunteer cohorts (40, 41, 42, 43). Two project staff independently analyzed if the volunteers’ goals were acceptable using the four PE criteria: specific, observable, significant, and directly related to the workshop material. The staff reconciled differences in scoring before calculating the final results. The responses to the beliefs surveys were summarized and analyzed using MS Excel™. During debriefs conducted after the workshop, staff shared lessons learned and planned improvements for training the subsequent MSTOP cohort.
Evaluation of the Adoption Step
Working with STOP program stakeholders we developed a logic model based on the CDC (1999) Framework for Program Evaluation in Public Health and used it to frame our evaluation of the Adoption step. The evaluation had four objectives: (1) determine progress on volunteers’ goals for intended action and changes in belief during their field assignments, (2) identify factors that promoted or hindered progress toward the goals for intended action, (3) whether and how MSTOP training had been useful during their assignment, and (4) whether stakeholder predictions that the MSTOP volunteers would become role models to the local health management staff were realized.
All MSTOP 40 and 41 volunteers were asked to complete an online survey and participate in either a telephone or in-person structured interview schedule. The same interview schedule and questions were used for the telephone and in-person interviews. Participants were assured that their name or other identifying information would not be associated with their responses and that the results would be used to improve MSTOP. Data were collected about usefulness and application of workshop content, sharing of skills and tools while on assignment, support received during assignment, the status of their goal(s), and the enablers and obstacles to volunteers’ performance. The beliefs survey was also readministered. SurveyMonkey® was used to collect and analyze online survey data. The interviews probed further about volunteers’ job responsibilities, accomplishments, challenges, progress toward their goals, and whether and how they used what they learned. To analyze and interpret the interview data, we transcribed and coded the interviews, identified recurrent themes, and extracted salient quotes. We shared results of the evaluation with the STOP program stakeholders and the volunteers. Lessons learned and feedback were incorporated into subsequent MSTOP workshops.
Results
Use of the Prediction Step for Workshop and Evaluation Design
The MSTOP curriculum addressed the management skills volunteers would need to complete their field assignments, such as process improvement, risk management, and communication. Class activities related to specific challenges in the field. For example, STOP volunteers must frequently address problems related to house marking, the process that indicates to vaccinators the status of vaccination in a household (GPEI, 2010).
The intervention also included a simple performance support system to maintain contact with the MSTOP volunteers and to provide them with continuing education and job aids during their field assignments.
Evaluation of the Intention Step
The volunteers’ goals and responses to the beliefs survey for the intention evaluation provided three sources of data: the quality of the goal statements, the topics that were noted, and the responses to the beliefs survey.
Based on the four PE criteria, the percentage of acceptable goals across the four cohorts ranged from 49% to 85%. Figure 3 summarizes the intention results of four cohorts for June 2012 to January 2014.
Goal statement results for four MSTOP cohorts, 2012-2014.
Figure 4 shows samples of successful goals and expected outcomes.
Sample successful goal statements and expected outcomes.
Topics were ranked for usefulness based on the number of times they were cited in the acceptable goal statements. For example, for STOP 40 “Analyzing the cause of variation” was named in 21 of the 35 acceptable goals. Figure 3 lists the frequency of citations of each workshop topic for the four cohorts under discussion.
Volunteers consistently indicated a high belief that despite the volatile environment in which they would be working, planning for risks can reduce unexpected events, and a high belief that an important part of working within a team is the ability to see the potential in other people’s ideas (Figure 5). In cohorts 40 and 41, volunteers’ beliefs related to their role as leaders in their STOP assignments were the lowest scoring (5.35 and 5.32 respectively), so for cohorts 42 and 43 we added a topic on Leadership and Advocacy that emphasized sustainability and empowerment.
Mean scores of belief statement results for four MSTOP cohorts, 2012-2014.
Evaluation of the Adoption Step
The response rate for the adoption step was 59% for cohort 40 and 89% for cohort 41. 1 A total of 88% of MSTOP 40 and 94% of MSTOP 41 volunteers reported achieving or making significant progress toward their goal. The adoption rankings of usefulness of topics were consistent with the rankings of the intention evaluation (Figure 6).
Comparison of usefulness of topics at Intention and Adoption steps for two MSTOP cohorts, 2012-2013.
In STOP 40, 14 (93.3%) respondents attributed their achievement or progress to being motivated and wanting to work on their goal. In STOP 41, 15 (93.8%) of respondents attributed their achievement or progress to the MSTOP sessions and/or tools.
“Analyzing the cause of variation” was the most useful session for both STOP 40 and 41 (70% and 87.5% respectively). Among STOP 41 respondents, 14 (87.5%) found the performance support provided during their assignment to be very helpful or helpful.
We used two attributes to determine if stakeholders’ predictions of volunteers being role models occurred: volunteers’ activities to build local capacity and volunteers’ beliefs about their role as a leader. For both cohort 40 and 41 volunteers reported facilitating process improvement sessions and implementing communications planning as their most common management activities. Overall, 87.5% of STOP 41 volunteers reported discussing MSTOP sessions and/or tools with local team members. For both STOP 40 and 41 the greatest increase in belief from the intention to the adoption evaluation was the statement “I can take a leadership role during my STOP assignment.” Analysis of the remaining statements on the beliefs survey for both STOP 40 and 41 indicated either no change or an increase: no beliefs declined.
Discussion
The PE approach provided a means to conduct concrete planning with project owners and use specific predicted behavior to inform training design and implementation. The precise nature of the PE intention evaluation, compared to a traditional Kirkpatrick satisfaction survey, provided timely and detailed information for improving our training program. Based on our analysis, we concluded that “Analyzing the cause of variation” and “Strategic communication” were the most useful topics. Volunteers’ responses to the beliefs survey assisted in surfacing attitudes that could enable or hinder the volunteers to succeed in their posts. We were greatly encouraged by the similarity of results between the intention and adoption findings for STOP 40 and 41. The adoption results of 88% and 94% for STOP 40 and 41, respectively, compare favorably with research that indicates learners generally apply 62% of what they learn immediately, with a drop to 44% 6 months after training (Saks & Belcourt, 2006).
With each MSTOP cohort we made improvements to both the materials and the intention evaluation process. For example, after we observed poor intention scores in the second cohort, we added daily review sessions, along with a review sheet for volunteers to note ideas of how they would apply what they learned. We also added a review period each morning for volunteers to share their ideas and think more concretely about their impending field assignments.
We learned valuable lessons about using PE during the design, implementation, and postworkshop follow-up of the MSTOP training.
Design Lessons
Stakeholders had not previously considered the value in measuring beliefs and behaviors, only knowledge transfer. It was through conversations and probing with the stakeholders that we were able to determine the priority beliefs, behaviors, and impacts they were seeking. These conversations informed the development of the workshop content and the intention evaluation instrument.
A second design lesson is English language capability. When working with an international audience, language can be a significant challenge. Because many STOP volunteers were not native English speakers, quite a number experienced difficulty writing coherently in English. Because the PE method places high value on coherent and specific statements, we found it difficult to determine how rigorously to judge vague or confusing goal statements. For example, we noticed that respondents frequently confused the goal and the outcome, so we reviewed both components together when we applied the four criteria. Since a number of volunteers came from French-speaking countries, we translated the worksheets and invited volunteers to write statements in French. Goals were analyzed in French to maintain the integrity of the writers’ intent.
Implementation Lessons
To support the intention evaluation, we introduced the concept of goal writing at the beginning of each workshop. Review sheets and discussions reinforced key points and encouraged volunteers to reflect about specific ways they could apply what they learned.
We were largely reliant on volunteers’ self-reporting to collect adoption evaluation data. However, volunteers’ field reports supported their responses in that they included extensive references to methods and tools that had been taught. Emails from the field and anecdotes from CDC field staff also supported the volunteers’ responses.
Postworkshop Follow-up Lessons
We found the postworkshop follow-up to be just as crucial as the workshop. Not only did volunteers obtain technical assistance, but they reported feeling encouraged while working in extreme hardship.
Since our initial experience using PE with MSTOP, we have implemented PE in other CDC-related programs around the world. In Jordan in 2012, an analysis of the intention evaluation results for a workshop for national program coordinators of the region’s field epidemiology training programs (FETP) surfaced the need for additional training on supervising staff and building strong work teams.
In 2013, the results of an intention evaluation from a training workshop for FETP mentors in Ethiopia indicated significant weaknesses in one of the workshop topics. We were able to immediately improve the workshop, rather than wait several months to evaluate on-the-job behavior change. When we did conduct the adoption evaluation four months after the workshop, the results were consistent with the findings from the intention evaluation: that weak topic was the least used. We also used PE to plan a training program for mid-level managers in Zimbabwe, and then implemented intention and adoption evaluations.
In all these circumstances, we have been using the goal adoption rates and other information from the adoption evaluations as a proxy to indicate impact.
The PE method has provided us the means to hold training to a higher standard. Framing a perceived need for training in the context of predicted beliefs and behaviors enables the client to think about training with performance in mind. Training design, delivery, and evaluation can be linked with these predictions, and training staff can be accountable. With health initiatives as challenging as the eradication of polio, it is imperative that training delivers performance results.
Footnotes
Acknowledgements
The authors are grateful to their colleagues at CDC for their assistance in the design and delivery of MSTOP, particularly Sara Clements, Casey Eastman, Yinka Kerr, Amy Loy MacKenzie, and Diana Miles. Special thanks to Wyeth Lawson and Elizabeth Ziegler who supported the project as public health fellows. Finally, we are most especially grateful to the thousands of STOP volunteers who endure extreme hardship to eradicate the world from polio.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: CDC provided funding for the MSTOP trainings.