Abstract
Problem
Effects of residents’ blended learning on their clinical performance have rarely been reported. A blended learning pilot program was instituted at Southern Illinois University School of Medicine's Obstetrics and Gynecology program. One of the modules was chronic hypertension in pregnancy. We sought to evaluate if the resident blended learning was transferred to their clinical performance six months after the module.
Intervention
A review of patient charts demonstrated inadequate documentation of history, evaluation, and counseling of patients with chronic hypertension at the first prenatal visit by Obstetrics and Gynecology (OB/GYN) residents. A blended learning module on chronic hypertension in pregnancy was then provided to the residents. A retrospective chart review was then performed to assess behavioral changes in the OB/GYN residents.
Context
This intervention was carried out at the Department of Obstetrics and Gynecology, Southern Illinois University. All 16 OB/GYN residents were enrolled in this module as part of their educational curriculum. A query of all prenatal patients diagnosed with chronic hypertension presenting to the OB/GYN resident clinics four months prior to the implementation of the blended learning module (March 2015–June 2015) and six months after (July 20, 2015–February 2016) was performed. Data were collected from outpatient charts utilizing the electronic medical record. Data were abstracted from resident documentation at the first prenatal visit.
Outcome
The residents thought that the blended learning module was applicable to performance improvement in the real-world setting. Patients evaluated before (n = 10) and after (n = 7) the intervention were compared. After the intervention, there was an increase in assessment of baseline liver enzymes, referral for electrocardiogram, and early assessment for diabetes in the obese patients. More patients were provided a blood pressure cuff after the module (71.4% vs. 20%). Data were provided to the residents in an informal setting. Discussion during this session suggested that inconsistent use of the algorithm and incomplete documentation were reasons for the findings.
Lessons Learned
This study suggests that blended learning may be a viable tool to support sustained changes in the performance of OB/GYN residents. Scheduled follow-up should be employed to facilitate and ensure continued learning and behavioral changes.
Keywords
Introduction
The ultimate purpose of an educational program in professional education is to improve performance in the workplace. However, there are very few efforts to demonstrate how a training program can improve workplace performance. In their review on training transfer, Burke and Hutchins 1 demonstrated that less than half of training results in personal or institutional improvements. Training without any performance evidence can result in inappropriate use of time and resources in a busy workplace. Therefore, it is important to understand how workplace training can provide intended performance outcomes.
In their extensive literature review, Burke and Hutchins 1 described three primary factors, namely, learning characteristics, intervention design and delivery, and work environment influences, that influence the transfer of training. Training outcomes appear to be influenced by a learner's intellectual capacity, self-efficacy, intrinsic motivation, and perceived utility associated with the training. Furthermore, Burke and Hutchins 1 suggested that the design and delivery of an intervention must address multiple aspects of active learning, including the identification of learning needs, learning goals, content relevance, prominent instructional strategies and methods, self-management strategies, and instructional media for training transfer. Finally, the work environment at a system level can have a significant influence on training transfer. This includes ability to practice, accountability, peer and supervisory support, and contextual factors that may preclude or support the ability to transfer what has been learned to the job environment.
While all three training transfer elements should be considered in improving training effectiveness, intervention design and delivery focusing on active learning processes is the central design practice in developing a training program. In this paper, we describe how a resident training program adopting a blended learning approach can lead to residents’ performance improvement. Our analysis will focus on what aspect of a blended learning approach would fit for residents’ performance change.
A blended learning pilot program of obstetric complications in pregnancy was instituted at Southern Illinois University School of Medicine's Obstetrics and Gynecology residency program. 2
Blended learning is a pedagogical approach to embrace organic learning process, rejecting dualism separating classroom from real world. 3 It may appear to be a simple use of both online learning technology and offline learning activities. Beyond this technocentric view, however, blended learning provides an instrumental process to facilitate residents’ seamless interaction between cognitive reflections and actions in community of practice.2,4 Blended learning techniques are unique as they foster similar characteristics noted in training transfer literature. To effectively promote the transfer of the blended learning modules, residents must assess the causes of inadequate performance. 5 Blended learning also provides residents with multiple forms of online resources, discussion forums, and face-to-face training sessions, which include opportunities for feedback. In addition, residents have the ability to complete these modules in their own time, which might address any barriers to the transfer of knowledge that may arise from the work environment, thus enhancing intrinsic motivation. The ability to play an active role in designing and co-creating knowledge with peers results in more investment in the design of training, thus furthering long-term learning transfer. 6
Although robust in human resource development, transfer training literature is limited in the field of undergraduate and graduate medical education. In fact, a literature search using the terms “training”, “training transfer”, “transfer of learning”, “professional improvement”, “professional development”, and “medical education” resulted in one article. 7 This particular article sought to describe a conceptual framework of learning transfer in postgraduate, continuing, medical education.
Given the paucity of literature of training transfer in graduate medical education, we aim to provide evidence of effectiveness of blended learning approach for behavioral change in residents’ clinical performance. Based on outcomes of our initial study, we decided to determine if short-term improvements in knowledge after a blended learning experience would result in sustained changes in resident clinical performance by assessing knowledge transfer in the ambulatory setting.
Methods
Educational design, development, and implementation
A blended learning module on chronic hypertension in pregnancy was created, as previously described. 2 All 16 Obstetrics and Gynecology (OB/GYN) residents participated. This learning experience took place between July 17 and July 24, 2015. Shortly, instead of providing the traditional instruction in a classroom over a one-hour face-to-face session, a blended instructional module was designed to be delivered over the course of a week. A brief, 10-minute, face-to-face session introduced the module to the residents during which time goals, course content, and sequence of instruction were reviewed. They had one week to complete the online content prior to the culminating face-to-face discussion.
The web-based instruction was housed on the learning management system, Moodle. The course content began with a pre-module quiz to assess background knowledge. The module comprised three sections, as follows: (1) diagnosis and initial management of chronic hypertension in pregnancy, (2) effect of chronic hypertension on pregnancy and vice versa, and (3) management of chronic hypertension during pregnancy and postpartum. Each of these sections included a short self-guided, interactive, self-paced multimedia presentation in accordance with the cognitive theory of multimedia learning 8 and supplemental reading materials.
There was no asynchronous online discussion that was due to findings of the pilot study. Rather, an open online forum was created in which the residents were tasked with either reviewing one of the articles posted in the module or performing their own literature search. The residents were asked to write a brief take-home message from the article they reviewed and describe how that information would impact the care of their patients in the future. They were also asked to include the citation for the article as a reference for their colleagues. The online component of the module concluded with a post-module quiz. Based on feedback from the pilot study, the pre- and post-module quizzes were shorter and only comprised multiple choice questions.
The blended learning module culminated in a face-to-face session over an hour. This hour was designed to foster a community of inquiry 4 and improve learning effectiveness through exceptional communication.9,10 Specific to the social nature of the practice of medicine, cognitive and teaching presence was explicitly fostered during this portion of the instructional unit. Cognitive and teaching presence functioned to maintain the focus and rigor of the educational experience. Case studies were used as the triggering event for the inquiry phase of cognitive processing. These case studies involved real-time review of a sample of resident obstetrics’ ambulatory clinic patient charts. This review demonstrated inadequate documentation of history, evaluation, and counseling of patients with chronic hypertension at the first prenatal visit.
After presenting these data to the residents, the rest of the face-to-face session was spent creating a comprehensive management algorithm for their patients with chronic hypertension in the ambulatory setting. This process was resident led and facilitated by the senior author (Fig. 1). This was an evidence-based algorithm, grounded on lessons learned during the blended learning experience over the course of the week. The intent was that each resident would follow this algorithm during the initial prenatal visit of a patient with chronic hypertension. The goal was to enable standardized and consistent patient care.
Evaluation and management algorithm created by the residents during the culminating face-to-face session, after conclusion of the online portion of the module.
Data sources
The primary data sources were resident quiz results, article review on the open online forum, and program evaluation survey. A retrospective electronic medical record chart review was performed six months after the blended learning module to assess ongoing positive behavioral changes in the OB/GYN residents’ care of pregnant women with chronic hypertension.
Data analysis
Pre- and post-module quizzes were scored in percentages ranging from 0% to 100% (all correct) based on the number of correct answers provided. A paired t-test was used to assess the measurement of learning achieved in the time between the pre- and post-module quizzes. Results were described with measures of central tendency (mean) and dispersion (standard deviation). A P-value < 0.05 was considered significant. Article review post was reviewed based on the thoroughness of answers to the two questions (what is the take-home message and how will the article impact future patient care). The program evaluation survey was a brief survey of resident perception of the blended learning unit and was described in a descriptive fashion. Data were abstracted from patient electronic medical record charts. Categorical variables were summarized as frequencies and percentages. To test for possible differences, proportions of completion for before and after implementation of the blended learning module for each charted question, data were analyzed by Fisher's exact test. A P-value < 0.05 was considered significant.
This study was deemed as nonhuman subjects research by the ethics review board of Southern Illinois University. This research complied with the principles of the Declaration of Helsinki.
Findings
Learning outcomes
All 16 residents completed the pre-module quiz. Only 14 residents completed the post-module quiz. When evaluating quiz scores of the group as a whole, there was a statistically significant improvement in mean quiz scores after the blended learning module, 64.0% ± 13.9% vs 77.8% ± 9.9% (P = 0.015). On average, mean scores increased by approximately 14% (Table 1). There was no statistically significant difference between the pre- and post-module quiz scores when examining the data by resident year (Table 2).
Chronic hypertension (paired sample t-test). Comparison of pre-module quiz and post-module quiz scores demonstrates a significant increase in quiz scores after conclusion of the online portion of the module when comparing the group as a whole. On average, scores increased by 14%. P-value = 0.015.
Chronic hypertension (paired sample t-test). When evaluating the group according to PGY-level, there was not a statistically significant difference between the pre-module and post-module quiz scores.
Online discussion post
All 16 residents posted an article review in the open online forum. All residents answered the questions posed. Three resident posts were considerably more robust in comparison to the others. No resident performed their own literature search. The articles they reviewed were the online resources that were included as part of the blended learning content.
Program evaluation survey
Thirteen of 16 residents completed the program evaluation survey. The blended learning module was well received by the respondents. The residents thought that the blended learning module was helpful and applicable to the real-world setting.
Five residents completed the online course at home, four residents completed the online component at work, and the remaining four residents completed the online modules at both work and home. Most of those who worked on the online content at work noted that they were frequently interrupted by clinical duties. Otherwise, they tended to utilize free time or downtime while at work to perform the online tasks.
Five of the 13 respondents stated that no factors precluded their ability to complete the online component of the module. Of the remaining eight respondents, several factors precluded their ability to seamlessly complete the tasks. These were organized as learner issues and access difficulties.
The main learner issue was failing to remember to perform the assignments. Access difficulties appeared to be a problem for some who attempted to complete the online tasks at home. Specific comments about access included:
server froze,
unable to access at home,
hard getting to load at home laptop, eventually worked, and
tried several times to log at home, worked after the 4th/5th time.
Aspects of the module that the residents liked included:
links to articles,
videos,
pre/posttest,
test with immediate feedback,
tests appropriate,
shorter quiz,
concise articles,
concise videos,
extremely helpful for learning style,
quizzes outstanding due to brevity and answers,
watch videos at convenient time,
combo of learning methods, video, tests, readings, lecture, creation of algorithm, and
case presentation.
When queried about what they would change about the module, 6 of the 13 residents stated that they would not change anything. The recurring theme from the other seven residents included:
no discussion, forum discussion not helpful, rather discuss as a group,
some videos redundant, shorter videos, 5–7 minutes, 11-minute videos are too long, and
more post-quiz questions.
Retrospective chart review
A total of 17 patients with chronic hypertension had their initial prenatal appointment performed by the residents within the four months prior to and six months after the blended learning module. Patients evaluated before (n = 10) and after (n = 7) the intervention were compared (Table 3). When compared to the four months prior to the blended learning module, there was an increase in assessment of baseline liver enzymes (100% vs 50%, P = 0.044), referral for an electrocardiogram (100% vs 40%, P = 0.035), and early assessment for diabetes in obese patients (85.7% vs 17%, P = 0.029). More patients were provided a blood pressure cuff after the module (71.4% vs 20%, P = 0.058).
Retrospective chart review results (paired sample t-test). comparison of resident clinical performance variables before and after instructional module on chronic hypertension.
After the intervention, there remained no documentation of prior work-up of chronic hypertension, infrequent documentation of duration of disease, and infrequent documentation of past medical therapy of chronic hypertension. Documentation of patient counseling was also rare, both before and after the intervention. Of those with an elevated blood pressure at their prenatal visit, none had a repeat blood pressure documented. In addition, no patient was initiated on baby aspirin for preeclampsia risk reduction.
Implications
There was a statistically significant improvement in quiz scores after completion of the online module when evaluating the group as a whole. It is possible that a statistically significant difference in quiz scores was not noted when evaluating the data by postgraduate level due to the small sample size. According to resident feedback, the online format, multimedia presentations, articles, and quizzes were seen as a strength of the module. Additional benefits of the module included the ability to complete the course at one's own pace and place, over time.
An important aspect of teaching and learning involves curriculum evaluation, especially determining whether or not the instruction has resulted in improvement in clinical performance. The teaching and learning of medical topics should assess not only retention but also deep understanding, transferability, and behavioral change. In the setting of medical education, true learning and understanding requires that the student consistently applies what has been learned to the clinical setting. The residents demonstrated some performance improvement in the initial management of the patient with chronic hypertension, specifically ordering baseline laboratory studies and electrocardiogram, and screening for diabetes mellitus. Unfortunately, documentation of patient history, prior work-up, initiation of baby aspirin, and counseling were lacking. Although only three parameters were consistently improved upon in the long term, it is important to note that these were the three objective parameters that were systematically discussed in the culminating face-to-face session and incorporated into the comprehensive management algorithm.
As a result of the findings in the retrospective chart review, the data were presented to 9 of the 16 residents. The residents agreed that they do not consistently inquire about duration and prior work-up of chronic hypertension. They also commented that they needed to consistently request that the patient's blood pressure be repeated if the initial blood pressure was elevated. They verified that they routinely perform a thorough physical examination. Despite this, they recognized that if they do not document their patient encounter, it is presumed that those aspects of the patient encounter did not occur or was not performed. At the conclusion of our discussion, they indicated that their desire was to thoroughly assess every patient and document 100% of the time. In addition, the residents were surprised that no patient had been initiated on baby aspirin for preeclampsia risk reduction.
Over the course of this discussion, the residents came up with two solutions: (1) put the algorithm they created somewhere centrally where it is readily available and (2) be more systematic and consistent with patient care and documentation. Overall, they found that the follow-up session was a “good review” of what they could improve upon in future patient encounters.
The limitations of this study include the study setting and small resident sample size. Other programs with different curricula or with a different resident population may have different baseline findings and dissimilar results post intervention. In addition, only 17 patients with chronic hypertension were identified as having their initial prenatal care provided by a resident. This is because nurse practitioners and faculty also provide this care to the patient population. As we were interested in the initial resident–patient encounters, only patient charts where the resident provided the first prenatal appointment were examined.
Due to sample size limitations, we are planning a multi-institutional collaboration to determine if these findings are reproducible and to what extent. We are also interested in examining what aspects of blended learning have the greatest effect on training transfer and to what degree. Finally, we intend to investigate how to harness these findings for future instruction and curriculum design.
The main strength of this study is demonstration of some long-term effects of blended learning on resident performance in certain aspects of patient care, specifically assessment of baseline patient status. We were able to document that acquisition of knowledge resulted in specific transferable change in clinical performance.
This study suggests that blended learning may be a viable instructional process to support the learning and performance of OB/GYN residents. The results propose that a blended learning experience may produce the desired outcome of sustained transferability of new knowledge and skills to patient care. In addition, the informal discussion of our findings after the chart review suggests that scheduled follow-up should be employed to facilitate and ensure continued learning, ongoing behavioral changes, and maintenance of proficiency.
Discussion
Our blended learning module was developed to reduce the risk of residents experiencing cognitive overload. 11 By delivering the training modules online, residents could choose when and where to complete learning tasks, allowing them to control the rate and amount of information they were exposed to at any given time. The multifaceted approach to training allowed residents to take a more active role in their learning, as opposed to the passive learning that takes place in lecture-based training. This active learning approach to training is believed to maintain the adult attention span, increase learning, and decrease negative outcomes in health and safety practice, 12 further enhancing transferability.
Online forums allowed residents to generate discussion regarding each other's findings, contributing to an open “transfer climate”. Although our findings suggested that a face-to-face group discussion was more favorable than the online forum, this mode of peer support and sharing of ideas about course content could help promote skill transfer. 13 This is an area that should be further investigated in terms of why residents did not utilize online forums, how residents’ social learning process occurs in their busy clinic days, and how they exchange information and learn from one another. Understanding this would be beneficial to facilitate their social learning process for knowledge and skill transfer.
Goal setting has been found to motivate the individual to develop relevant strategies for goal attainment, which is necessary for transfer.14,15 While our residents were able to set goals during an initial face-to-face session, implementing a more involved goal setting exercise into the training module, in which residents shared goals with their peers, would likely increase transfer, accountability, and self-management strategies. By sharing their goals, residents would be able to assess each other's transfer, creating a sense of accountability within the work environment. The assessment of transfer not only makes trainees accountable for transfer success but also creates a culture that values learning and its application on the job. 16
Providing feedback, reinforcement, and remediation opportunities for learning mastery has been shown to result in significantly higher transfer scores. 17 While residents had opportunities for feedback through face-to-face sessions and discussion forums, more frequent opportunities for feedback may enhance long-term maintenance and application of skills. 18
An important consideration for future modules include organizing subsequent follow-up sessions to support continued improvement of clinical performance. In addition, a detailed process for dealing with technical problems should be designed and provided to the residents. Based on the findings of this study, future studies should involve formal design, implementation, organizational support, and comprehensive assessment of a blended learning curriculum in OB/GYN.
Author Contributions
Conceived and designed: FT. Abstracted data: AG and FT. Analyzed the data: KD. Wrote the first draft: FT. Contributed to writing: FT, AG, and HH. Agreed with manuscript results and conclusions: FT, HH, AG, and KD. Jointly developed structure and arguments: FT, AG, and HH. Made critical revisions and approved the final version: FT, AG, and HH. All the authors reviewed and approved the final manuscript.
Footnotes
Acknowledgments
We greatly thank our colleagues Ms. Kathy Groesch, RN, Dr. Paula Diaz-Sylvester, PhD, and Ms. Teresa Wilson, BA, who provided insight that greatly assisted the research.