Abstract
Background & Aims
We utilized a triangulation method of a faculty development program's (FDP) evaluation comprising short-course workshops on classroom behaviors and lecturing skills of basic sciences faculty in a medical school.
Methods & Results
This study utilized data from the pre and post evaluation of classroom lectures by an expert observer. Course participants were observed before the inception of a 4-month FDP and after 6-months of program completion. Findings at 6-month post-FDP interval were supplemented with students’ and participant's self-evaluation. Expert evaluation of 15 participants showed that more participants were summarizing lectures at the end of their class (p = 0.021), utilizing more than one teaching tool (p = 0.008) and showing a well-structured flow of information (p = 0.013). Among the students, majority (95.5%, n = 728) agreed on “teachers were well-prepared for the lecture”, however, a low number (66.1%, n = 504) agreed on “teachers were able to make the lecture interesting”. On self-evaluation (n = 12), majority of the participants (91.7%, n = 11) thought these FDP workshops had a positive impact on their role as a teacher.
Conclusions
Gathering feedback from multiple sources can provide a more holistic insight into the impact of an FDP and can provide a robust framework for setting up future FDP targets.
Introduction
Medical education is evolving rapidly, and the role of a medical educator is challenging more than ever. However, the early-career medical teachers are thought to be ready and well prepared for this role right away without any formal training. 1 Faculty Development programs (FDPs)play a vital role in developing early-career teachers into professional educators. Of various continuing medical education (CME) formats, short-course workshops are not only easy to administer but are highly effective in there source-limited settings. 2 To ascertain the impact of these FDP workshops on the actual classroom behavior of medical teachers (Level 3 of Kirkpatrick's evaluation model), three data sources are commonly utilized:1) evaluation by self or participant's self-evaluation, 2) evaluation by the students, and 3) peer-evaluation.3, 4 We postulate that if used in combination (a triangulation model), these measures can improve the overall strength and reliability of findings especially if the sample size is low, which is the case in FDPs being run in a single institution.
Materials & methods
We analyzed an FDP comprising CME workshops on lecturing skills of basic sciences faculty in a private medical school of Lahore, Pakistan after seeking an ethical approval. A single workshop session was of 4-h duration and was conducted at least twice every month by the medical education experts. The course content was delivered mainly through didactic sessions followed by interactive group discussions. The course encompassed six themes: 1) Lecture layout & Time management, 2) Communication Skills, 3) Proficiency in Subject Matter & Content Delivery, 4) Teacher-Participant interaction, 5) Persona & Composure, 6) Quality of the PowerPoint presentation. (Figure 1).
FDP's timeline along with the feedback intervals and study design.
A blinded evaluation of one randomly picked classroom lecture was done to ascertain the change in classroom behavior of the participants. An expert observer rated lectures one month prior to initiation and six months after the completion of the FDP workshops on a 38-item questionnaire with ‘yes-no’ responses. To strengthen the findings of the post-FDP evaluation, students’ and participant's self-evaluation were added (Triangulation model). Students rated their teacher's lectures on a 13-item questionnaire with yes-no responses which was adapted from existing literature and was modified to our FDP's course content. 5 All questionnaires were face validated by medical education experts.
Data analysis was done on SPSS 21.0 and Microsoft Excel. Paired dichotomous categorical variables from pre- and post- evaluation were analyzed through McNemar's exact test of significance (2-tailed) using the binomial distribution. Percent of differences between pre- and post-FDP data with 95% confidence intervals were calculated. A p-value of less than 0.05 was considered statistically significant for inferential purposes.
Study outcomes
The results from the three-pronged feedback were used to devise the future FDP goals and targets. Triangulation of findings can provide a holistic view in this regard as some technical aspects of lecturing can only be ascertained by an expert observer. Likewise, it is also necessary to include the students’ perspective on how effective the teacher is in the classroom as far as their learning is concerned. It is noteworthy that the findings validate each other as well and increase the overall strength with less robust data. This is evident in expert evaluation on question 2 (Table 1) and student evaluation on question 2, 3 and 5 (Table 2), which show similar results.
Difference of percentages in participant's pre- and post-FDP evaluation by the expert.
Legend: a Confidence intervals are calculated using z-scores due to a smaller sample size (df = 14). bp-value generated from McNemar exact test (2-sided significance). * p-value significant(-) denotes a negative attribute i.e. undesired.
Evaluation of participant's classroom behavior and lecture by the students at 6-month post-FDP interval.
A significant improvement was seen in the faculty's behavior on expert evaluation except in some areas like the PowerPoint presentations. (Table 1). This could be because the teachers were using same outdated slides of other topics at the post-FDP interval. This calls for a special attention towards designing workshops aimed at the effectiveness of PowerPoint presentations which should be coupled with engagement of faculty to review their whole course content or make a standardized template for all lecture presentations to make them more effective. Another reason can be teacher's commitment in delivery of up-to-date material and can be further evaluated by behavioral centered investigations. Accordingly, a majority of the participants expressed their interest in future workshops on PowerPoint multimedia on self-evaluation, 75.0% (n = 9). Majority of them believed the FDP workshops had a positive impact on their teaching skills and also expressed an interest in attending future programs, 91.7% (n = 11).
On expert evaluation, we found out that less than half of the teachers were integrating short breaks like mild jokes, anecdotes, examples or a story in their lectures, which was supported by the question 1 and 11 in student's evaluation. Half of the faculty participants opted for more training in ‘teacher-participant interaction’ domain.
Kirkpatrick's level 3 (change in workplace behavior and attitude) is considered a reliable indicator of the effectiveness of CME activities and FDPs. 6 However, majority of the studies which do report evaluation at level 3 do so in the form of participant's self-reported perceptions and changes. 7 We focused on a comprehensive evaluation by an expert observer and students instead and added participant's self-evaluation to these. Evaluations from other FDPs in medical and clinical education also support the use of multi-source feedback data to drive the future policymaking.8, 9 Our post-FDP evaluation was done at 6-month interval which is a reasonable amount of time where learners can translate learning into practices. 3
It is essential to understand that these findings are based on a single institution with limited participants and high drop-out rate. Also, the findings are from a single randomly selected routine lecture of each participant and with only one expert evaluator. This limitation should be known when interpreting the findings. Despite a smaller sample, triangulating the findings can help in designing newer FDP models, and potentially assist in learning multicenter approach with wider variety of classrooms, and educational settings. It should be known that the self-evaluation was based on open-ended survey questions and no objective scale was applied, however, a validated scale can be an expedient approach. Also, ratings from the evaluators could be done on a Likert scale, we opted for yes-no responses to avoid complexity.
Conclusion
FDPs based on short-course workshops can positively influence medical teacher's behavior in the classroom and their lecturing skill through reinforcement of pertinent attributes. Triangulation method correlates and validates findings from multi-source feedback which can provide a practical framework for improvement and development of future FDPs targets. This is particularly important for developing countries with limited resources and less robust data.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
