Web-Conferenced Simulation Sessions: A Satisfaction Survey of Clinical Simulation Encounters via Remote Supervision

Abstract

Purpose: A critical barrier to expanding simulation-based instruction in medicine is the availability of clinical instructors. Allowing instructors to remotely observe and debrief simulation sessions may make simulation-based instruction more convenient, thus expanding the pool of instructors available. This study compared the impact of simulation sessions facilitated by in-person (IP) faculty versus those supervised remotely using Web-conferencing software (WebEx^®, Cisco [www.webex.com/]). Subjects and Methods: A convenience sample of preclinical medical students volunteered to “care for” patients in a simulation laboratory. Students received either standard IP or Web-conferenced (WC) instruction. WC sessions were facilitated by off-site instructors. A satisfaction survey (5-point Likert scale, where 1=strongly disagree and 5=strongly agree) was completed immediately following the sessions. Results: Forty-four surveys were analyzed (WC n=25, IP n=19). In response to the question “Was the communication between faculty and students a barrier to understanding the case?,” the average student responses were 2.8 (95% confidence interval [CI] 2.4–3.2) for WC and 4.5 (95% CI 4.0–5.0) for IP (p<0.0001). In response to the question “Would you participate again in such a session?,” the average student responses were 4.2 (95% CI 4.0–4.5) for WC and 4.9 (95% CI 4.6–5.2) for IP (p=0.0003). Both groups agreed that they acquired new skills (4.2 for WC, 4.5 for IP; p=0.39) and new knowledge (4.6 for WC, 4.7 for IP; p=0.41). Conclusions: Telecommunication can successfully enhance access to simulation-based instruction. In this study, a Web interface downgraded the quality of student–faculty communication. Future investigation is needed to better understand the impact of such an effect on the learning process and to reduce barriers that impede implementation of technology-facilitated supervision.

Introduction

Healthcare simulation laboratories have now enabled trainees to “care for” patients in an environment that does not endanger real patients' safety^1

–4 while the trainees perform under direct observation of an experienced clinician. However, even when trained instructors exist at an institution, faculty time away from clinical duties is limited. One potential option to increase faculty participation in simulation sessions is to allow faculty to facilitate (observe and debrief) a simulation session from a remote location using Web-based videoconferencing technology.

Use of telecommunication to facilitate learning and patient care is not new. In education in general, telecommunication has been used to provide educational content to geographically remote sites. Healthcare education has used this modality with providing didactic materials to students, residents,⁵ or staff at distant sites. The surgical fields have used telecommunication in the form of telementoring, where telecommunication is used to guide trainees through procedures with a supervising instructor at a geographically remote site, as opposed to only content delivery in the prior examples.^6,7 There are a few reports of using telecommunication in simulation education. The examples available in the literature are surgical procedure based: Panait et al.⁸ compared medical students learning on a virtual laparoscopy simulator, one group with an in-person (IP) instruction and the other group with remote instruction using telecommunication; and Okrainec et al.⁹ remotely guided surgeons in Botswana in their simulated laparoscopic surgical practice using a laparoscopy simulator in Botswana. Another example of how telecommunication has been used for simulation education is given by Cooper et al.¹⁰ using Web conferencing of a simulation session in progress to be shown to a remote audience, the purpose of which was to educate the audience by using the simulation as a driver of a clinical discussion.

There is little evidence found in the literature describing this study's form of Web-conferenced (WC) instruction, where the debriefing instructor of a cognitive or team-training scenario is remote from the actual simulation but can give real-time feedback and discussion on the performance of a group of simulation participants. This novel use of telecommunication was felt to be a possible solution to the lack of trained instructors available on-site at the simulation laboratory.

The question for this study was as follows: Can direct facilitation (supervision and instruction) of simulation sessions be as effectively accomplished using telecommunication tools? The null hypothesis of this study was that there would be no difference between WC simulation sessions and IP simulation sessions.

Subjects and Methods

Study Design

This was a prospective pilot study of student and instructor satisfaction with two methods of simulation supervision and instruction. The institutional review board for the participating institution approved the study.

Study Setting and Population

A convenience sample of preclinical Harvard medical students participated in this study, along with simulation instructors from the Gilbert Program in Medical Simulation (Harvard Medical School, Boston, MA). All simulation instructors were clinical faculty at one of the Harvard-affiliated hospitals from the fields of emergency medicine, internal medicine, and anesthesia. The eligible student participants had already self-selected their own groups (3–5 students per hour-long session), and each student group was already assigned to one instructor by the simulation laboratory administration. The self-formed student groups met throughout the semester in the simulation laboratory for simulation cases and participated in the study simulation sessions during a discrete portion of the semester. All data collection occurred between March and June 2009.

Student groups and instructors were assigned to IP and WC sessions. Opportunities were provided such that each student group had the chance to participate in both an IP and a WC session. The opportunities were provided by convenience according to the time that the student group and faculty instructors met as well as the availability of the simulation room with the Web-conferencing capabilities.

All simulation sessions occurred in the Gilbert Program in Medical Simulation, located within the buildings of Harvard Medical School. In this program, students were able to “care for” a patient in simulated clinical scenarios, including performing a history and physical, ordering diagnostics, and performing interventions. The simulated patient was a high-fidelity robot mannequin (ECS; Medical Education Technologies Inc., Sarasota, FL) with the voice and the physiologic parameters manipulated by a simulation technician. For the WC sessions, instructors were located in another room in the medical school, in their office at the hospital, or in another building in another city. WebEx^® (Cisco) Web-conferencing software (www.webex.com) was used.

Study Protocol

In the simulation sessions, preclinical students interviewed, examined, and managed simulated patients while working in their self-formed groups. They discussed the case during the session as well as kept notes on a whiteboard that were referenced during debriefing. Clinical faculty members were present with the students to directly observe the encounter and to provide immediate feedback to the students.

All simulated cases were developed by the instructor participants in the usual manner consistent with other simulation sessions. The cases used for this study were rhabdomyolysis, renal colic, anaphylaxis, pulmonary embolism, subdural hematoma, and septic shock.

The student and instructor participants were familiar with the IP simulation format (a clinical encounter followed by a debriefing segment with the instructor in the room) as they both had participated in similar sessions in the past.

In the WC sessions, the students and a simulation specialist (technologist) were on-site in the simulation room, and the instructor was in a remote location. A Webcam was set up to show the mannequin and its immediate surroundings, which would include most of the 3–5 students standing around the mannequin's bed. The instructor was able to use the Web-conferencing software to observe the students around the simulation mannequin and could converse with the students via a speaker phone in the simulation room. The instructor was able to visualize and hear the students in real-time; the students were only able to hear the instructor in real-time via a speaker phone in the simulation room.

Prior to the first WC sessions, the instructors spent 5–15 min working with a simulation technician on how to remotely interact with the students use the WC software.

At the beginning of the WC sessions, the student participants were told that their instructor was at a remote site watching and listening to them via Webcam and phone and that the instructor could speak to them over the speaker phone.

Both IP and WC sessions proceeded with the usual format: orientation to the case, simulation session, and debriefing with the instructor. Immediately following the sessions, both students and instructors completed the surveys, which were collected by the researchers.

Measurements

User surveys were given to the student participants immediately after the debriefing of the simulation sessions. The student surveys had seven questions, each answered with a 5-point Likert scale from “strongly disagree” (1) to “strongly agree” (5). A section for unstructured comments was at the end of the survey. Written informal feedback from the instructors was gathered after each session. Each survey had areas for unstructured comments (Table 1).

Table 1.

Description of WebEx Student Survey Instrument

FACTOR	DESCRIPTION
Demographics	Gender
	Age
	Year of study
	Medical or dental school affiliation
	Estimated number of previous simulation sessions
Survey questions	On a scale of “Strongly disagree” (1) to “strongly agree” (5)
Q1.	Do you have a better understanding of the clinical case in which you participated today?
Q2.	Did you feel comfortable with today's learning session?
Q3.	Do you feel you learned new knowledge?
Q4.	Do you feel you learned new skills?
Q5.	Do you feel you gained new insight regarding your professional role?
Q6.	Was the communication between faculty and students a barrier?
Q7.	Would you participate again in such a session?
Comment section	Please use the back of the page for any comments.

Q, survey question.

Data Analysis

Student data were analyzed using SPSS software. Statistical differences between the IP and WC formats were tested using the independent-samples t test. Subgroup analysis for the instructors who participated in both IP and WC formats was performed using a two-sample t test limited to the sample within each instructor. Narrative comments in both the student and instructor surveys were analyzed, and themes were found among the anecdotal comments.

Results

The student respondents' demographic description included an age range of 20–29 years (mean, 23.3 years), and current academic level was divided between first year of medical school (86%) and second year (14%).

Forty-four surveys were returned from a total of 23 total study simulation sessions. Nineteen surveys (43%) were from the IP sessions, and 25 (57%) were from the WC sessions, and the distribution of IP and WC sessions among the three instructors is shown in Table 2. Eight of the student participants who responded to the survey participated in both IP and WC sessions. Eleven of the 44 student surveys contained additional written comments. The data also included narrative comments from the three instructors.

Table 2.

Distribution of Sessions Among Instructors

INSTRUCTOR	IP	WC	TOTAL
1	3	0	3
2	8	14	22
3	8	11	19

IP, in-person session; WC, Web-conferenced session.

Both IP and WC groups responded favorably (“agree” or “strongly agree”) to questions regarding perceived educational value and efficacy of the experience; at least 75% of the respondents in both groups reported that they achieved a better understanding of the clinical case, learned new knowledge, learned new skills, felt they gained new insight into their professional role, and reported that they would participate again in such a session (Table 3). In relation to the satisfaction with the simulation encounter, only 11% of the IP surveys listed communication as a barrier between students and faculty, whereas 40% of WC surveys stated communication as a barrier.

Table 3.

Survey Results

	IP (N=19)	WC (N=25)
Demographics
Gender	63% M, 37% F	50% M, 50% F
Age (years)	23.3	23.3
Year of study	1st	1st
Medical or dental school affiliation	All medical	All medical
Number of previous simulation sessions	10	8
Survey questions (on a scale of “strongly disagree” [1] to “strongly agree” [5])	“Strongly agree” (5) or “agree” (4)	“Strongly agree” (5) or “agree” (4)
Q1. Better understanding of case	100%	88%
Q2. Felt comfortable with session	100%	88%
Q3. Learned new knowledge	100%	92%
Q4. Learned new skills	95%	76%
Q5. Gained insight for professional role	89%	76%
Q6. Communication was barrier^a	11%	40%
Q7. Participate again in such a session	100%	92%

Survey question (Q) 6 was a negatively asked question.

F, female; IP, in-person sessions; M, male; WC, Web-conferenced sessions.

Further comparison of the two groups was accomplished through a series of independent-samples t tests to assess differences between the two groups on the seven-question survey. Results indicated that three questions (comfort with the session, communication between faculty and students, and if the students would participate in similar session in the future) were statistically significantly different between the IP and WC surveys (Table 4). Subgroup analyses were performed for each of the two instructors who participated in both the IP and WC formats (Instructors 2 and 3) and found that the comparisons between IP and WC were consistent between the instructors who taught both types of sessions.

Table 4.

Group Comparison Results

DEPENDENT VARIABLE	GROUP	MEAN	LOWER BOUND CI	UPPER BOUND CI	P VALUE
Q1. Better understanding of case	IP	4.789	4.475	5.104	0.11
	WC	4.480	4.206	4.754
Q2. Felt comfortable with session	IP	4.895	4.611	5.179	0.0002
	WC	4.200	3.953	4.447
Q3. Learned new knowledge	IP	4.737	4.473	5.001	0.41
	WC	4.600	4.370	4.830
Q4. Learned new skills	IP	4.474	4.049	4.898	0.39
	WC	4.240	3.870	4.610
Q5. Gained insight for professional role	IP	4.316	3.931	4.701	0.15
	WC	3.960	3.624	4.296
Q6. Communication was barrier^a	IP	4.526	4.028	5.025	<0.0001
	WC	2.800	2.365	3.235
Q7. Participate again in such a session	IP	4.895	4.624	5.165	0.0003
	WC	4.240	4.004	4.476

Survey question (Q) 6 was recoded in the inverse for this analysis.

CI, 95% confidence interval; IP, in-person session; WC, Web-conferenced session.

Eleven of the students' surveys had written comments. All comments were from the WC sessions' surveys. Using a modified constant comparison method, emergent themes among the comments were grouped. Ten of the comments stated that the WC sessions created a barrier in communication. Two of the comments stated that it was preferred to have the instructor WC for the simulation scenario but IP for the debriefing, such as this comment: “I like being more on our own at this stage (after being [in] 10 [simulation sessions] with someone in the room) but I think the teaching could be better with the instructor in the room for wrapup.”

The instructor participants' comments in general stated the technology was a barrier to communication and that the technology needed improvement for future sessions. Some comments supported the use of Web conferencing and noted the potential benefits of it; however, there was general preference for IP instruction.

Discussion

In simulation education, trained faculty instructors may exist at an institution; however, they may not be available to participate as an instructor in a simulation session. Several obstacles can exist for faculty to participate, one of which could be the lack of time to travel to a simulation center located far from their clinical responsibilities. This study explored the use of Web-conferencing technology as a possible solution for faculty instructors who are located in areas remote from the simulation center. Specifically, this study examined the perceptions of student and faculty participants regarding the use of WC technology for instruction in simulation sessions.

This study demonstrated that the use of WC software to deliver clinical simulation sessions was generally perceived to be as educationally effective as IP sessions by the student participants. The survey responses for both IP and WC indicated that the student participants felt they learned new knowledge, skills, and attitudes from the simulation sessions. Little information is available in the literature describing teleconferencing in simulation education using an instructor at a remote site. It is interesting that some of the student comments expressed a preference to have the instructors at a remote site for the simulation case but to have the instructor be IP for the debriefing discussion. Future studies could evaluate such hybrid sessions using IP and WC methods in a single session.

This study also demonstrated that the Web conferencing created a perceived barrier to communication. The surveys did not include questions asking why participants felt there was a barrier to communication, but some of the unstructured comments offered some ideas, such as the comments that explained frustration that the instructor could not see the group's thought process written on the whiteboard (out of view of the camera). A difference in the perception of the quality of communication between IP and WC sessions could be due to other difficulties in the implementation of the Web-conferencing platform, such as microphones not being placed in appropriate locations for the instructor to hear the students.

It is interesting that even though there was a perceived barrier to communication with the WC sessions, the students still responded that they would participate again in such sessions. Although this initial pilot study was small, it does show that there is potential for the use of Web conferencing in the simulation laboratory as a means to increase the availability of simulation instructors.

For Web conferencing to be used most effectively as a teaching adjunct in the simulation laboratory, some of the factors creating a perceived barrier to communication will need to be addressed. One of the barriers mentioned in the anecdotal comments from the faculty instructors centered on the limited views available to the remote instructors of the simulation room. For future sessions, the placement and view of the cameras will need to capture all important information in the encounter. As in the example above, the students often use a whiteboard next to the mannequin to document their history, physical findings, differential diagnosis, and laboratory and imaging data. Typically in IP simulation sessions, the simulation instructors refer to this written information during their debriefings. However, in the WC sessions the whiteboard was not captured in the Webcam's view, and thus useful information was lost for use in the debriefing discussions. Future studies can focus on what creates the barrier to communication in WC sessions and thus improve the use of the technology, or even improve the technology itself.

Limitations

There are several limitations of this study that should be noted. This study was conducted with a small convenience sample of volunteer medical students in self-formed simulation groups. Although attempts were made to allow all participants to experience both WC and IP, because of scheduling complications only some of the participants were able to experience and evaluate both formats. This study included only medical students from one institution. Another limitation to note is that the WC format was not an identical replica of the IP format because the students were unable to see the instructor and the instructor could not see the whiteboard. These features limit the generalizability of the results of this pilot study.

Conclusions

This was a novel example of the use of telecommunication to facilitate cognitive-based simulation scenarios. Although overall ratings for both IP and WC sessions were high, there was a perceived difference between the WC and IP simulation session. In this study, a Web interface downgraded the quality of student–faculty communication. Future investigation is needed to better understand the impact of such an effect on the learning process, to explore new technologies to facilitate direct supervision, and to compare the learning outcomes between the IP and WC formats.

Footnotes

Disclosure Statement

No competing financial interests exist.

References

Gordon

, Hayden

, Ahmed

, Pawlowski

, Khoury

, Oriol

. Early bedside care during preclinical medical education: Can technology-enhanced patient simulation advance the Flexnerian ideal? Academic Med, 2010; 85:370–377.

Gordon

, Oriol

, Cooper

. Bringing good teaching cases "to life": A simulator-based medical education service. Academic Med, 2004; 79:23–27.

McMahon

, Monoghan

, Falchuk

, Gordon

, Alexander

. Development of a simulator-based curriculum to promote comparative and reflective analysis in an internal medicine clerkship. Academic Med, 2005; 80:84–89.

Gordon

, Pawlowski

. Education on-demand: The development of a simulator-based medical education service. Academic Med, 2002; 77:751–752.

Markova

, Roth

. E-conferencing for delivery of residency didactics. Academic Med, 2002; 77:748–749.

Rosser

, Young

, Klonsky

. Telementoring: An application whose time has come. Surg Endosc, 2007; 21:1458–1463.

Moore

, Adams

, Partin

, Docimo

, Kavoussi

. Telementoring of laparoscopic procedures: Initial clinical experience. Surg Endosc, 1996; 10:107–110.

Panait

, Rafiq

, Tomulescu

et al. Telementoring versus on-site mentoring in virtual reality-based surgical training. Surg Endosc, 2006; 20:113–118.

Okrainec

, Henao

, Azzie

. Telesimulation: An effective method for teaching the fundamentals of laparoscopic surgery in resource-restricted countries. Surg Endosc, 2009; 24:417–422.

10.

Cooper

, Barron

, Blum

et al. Video teleconferencing with realistic simulation for medical education. J Clin Anesth, 2000; 12:256–261.