Developing a 3D Gestural Interface for Anesthesia-Related Human-Computer Interaction Tasks Using Both Experts and Novices

Abstract

Objective:

The purpose of this research was to compare gesture-function mappings for experts and novices using a 3D, vision-based, gestural input system when exposed to the same context of anesthesia tasks in the operating room (OR).

Background:

3D, vision-based, gestural input systems can serve as a natural way to interact with computers and are potentially useful in sterile environments (e.g., ORs) to limit the spread of bacteria. Anesthesia providers’ hands have been linked to bacterial transfer in the OR, but a gestural input system for anesthetic tasks has not been investigated.

Methods:

A repeated-measures study was conducted with two cohorts: anesthesia providers (i.e., experts) (N = 16) and students (i.e., novices) (N = 30). Participants chose gestures for 10 anesthetic functions across three blocks to determine intuitive gesture-function mappings. Reaction time was collected as a complementary measure for understanding the mappings.

Results:

The two gesture-function mapping sets showed some similarities and differences. The gesture mappings of the anesthesia providers showed a relationship to physical components in the anesthesia environment that were not seen in the students’ gestures. The students also exhibited evidence related to longer reaction times compared to the anesthesia providers.

Conclusion:

Domain expertise is influential when creating gesture-function mappings. However, both experts and novices should be able to use a gesture system intuitively, so development methods need to be refined for considering the needs of different user groups.

Application:

The development of a touchless interface for perioperative anesthesia may reduce bacterial contamination and eventually offer a reduced risk of infection to patients.

Keywords

gestures human-computer interaction expertise anesthesiology and perioperative care

Introduction

Gestures are a natural means of communication (Efron, 1972; Freedman, 1972; Kendon, 1988; McNeill, 1992) used to enhance, extend, and replace speech where verbal communication is hindered or infeasible (Kendon, 1988; McNeill, 1992). Consequently, gestures have the potential to serve as a natural way to interact with computers and other devices (Karam & Schraefel, 2005). Gesture-based human-computer interaction is most common in the form of contact-based, 2D gestures on touchscreen devices (e.g., smartphones, tablets). More analogous to natural human communication, but not as widely adopted, is vision-based, 3D, gestural technology. Vision-based, 3D, gestural technology uses cameras to detect motion so that the user does not have to wear additional sensors or components when performing 3D gestures. This creates a more natural and nonintrusive experience (Baudel & Beaudouin-Lafon, 1993). The touchless feature of 3D, vision-based, gestural input is particularly useful in environments such as operating rooms (ORs) (Maggioni & Kämmerer, 1998; Wachs et al., 2008) since 3D gestural input can help maintain sterility and limit the spread of bacteria that can occur through physical contact. In this study, we explore the application of touchless, 3D, gestural input for anesthesia providers in ORs.

The majority of gestural research in the OR is focused on the manipulation of medical images during a surgical case wherein the hand is an input device and acts as a pointer or mouse (Bizzotto et al., 2014; Jacob & Wachs, 2014; Jacob, Wachs, & Packer, 2013; Mewes, Saalfeld, Riabikin, Skalej, & Hansen, 2016; O’Hara et al., 2014). However, anesthesia providers could additionally benefit from gestural input technology as they also interact with patients throughout the surgical process (Miller & Pardo, 2011). Multiple studies have linked anesthesia providers to bacterial transfer during the surgical process (Biddle & Shah, 2012; Birnbach, Rosen, Fitzpatrick, Carling, & Munoz-Price, 2015; Loftus et al., 2008; Loftus et al., 2011; Munoz-Price et al., 2013; Rowlands et al., 2014), with contaminated hands’ playing a key role in bacterial transfer (Loftus et al., 2012). Anesthesia providers only intermittently clean their hands while moving throughout the anesthesia work area (Biddle & Shah, 2012; Munoz-Price et al., 2013), and their potentially contaminated hands touch objects such as the patient bed (Rowlands et al., 2014), anesthesia cart and monitors, and computer keyboards (Birnbach et al., 2015). If anesthesia providers can reduce the number of surfaces and objects they physically touch by replacing particular tasks with 3D gestures, then there is a potential for reduced bacterial transfer in the anesthesia environment.

An intuitive gesture vocabulary is the foundation for a gestural system (Nielsen, Störring, Moeslund, & Granum, 2004). Gesture vocabularies either can be elicited through the users, thus maximizing usability of the system, or can be defined by researchers or developers to maximize recognition accuracy of the technology (Stern, Wachs, & Edan, 2008). Although both are widely used in practice, it has been shown that user-defined gesture vocabularies are preferred to researcher-defined gestures (Morris, Wobbrock, & Wilson, 2010). Vision-based gestural input systems offer a large degree of freedom and flexibility in the user interactions (Galitz, 2007; Schneiderman & Plaisant, 2003), but there is a lack of research investigating how different users’ gesture interactions vary, specifically the similarities and differences of gestural mappings across users. Ideally, the gesture sets of multiple user groups would converge into a single intuitive set or at least a compatible set of gestures. Both expert and novice users should be able to interact with the gestural system naturally (Wigdor & Wixon, 2011), so it is important to capture the user behavior of both domain experts and domain novices. Using domain novices represents one end of this spectrum, and domain novices may indeed be asked to interact with anesthesia equipment at various points in the device’s life cycle. The gestural system should additionally be easy to use, so the amount of mental effort should be minimized (Wigdor & Wixon, 2011). A gesture vocabulary that maps poorly to the tasks and functions will create a greater cognitive load, which is undesirable. Environmental context must also be considered when eliciting gestures from users (Ardito, Costabile, & Jetter, 2014; Jacob & Wachs, 2014; Jacob et al., 2013; Nielsen et al., 2004; Wigdor & Wixon, 2011), and complete novices should be included from the very beginning of developing natural user interfaces (Wigdor & Wixon, 2011). Thus, if a gesture vocabulary is to be developed in the application of anesthesia tasks and functions in the OR, gestures should be elicited from both expert and novice users, within a representative environmental context, focusing on similarities and differences of gesture behaviors across users.

This research sought to compare the mappings of gestures to functions generated for domain experts and novices exposed to the same OR anesthesia context. The experts were anesthesia providers, and the novices were students who were unfamiliar with anesthesia tasks in the OR. We expected that the two user groups would generate some similar as well as different gesture-function mappings that would lead to differences in the gesture vocabularies. In addition, we expected novice users to exhibit longer reaction times, thus potentially suggesting greater cognitive load related to gesture-function mappings compared to expert users.

Methods

Nielsen et al. (2004) created a procedure for developing intuitive and ergonomic gesture interfaces. As part of this procedure, gesture vocabularies can be elicited from end users in either a bottom-up or a top-down fashion. The bottom-up approach presents functions to find matching gestures, and the top-down approach presents gestures to find a function mapping (Nielsen et al., 2004). The top-down approach is useful for testing a gesture vocabulary (Nielsen et al., 2004), so since this study is generating gesture-function mappings, the bottom-up approach was used. In this approach, a command or function is shown to the user, and the user chooses a gesture that he or she believes maps to the function. The gesture that is most frequently performed across all users is mapped to a function as the most intuitive gesture. This approach was applied in the current study investigating two cohorts: the responses of undergraduate and graduate students and the responses of anesthesia providers. This research complied with the American Psychological Association code of ethics and was approved by the Clemson University institutional review board (IRB) (IRB Number 2016-110) to study the responses of undergraduate and graduate students. This research was additionally approved by the Medical University of South Carolina IRB (IRB Number Pro00048787) to study the responses of anesthesia providers. The two cohorts were studied at separate times due to the availability of novice and expert participants and the distance between the hospital and the university.

Participants

All participants needed to be able to move their fingers, wrists, and arms without issue in their nondominant hand and needed to be able to read, write, and speak in English. Participants were domain novices (N = 30) and domain experts (N = 16). The domain novices were undergraduate and graduate students, and the domain experts were anesthesia providers including attending anesthesiologists, certified registered nurse anesthetists (CRNAs), and anesthesia residents.

Design

This study employed a repeated-measures design where the functions (N = 10; see Figure 1) were repeated across three blocks. Each block included all 10 functions, and the presentation order of the functions was randomized within each block. The function displays were placed in a PowerPoint presentation according to a randomized order for each participant. The functions tested in the experiment were representative of typical tasks done by anesthesia providers in the OR and were selected after performing in-person and video observations in the OR (Betza et al., 2016). Some functions were generic examples used to elicit gestures for question answering (e.g., “Is the heart rate normal?”), and Function 9 was used for making choices among different options. Function 9 (“Select Heart Rate”) is not a task currently done by anesthesia providers but is a function that could be implemented as part of a 3D gestural system for anesthesia, as testing new gestural functionalities when building gestural systems is recommended (Wigdor & Wixon, 2011).

Figure 1.

Anesthesia functions tested in both experiments.

Equipment

The study of the two cohorts was completed at the same table with the same standard desktop computer with two monitors side by side (see Figure 2). The study equipment (i.e., desk, two Dell 22-inch LED monitors, an Intel RealSense F200 Camera gestural camera, a PC running Windows 10, and medical gloves) was used at both locations with the position of the monitors and the gestural camera marked on the desk. Participants primarily interacted with the right monitor as this monitor presented the function displays and had the 3D camera attached on the top. A digital clock with the computer system time and depth-feedback of the 3D camera view were displayed on the left monitor. The setup of the computer and monitors did not differ between novices and experts; however, the study occurred in different rooms due to the participants’ being located at either the hospital or the university. The domain experts were in a conference room at the hospital that had additional tables, chairs, and a TV. The domain novices were at the university in an experimental room in a research lab without windows.

Figure 2.

Experimental setup.

The experimental session duplicated certain features of an anesthesia setting in the OR by sounding continuous and intermittent patient alarms and by having participants wear medical gloves. The World Health Organization (2009) recommends health care providers wear gloves when working with a patient, so wearing the gloves helped emulate the anesthesia workstation. The alarms additionally helped to establish environmental context. The participants interacted with the camera with their nondominant hand as dexterity is typically lower in this hand compared to the dominant hand (Oxford Grice et al., 2003), which allowed for observation of how dexterity may influence gesture behaviors.

Procedure

The same study procedure was followed for both cohorts. Upon his or her arrival, the participant completed the informed consent process and filled out a demographics survey. There were two demographics surveys created due to the differences in characteristics of the two cohorts. For example, anesthesia providers were not asked any questions about their major as these questions were not applicable. After completing the demographics survey, the participant familiarized him- or herself with the technology by practicing with the set of 14 gestures provided by the Intel RealSense SDK (Intel Corporation, 2016). Each gesture was performed 15 times according to Nielsen et al.’s (2004) approach for assessing the comfort of gestures in a user elicitation study.

The participant then completed the experimental task. A “Wizard of Oz” technique was used in the experimental session, which has been shown to be valuable in gesture user-elicitation studies (Aigner et al., 2012; Freeman, Benko, Morris, & Wigdor, 2009; Höysniemi, Hämäläinen, & Turkki, 2004; Morris et al., 2010). In the Wizard of Oz technique, the experimenter takes the place of an automatic system, interpreting inputs and controlling outputs. This is done to evaluate functions and interfaces prior to investing in the technology required for automatic input and output. In our experiment, the system is perceived to be controlled by a participant’s gestural input, but the experimenter manually progresses to the next function after a gesture is performed; therefore, this is not a complete Wizard of Oz study as the experimenter is physically in the room with the participant. Having the experimenter manually progress to the next function generates an effect-cause relationship between gesture and function that would be expected if the gestural system were actually implemented and working. The function display (the effect) was always presented first, and then the participant would choose a gesture (the cause) that he or she believed initiated the function. Participants performed gestures of their choosing and whichever gesture was their “first guess” to complete the function.

Intuitive Gestures Measure

The intuitive gesture-function mappings were analyzed separately for the experts and novices to identify the differences between the two gesture sets. Videos of the participants’ hands and fingers were recorded and analyzed to determine which gestures were performed for each function. A list of potential gestures performed was built by the research team to aid in the gesture analysis. The gesture list included the name and definition of all gestures used in the practice session, gestures from other studies, and commonly known cultural gestures. The list of potential gestures was created to provide standardization in gesture classification among the researchers. All videos were analyzed by three researchers separately, and gestures were classified according to the best-fit definition in the gesture list. Any discrepancies in gesture classification were discussed until all researchers agreed on which gesture was chosen by the participant.

There was the potential for participants to choose up to three unique gestures for a function since a repeated-measures design was utilized. When participants perform three gestures for one function across the blocks it reveals that they could not identify an intuitive gesture-function mapping. Therefore, these data should not contribute to the assessment of intuitive gesture-function mappings across the experimental group and thus were removed. Only gesture responses that were either completely consistent (i.e., same gesture performed in all three blocks) or partially consistent (i.e., same gesture performed in two of the three blocks) for a function were considered in the analysis. According to Nielsen et al.’s (2004) approach, the intuitive gesture for a function is the gesture that is most frequently chosen across a group. The gesture responses for each function were compiled in a table, and the gesture response that was performed most frequently across the experimental group was chosen as the intuitive gesture-function mapping.

Reaction Time Measure

The reaction time from presentation of the function display to completion of a gesture was recorded by a computer program for every gesture-function pair. These data were collected to complement the analysis of the intuitive gesture-function mappings as it has been shown that shorter reaction times are associated with higher convergence of gestures performed for a function (Pereira, Wachs, Park, & Rempel, 2015). Different from the intuitive gesture-function mappings analysis, the reaction time data were combined into one analysis for the experts and novices. A mixed linear regression model with participant ID as the random effect was used to identify gesture-function mappings that exhibited longer reaction times. A mixed linear regression model was used to account for both fixed and random effects. The fixed effects in the model were handedness, video game experience, virtual reality experience, functions, and participant type. Interaction effects between function and participant type were included in the model to identify differences between experts and novices. Only reaction times from the first block were analyzed to separate the first instance the participant was exposed to a function and to avoid any issues in the statistical model related to learning effects that could be present in the other blocks. R version 3.2.2 was used for all data analysis; the lmer function of the lme4 package (Bates, Mächler, Bolker, & Walker, 2014) was used to build the mixed linear regression model, and the ggplot2 package (Wickham, 2009) was used to plot the data.

Results

The mean age of novices was 21.9 (SD = 2.23) years, and half (n = 15) of the novices identified as female. Most of the novices were right-handed (n = 26), and only 3 were left-handed. About half (n = 16) of the novices had previous virtual reality gaming experience, and half (n = 15) regularly used video games. A majority of the experts were right-handed (n = 14), and only 2 were left-handed. There were 12 experts who had previous virtual reality gaming experience, and 11 experts regularly used video games.

The mean reaction time across all blocks for the novices was 4.77 s (SD = 2.93 s), and the mean reaction time across all blocks for the experts was 4.31 s (SD = 2.29 s). In the novice cohort, 900 gestures were recorded, but due to inconsistent gesture responses for some participants (e.g., three gestures chosen across blocks), 852 gestures were analyzed. In the expert cohort, 438 out of the 480 possible gestures were analyzed after removing inconsistent gesture responses. Overall, 40 unique gestures were recorded for the novices (see Table 1), and 27 unique gestures were recorded for the experts (see Table 2). All gestures performed across both study groups are included in each table, so the blank cells indicate that the specific gesture was not performed by any participant. Most of the gesture names are relatively self-explanatory (e.g., thumbs up, swipe hand up), but some of the gestures are not as obvious. For example, the “click” gesture is a pinch of only the thumb and index finger; the “pump” gesture was imitating a heart pumping by repeatedly going from an open hand to a closed fist; the “rotate” gesture was rotations of the hand back and forth rather than only one direction; and the “X” gesture was making the letter “X” in the air with the hands. The “down down” and “up up” gestures were similar to “swipe hand up” and “swipe hand down” gestures; they were two movements that were very quick rather than a smooth swiping motion. Out of 10 final gesture mappings, 9 were gestures performed during the familiarization training for the novice group. These were thumbs up, five up, swipe hand up, swipe hand down, and swipe hand left. For the experts, 5 out of 10 final gesture mappings were gestures performed during the familiarization training. These were push hand, thumbs up, and swipe hand left.

Table 1:

Gestures Chosen for Each Function by the Novice Participants

	Gestures
Function [Number]	Click	Come	Down down	Down right	Fist	Five up	Four up	Full grab	Full pinch	Hand facing up	Hand to click	Okay	One up	One down	One down five up	One up	Point	Pump	Push fingers	Push hand	Reverse full grab	Right down	Right up	Rotate	Rotate left	Rotate right	Swipe fingers down	Swipe hand down	Swipe fingers left	Swipe hand left	Swipe fingers right	Swipe hand right	Swipe fingers up	Swipe hand up	Three up	Thumbs down	Thumbs up	Two up	Up up	Wave hand	X
Start the flow [1]				2	1	3						5							3	7								1		2	4	3	7	14			29		3
Stop the flow [2]					12	17			3			2							2	14		2						6	3	10		3				4				2
Increase the flow [3]	1					2										4							3		2	7					2	2	10	33		1	16		7
Decrease the flow [4]			4			2						2		6			1					3		1	6		6	33	3	2				2		17	2
Silence alarm [5]					2	9		1	9			2				3			5	4								5		15	2	12				6				9
Acknowledge the message [6]	5					3					1	16							2																		46	2		5
Heart rate normal? [7]	3					3				6		7						3																			54	1		3
Pulse oximeter normal? [8]	3				2	4				3		10																		4		1				23	25			6
Select heart rate [9]	6				6	5	1	1				6			3	3			25	5												1		2	19		3	3		3
Cancel the message [10]	3	1			10	4						1							2	5	1									25		12				13	3	1		4	2

Note. Blank cells indicate that the gesture was not performed by any participant.

Table 2:

Gestures Chosen for Each Function by the Expert Participants

	Gestures
Function [Number]	Click	Come	Down down	Down right	Fist	Five up	Four up	Full grab	Full pinch	Hand facing up	Hand to click	Okay	One up	One down	One down five up	One up	Point	Pump	Push fingers	Push hand	Reverse full grab	Right down	Right up	Rotate	Rotate left	Rotate right	Swipe fingers down	Swipe hand down	Swipe fingers left	Swipe hand left	Swipe fingers right	Swipe hand right	Swipe fingers up	Swipe hand up	Three up	Thumbs down	Thumbs up	Two up	Up up	Wave hand	X
Start the flow [1]																			4						2	16	3			2		4		10			6
Stop the flow [2]					3	3														8					11	3		2		2		3									1
Increase the flow [3]						1							1												4	16						3		12			3		2
Decrease the flow [4]			2																						15	3	3	13				2				4
Silence alarm [5]									4			2							7	20										7		2
Acknowledge the message [6]									2			15	3						1	1								1		2		2					18
Heart rate normal? [7]									2			6																		1							33	1
Pulse oximeter normal? [8]									2			8																		1							32				1
Select heart rate [9]					1														26											3					9			3
Cancel the message [10]						3												3	2	7				3						11		8								2

Note. Blank cells indicate that the gesture was not performed by any participant.

Intuitive Gesture Mappings

The intuitive gesture-function mappings are shown in Table 3. Pictorial representations of the gestures that were intuitively mapped are shown in Figure 3. There were several functions with different intuitive mappings for the novices and the experts. Functions 1 through 5 were associated with different gestures for the novices and the experts. Functions 6 through 10 resulted in the same gestures for both the novices and the experts. Functions 6, 7, and 8 were all mapped to the “Thumbs up” gesture for both cohorts.

Table 3:

Intuitive Gesture-Function Mappings for Novices and Experts

Function [Number]	Intuitive Gesture Mapped
Function [Number]	Novices	Experts
Start the flow [1]^a	Thumbs up	Rotate right
Stop the flow [2]^a	Five up	Rotate left
Increase the flow [3]^a	Swipe hand up	Rotate right
Decrease the flow [4]^a	Swipe hand down	Rotate left
Silence alarm [5]^a	Swipe hand left	Push hand
Acknowledge the message [6]	Thumbs up	Thumbs up
Heart rate normal? [7]	Thumbs up	Thumbs up
Pulse oximeter normal? [8]	Thumbs up	Thumbs up
Select heart rate [9]	Push fingers	Push fingers
Cancel the message [10]	Swipe hand left	Swipe hand left

Dissimilar mappings for novices and experts.

Figure 3.

Pictorial representation of gestures mapped to anesthesia functions. Arrow indicates movement direction.

Reaction Times

The mean reaction time for Block 1 data for novices was 5.90 s (SD = 3.66 s), and the mean reaction time for Block 1 data for experts was 4.34 s (SD = 2.28 s). Figure 4 shows the raw data of the reaction times for both groups. A summary of the mixed linear regression models is shown in Table 4. Handedness, video game experience, and virtual reality experience were not significantly associated with longer reaction times. For variables with interactions, only the interaction terms are evaluated, and main effects are not discussed. There were significant interactions between the user groups and some of the specific functions. Novices had significantly longer reaction times than experts for Functions 7 through 10. These functions included “Is heart rate normal?” (p = .007), “Is Pulse oximeter normal?” (p < .001), “Select heart rate” (p < .001), and “Cancel the message” (p = .005).

Figure 4.

Jitter plot of reaction times for all trials, for all participants in the first experimental block.

Table 4:

Summary of Mixed Linear Regression Model for Reaction Time

	β	SE	t	p > \| t \|
(Intercept)	1.54	0.12	13.26	<.001*
Handedness, right	0.08	0.12	0.66	.512
Handedness, ambidextrous	−0.04	0.28	−0.13	.901
Video game experience	0.09	0.10	0.89	.376
Virtual reality experience	−0.02	0.10	−0.19	.852
Stop the flow [2]	0.07	0.14	0.52	.603
Increase the flow [3]	−0.07	0.14	−0.48	.634
Decrease the flow [4]	−0.08	0.14	−0.59	.555
Silence alarm [5]	−0.32	0.14	−2.29	.022*
Acknowledge the message [6]	−0.24	0.14	−1.73	.084
Heart rate normal? [7]	−0.38	0.14	−2.76	.006*
Pulse oximeter normal? [8]	−0.34	0.14	−2.45	.015*
Select heart rate [9]	−0.25	0.14	−1.80	.073
Cancel the message [10]	−0.51	0.14	−3.72	<.001*
Novice	−0.01	0.17	−0.06	.954
Stop the Flow [2] × Novice	−0.31	0.17	−1.81	.071
Increase the Flow [3] × Novice	−0.09	0.17	−0.51	.611
Decrease the Flow [4] × Novice	−0.11	0.17	−0.66	.508
Silence Alarm [5] × Novice	0.27	0.17	1.60	.111
Acknowledge the Message [6] × Novice	0.19	0.17	1.08	.280
Heart Rate Normal? [7] × Novice	0.46	0.17	2.70	.007*
Pulse Oximeter Normal? [8] × Novice	0.61	0.17	3.56	<.001*
Select Heart Rate [9] × Novice	0.59	0.17	3.43	<.001*
Cancel the Message [10] × Novice	0.48	0.17	2.81	.005*

p < .05.

Discussion

The objective of this study was to evaluate the differences in intuitive gesture-function mappings between novices and experts. The students generated 40 unique gestures, and the anesthesia providers generated only 27 unique gestures. The context did not change between anesthesia providers and students, and all participants were exposed to the same functions and displays, yet the gesture-function mapping sets differed between the anesthesia providers and the students. Five of the functions mapped to different gestures, and five mapped to the same gesture. The main finding of this paper is that experts and novices differ in terms of intuitiveness of gestures, thus emphasizing the need for domain expertise in the creation of a gesture vocabulary. Furthermore, the novice user group had significantly longer reaction times for four functions compared to the expert group.

There are characteristics of both the similar and the different mappings that reveal insight into the gesture behavior of novices and anesthesia providers. For the set of functions that had different mappings (see Functions 1–5 in Table 3), the anesthesia providers showed associations between the OR’s physical environment and the gesture-function mapping. Specifically with the functions related to manipulating anesthesia gas, there were more rotational gestures, similar to how anesthesia providers currently perform this task in the OR (Betza et al., 2016). Similarly, the “push hand” gesture of “Silence the alarm” is related to the physical interaction with the computers and monitors in the OR (Betza et al., 2016). Thus, the anesthesia providers’ gesture mappings of these functions seem to show a strong contextual relationship to the physical environment. On the other hand, the gesture mappings that were the same (see Functions 6–10 in Table 3) do not show this same level of association to the physical environment. The functions that had the same gesture mappings are more or less general human-computer interaction tasks such as canceling, selecting, providing yes/no answers, and acknowledging.

The differences in intuitive gesture-function mappings have design implications that should be considered when developing an intuitive, context-specific gesture system. The anesthesia provider group exhibited a degree of domain expertise and contextual knowledge that was not inherent within the student group. Because of their expertise in the anesthesia domain and OR, the anesthesia providers chose gestures that were related to the anesthetic tasks as well as the physical and technological components in the anesthesia environment, such as rotational knobs and buttons. Conversely, the students demonstrated very few rotating gestures when the same contextual interface was presented to them. This suggests that in addition to context’s being important (Ardito et al., 2014; Jacob & Wachs, 2014; Jacob et al., 2013; Nielsen et al., 2004; Wigdor & Wixon, 2011), domain expertise is meaningful when creating the gestural vocabularies. However, the fact that both novices and experts chose similar gestures for half of the functions suggests that some functions may not necessarily depend on domain expertise. For gesture-function mappings that were the same across both user groups, potentially a more general population could be used to map gestures to functions, but a general population could not be used solely for all functions because of the gesture-function mappings that were different. The domain expertise of the anesthesia provider group generated about half as many gestures compared to those generated by the student group. Having a narrower set of gestures reveals some homogeneity within the anesthesia providers and may indicate convergence in gesture mapping agreements as a user group.

In addition, the differences in reaction times between novices and experts for some of the functions further support our main finding that there is a need to consider domain expertise when building an intuitive gestural system. Longer reaction times may indicate that participants have difficulty generating a gesture-function mapping as previous studies have used reaction times as indicators for cognitive load (Horsky, Kaufman, Oppenheim, & Patel, 2003). This set of functions (7–10) all included language specifically related to the medical field (e.g., heart rate, pulse oximeter, attending anesthesiologist), and the lack of clinical knowledge in the novice group may have provoked longer reaction times for these functions. The longer reaction times also may have been due to the difficulty of generating a gesture-function mapping as reaction times may be used to indicate cognitive load (Horsky et al., 2003).

Limitations and Future Research

There are some limitations associated with this research. We were able to recruit 30 novices to do the study and only 16 experts, and this difference in sample size may have affected the results. Specifically, the larger number of unique gestures generated in the novice cohort could be due to the larger sample of novices in the study. Furthermore, allowing participants to choose their own gestures for functions may have contributed to greater use of the same gesture for different functions; however, we believe this was the most appropriate way to capture which gestures were intuitive to users by having participants perform their “first guess.” As part of our methodology, the familiarization training with the technology may have influenced gestures chosen during the experiment. However, there was a large number of unique gestures recorded among the students (40) and among the anesthesia providers (27), and only 14 gestures were practiced as part of the familiarization training. Future research should evaluate how different practice gestures affect participant-derived gestures.

Our methodology for determining intuitive gesture-function mappings may be a limitation for this study. Nielsen et al.’s (2004) approach states that the most frequently performed gesture across users is the most intuitive gesture for a function and does not account for variation between the top few gestures chosen. However, the gestures chosen—shown in Tables 1 and 2—indicate that there may not be much variation between the most frequently chosen gestures. For example, in Table 1, the two most frequently chosen gestures for Function 9 (“Select heart rate”) for the novices were “Push fingers” (n = 25) and “three up” (n = 19), and the two most frequently chosen gestures for Function 7 (“Is heart rate normal?”) were “thumbs up” (n = 54) and “okay” (n = 7). This may indicate that Function 7 has a stronger intuitive mapping than Function 9. Future research should focus on refining methodologies for determining the strength of intuitiveness of gestures.

Our study’s main finding was the differences in intuitive gestures between domain experts and domain novices. However, it becomes unclear how to design gesture-function mappings that are intuitive to all users when mappings between user groups exist. For example, if the expert’s choice is mapped as the intuitive gesture for a function, then novices may not find the gesture-function mapping to be intuitive, resulting in errors and frustration. Current gestural research methodologies do not account for differences between user groups; thus, there is a need to develop gesture vocabulary design principles to account for the needs of multiple user groups. There may be certain properties or features of gestures that may be intuitive across all groups, and researchers may be able to design an optimal gesture set that meets all users’ needs using intuitive gesture features, especially after considering other design aspects of gesture sets such as comfort and accuracy (Stern, Wachs, & Edan, 2006). It may be possible that multiple gestures can be mapped to a function to account for user differences in intuition. In addition, it is not clear why domain novices and experts would have differences in reaction times for particular functions, and future researchers should seek to understand why some functions provoke longer reactions and why other functions do not.

Furthermore, which gestures are intuitive may differ within the varying roles of domain experts. Future work should investigate whether anesthesia-related expertise (e.g., CRNA vs. attending anesthesiologist) as well as specific demographics variables, such as age and gender, influence intuitive gestural interactions and how gestures may change as domain expertise is acquired. Future research should examine the gestural interactions of anesthesia providers who work at different hospitals or have exposure to different types of anesthesia equipment. It will also be critical to evaluate specific interactions and an anesthesia provider’s ability to make gestures during tasks such as intubation (i.e., when the anesthesia provider places the tubing down the patient’s throat). This study focused on the two user groups that differ the greatest in domain expertise—those who are domain experts and those who are domain novices—as it is recommended to include complete novices in gestural system development from the beginning; however, it may be interesting to specifically investigate differences between attending anesthesiologists and medical students pursuing careers in anesthesia.

In addition, alternative natural user interfaces (e.g., voice controlled, stylus, gaze tracking, brain-machine) and combinations of technologies should be investigated to determine the best fit for the anesthesia domain. Future studies should incorporate fully developed gestural systems in a high-fidelity simulated environment and look at user behavior over extended periods. Incorporating gestural systems into a simulated environment will allow us to understand how the physical layout and arrangement of equipment affects gesture behavior, such as whether providers more readily perform gestures with their left or right hands and the effects this would have on the gestural system. Last, gestures differ across cultures (Efron, 1972), so the outcomes should be confirmed across cultures and languages, and natural user interfaces should be adapted accordingly.

Conclusion

A gestural input system may be useful within the anesthesia domain as it has been shown that anesthesia providers’ hands contribute to bacterial transfer in the OR (Loftus et al., 2012). By replacing physical touch with 3D hand gestures, anesthesia providers may reduce the number of surfaces, equipment, and other devices they come in contact with, which could potentially lead to a reduced risk of infection. Furthermore, due to a high number of false alarms in the OR (Seagull & Sanderson, 2001), it is good practice for anesthesia providers to look at the patient rather than relying on monitoring alone, and hand gestures would not require an anesthesia provider to turn his or her back to a patient. Therefore, anesthesia providers have the added benefit of not having to disengage from the patient to interact with a computer via hand gestures. Although a gestural input system for anesthesia tasks in the OR may be valuable, this study has shown that novices and experts differ in the intuitiveness of gestures and domain expertise matters when creating a gestural vocabulary. This is an important consideration during design and development as there are a variety of users who could potentially use this system: CRNAs, attending anesthesiologists, anesthesia residents, anesthesia fellows, and medical students. While it is easy to design for one group of users and difficult to design for several groups of users (Schneiderman & Plaisant, 2003), a gestural system for anesthesia should be easy to all potential users.

When providers reduce the number of surfaces they come in contact with, the spread of bacteria is potentially reduced, thus minimizing the risk of infection. While gestural input systems have found use within the OR for surgical tasks, the value of using a 3D gestural vocabulary for anesthetic tasks has yet to be fully explored. Despite allowing our participants to select any gesture they felt appropriate, we found considerable homogeneity in their responses, particularly among domain experts. This is encouraging for the development of no-touch interfaces for perioperative anesthesia that might eventually offer considerable reductions in the risk of infection for surgical patients.

Key Points

Vision-based gestural input technology allows for natural and nonintrusive human-computer interactions and has the potential to benefit anesthesia providers in the OR.

A user-centered gesture-elicitation approach was used in two studies that investigated the gesture behavior of experts (i.e., anesthesia providers) and novices (i.e., non–medical students).

The gesture vocabularies generated from experts and novices displayed differences in gesture-function mappings, and the experts’ mappings that differed revealed a relationship to physical components in the contextual environment.

Although experts and novices displayed different gesture-function sets, it is unclear how to design a gesture vocabulary that is intuitive and easy to use for all users.

Footnotes

Acknowledgements

This project was supported by the Agency for Healthcare Research and Quality (Contract Number P30HS2438001). The authors would like to acknowledge the members of the Realizing Improved Patient Care Through Human-Centered Design in the Operating Room (RIPCHD.OR) study group.

ORCID iD

David M. Neyens

Katherina A. Jurewicz is currently (as of 2018) a PhD student in the Department of Industrial Engineering at Clemson University in Clemson, South Carolina. She received her MS degree in industrial engineering from Clemson University in 2016.

David M. Neyens is an associate professor in the Department of Industrial Engineering at Clemson University. He received his PhD in industrial engineering from the University of Iowa in 2010.

Ken Catchpole is the South Carolina SmartState Endowed Chair in Clinical Practice and Human Factors and is a professor in the Department of Anesthesia and Perioperative Medicine at the Medical University of South Carolina in Charleston. He received his PhD in psychology and physiology at the University of Leeds in West Yorkshire, United Kingdom, in 1999.

Scott T. Reeves is a professor in and the chair of the Department of Anesthesia and Perioperative Medicine at the Medical University of South Carolina in Charleston. He received his MD at the Medical University of South Carolina in 1987.

References

Aigner

Wigdor

Benko

Haller

Lindbauer

Ion

. . . Koh

(2012). Understanding mid-air hand gestures: A study of human preferences in usage of gesture types for HCI. Microsoft Research TechReport MSR-TR-2012-111. Retrieved from https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/GesturesTR-20121107-RoA.pdf

Ardito

Costabile

M. F.

Jetter

H.-C.

(2014). Gestures that people can understand and use. Journal of Visual Languages & Computing, 25(5), 572–576.

Bates

Mächler

Bolker

Walker

(2014). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48.

Baudel

Beaudouin-Lafon

(1993). Charade: Remote control of objects using free-hand gestures. Communications of the ACM, 36(7), 28–35.

Betza

S. M.

Jurewicz

K. A.

Neyens

D. M.

Riggs

S. L.

Abernathy

J. H.

Reeves

S. T.

(2016). Anesthesia maintenance and vigilance. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 608–612). Santa Monica, CA: Human Factors and Ergonomics Society.

Biddle

Shah

(2012). Quantification of anesthesia providers’ hand hygiene in a busy metropolitan operating room: What would Semmelweis think? American Journal of Infection Control, 40(8), 756–759.

Birnbach

D. J.

Rosen

L. F.

Fitzpatrick

Carling

Munoz-Price

L. S.

(2015). The use of a novel technology to study dynamics of pathogen transmission in the operating room. Anesthesia & Analgesia, 120(4), 844–847.

Bizzotto

Costanzo

Bizzotto

Regis

Sandri

Magnan

(2014). Leap motion gesture control with OsiriX in the operating room to control imaging: First experiences during live surgery. Surgical Innovation, 1, 655–666.

Efron

(1972). Gesture, race and culture. The Hague, the Netherlands: Mouton.

10.

Freedman

(1972). The analysis of movement behavior during the clinical interview. In Siegman

A. W.

Pope

(Eds.), Studies in Dyadic Communication (pp. 153–175). New York, NY: Pergamon.

11.

Freeman

Benko

Morris

M. R.

Wigdor

(2009). ShadowGuides: Visualizations for in-situ learning of multi-touch and whole-hand gestures. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (pp. 165–172). New York, NY: ACM.

12.

Galitz

W. O.

(2007). The essential guide to user interface design: An introduction to GUI design principles and techniques (3rd ed.). New York, NY: Wiley.

13.

Horsky

Kaufman

D. R.

Oppenheim

M. I.

Patel

V. L.

(2003). A framework for analyzing the cognitive complexity of computer-assisted clinical ordering. Journal of Biomedical Informatics, 36(1), 4–22.

14.

Höysniemi

Hämäläinen

Turkki

(2004). Wizard of Oz prototyping of computer vision based action games for children. In Proceedings of the 2004 Conference on Interaction Design and Children: Building a Community (pp. 27–34). New York, NY: ACM.

15.

Intel Corporation. (2016). Handling gestures. Intel® RealSense^TM SDK 2016 R2 documentation. Retrieved from https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?doc_devguide_whats_new_in_sdk.html

16.

Jacob

M. G.

Wachs

J. P.

(2014). Context-based hand gesture recognition for the operating room. Pattern Recognition Letters, 36, 196–203. Retrieved from https://doi.org/10.1016/j.patrec.2013.05.024

17.

Jacob

M. G.

Wachs

J. P.

Packer

R. A.

(2013). Hand-gesture-based sterile interface for the operating room using contextual cues for the navigation of radiological images. Journal of the American Medical Informatics Association, 20(1), 183–186.

18.

Karam

Schraefel

M. C.

(2005). A taxonomy of gestures in human computer interactions. Southampton, UK: University of Southampton, Computing Service.

19.

Kendon

(1988). How gestures can become like words. Cross-Cultural Perspectives in Nonverbal Communication, 1, 131–141.

20.

Loftus

R. W.

Brown

J. R.

Koff

M. D.

Reddy

Heard

S. O.

Patel

H. M.

. . . Jensen

J. T.

(2012). Multiple reservoirs contribute to intraoperative bacterial transmission. Anesthesia & Analgesia, 114(6), 1236–1248.

21.

Loftus

R. W.

Koff

M. D.

Burchman

C. C.

Schwartzman

J. D.

Thorum

Read

M. E.

. . . Beach

M. L.

(2008). Transmission of pathogenic bacterial organisms in the anesthesia work area. Anesthesiology, 109(3), 399–407. Retrieved from https://dx-doi-org.web.bisu.edu.cn/10.1097/ALN.0b013e318182c855

22.

Loftus

R. W.

Muffly

M. K.

Brown

J. R.

Beach

M. L.

Koff

M. D.

Corwin

H. L.

. . . Yeager

M. P.

(2011). Hand contamination of anesthesia providers is an important risk factor for intraoperative bacterial transmission. Anesthesia & Analgesia, 112(1), 98–105.

23.

Maggioni

Kämmerer

(1998). GestureComputer—History, design and applications. In Cipolla

Pentland

(Eds.), Computer vision for human-machine interaction (pp. 23–52). Cambridge, UK: Cambridge University Press.

24.

McNeill

(1992). Hand and mind: What gestures reveal about thought. Chicago, IL: University of Chicago Press.

25.

Mewes

Saalfeld

Riabikin

Skalej

Hansen

(2016). A gesture-controlled projection display for CT-guided interventions. International Journal of Computer Assisted Radiology and Surgery, 11(1), 157–164.

26.

Miller

R. D.

Pardo

(2011). Basics of anesthesia (6th ed.). Philadelphia, PA: Elsevier.

27.

Morris

M. R.

Wobbrock

J. O.

Wilson

A. D.

(2010). Understanding users’ preferences for surface gestures. In Proceedings of graphics interface 2010 (pp. 261–268). Mississauga: Canadian Information Processing Society.

28.

Munoz-Price

L. S.

Lubarsky

D. A.

Arheart

K. L.

Prado

Cleary

Fajardo-Aquino

. . . Birnbach

D. J.

(2013). Interactions between anesthesiologists and the environment while providing anesthesia care in the operating room. American Journal of Infection Control, 41(10), 922–924. Retrieved from https://doi.org/https://dx-doi-org.web.bisu.edu.cn/10.1016/j.ajic.2013.03.306

29.

Nielsen

Störring

Moeslund

Granum

(2004). A procedure for developing intuitive and ergonomic gesture interfaces for HCI. In Camurri

Volpe

(Eds.), Gesture-based communication in human-computer interaction SE-38 (Vol. 2915, pp. 409–420). Heidelberg, Germany: Springer Berlin.

30.

O’Hara

Gonzalez

Sellen

Penney

Varnavas

Mentis

. . . Dastur

(2014). Touchless interaction in surgery. Communications of the ACM, 57(1), 70–77.

31.

Oxford Grice

Vogel

K. A.

Mitchell

Muniz

Vollmer

M. A.

(2003). Adult norms for a commercially available nine hole peg test for finger dexterity. American Journal of Occupational Therapy, 57(5), 570–573. Retrieved from https://dx-doi-org.web.bisu.edu.cn/10.5014/ajot.57.5.570

32.

Pereira

Wachs

J. P.

Park

Rempel

(2015). A user-developed 3-D hand gesture set for human-computer interaction. Human Factors: The Journal of the Human Factors and Ergonomics Society, 57(4), 607–621.

33.

Rowlands

Yeager

M. P.

Beach

Patel

H. M.

Huysman

B. C.

Loftus

R. W.

(2014). Video observation to map hand contact and bacterial transmission in operating rooms. American Journal of Infection Control, 42(7), 698–701.

34.

Schneiderman

Plaisant

(2003). Designing the user interface (4th ed.). New York, NY: Pearson Education.

35.

Seagull

J. F.

Sanderson

P. M.

(2001). Anesthesia alarms in context: An observational study. Human Factors, 43, 66–78. Retrieved from https://doi.org/10.1518/001872001775992453

36.

Stern

H. I.

Wachs

J. P.

Edan

(2006). Human factors for design of hand gesture human-machine interaction. IEEE International Conference on Systems Man and Cybernetics, 5, 4052–4056. Retrieved from http://web.ics.purdue.edu/~jpwachs/papers/smc06_human.pdf

37.

Stern

H. I.

Wachs

J. P.

Edan

(2008). Optimal consensus intuitive hand gesture vocabulary design. 2008 IEEE International Conference on Semantic Computing. Retrieved from https://doi.org/10.1109/ICSC.2008.29

38.

Wachs

J. P.

Stern

H. I.

Edan

Gillam

Handler

Feied

Smith

(2008). A gesture-based tool for sterile browsing of radiology images. Journal of the American Medical Informatics Association, 15(3), 321–323.

39.

Wickham

(2009). ggplot2: Elegant graphics for data analysis. New York, NY: Springer-Verlag. Retrieved from http://ggplot2.org

40.

Wigdor

Wixon

(2011). Brave NUI world: Designing natural user interfaces for touch and gesture. Burlington, MA: Elsevier.

41.

World Health Organization. (2009). WHO guidelines on hand hygiene in health care: First global patient safety challenge. Clean care is safer care. Geneva, Switzerland: Author.