Demonstration of Facial Communication of Emotion Through Telehospice Videophone Contact

Introduction

In the absence of visual data within telenursing, nonverbal emotional communication depends on the nonlanguage features of speech. ¹ Nonverbal behavior is crucial to effective nurse–client relationships. ^1,2 Telephone triage nurses cite “not seeing the patient” as a major challenge, impacting quality and speed of their patient assessments and the nature of their interventions. ^3,4 Family caregivers with access to telephone-based triage services for information and emotional support also report that their needs sometimes go unmet. ^{5 –7} The absence of visual nonverbal signals of emotion may partly explain this finding.

Building on prior work examining conversation content and flow between hospice nurses and family caregivers, we explored whether an analog (i.e., plain old telephone service) videophone was capable of providing meaningful data on facial expressions of emotion. We hypothesized that a nurse providing telehospice care via videophone would gain access to rich nonverbal emotional signals from the caregiver and communicate her own social presence to the caregiver, potentially enhancing the building of empathy between nurse and caregiver.

Methods

This report describes an early case exemplar from a larger study using commercially available videophones (Vialta, Milpitas, CA) to enhance communication between after-hours triage nurses and family caregivers of hospice patients. Nurse and caregiver were able to view each other via television or videophone screen during the call. Recording of the videophone conversation produced a record of changing nonverbal expression. This study was conducted with approval from the University of Pittsburgh Institutional Review Board and written informed consent for participation and for publication of images was obtained.

Facial expressions of emotion were measured using the standardized facial action coding system [FACS] ⁸ ), for accurate and complete description of facial expressions in the videophone interaction. FACS provides detailed descriptions of the actions of individual muscles of facial expression (action units [AUs]) coded as changes in appearance of the skin. The typical joyful smile includes AUs in the upper face around the eye (wrinkles at eye corners; AU 6) as well as AUs in the lower face (upturned lip corners; AU 12). Mixed emotion smiles result from the fact that upward lip corner movement in smiling (AU 12) can be counteracted by AUs moving them downward. For this analysis, dampening AUs included AUs 14, 15, 17, and 24. Videophones record frames in a slightly different manner than typical video. Interframe intervals varied, with video refreshed at frequent, though not regularly spaced, intervals of 1/15–1/4 s. Each distinct video frame (separate facial still image) was individually assessed. Because frames may not show independent expressions—an intensification of an existing smile can appear as a newly refreshed frame—facial expression codes in this context are best conceptualized as continuous emotion change (Fig. 1).

OPEN IN VIEWER

Fig. 1.

Emotion expression changes over the course of a telehospice videophone consultation.

Facial movements (AUs) within frames were coded. A subset of frames (20%) was independently coded by two certified FACS coders to establish reliability (interrater agreement=0.85). Remaining frames were assessed by a single coder (K.L.S.). Results were summarized as the number of AUs per frame.

Global assessment of facial emotion was based on AU combinations recognized as emotion indicators. Expressions were assigned an integer label from −2 to 5, ranging from negative emotion (disgust or sadness) through highly positive joy expressions. Labels were assigned as follows: −2, moderate fear (AU 20), disgust (AU 9), anger (AU 4+anger AU), or sadness (AU 1+4); −1, concern (AU 4 only, AU 24 only); 0, neutral; 1, interest, slight surprise (AU 1+2, AU 5); 2, dampened smile (AU 12+dampening AU ² ); 3, social smile (AU 12 only); 4, dampened joyful smile (AU 6+12+dampening AU); 5, joyful smile (AU 6+12). Smiles are typically seen as expressions of positive emotion, but can also exhibit underlying or masked negative emotion. ⁹

Results

During the telehospice call, 104 distinct caregiver frames (expression changes) comprising a total of 522 facial AUs were recorded, for a mean ratio of 5 AU per frame. The nurse had 69 distinct frames, comprising a total of 312 AUs (4.5 AUs per frame). Positive emotion (joy), mixed emotion (dampened smiles and dampened joyful smiles), neutral expression or slight interest, as well as mild-to-moderate negative emotion are displayed in the course of this 28.3-s-long telehospice phone call (Fig. 1). Shifts between emotions are more frequent in the caregiver (41) than in the nurse (23). At the beginning of the call, the nurse displays neutral expression, but the caregiver is positive or mixed. The nurse's first negative emotion is followed by mixed emotion in the caregiver. Caregiver expression becomes more negative after the midpoint of the call. Throughout the call, however, strong positive emotion—undampened joyful smile—is displayed frequently by the nurse and intermittently by the caregiver.

Discussion

This case study provides evidence that emotional expression can be captured and communicated using plain old telephone service–based videophones. Both nurse and caregiver displayed varying facial expressions, including mixed negative and positive emotions. Caregiver expression changed more frequently during the call, reflected in the greater number of distinct frames recorded and greater variety of facial emotions. Both nurse and caregiver smiled frequently, with highly positive joyful smiles and simpler, social, or mixed emotion smiles. The nurse's smiling follows an initial period of smiling by the caregiver. Smiling is a known affiliative behavior enhancing interaction and displaying social intent; thus, social presence does appear to have been experienced on both sides. ¹⁰ Given the sequencing of facial emotions within the call, results also suggest that nurses and caregivers can recognize and respond to visual emotional nonverbal data using this technology. In particular, observations of mixed emotions could be a sign of conflicted feelings in the participant. This display could represent masked negative emotion and thus be an important cue for the nurse to explore. Despite limitations of this study in videorecording quality, the quality of facial emotion information available has important implications for development of existing and future video communication applications in telenursing.