Automatic Detection of Confusion in Elderly Users of a Web-Based Health Instruction Video

Abstract

Background: Because of cognitive limitations and lower health literacy, many elderly patients have difficulty understanding verbal medical instructions. Automatic detection of facial movements provides a nonintrusive basis for building technological tools supporting confusion detection in healthcare delivery applications on the Internet. Materials and Methods: Twenty-four elderly participants (70–90 years old) were recorded while watching Web-based health instruction videos involving easy and complex medical terminology. Relevant fragments of the participants' facial expressions were rated by 40 medical students for perceived level of confusion and analyzed with automatic software for facial movement recognition. Results: A computer classification of the automatically detected facial features performed more accurately and with a higher sensitivity than the human observers (automatic detection and classification, 64% accuracy, 0.64 sensitivity; human observers, 41% accuracy, 0.43 sensitivity). A drill-down analysis of cues to confusion indicated the importance of the eye and eyebrow region. Conclusions: Confusion caused by misunderstanding of medical terminology is signaled by facial cues that can be automatically detected with currently available facial expression detection technology. The findings are relevant for the development of Web-based services for healthcare consumers.

Introduction

Long-standing research in the field of medical communication has shown that the way in which information is conveyed to patients has an impact on several important patient outcomes,^1

–5 including the quality of the diagnosis, patient satisfaction, reported enhanced physical and emotional health, understanding of medical information, recall and compliance/adherence to instructions, and better performance in daily activities, as well as improvements in markers of disease. In fact, next to expert knowledge, the communicative process is considered to be one of the two fundamental components of medical care.³ It is alarming that patients tend to misunderstand or forget 20–75% of the information they receive,⁶ which may result in additional hospital time and treatments for side effects, for instance, due to patients' errors in taking medication.

Misunderstanding medical information may be due to various factors. First of all, when faced with unexpected bad news or having to deal with a new diagnosis, patients may become overwhelmed and confused.⁷ Second, the way information is framed (i.e., positively, rather than negatively) affects the decision-making process, particularly in older patients who tend to rely on heuristics rather than on analytic information processing.⁸ Finally, in particular, older patients may misunderstand the information they receive because of poor health literacy.⁹ Past research has indicated that checks of understanding are infrequent in medical interactions.^10,11 In addition, older patients might be reluctant to signal a communication problem because they value the competence of medical professionals and their authority and prefer not to challenge it in any way.¹² In this context, correct detection of nonverbal cues provides valuable information as to the patients' degree of comprehension. In addition, an automatic detection of confusion is of high relevance for Web-based information services for the elderly¹³ (e.g., in the form of online instruction videos or electronic visits).¹⁴

Several experimental studies have demonstrated that confusion and uncertainty can be detected with the help of facial cues.^15,16 To describe the relevant cues, research in nonverbal communication makes use of various annotation systems. The most common type of facial cue analysis is the Facial Action Coding System (FACS), originally developed by Ekman and Friesen.¹⁷ The FACS offers a comprehensive anatomically based coding of facial expressions by decomposing them into 44 Action Units (AUs). According to Ekman and Friesen,¹⁷ AUs (e.g., lip raise, jaw drop, and blink/eye closure) are the building blocks of facial expressions, including emotional expressions. For example, the facial expression of happiness is detected by the simultaneous presence of cheek raiser (AU 6) and lip corner puller (AU 12). For confusion, studies performed in learning environments involving an online tutor indicated that it is typically associated with a lowered brow (AU 4), tightening of the eye lids (AU 7), and a notable lack of a lip corner puller (AU 12).^18,19

In addition to manual annotations, the AUs can be detected with the Computer Expression Recognition Toolbox (CERT), a toolbox that offers a real-time recognition of the facial units described by the FACS and can perform with 80% accuracy on a dataset of videotaped spontaneous behavior.²⁰ Given that some of the cues may be barely detectable for humans,²¹ the first objective of this study was to compare the results of the automatic detection with the performance of human raters. Because the issue of confusion detection in older age groups is particularly pertinent in the context of doctor–patient interactions, the raters in the perception study were all medical students. The second objective was to identify the relevant cues; we assumed that the same AUs that have been found to be indicative of confusion in learning environments (lowered brow, tightening of the eye lids, lack of a lip corner puller) might also play a role in the detection of confusion in medical interactions involving the elderly.

Materials and Methods

Recordings

In total, 24 participants (50% male) ranging in age from 70 to 90 years (mean=79.6; standard deviation [SD]=6.2) were recorded in the production study. They were recruited in a nursing home in Rotterdam (n=16) and an activity center for the elderly in Tilburg (n=8) in The Netherlands. Preliminary selection was conducted with the help of the organization staff in order to exclude participants suffering from deafness, dementia, and other cognitive impairments that could obstruct the effects of the experimental manipulation. Prior to each session, the participants were informed of the purpose of the study and gave a written consent for the recorded material to be used for research and educational purposes. Approval for the study was obtained from the Institutional Review Board, Faculty of Humanities, Tilburg University, Tilburg. The recruitment and the experimental procedure followed the guidelines of the Institutional Review Board.

All participants were presented with two 2-min medical instruction videos: a complex one and a control video. A pilot of the experimental procedure conducted prior to the study demonstrated that the presentation of the complex video instructions needed to take place after the control condition. The control video served as a baseline for the participants; moreover, presenting the complex condition first in some cases led to participants' unwillingness to continue because the task was perceived as too difficult. In the first video, a female medical professional conveyed information about the symptoms of a health problem (heart attack or stroke, divided equally across the participants) using everyday medical terminology; in the second video, the same speaker presented information about a different problem (stroke/heart attack, counterbalanced) using complex medical terminology. After each video, participants were asked to indicate the comprehensibility of the medical instructions they received on a scale from 0=“not comprehensible” to 10=“comprehensible” and to name at least one symptom of the health problems discussed in the instruction material. The pilot showed that some elderly participants were reluctant to evaluate the complex video critically. Therefore, next to the oral evaluation, an additional form of manipulation check was used in each experimental session, with participants asked to evaluate the instructions in a text form using pen and paper.

The information presented in the videos included the listing of symptoms associated with the health problem. The speaker in the video was looking straight into the camera while slowly reciting the instructions. Prior to the study, the content of the instruction videos and the experimental procedure were pretested with a male and a female participant from the same age group (not part of the experimental group). The recordings of the 24 participants were made with a Samsung (Seoul, Korea) optical image stabilizer (IOS) H300 HD camera located above and behind the computer screen on which the medical instruction videos were played. The distance between the screen and the participant was approximately 70 cm. The volume of the sound recording was adjusted for each participant individually to ensure optimal listening conditions.

Data Evaluation

We first compared the evaluations of the medical instruction videos obtained from the elderly participants. A two-way analysis of variance with Complexity (low, control) and Scenario (heart attack, stroke) as independent variables and the Comprehensibility scale as the dependent variable was conducted to test the effect of the experimental manipulation. With respect to the oral evaluation of the medical instruction videos, complex videos were evaluated as less comprehensible than the control videos: F _1,20=4.67, p=0.04, η_p ²=0.19. The interaction effect of Scenario and Complexity showed a similar effect as the main effect of Complexity (F _1,20=4.67, p=0.04, η_p ²=0.19), with no main effect of Scenario (F _1,20=0.01, p=0.93): the effect of the experimental manipulation was stronger for the scenarios involving stroke (mean_complex=8.22, SD=0.38; mean_control=7.45, SD=0.35) than for the scenarios involving heart attack (mean_complex=7.92, SD=0.32; mean_control=7.92, SD=0.35). The evaluation of the written scenarios using pen and paper showed that complex videos were evaluated as significantly less comprehensible by the elderly viewers than the control videos: F _1,20=57.94, p<0.001, η_p ²=0.74; mean_complex=7.14, SD=0.26; mean_control=8.10, SD=0.21.

Given the measured differences between the complex and control videos, the video recordings were judged to be valid for a perception study by third-party observers, as well as an automatic facial expression analysis. For these purposes, we made use of thin slices (5-s fragments²²), obtained in the complex and control conditions immediately after a complex medical term or its matched “easy” equivalent were used by the health professional in the video. The recordings of one female participant were excluded from the collection because of her positioning of the face outside of the camera view.

Perception Study

The goal of the perception study was to investigate if third-party human observers can correctly interpret elderly viewers' reactions to medical instructions involving complex terminology. The study was conducted with 40 participants (50% male) assigned the task of third-party observers, all advanced medical students at Erasmus University in Rotterdam between the ages of 18 to 26 years (mean=23.9, SD=1.8). They participated on a voluntary basis. Approval for the study was obtained from the Institutional Review Board, Faculty of Humanities, Tilburg University.

The study consisted of two within-participant conditions (Complexity: high and control) with stimulus order randomized per observer. After a brief introduction to the experiment, observers were instructed to evaluate short video fragments without auditory input on a 7-point scale, indicating the perceived level of confusion signaled by the facial cues of the person in the video. The scale was anchored at 1=absolutely [does] not [understand] and 7=[understands] extremely well. At the end of the questionnaire, observers were asked which facial features they considered to be indicative of confusion. To prevent public access to the experimental recordings, the study was conducted with the help of the LimeSurvey questionnaire software installed on a local lab server.

The stimulus set consisted of 92 short fragments collected from 23 elderly viewers with four fragments per viewer: two in the high-complexity condition and two in the control condition. In the high-complexity condition, the stimuli involved 5-s fragments of recordings of elderly viewers collected immediately after an infrequent medical term (e.g., “myocardium” or “diplopia”) was introduced in the medical instruction videos described in the production study. In the control condition, we used fragments of the same length collected after the presentation of matched terms in the control condition, as previously described. The video recordings were displayed in mp4 format fixed to 720×480 size, with a frame rate of 30 frames/s.

Results

To evaluate the ability of human observers to detect confusion, we first examined the effect of the within-participant factor Complexity (two levels: high and control) and observer's Gender as the between factor on the perceived level of confusion. A mixed design analysis of variance showed a significant main effect of Complexity: F _1,38=38.72, p<0.001, η_p ²=0.51. The elderly viewers in fragments from the high-complexity condition were evaluated as showing a lower level of understanding of the received instructions (mean=2.83, SD=0.05) than viewers from the control condition (mean=3.01, SD=0.06). There was no main effect of observer's Gender (F _1,38<1, p=0.63) and no interaction effect of Complexity and Gender (F _1,38<1, p=0.50).

In order to calculate the human observer sensitivity to confusion cues, we created a new binomial variable based on the average value of all human judgments (mean=2.93, SD=0.07). A video stimulus was classified in the category “understanding” if the mean value was higher than the median=2.91 of the average value of judgments; it was classified as “not understanding” if the mean value was lower than or equal to the median. Calculated on the basis of the median values, the success of the human observer classification was 41%, with sensitivity to confusion (probability that a complex video was classified into the “not understanding” category) of 0.43.

In order to explore possible individual differences in the display of confusion, we made use of a linear mixed-model analysis with Complexity as the fixed factor and Viewer ID, the identity of the elderly viewer in the recording, as a random factor. The dependent variable was the mean perceived level of understanding, averaged over all observers in the perception study. The relationship between complexity of instructions and perceived level of understanding showed significant variance in intercepts across elderly viewers: var(u_0j )=0.32, χ ²(1)=55.91, p<0.001. The results of the mixed-model analysis, summarized in Table 1, show that not all expressions of confusions were decoded as such, with some elderly viewers exhibiting facial behavior that was incorrectly interpreted by the third-party observers.

Table 1.

Mixed-Models Results of the Perception Study

PARAMETER	ESTIMATE	SE	DF	T	WALD Z	SIGNIFICANCE	95% CONFIDENCE INTERVAL (LOWER BOUND, UPPER BOUND)
Estimates of fixed effects	−0.2557	0.0760	65	−3.36		0.001	−0.4075, −0.1039
Intercept	3.1011	0.1316	24.93	23.57		<0.001	2.8301, 3.3722
Estimates of covariance parameters
Residual	0.1271	0.0223			5.70	<0.001	0.0901, 0.1792
Intercept	0.3174	0.1079			2.94	0.003	0.1630, 0.6180

SE, standard error.

We next performed computational analyses of the videos²³ to determine (1) the extent to which facial expressions provide cues to confusion regarding medical instructions, (2) the degree of agreement between the computer performance and human performance, and (3) which facial cues are diagnostic for confusion detection, using the CERT.²⁰

For each video frame, CERT detected the face region and analyzed the face in order to determine the presence of 20 AUs. When presented with a video containing a human face, CERT outputs (among other measures) 20 real-valued AU scores. To determine the extent to which the facial expressions of the elderly participants reveal their confusion, we trained a learning algorithm, the 1-nearest neighbor classifier (e.g., Hastie et al.²⁴), on the binary task of determining if a 5-s fragment corresponds to a participant that just heard a control (class 1) or a complex instruction (class 2). The classifier was trained using CERT AU scores averaged over the 5-s period (i.e., given a frame rate of 30, the averages were taken over 150 AU estimates). A so-called leaving-one-subject-out cross-validation procedure was used to assess the performance of the classifier.²⁴ This procedure provides a reliable estimate of the prediction accuracy, a measure of how well the trained classifier predicts unseen video fragments. A prediction accuracy of 50% would indicate that the trained classifier performs at chance level, whereas a prediction accuracy of 100% indicates perfect prediction.

Using the 20 AUs, the prediction performance obtained was higher than the performance in the perception study with human observers. The facial expressions provided cues to the difficulty of the instructions, in that the confusion of a previously unseen participant could be inferred with an estimated certainty of 64%. This compares favorably with the chance level of 50% and with the human observer performance of 41%. The sensitivity (true-positive rate) to confusion was 0.64, compared with the human observer sensitivity of 0.43. Human observers and the computer classifier were in agreement for 59% of the videos.

Subsequently, we determined the visual cues that were most informative in predicting confusion. To this end we performed feature selection by means of exhaustive grid search. Instead of taking (averages of) all 20 AUs as inputs to the classifier, we determined the prediction performances for individual AUs, all possible pairs, triples, and quartets of AU. (Evaluating the prediction performance obtained with combinations of 5 or more AUs would be too computationally demanding to be performed exhaustively.) The intention was to find those combinations of AUs that would give the best prediction performances because they would form the most informative visual cues for confusion. (It is important to note that the prediction performances obtained in feature selection should not be interpreted as indicating how well we are able to predict confusion. Instead, they indicate how well we would have predicted confusion if we had known the optimal AUs beforehand.) Table 2 lists the optimal combinations of AUs obtained in our feature selection experiment. For a single AU, chin-raising was the most informative visual cue, yielding a prediction performance of 63%. For 2, 3, and 4 AUs, the optimal combinations of AUs consist of lip-related visual cues (lip corner pull, lip pucker, and lip stretch), jaw drop (for 2 and 3 AUs), and cheek and eyebrow cues. Examination of the (nonaveraged) intensities for the optimal combination of 2 AUs (lip corner pull and jaw drop) revealed considerable individual differences in the separation between the scores in the two conditions.

Table 2.

Results of the Feature Selection Experiment

NUMBER OF AUS	OPTIMAL COMBINATION OF AUS	PREDICTION PERFORMANCE
1	Chin raise (AU 17)	63%
2	Lip corner pull (AU 12)	68%
	Jaw drop (AU 26)
3	Lip corner pull (AU12)	68%
	Lip pucker (AU 18)
	Jaw drop (AU 26)
4	Brow lower (AU 4)	72%
	Cheek raise (AU 6)
	Lip corner pull (AU 12)
	Lip stretch (AU 20)

The left column list the number of action units (AUs) (out of the total of 20 AUs) for which the optimal combination was sought. The center column lists the corresponding number of AUs that yield the best prediction performance. The right column lists the prediction performances.

Discussion

Misunderstanding of medical information due to patients' low health literacy or cognitive limitations is a relatively common phenomenon in medical interactions. In our study, we investigated if confusion can be automatically detected from the facial cues and compared the performance of automatic classification with that of human observers. The analysis of facial movements was performed with the help of a computer toolbox that annotates facial movements within the FACS system. FACS has been widely used in research on spontaneous and simulated facial expressions of emotions and several mental and physical health variables, including expressions of pain.²⁵ Manual annotation of the AUs is laborious, but recent developments in the field of affective computing demonstrate that automatic annotation is possible.

In our study, the performance of the automatic detection analysis was slightly better than the human performance, with the combinations of 4 AUs (AU4, brow lower; AU6, cheek raise; AU12, lip corner pull; and AU20, lip stretch) achieving the highest score. These results are in agreement with cues to confusion identified in other settings.¹⁹ They are also comparable to the findings reported by Durso et al.,¹⁶ who explored perceived expressions of confusion by means of surface facial electromyography (a technique that uses electrodes placed on the skin to detect electrical signals emitted by muscles when they contract). In their study, confusion was mainly associated with the movement of right and left corrugator supercilii (eyebrow movement) and the right depressor anguli oris (right lip corner pull), movements typically involved in negatively valenced expressions.²⁶

Statistical analysis of perceived levels of understanding indicated significant individual differences in the encoding of confusion, with some elderly participants being judged inversely to the original design. It is interesting that individual variety in nonverbal cues has been observed in the past for expressions of confusion,²⁶ with some participants making primarily use of the eyebrow regions, whereas others were especially expressive in the mouth area. The results of our study suggest, however, that facial cues associated with confusion may, at least in some cases, be misinterpreted for expressions of understanding by human observers.

Future research should focus on the origin and motivation for misperceptions due to the ambiguity of some facial expressions. For example, it could be the case that some of the recorded older individuals engaged in prosocial behavior to hide negative emotions associated with misunderstanding. Deciphering the mechanisms behind an emotion/stand display is likely to support the process of patient empowerment and improved decision making. Second, the use of Web-based health instruction videos is by no means limited to a particular age category. In the past, Web-based health programs have been tested not just with adults, but also with children²⁷ and adolescents.²⁸ Apart from dealing with different sources of comprehension difficulties, these age groups may, in fact, signal confusion differently because of different degrees of expressivity as a function of variance in emotional intensity²⁹ and emotion regulation³⁰ when contrasted with older adults. Therefore, it is important to compare their use of nonverbal cues in follow-up studies that would include age-suitable methods for collecting self-reported measures. Finally, the use of automatic confusion detection methods could potentially be beneficial during the development and testing phase of Web-based health applications, providing an alternative to performance-based measures and thinking out loud protocols currently used in user experience research.

Conclusions

Confusion caused by misunderstanding of medical terminology is signaled by facial cues that can be automatically detected with currently available facial expression detection technology. Automatic detection of facial movements provides a nonintrusive basis for building technology used in healthcare delivery applications on the Internet, including personalized health information and electronic visits.

Footnotes

Acknowledgments

The authors would like to thank Elise Peters and Tineke van den Hoek for their help with collecting the data for this study, as well as all the participants for their time.

Disclosure Statement

No competing financial interests exist.

References

Laine

, Davidoff

, Lewis

, Nelson

, Kessler

, et al. Important elements of outpatient care: A comparison of patients' and physicians' opinions. Ann Intern Med, 1996; 125:640–645.

Pawlikowska

, Zhang

, Griffiths

, Van Dalen

, Van der Vleuten

. Verbal and non-verbal behavior of doctors and patients in primary care consultations—How this relates to patientenablement. Patient Educ Couns, 2012; 86:70–76.

Roter

, Hall

. Doctors talking with patients/patients talking with doctors: Improving communication in medical visits. Portsmouth, NH: Greenwood Publishing Group, 2006.

Stewart

. Effective physician-patient communication and health outcomes: A review. CMAJ, 1995; 152:1423–1433.

Zandbelt

, Smets

, Oort

, Godfried

, de Haes

. Satisfaction with the outpatient encounter: A comparison of patients' and physicians' views. J Gen Intern Med, 2004; 19:1088–1095.

Jansen

. Communicating with older cancer patients: Impact on information recall [PhD dissertation]. Utrecht, The Netherlands: Utrecht University, 2009.

Jacobsen

, de Lima

Thomas J

, Jackson

. Misunderstandings about prognosis: An approach for palliative care consultants when the patient does not seem to understand what was said. J Palliat Med, 2013; 16:91–95.

Zamarian

, Benke

, Buchler

, Wenter

, Delazer

. Information about medications may cause misunderstanding in older adults with cognitive impairment. J Neurol Sci, 2010; 298:46–51.

Marks

, Schectman

, Groninger

, Plews-Ogan

. The association of health literacy and socio-demographic factors with medication knowledge. Patient Educ Couns, 2010; 78:372–376.

10.

Jenkins

, Solis-Trapala

, Langridge

, Catt

, Talbot

, Fallowfield

. What oncologists believe they said and what patients believe they heard: An analysis of phase I trial discussions. J Clin Oncol, 2011; 29:61–68.

11.

West

, Baile

. “Tell me what you understand:” The importance of checking for patient understanding. J Support Oncol, 2010; 8:216–218.

12.

Ekdahl

, Andersson

, Friedrichsen

. “They do what they think is the best for me.” Frail elderly patients' preferences for participation in their care during hospitalization. Patient Educ Couns, 2010; 80:233–240.

13.

Jones

, Cassie

, Thompson

, Atherton

, Leslie

. Delivering healthcare information via the Internet: Cardiac patients' access, usage, perceptions of usefulness, and Web site content preferences. Telemed J E Health, 2014; 20:223–228.

14.

Albert

, Shevchik

, Paone

, Daniel

. Internet-based medical visit and diagnosis for common medical problems: Experience of first user cohort. Telemed J E Health, 2011; 17:304–308.

15.

Krahmer

, Swerts

. How children and adults produce and perceive uncertainty in audiovisual speech. Lang Speech, 2005; 48:29–53.

16.

Durso

, Geldbach

, Corballis

. Detecting confusion using facial electromyography. Hum Factors, 2012; 54:60–69.

17.

Ekman

, Friesen

. Unmasking the face: A guide to recognizing emotions from facial cues. Upper Saddle River, NJ: Prentice Hall, 1975.

18.

Grafsgaard

, Boyer

, Phillips

, Lester

. Modeling confusion: Facial expression, task, and discourse in task-oriented tutorial dialogue. Artif Intell Educ, 2011; 6738:98–105.

19.

McDaniel

, D'Mello

, King

, Chipman

, Tapp

, Graesser

. Facial features for affective state detection in learning environments. In: McNamara

, Trafton

, eds. Proceedings of the 29th Annual Meeting of the Cognitive Science Society. Austin, TX: Cognitive Science Society, 2007; 467–472.

20.

Littlewort

, Whitehill

, Wu

, Fasel

, Frank

, Movellan

, Bartlett

. The Computer Expression Recognition Toolbox (CERT). IEEE Int Conf, 2011; 298–305.

21.

Visser

, Postma-Nilsenová

, Krahmer

, Swerts

. Nonverbal cues of meta-memory awareness in older adults. In: Bello

, Guarini

, McShane

, Scassellati

, eds. Proceedings of the 36th Annual Meeting of the Cognitive Science Society. Austin, TX: Cognitive Science Society, 2014; 1694–1699.

22.

Ambady

, Rosenthal

. Half a minute: Predicting teacher evaluations from thin slices of nonverbal behavior and physical attractiveness. J Pers Soc Psychol, 1993; 64:431–441.

23.

Mattheij

, Postma-Nilsenová

, Postma

. Vocal and facial imitation of humans interacting with virtual agents. In: Affective computing and intelligent interaction. New York: IEEE, 2013; 815–820.

24.

Hastie

, Tibshirani

, Friedman

. The elements of statistical learning. Data mining, inference, and prediction. New York: Springer-Verlag, 2009.

25.

Ekman

, Rosenberg

. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). New York: Oxford University Press, 1997.

26.

Rozin

, Cohen

. High frequency of facial expressions corresponding to confusion, concentration, and worry in an analysis of naturally occurring facial expressions of Americans. Emotion, 2003; 3:68–75.

27.

Palmer

, Graham

, Elliott

. Effects of a Web-based health program on fifth grade children's physical activity knowledge, attitudes and behavior. Am J Health Educ, 2005; 36:86–93.

28.

Long

, Stevens

. Using technology to promote self-efficacy for healthy eating in adolescents. J Nurs Scholarsh, 2004; 36:134–139.

29.

Carstensen

, Pasupathi

, Mayr

, Nesselroade

. Emotional experience in everyday life across the adult life span. J Pers Soc Psychol, 2000; 79:644–655.

30.

Charles

, Carstensen

. Emotion regulation and aging. In: Gross

, ed. Handbook of emotion regulation. New York: Guilford Press, 2009:307–329.