Identifying Anxiety Through Tracked Head Movements in a Virtual Classroom

Abstract

Virtual reality allows the controlled simulation of complex social settings, such as classrooms, and thus provides an opportunity to test a range of theories in the social sciences in a way that is both naturalistic and controlled. Importantly, virtual environments also allow the body movements of participants in the virtual world to be tracked and recorded. In the following article, we discuss how tracked head movements were correlated with participants' reports of anxiety in a simulation of a classroom. Participants who reported a high sense of awareness of and concern about the other virtual people in the room showed different patterns of head movement (more lateral head movement, indicating scanning behavior) from those who reported a low level of concern. We discuss the implications of this research for understanding nonverbal behavior associated with anxiety and for the design of online educational systems.

Introduction

Many social elements in the classroom, including characteristics of teachers and other students, can affect learning. The impact of such factors on students' state of mind can be measured in a variety of ways, such as a postexperiment survey or behavioral methods. One set of important behavioral measures is body movements.

Body movements can be measured by having trained observers watch participants or videos, or by automatically tracking and recording movements. Analyzing body movements automatically allows researchers to look for patterns in behavior that may not be apparent to a human observer and allows researchers to assess these patterns across a large sample size, in real time. If learners' patterns of movement are correlated with their mental states, and such patterns can be identified, this has important implications for the design and analysis of the next generation of digital learning environments.¹

Much previous research supports this. For example, a study in a virtual classroom used the measurement of total body movement to distinguish children with attention deficit hyperactivity disorder (ADHD), who moved more, from children without such a diagnosis, who moved less.² Body movements of teacher and student have been shown to be predictive of outcome in a teaching and learning task,³ and automatically detected posture of students to be predictive of attention.⁴

Feeling like one belongs in a learning environment is important in virtual as well as physical classrooms.⁵ However, some learners feel a sense of anxiety rather than a sense of belonging. Anxiety, which has been shown to increase arousal and diminish performance on difficult tasks,⁶ is another state of mind that may be revealed by gesture.

In a 2006 meta-analysis of attention to threatening stimuli, the authors state “there is a consensus at the theoretical level that anxiety is associated with biases in attending to threat-related information.”⁷ In our data set, we could infer gaze from head movement.⁸ We might then expect that participants who are anxious about some aspect of the classroom environment might reveal their anxiety through head movement as they attend to the source or sources of their anxiety.

We examined an unpublished data set collected in a study on classroom gender composition. In this experiment, participants took a short lesson on computer science in a virtual classroom filled with digital agents. Participants were told that these agents were avatars controlled by other student participants at other locations and that their own avatar was similarly visible to these other participants.^a Participants took a post-test on the material covered in the lesson and completed a survey on their feelings about the experience. Our research question was whether body movements, in particular head movements, would correlate with self-reported anxiety and presence in a virtual classroom, and if we could find meaningful patterns in the data that might account for such correlations.

We discuss our findings, provide an explanation for the patterns observed, and discuss the implications for both online and in the real world.

Materials and Methods

Participants

Fifty-seven participants (37 female) between the ages of 18 and 31 were recruited from the graduate and undergraduate student population of a medium-sized private university. To reduce the likelihood of significant prior knowledge of the material, STEM majors were not eligible for the study.

Materials

Participants donned a head-mounted display (HMD) with integrated headphones through which they could see and hear the virtual environment. The HMD nVisor SX60 presents one LCD panel with 1,280 horizontal pixels and 1,024 vertical pixels for each eye, with a 60° diagonal field of view. An IC3 Intersense cube tracked the pitch, yaw and roll of the participant's head, and an optical infrared tracker captured the X, Y, Z position of the head. This tracking information was used to continuously update the simulated viewpoint and was recorded at 60 frames a second. The frame update rate was 60 frames per second, and the system latency was between 40 and 50 ms across all components. Vizard 4.0 software was used to assimilate tracking and rendering. Wrist trackers were also placed on the participant, but the trackers were not used to update information in the HMD, only to gather information about the participant's body movement. Figure 1 shows a representative participant in the HMD.

FIG. 1.

A representative participant wearing the HMD and marker in the experiment room. HMD, head-mounted display.

On donning the headset, participants saw a virtual classroom (Figs. 2 and 3) that appeared to be filled with avatars controlled by other students. These were actually virtual agents controlled by the computer, selected from the stock avatars provided via Vizard HD Complete Characters. Participants could look around the room freely but were instructed to remain seated.

FIG. 2.

Shows the classroom and other students from the point of view of the participant. The image on the left shows the teacher lecturing, and the image on the right shows an example slide.

FIG. 3.

Bird's eye view of the classroom, showing participant position.

Design and procedure

Procedure

Before entering the virtual reality classroom, participants completed a pretest, including questions on major and gender. (Because the experiment focused on gender composition, gender was included on the pretest to make sure that it was salient.) Participants were then told that the experiment was testing a virtual reality implementation of online classes. This was to ensure that the participants' feelings about their virtual colearners would be similar to how they felt about fellow students in the real world. Therefore, although the other “students” were agents whose movements were controlled by the computer, participants were told that the avatars were controlled by other participants in the research study located at other sites. They were also told that experimenters had selected avatars for all participants that matched their real-world appearance and that, while their own avatar was not visible to them, it would be visible to other participants in the virtual classroom. Manipulating the belief that a virtual human is controlled by a real person or an agent is common in virtual reality experiments.⁹ Finally, participants were told to pay attention to learn the material, as in a real classroom.

Wrist trackers and the HMD were placed on the participant. The participant was then directed to “look around and get comfortable in the environment” while other participants connected to the shared virtual classroom. At the end of 2 minutes, the participant was informed that there was a connection problem that needed to be debugged. Participants were instructed to look at the “classmates” on either side to see if they were nodding and to nod at the instructor in the front of the view. The participant was then informed, “it looks like the problem is between us and the other sites. The tech is trying to fix the connection.” Twenty seconds later, the participant was informed that the problem was likely fixed and asked to look at the agents on either side again. This time the participant could see the agents nodding. The participant was then asked to nod at the instructor. Shortly thereafter, the lecture commenced. The purpose of these manipulations was twofold: first, to ensure that, regardless of their movements during the “getting comfortable” phase of the environment, all participants would have moved their gaze across the entire classroom at least once, and second, to re-emphasize that the participants were seeing avatars controlled in real time by other students in a networked virtual classroom.

Participants then saw a 10-minute lecture explaining the binary number system, a nonprogramming computer science topic, delivered by a male agent “teacher.” Afterward, the participant was removed from the HMD and wrist trackers and directed back to the adjacent survey room to take the post-test.

Measures

The X, Y, Z position of participants' heads and hands, and the pitch, yaw, and roll of the head were recorded for the entire interaction. Pitch represents the movement of the head around the X-axis, as when nodding the head. Yaw represents the movement of the head around the Y-axis, as when shaking the head to indicate “no.” Roll represents the movement of the head around the Z-axis, as when tilting one's head to the side. Figure 4 shows these movements.

FIG. 4.

Demonstrates head rotations: pitch, yaw, and roll.

Participants' experiences in the virtual classroom consisted of two components. The first was the 2-minute period during which they were examining the virtual classroom. The second was the 10-minute lecture. To distinguish these different experiences, we separated the data into the “look-around” period and the “learning” period, removing the portion of the data that contained the experimenter-directed movements of back and forth head movements and up-and-down nods. Following Won et al.,³ we created summary measures from our tracked data for both periods: mean, standard deviation, and skewness. Standard deviation, or “Scanning,” reflects the tendency of the participant to move their head, changing the direction of their gaze horizontally in the virtual environment. Skewness reflects the deviance of the gaze from the mode, or where the participant was most frequently looking. For example, if the participant were mostly looking at the front of the classroom, but occasionally glanced out the window, those glances would show in a change of the skewness of the gaze toward the window.

Our outcome measures consisted of two main components: social anxiety and presence. As part of the post-test following participants' experience in the virtual classroom, they were asked to respond to a variety of statements using a seven-point Likert scale from strongly disagree to strongly agree.

Because the main experiment examined anxiety as derived from stereotype threat based on gender, the Anxiety scale consisted of the mean of six items (Cronbach's α = 0.76) adapted from prior work on feelings of belonging and anxiety in learning environments¹⁰ such as “In the virtual classroom, I wondered what the other students thought of me” and “While in the virtual classroom, I felt self-conscious about my gender.”

Presence has been linked to affective states in virtual reality by a number of researchers,¹¹ particularly when the affective state is anxiety or fear.¹² Thus, we also examined participants' self-report of different categories of presence, such as environmental (or spatial) presence, social presence, and self-presence.¹³ We adapted a four-item scale to measure environmental presence.¹⁴ We also asked four questions each to capture the sense of social presence with the student agents (Student Social Presence) and with the teacher agent (Teacher Social Presence).¹⁵ Combining these environmental and social presence measures gave us a 12-item presence scale (Cronbach's α = 0.92). To create the general measure “Presence,” we took the mean of all 12 items.

Please see Appendix A for the wording of all questions.

Results

Since participants received instructions to remain seated during the course of the experiment, limiting translation, we did not use the measures of head position for further analysis. The other measures from the tracking data were the pitch, yaw, and roll of the head. By observation, all participants remained relatively still during the lesson, directing their gaze toward the lecturer. In contrast, students were more active in the first section of the experiment, described as “getting comfortable in the classroom.” Thus, we focused on head movements in the first section as an indicator of participants' feelings in the classroom environment.

Visual inspection of the roll, pitch, and yaw over time revealed different patterns of movement. While the roll and pitch data were relatively static, suggesting that participants did not significantly nod or tilt their heads, the yaw data varied. Some examples can be seen in Figure 5. During their first, unstructured minutes in the classroom, some participants remained relatively still, while other participants were more active, repeatedly shifting their gaze around the environment. To summarize these movement differences, we created a variable called Scanning. This variable consisted of the standard deviation of the yaw measurement over a 100-second period of the prelecture time after the instructor-directed movements.

FIG. 5.

These plots are derived from head movements that the experimenters intentionally produced by enacting low/high scanning behavior. The plot on the left shows head movements that produce a high standard deviation of 45.5. The plot on the right shows head movements that produce a low standard deviation of 19.9 (right). The Y axis represents the angle of the head, and the X axis represents time.

To verify our interpretation of these results, we created two test files as shown in Figure 5. These files were created by an experimenter donning the HMD and scanning the room repeatedly (left side, Fig. 5a) and then briefly glancing around the room (right side, Fig. 5b). The high-movement file showed a high standard deviation, while low movement had a low standard deviation.

Overall, there was a positive correlation between Anxiety and Scanning (the standard deviation of yaw) (r = 0.33, p = 0.011). There was no correlation between Scanning and the average of all the questions measuring Presence (r = −0.02, p = 0.865). However, there was a positive correlation between Presence and Anxiety (r = 0.28, p = 0.034), consistent with other findings on Presence and Anxiety.¹⁶ Breaking this down further, we found a positive correlation between the average of the four questions on Student Social Presence and Anxiety (r = 0.36, p = 0.006). We interpret these findings to mean that Scanning or the head movements characterized by the standard deviation of yaw were specifically linked to participant anxiety about the agents, rather than just participants' belief that the agents in the room with them were real. However, to feel socially anxious about digital agents, the viewer must feel that those agents are socially present.

We broadened our analysis to investigate what the correlations might represent. Because of the perception of computer science as a masculine field,¹⁷ the classroom environment might be expected to make female participants more uncomfortable, a feeling that may be exacerbated by gender being made salient in the pretest as a part of the original experiment. Therefore, we next examined the difference between female and male participants. There was indeed a significant effect of gender on feelings of anxiety [F(1, 55) = 8.66, p = 0.005, Adjusted R Squared = 0.12]. In addition, when the genders were separated, the correlation between anxiety and head movement was statistically significant for women (r = 0.44, p = 0.006) but not for men (r = 0.28, p = 0.227). However, there was no difference overall in Scanning, the standard deviation of head movement about the y-axis between women and men [F(1, 55) = 0.59, p = 0.446, Adjusted R Squared = −0.01]. Correlations between each measure by gender are shown in Tables 1 and 2, and a plot showing the relationship between Scanning and Anxiety in both genders is shown in Figure 6.

FIG. 6.

Plot showing the relationship between Scanning and Anxiety in both genders.

Table 1.

Correlations Between Variables for Male Participants

	Anxiety	Social presence	Scanning
Anxiety	1.00	0.32	0.28
Social presence	0.32	1.00	−0.06
Scanning	0.28	−0.06	1.00

Table 2.

Correlations Between Variables for Female Participants

	Anxiety	Social presence	Scanning
Anxiety	1.00	0.43^**	0.44^**
Social presence	0.43^**	1.00	0.11
Scanning	0.44^**	0.11	1.00

p < 0.01

By visual comparison of the traces of head movement over the course of the entire interaction, female participants who reported feeling concerned about how the other digital humans in the room perceived them had a distinctive pattern of head movement, as can be seen in Figures 7a and b.

FIG. 7.

The first six images (a) show the traces of Scanning head movements over time of female participants who reported the highest level of anxiety (scores above 4.5). The remaining five images (b) show the traces of Scanning head movements over time for female participants who reported the lowest level of anxiety (scores below 2.0). The Y axis represents the angle of the head, and the X axis represents time.

Discussion

In this exploratory study, we demonstrated that participants' self-reported feeling of social anxiety that they felt in a virtual classroom could be correlated to specific patterns of tracked movements that related to real-life behaviors. On visual inspection, the head movements of participants who reported high anxiety in the virtual classroom differed from those of low-anxiety participants. This difference was captured by the standard deviation of the “yaw” movements, indicating scanning of the environment.

Given the limitations of this study, we cannot determine the cause of participant anxiety. However, we can consider some possibilities. Based on the higher levels of reported anxiety by female participants, one possibility is that gender was part of their feelings of anxiety, especially since gender was purposely made salient in the experiment from which this data set was taken. Prior research¹⁸ has shown that computer science is often perceived as a “male” discipline, and so, the perceived mismatch between participants' gender and the material presented in the classroom may also have been a factor.

Another possibility is that since participants in this study were selected for their non-STEM majors, the presentation of material outside their areas of study may have provided an additional source of anxiety. While this situation would not normally be considered to lead to stereotype threat,¹⁹ the recently developed Multi-Threat Framework²⁰ provides a new model to suggest that a form of stereotype threat may explain anxiety in participants not highly associated with a given domain.

However, the proposed relationships between anxiety and gender in this study remain tentative. For this reason, future work should more precisely determine the source of participants' reported anxiety. For example, is the relationship between anxiety and movement typical for women in all environments, or only environments that are designed to make gender salient? Do men also demonstrate such behaviors in learning environments not traditionally part of a male domain? Or can these behaviors be used more broadly to characterize social anxiety?

Finally, it is notable that this experiment was performed with a HMD that weighed ∼1 kg. Thus, head movements in the “pitch” and “roll” axes may have been constrained by the weight of the headset. Future work should utilize the new generation of lighter HMDs that may allow the capture and measurement of more subtle head movements.

We feel that the greatest contribution of this work is identifying anxiety unobtrusively using the inherent tracking capabilities of HMDs while in a virtual classroom. This provides a useful supplement to self-report about a state of mind that is important for learning in general. Using this tracking input allows for behavior to be assessed in real time, while the user is actually in the virtual environment. Given that virtual reality may be uniquely adapted to assess and mitigate threat,²¹ these findings have important implications for the design of virtual and real classrooms. However, this potential measure of anxiety may also be applied using other tracking methodology in non-VR classrooms or other public spaces such as hospitals. When people's internal mental states can be assessed using automatic and unobtrusive measures, then we may speculate on how environments may adapt to address their needs before such feelings of anxiety become insurmountable.

Footnotes

Acknowledgments

The work presented herein was funded, in part, by Konica Minolta as part of a Stanford Media-X grant and by the Amir Lopatin Fellowship. The conclusions reached are those of the investigators and do not necessarily represent the perspectives of the funders. We thank Konica Minolta for the valuable insights provided by their visiting researchers, especially Dr. Haisong Gu. We also thank the staff of the Stanford Virtual Human Interaction Lab (VHIL), especially lab manager Cody Karutz, as well Mark Peng and Melvin Low for their programming assistance.

Notes

a. The percentage of female classmates in the virtual classroom varied to induce various levels of anxiety through stereotype threat. Participants either had 100 percent male, 75 percent male, 50 percent female, 75 percent female, or 100 percent female classmates. However, there was no significant effect of number of female colearners on participants' feelings of anxiety or movement. Our findings presented here collapse these five conditions into one.

Author Disclosure Statement

No competing financial interests exist. Funding for this study was provided by the Amir Lopatin Fellowship.

Appendix A. Self-Report

All self-report questions were on a 1–7 Likert Scale, and all were preceded by the following statement: “Please rate your agreement with the following statements.”

References

D'Mello

, Picard

, Graesser

. Towards an affect-sensitive autotutor. IEEE Intelligent Systems, 2007; 22:53–61.

Parsons

, Bowerly

, Buckwalter

, et al. A controlled clinical comparison of attention performance in children with ADHD in a virtual reality classroom compared to standard neuropsychological methods. Child Neuropsychology, 2007; 13:363–381.

Won

, Bailenson

, Janssen

. Automatic detection of nonverbal behavior predicts learning in dyadic interactions. Affective Computing, IEEE Transactions on, 2014; 2:112–125.

D'Mello

, Graesser

. Automatic detection of learner's affect from gross body language. Applied Artificial Intelligence, 2009; 23:123–150.

Cheryan

, Plaut

, Davies

, et al. Ambient belonging: how stereotypical cues impact gender participation in computer science. Journal of Personality and Social Psychology, 2009; 97:1045–1060.

Cheryan

, Meltzoff

, Kim

. Classrooms matter: the design of virtual classrooms influences gender disparities in computer science classes. Computers & Education, 2011; 57:1825–1835.

Stiefelhagen

, Yang

, Waibel

. Estimating focus of attention based on gaze and sound. In: Proceedings of the 2001 workshop on perceptive user interfaces. Orlando, FL: ACM, pp. 1–9.

Bar-Haim

, Lamy

, Pergamin

, et al. Threat-related attentional bias in anxious and nonanxious individuals: a meta-analytic study. Psychological bulletin, 2007; 133:1–24.

Bailenson

, Blascovich

, Beall

, et al. Interpersonal distance in immersive virtual environments. Personality and Social Psychology Bulletin, 2003; 29:1–15.

10.

Murphy

, Steele

, Gross

. Signaling threat how situational cues affect women in math, science, and engineering settings. Psychological Science, 2007; 10:879–885.

11.

Riva

, Mantovani

, Capideville

, et al. Affective interactions using virtual reality: the link between presence and emotions. CyberPsychology & Behavior, 2007; 10:45–56.

12.

Diemer

, Alpers

, Peperkorn

, et al. The impact of perception and presence on emotional reactions: a review of research in virtual reality. Frontiers in psychology, 2015; 6:26.

13.

Lee

. Presence, explicated. Communication Theory, 2004; 14:27–50.

14.

Bailenson

, Beall

, Loomis

, et al. Transformed social interaction: decoupling representation from behavior and form in collaborative virtual environments. Presence: Teleoperators and Virtual Environments, 2004; 13:428–441.

15.

Bailenson

, Yee

. A longitudinal study of task performance, head movements, subjective report, simulator sickness, and transformed social interaction in collaborative virtual environments. Presence: Teleoperators and Virtual Environments, 2006; 15:699–716.

16.

Kirschner

, Williams

. (2014) A microsociological perspective on non-verbal communicative strategies in MMORPGs In: Tanenbaum

, El-Nasr

, Nixon

, eds. Nonverbal communication in virtual worlds: understanding and designing expressive characters. Pittsburgh, PA: ETC Press.

17.

Shapiro

. (2012) Types of threats: from stereotype threat to stereotype threats. In Schmader

, Inzlicht

, eds. Stereotype threat: theory, process, and application. New York: Oxford University Press, pp. 71–88.

18.

Correll

. Constraints into preferences: gender, status, and emerging career aspirations. American Sociological Review, 2004; 69:93–113.

19.

Steele

, Aronson

. Stereotype threat and the intellectual test performance of African Americans. Journal of Personality and Social Psychology, 1995; 69:797.

20.

Shapiro

, Neuberg

. From stereotype threat to stereotype threats: implications of a multi-threat framework for causes, moderators, mediators, consequences, and interventions. Personality and Social Psychology Review, 2007; 11:107–130.

21.

Parsons

, Rizzo

. Affective outcomes of virtual reality exposure therapy for anxiety and specific phobias: a meta-analysis. Journal of Behavior Therapy and Experimental Psychiatry, 2008; 39:250–261.