Short report: Evaluating the safety and usability of head-mounted virtual reality compared to monitor-displayed video for children with autism spectrum disorder

Abstract

Virtual reality provides a relatively inexpensive way to learn and repeatedly practice skills in personalized, controlled, and safe computer-generated settings. These systems are increasingly receiving attention as an innovative medium for delivering interventions to children with autism spectrum disorder. Although many virtual reality systems are commercially available and their use is increasing, little is known about the safety and usability of these systems for children with autism spectrum disorder. The aim of this study was a first step in addressing this gap. A convenience sample of 35 children with a diagnosis of autism spectrum disorder participated in an immersive head-mounted display virtual reality experience and a control condition (monitor-displayed video). Levels of anxiety and negative effects experienced were not significantly different between the two conditions. Participants reported significantly enhanced spatial presence (p = 0.003; d = 0.3) and naturalness (p = 0.002; d = 0.47) for the head-mounted display–virtual reality condition, and 74% of participants preferred using head-mounted display–virtual reality over monitor-displayed video. These findings provide preliminary evidence to support the safety and usability of head-mounted display–virtual reality for children with autism spectrum disorder. Future studies are needed to replicate the results in a larger sample, a range of virtual reality experiences, and in the context of long-term exposure.

Lay abstract

This study investigated the safety and usability of a virtual reality experience for children with autism spectrum disorder in a laboratory setting. In our study, the negative effects of head-mounted display–virtual reality were similar to monitor-displayed video watching. At the same time, the participants indicated that the head-mounted display–virtual reality experience provided improved realism and sense of presence. This study is a first step in understanding the impact of head-mounted display on children with autism spectrum disorder.

Keywords

anxiety autism spectrum disorder autistic disorder children technology virtual reality

Introduction

Virtual reality (VR) is becoming increasingly common as a medium for delivering interventions to individuals with autism spectrum disorder (ASD). VR is a computer-generated, interactive environment that simulates the real world by presenting the user with three-dimensional imagery. These systems can provide elevated levels of situational realism and a relatively inexpensive way to learn and practice skills in a personalized, controlled, and safe setting.

A VR experience can be presented to the user through a variety of VR technologies, including head-mounted displays (HMDs), glasses-like displays that provide three-dimensional experiences to users. HMDs offer several advantages over other VR technologies which may increase usability and ultimately impact the uptake and adherence in interventions for children with ASD. These include stereoscopy (illusion of depth), motion tracking (continuously updated viewing based on head movements), and a strong sense of presence. These properties enable HMDs to provide fully immersive experiences that can potentially improve the ecological validity of delivered interventions and generalizability of learned skills. Despite their low-cost, high-quality, and growing popularity, the safety and usability of HMD systems remain relatively unexplored for children with ASD. Safety is a concern for HMD-based systems as the user is visually isolated from outsiders and this may provoke anxiety. Even newer generation HMDs (e.g. Oculus Rift) have been associated with “cybersickness” (characterized by eye strain, headache, dizziness, and/or nausea) in both neurotypical populations (Palmisano et al., 2017) and adults with ASD (Newbutt et al., 2016). Due to sensory differences prevalent among children with ASD, the experience of cybersickness may be exacerbated in this population. To this end, the first objective of this study was to examine the safety of an HMD system for children with ASD.

VR response is often characterized by the sense of presence in the simulated world, the degree of involvement or engagement with simulated content, and the perceived level of realism in behaviors of simulated characters and their interactions with participants. HMD-VR experiences are associated with improved user responses in the aforementioned dimensions compared to traditional videos in the general population (Van Damme et al., 2019) as well as in adults with ASD (Newbutt et al., 2016). However, it is unclear if the advantages of HMD systems extend to experiences of children with ASD, and only one other study to date has examined the experiences of children with ASD using HMD-VR (Newbutt et al., 2020). In this context, the second objective of our study was to examine the usability of HMDs for children with ASD.

Methods

The study protocol was designed in partnership with clinicians and families and approved by the Research Ethics Boards of the authors’ intuitions. Participants deemed to have capacity for consent provided written consent. Others provided assent and their caregiver provided written consent.

Participants

A convenience sample of 35 children with a diagnosis of ASD was recruited for the study (mean age: 13.0 ± 2.6 years; 10 females). Inclusion criteria were a clinical diagnosis of ASD supported by the Autism Diagnostic Observation Schedule (Lord et al., 2003), 8–18 years of age, full-scale IQ and verbal IQ greater than 70, and normal or corrected-to-normal hearing and vision. As one of the first studies of HMDs in a population of children with ASD, exclusion criteria were designed to limit the potential risk to participants. These included the use of beta-blockers, history of migraines, seizures, vestibular conditions, hypertension, cardiovascular and circulatory diseases, history of difficulty differentiating between reality and fiction, and parent-reported predisposition to motion sickness.

Measures

Participant characteristics

Participant characteristics are detailed in Table 1. IQ was measured using the Wechsler Abbreviated Scale of Intelligence, 2nd edition (Wechsler, 2011). ASD symptomatology and baseline anxiety symptoms were measured by the Social Communication Questionnaire (Rutter et al., 2003) and the Screen for Child Anxiety Related Emotional Disorders (Birmaher et al., 1997).

Table 1.

Participant information. The statistics are reported as median (IQR).

	All (N = 35)	Completed (N = 32)	Did not Complete (N = 3)
Age	13.0(5)	13.5(4.75)	9.0(4)
Sex (male: female)	25:10	22:10	3:0
Verbal IQ	94(15.75)	93(18)	95(9)
Non-verbal IQ	103(24)	105(23)	89(15)
Full-scale IQ	99(23.5)	103(25)	87(9)
SCQ	22(9)	22(8.25)	13(23)
SCARED	24(16)	24(16)	15(2)
Experience with VR (Y/N)	22:13	20:12	2:1
Experience with bus (< 10 times:> 10 times)	6:28	4:27	2:1
Video games (hours/week)	10(11)	12(12.5)	7(20.5)
Watch movies/videos (hours/week)	13(16)	13(16)	13(19)

IQR: interquartile range; SCQ: Social Communication Questionnaire; SCARED: Screen for Child Anxiety Related Disorders; VR: virtual reality.

Side effects

Anxiety was quantified using the State-Trait Anxiety Inventory (STAI; Chlan et al., 2003) administered following baseline, HMD-VR, and control conditions.

Cybersickness symptoms were quantified using the negative effects subscale of the Independent Television Commission–Sense of Presence Inventory (ITC-SoPI; Lessiter et al., 2001), a 39-question self-report that measures spatial presence, engagement, ecological validity/naturalness, and negative effects; a measure with good psychometric properties and previously used in adults (Newbutt et al., 2016) and adolescents with ASD (Wallace et al., 2017). In a follow-up call 1 month after the visit, caregivers were asked about the presence of these symptoms in their child after the study visit as well as the overall impact of the study on their child’s experience of school buses.

Usability

Usability was assessed based on effectiveness, efficiency, and satisfaction. Effectiveness was operationalized as (1) the ability of the system to convey the intended setting with high realism as measured by the ITC-SoPI spatial presence, engagement, and naturalness subscales, and (2) the perceived usefulness measured using a custom questionnaire. Efficiency was captured through self-report on ease of use and fatigue items of a custom usability questionnaire. Satisfaction was defined as the user’s self-reported comfort level and overall positive attitude toward the project. All measures were administered as shown in Figure 1.

Figure 1.

Schematic outline of the research protocol.

Procedure

Participation in this study entailed a single 2–3 h study visit to a research laboratory and a follow-up phone call 1 month after the visit. During the study visit, participants were familiarized with the study protocol using a visual storyboard. They were seated on a stationary chair in a laboratory (Figure 2) and were instructed to explore the scenario by moving their head (HMD-VR) or a computer mouse (control). The protocol consisted of an HMD-VR (Oculus Rift, 2160 by 1200 resolution) and monitor-displayed 360° video control condition (ViewSonic VP2468, 1920 by 1080 resolution), both depicting the same 5-min scenario developed by Shaftesbury Films (shaftesbury.ca, 4096 by 2048 resolution, 29.97 frames/s). The HMD’s integrated audio system was used for the HMD-VR condition, and a Sony on-ear headset was used for the control condition.

Figure 2.

Schematic of the experiment room.

The HMD-VR and control conditions were each repeated twice and separated by a baseline task in which participants watched 5-min clips from the Blue Planet series on a computer monitor. The presentation order (HMD-VR or control first) was randomized. The scenario placed the user seated inside a stationary school bus, with a driver and other children on the bus (Figure 3). During the scenario, seven children entered the bus and engaged in verbal interactions among each other. Several sensory and social triggers were included in the scenario (e.g. street noise, children, and driver).

Figure 3.

Screenshots from the scenario developed by Shaftesbury Films that depict a panoramic view of the bus from the user’s perspective.

Analyses

To compare HMD-VR and control conditions, continuous scores that passed Kolmogorov–Smirnov and Shapiro–Wilk normality tests were analyzed by repeated measure analysis of variance; otherwise, the nonparametric method (Wilcoxon rank-sum) was used. Discrete scores on Likert-type-scale were compared by chi-square test of homogeneity. All tests performed with a significance level of 5% and corrected for multiple comparisons using Bonferroni correction where appropriate.

Results

Thirty-two participants completed the full protocol (Table 1). Three participants dropped out of the study due to previous negative experience with school buses (n = 1), irritation due to the whistling presented in the scenario (n = 1), and excessive fidgeting (n = 1).

Side effects

Relative to baseline (mean STAI = 28.64), there was a significant increase in self-report of anxiety after the HMD-VR task (mean STAI = 32.19; Wilcoxon rank-sum: Z = 183.5, p < 0.001). This was not significantly different than control condition (mean STAI = 31.67; Wilcoxon rank-sum: Z = 10, p = 0.84).

HMD-VR and control conditions did not differ significantly on the ITC-SoPI negative effects scores (control = 1.77 ± 0.71; HMD-VR = 1.77 ± 0.68; Wilcoxon rank-sum: Z = 0.234, p = 0.81). Nine participants endorsed at least one negative effect following HMD-VR (score ≥ 4 on the ITC-SoPI) but none endorsed the nausea item (tiredness (n = 5), dizziness (n = 1), eye strain (n = 3), and headache (n = 3)).

Usability

Effectiveness

Compared to control, the HMD-VR condition received significantly greater scores on spatial presence (control = 2.76 ± 0.92; HMD-VR = 3.03 ± 0.88; t(30) = −3.25, p = 0.003; d = 0.3) and naturalness (control = 3.40 ± 0.77; HMD-VR = 3.78 ± 0.83; t(30) = −3.32, p = 0.002; d = 0.47), but not engagement.

Participants’ report of perceived usefulness was not significantly different between the HMD-VR and control conditions: 30% (HMD-VR) and 37% (control) responded that the experience helped them feel less nervous about being on a real bus; 67% (HMD-VR) and 70% (control) responded that the experience can help other kids be less nervous about being on a real bus. All parents completed the 1-month follow-up; three reported that participation had positively changed the way their child felt about being on the school bus.

Efficiency

Ease of use and fatigue were not significantly different between the HMD and control conditions (understood how to use the experience: HMD-VR: 68%, control: 69%; experience was easy to use: HMD-VR: 84%, control: 69%, could easily navigate the bus: HMD-VR: 58%, control: 75%; they were focused: HMD-VR: 45%, control: 47%; got tired when watching the experience: HMD-VR: 13%, control: 26%).

Satisfaction

Seventy-one percent, 58%, and 68% of participants reported that they enjoyed the HMD-VR experience, the experience was fun to watch, and the HMD was comfortable, respectively. The responses to the satisfaction questionnaire were different between the two conditions only for the question probing enjoyment, with HMD-VR being deemed more enjoyable than the control condition (χ²(4, N = 63) = 13.12, p = 0.010). Seventy-four percent of participants indicated that they preferred HMD-VR over the control condition.

Discussion

We did not find statistically significant differences between the HMD-VR and control condition in self-reported anxiety or cybersickness symptoms. Previous research in other populations has been mixed, reporting both increased (Dennison et al., 2016; Sharples et al., 2008) and similar (Huygelier et al., 2019) levels of cybersickness in HMDs compared to a monitor-displayed video. In our study, 26% of participants experienced one or more side-effects. Although this is a modest proportion based on the existing literature using the same HMD, the Oculus Rift (22% - 56%, Munafo et al., 2017), similar scores on the negative effects subscale of the ITC-SoPI have been demonstrated for VR experience of ASD populations (2.03 ± 0.95, Newbutt et al., 2016; 1.8 ± 0.7, Wallace et al., 2017). The relatively low rate of cybersickness in our study may be related to our conservative inclusion criteria that were in place to limit potential risks to participants in this first safety study. In particular, we used a convenience sample and excluded children with a predisposition to motion sickness. We also studied a single short VR scenario with stationary users in a laboratory setting with only one HMD (Oculus Rift) and control (monitor-displayed 360° video). As cybersickness is strongly linked to the interaction of user motion and motion in the VR environment (Palmisano et al., 2017), multiple scenarios involving different forms of motion would allow future studies to examine predictors of cybersickness more thoroughly. Finally, many concerns exist regarding extended exposure to VR, especially in naturalistic settings, and these were not investigated in this study. Future studies are needed to examine the safety of multiple HMD-VR systems for a greater diversity of children with ASD, especially in the context of long-term exposure across multiple settings.

Overall, our results supported the usability of the HMD-VR experience in dimensions of effectiveness, efficiency, and user satisfaction under experimental conditions. In agreement with HMD-VR research in an adult ASD population (Newbutt et al., 2016), HMD-VR was associated with greater spatial presence and naturalness as compared to monitor-displayed video in this study, and 74% of participants preferred the HMD-VR. Our findings on HMD usability for children with ASD echo the recent findings of Newbutt et al. (2020), who highlight a positive response to HMD-VR across their sample. These results provide preliminary evidence to suggest that HMDs may have the potential to enhance user experiences over monitor-displayed video for children with ASD. Future studies are needed to examine user, content, and exposure characteristics under which the advantages of HMD-VR can be fully realized.

Conclusion

Our study examined the safety and usability of short-exposure HMD-VR (Oculus Rift) for children with ASD under controlled laboratory conditions. Results showed that side effects of HMD-VR are similar to those of monitor-displayed video, but HMD-VR provides enhanced usability and user experience. These results add to the growing body of evidence supporting the use of HMDs for delivering interventions to children with ASD; however, they must be interpreted in the context of several limitations.

Footnotes

Acknowledgements

The authors would like to thank participants and their families for contributing to this research by volunteering their time. The authors would also like to thank Holland Bloorview Foundation for their support.

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Dr Kushki serves on the Board of Advisors for Shaftesbury Technology, a media company developing VR products for children with ASD.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The study was supported by Shaftesbury, a media company developing VR experiences for children, including those with ASD, through contribution to Natural Sciences and Engineering Research Council /Ontario Centres of Excellence [26308] funding, and by the Holland Bloorview Foundation Graduate Student Scholarship.

ORCID iD

Azadeh Kushki

References

Birmaher

Khetarpal

Brent

Cully

Balach

Kaufman

Neer

S. M.

(1997). The Screen for Child Anxiety Related Emotional Disorders (SCARED): Scale construction and psychometric characteristics. Journal of the American Academy of Child & Adolescent Psychiatry, 36(4), 545–553.

Chlan

Savik

Weinert

(2003). Development of a shortened state anxiety scale from the Spielberger State-Trait Anxiety Inventory (STAI) for patients receiving mechanical ventilatory support. Journal of Nursing Measurement, 11(3), 283–293.

Dennison

M. S.

Wisti

A. Z.

D’Zmura

(2016). Use of physiological signals to predict cybersickness. Displays, 44, 42–52.

Huygelier

Schraepen

van Ee

Abeele

V. V.

Gillebert

C. R.

(2019). Acceptance of immersive head-mounted virtual reality in older adults. Scientific Reports, 9(1), 4519.

Lessiter

Freeman

Keogh

Davidoff

(2001). A cross-media presence questionnaire: The ITC-Sense of Presence Inventory. Presence: Teleoperators & Virtual Environments, 10(3), 282–297.

Lord

Rutter

DiLavore

P. C.

Risi

(2003). Autism Diagnostic Observation Schedule: ADOS. Western Psychological Services.

Munafo

Diedrick

Stoffregen

T. A.

(2017). The virtual reality head-mounted display Oculus Rift induces motion sickness and is sexist in its effects. Experimental Brain Research, 235(3), 889–901.

Newbutt

Bradley

Conley

(2020). Using virtual reality head-mounted displays in schools with autistic children: Views, experiences, and future directions. Cyberpsychology, Behavior, and Social Networking, 23(1), 23–33.

Newbutt

Sung

Kuo

H.-J.

Leahy

M. J.

Lin

C.-C.

Tong

(2016). Brief report: A pilot study of the use of a virtual reality headset in autism populations. Journal of Autism and Developmental Disorders, 46(9), 3166–3176.

10.

Palmisano

Mursic

Kim

(2017). Vection and cybersickness generated by head-and-display motion in the Oculus Rift. Displays, 46, 1–8.

11.

Rutter

Bailey

Lord

(2003). The Social Communication Questionnaire. Western Psychological Services.

12.

Sharples

Cobb

Moody

Wilson

J. R.

(2008). Virtual reality induced symptoms and effects (VRISE): Comparison of head mounted display (HMD), desktop and projection display systems. Displays, 29(2), 58–69.

13.

Van Damme

All

De Marez

Van Leuven

. (2019). 360 video journalism: Experimental study on the effect of immersion on news experience and distant suffering. Journalism Studies, 20(14), 2053–2076.

14.

Wallace

Parsons

Bailey

(2017). Self-reported sense of presence and responses to social stimuli by adolescents with autism spectrum disorder in a collaborative virtual reality environment. Journal of Intellectual & Developmental Disability, 42(2), 131–141.

15.

Wechsler

(2011). Wechsler Abbreviated Scale of Intelligence, Second Edition (WASI-II). NCS Pearson.