Evaluation of the Usability of a Serious Game in Virtual Reality with a Focus on the Perception and Experience of Health Professionals for Motor Rehabilitation in Children with Cerebral Palsy

Abstract

Introduction:

Cerebral palsy (CP) is a nonprogressive neuropathological condition that requires lifelong neurocognitive-motor rehabilitation. Evidences indicate that the use of new technologies to assist in rehabilitation processes, such as serious games in virtual reality (VR), have served as adjuncts to therapy and capable of promoting engagement, motivation, and motor activation for these patients.

Objective:

To investigate the usability of a serious game in VR to help with the stability and balance of the head and trunk of children with CP, focusing on the perception and experience of health professionals.

Methods:

The collection was carried out with health professionals, and the results were comprehensively evaluated through viability by means of the total score, number of correct answers, number of errors, and level of difficulty during the execution of the game, which were collected from the performance report generated by the application. System satisfaction was also verified by the System Usability Scale (SUS).

Results:

The mean obtained from the total score of the SUS was 82.10 ± 12.66 points, being considered of high usability for the suggested purpose. The professionals' opinion about the usability of the system did not change due to the performance during the game.

Conclusion:

The study demonstrated that the developed rehabilitation program has successfully delivered the experience to exercise the head control and trunk balance of subjects with CP.

Introduction

Cerebral palsy (CP) is a nonprogressive neuropathological condition that results in heterogeneous clinics, accompanied by a variety of comorbidities that can affect social participation, causing functional impact and reduced quality of life.^1–5 The most promising interventions are neuroprotective, based on the practice of tasks and activities of daily living, as they stimulate neuroplasticity, improving functionality, and, thus, enabling inclusion in the community.^1,2,5–8

Evidence suggests that the use of new technologies such as virtual reality (VR) has the potential to increase adherence to treatment, making interventions more interesting, helping with learning and motor and functional development. Indications that these effects may be due to active sensorimotor experience associated with neuroplasticity and increased motivation and engagement.^8–11

Specific programs or applications for the use of VR in rehabilitation have been tested as an alternative to produce learning in an ecological context, stimulating the acquisition of new motor, functional, and/or cognitive tasks.¹¹ Some studies present the development of programs for motor rehabilitation through game technology platforms envisioning more fun and less frustrating interventions. These characteristics allow a greater number of repetitions of the task with multisensory feedback, achieving functional recovery in a challenging, pleasant, and safe environment.^8,12,13

With this foundation, we created a rehabilitation program based on the application of a serious game in VR. The application was developed by the Neuroscience and Experimental Virtual Reality research team to exercise head control and trunk balance in people with CP, classified in the Gross Motor Function Classification System (GMFCS) levels III, IV, and V. The serious game produced by NERV has been studied and established, being tested and reviewed by the team of researchers and clinicians, and with children and/or adolescents with CP. The researchers looked at the experience, reinforcing the goal of motor rehabilitation without losing focus on user engagement and enjoyment. To increase the patient's involvement with the proposed activity and actively induce postural control, the application offers real-time feedback for motor activation. Therefore, immersive VR technology was proposed, displayed through a Head-Mounted Display (HMD) with the aim of provoking a greater sense of presence and immersion to patients.¹⁴

The primary objective of this study was to investigate the usability of a serious game in VR to assist in the stability and balance of the head and trunk of children with CP, focusing on the perception and experience of health professionals, as well as verifying the performance of these clinicians during the game.

Methods

Virtual environment

To display the virtual rehabilitation environment, a mobile HMD device (VR BOX—Virtual Reality Glasses^®) is used, coupled to a compatible smartphone—Samsung Galaxy S8^®—model that sets the image at high frequency, which reduces and/or avoids adverse effects such as nausea (cybersickness/motion sickness) to integrate real-time computer graphics, sounds, and other sensory data and create a virtual environment for user interaction.¹⁵ The gyroscope and accelerometer included in the smartphone allow tracking of the position and orientation of the patient's head, both for data collection and to provide real-time feedback through the virtual environment, which simulates up to 3 degrees of freedom. The HMD enables immersion in the virtual world, increasing the sense of presence and providing feedback to the central nervous system as a result of the motor act.¹⁶ The application can be used on any smartphone compatible with the ANDROID^® VR platform.

Serious game description

The serious game developed by the team of researchers consists of an airplane flying through a forest on a sunny day, in which the patient helps to guide the airplane, using head movements (Fig. 1). Following a high commercial standard in the development of the experience, it was decided to maximize responsiveness in the game's audiovisual feedback, to maximize the feeling of immersion, and to reduce the symptoms of cybersickness.¹⁷ As a main objective, the patient is invited to look for striped and colored arcs that are floating in this virtual environment, with the intention of guiding the plane toward them, thus performing a movement with the head. The system allows head and trunk movements in different ranges of motion, providing a playful and fun training. In addition to the stylized virtual environment with vibrant colors in pleasant contrasts and well-defined lines to facilitate visualization even on low-resolution screens, the game also includes an original soundtrack, visual and sound effects, and animations in an attempt to support the acquisition and retention of new motor skills, in addition to reinforcing users' motivation and involvement in therapy.

FIG. 1.

Virtual reality game environment.

A dynamic difficulty adjustment has been implemented in the game to provide an appropriate level of challenge, increased engagement, and player retention.¹⁸ The central task of the game is to direct a virtual plane to the center of floating colored arcs that appear scattered in the scenario, an action that leads the patient to perform head movements. The implicit difficulty system influences the position and height of these floating colored arches, making it possible for patients with different motor skills to use the game successfully.¹⁹

Initially, the arches are positioned very close to the center of the scene, both in height and in horizontal position, allowing even small head movements to reach them. As the patient stabilizes the head and trunk, the game progresses through 10 implicit levels of difficulty, causing the next colored arcs to appear at more distal positions from the center of the scene, causing the patient to perform more complete head movements. If the patient fails to achieve a minimum number of arcs at their current difficulty level, the difficulty will automatically regress to the previous level until it advances again. At the highest levels of difficulty, the colored arcs appear widely distributed across the scene, near the grass, in the sky, and near the trees on the right or left. It should be noted that the implementation of adaptive difficulty in the development of the game strongly took into account the possibility of users having atypical cognitive and neurological deficiencies; therefore, the implementation of numerical systems to indicate punctuation and complex textual information present in the graphical interface is avoided. In addition, the underlying adaptive difficulty programming was used to implement visual elements such as fireworks and sound elements such as a soundtrack progressively linked to reaching higher difficulty levels.

Throughout the development of the software, the game was tested and reviewed by the team of researchers and clinicians, and with children and/or adolescents with CP, with a focus on reinforcing the experience and obtaining user engagement and satisfaction and emphasizing the objective of motor rehabilitation. The time that the player must remain immersed in the scenario is the responsibility of the professional conducting the intervention. At the end of the game, the professional can view the patient's performance data through spreadsheets exported by the application, containing the session's game data, which include the total score of the game, the number of hits and misses, the level of difficulty achieved, the total duration of the game and in which time the patient reached a certain difficulty level.

Study design

The research was conducted with health professionals with experience in CP. The experts were invited to use the system, after receiving guidance on the game, in a single session lasting 10 minutes and after recording their experience through the System Usability Scale (SUS). The researchers applying the task explained the nature of the task so that the participants could interact with the system. Right after familiarization, the team explained how to access the data generated by the program.

Outcome measures

The results were measured through the feasibility and satisfaction expressed by the user with the experience of using the product. Feasibility was comprehensively measured by (1) total score, (2) number of hits, (3) number of misses, and (4) level of difficulty while running the game.

Data were collected from the performance report generated by the application. Satisfaction was measured by the SUS. The SUS, developed by Brooke as a “quick and dirty” survey scale, is a quick and easy self-administered instrument that assigns a benchmark score on professionals' views of a wide range of products, services, and interfaces.^20,21 The SUS is composed of 10 items that are scored on a 5-point Likert scale, (1) strongly disagree, (2) disagree, (3) neither agree nor disagree, (4) agree, (5) strongly agree.²¹ The final score can vary from 0 to 100 and must be calculated from the formula proposed by Brooke, the higher the score, the greater the degree of usability.^20,21 For odd-numbered questions (1, 3, 5, 7 and 9), 1 should be subtracted from the individual score, whereas for even-numbered questions (2, 4, 6, 8 and 10), the score obtained should be reduced by 5, thus, all the converted answers are added and multiplied by 2.5, obtaining the total value of the SUS.^20,21 With the definition of the final score, it is possible to classify the evaluated system: Below 25 points: worst imaginable; 25–39: poor; 40–55: ok; 56–75: good; 76–87: excellent and 88–100: best imaginable.^22,23 A score above 68 in the SUS means that the product is acceptable.²⁰ The estimated time for applying the SUS scale is ∼5 minutes.

Data analysis

Information was collected using standardized forms before being stored. The clinical and demographic characteristics of the participants were descriptively analyzed using absolute and relative frequency for categorical variables and mean and standard deviation or median and interquartile range for quantitative variables. Data normality was assessed using the Kolmogorov–Smirnov test. Spearman's coefficient was used to assess a correlation between SUS, the total game score, and the number of correct answers. It also verified whether there was any association, using the Pearson Chi-Square test, between the SUS and the variables age, sex, profession, specialty, and level of education. A 95% confidence interval was considered to estimate means. Data were analyzed using the statistical package SPSS version 22.0 (IBM Corp.) and a significance level of 5% (P < 0.05).

Results

Fifty-seven health professionals, including physiotherapists, doctors, and occupational therapists, took part in the study. The mean age was 35.05 ± 9.57 years, with a minimum age of 22 and a maximum of 60 years. Table 1 shows the characteristics of the users.

Table 1.

Demographic Characteristics of Professionals

Characteristics		Total (n = 57)
Sex, %	Male	24.6
Sex, %	Female	75.4
Professionals, %	Physiotherapist	80.7
	Occupational therapist	10.5
	Physician	8.8

Maximum education level	Physiotherapist, %	Occupational therapist, %	Physician, %
Specialty	52.87	3.50	7.04
Master's	22.26	3.50	0
PhD	5.56	3.50	1.76

Performance	Physiotherapist, %	Occupational therapist, %	Physician, %
Generalist	40.35	10.50	0
Neurofunctional	24.56	0	0
Physiatrics	0	0	7.04
Others	15.78	0	1.76

After the researcher instructed about the objective and how to access it, the professionals experienced the game for ∼10 minutes. During the task, the application recorded the total score, successes, errors, and the level of difficulty reached during the game. The mean total score among the participants was 131.86 ± 59.68 points, with the lowest being 32 and the highest 295 points. The mean number of correct answers was 85.79 ± 45.55 with a minimum of 6 and a maximum of 261 hits and the mean number of errors was 46.07 ± 41.32 with a minimum of 0 and a maximum of 198 errors. As for the difficulty level reached during the game, the average was 9.58 ± 1.47, with a minimum degree of 2 and a maximum of 10. It should be noted that 51 people reached the maximum level of difficulty.

Regarding the usability questionnaire, SUS, all participants completed all questions. The mean obtained from the total score was 82.10 ± 12.66, minimum value of 50, and maximum value of 100. It is noted that 36 professionals considered the task above 75 points on the SUS scale, which classifies it from Excellent to Best Imaginable, while 21 participants considered the game from Ok to Good (Table 2). Remembering that a score above 68 in SUS considers the product as acceptable.

Table 2.

Values of Mean, Standard Deviation, and Interquartile Range of the Score of Professionals Who Classified the Game in System Usability Scale as Ok and Good e Excellent and Best Imaginable

Rank SUS	Mean (± DP)	Q25–Q75	p
Ok and good (n = 21)	61.61 (±20.80)	50.75–72.49	0.14
Excellent and best imaginable (n = 36)	68.32 (±18.99)	57.78–85.65	0.14

DP, standard deviation; p, minimum level of significance of the Wilcoxon test; Q25–Q75, interquartile range; SUS, System Usability Scale.

Spearman's coefficient showed that there was no correlation between the SUS and the total score (r = −0.047; P = 0.72) with the number of correct answers (r = −0.207; P = 0.12) in the game, demonstrating that the opinion of professionals in relation to the product did not change due to the performance during the game.

There was no association between the SUS score and the variables age, gender, profession, and specialty, which were not statistically significant, with P = 0.89, P = 0.24, P = 0.39, and P = 0.80, respectively. There was also no association between SUS and the neurofunctional specialty (P = 0.61). However, it is observed that the higher the professional's education, the better the classification of the product in the SUS (P < 0.05).

Discussion

The SUS scale was applied to evaluate a rehabilitation program interconnected with a serious game in VR. The instrument showed that participants rated the rehabilitation program developed to exercise head control and trunk balance in subjects with CP between excellent and best imaginable, which makes the game a viable supporting tool for application in the functional recovery of people with CP.

A product and/or system rated by users with 82 points on the SUS not only demonstrates how much users appreciate it but also shows how likely they are to recommend it to others.²⁴ In this sense, this study demonstrates that the developed VR game would probably be used by specialists to assist in the rehabilitation of head control and trunk balance of patients with CP during professional activity.

Therefore, the serious game in VR created by the group has the potential to improve the final results of the interventions and allows patients to experience challenges with motivation and engagement, facilitating the specificity and variation of the task, promoting motor learning and, consequently, improvement of performance and functionality.^6,25,26

Other groups report results on the application of VR games intended for the rehabilitation of children with CP and the application of the SUS scale to assess the effectiveness of therapies.²⁷ However, the number of participants/evaluators was six with positive feedback on the usability of the games and the system, reporting that the experience in VR was pleasant and therapeutically relevant.²⁷

It is observed in this study that there was no correlation between the performance of professionals during the game and the judgment about the usability of the system. This is probably because the researchers, during the instructions about the game, did not ask the professionals to make an effort to obtain the best score possible. This finding had already been reported by Brooke who states that user assessments may not be adequate for the outcome of using the system, demonstrating a weak correlation between SUS scores and performance during the task.²⁴

It should be noted that our results also found no correlation between sex and the SUS score, as well as other studies also show low or no correlation between these variables.^28,29 However, Melnick et al. reports that older specialists had lower SUS records than younger ones,²⁹ which did not happen to professionals in this study, with no association between age and SUS score. Padrini-Andrade et al.²⁸ stated that there is no association between schooling and SUS. On the other hand, this study points out that professionals with higher education, such as postgraduate studies, considered the product better than those who had only undergraduate degrees.

Game Limitations

In terms of health applications, access to and familiarity with VR equipment remain considerably restricted. This lack of familiarity can be considered a limitation in terms of reducing users' ability to understand the virtual experience. Some of the participants suggested that the game contains more than one scenario because they considered that the game could become monotonous after some exposure time. However, it is important to consider that the participants know the patients' experience from an external perspective, in the 3rd person, and do not have the physical and/or intellectual limitations of the patients. Ideas such as piloting a submarine in the ocean, driving a motorcycle using head movements, or even incorporating fantasy scenarios, flying through the clouds like superheroes were suggested by clinicians, which could increase the potential of serious play, increasing repetition in physiotherapeutic treatments of long term.

Future of Game

Following the results obtained by this study, the game could be updated to improve its effectiveness as a motor training tool for CP, and potentially evolve its adjacent quality as a cognitive training. Based on feedback from health care professionals, it was possible to generate new ideas for serious games in VR for different clinical conditions. Finally, the distribution of this software in the finalized academic environment will allow the generalization of this study, providing more evidence and greater understanding about the use of VR for CP.

Conclusion

It is observed that the serious game in VR developed successfully delivered the experience to exercise head control and trunk balance of subjects with CP, offering a pleasant experience with wide applicability, in addition to being relevant for motor rehabilitation, according to the judgment of professionals of health. The usability tests effectively identified the serious game being excellent and the best imaginable. The study indicates that there is the possibility of improving the serious game in VR to optimize user practice and therapeutic relevance.

Ethical Approval

The study was approved by the Research Ethics Committee of the Federal University of Health Sciences of Porto Alegre under CAEE: 88204418.9.0000.5345. All participants gave written informed consent prior to the start of data collection.

Footnotes

Author Disclosure Statement

The authors report no conflicts of interest.

Funding Information

This research did not receive any specific grants from funding agencies in the public, commercial, or not-for-profit sectors.

References

Graham

, Paget

, Wimalasundera

. Current thinking in the health care management of children with cerebral palsy. Med J Aust, 2019; 210(3):129–135; doi: 10.5694/mja2.12106.

Gulati

, Sondhi

. Cerebral palsy: An overview. Indian J Pediatr, 2018; 85(11):1006–1016; doi: 10.1007/s12098-017-2475-1.

NICE. Cerebral Palsy in under 25s: Assessment and Management: Methods, Evidence and Recommendations. National Institute for Health and Care Excellence: Clinical Guidelines (UK) (NICE Guideline, No. 62.) NICE: London; 2017.

Novak

, Morgan

, Adde

, et al. Early, accurate diagnosis and early intervention in cerebral palsy. JAMA Pediatr, 2017; 171(9):897; doi: 10.1001/jamapediatrics.2017.1689.

Vitrikas

, Dalton

, Grant

, et al. Cerebral palsy: An overview. Am Fam Physician, 2020; 101(4):213–220.

Novak

, Morgan

, Fahey

, et al. State of the evidence traffic lights 2019: Systematic review of interventions for preventing and treating children with cerebral palsy. Curr Neurol Neurosci Rep, 2020; 20(2):3; doi: 10.1007/s11910-020-1022-z.

Novak

, Mcintyre

, Morgan

, et al. A systematic review of interventions for children with cerebral palsy: State of the evidence. Dev Med Child Neurol, 2013; 55(10):885–910; doi: 10.1111/dmcn.12246.

Bortone

, Leonardis

, Solazzi

, et al. Serious game and wearable haptic devices for neuro motor rehabilitation of children with cerebral palsy. Biosyst Biorobotics, 2017; 15:443–447; doi: 10.1007/978-3-319-46669-9_74.

Gatica-Rojas

, Méndez-Rebolledo

. Virtual reality interface devices in the reorganization of neural networks in the brain of patients with neurological diseases. Neural Regen Res, 2014; 9(8):888–896; doi: 10.4103/1673-5374.131612.

10.

Weiss

, Tirosh

, Fehlings

. Role of virtual reality for cerebral palsy management. J Child Neurol, 2014; 29(8):1119–1124; doi: 10.1177/0883073814533007.

11.

Ravi

, Kumar

, Singhi

. Effectiveness of virtual reality rehabilitation for children and adolescents with cerebral palsy: An updated evidence-based systematic review. Physiother (United Kingdom), 2017; 103(3):245–258; doi: 10.1016/j.physio.2016.08.004.

12.

Corrêa

AGD

, Kintschner

, Blascovi-Assis

. System of upper limb motor rehabilitation training using leap motion and gear VR in sessions of home game therapy. In: Proceedings—IEEE Symposium on Computers and Communications Institute of Electrical and Electronics Engineers Inc.;, 2019; doi: 10.1109/ISCC47284.2019.8969586.

13.

Shum

, Valdes

, Hodges

, et al. Error augmentation in immersive virtual reality for bimanual upper-limb rehabilitation in individuals with and without hemiplegic cerebral palsy. IEEE Trans Neural Syst Rehabil Eng, 2020; 28(2):541–549; doi: 10.1109/TNSRE.2019.2959621.

14.

Slater

. Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philos Trans R Soc L B Biol Sci, 2009; 364(1535):3549–3557; doi: 10.1098/rstb.2009.0138.

15.

Gregg

, Tarrier

. Virtual reality in mental health. A review of the literature. Soc Psychiatry Psychiatr Epidemiol, 2007; 42(5):343–354; doi: 10.1007/S00127-007-0173-4.

16.

Wang

, Reid

. Virtual reality in pediatric neurorehabilitation: Attention deficit hyperactivity disorder, autism and cerebral palsy. Neuroepidemiology, 2011; 36(1):2–18; doi: 10.1159/000320847.

17.

Le May

, Tsimicalis

, Noel

, et al. Immersive virtual reality vs. non-immersive distraction for pain management of children during bone pins and sutures removal: A randomized clinical trial protocol. J Adv Nurs, 2021; 77(1):439–447; doi: 10.1111/JAN.14607.

18.

Hendrix

, Bellamy-Wood

, McKay

, et al. Implementing adaptive game difficulty balancing in serious games. IEEE Trans Games, 2019; 11(4):320–327; doi: 10.1109/TG.2018.2791019.

19.

Maung

, Crawfis

, Gauthier

, et al. Development of Recovery Rapids—A game for cost effective stroke therapy. International Conference on Foundations of Digital Games (2014).

20.

Martins

, Rosa

, Queirós

, et al. European Portuguese Validation of the System Usability Scale (SUS). Procedia Comput Sci, 2015; 67(Dsai):293–300; doi: 10.1016/j.procs.2015.09.273.

21.

Brooke

. SUS: A “Quick and Dirty” Usability Scale. Usability Eval Ind, 1996;(November 1995):207–212; doi: 10.1201/9781498710411-35.

22.

Bangor

, Kortum

, Miller

. An empirical evaluation of the System Usability Scale. Int J Hum Comput Interact, 2008; 24(6):574–594; doi: 10.1080/10447310802205776.

23.

Bangor

, Staff

, Kortum

, et al. Determining what individual SUS scores mean: Adding an adjective rating scale. J usability Stud, 2009; 4(3):114–123.

24.

Brooke

. SUS: A retrospective. J usability Stud, 2013; 8(2):29–40.

25.

Booth

ATC

, van der Krogt

, Buizer

, et al. The validity and usability of an eight marker model for avatar-based biofeedback gait training. Clin Biomech, 2019; 70(August):146–152; doi: 10.1016/j.clinbiomech.2019.08.013.

26.

Booth

ATC

, Buizer

, Harlaar

, et al. Immediate effects of immersive biofeedback on gait in children with cerebral palsy. Arch Phys Med Rehabil, 2019; 100(4):598–605; doi: 10.1016/j.apmr.2018.10.013.

27.

, Fehlings

, Biddiss

. Design and evaluation of virtual reality-based therapy games with dual focus on therapeutic relevance and user experience for children with cerebral palsy. Games Health J, 2014; 3(3):162–171; doi: 10.1089/G4H.2014.0003/ASSET/IMAGES/LARGE/FIGURE 2.JPEG.

28.

Padrini-Andrade

, Xavier Balda

RdC

, Nema Areco

, et al. Evaluation of usability of a neonatal health information system according to the user's perception. Rev Paul Pediatr, 2019; 37(1):90–96; doi: 10.1590/1984-0462/;2019;37;1;00019.

29.

Melnick

, Dyrbye

, Sinsky

, et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc, 2020; 95(3):476–487; doi: 10.1016/J.MAYOCP.2019.09.024/ATTACHMENT/3B700C06-056E-441A-9639-76B4D0E60977/MMC2.MP4.