Can serious play and clinical cognitive assessment go together? On the feasibility and user-experience of virtual reality simulations in paediatric neurorehabilitation

Abstract

PURPOSE:

Cognitive impairments frequently occur in children and adolescents with acquired brain injury (ABI), causing significant disabilities in daily life. Current paper-and-pencil neuropsychological tests do not capture the complexity of daily life activities, often failing to objectify subtle cognitive impairments. Virtual Reality (VR) simulations might overcome this discrepancy, as it resembles daily life situations. The aim of this study was to evaluate the feasibility, user-experience and preference of a VR simulation with a non-immersive (computer monitor; CM)) and immersive (head mounted display; HMD)) VR setup.

METHODS:

Children and adolescents with ABI (n = 15) and typically developing children and adolescents (n = 21) completed a VR-task with a CM and HMD.

RESULTS:

Both VR setups were feasible for children and adolescents with ABI. User-experience was enhanced with the HMD compared to CM in both groups. Side effects were low and comparable for both groups, and there were no differences between setups (HMD and CM). The majority of the children and adolescents with ABI preferred the HMD.

CONCLUSION:

VR simulations appear feasible to use in paediatric rehabilitation. The preference for a VR setup should be discussed with the child. Further research is needed to develop more sensitive measures to further explore the potential of VR for cognitive assessment.

Keywords

Virtual reality feasibility cognition computerized assessment paediatric rehabilitation

1 Introduction

Acquired brain injury (ABI) is a common disorder with an estimated incidence of 19,000 children and adolescents (0–24 years) each year in the Netherlands [1]. Traumatic brain injury is the most common cause of ABI, followed by meningitis/encephalitis, tumours and stroke [1]. Impairments following ABI are heterogeneous, from physical to cognitive impairments [2, 3]. Cognitive impairment can tremendously hamper activities of daily life, such as social interaction or school attendance [2, 4]. Consequently, it is therefore of great importance to assess cognitive impairment in order to predict the level of anticipated performance during daily life activities and to treat cognitive impairment through psychoeducation (e.g., educating children, family members and teachers on compensatory strategies for cognitive impairment in daily life).

Currently, a neuropsychological assessment (NPA) is used to assess cognitive impairment. Results systematically evaluate cognitive functions in order to gain insight into cognitive strengths and weaknesses. NPA helps the clinician to provide appropriate psychoeducation and create a patient-centred, tailor-made treatment plan. However, NPA does not capture the complexities of daily life functioning; for example the ability to attend to classwork in a noisy classroom or to participate in birthday parties [5, 6]. NPA mostly consists of multiple, sequentially administered paper-and-pencil tests in a quiet room without external distractors. Cognitive impairment may be present only in daily life situations, and therefore not be objectified or anticipated. In order to bridge this gap, it is preferable to assess cognition in more realistic, daily life situations.

Employing virtual reality (VR) enables natural interaction and immersion in a realistic, safe and controlled 3D environment [6]. Predominantly, there are two VR setups that are most widely used [7]: non-immersive VR, for example by using a computer monitor (CM); and immersive VR, for example a head mounted display (HMD). An advantage of immersive VR is the enhanced feeling of being physically present, making interaction with the environment more natural. VR demonstrates great promise in capturing cognitive impairments which cannot be captured with paper-and-pencil tests [5, 6].

Non-immersive VR has been used in several studies and appears feasible for use in children with ABI [8]. However, the use of a HMD with bright light might trigger side effects for children and adolescents with cognitive complaints due to ABI. A feasibility study determines whether an intervention is appropriate for further testing [9]. To the author’s knowledge, this is the first article addressing the feasibility of immersive VR for children and adolescents with ABI. In other paediatric populations (children with deafness, cancer, autism) immersive VR has shown to increase game enjoyment [10, 11], although some children also feel too overwhelmed [11]. The VR systems appeared easy to understand and most children completed the tasks [10 –12]. Side effects, such as nausea or dizziness, were reported in 0–9%, however none ended the study due to this [11, 12]. Furthermore, the possibility of choosing a preferred VR setup (either a CM or HMD) might enhance motivation to use a VR program. The most appropriate VR setup for children and adolescents with ABI is unknown.

The aim of the current study was threefold: (1) to evaluate the feasibility of the VR simulation by determining whether children and adolescents (8–18 years) with ABI were able to use a non-immersive (CM) and/or immersive (HMD) VR setup; (2) to examine user-experience, with particular focus on cybersickness, per VR setup (CM or HMD); and (3) to determine which VR setup is preferred. In comparison to other factors (e.g., feeling immersed or forgetting time), cybersickness may be a major reason for prematurely terminating assessment. With respect to the second and third sub aims, the results of the children and adolescents with ABI were compared to their typically developing peers.

2 Methods

2.1 Participants

Children and adolescents with ABI were included in De Hoogstraat Rehabilitation Centre in Utrecht, the Netherlands, in the paediatric rehabilitation ward and outpatient clinic. Certified practitioners (i.e., rehabilitation doctor or child psychologist) informed the researcher when children and adolescents met the inclusion criteria, which for those with ABI were: (1) 8–18 years old; (2) participating in an in- or outpatient rehabilitation program at De Hoogstraat Rehabilitation Centre; (3) adequate stamina to participate, assessed by a certified practitioner; (4) voluntary participation; and (5) written informed consent. Exclusion criteria for children and adolescents with ABI were: (1) diagnosis of epilepsy (photosensitive epilepsy might be triggered as images in VR flash very quickly) and (2) being unable to use a game controller (e.g., due to hemiparesis). Typically developing children and adolescents were recruited amongst colleagues, family and friends of the research group. They were included when they met the following criteria: (1) 8–18 years old; (2) no history of neurological/psychiatric disease; (3) voluntary participation; and (4) written informed consent. The inclusion period started in June 2016 and finished in June 2019.

2.2 Virtual reality simulation and setups

A virtual supermarket was developed by Atoms2Bits for commercial purposes. The software of Unity was used. For research and potential clinical purposes, the supermarket was adapted in collaboration with the Utrecht University, University Medical Centre Utrecht, and De Hoogstraat Rehabilitation Centre. The software was designed to run on a regular computer in combination with two VR setups: a CM and a HMD. The virtual supermarket was modelled according to a regular Dutch supermarket and contained 18 shelves, eight cash registers, several product displays (e.g., bread, fruit, vegetables) and freezing compartments. Approximately 20,000 products were designed resembling real brands and packages from well-known products in Dutch supermarkets. The supermarket had a surface of 50×30 virtual meters. The maximum navigation speed was 0.5 meter per second.

The CM was 24-inch with a resolution of 1920×1200 pixels. For navigation, a wired controller was used (Xbox 360©). Participants sat on an office chair in front of the CM, at a distance of approximately 90 cm from their eyes. Two types of HMDs were used. Participants were assessed between July 2016 and October 2016 with the Oculus Rift DK2© (further referred to as ‘Oculus’) and between April 2018 and July 2019 with the HTC Vive© (further referred to as ‘HTC’). The Oculus had a 100° field of view, a resolution of 960×1080 per eye, and a refresh rate of 75 Hz. The HTC had a 110° field of view, a resolution of 1080×1200 per eye, and a refresh rate of 90 Hz. Navigation with the HTC was possible with use of two controllers and two base stations with a tracking system to navigate through real time movement in the virtual environment (maximum space of 3×3 meters). For safety reasons, all participants were seated in an office chair, therefore allowing for direct comparisons between children and adolescents with ABI and typically developing children and adolescents.

2.3 Procedure and tests

Prior to commencing assessment, participants were asked whether they had any questions relating to the study and to provide written informed consent. First, they completed a questionnaire regarding demographics and gaming experience (see also [13]). Participants were interviewed about their experience with six forms of games/technologies: (1) 2D games; (2) 3D games with third-person view; (3) 3D games with first-person view; (4) use of keyboard/touchscreen; (5) use of controller; and (6) VR. Subsequently, a short cognitive screener of cognition was administered (Dutch version of the Montreal Cognitive Assessment [14]).

In addition, participants were asked to perform a VR-based task twice using both CM and HMD setups. To avoid potential bias in the results caused by learning or boredom, the order of the two conditions was randomized: half of the participants started with the CM and half started with the HMD. Participants started with a practice trial to get familiar with the VR apparatus and environment (i.e., virtual supermarket with empty shelves). After the practice trial, they were instructed to (1) start the VR-based task by passing through the entry gates, (2) find three products from a shopping list, and (3) pass the cash registers to finish. A shopping list was presented three times, and participants were asked to recall the products prior to the VR-based task. There were two different shopping lists and the order was semi-randomised across conditions. One list contained M&Ms, Donald Duck magazine, and fizzy peaches, the other one contained matches, crunchy fruit sprinkles, and hotdogs. After 15 minutes, participants were asked to stop if they had not finished yet.

After the first VR-task, a questionnaire was administered to assess user-experience with the specific VR setup (CM or HMD) (Appendix 1; for information on the development and validity of this questionnaire see Spreij et al., 2022). Participants were required to indicate to what extent they agreed or disagreed with 15 items on a 6-point scale. Three items belonged to one of five categories: (1) Engagement; (2) Transportation; (3) Flow; (4) Presence; and (5) Side effects (for more details, Spreij et al., 2022). Response options were based on a 6-point Likert scale ranging from negative (--- [031-May-22]) to positive (+++[5]). Each category was described by a summed score of three items (Appendix 1), resulting in a scale-score ranging from 0–15. This was repeated with the other VR setup and the questionnaire was filled out a second time. Last, a final questionnaire was administered to assess the preference for one of the two user interfaces containing five items regarding motivation and enjoyment (Appendix 2). Participants indicated whether they preferred the CM, the HMD, or did not have a specific preference for VR setup.

2.4 Outcome measures and statistical analyses

With respect to feasibility, the following outcome measures were administered: number of children and adolescents who were able to complete the task, total duration of assessment (ranging 0–15 minutes), and total number of products found (ranging 0–3). With respect to user-experience, summed scores were calculated on items belonging to the same category, resulting in five scores (i.e., Engagement, Transportation, Flow, Presence, Side effects) ranging from 0–15 (0 = very negative, 15 = very positive; except for ‘Side effects’ where a higher score indicated more side effects). Preference was indicated using multiple choice for CM, HMD, or no preference. Feasibility, user-experience and preference were compared between the user interfaces (CM vs. HMD) and between children and adolescents with ABI and typically developing children and adolescents. Cybersickness was assessed by the number of participants scoring > 3 (> [+]) on the item ‘I felt nauseous’ (item 5) of the user-experience questionnaire.

Demographic and clinical characteristics were collected from patient files. Demographic characteristics were compared using non-parametric tests (Fisher’s Exact test for categorical variables and Mann-Whitney U test for continuous variables). A mixed analysis of variance (ANOVA) was conducted with “VR setup” as a within-subject factor (CM vs. HMD; Oculus vs. HTC; as two types of HMDs were used in this study, potential differences within the HMD setup were also investigated) and “group” as a between-subject factor (children and adolescents with ABI vs. typically developing children and adolescents) for continuous variables (i.e., total duration, number of products found, scale-scores). These ANOVAs also allowed for exploration of possible significant interactions between these factors. The pre-specified level of statistical significance for all tests was 0.05. STATISTICA (StatSoft, USA) was used to perform the statistical analyses.

3 Results

3.1 Feasibility

3.1.1 Early termination

In total, 40 children and adolescents were included (25 typically developing, 15 children and adolescents with ABI). Sixteen were assessed with the Oculus (10 typically developing, 6 children and adolescents with ABI) and twenty were assessed with the HTC (11 typically developing, 9 children and adolescents with ABI). All were able to complete the test when working with the CM. Assessment with the Oculus was never prematurely terminated. Early termination with the HTC was needed with 4 typically developing children and adolescents; due to software failure (n = 2) and due to fatigue and motivational issues (n = 2). None with ABI terminated the VR-based task when using the HMD (Oculus as well as HTC).

After exclusion of the early terminated assessments, data of 21 typically developing children and adolescents and of 15 with ABI were included for analyses (demographic characteristics can be found in Table 1). Sex, age, type of education, and gaming experience were comparable between groups (sex Fisher’s exact, p = 1.00; age U = 126.0, z = 1.01, p = 0.31; type of education Fisher’s exact p = 0.17; gaming experience U = 130.0, p = 0.38). Clinical characteristics of the children and adolescents with ABI are presented Table 1.

Table 1
Demographic, clinical and cognitive characteristics split on children and adolescents with ABI and typically developing children and adolescents (where applicable)

Demographic characteristics Children and adolescents with ABI (n = 21) Typically developing (n = 15) Statistical analyses

Sex (% male) 60.0 61.9 p = 1.00

Age in years (mean, SD) 15.1 (2.6) 14.6 (1.7) p = 0.31

Type of education (%) p = 0.17

Regular primary education 13.3 0

Regular secondary education 86.7 100

Dedicated special needs 0 –

Gaming experience questionnaire

(0–18 (mean, SD)) 13.9 (2.5) 14.4 (3.0) p = 0.38

ABI type (%)

Trauma 53.3 –

Haemorrhage 13.3 –

Oncology 20 –

Infection 13.3 –

Hemisphere of damage (%)

Left 6.7 –

Right 20 –

Bilateral 53.3 –

Unknown/other/not applicable 20 –

Days post-ABI (mean, range) 662 (32–2921) –

Montreal Cognitive Assessment 0–30 (mean, SD) 27.5 (2.1) 28.4 (2.9)

Demographic characteristics	Children and adolescents with ABI (n = 21)	Typically developing (n = 15)	Statistical analyses
Sex (% male)	60.0	61.9	p = 1.00
Age in years (mean, SD)	15.1 (2.6)	14.6 (1.7)	p = 0.31
Type of education (%)			p = 0.17
Regular primary education	13.3	0
Regular secondary education	86.7	100
Dedicated special needs	0	–
Gaming experience questionnaire
(0–18 (mean, SD))	13.9 (2.5)	14.4 (3.0)	p = 0.38
ABI type (%)
Trauma	53.3	–
Haemorrhage	13.3	–
Oncology	20	–
Infection	13.3	–
Hemisphere of damage (%)
Left	6.7	–
Right	20	–
Bilateral	53.3	–
Unknown/other/not applicable	20	–
Days post-ABI (mean, range)	662 (32–2921)	–
Montreal Cognitive Assessment 0–30 (mean, SD)	27.5 (2.1)	28.4 (2.9)

3.1.2 Total duration

Total duration of assessment was comparable between the typically developing children and adolescents and those with ABI (F(1,34) = 3.30; p = 0.078). Duration was significantly shorter with the CM setup compared to the HMD setup (F(1,34) = 5.81; p = 0.022). When comparing the duration between groups for CM versus HMD, there was a marginally significant interaction, with no apparent difference in duration between CM and HMD assessment for the children and adolescents with ABI, yet a shorter duration for the CM assessment compared to the HMD assessment for the typically developing children and adolescents (F(1,34) = 3.96; p = 0.055; Fig. 1c). When HMD assessment was split for Oculus and HTC, total duration of assessment was again comparable between typically developing children and adolescents and those with ABI (F(1,32) = 0.59; p = 0.81). Duration was significantly shorter with the Oculus setup compared to the HTC setup (F(1,32) = 6.93; p = 0.013), which was comparable for typically developing children and adolescents and those with ABI (F(1,32) = 0.016; p = 0.90; see Fig. 1d).

Fig. 1

a) Distributions of total duration of assessment (in seconds). Even though the data is not normally distributed, the distributions of both groups (red line: head mounted display; blue line: computer monitor) are similar. Therefore we were able to use an ANOVA for analyses, despite the data not being normally distributed; b) Distributions of the number of products found. Even though the data is not normally distributed, the distributions of both groups (red line: head mounted display; blue line: computer monitor) are similar. Therefore we were able to use an ANOVA for analyses, despite the data not being normally distributed; c) Total duration (and standard deviation) of the assessment for children and adolescents with ABI and typically developing children and adolescents, split for setup (CM versus HMD); d) Total duration of the assessment for children and adolescents with ABI and typically developing children and adolescents, split for setup (Oculus versus HTC); e) Number of products found (and standard deviation) for children and adolescents with ABI and typically developing children and adolescents, split for setup (CM versus HMD); f) Number of products found for children and adolescents with ABI and typically developing children and adolescents, split for setup (Oculus versus HTC).

3.1.3 Number of products found

Total number of products found was significantly higher for the typically developing children and adolescents compared to those with ABI (F(1,34) = 7.81; p = 0.008). There was no difference in number of products found between the CM setup compared to the HMD setup (F(1,34) = 1.94; p = 0.17), and there was no significant interaction between group and setup (F(1,34) = 1.94; p = 0.17; Fig. 1e). When HMD assessment was split for Oculus and HTC, the total number of products found was significantly higher for the typically developing children and adolescents compared to those with ABI (F(1,32) = 10.2; p = 0.003). There was no difference in total number of products found between the Oculus setup and the HTC setup (F(1,32) = 0.36; p = 0.550), which was comparable for typically developing children and adolescents and those with ABI (F(1,32) = 0.001; p = 0.98; see Fig. 1f).

3.2 User-experience

With respect to Engagement, Transportation, Flow, and Presence, children and adolescents with ABI reported significantly lower scores compared to typically developing children and adolescents (F(1,34) = 5.55, p < 0.02; Fig. 2). Furthermore, these scores were overall significantly lower for the CM setup compared to HMD (F(1,34) > 7.7, p < 0.009), which was comparable for children and adolescents with ABI and typically developing children and adolescents (F(1,34) < 1.17, p > 0.29). Reported side effects were comparable between children and adolescents with ABI and typically developing children and adolescents (F(1,34) = 1.22, p = 0.28) and between setups (F(1,34) = 1.57, p = 0.22).

Fig. 2

Mean scores with 2 standard deviation error bars on the user-experience questionnaire, split on sub-categories Flow, Transportation, Engagement, Side effects, and Presence; a) overview of scores for the HMD setup for both groups; b) overview of scores for the CM setup for both groups.

When split for type of HMD, reported scores for Transportation, Flow, and Presence were lower for children and adolescents with ABI compared to typically developing children and adolescents (F(1,34) > 4.60, p < 0.04) and this was the case for both the Oculus and the HTC (F(1,34) < 0.54, p > 0.47). Only for Engagement and Side effects, scores did not differ between children and adolescents with ABI. Typically developing children and adolescents scored comparably on Engagement and Side effects (F(1,34) < 3.56, p > 0.07) and this was the case for both the Oculus and the HTC (F(1,34) < 0.13, p > 0.72).

3.3 Preference

Considering the typically developing children and adolescents, 51% preferred to use the HMD, whereas 49% did not report a specific preference. Considering those with ABI, 40% had a specific preference for the HMD, 36% had a specific preference for the CM, and 24% did not report a specific preference for setup. When split on Oculus versus HTC, there was an increase of children and adolescents with ABI reporting a specific preference for the HMD (20% versus 50%, respectively) and a decrease of children and adolescents reporting a specific preference for the CM (60% versus 24%, respectively). For the typically developing children and adolescents, the percentage remained comparable (Oculus: 52% preference for the HMD, 48% no preference; HTC: 51% preference for the HMD, 49% no preference). The percentage of children and adolescents without a specific preference for setup was comparable: 20% of the children and adolescents assessed with the Oculus versus 26% of the children and adolescents assessed with the HTC. For children and adolescents reporting cybersickness (n = 7; scores > 3 on item 5 of the user-experience questionnaire), 20% had a specific preference for the HMD setup, 31.4% for the CM setup, and 48.6% did not report a specific preference. For those reporting no cybersickness (n = 29; scores < 3 on item 5 of the user-experience questionnaire), 53.1% had a specific preference for the HMD setup, 11.1% for the CM setup, and 35.9% did not. As such a small number of children and adolescents reported cybersickness, these distributions should be interpreted cautiously.

4 Discussion

The aim of the current study was threefold: (1) to evaluate the feasibility of a VR simulation, determining whether children and adolescents with ABI were able to use non-immersive (CM) or immersive (HMD) VR setups; (2) to examine the user-experience per setup (CM or HMD); and (3) to determine which setup the children and adolescents with ABI prefer.

To the best of the authors’ knowledge, the feasibility of immersive VR for children and adolescents with ABI has not been studied before. This study demonstrated that the feasibility was high; children and adolescents with ABI were able to work with both the VR setups (CM and HMD), could find nearly all products on their shopping lists, and find the cash registers to pay for their products. This was comparable to the performance of typically developing children and adolescents. Moreover, the whole assignment was an appropriate length of time and comparable to conventional neuropsychological testing. The most notable, clinically relevant outcome of this study concerns the feasibility for both non-immersive and immersive VR in children and adolescents with ABI; preference for setups was equally distributed in our sample of children and adolescents and both setups were highly feasible to use. It appears that setup could be chosen to best suit the child as well as the clinical or research aim (see also below, where progression in hardware and software development is discussed).

The user-experience of immersive VR (HMD) was better compared to non-immersive VR (CM) in both children and adolescents with ABI and typically developing children and adolescents. Scores at the level of sub-categories for experience with the HMD were moderate up to good (approximately 8–12), depending on the specific category. Engagement, Transportation, Flow, and Presence are obviously important factors for a better VR user-experience, especially when aiming to use VR simulations in clinical settings to mimic the dynamic and complex situations of daily life. Even though not all children and adolescents reported high scores on Transportation and Flow (approximately 8–10), scores on Presence and Engagement were good (9–12). It is considered that with a high feeling of Presence it is easier to interact with the environment and to show natural behaviour [15]. This means children and adolescents probably demonstrate more natural behaviour in the HDM environment, which should be considered when designing future studies.

Virtual reality offers the opportunity to choose any possible environment. In this case, a virtual supermarket was chosen to appeal to a broad age range. A supermarket is a well-known environment for children and appeals to several cognitive aspects such as natural exploration, search behaviour, as well as a focused mission. Children and adolescents scored ‘Engagement’ as good, which includes being curious about the environment, attention was drawn to the environment and wanting to explore the environment (items 4, 9 and 13 on the user-experience questionnaire). Additionally, it is important to minimise side effects, such as nausea or dizziness, which are reported in the adult population [16]. Studies on side effects of VR simulations in children and adolescents are scarce [11, 12]; it appears that side effects or cybersickness are not often reported and do not tend to result in premature termination of assessment. In this study, the reported side effects were also low and comparable between children and adolescents with ABI and typically developing children and adolescents. There was no difference between setups (HMD and CM). Last, only six children and adolescents reported cybersickness in our study and none prematurely terminated the session. Despite a somewhat larger preference for the CM in the children and adolescents not reporting cybersickness, those in both groups did not report a specific preference.

Considering the current software and hardware, VR simulations are appropriate for use in paediatric rehabilitation. During and after the study, superior hardware (and software) has been released, enhancing the VR user-experience. However, this may have important consequences for the assessment that was observed in the current study as well. For example, visual acuity was enhanced in the HTC compared to the Oculus. It was therefore easier to read the advertisements and the labels of products or prizes in the supermarket when performing the task wearing the HTC compared to the Oculus. Observations and verbal feedback (not formal, informal verbalisation (of strategy maybe) during the assessment) during the assessment were that children and adolescents read more of the advertisements and labels and/or compared labels of products while verbalising their strategy. Superior hardware likely improves the experience inviting children and adolescents to explore further.

Improved VR software and hardware is important for future clinical and research aims. As previously mentioned, improved hardware changed the preference of VR set up in children and adolescents. Hence, enhanced experience most likely will lead to a more pronounced preference. Other outcome measures (i.e., eye tracking features and navigation features) will likely provide better insight into the (quality of) cognitive functioning in children and adolescents, leading to potentially more nuanced and detailed cognitive assessment.

This was a first exploration of the use of VR in the rehabilitation for children and adolescents with ABI. Although there is a great promise for VR in capturing cognitive strengths and weaknesses in a realistic environment, considerations should be made. The sample size of both groups was small. Also, there was relatively high bias in the type and/or level of education for both the typically developing children and adolescents and the children and adolescents with ABI. At time of testing, no children and adolescents with ABI were referred to dedicated special needs schools. Therefore, the feasibility, user-experience and preference of more severely affected (e.g., cognitive, motor, learning disabilities) children and adolescents remain unclear. The current sample included children and adolescents with ABI admitted for both inpatient and outpatient rehabilitation. As the subgroups were small, there was not enough power to analyse them. However, these participants did not prematurely abort the session or report side effects, such as nausea, which would warrant early termination and decrease the feasibility. Considering this was a monocentre study, generalisation to other settings is not possible. Nevertheless, major issues are not foreseen when using VR simulations in other clinical paediatric settings, as De Hoogstraat Rehabilitation Centre provides both inpatient and outpatient rehabilitation for a wide range of disabilities, according to Dutch standards and triage criteria.

Importantly, this study was not designed to assess cognitive function in a very precise and detailed manner. For feasibility and especially user-experience and preference, it works a lot better when children and adolescents have a task at hand. Larger shopping lists can be created to increase the assessment’s difficulty level. It remains unclear what the optimal time for cognitive assessment is. Also, it would potentially be desirable to consider a difficulty level or duration time that challenges the children and adolescents cognitively, which does not overstimulate them. In paediatric rehabilitation, it is important to adjust the rehabilitation programmes for complaints experienced in daily life. Children and adolescents frequently wish to be ‘normal’. There may be room for adjustments at (high) schools and in social groups, but children and adolescents would rather blend in. Nowadays, neuropsychological assessment and clinical judgement is used to anticipate the problems that can occur when children and adolescents with ABI return to daily life situations. VR simulations may help to signal realistic problems in multitasking environments. Alternatively, it could be argued that a VR simulation has the potential to gradually teach how to overcome overstimulation. This is essential for children and adolescents, given that social interactions at school, sports, (birthday) parties and during daily activities, such as doing errands, are of utmost importance for social and cognitive development.

Collectively, VR simulations demonstrate the potential to be developed into a cognitive assessment tool. It appears feasible to use it in both inpatient and outpatient settings for paediatric rehabilitation. Future studies should evaluate the feasibility of using VR for cognitive assessment in groups of children and adolescents with more severe motor, cognitive impairments, and/or intellectual disabilities. Additionally, more sensitive outcome measures derived from the assessment should be developed to predict cognitive functions and further relate theses outcome measures to cognitive complaints in daily life.

Footnotes

Acknowledgments

The authors thank Atoms2Bits for providing the materials and software assistance. A special thanks to all children and adolescents for their contribution. Additionally, the authors thank Marc van den Heerik for his help with collecting the data, Charlotte Southcombe for proofreading the manuscript and Andries van Iperen and Marie-Anne Kuiper for useful discussions.

Conflict of interest

The authors have no conflicts of interest to report.

Ethical considerations

Ethical approval was obtained with the Medical Ethical Committee of the University Medical Centre Utrecht (registration number 16–113). The study was subsequently approved by the Medical Ethics Review Committee of De Hoogstraat Rehabilitation Centre. Written informed consent was obtained from all children and adolescents and also from both parents from children under the age of 16.

Funding

This work was supported by HandicapNL under Grant [R2015010 and R201705758] to Tanja C.W. Nijboer, seed money grants by Focus Areas DataScience and Research IT from Utrecht University to Tanja C.W. Nijboer.

The appendices are available in the electronic version of this article: .

References

De Kloet

, Hilberink

, Roebroeck

, Catsman-Berrevoets

, Peeters

, Lambregts

SAM

, et al., Youth with acquired brain injury in the Netherlands: A multicentre study. Brain Inj. 2013;27(7–8):843–9. doi: 10.3109/02699052.2013.775496.

Naunheim

, Matero

, Fucetola

Assessment of patients with mild concussion in the emergency department, J Head Trauma Rehabil 2008;23(2):116–22. doi: 10.1097/01.HTR.0000314530.30401.70.

Bazarian

, McClung

, Shah

, Cheng

, Flesher

, Kraus

Mild traumatic brain injury in the United States, Brain Inj 2005;19(2):85–91. doi: 10.1080/02699050410001720158.

Hermans

, Winkens

, Te Winkel-Witlox

, van Iperen

, Caregiver reported problems of children and families 2–4 years following rehabilitation for pediatric brain injury, NeuroRehabilitation. 2012;30(3):213–7. doi: 10.3233/NRE-2012-0747.

Javouhey

, Guérin

, Amoros

, Haddak

, Ndiaye

, Floret

, et al., Severe outcome of children following trauma resulting from road accidents, Eur J Pediatr 2006;165(8):519–25 doi: 10.1007/s00431-006-0118-z.

Anderson

, Catroppa

, Memory outcome at 5 years post-childhood traumatic brain injury. Brain Inj. 2007;21 (13–14):1399–409. doi: 10.1080/02699050701785070.

Catroppa

, Anderson

, Godfrey

, Rosenfeld

, Attentional skills 10 years post-paediatric traumatic brain injury (TBI), Brain Inj 2011;25(9):858–69. 10.3109/02699052.2011.589794.

van den Heerik

, Spreij

, Visser-Meily

JMA

, Rentinck

ICM

, Verhoef

, Nijboer

TCW

, Spelend revalideren: Is er toekomst voor virtual reality in de cognitieve kinderrevalidatie? Een literatuuroverzicht, Neuropraxis 2016;20(5):134–46. doi:10.1007/s12474-016-0135-2.

Bowen

, Kreuter

, Spring

, Cofta-Woerpel

, Linnan

, Weiner

, et al., How we design feasibility studies, Am J Prev Med 2009;36(5):452–7. doi:10.1016/j.amepre.2009.02.002.

10.

Adamo-Villani

, Wilbur

, Effects of platform (immersive versus non-immersive) on usability and enjoyment of a virtual learning environment for deaf and hearing children. In: Mohler B, van Liere R, editors. EGVE Symposium. The Eurographics Association; 2008, pp. 17-20.

11.

Birnie

, Kulandaivelu

, Jibb

, Hroch

, Positano

, Robertson

, et al., Usability testing of an interactive virtual reality distraction intervention to reduce procedural pain in children and adolescents with cancer, J Pediatr Oncol Nurs 2018;35(6):406–16. doi:10.1177/1043454218782138.

12.

Ravindran

, Sazawal

, Solorzano

, Turnacioglu

, Experiences with virtual reality-based training for joint attention: A usability and initial feasibility pilot of FloreoVR’s Joint Attention Module in a special education school setting. Unpublished Manuscript. 2017.

13.

Spreij

, Visser-Meily

JMA

, Sibbel

, Gosselt

, Nijboer

TCW

, Feasibility and user-experience of virtual reality in neuropsychological assessment following stroke, Neuropsychol Rehabil 2022;32(4):499–519. doi:10.1080/09602011.2020.1831935.

14.

Nasreddine

, Phillips

, Bédirian

, Charbonneau

, Whitehead

, Collin

, et al., The montreal cognitive assessment, MoCA: A brief screening tool for mild cognitive impairment, J Am Geriatr Soc 2005;53(4):695–9. doi:10.1111/j.1532-5415.2005.53221.x.

15.

Parsons

, Virtual reality for enhanced ecological validity and experimental control in the clinical, affective and social neurosciences, Front Hum Neurosci 2015;9:660. doi: 10.3389/fnhum.2015.00660.

16.

Sharples

, Cobb

, Moody

, Wilson

, Virtual reality induced symptoms and effects (VRISE): Comparison of head mounted display (HMD), desktop and projection display systems. Displays. 2008;29(2):58–69. doi: 10.1016/j.displa.2007.09.005.