Development,concurrent validity and internal consistency of a simulator tool for assessing continued car driving after a brain injury/disease

Abstract

Introduction

Beyond relevant cognitive tests, a simulated driving activity could be useful to observe performance and behaviour in a standardized ‘driving-like’ situation and provide information on attention, reaction time and information processing speed when evaluating the ability to continue driving after a brain injury or disease. The aim was to develop and evaluate concurrent validity and internal consistency of a computerized simulated driving task tool.

Methods

Results from a new tool (CyberSiM, including three subtests and four result variables) were compared with results from the Trail Making Test, Useful Field of View test and Nordic Stroke Driver Screening Assessment. There were 126 healthy adults included in the study.

Results

The correlation analysis showed significant correlations (p<0.001) for CyberSiM reaction time and all cognitive tests except for Useful Field of View 1. The CyberSiM showed good internal consistency, with Cronbach’s alpha=0.85.

Conclusion

The analysis of concurrent validity showed conformity to most cognitive tests. CyberSiM might be a useful complement to cognitive testing with the opportunity to observe some behaviours ‘in a driving-like activity’. Further studies on clinical groups are needed to confirm its usefulness.

Keywords

Age attention cognition evaluation traffic safety occupational therapy

Introduction

A medical driving assessment after a brain injury or disease aims to decide whether the injury or disease has caused impairments in motor and/or cognitive functions that will seriously affect a person’s ability to resume driving or cause an increased risk for traffic accidents (Handley et al., 2017). At the operational level, driving is mainly automatic and relatively easy, especially because there are a lot of options to assist the driver and compensate for difficulties related to motor function limitations. However, driving a car safely in traffic is an extremely complex and cognitively demanding task. Even though there are assistive solutions available to compensate for some situations (such as support in keeping distance, parking and maintaining a certain speed), driving behaviour is based on a number of factors and continuous information from the environment. Thus, the driver still needs to make a lot of decisions. Occupational therapists are often involved in the assessments necessary for a physician to make a decision on a patient’s fitness to resume driving after an injury or disease affecting the brain. Because there is no consensus on which evaluation tools should be used for this purpose, the quality of such assessments varies between countries, specialist departments and individual therapists. Several studies have found that no one assessment tool is sufficient to draw conclusions on driving fitness (Larsson et al., 2007; Samuelsson et al., 2018; Schultheis and Whipple, 2014). The recommendation is to collect information from several areas of cognitive function by looking at test results as well as observing behaviour in simulated or real traffic activities in order to get as much information as possible about the cognitive skills and behaviour important for driving.

Three types of driving behaviour have been described in a hierarchical modelling approach: an operational control level where most behaviour is automatic and consists of online decisions on immediate control actions responding to the changing conditions (such as braking when something or someone crosses the street). A tactical control level includes mastering the vehicle according to information from ongoing traffic situations, referred to as behaviour. Driving behaviour is based on conscious decisions and thus judgement of traffic situations and risk avoidance. The final level is a planning or strategic level, including long-terms decisions such as which route to choose or at what time of the day to drive. The three levels can be identified and separated by task requirements and the cognitive processes (Shinar and Oppenheim, 2011). Interaction of several cognitive abilities is required, especially at the tactical level, where the most important factors for predicting motor vehicle collisions has been found to be executive function, divided and selective attention, visuospatial function, decision-making, processing speed and memory (Fields and Unsworth, 2017; Hird et al., 2016; Lafont et al., 2008; Seong-Youl et al., 2014). The planning or strategic level is the most difficult level to assess in a standardized way because it mainly concerns risk judgement and avoiding risks before the driver sits behind the wheel. Thus, it depends on personal as well as environmental factors.

Some commonly used cognitive tests to address the operational level assessing executive function, attention and visuospatial function have been found to be able to predict driving outcomes (Ball et al., 1993; Hird et al., 2016; McManus et al., 2015; Novack et al., 2006).

As a complement to traditional cognitive assessment, several studies have described the use of some kind of simulated driving (Hird et al., 2016; Samuelsson et al., 2018). The predictive value of simulated driving to identify safe versus unsafe drivers has been discussed, and diverse results have been presented (Matas et al., 2016; Samuelsson et al., 2018). Simulated driving is not a homogeneous concept but includes several different simulator models and software scenarios, which makes it difficult to draw general conclusions concerning its applicability. However, simulated driving offers an opportunity to observe performance and behaviour in a standardized ‘driving-like’ situation and at the same time obtain results at an operational and a tactical level. However, simulated driving might not be useful for all patients because of simulator sickness, especially in the older population (Brooks et al., 2010). Brooks et al. (2010) state that simulator sickness can potentially confound data and influence participant dropout rates.

The aim was to develop and evaluate concurrent validity and internal consistency of a computerized simulated driving task tool.

Methods

Study process

To validate the CyberSiM driving tool (Cybercom, Sweden AB), results from CyberSiM were analysed and compared with results from three commonly used driving assessment tools: Useful Field of View (UFOV) (Clay et al., 2005); Trail Making Test (TMT) (Mathias and Lucas, 2009) and the Nordic Stroke Driver Screening Assessment (NorSDSA) (Larsson et al., 2007).

Two trained test leaders (one psychology student and one psychologist) administrated the tests individually to the participants over a 6-month period. All tests were presented at the same occasion during a 2-hour session without any break. The tests were presented in a specific order (UFOV, TMT, NorSDSA, CyberSiM) with standardized instructions. After each test session, all results were transferred into an anonymized dataset, which was used for further analyses. Information on age, gender and educational level were collected through a study survey before each test session.

Participants

Healthy individuals were recruited by advertising on a digital forum at one university in Sweden, information sent to retirement organizations in two cities in Sweden, and through snowball sampling. Information about the study aim and study content was presented to each participant in writing. Inclusion criteria were age ≥20 years, no known disease or injury affecting the brain, having a driving licence and being a regular car driver.

CyberSiM

CyberSiM was developed by a local company participating in a competition financed by Sweden’s innovation agency (VINNOVA, a state agency). The specification of the expected outcome was identified by the Traffic Medical Centre at the University Hospital in Linköping, Sweden, where an old simulator with old, inappropriate software was in use. The development of the equipment was initiated through collaboration among the University Hospital (IT Centre and Traffic Medical Centre), Linköping University and the local company, Cybercom AB. CyberSiM consists of specially developed software installed on a laptop attached to a steering wheel. The steering wheel is used for steering and for responding to visual stimuli on the screen by pressing the paddles located on the left and right of the steering wheel. No gas or brake pedals are used, and the speed is constant (40 km/h). The participant is asked to ‘drive on a road’ while steering and reacting to stimuli shown on the left or right or both sides of the screen simultaneously (Figure 1).

Figure 1.

CyberSiM: the steering wheel with two paddles and the screen.

CyberSiM includes three subtests representing increasing challenges regarding procedural memory, attention, information processing and reaction speed.

The first subtest includes a simple stimulus–response task where the person is asked to press the right button every time an arrow appears on the screen (pointing to the right and shown on the right or left side of the screen) while ‘driving’.

The second subtest shows an arrow pointing to the left or right on the left or right side of the screen. The person is asked to respond by pressing the right or left button depending on the direction of the arrow regardless of which side of the screen.

The last subtest shows two arrows at the same time, one on the left and one on the right side of the screen. When the arrows are pointing in opposite directions, the person is informed not to press any button (divergent response); when both arrows point in the same direction, the person is expected to press the button representing the direction of the arrows.

A short exercise before each test was used to inform and check that each participant had understood each subtest. He/she was told to drive on the right side of the road, and steer around obstacles such as a parked car at the side of the road. The participant was also asked to respond each time a traffic sign popped up on the screen (arrow to the left or right) by pressing one of the two paddles on the steering wheel as quickly as possible.

Reaction time (in milliseconds) and the number of missed and wrong responses were registered and reported in the software protocol. For the first two subtests, results were presented for stimuli showing on the left and on the right of the screen. For the last subtest, results were presented for both sides together. In addition, an algorithm was developed to illustrate how well the person had managed to keep the ‘car’ on the right side of the road (that is, how well the person managed to avoid driving on or over the road markings to the right or left); this algorithm was presented as the wobbling factor.

Three commonly used assessment tools were used for validation of the CyberSiM test.

Useful Field of View test

UFOV is a computer-based test recommended as a screening measure for fitness to drive (Marshall et al., 2007; Visual Awareness Research Group, 2009). UFOV 6.1.4 consists of three subtests that assess the accuracy of visual processing under increasingly complex tasks. The participant must detect, identify and localize briefly presented targets and respond to them by pointing to the right spot on a touch screen or using a mouse. For each of the three subtests, UFOV automatically adjusts the duration of stimuli presentation as needed, depending on the accuracy of the participant’s responses (Visual Awareness Research Group, 2009). This process of tracking the perceptual threshold continues until a stable estimate of 75% correct is calculated. The accuracy of each response is measured.

This subtest is the least challenging and requires processing speed. The participant must detect, identify and localize briefly presented vehicles: a car or truck presented centrally on the screen.

The second subtest requires divided attention. The participant is required to simultaneously identify a centrally located car or truck as well as the location of a car on the periphery.

The third subtest requires selective attention and is identical to subtest 2 except that the car on the periphery is embedded in distractors (47 triangles).

UFOV provides one score, reported in milliseconds, for each of the three subtests as well as a total score including all subtests. A risk report related to car driving is automatically calculated based on all three subtests, including 14 outcomes or combinations of scores. Test–retest reliability for risk category (composite) was 0.88 (Visual Awareness Research Group, 2009). A meta-analysis based on eight independent studies showed high validity (Cohen’s d=0.945) when an older version of UFOV was compared with driving measures (Ball et al., 2005). The old version as well as the newer version of UFOV have been used for evaluation of driving fitness because the old version has been shown to have a high level of prediction of crash risk (Ball et al., 2006; Clay et al., 2005; McManus et al., 2015).

Trail Making Test

There are several versions of the TMT. The one used in this study is a paper-based test consisting of two subtests.

TMT-A includes 25 scattered numbers where the participant draws lines sequentially connecting the numbers as quickly as possible without lifting the pencil (Tombaugh, 2004).

TMT-B is more complex than TMT-A, because the participant must alternate between 13 numbers and 12 letters in two sequences (Vaucher et al., 2014). The results are expressed as the time taken to complete each subtest (Tombaugh, 2004).

According to Tombaugh (2004), the TMT assesses visual searching, scanning, processing speed, mental flexibility and executive functions. In their review, however, Sanchez-Cubillo et al. (2009) explained that there is a lack of consensus on the exact nature and contributions of different cognitive functions and concluded that TMT-A mainly requires visuoperceptual function and TMT-B requires primarily working memory and secondarily task-switching ability. The TMT results are related to age and education, and norm values based on that are available (Tombaugh, 2004). Roy and Molnar (2013) found in their review that different cut-off scores are used for TMT-B for predicting fitness to drive. In addition, TMT-B has been found to be useful in dementia-screening as a means to assess driving concerns. However, results from the TMT should be combined with other driving assessments and should not be used as a standalone tool (Patel, 2014).

Nordic Stroke Driver Screening Assessment

The NorSDSA (Lundberg, 2003) is a paper-based test revised and further developed by the British Stroke Driver Screening Assessment (SDSA) (Lincoln et al., 2012; Nouri and Lincoln, 1992). The Nordic version of the SDSA has been adapted for right-hand traffic, and some road signs in the original English version have been replaced. In addition, a new discriminant analysis has been conducted for the Nordic version, resulting in an adjusted base for classification of pass or fail (Lundberg et al., 2003).

The NorSDSA consists of four subtests:

Dot cancellation, which consists of dots lined up in groups of three, four or five. The participant crosses out all groups with four dots. Results are expressed as time taken in seconds, missed groups of four and crossed out groups of three or five (so-called false alarms).

The direction test consists of a 4 × 4 square matrix and 16 stimulus cards showing a lorry and a car travelling in different directions. Four large (representing the lorry) and four small (representing the car) directional arrows are placed along the side and top of the matrix. The participant then places the stimulus cards in the square that corresponds correctly to one small and one large arrow. Maximum time is 5 minutes and results are expressed in points, one point given for each correctly placed vehicle.

The compass test is similar to but more complex than the direction test. The same matrix is used, but the arrows represent eight compass directions and the stimulus cards show a roundabout with two cars leaving or driving into the roundabout on two of eight different roads. The participant is given 28 cards, including one practice card, of which only 16 can be correctly placed. He/she places the card so that the compass direction corresponds to the road each car is travelling on. Maximum time and scoring is the same as for the direction test.

The road sign recognition test consists of 12 cards depicting different traffic situations; they are placed in front of the participant. He/she is given 20 cards with traffic signs, 12 of which are to be placed on top of the correct traffic situation. Thus, the participant has to discard redundant cards. A first result is calculated after 3 minutes and a second after another 2 minutes.

According to the NorSDSA manual (Lundberg, 2003), dot cancellation puts demands on visual scanning and visual perception, sustained and selective attention, and speed. The direction test assesses simple visuospatial functions and the compass test assesses scanning and processing of visual material such as psychomotor speed and attention. Road sign recognition is described as requiring visual scanning and processing as well as verbal and visuospatial memory. Based on the results for dot cancellation, compass and road sign recognition, the test provides a weighted overall total score. The higher the score the better the result.

SDSA and NorSDSA have been found to correctly classify a sufficient number of unsuitable drivers in some studies (Lundberg et al., 2003) but has been found to be inaccurate for neurological conditions other than stroke and to require additional cognitive tests in some studies (Lincoln et al., 2012; Selander et al., 2010). There is a lack of studies describing norm values and results for validity and reliability for the SDSA and NorSDSA, which should be taken into account when using these tools.

Statistics

Statistical analyses were conducted using the Statistical Package for the Social Sciences (SPSS) 23.0. Non-parametric as well as parametric statistical methods were used. UFOV subtests and the sum of the subtests (total score), time taken for TMT-A/-B and NorSDSA total score were used for validation of the three subtests of the simulator tool. Spearman’s rank order correlation test was used for correlation analyses. Reaction time for the subtest in CyberSiM regarding stimuli to the left versus the right side of the screen showed no difference and was thus merged and a mean value was used. CyberSiM 1 t=0.002, p=0.998 and CyberSiM 2 t=1.445, p=0.151. The accepted level of significance was p≤0.001. Student’s t test was used for comparisons on demographic data and to compare reaction time for stimuli to the left versus the right on the CyberSiM subtests 1 and 2. Internal consistency of the entire CyberSiM test was tested using Cronbach’s alpha. A linear regression analysis was performed using CyberSiM 3 as the dependent variable.

Results

Participant characteristics

In total, 127 individuals were included in the study (Table 1). One person was excluded from the analysis due to inability to perform all the tests. Thus, the final research group included 126 individuals, 72 women (57%) and 54 men (43%), who were all recruited from two cities in Sweden. Mean age was 48.8 years (SD, ±19.1 years; range, 20–85 years).

Table 1.

Demographic data for the participants (n=126).

	Percentage
Gender
Men	43
Women	57
Education
≤9 years	6
10–12 years	17
≥12 years	77

The results for the different tests are presented as means and standard deviation for all tests and subtests (Table 2). The participants included in the study had very few misses and wrong responses for the CyberSiM subtests, explaining the low mean values.

Table 2.

A summary of the results for all subtests (n=126).

	Mean (SD)
UFOV subtest 1: processing speed	15.1 (1.6)
UFOV subtest 2: divided attention	35.4 (52.9)
UFOV subtest 3: selective attention	82.6 (69.0)
UFOV total subtests 1–3	133.1 (112.3)
TMT-A (s)	24.7 (7.5)
TMT-B (s)	61.5 (22.2)
NorSDSA, total score	2.8 (0.8)
CyberSiM subtest 1 (ms)	549 (132)
Wobbling	0.45 (0.2)
Misses	0.14 (0,4)
Wrong response	0.01 (0.1)
CyberSiM subtest 2 (ms)	936 (173)
Wobbling	0.49 (0.2)
Misses	0.07 (0.3)
Wrong response	1.0 (1.7)
CyberSiM Subtest 3 (ms)	1186 (204)
Wobbling	0.50 (0.25)
Misses	0.02 (0.2)
Wrong response	0.46 (0.8)

UFOV: Useful Field of View test; TMT: Trail Making Test; NorSDSA: Nordic Stroke Driver Assessment.

There was no difference between men and women regarding the UFOV subtests or total score, CyberSiM subtests, TMT-A or NorSDSA. Women had a lower mean (faster) than men for results on TMT-B; the mean for women was 56s (95% CI, 47.1–60.4s) and the mean for men was 69s (95% CI, 61.6–76.6s, p=0.007). Educational level had an impact only on the results for UFOV subtest 3 (p=0.011). Age was significantly correlated with all test results (Table 3).

Table 3.

Correlation analysis for validation of CyberSiM.

	UFOV 1 (ms)	UFOV 2 (ms)	UFOV 3 (ms)	UFOV total (ms)	TMT-A (s)	TMT-B (s)	NorSDSA total	CyberSiM 1 (ms)	CyberSiM 1 (wobbling)	CyberSiM 2 (ms)	CyberSiM 2 (wobbling)	CyberSiM 3 (ms)	CyberSiM 3 (wobbling)
UFOV 1 (ms)	1.000
UFOV 2 (ms)	0.57***	1.000
UFOV 3 (ms)	0.33***	0.61***	1.000
UFOV total (ms)	0.36***	0.66***	0.99***	1.000
TMT-A (s)	0.17	0.40***	0.63***	0.63***	1.000
TMT-B (s)	0.04	0.31***	0.44***	0.45***	0.65***	1.000
NorSDSA total	−0.08	−0.35***	−0.50***	−0.50***	−0.44***	−0.48***	1.000
CyberSiM 1 (ms)	0.11	0.34***	0.40***	0.40***	0.42***	0.37***	−0.36***	1.000
CyberSiM 1 wobbling	0.10	0.34***	0.40***	0.41***	0.31***	0.11	−0.15	0.27	1.000
CyberSiM 2 (ms)	0.23	0.36***	0.58***	0.56***	0.50***	0.40***	−0.37***	0.61***	0.27	1.000
CyberSiM 2 wobbling	0.10	0.37***	0.43***	0.45***	0.36***	0.18	−0.22	0.23	0.88***	0.36***	1.000
CyberSiM 3 (ms)	0.11	0.40***	0.65***	0.64***	0.60***	0.46***	−0.42***	0.57***	0.29***	0.82***	0.37***	1.000
CyberSiM 3 wobbling	0.16	0.42***	0.50***	0.52***	0.40***	0.22	−0.23	0.26	0.85***	0.40***	0.91***	0.43***	1.000
Age (years)	0.29***	0.56***	0.78***	0.79***	0.54***	0.43***	–0.45***	0.36***	0.34***	0.70***	0.42***	0.67***	0.49***

A correlation coefficient of significance (***p<0.001) was considered acceptable.

UFOV: Useful Field of View test; TMT: Trail Making Test; NorSDSA: Nordic Stroke Driver Assessment.

Correlation analysis

Concurrent validity of the CyberSiM assessment tool was analysed by a correlation analysis including the following commonly used tests: UFOV, TMT-A, TMT-B and the NorSDSA. Results for reaction times for CyberSiM 1 and 2 are presented based on mean values for responses to the left and right (Tables 2 and 3). Regarding UFOV risk categories, 91% of participants were classified as category 1 (very low risk) and 9% were classified as category 2 (low risk) (Visual Awareness Research Group, 2009). Thus, the risk category results were not used for the correlation analyses. All test results were analysed according to their inter-correlation. Interpretation of the correlation coefficient as described by Mukaka (2012) was used. A correlation coefficient r_s<0.3 was considered negligible; r_s>0.3 as low; r_s>0.5 as moderate; r_s>0.7 as high and r_s>0.9 as very high. In addition, significant results (***p<0.001) are shown in Table 3. Results for the different subtests of CyberSiM showed that reaction time for CyberSiM subtests 2 and 3 had a moderate correlation with UFOV subtest 3 (r_s=0.58 and r_s=0.65, respectively) as well as UFOV total score (r_s=0.56 and r_s=0.64, respectively; see Table 3). In addition, reaction time for CyberSiM subtest 3 had a moderate correlation with TMT-A (r_s=0.60). CyberSiM subtest 1 had a low correlation with all other tests (r_s<0.5). However, the statistical analysis showed that correlation coefficients were significant (p<0.001) for CyberSiM reaction time and all the cognitive tests except for UFOV 1.

The correlation analysis on misses and wrong responses reported for the three different subtests on CyberSiM showed no significant correlation with any of the other variables (p>0.05). Results for the variables included in CyberSiM showed good internal consistency with Cronbach’s alpha=0.85. A linear regression analysis with CyberSiM 3 as the dependent variable (because this seemed to be the most demanding task) showed a significant value for UFOV 3 (p=0.008, β=0.323) and age (p=0.006, β=0.252) (Table 4). The adjusted R² was 0.360 and the F value was 11.032.

Table 4.

Results from a linear regression model with CyberSiM 3 as the dependent variable.

	β	t	Significance
(Constant)		6.163	0.000
UFOV I	−0.094	−1.255	0.212
UFOV II	−0.087	−0.848	0.398
UFOV III	0.323	2.693	0.008
TMT-A (s)	0.207	1.905	0.059
TMT-B (s)	0.054	0.525	0.600
NorSDSA total score	0.001	0.010	0.992
Age	0.252	2.828	0.006

UFOV: Useful Field of View; TMT: Trail Making Test; NorSDSA: Nordic Stroke Driver Assessment.

Discussion

The aim was to develop and evaluate concurrent validity and internal consistency of a computerized simulated driving task tool. Healthy individuals were recruited for data collection, and measurements for comparison were chosen based on former studies and clinical experience.

The results showed that the correlation coefficient for CyberSiM was strongest when comparing results from subtests 2 and 3 with UFOV subtest 3, which is considered the most demanding, and UFOV total score (r_s=0.56–0.64). This indicates that CyberSiM subtests 2 and 3 place demands on divided attention. CyberSiM subtest 1 had low correlation coefficients with UFOV 2 and 3 and total score as well as with TMT-A, TMT-B and NorSDSA (r_s<0.42). UFOV 1 had negligible and non-significant correlation coefficients with every other test in this study (r_s<0.3), indicating that this subtest might be seen as a training session. Results for CyberSiM 3 were considered as dependent variables in the linear regression analysis, showing that UFOV 3 and age were the only significant variables (Table 4). None of the other test results were significant explanatory variables for CyberSiM 3, indicating that this test might add other cognitive aspects, yet to be studied, preferably in an on-road test.

The wobbling factor illustrates how well the participant manages to perform the driving task. Based on the group of healthy individuals in this study, this ‘driving’ activity should be performed without any problems and with no misses and wrong responses. However, these variables are assumed to be of more relevance in a patient group, which remains to be studied. For both UFOV and CyberSiM, participants are asked to identify and respond to visual stimuli on the screen as quickly as possible. When using CyberSiM, participants are simultaneously steering a ‘car’ on the right side of the road, which demands a simultaneous activity not comparable with the UFOV scenario or with any of the other assessment tools. CyberSiM subtest 3 also includes a scenario with a divergent task, which is not comparable with other tests, which might affect concurrent validity. The participant is requested to make a response every time the stimulus includes two arrows pointing in opposite directions. This part of the test thus includes procedural and working memory, cognitive processing and a decision-making process, as well as the attention, stimuli response and steering tasks. This subtest thus provides an executive challenge, which is difficult to identify in the other assessment tools and thus might explain the relevance of the moderate correlation coefficients.

TMT-B has been found to be a predictor of driving performance and motor vehicle collisions (Seong-Youl et al., 2014) and is therefore useful in driving assessments. Results from the present study showed low correlation coefficients (r_s<0.5) between TMT-B and the other assessment tools, indicating that TMT-B might contribute to an assessment by measuring other aspects of cognitive abilities, such as task-switching ability (Sanchez-Cubillo et al., 2009). CyberSiM subtest 3 (reaction time) and TMT-A had a correlation coefficient of r_s=0.60, indicating that these subtests place some demands on visuoperceptual abilities, which is not surprising (Sanchez-Cubillo et al., 2009).

Deficits in vision, visual field, selective and divided attention, information processing speed and executive functioning have all been found to have an impact on driving safety among individuals with a brain injury or brain disease (Ortoleva et al., 2012). Even though several retrospective studies have shown that there is an increased risk of traffic accidents in the brain-injured population, in people with diagnosed dementia and among the elderly, there is still a lack of consensus on how different cognitive dysfunctions affect driving performance. Which assessment tools should be used and the cut-offs for different assessment tools are still topics for discussion. Age was shown to be significantly correlated with the results on all tests included in this study even though the correlation coefficient varied in strength. In several studies, age has been found to be a strong predictor for cognitive test results related to car driving, thus age-related norm values are available for some of the commonly used tools (Edwards et al., 2006; Tombaugh, 2004). Age-related norm values should be considered as a useful comparison tool; however, when using these instruments on healthy individuals, there is a risk for a ceiling or floor effect, such as for the wobbling, misses and wrong responses in the CyberSiM tests.

CyberSiM was developed through the cooperation of an experienced multi-professional medical team at one university hospital, one software developing company and the medical technician department at the same university hospital. The need to observe patients in different situations without the risk of simulator sickness (Matas et al., 2016) or inability to understand the task due to age and/or computer inexperience served as guidance in deciding on the design and appearance of the equipment. Participants included in the present study did not experience any simulator sickness symptoms.

The therapist can observe every part of the assessment process and identify difficulties based on inattention, processing speed, fatigue or an inability to interpret information. Results are presented for the left versus right side of the screen to identify visual deficits due to, for example, visual hemi-inattention. Face validity was discussed throughout the development process. Concurrent validity was evaluated by comparing results from the new computer-based tool with other well-established assessment tools; although not all tools (NorSDSA) are considered valid for all diagnoses, they are commonly used and available to occupational therapists, and used within traffic medicine in many countries (Edwards et al., 2006; Schultheis and Whipple, 2014; Selander et al., 2010). In order to further validate the new tool for assessment, further studies are needed. Results from clinical studies on patient groups as well as results from on-road testing will be needed to confirm the validity of the CyberSiM test.

Practical applications

For many people, continued driving is associated with community re-integration, wellbeing and possible role keeping (Winter et al., 2017). A number of assessment tools have been suggested and described as useful when making decisions on continued car driving after a brain injury or disease. However, no single tool or combination of tools has yet been found to explain every outcome of an on-road or multi-professional final evaluation (Ball et al., 2006; Hird et al., 2016; Samuelsson et al., 2018). Because driving is an activity that requires many different motor and cognitive abilities besides strategic decision-making, as described earlier by Shinar and Oppenheim (2011), the ability to continue this activity should be thoroughly and carefully evaluated. CyberSiM might serve as a complementary tool in an assessment process for continued driving; however, further studies are needed to confirm this. The new equipment seems to capture relevant information for car driving (especially subtests 2 and 3) based on significant correlation coefficient results for commonly used tests in clinical practice. Based on relatively low correlations between CyberSiM subtest 1 and other measurement results, this subtest might best be considered as a training session. In addition, CyberSiM gives the examiner an opportunity to study how an individual manages simultaneous tasks because CyberSiM requires the ‘driver’ to steer the ‘car’ and at the same time respond to visual stimuli. CyberSiM has not been correlated to on-road performance and should not be considered to replace that.

Behaviour-related information as described by Shinar and Oppenheim (2011) is still not fully addressed by the tests included in this study and thus needs to be further assessed using other information methods, such as on-road observation, questionnaires and interviews. An on-road evaluation includes both actual driving skills in a complex environment as well as decision-making based on environmental as well as cognitive aspects.

Study limitations and strengths

The study was conducted using a sample of healthy individuals from two different cities. The study population was healthy and does not correspond to the population that the instruments are designed for. Some of the instruments have signs of ceiling and/or floor effects, and thus the tasks might be too easy for the study population compared with those with cognitive deficits.

In addition, the study group was not fully comparable with the adult norm population in Sweden because the participants had a higher mean education level. This might have affected the results because some studies have shown that educational level might have an impact on the results for UFOV as well as TMT but not for the NorSDSA (Edwards et al., 2006; Selander et al., 2010; Tombaugh, 2004). However, age has been found to have a much stronger impact on results than education in several studies (Novack et al., 2006; Selander et al., 2010; Tombaugh, 2004), and this was confirmed by the results from the present study.

The same test leader performed all the tests, which should be considered as a strength, because we do not know the interrater reliability of all the tests. In addition, all tests were presented in the same order and in the same way for all participants, which should have a positive effect on reliability. The tests chosen for concurrent validation were all identified as measurement tools of relevant cognitive factors that have been found to be linked to driving performance and thus valid for the research question (McManus et al., 2015; Roy and Molnar, 2013; Selander et al., 2010).

The newly developed assessment tool, CyberSiM, was tested for internal consistency but not yet for repeated measures, which is a limitation. Participants were not allowed to repeat their ‘driving’ at the time of data collection, which was a way to deal with the lack of knowledge on test–retest reliability; however, this should be further studied. The tests were presented in the same order for all participants, which might lead to order effects. In addition, we plan to use the new tool in a study including relevant patients, which will make it possible to make a final recommendation/decision to compare results.

Key findings

CyberSiM has concurrent validity in relation to other commonly used tools for cognitive assessments related to car driving.

The different CyberSiM variables were found to have good internal consistency.

What the study has added

CyberSiM did not contribute to the explanation of the results for UFOV, which was considered to be the most relevant dependent variable, but might illustrate other aspects of importance.

Footnotes

Acknowledgements

We would like to thank all participants who contributed to this study. We would also like to thank Professor Birgitta Bernspång and occupational therapist Eva Lundström, Umeå University and Umeå University Hospital, for all practical support during the time for data collection.

Research ethics

Ethical approval for this study was obtained from the Ethics Regional Board in Linköping (Dnr 2016–353-31). No medical or other risks related to participation in the study could be identified.

Consent

All participants were given written information about the study and provided informed consent. All data forms were coded and no individual could be identified by the researchers. All procedures were in accordance with the ethical standards of the Declaration of Helsinki.

Declaration of conflicting interests

The authors declared no potential conflict of interest with respect to the research, authorship and/or publication of this article.

Funding

The authors disclose receipt of the following financial support for the research and authorship of this article: VINNOVA (2013-04010, 2013); County Council of Östergötland, Sweden (LIO-582071, 2016) and the Medical Research Council of Southeast Sweden (FORSS-654091, 2016; FORSS-755541, 2018).

Contributorship

All authors researched literature and contributed to the methodology of the project, as well as the statistical analysis plan. Maria Tropp recruited all participants and did all data collection. Kersti Samuelsson applied for ethical approval. Kersti Samuelsson and Ewa Wressle did all statistical analysis and all authors interpreted data. All authors reviewed and edited the manuscript and approved the final version.

References

Ball

Clay

Wadley

, et al. (2005) Predicting driving performance in older adults with the useful field of view test: A meta-analysis. In: Proceedings of the Third International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, Rockport, Maine, June 27–30, Iowa City. Iowa: University of Iowa, Public Policy Center, pp.51–57.

Ball

Owsley

Sloane

, et al. (1993) Visual attention problems as a predictor of vehicle crashes in older drivers. Investigative Ophthalmology and Visual Science 34(11): 3110–3123.

Ball

Roenker

Wadley

, et al. (2006) Can high-risk older drivers be identified through performance-based measures in a department of motor vehicles setting? Journal of American Geriatric Society 54(1): 77–84.

Brooks

Goodenough

Crisler

, et al. (2010) Simulator sickness during driving simulation studies. Accident Analysis and Prevention 42(3): 788–796.

Clay

Wadley

Edwards

, et al. (2005) Cumulative meta-analysis of the relationship between useful field of view and driving performance in older adults: Current and future implications. Optometry and Vision Science 82(8): 724–731.

Edwards

Ross

Wadley

, et al. (2006) The useful field of view test: Normative data for older adults. Archives of Clinical Neuropsychology 21: 275–286.

Fields

Unsworth

(2017) Revision of the competency standards for occupational therapy driver assessors: An overview of the evidence for the inclusion of cognitive and perceptual assessments within fitness-to-drive evaluations. Australian Occupational Therapy Journal 64(4): 328–339.

Handley

Thomas

McKenna

, et al. (2017) On the road again: Assessing driving ability in patients with neurological conditions. Practical Neurology 17(3): 203–206.

Hird

Egeto

Fischer

, et al. (2016) A systematic review and meta-analysis of on-road simulator and cognitive driving assessment in Alzheimer’s disease and mild cognitive impairment. Journal of Alzheimer’s Disease 53(2): 713–729.

10.

Lafont

Laumon

Helmer

, et al. (2008) Driving cessation and self-reported car crashes in older drivers. The impact of cognitive impairment and dementia in a population-based study. Journal of Geriatric Psychiatry and Neurology 21(3): 171–182.

11.

Larsson

Lundberg

Falkmer

, et al. (2007) A Swedish survey of occupational therapists’ involvement and performance in driving assessments. Scandinavian Journal of Occupational Therapy 14: 215–222.

12.

Lincoln

Radford

Nouri

(2012) Stroke Drivers Screening Assessment: Revised Manual. Nottingham, UK: University of Nottingham.

13.

Lundberg

(2003) Manual for the Nordic Stroke Driver Assessment (NorSDSA). Blidö, Sweden: NorSDSA Produktion Görel Caneman and Catarina Lundberg [in Swedish].

14.

Lundberg

Caneman

Samuelsson

, et al. (2003) The assessment of fitness to drive after a stroke: The Nordic Stroke Driver Screening Assessment. Scandinavian Journal of Psychology 44(1): 23–30.

15.

McManus

Cox

Vance

, et al. (2015) Predicting motor vehicle collisions in a driving simulator in young adults using the Useful Field of View Assessment. Traffic Injury Prevention 16(8): 818–823.

16.

Marshall

Molnar

Man-Son-Hing

, et al. (2007) Predictors of driving ability following stroke: A systematic review. Topics in Stroke Rehabilitation 14(1): 98–114.

17.

Matas

Nettelbeck

Burns

(2016) Assessment of driving simulator validity and acceptability for older adult drivers. Advances in Transportation Studies 39: 69–88.

18.

Mathias

Lucas

(2009) Cognitive predictors of unsafe driving in older drivers: A meta-analysis. International Psychogeriatrics 21(4): 637–653.

19.

Mukaka

(2012) A guide to appropriate use of correlation coefficient in medical research. Malawi Medical Journal 24(3): 69–71.

20.

Nouri

Lincoln

(1992) Validation of a cognitive assessment: Predicting diving performance after stroke. Clinical Rehabilitation 6: 275–281.

21.

Novack

Baños

Alderson

, et al. (2006) UFOV performance and driving ability following traumatic brain injury. Brain Injury 20(5): 455–461.

22.

Ortoleva

Brugger

Van der Linden

, et al. (2012) Prediction of driving capacity after traumatic brain injury: A systematic review. Journal of Head Trauma Rehabilitation 27(4): 302–313.

23.

Patel

(2014) Neuropsychological factors associated with Useful Field of View. MSc Thesis, College of Arts and Sciences, Ohio University.

24.

Roy

Molnar

(2013) Systematic review of the evidence for Trails B cut-off scores in assessing fitness-to-drive. Canadian Geriatrics Journal 16(3): 120–142.

25.

Samuelsson

Modig-Arding

Wressle

(2018) Driving after an injury or disease affecting the brain: An analysis of clinical data. British Journal of Occupational Therapy 81(7): 376–383.

26.

Sanchez-Cubillo

Perianez

Adrover-Roig

, et al. (2009) Construct validity of the Trail Making Test: Role of task-switching, working memory, inhibition/interference control, and visuomotor abilities. Journal of the International Neuropsychological Society 15(3): 438–450.

27.

Schultheis

Whipple

(2014) Driving after traumatic brain injury: Evaluation and rehabilitation interventions. Current Physical Medicine and Rehabilitation Reports 2(3): 176–183.

28.

Selander

Johansson

Lundberg

, et al. (2010) The Nordic Stroke Driver Screening Assessment as predictor for the outcome of an on-road test. Scandinavian Journal of Occupational Therapy 17(1): 10–17.

29.

Seong-Youl

Jae-Shin

A-Young

(2014) Cognitive test to forecast unsafe driving in older drivers: Meta-analysis. NeuroRehabilitation 35: 771–778.

30.

Shinar

Oppenheim

(2011) Review of models of driver behaviour and development of a unified driver behaviour model for driving in safety critical situations. In: Cacciabue

Hjälmdahl

Luedtke

, et al. (eds) Human Modelling in Assisted Transportation. Milan: Springer-Verlag, p. 215.

31.

Tombaugh

(2004) Trail Making Test A and B: Normative data stratified by age and education. Archives of Clinical Neuropsychology 19(2): 203–214.

32.

Vaucher

Herzig

Cardoso

, et al. (2014) The trail making test as a screening instrument for driving performance in older drivers; A translational research. BMC Geriatrics 14(1): 123.

33.

Visual Awareness Research Group (2009) UFOV User’s Guide Version 6.1.4. Birmingham, Alabama: Visual Awareness.

34.

Winter

Moriarty

Short

(2017) Self-reported driving difficulty in veterans with traumatic brain injury: Its central role in psychological well-being. Physical Medicine and Rehabilitation 9(9): 901–909.