Abstract
In rehabilitation settings for people with possible or confirmed neurological conditions, occupational therapy practitioners must consider multiple areas of function. Neurological conditions can result in a wide variety of impairments that affect daily activities, including the ability to drive. Cognitive functions of attention, working memory, visual–motor coordination, motor and mental speed, and visual scanning significantly contribute to predicting driving status of people after neurorehabilitation (Perumparaichallai et al., 2014). In addition, measures of executive functioning, visual search, and psychomotor speed accurately predict road test performance (Niewoehner et al., 2012). Proper assessment can allow therapists to determine whether an individual can likely complete the activity, and it can also help to guide treatment before in situ assessment and training, which is especially important in safety-sensitive tasks such as driving (Niewoehner et al., 2012).
Visual attention refers to one’s ability to carefully observe objects to determine information about their features and their relationship to themselves and other objects in the environment (Warren, 2006). One product of visual attention is visual scanning. Visual scanning consists of the ocular strategies used to obtain visual input; it is necessary to properly process and interpret visual information (Hutman, 2013).
The Brain Injury Visual Assessment Battery for Adults (BIVABA; Warren, 2006) consists of multiple visual assessments that measure acuity, contrast sensitivity, oculomotor function, visual field, and visual search. The ScanCourse is a component of the BIVABA and assesses a person’s visual attention and scanning capacity while ambulating (Warren, 2006). The ScanCourse is administered by placing numbered cards on both sides of a hallway at varied heights and asking the test takers to verbalize or point out which cards they see as they mobilize down the hallway.
The ScanCourse has been recommended to assess visual scanning abilities in driver rehabilitation, because it is easy to access, administer, and score and combines visual search with mobility, a feature not seen in paper-based assessments (Warren, 2006). This combination of functional mobility and visual scanning classifies the ScanCourse as a dual-task assessment. Dual tasking is important in many aspects of daily life, including observing surroundings while crossing the street, conversing while walking, and driving (Yang et al., 2016).
Research on the BIVABA is limited, especially for the ScanCourse component. On the basis of our review of the literature, which included CINAHL and MEDLINE database searches as well as consultation with the developer of the ScanCourse, we were unable to identify any available studies that examined the measurement properties of the ScanCourse. Therefore, the objective of this study was to determine the measurement properties of the ScanCourse, including interrater reliability among occupational therapists using the ScanCourse with neurological clients, test–retest reliability of the ScanCourse in people with neurological impairments, and construct validity by comparing scores on related measures. We hypothesized that with the provision of standardized administration guidelines, the ScanCourse would have an intraclass correlation coefficient (ICC) greater than .8 for interrater and test–retest reliability and would be moderately correlated with measures of visual scanning, the Bells Test, and the Trail Making Test Parts A and B (between .3 and .6). Moderate correlations were expected because all three assessments measure visual attention and scanning; however, the ScanCourse differs from the other two because it is a divided attention task that involves movements while scanning, and it is not paper based. Dual tasks involving divided attention are typically more difficult as a result of task interference (Alavash et al., 2015).
Method
Study Design
This measurement study involved data collection by multiple raters. The goal was for each study participant to be evaluated at two time points and on one of those occasions by two therapists. Ethical approval was obtained from the local university and institutional ethics boards at each site before beginning the study. Researchers used a convenience sample from three hospital-based rehabilitation centers in the same geographic region. Data were collected over approximately 4 mo from February 2017 through May 2017.
Participants and Recruitment
Inpatients and outpatients with potential problems with visual scanning, as identified by their therapists, were considered for the study. Inclusion criteria included the ability to independently mobilize, by walking or wheelchair; either verbalize or point to their observations; and make consent clear. Individuals who were unable to verbalize or otherwise communicate their observations, or who were unable to provide consent, were excluded from the study.
Forty-one participants were recruited from the active caseload of occupational therapists working in neurorehabilitation at inpatient and outpatient facilities of three hospitals. Participants were recruited by their treating therapist, who planned to administer the tool as part of their normal practice. Each participant was provided with a letter of initial contact and a consent form. Participants were informed that there was no obligation for them to participate and that they could withdraw from the study at any time. Informed consent and demographic information were obtained before administration of the test.
Procedure
Occupational therapists administered the ScanCourse using a standardized administration protocol that was developed by the research team. The protocol included a script to read to the participant before completing the ScanCourse, reminders to not prompt the participant with visual scanning suggestions, and instructions on how to complete and score the ScanCourse. Occupational therapists were also provided with a standardized recording form, which contained the script, order of numbers in a table format for easy marking, and a checklist of visual scanning techniques potentially used by participants. The therapists’ experience levels ranged from <1 yr to >10 yr. Therapists assessed their own clients; when acting as a second rater, they assessed other clients. One to two therapists were involved at a given time.
Researchers set up a partially standardized ScanCourse at each site. Numbers 1–20 were printed at 1.25 in. high using Arial font on a 3.25-in. × 4.25-in. card, which was laminated. The numbers were placed with the middle of the card at wall heights of 71 in., 42 in., and 20 in. Fifty percent of the numbers were on the left side of the hallway, and 50% were on the right side of the hallway. Hallway width and length were not standardized because of environmental constraints and therefore varied from 7 ft to 8 ft and 72 ft to 100 ft, respectively.
The Bells Test (Gauthier et al., 1989) and Trail Making Test Parts A and B (Trails A and Trails B; Marvin, 2012) were used to assess the level of convergent construct validity. The Bells Test is a cancellation test that measures visual neglect by asking individuals to cross out 35 bells embedded within 280 distractors on an 11-in. × 8.5-in. page (Zeltzer & Menon, 2011). The Bells Test demonstrates extreme group validity in that there is a statistically significant difference between scores of individuals with and without neglect (Gauthier et al., 1989; Vanier et al., 1990).
The Trails A and Trails B (Marvin, 2012) involve a variety of skills, including letter and number recognition, visual scanning, and motor function and primarily measure individual differences in speed and fluid cognitive abilities (Salthouse, 2011). Trails A involves connecting randomly distributed numbers in numerical sequence. Trails B involves connecting randomly distributed letters and numbers, alternating between a letter and a number in ascending order (i.e., 1, A, 2). The Trail Making Test demonstrates test–retest reliability of r = .78 for Trails A and r = .67 for Trails B as well as predictive validity for driving test outcomes for Trails B (Goldstein & Watson, 1989; Marshall et al., 2007)
Data Collection
To measure interrater reliability, during one of the two ScanCourse administrations, two raters observed participants and scored them on the standardized recording form provided. The second rater was another occupational therapist or an occupational therapy student who had previous experience administering the ScanCourse. Occupational therapy students were on first- or second-year clinical placements and were trained by their supervising therapist before administering the assessment.
To measure test–retest reliability, level of agreement, and standard error of measurement (SEM), the same occupational therapist administered the ScanCourse twice to the same participant within a 3- to 14-day period. Occupational therapists were instructed not to give any training, education, or feedback between the sessions. These measures were put in place to minimize a learning effect for the participants.
To examine the level of convergent construct validity between the ScanCourse and the Bells Test or Trails A and B, occupational therapists administered these assessments at baseline. These tests were chosen because they are used to determine potential difficulty with visual scanning tasks, such as driving (Vrkljan et al., 2011).
Data Analysis
Statistical analysis was completed using IBM SPSS Statistics (Version 23.0; IBM Corp., Armonk, NY). To determine the level of interrater reliability among occupational therapists using the ScanCourse with neurological clients and test–retest reliability among neurological clients, ICCs (1,1) were calculated. We chose ICC (1,1) (i.e., one-way random effects model) because it is used in cases in which participants are not assessed by the same set of raters (Koo & Lee, 2016). In our study, participants were evaluated at multiple sites at which different therapists were used. A Bland–Altman plot was created to demonstrate the level of agreement between the difference in scores for ScanCourse Trial 2 and Trial 1 and the mean scores between these two trials (Figure 1). To evaluate absolute reliability (i.e., the degree of variability from trial to trial; Atkinson & Nevill, 1998), the SEM was calculated using the formula
Data were skewed because most participants scored high on the ScanCourse and other measures. Because data were found to be nonnormally distributed, Spearman’s correlations were used instead of Pearson’s correlations for construct validity testing, which compared ScanCourse Trial 1 scores with Bells Test, Trails A, and Trails B scores (Mukaka, 2012).
Mean ScanCourse scores between Trial 1 and 2.
Results
Demographics
The demographic characteristics of the study population are shown in Table 1. The majority of participants were male and spoke English as a first language, and the most common diagnoses were stroke, brain injury, and spinal cord injury. The mean scores for the measures used for this assessment are reported in Table 2. The mean score for the ScanCourse on Trial 2 was slightly higher than the mean score for Trial 1. Note that the number of participants vary for each category because not all participants completed all components of the study as a result of time constraints or limitations of service delivery, for example, outpatients who were seen less frequently than the 2-wk time period required for readministration.
Participant Demographics
Mean Scores of Study Measures
Reliability
For the participants assessed for interrater reliability (N = 36), the ICC was .998 (95% confidence interval [CI] [.996–.999]). The ICC for test–retest reliability (N = 28) was .912 (95% CI [.811–.959]). The SEM was .503.
Level of Agreement
Figure 1 displays the Bland–Altman plot. Spearman’s correlation coefficient was calculated to determine the level of correlation of the ScanCourse to related measures. The ScanCourse was found to be significantly correlated to Trails A (N = 35; r s = –.436; p = .009) and Trails B (N = 36; r s = –.364; p = .029). It was not significantly correlated to the Bells Test (N = 41; r s = .140; p = .383).
Discussion
This study determined the interrater reliability, test–retest reliability, and construct validity of the ScanCourse compared with the Bells Test and Trails A and B. There was good support for our first two hypotheses, that the ScanCourse would have an ICC greater than .8 for interrater and test–retest reliability. These values are sufficient for individual comparisons (>.90; Aaronson et al., 2002). The Bland–Altman plot shows a slight increase over time, which could be due to a learning effect of the assessment or to improvement in the participants’ neurological impairment over time. The SEM indicates that a 1-point change in score would not be considered an error (i.e., a 1-point difference is considered to be statistically significant). However, further research is needed to determine which values represent clinically significant changes.
We found support for only a portion of our final hypothesis: The ScanCourse was significantly correlated to the Trails A and B, but not to the Bells Test. These results suggest that the Trail Making Test and the ScanCourse measure different skills than the Bells Test. For example, the Bells Test is a static test of near space and visual neglect, whereas the Trail Making Test and the ScanCourse involve following a path of targets. The ScanCourse and Trail Making Test also both involve recognition of numbers versus the symbols used in the Bells Test.
The correlation between Trails B and the ScanCourse provides support for the potential use of the ScanCourse in driver rehabilitation. Multiple studies support the use of the Trails B in predicting fitness to drive (Classen et al., 2013; Devos et al., 2011; Gibbons et al., 2017; Roy & Molnar, 2013; Vrkljan et al., 2011). Previous studies have found that participants who completed the Trail Making Test B in more than 90 s required further on-road assessment and that the Trail Making Test B was the best predictor of driving ability compared with the Montreal Cognitive Assessment, Motor-Free Visual Perceptual Test, and clock drawing test (Devos et al., 2011; Gibbons et al., 2017). Therefore, an area for future research would be to examine the use of the ScanCourse in predicting fitness to drive, especially because it involves dual tasking, which is critical for driving (Baldwin & Schieber, 1995).
The study had two main limitations. We were unable to standardize hallway width, length, color, or amount of clutter because of constraints of the facilities used. Data were collected at the site level; therefore, the test–retest reliability or interrater reliability for each clinician or pair cannot be calculated.
Implications for Occupational Therapy Practice
This study has the following implications for occupational therapy practice:
When administered using a standardized protocol, the ScanCourse demonstrates high interrater and test–retest reliability.
Because the ScanCourse was not significantly correlated to the Bells Test, it should not be used to identify near space visual neglect.
Because the ScanCourse is significantly correlated to the Trails A and B, which are currently used to predict driving capability (Marshall et al., 2007), the ScanCourse warrants additional study to determine how well it predicts driving capability.
Conclusion
The results of this study provide support for the use of the ScanCourse as a visual scanning assessment in clients with neurological impairment; it was found to have excellent interrater and test–retest reliability and to be significantly correlated to Trails A and B. The correlation between scores on the ScanCourse and Trail Making Test B may indicate that the ScanCourse may be appropriate for use in a driver rehabilitation setting.
Footnotes
Acknowledgments
The authors thank the occupational therapists who contributed to the data collection. W. Ben Mortenson was supported by a New Investigator Award from the Canadian Institutes of Health Research. This is an unfunded study completed in fulfillment of a Master of Occupational Therapy degree.