Inter-rater and test-retest reliability of computerized clinical vestibular tools

2.1 Ethical considerations

The study was approved by the University of Al-abama at Birmingham’s Institutional Review Board.

2.2 Participants and testers

A sample of convenience (n = 60) from the 1st and 2nd year University of Alabama at Birmingham (UAB) Doctor of Physical Therapy (DPT) students participated in this study. Participants committed to two 1-hour long testing sessions no more than one week apart. To be included, participants had to be 18–35 years of age, report normal vision with or without the use of corrective lenses, have no current or previous neurological diagnosis and no self-report of hearing loss, headache, migraine, neck pain, or dizziness. Participants were asked to refrain from taking medications known to affect the vestibular system (e.g. meclizine) as well to refrain from drinking alcohol 24 hours prior to testing. If a participant had a headache or dizziness at the time of either session, they were not allowed to test that day. If during testing, the participant developed a headache or dizziness rated 3/10 or greater, testing stopped immediately. If the headache and/or dizziness was rated 2/10 or lower the participant was instructed to rest to allow symptoms to return to 0/10 before proceeding.

All testers were DPT second year students who trained using methodology recommended by the manufacturer for data collection until they could confidently perform all tests. Three testers collected data for the full study sample. To assess intertester reliability, the first 30 participants were tested by a different examiner on each tool at Session 1 and Session 2 (i.e., Examiner 1 was the tester for Tool A at Session 1 for Participant 1, but Examiner 2 was the tester for Tool A at Session 2). Each examiner was the tester for each tool a total of 20 times, and examiners were randomized for each participant/session (i.e., Examiner 2 was not always the tester for Tool A at Session 2 as in the example above). The second 30 participants were tested by the same tester on each tool (i.e. Examiner 1 was the tester for Tool A at Session 1 and Session 2 for Participant 31) to assess each tools’ test-retest reliability. Again, each examiner was the tester for each tool a total of 20 times.

Participants were tested in a quiet environment. To control for the effect of test order, the four tests were organized into eight predetermined orders. On the first test day, participants drew a number (1–8) to determine order of tests. Once a number had been drawn eight times, that order was removed. This ran-domization ensured that, if the battery was fatig-uing, participants were not always fatigued on the same tools. The four clinical tools included: 1) Ber-tec Visual Advantage, 2) Synapsys vHIT, 3) EyeSeeCam® vHIT, and 4) Smartphone Accelerometer-Assisted SVV Bucket Test.

2.3 Computerized dynamic visual acuity

The Bertec Visual Advantage computerized dy-namic visual acuity system (BVA-cDVA) was used to assess static visual acuity (SVA), visual processing speed, and either the DVAT or GST in yaw. Participants were not asked to complete both tasks to reduce fatigue effects that may influence performance on the DVAT/GST [13]. At their first session, participants randomly picked a 1 or 2 to determine whether they would complete the DVAT or GST. Once the DVAT or GST reached 15 participants for each set of 30 participants, the remaining participants up to n = 30 completed the other task so that there was n = 15 participants for both subtests for inter-rater and test-retest reliability calculations. Tests for SVA and visual processing speed were determined before the DVAT or GST. Both SVA and DVAT are presented as LogMAR units (the log base-10 of the smallest minute of visual angle the participant can correctly identify) over Snellen scores due to their continuous nature.

The BVA-cDVA subtests require participants to identify the direction of a tumbling E optotype. For visual acuity tests, the size of the optotype changes based on participant response. For visual processing speed and the GST, the optotype remains the same size for each trial but is presented at different speeds (visual processing speed) or requires a certain yaw head velocity to be presented (GST). The BVA-cDVA utilizes an adaptive, threshold-search algorithm based on the Hughson-Westlake method to determine which optotype size, presentation time, or head velocity to use for subsequent trials [9]. After each trial, the software determines if the number of completed trials is enough to reliably determine the threshold with approximately equal numbers of correct and incorrect responses around that value. If not, trials continue. The BVA-cDVA has the following ceiling values: SVA and DVAT: –0.3 LogMAR, GST head velocity: 150 °/s, and visual processing speed: 30 ms.

The DVAT was conducted by having the tester move the participant’s head in the yaw plane sinusoidally at a constant velocity between 90–120°/s as measured by a head mounted accelerometer. When the participant’s head movement was within this ra-nge, the optotype presented during a leftward or right-ward head movement, depending on the current test direction, to test the participant’s ability to accurately identify the optotype during movement towards the testing side. The variables of interest were the dyn-amic acuity (LogMAR units) and the number of lines lost (equivalent to 0.1 LogMAR each) between SVA and DVAT conditions for both rightward and leftward rotations. The leftward DVAT was completed first for the first 30 participants and the rightward DVAT was completed first for the second 30 participants.

The GST was conducted by having the tester move the participant’s head at faster or slower speeds depending on their performance on their previous trial, with a ceiling velocity of 150 °/sec, to determine the upper limit of the participant’s VOR’s ability to stabilize their vision. The variable of interest was maximum velocity of head movement (°/sec) rightward and leftward with accurate identification of the presented optotype. The left side was tested first for the first 30 participants and the right side was tested first for the second 30 participants.

2.4 Video head impulse testing

EyeSeeCam® (E-vHIT) and Synapsys (S-vHIT) were used to assess the gain and asymmetry of the VOR in all semicircular canals using video head impulse test (10 trials per canal) using the instructions from each manufacturer. The E-vHIT uses a tight-fitting goggle with a camera placed over the left or right eye. For this study, the camera was kept over the left eye. Calibration was completed for both the eye-tracker and accelerometer. The E-vHIT calculates 6 total canal gains: the leftward and rightward lateral canal gains at 60 ms and 100 ms after head impulse and the left/right posterior/anterior canal gains at 100 ms after head impulse. The gain of the VOR in each canal was the variable of interest.

The S-vHIT uses a single remote camera connected to a base placed 1 meter in front of the participant and focused on the participant’s eyes. No calibration is necessary. The S-vHIT calculates the VOR gain for each canal and asymmetry.

For both tools, the following thresholds were used to determine whether a trial was valid or not: for lateral canals, acceleration = 1,500–4,000 °/s², velocity = 120–310 °/s; for posterior and anterior canals, acceleration = 1,000–4,000 °/s², velocity = 76–219 °/s. The peak velocity velocity needed to occur within 80 ms of impulse onset. While covert saccades were recorded by the tools (> = 5,000 °/s eye velocity after the peak velocity), these were not analyzed for this study due to a lack of pathology in our sample leading to very low numbers of corrective saccades. It is important to note that the E-vHIT validates trials through both the head-mounted accelerometer to calculate impulse acceleration and the eye-tracking camera to calculate eye velocity whereas the S-vHIT relies solely on its eye-tracking camera to make all calculations.

2.5 Smartphone bucket test

The smartphone-assisted bucket test (SA-SVV) used a commercially available smartphone application developed by one author (Graham Cochrane, who was involved in the training of the tool but not the data collection). This test involved placing a bucket over the participant’s face, who was sitting with their head straight to remove potential body-tilt bias, with a computer-generated printed line on the inner base of the bucket [10]. A smartphone was mounted onto the outer base that records the angle of the phone in the phone’s yaw plane. The creator of the bucket attempted to align the smartphone to be parallel with the inner line; a perfect alignment may not have been made based on the average deviations presented. The participant was instructed to close their eyes and the examiner holding the bucket turned the line to a preset angle identified randomly from a list of 12, 15, 20, –12, –15, and –20 degrees, to ensure variable, random, preset angles [17]. The participant then opened their eyes and the bucket was slowly turned by the examiner to true vertical at a speed < 8°/s (measured by the phone which prompted the tester to slow down if exceeded). When the participant believed the line was perfectly vertical, they pressed a Bluetooth button that recorded the angle of the phone at that moment. This was repeated for 12 trials. The variable of interest for this test was average response (degrees from 0) and variance in response.

2.6 Examiner training

Before official testing began, the examiners were trained by video chat and in person sessions by tool manufacturers, then given as much time as needed to become confident in the use of each tool. “Confidently” was defined through two methods: 1) Time to where the examiner could set up / break down the tool, set up a patient and correct parameters, and record results without needing input from other examiners; 2) Ability to complete a testing session on the tool while making 0–2 errors, defined as improper impulses per canal on E-vHIT and S-vHIT and or too fast or too slow trials on the DVAT and the GST. The SA-SVV does not have easily identifiable errors on the examiner’s part besides moving the bucket too fast which is monitored and can be fixed in the moment.

2.7 Statistics

All statistics were computed in SPSS v26. De-scriptive statistics consist of means and standard deviations for normal distributions, and medians and interquartile ranges for skewed data (determined by visual inspection). Two-way mixed intra-class correlations (ICCs) were computed for both sets of 30 participants representing inter-rater absolute agreement and test-retest absolute agreement [11].

ICCs were interpreted as follows: 0.00–0.49 = Poor, 0.50–0.75 = Moderate, 0.76–0.90 = Good, 0.91–1.00 = Excellent [19].

To improve potential clinical utility, ICCs were converted to smallest real differences (SRDs), also known as minimal detectable change scores. These SRD numbers help better represent the expected test-retest range of a clinical testing instrument [1]. Determining this range allows clinicians to better identify whether a change in performance on a measure is indicative of true change in the patient or a result of the error of a tool/examiner. The equation used to calculate the SRDs for this study was:[3]

SRD =  (Test 1& 2 Combined Standard Deviation) ^* sqrt(1-ICC) ^* 1.96 ^* sqrt(2)

As the SRD is determined by the standard deviation of the tool, it is only accurate for measures that are normally distributed. As such, SRDs were only calculated for non-skewed data.

A simple Student’s t-test was run to determine if there was a systematic difference between the inter-rater and test-retest groups in terms of their change in performance (session 1 score –session 2 score) from session 1 to session 2 for all normally distributed tool measures.

3.1 Training

The testing team (2nd year DPT students) reported the following hours of training (per tester) needed to confidently perform each test before data collection began: BVA-cDVA: 3 hours, E-vHIT: 3 hours, S-vHIT:>20 hours, SA-SVV: 1 hour. On the S-vHIT system, it was often difficult to obtain 10 accepted impulses per canal without making > 2 errors even after this training period.

3.2 Participant demographics

Participants were on average 24.1±1.9 years of age; n = 42 (70%) were female and n = 18 (30%) were male. n = 48 (80%) identified as white, 8 (13.3%) as black, 2 (3.3%) as Asian, 1 (1.7%) as Native American / Pacific Islander, and 1 (1.7%) as more than one race.

3.3 Testing

All 60 participants finished the first day protocol without an increase in headache or dizziness during the E-vHIT task and the bucket task. One individual reported an increase in dizziness (from 0–2) during the S-vHIT task on both Day 1 and Day 2. Four individuals reported an increase in dizziness during the BVA-cDVA tasks (0-1 or 0-2); two of these four reported increased dizziness on both days while the other two only reported dizziness on one of the days. All participants completed their second testing session within 1–7 days from the first session. One individual did not finish the second day protocol, reporting dizziness at 3/10 during the BVA-cDVA, their first tool of the day, which was unexpected based on the tolerance of other participants. Therefore, the test-retest reliability analyses were completed with 29 individuals instead of 30. Intertester reliability statistics are reported in Table 1 and test-retest reliability statistics are reported in Table 2 for all variables. There were no significant differences on change in any tool measure between the two groups (p-value = 0.13–0.94).

The SVA and DVAT LogMAR data showed good to excellent inter-rater reliability and moderate to good test-retest reliability. Lines Lost for the DVAT had poor inter-rater reliability and test-retest reliability. The GST had poor to moderate inter-rater reliability and poor test-retest reliability.

3.5 Video head impulse testing

The E-vHIT VOR gains of the lateral semicircular canals at 60 ms and 100 ms had poor inter-rater reliability and moderate to good test-retest reliability. Posterior and anterior canal VOR gains had poor to moderate inter-rater reliability and moderate to good test-retest reliability. Left anterior and right posterior (LARP) gains were 30% larger on average than right anterior and left posterior (RALP).

The S-vHIT VOR gains of the lateral semicircular canals had poor to moderate inter-rater reliability and good to excellent test-retest reliability. Posterior and anterior canals had moderate inter-rater reliability and moderate test-retest reliability. Asymmetries calculated by S-vHIT had poor inter-rater reliability and poor to moderate test-retest reliability.

3.6 Smartphone bucket test

Average degrees from vertical and variance had moderate inter-rater and test-retest reliability. The ICCs for variance were also moderate but SRDs were not calculated due to the large right-skew of the data.

4.1 Computerized dynamic visual acuity

Overall, the measures from the BVA-cDVA with the strongest reliability were the SVA and DVAT LogMAR data. While clinicians typically utilize the resulting comparison of these two measures, Lines Lost, this measure did not show convincing ICCs. Despite low ICCs, the SRD was still 1.5–2.0 lines which does not appear unreasonable as goal endpoint for a treatment or therapy. It should be noted that the participants in this study did not have vestibular deficits and the mean Lines Lost was only one line. Therefore, the low inter-subject variability likely affected the ICC values [13]. Similarly, although the GST demonstrated low ICCs, the SRDs may guide clinicians who are using the GST to determine response to gaze stabilization training.

Our good to excellent reliability results differ from other studies who report poor reliability for the computerized DVAT instead [19]. The difference in our findings may be attributed to the fact that we required the examiner to assist the participant with turning the head, rather than having the participant turn their head independently. During the DVAT or GST, the optotype will not appear until the head velocity is within a certain range. Participants, especially patients, often have difficulty obtaining the correct velocity which makes the test last longer potentially resulting in fatigue. Comparing inter-rater to test-retest reliability suggests that inter-rater testing resulted in slightly higher ICCs but nearly identical SRDs.

4.2 Video head impulse testing

The E-vHIT appears to be the device most affected by change in examiner from one administration of the test to the next, with significantly different reliability scores when comparing inter-rater reliability and test-retest reliability. This indicates that either the E-vHIT is sensitive to the exact user or that it is the tool that requires the most training to produce similar results. Since the test-retest analysis was done on the second group of 30 participants, the testers were more skilled on the tools by that point. This does not coincide with the testers’ reported time to perceived confidence vs the S-vHIT, however. On the E-vHIT system, VOR gains were close to 1.00 as expected, except for the gains of the LARP canals, which were 30% larger than the other canals. Through discussion with the E-vHIT vendor experts, it is likely these higher gains were due to the accelerometer system and camera being fastened to the left side of the goggles. This pattern was not fully appreciated until multiple participants had been tested and the research team decided to stay consistent with our original testing paradigm. Clinicians should be mindful of this and collect their own normative data using standardized methodology. It is possible there may be an exact correction factor to account for this issue. Although the VOR gains were higher in LARP compared to RALP, reliability coefficients were similar. However, this was not explored further.

The ICCs for the S-vHIT system were similar for inter-rater and test-retest across the six canals. While the asymmetry ICCs were very low, it is important to note that the sample tested were all healthy individuals with no history of vestibular disease. Therefore, it is not surprising that few individuals had any pronounced asymmetry. As with the BVA-cDVA tasks, ICCs can be heavily influenced by low inter-subject variability; the low ICCs here should be attributed to a statistical weakness of the correlation and not an indication that asymmetries are unreliable [13]. Studies investigating asymmetries in a population with more variability (e.g. those with differing levels of unilateral vestibular hypofunction) will be better equipped to determine the reliability of asymmetry measures. It is important to emphasize that the S-vHIT was the most difficult for the testers to perform confidently and that, even after a 20 + hour training period and 120 examinations still felt as though they were learning nuances to the system.

4.3 Smartphone accelerometer assisted SA-SVV

Mean degree angle and variance in angle response had similar inter-rater and test-retest reliability ICCs suggesting that changing examiners between administrations may not affect scores. The SA-SVV helps this by providing the preset angle for each trial, monitoring the speed of the bucket rotation, and making sure the bucket and phone are parallel to the floor. Average deviations from the mean were tilted slightly from vertical (–0.36 and –0.21), likely indicating that our phone was not perfectly aligned with the visual line. While this bias did not affect the reliability calculations here since the same biased tool was used for both administrations, this is important to consider when using this system clinically. Assuming a population average of 0.0 degrees from vertical would allow a clinician to determine their bucket’s bias if there is any. This potential for error is not unique to the SA-SVV and needs to be considered in studies of and use of the traditional bucket test. The results suggest the ICC of the smartphone app is near, but less than, the Pearson correlation coefficients recorded for the traditional bucket test by Zwergal et al., greater than that of the bucket test measured by Michelson, et al., and comparable to the virtual SVV system [12, 15]. As mentioned previously, ICC is a more robust measure of agreement than Pearson r, so a lower ICC here does not necessarily indicate the SA-SVV was less reliable than the traditional method [2]. As this particular app is used identically to the traditional angle finder / goniometer bucket test and requires only a smartphone instead of a computer and software we believe it is more clinically feasible, cheaper, and would require less training to shift to from traditional angle finder / goniometer bucket test when compared to other available virtual SVV tools.

4.4 Feasibility

The testing battery of BVA-cDVA, E-vHIT and S-vHIT, and SA-SVV was well tolerated by participants and reasonable to use for the vestibular evaluation in a single clinic visit from a fatigue point of view. However, it is possible that length of the visit or symptoms exacerbated by patients with actual vestibular impairments may make using all tools unfeasible. In addition, the cost of all tools may be prohibitive. Only one individual was unable to complete both sessions due to their reported dizziness severity. The testers required 1 hour of training to confidently use the SA-SVV app, 3 hours to confidently use the BVA-cDVA and E-vHIT, and > 20 hours to confidently use the S-vHIT. It is possible that clinicians well trained in administration of vestibular testing may not require the same level of training that these students required.