Evaluation of a Psychomotor Vigilance Task for Touch Screen Devices

Introduction

The psychomotor vigilance task (PVT) is considered the gold standard for detecting fatigue in laboratory (Belenky et al., 2003; Doran, Van Dongen, & Dinges, 2001; Van Dongen, Maislin, Mullington, & Dinges, 2003) and field (Åkerstedt & Wright, 2009; Russo et al., 2004, Dijk et al., 2001) studies due to its documented sensitivity in detecting sleepiness (Basner & Dinges, 2011; Howard et al., 2003; Rogers & Kloss, 2004). The original form of the PVT (PVT-192) is a handheld test with a 3-mm display that delivers a visual stimulus in the form of a reaction-time counter, presented with an interstimulus interval varying between 2 and 10 s (Dinges & Kribbs, 1991). Participants are instructed to rest their thumbs on two physical buttons and depress one button with their dominant thumb as quickly as possible as soon as the stimulus appears. If participants press the button too soon, the phrase “false start” appears on the display. The total task duration for each PVT trial is 10 min. The outcomes from the test that have been shown to be most sensitive to fatigue are mean reaction time (RT), number of performance lapses (trials in which the participant failed to generate a response within 500 ms), inverse RT, median RT, 10% fastest RT, and 10% slowest RT (Dinges & Kribbs, 1991). The PVT has no significant learning curve (Dinges & Kribbs, 1991), making this task ideal for evaluating fatigue arising from sleepiness and circadian misalignment in field settings.

With the rapid development of new mobile technologies, several versions of the PVT have been developed for handheld devices. The portability of mobile devices provides a desirable medium for assessment of fatigue in remote settings; however, there are many methodological considerations related to administration of a PVT on a handheld touch screen device that remain unresolved. The most widely used handheld version of the PVT is a 5-min test that was developed for use on Palm OS and validated against the PVT-192 (Thorne et al., 2005); however, that version of the PVT utilized a physical button to register RT. In contrast, implementation of the PVT on touch screen devices requires participants to hover their fingers over the screen. Recent evidence suggests that under carefully controlled laboratory conditions, touch screen versions of the PVT yield changes in RT consistent with those recorded by computer versions of the test (Honn, Riedy, & Grant, 2015). In addition, the RT registered following a finger deflection on a touch screen is similar to that obtained with a physical button press (Kay et al., 2013). It is not clear, however, whether the orientation that one uses to hold a device affects RT. It is also not clear whether the finger that a participant uses to respond to a test stimulus alters the recorded RT.

Given the ubiquity of handheld devices, creating a reliable, accurate PVT software application and standardized methodology for taking the test is imperative, although not necessarily easy given that the precise timing is crucial to determine accurate levels of alertness. Limitations in the current hardware specifications of most mobile devices cause concern for accurately measuring response time with touch. A persistent problem with touch screen devices is latency, the time between user action (touching the screen) and the system’s response (Jota, Ng, Dietz, & Wigdor, 2013; Kaaresoja & Brewster, 2010; Steed, 2008). Using high-speed cameras, Ng and colleagues evaluated basic navigation tasks on several touch screen platforms and found that latency between the touch and documented response time ranged between 50 and 200 ms (Ng, Lepinski, Wigdor, Sanders, & Dietz, 2012). Similarly, Agawi developed a “touchscope” that measured how quickly applications could respond to touch events and found that even top-of-the-line tablets have latencies that reach 75 ms (Takahashi, 2013). Given the importance of accuracy in estimating fatigue using the PVT, the latency of a touch screen device needs to be evaluated in order to ensure response times are accurate and reliable.

Given the limited information available regarding how the orientation of a touch screen device may affect RT, we aimed to compare RTs of participants with the device in portrait and landscape positions, using the index finger or the thumb to respond to the stimuli. Second, we aimed to identify the device latency between the actual time of a touch and the RT recorded by a commonly used touch screen device in order to adjust the PVT trials before analyses.

Method

Materials and Procedure

The study was approved by the NASA Institutional Review Board (HRII-14-17). Participants signed a consent form prior to participating in the study. Following consent, they completed a demographic questionnaire and Epworth Sleepiness Scale (ESS; Johns, 1991). They next received a demonstration showing them how to hold the device for each orientation and completed a short practice version (eight trials) of the NASA PVT. Participants were asked to use their dominant hand to respond to the stimuli throughout testing.

The NASA PVT was developed at NASA Ames Research Center to be used on a touch screen personal digital assistant (PDA) and was implemented on a fifth-generation, 32-GB Apple iPod (Apple Inc., Cupertino, CA) running Operating System 5.1.1. The NASA PVT was a 5-min visual PVT. As with the traditional PVT, the interstimulus interval varied from 2 to 10 s, with each PVT consisting of multiple trials (80–100 stimulus-response events). The delays for individual trials were chosen by drawing a value from an exponential distribution with a mean of 750 ms, which was then added to the minimum time of 2,000 ms. This procedure means that half of the trials had delays that were less than 2.75 s, but a small number had delays that were much longer, as long as 10 s. The exponential distribution was chosen because the distribution of the foreperiods resulted in a constant hazard function. In other words, the probability that the stimulus will appear in the next interval, given that it has not appeared yet, was constant. Drawing the foreperiods from a uniform distribution, on the other hand, had the property that the appearance of the stimulus became more likely the more time passed without the appearance of the stimulus.

The stimulus was represented by a white square that appeared on the black screen and was replaced by the RT of that event after the participant responded to the stimulus. A participant response was recorded when the participant touched any part of the screen. The RT was the time between the appearance of the stimulus and the response of the participant. After a response, the RT was displayed in white. This display remained on until the next stimulus. If a tap was made before the stimulus came up, the tap was noted as an anticipatory response, but the display did not change until the stimulus timer ran out, at which time the amount of anticipation was displayed in red. The RTs were stored in the iPod and uploaded on a PC, where the data were processed into summary statistics. The outcome metrics that we used for comparing different orientations of the device included mean 1/RT, number of lapses (RT > 500 ms), median RT, fastest 10% RT, and slowest 10% 1/RT. A PVT response was considered valid if RT was >100 ms. Consistent with previous studies, responses with an RT <100 ms were counted as false starts (Basner & Dinges, 2011). Participants were asked to take the 5-min PVT three times in the same order, with a break of 1 min between test sessions.

On the first session, the PDA device was placed on the table in the portrait orientation, and the participants were asked to tap the screen under the stimulus using the index finger from their dominant hand. During the second session, they were asked to hold the PDA device in the portrait orientation with the nondominant hand and use the index finger from the dominant hand to respond to the stimuli. During the third session, participants held the PVT in landscape position using both hands and responded to the stimuli using their thumb from the dominant hand. A white bar was displayed at the top of the device to orient the participant for positioning the device. The three experimental techniques are represented in Figure 1.

OPEN IN VIEWER

Figure 1.

Illustration of the three psychomotor vigilance task techniques. (1) Device is placed on the table in portrait orientation. (2) The device is handheld in portrait orientation. (3) The device is handheld in landscape orientation.

In addition, the first session of PVT was recorded using a high-speed video camera (Point Grey Research FL3-U3-13Y3M) recording 500 frames per second. The testing device was oriented to lay flat on a table to allow a mirror to capture the screen touch below the touch surface. The camera captured the finger motion, the mirror image, and the display of the stimulus. The camera was connected to a computer controlled by the experimenter, which did not provide any visual feedback to the participant during the execution of the trials. The participant was seated and the chair was adjusted so that the participant could comfortably operate the PDA device from the table. The participants used the PDA and were asked to respond to the stimuli using their index finger and to tap on the lower part of the screen, under the stimulus. This test lasted 5 min. The device Wi-Fi was turned off during all tests in order to reduce the introduction of variability arising from system processing. Figure 2 shows the picture of the experimental apparatus.

OPEN IN VIEWER

Figure 2.

Experimental apparatus: (1) high-speed video camera, (2) frame counter, (3) mirror, (4) touch device, (5) infrared illuminator.

Two independent raters recorded the video frame numbers when the stimulus appeared and when the touch occurred. The time elapsed from when the stimulus appeared on the screen and when the touch occurred was subtracted from the RT recorded by the device. The remaining time (in milliseconds) from the occurrence of the touch and the RT recorded by the device is the device latency.

Results

Thirteen participants (nine men, four women) completed the study. Their ages ranged from 24 to 55 years (M = 33.54, SD = 8.77), and all of them had experience with touch screen devices. Two participants were left-handed. Participants reported that their average sleep duration ranged between 5 and 9 hr per day (M = 7.58, SD = 0.99), with regular bedtimes between 10:00 p.m. and 2:00 a.m. and wake-up times between 5:00 a.m. and 11:00 a.m. One participant traveled across three time zones within the past month of entering the study. The mean ESS score was within normal limits (M = 7.69, SD = 4.09) with a range between 2 and 17. Four participants had a score between 10 and 15, suggesting that they may be excessively sleepy, and one participant had a score above 15, suggesting that he should consider sleep counseling.

We evaluated the influence of the orientation of the device on RT and found significant differences based on how participants held the device. A paired-sample t test revealed a significant difference, t(12) = −3.16, p = .024, of the mean 1/RT between handheld portrait (M = 4.57, SD = 0.68) and handheld landscape PVT (M = 4.80, SD = 0.64). We also found a statistically significant difference, t(12) = 3.05, p = .03, of mean lapses between the handheld portrait (M = 2.38, SD = 1.71) and handheld landscape PVT (M = 1.08, SD = 1.32) and of median RT, t(12) = 2.97, p = .012, between the handheld portrait (M = 224.09, SD = 39.30) and handheld landscape (M = 211.50, SD = 30.74). In addition, we found statistically significant differences, t(12) = 2.45, p = .03; t(12) = 6.10, p < .001; of fastest 10% RT between the table portrait (M = 167.87, SD = 21.46) and handheld landscape (M = 157.39, SD = 20.51) and between handheld portrait (M = 167.87, SD = 20.31) and handheld landscape. We found no significant differences of slowest 10% 1/RT between any of the techniques. Detailed results of the mean changes and the 95% confidence interval between the three techniques are included in Figure 3.

OPEN IN VIEWER

Figure 3.

The changes and the 95% confidence interval of psychomotor vigilance task (PVT) metrics (mean 1/reaction time (RT), lapses, slowest 10% 1/RT, median RT, fastest 10% RT) for the comparisons of the three techniques: table portrait (TP), handheld portrait (HHP), and handheld landscape (HHL). The x-axis displays the range of the statistical changes of the PVT variables. The vertical line at zero was added for convenience, and the length of the x-axis was made symmetric around the reference line for easy comparison of the magnitude of coefficients. *p < .05. **p < .01. ***p < .001.

Mean 1/RT, mean lapses, median RT, fastest 10% RT, and slowest 10% 1/RT for each technique are illustrated in Figure 4.

OPEN IN VIEWER

Figure 4.

The means and standard deviations of the psychomotor vigilance task metrics (mean 1/reaction time (RT), mean lapses, median RT, fastest 10% RT, slowest 10% 1/RT). *p < .05. **p < .01. ***p < .001.

Overall, the landscape-oriented PVT had the higher mean 1/RT, had the smaller number of lapses, and showed the best performance on the fastest 10% RT. Furthermore, the landscape-oriented PVT produced roughly half as many lapses (n = 14) compared with table and handheld portrait-oriented PVTs (Figure 5).

OPEN IN VIEWER

Figure 5.

Total number of lapses for each individual participant by psychomotor vigilance task technique.

Eighty-five percent of the participants preferred the landscape-oriented PVT, reporting that it was the easiest to use. Two participants preferred the portrait-oriented PVT placed on the table. No participants preferred the handheld portrait orientation.

Device Latency Evaluation

The participants completed 1,241 unique touch events, which were scored by two raters who viewed the high-speed camera recording and documented the exact timing of the touch by logging the frame when the stimulus and the touch event occurred. An interrater reliability analysis was performed to determine consistency between raters. The overall agreement between raters was 100% for both stimulus present on frame and for the touch frame within one frame (2 ms). The threshold of 2 ms was chosen based on the raters’ agreement for each frame. Detailed examination of raters’ differences revealed that the average disagreement was one frame (2 ms), and the dissimilarity was due to noting either the clear presence of the stimulus or the weakened appearance of the stimulus on frame. The means of the responses from the two raters were used further to calculate the system latency. Through video analyses, we identified several occasions when the participants executed a double touch, that is, the device did not respond to the first tap and a second tap was necessary to record the response. These double touches represented 1% of the touch events and were removed from the analysis of device latency in order to maximize the accuracy of the responses. We determined the mean latency of the device to be 68.53 ms (SD = 18.09). The device latency was subtracted from each PVT trial before PVT analyses in order to maximize the accuracy of PVT results. Figure 6 shows the average results for each participant.

OPEN IN VIEWER

Figure 6.

Mean latency and 95% confidence interval of the means for each participant from the time of the actual touch on the screen to the response time recorded by the device. The reference line to the y-axis represents the mean latency for the whole group of participants.

We found some evidence of system binning in our data, which is evident in the standard deviation of the touch events. Figure 7 shows a scatter plot of touch time versus device RT for a single participant.

OPEN IN VIEWER

Figure 7.

Touch time versus device reaction time (in milliseconds).

The quantization of the device RTs in this figure is evidenced by the fact that the points cluster into vertically aligned groups. The touch times, on the other hand, show no evidence of quantization, as expected. The results in Figure 7 suggest that the variability in the time required to register the touch was much larger than the touch screen scan period of 16.7 ms. If there were a simply fixed software latency, we would expect to see data displayed below the main diagonal but tightly grouped about a line with perhaps a single scan period of scatter. Instead, for a given RT reported by the device, the touch times (true RT) were distributed over a range corresponding to four to five scan periods. This finding could be due to subject-dependent factors that interact with the touch screen sensor.

In order to illustrate the importance of obtaining accurate PVT results, we analyzed the number of lapses before and after the offset of 68.53 ms was applied to the PVT raw data. We found a 31% reduction of lapses for both table portrait and handheld portrait and a 48% reduction of lapses for landscape portrait (Figure 8).

OPEN IN VIEWER

Figure 8.

The sum of lapses before and after the offset was applied to the psychomotor vigilance task data for each technique.

Discussion

We found that RTs differed depending on the orientation of the device and the finger used to respond to the stimulus. We also found that there was substantial response latency between the actual time of an individual’s touch response and the time recorded by the touch screen device that we used. These findings have significant implications for researchers conducting studies using touch screen devices to administer the PVT.

On average, mean 1/RT and median RT were faster and lapses were fewer when participants used the PDA device in the landscape position with their thumb, compared with portrait position with the index finger, despite the fact that the landscape testing condition was ordered last in our testing sequence. The landscape condition occurred when it would be expected that RT would be highest due to fatigue related to increased time on task (Lim & Dinges, 2008). As the learning curve of the PVT is very small (Dinges & Kribbs, 1991), it is unlikely that the better performance that we observed in the handheld landscape orientation is due to learning. We found no differences on the slowest 10% 1/RT between the three techniques but significant differences on the fastest 10% of RT. The table portrait and handheld portrait had similar fastest 10% RT, and both were significantly lower compared with the handheld landscape PVT. This finding suggests that the major difference between the three techniques is shown by participants performing best on the landscape orientation while showing constant vigilance response slowing across the three techniques.

There are several possible explanations for why the PVT outcomes were better when participants oriented the device in the handheld landscape position. It is possible that some of the difference that we observed relates to the distance of the finger from the device in the different conditions. We instructed the participants to respond to the PVT stimulus using their dominant thumb during the handheld landscape testing condition. It is possible that the landscape orientation allows individuals to naturally hover their thumb closer to the screen compared with when the device is in portrait position. In addition, the portrait orientation might require a more complex movement with the index finger to produce the touch response (Zhang, Zatsiorsky, & Latash, 2006). The faster RTs when participants held the device in landscape position may also be related to the participants preferring that orientation. Kay and colleagues (2013) compared several touch screen input techniques to a physical button as used with traditional PVT. They found that the touch-down technique, in contrast to a swipe, finger lift, or finger tilt, was preferred by participants and had an execution time similar to using a physical button. Together with our results, these findings suggest that directing participants to hold the device in landscape position and to use the dominant thumb to touch down on the screen in response to a stimulus would most similarly mimic the conditions of a traditional PVT and would be acceptable for most individuals.

We found that there was 68.53-ms device latency for the touch screen device that we used. Although device latency has been identified by developers of gaming applications for touch screen devices, such information is not openly available, and we were unable to find published results of system latency for the fifth-generation iPod. It is critical to know the device latency before analyzing touch screen PVT results; of particular concern is the impact on the PVT lapses as an outcome. Lapses are defined as RTs that are greater than 500 ms, and they are the most indicative measure of sleepiness; therefore any application that does not account for system latency could provide inaccurate results. Therefore, there may be indications of higher fatigue levels than they actually are. Our results demonstrated how the calculation of lapses changed when we did not account for device latency.

There are many factors that could contribute to the length of the latency and variability, including the device, the operating system, or the application itself (Ng et al., 2012). It is not possible to determine which of these factors was responsible for the response latency and variability that we observed without access to proprietary information from the manufacturer of the device. Also using a different finger (index vs. thumb) to respond to the stimuli could induce some variability as well. Our results showed that refresh rate of the touch screen could induce variability in the responses. Although such sources of variability cannot be controlled, it would be prudent for researchers using touch screen devices to minimize all other potential sources of variability, such as turning off other applications and Wi-Fi, and using the same device, operating system, PVT application, and experimental procedure for all tests in a given study.

Although our goal was straightforward—to identify differences in the PVT response arising from the orientation of the device and to quantify the response latency—our study is not without limitation. Although we found that there were differences in RT based on the orientation of the device, we did not evaluate how such differences could alter results obtained during episodes of sleep deprivation or during the circadian nadir. However, in a carefully controlled laboratory experiment, Honn and colleagues (2015) found that a touch screen version of the PVT showed results similar to those obtained from a laptop version of the PVT. This finding suggests that when the method for collecting the PVT is standardized, the results are consistent with expected results. We evaluated only one type of touch screen device running one operating system. Others have found that the latency of other touch screen devices is different from the latency of the device that we tested (Takahashi, 2013). This finding suggests that researchers should use caution when administering a touch screen version of the PVT on multiple devices. Similarly, we evaluated a PVT developed at NASA for use in field studies. Although there are many PVT applications available for use on touch screen devices, we have no information on whether or not such tests account for the system latency of a given device. As such, we suggest that researchers use only touch screen–based PVT tests that have clearly documented how the latency is determined for the device that will be used.

In addition, we used a high-speed video camera and had two individuals document the time of the touch response to measure the latency of the device. Others have used different methodology to determine the device latency between a response box and RT documented on a computer-based test (Khitrov et al., 2014). Although we could have connected an RT box to the touch screen device to determine the latency arising from software processing, we chose to evaluate the latency using a high-speed camera in order to also capture differences between individuals relating to the administration of the PVT on a touch screen device. We did not counterbalance condition order. We wanted to measure the latency of the device, so we chose to keep the videotaped PVT first in order to minimize any variability that could be due to unknown underlying factors of the system (e.g., possible application lagging). Nevertheless, we believe that our findings are still valid because Balkin et al. (2004) and Dinges et al. (1997) found that PVT performance did not improve as a function of repeated administrations. Rather, sleep deprivation causes an overall slowing of PVT response times and an increase in the number of lapses (Doran et al., 2001), and generally these effects increase with time on task (Lim et al., 2010). We suspect that our results were likely attenuated and that randomization would have shown an even stronger effect for handheld landscape orientation.

Conclusions

Although the PVT is a sensitive and reliable indicator of fatigue, our findings suggest that researchers should be cautious when administering touch screen versions of the test. We found that the way that an individual orients the touch screen device while taking the PVT alters the recorded RT. The handheld landscape-oriented PVT using the thumb to respond to the stimuli produced the best results, and it was the most preferred by subjects. Therefore, we recommend that researchers using touch screen versions of the PVT should instruct participants to orient the device in landscape position and use their dominant thumb to respond to the stimuli. We recommend that developers of touch screen PVTs provide documentation of how the device latency and variability is calculated and incorporated into the results output. Researchers using the PVT on touch screen devices should determine the latency of their selected device and ensure that all devices used in a single study are the same to reduce the contribution of hidden factors that may alter response latency (e.g., Wi-Fi should be off, no other applications should be running, the same operating system should be used for all devices). Standardization of the hardware, software, and methodology used to administer the PVT on touch screen devices is essential for using such devices to estimate fatigue.

Key Points