Toward the Use of Pupillary Responses for Pilot Selection

Individual Differences in Pupillary Responses

Early empirical research about task-induced variations in pupil diameter found that pupil diameter increased with the difficulty of a memory load task (Hess & Polt, 1964; Kahneman & Beatty, 1966). Since these first research works, task-induced variations in pupil diameter have been widely studied with some research focusing on individual differences. Findings from studies about individual differences in task-evoked pupillary responses are quite inconsistent.

A first group of studies has found that low performers showed greater pupil dilations (e.g., Ahern & Beatty, 1979; Heitz et al., 2008 in the condition without incentives; Jainta & Baccino, 2010), suggesting that low performers needed to recruit more resources to complete the tasks, particularly when difficulty increased. In the present paper, the term “resources” refers to both working memory and controlled attention. Ahern and Beatty (1979) have shown that low performers at the scholastic aptitude test (used for college admissions in the United States) had larger task-induced pupillary responses than high performers during simple tasks (multiplications, digit span, sentence comprehension). Heitz et al. (2008) have measured pupillary responses during a reading span task and compared high versus low performers during an operation span task. The task-evoked pupillary responses during the recall phase of the reading span task revealed that low-span individuals had greater pupil diameter changes from their baseline than high-span individuals, specifically when no incentive was provided. Similarly, Jainta and Baccino (2010) have observed larger pupil increases for individuals who had made the most errors on a multiplication task compared to those who had made fewer errors.

On the contrary, a second group of studies has highlighted greater pupil dilations for high performers (e.g., Tsukahara et al., 2016 for the high-load trials; van der Meer et al., 2010 with the geometry analogy task), suggesting that high performers have more spare resources, specifically when the tasks are difficult. van der Meer et al. (2010) found that on a difficult task (geometric analogies), individuals with high fluid intelligence scores (assessed through the Advanced Progressive Matrices [APM] test) had greater pupil dilations than individuals with average APM scores. Recently, Tsukahara et al. (2016) found that high performers on working memory tasks (and also with high performers on fluid intelligence tasks) had higher pupil dilations on a simple memory task than low performers, specifically when the task became very demanding (from 9 to 12 letters to be memorized).

Finally, a third group of experiments found no differences in pupil responses for high and low performers (Heitz et al., 2008 in the condition with incentives; van der Meer et al., 2010 with the choice reaction time task). Importantly, all these studies varied in the way they identified individual differences (e.g., with a preexperimental assessment or based on task performances) and in the nature of the task performed during pupil measurement (e.g., simple multiplications or difficult analogical reasoning).

Depending on the specificity of the task and of the context, high performers may be characterized by a greater efficiency in the way they use their resources or by a greater level of spare resources to cope with a demanding situation. High performers may have more resources available, specifically with unknown tasks or at the beginning of the training, and may be more efficient in the way they use them after practicing these tasks. As a consequence, for difficult tasks or conditions, the pupil size of high performers should be higher than that of low performers; and for easy tasks or conditions, the pupil size of high performers should be smaller than that of low performers. This hypothesis is in line with the neural efficiency hypothesis of intelligence (Neubauer & Fink, 2009) and specifically with its interaction with task difficulty. Indeed, many studies based on brain measurements have found that more proficient individuals displayed lower brain activations when performing low to moderate difficulty tasks or after sufficient practice of complex tasks. On the other hand, they invested more brain resources when the task was very difficult, compared to less proficient individuals (e.g., Di Domenico et al., 2015; Doppelmayr et al., 2005; Dunst et al., 2014; Lipp et al., 2012; Puma et al., 2018). Therefore, in a context of selection, it would be helpful to identify those individuals who are likely to invest spare resources when the task becomes more demanding. The objective of the present study is to assess the predictive validity of task-evoked pupillary responses for a highly demanding training, namely airline pilot training.

Pilot Training

Pilot training is a highly complex learning process that requires trainees to acquire a large amount of knowledge (procedures, rules, aircraft laws, meteorology, etc.) and to develop many skills. For a student with no flying experience, airline pilot training lasts 2.5–3 years and is composed of theoretical and practical training. The practical training includes flying hours with a flight instructor and simulator flights. In case of learning difficulties, the attribution of additional flying hours or exclusion from the training is decided by the training organization. A frequently reported difficulty of pilot trainees during practical training is the lack of spare mental resources in order to manage the flight. In a similar context, military flight instructors reported that the most frequent difficulties of pilot trainees were related to “attention control” and inability to deal with the “load of flight” (Gopher et al., 1994). Indeed, managing a flight is a challenge for pilot trainees, as they have to manage the attitude of the aircraft (pitch, roll, and heading), anticipate its trajectory, communicate with other actors (air traffic controllers or their flight instructor), and manage the systems in the cockpit—all this under time pressure. Consequently, the pilot trainee might feel overloaded. The question that arises is whether this lack of spare mental resources during flight training could be associated with a lack of mental resources while performing a demanding laboratory task. Many pilot selection processes include a multitasking test in order to assess the ability of the applicant to perform several tasks concurrently. A meta-analysis of the predictive validity of performance scores derived from such tests has revealed a significant correlation with training outcome. Still, the global effect remains small (r = .23, Damos, 1993). This means that obtaining a high performance on such a test is not necessarily linked with the pilot’s training success. Indeed, a high performance may be obtained with a more or less high level of effort (Wickens, 2002).

Overview of the Study

The present paper focused on the capacity of individual differences in task-evoked pupillary responses to predict learning individual differences in an ecologically valid environment. More precisely, we studied the relationship between pupillary responses on a demanding laboratory task and the outcome of airline pilot training. The rationale of the study is that if the differences in pupillary responses in a laboratory task are meaningful and ecologically valid, then they should be associated with differences in training outcome.

In the present study, pupillary responses of pilot trainees were recorded while they were performing a psychomotor multitasking scenario before the beginning of their practical training. This task consisted of six successive stages with one to four subtasks to be performed concurrently. Therefore, the mental workload induced by this task was supposed to increase from low load to high load. Then, two groups of pilot trainees were identified after practical flight training completion (1.5–2 years after the recording of the pupillary responses): (a) trainees who successfully completed the training without notable difficulties and (b) trainees who completed the training with more notable difficulties. The aim of the study was to identify whether these two groups of trainees would have different patterns of pupillary responses measured during the multitasking laboratory task.

Hypotheses

Following the neural efficiency hypothesis and, more specifically, its interaction with task difficulty, the following hypotheses could be derived:

H1: Given their higher level of spare mental resources, the most proficient pilot students should show greater pupil size changes from low-load to high-load stages when performing the complex psychomotor multitasking scenario, in comparison to less proficient pilot students.

H2: Given their higher mental efficiency, the most proficient pilot students should show lower pupil sizes during the lowest load stage of the psychomotor task, in comparison to less proficient pilot students.

Participants

Twenty pilot students (18 men, 2 women) from the French civil aviation university volunteered and received a movie ticket after the experiment. They were all aged between 18 and 23 years (M = 20.9, SD = 1.3). They had been selected after a multiple-stage selection process comprising scientific knowledge tests, English tests, cognitive ability tests, group exercises, and individual interviews. Moreover, all of them came from scientific preparatory classes for competitive admission to elite universities. The Priority Management Task used in the present paper was the same as the one used during the cognitive ability tests stage of the selection. Thus, all participants performed sufficiently well at this task to be selected. At the time of the experiment, all participants were following their pilot theoretical training and had not started the practical phase of the training. This study was conducted in accordance with the principles expressed in the Declaration of Helsinki. Informed consent was obtained from each participant.

Procedure

First, the pilot students performed a psychomotor multitasking scenario individually. Participants sat in front of a 19-inch, 1024 × 768 resolution computer screen and interacted through two joysticks and a keyboard. Pupil size was measured using an EyeLink 1000 desktop eye tracker (SR Research Ltd., Mississauga, Ontario, Canada). This eye-tracker possesses a spatial accuracy greater than 0.5° and a 0.01° spatial resolution. The sampling rate was set to 1000 Hz. A chin and forehead rest was used to maintain these distances and to avoid head movements. All eye-tracking data were extracted using the SR Research default algorithm. The experiment took place in a quiet simulation room. The volunteers were seated on a comfortable chair. The eye tracker camera was placed at a distance of 20 cm from the screen and the eye camera was at a distance of 60 cm from the screen. The lighting of the room was maintained constant, and luminance was controlled and remained constant throughout the experiment (212 lx as measured a posteriori with an analogical luxmeter Extech 401 025). Before starting the experimental phase, participants performed a short calibration phase in order to adjust the eye tracker. Participants were then told to perform the task in strict accordance with the instructions provided on the screen.

The multitasking scenario, labeled the Priority Management Task (Figure 1), was organized in six successive 4-min stages, except the first stage that lasted 3 min (see Matton et al., 2016, for a detailed description of this task and computation of the performance). To summarize, participants had to complete four concurrent subtasks that were successively added (from only one subtask at the first stage [S1], to four subtasks at the fourth stage [S4]): monitoring, tracking, detecting, and calculating. The “monitoring” task consisted of maintaining the levels of four gauges within an interval by using the first joystick. Every 15 s, one of the gauges deviated from its position at a speed of 10–70 pixels per second. To maintain the level of the gauge in the target interval, the participant had to press several buttons to select the right gauge and then adjust the gauge to the desired value with the joystick. The “tracking” task consisted of keeping a cross positioned in a moving circle of 50-pixel diameter, in an area bordered by a large circle of 300-pixel diameter, through the second joystick. The circle moved every 15 s at a speed of 3–12 pixels per second. The “detecting” task consisted of detecting the presence of three target letters (which varied from stage to stage) in a block of nine letters. Participants had to press one of nine keyboard keys as quickly as possible when a target letter appeared in the corresponding zone. A new block of letters was presented every 15 s. The “calculating” task consisted of simple arithmetic problems (e.g., deducing a distance from given speed and time); participants had to type the numeric answer as quickly as possible. Whether an answer had been given or not, a new problem was presented every 15 s. At each stage, instructions indicated that each subtask was equally important and the number of concurrent subtasks increased from S1 (monitoring subtask only) to S4 (all four subtasks concurrently). Two final stages, S5 and S6, varied the assigned priorities of the four subtasks. Subtasks were appended in the same order for all participants. Characteristics of all subtasks were exactly the same for each participant: each subtask required an action from the participant every 15 s at the same time. Thus, mental workload was supposed to increase from S1 to S4 with the addition of a new subtask at each stage. Performance indices were computed in 100 ms steps. For each of the subtasks, this ranged from 0 (worst) to 100 (best). The performance for “tracking” was proportional to the distance between the cross and the moving circle. It was given a score of 100 whenever the cross remained within the moving circle. Depending on the speed of the moving circle, performance could decrease at a rate of 0%–20% per second. The performance for “monitoring” was proportional to the maximum distance between the four gage levels and their corresponding target intervals. Moreover, the performance was given a score of 0 whenever one gage level went beyond a 60% tolerance interval. Depending on the speed of the gage, performance could decrease at a rate of 0%–33% per second. Performances for “detecting” and “calculating” followed the same principle: they were given a score of 100 when the block of letters or the arithmetic problem was presented. Then, the performance level gradually fell at a rate of 6.67 per second until the correct answer was keyed in. If a wrong answer was supplied, performance was more substantially decreased. At each stage, a global performance was computed by averaging all subtask performances. The instantaneous performance level of each subtask was displayed through a corresponding gauge in the top center of the screen. When performance on one of the subtasks dropped below a 10% performance threshold, the global performance was set to 0, in order to avoid participants neglecting one or several subtasks.

OPEN IN VIEWER

Figure 1

A screenshot of S4 of the Priority Management Task. The letters (T = tracking, M = monitoring, D = detecting, C = calculating) have been superimposed here for the reader but were not displayed during the task. P stands for performance gauges.

Second, we collected data of the training outcome of these pilot students after completion of practical training, that is, 2 years later. We focused on the number of additional practical training hours they had received and we also collected some qualitative information regarding the training difficulties encountered. Additional practical training hours are requested by flight instructors and decided by the training organization. Given the cost of flight hours, a high number of additional flight hours is only given in case of substantive learning difficulties. Pilot students were divided into two groups, medium (n = 6) and high (n = 14) levels, according to whether they had experienced such learning difficulties or not (threshold was set at 3 hr of remedial training; see “Results” for more details). None of the 20 pilot students who took part in this study were excluded from the training; thus, no “low” level group was identified. The term “low” was kept unused to account for potential training failure during potential replications. Pilot students took up the Priority Management Task with eye-tracking recording during theoretical training, which means before flight training; none of them had yet received any remedial training. Therefore, none of the events that led to pilot categorization could have any influence on pupillometry or performance on the psychomotor task. Moreover, during training completion, pilot instructors were unaware of the experiment being run and prescribed complementary flight hours as they usually did to students unable to achieve the mandatory level for final certification of air pilot training. Flight instructors discussed the threshold to consider medium or high performer students without consideration of the experiment.

Analyses

Fixations were extracted by the Eyelink standard algorithm and pupil sizes were only analyzed for fixations. Thus, filtering for blinks or saccades was not needed. Pupil sizes were extracted with the Eyelink arbitrary unit and then converted into millimeters according to the formula given in the Eyelink manual (i.e., pupil size [mm] = square root(pupil size(arbitrary unit))/10). First, we computed average pupil size per 1-min periods for each stage (three blocks for the 3-min stage S1 and four blocks for the other 4-min stages) that were used as repeated measurements within a stage. As the first stage (S1) consisted of performing one simple monitoring task, it was identified as the lowest load stage. As we were interested in assessing the increase in use of mental resources when the workload rose, we decided to compute differences between each block average pupil size of the subsequent stages (S2 to S6) and the mean of the pupil size of the lowest load stage (S1) and called it pupil size variation from S1. In other words, we further analyzed the following mean differences: (S2−S1), (S3−S1), (S4−S1), (S5−S1), and (S6−S1). Thus, we did not subtract any resting baseline value but the mean pupil size at the lowest load stage. Beatty and Lucero-Wagoner (2000) recommended the baseline value to be subtracted from peak dilation or average pupil diameter. We preferred average pupil diameter because of its higher robustness. A 2 × 6 mixed analysis of variance (ANOVA) was then conducted on the pupil size changes from S1 with a between-subject factor (“training outcome”—high or medium) and a within-subject factor (“stage”—from S1 to S6). Second, raw pupil sizes were analyzed as they provided further explanation on the dynamics observed when considering pupil size variations.

Statistical analyses. All analyses were computed using R (R Core Team, 2018) and all ANOVAs were conducted with the ez package. Effect sizes for ANOVAs were generalized eta-squared (ges) (e.g., Olejnik & Algina, 2003). Post-hoc analyses consisted of pairwise t tests with Bonferroni’s adjustment method (with the rstatix package). Eye-tracking variations and raw sizes as well as performances were analyzed using a repeated measures ANOVA with the “group” factor as a between-subject variable and the “phase” factor as a six-level intra-individual variable. A 2 × 6 mixed ANOVA was then conducted on the pupil size variations from S1 with a between-subject factor (“training outcome”—high or medium) and a within-subject factor (“stage”—from S1 to S6). When Mauchly’s test assumption of sphericity was violated, degrees of freedom of within-subject factors were corrected using Greenhouse-Geisser estimates of sphericity and corrected p values were provided. Given the non-normal distribution of raw pupil sizes, the analyses were made using Mann-Whitney U test and Bonferroni corrections for p values.

Pilot Training Outcome

Among the 20 pilot students, all of them succeeded in the practical airline pilot training but some of them received additional training (Figure 2). After discussions with flight instructors, a threshold of 3 hr of additional practical training was set to contrast those students who faced little or no difficulty during training (n = 14, labeled “high level”) and those who faced more difficulties during training (n = 6, labeled “medium level”). Among the 14 high-level pilot students, 12 received no additional training and the other two received 1 hr and 1.5 hr, respectively. The six medium-level pilot students received additional training because they were “below standard at a progress check” (n = 1, receiving 3 hr additional training), because they failed the commercial pilot license at the first attempt (n = 2, receiving 3 hr and 5 hr additional training, respectively), because of “problems visualizing their position in a three-dimensional space” (n = 1, receiving 5 hr additional training), “lack of mental resources during the flight” (n = 1, receiving 9 hr additional training), and “difficulty in workload management” (n = 1, receiving 11.5 hr additional training). However, the additional training hours were sufficient to enable final success of the airline pilot training for all medium level students.

OPEN IN VIEWER

Figure 2

Barplot of additional training hours during practical training for the 20 pilot students. An arbitrary threshold of 3 hr of additional training has been set to identify those students who experienced some difficulties. Fourteen pilot students received less than 3 hr of additional training (high level) and six of them received 3 hr of additional training or more (medium level).

Pilot Training Outcome and Task-Evoked Pupil Variation

A mixed ANOVA was run on the pupil size variation from S1 with one between-subject factor (“training outcome”—high or medium) and one within-subject factor (“stage”—from S1 to S6). A significant interaction between the factors “training outcome” and “stage” was observed for the pupil size variation from S1, F(3.05, 54.9) = 6.10, p = .001, ges = .10 (Figure 3). The simple effect of training outcome was significant for high-level (p < .001) and medium-level students (p < .001). However, pairwise comparisons showed that the pupil size variation from S1 increased significantly from S1 to S4 only for high-level pilot trainees (S2 vs S1, p = .002; S3 vs S2, p = .038; S4 vs S3, p = .008). Differences for successive stages were all nonsignificant for medium-level pilot trainees (the only significant difference was between S1 and S4 and between S1 and S5). For information, conclusions were identical after changing the criterion for categorizing high- versus medium-level pilot students (with 0 additional hours and with 5 additional hours). Therefore, the variation in pupil size over levels of mental workload was larger for high-level than medium-level pilot trainees. Thus, H1 was confirmed: high-level pilot students showed greater pupil size changes from low-load to high-load stages than medium-level pilot students. Moreover, the two pilot students who experienced difficulties in the management of mental resources, as explicitly identified by their flight instructors, received 9 and 11.5 hr additional training and would have been correctly classified in the medium-level pilot student group. The difference between the two groups was the largest at stage S4, that is, when the mental workload was at its highest. An inspection of individual patterns revealed that the pupil size variation from S1 at S4 comprised between 0.36 and .67 mm for all but one high-level pilot student and that all but one medium-level pilot student presented a pupil size variation from S1 at S4 of between 0.17 and .26 mm (Figure 6, Appendix 1). Hence, based on a threshold at .30 mm for example, in each group all but one pilot student would have been correctly classified in the high or medium category. Thus, the two pilot student groups could be quite clearly discriminated regarding each individual pupil size variation from S1.

OPEN IN VIEWER

Figure 3

Pupil size variation from mean pupil size at S1 to mean pupil size at another stage of the Priority Management Task for pilot students who experienced difficulties during the 2-year practical training (medium-level) or not (high-level). The number of subtasks increased from one to four from S1 to S4 and remained at four from S4 to S6. Error bars represent 95% confidence intervals of the means. For each subject, data were averaged in blocks of 1 min.

An analysis of raw pupil sizes (Table 1 for descriptive statistics) revealed that at the lowest load stage, the high-level pilot students showed lower pupil sizes than medium-level pilot students (Figure 4), W = 145, p < .001 (Mann-Whitney U test). Thus, H2 was also confirmed. The five other paired comparisons for S2 to S6 highlighted lower pupil sizes for the high-level group except for S6 (Mann-Whitney U tests with Bonferroni correction for p value computation). Thus, differences were not limited to the lowest level of workload but this does not contradict our hypothesis. The global ANOVA revealed a significant interaction between the factors “training outcome” and “stage” for the raw pupil size, F(3.12, 56.07) = 6.37, p < .001, ges = .01, confirming the difference in pattern of pupil size variation across the task between high-level and medium-level pilot students. Individual data (Figure 7, Appendix 2) did not reveal the same discrimination between raw pupil diameter patterns as for the pupil size variation from S1 patterns. Indeed, the two groups of pilot students were less clearly discriminated with the raw pupil sizes than with the pupil size variation from S1.

OPEN IN VIEWER

Figure 4

Raw mean pupil size (mm) at each stage of the Priority Management Task for pilot students who experienced difficulties during the 2-year practical training (medium-level) or not (high-level). Error bars represent 95% confidence intervals of the means. For each subject, data were averaged in blocks of 1 min.

OPEN IN VIEWER

Table 1

Means and Standard Deviations of Raw Pupil Diameters in Millimeters for Each Stage (S1–S6) of the Multitasking Scenario and for Each Group of Pilot Students (Medium and High)

Stage	High-level pilot studentsM (SD)	Medium-level pilot studentsM (SD)
S1	7.52 (0.51)	8.08 (0.38)
S2	7.69 (0.51)	8.18 (0.46)
S3	7.82 (0.49)	8.27 (0.41)
S4	7.98 (0.52)	8.34 (0.43)
S5	7.96 (0.51)	8.33 (0.43)
S6	7.92 (0.50)	8.24 (0.45)

Pilot Training Outcome and Multitasking Performance

A question that arose is whether these differences in pupil dilation for the two groups of pilot students would have also been observed for performance measurements. No significant interaction between the factors “training outcome” and “stage” was observed, F(2.10, 37.79) = 1.76, p = .18, ges = .05. Moreover, only a marginally significant difference of multitasking performance was found regarding training outcome, F(1, 18) = 3.10, p = .095, ges = .06 (Figure 5 and Table 2 for descriptive statistics). Besides, “stage” was significant, F(2.10, 37.79) = 115.61, p < .001, ges = .79. The slight performance difference at S5 was confirmed with additional data, published elsewhere (Matton & André, 2014) and confirmed by the individual performance data (Figure 8, Appendix 3). Interestingly, performance differences between the two groups of pilot students were small (except at S5), compared to differences in pupil diameter patterns. Moreover, as the maximum performance score is 100 at each stage, performance data are consistent with the claim that S1 is an easy stage.

OPEN IN VIEWER

Table 2

Means and Standard Deviations of Performance Score for Each Stage (S1–S6) of the Multitasking Scenario and for Each Group of Pilot Students (Medium and High)

Stage	High-level pilot studentsM (SD)	Medium-level pilot studentsM (SD)
S1	97.8 (0.5)	97.3 (1.1)
S2	97.3 (0.9)	96.5 (1.7)
S3	95.1 (1.3)	94.5 (1.8)
S4	87.6 (3.1)	86.2 (2.6)
S5	84.9 (5.0)	80.3 (5.4)
S6	88.2 (3.0)	86.7 (3.2)

OPEN IN VIEWER

Figure 5

Global performance (average of subtask performance) for the Priority Management Task for pilot students who experienced some difficulties during the 2-year practical training (medium-level) or not (high-level). The number of subtasks increased from one to four from S1 to S4 and remained at four for S4 to S6. Error bars represent 95% confidence intervals of the means.

Limitations

Heitz et al. (2008) found that high-span individuals had larger resting pupil diameters than low-span individuals (for preexperimental and pretrial pupil baselines). Recently, this result has been replicated: individuals of the upper quartile of a working memory capacity composite score had larger baseline pupil sizes (preexperimental and pretrial pupil baselines) than individuals of the lower quartile (Tsukahara et al., 2016). Unfortunately, we did not collect resting baseline pupil measurements. This will be the case for future investigations.

We observed the expected patterns when comparing proficient and less proficient pilot students. However, all the pilot students of this study finally completed the flight training. The question remains open whether pilot students who fail the training would have a different pattern of pupil size variation or a similar pattern to the less proficient students.