Cognitive Neurophysiologic Synchronies

Tasks

For this study we used navigation training tasks that are integral components of the Submarine Officer Advanced Course (SOAC) at the U.S. Navy Submarine School, Groton, Connecticut. Submarine Piloting and Navigation (SPAN) is a high-fidelity simulation containing dynamically programmed situation events such as encounters with approaching ship traffic, changing weather conditions, and instrument failure. There are task-oriented cues to guide the mission, team member cues providing information on how other members of the team are performing/communicating, and adaptive behaviors that help the team adjust.

The teams contain 11 or 12 members in positions officer on deck (OOD), navigator (NAV), assistant navigator (ANAV), contact coordinator (CC), fathometer (FATH), helm (HELM), quartermaster on watch (QMOW), radar operator (RAD), recorder (REC), periscope operator (SCOPE), and captain (CAPT) and/or instructor (INST). The simulations require a mixture of task work and teamwork. For example, the task work for the RAD would be adjusting the range and bearing line on the radarscope, whereas the teamwork would be appropriately conveying this information to the CC in case of a new contact. Although we have collected EEG teamwork data from 21 SPAN sessions, the data reported here were derived from a subset of 12 of those sessions selected as (a) the persons in the same six crew positions were monitored by EEG, (b) there were no role or membership shifts in the teams across training sessions, and (c) there were three SOAC teams and three experienced submarine navigation teams that each performed two SPAN simulations. SOAC teams and sessions are designated with a T for the team and S for the session (e.g., T4S1); for expert teams, E is substituted for the team designation (e.g., E1S1). Each SPAN session begins with a briefing outlining the mission goals and providing information on position, contacts, weather and sea state. The scenario segment is more dynamic and contains easily identified processes of teamwork along with others which are less well defined. One regular process is the updating of the ship’s position termed rounds. Here, three navigation points are chosen, usually visually, and the bearing of each from the boat is measured and plotted on a chart. This process occurs every 3 min with a countdown from the 1-min mark. The REC counts down to the fix and logs the data. The regularity of this process is shown by the speech patterns of the REC for five SPAN sessions (Figure 1).

OPEN IN VIEWER

Figure 1.

Mapping the periodic updating of the submarine position with the recorders’ speech. During Submarine Piloting and Navigation the recorder uses a five-step countdown (shown to the left) to the moment when a round is taken. The timing of these five steps is shown for five teams where each row represents one of the steps. The team designations beginning with an E are experienced submarine navigation teams; those with a T are student teams.

The two expert sessions, E1S1 and E1S2, mostly show complete five-step rounds countdowns. The patterns were less regular for SOAC teams T4S2 and T5S5 where steps were omitted and occasionally fixes were missed. Another expert team E4S2 began the scenario with four effective fixes and then began having difficulties conducting regular rounds. This example is shown as it indicates there are likely levels of expertise. For all teams there were periods when the rhythm of rounds was broken, which was often indicative of stressful conditions.

Interleaved with these deterministic events are situations arising from new ship traffic, increased proximity to hazards, equipment malfunctions, or reduced visibility. In contrast to the regular updating of the submarine’s position, these events are more perturbations to the regular functioning of the team. Some are rapid such as a man overboard, whereas others evolve over 5 to 10 min and may be based on previous decisions. The speech patterns of the team in response to the evolving situation are much less regular than were those of the REC during rounds (Figure 2). The debriefing is the most structured segment of the training with team members reporting in order, beginning with the NAV. The task times ranged from 75 to 120 min, and the proportion of time allocated to the scenario and debriefing was variable depending on the team.

OPEN IN VIEWER

Figure 2.

Speech patterns during sample Submarine Piloting and Navigation (SPAN) team performances. The dynamics of speech are shown for an experienced submarine navigation team (labeled E2S1), and one junior officer navigation team that was midway through their required SPAN training (labeled T5S5). The demarcations between the briefing, scenario and debriefing segments are shown by arrows. ANAV = assistant navigator; XO/INST = commanding officer/instructor; CC = contact coordinator; FATH = fathometer; HELM = helm; NAV = navigator; ODD = officer on deck; QMOW = quartermaster on watch; RAD = radar operator; REC = recorder.

Electroencephalography

The ABM B-Alert® system contains an easily applied wireless EEG system that includes intelligent software designed to identify and eliminate multiple sources of biological and environmental contamination and allow real-time classification of cognitive state changes even in challenging environments. The nine-channel wireless headset includes sensor site locations F3, F4, C3, C4, P3, P4, Fz, Cz, POz in a monopolar configuration referenced to linked mastoids. ABM B-Alert software acquires the data and quantifies alertness, engagement, and mental workload in real time using linear and quadratic discriminant function analyses with model-selected power spectral density (PSD) variables in each of the 1 Hz bins from 1 to 40 Hz, ratios of power bins, event-related power, and/or wavelet transform calculations.

The data processing uses eye-blink decontaminated EEG files containing second-by-second calculations of the probabilities of high EEG-engagement (EEG-E), which is related to processes involving information gathering, visual scanning, and increased attention (Berka et al., 2004, 2007). This measure is generated from 22 PSD variables obtained from electrode combinations FzPOz and CzPOz over 1 to 40 Hz bins.

The neuropsychological tasks used to build the algorithm and subsequently used to individualize the algorithm’s centroids were presented using proprietary acquisition software. The algorithm was trained using EEG data collected during the OSLER maintenance of wakefulness task (OSLER; Krieger & Ayappa, 2004), eyes closed passive vigilance (EC), eyes open passive vigilance (EO), and three-choice active vigilance (3CVT) tasks to define the classes of sleep onset (SO), distraction/relaxed wakefulness (DIS), low engagement (LE), and high engagement (HE), respectively.

Simple baseline tasks are used to fit the EEG classification algorithms to the individual so the cognitive state models can be applied to increasingly complex task environments. These methods have proved valid in EEG quantification of drowsiness–alertness during driving simulation, in simple and complex cognitive tasks, and in military, industrial, and educational simulation environments (Berka et al., 2004, 2007; Stevens et al., 2007).

Layered Models of Neurophysiologic Synchronies

The first modeling step (Figure 3a) normalizes the second-by-second EEG-E measures to a team member’s average levels. This normalization identifies when a particular team member was experiencing above or below average levels of EEG-E and, by comparing across team members, whether the team as a whole was experiencing above or below their individual average levels for the session. In this normalization the EEG-E levels are partitioned into the upper 33%, the lower 33%, and the middle 33%, and these are assigned values of 3, –1, and 1, respectively, values chosen to facilitate further processing and enhance visualizations (Stevens et al., 2010a; Figure 3).

OPEN IN VIEWER

Figure 3.

Data normalization and modeling. The top panel (A) shows the raw levels of EEG-E being normalized into training vectors for neural network classification (Panel B). ANN = artificial neural network; QMOW = quartermaster on watch; NAV = navigator; ODD = officer on deck; ANAV = assistant navigator; CC = contact coordinator; RAD = radar operator.

The next step (Figure 3b) combines these values at each epoch for each team member into a vector representing the EEG-E levels for the team as a whole; these vectors are used to train unsupervised artificial neural networks (ANN) to generate different NS_E patterns of the team (Stevens et al., 2010a). In this process EEG-E training vectors are repeatedly (2,000–4,000 times) presented to a 1 × 25 node ANN where the output neurons were organized in a linear architecture. Pilot studies with architectures varying from 16 to 100 nodes indicated that the 25 node architecture provided a balance of speed and sensitivity. The ANN acts as a classifier similar to K-means clustering with the advantage that each node competes with its neighbors to the left and right for the training vector and a topology develops where vectors most similar to each other become closer and more disparate vectors are pushed away. The output of this training is 25 histogram patterns, termed NS patterns, showing the relative levels of EEG-E for each team member on a second-by-second basis. Profiles of three NS_E patterns are shown in Figure 4. Each pattern contains six histograms, one for each team member, and the height shows the relative levels of engagement. NS_E Pattern 1 represents a team where Members 2 and 4 had average levels of engagement, Team Member 6 showed above average engagement levels, and Team Members 1, 3, and 5 were below average. NS_E Pattern 10 represents a team where Members 1, 4, and 6 had above average levels of engagement and the other team members were below average. NS_E Pattern 24 was where all team members had above average levels of engagement except Team Member 5, who showed average engagement.

OPEN IN VIEWER

Figure 4.

Examples of neurophysiologic synchrony pattern profiles. The pattern designations above each figure are the same as in Figure 5. The high, average and low EEG-E designations are derived from the coding of the training vectors in Figure 3a.

To enable comparisons across teams, ANN models were generated using pooled data from 8 six-person teams. This resulted in a training set of 31,450 team vectors (∼9 hr of teamwork).

The resulting 25 NS_E patterns obtained following modeling are shown in Figure 5. NS_E Patterns 1 through 4 represented times when most team members had low EEG-E levels, whereas NS_E Patterns 13–15 and 22–24 represented times when most team members had high EEG-E levels.

OPEN IN VIEWER

Figure 5.

NS_E pattern and state classifications. The NS_E patterns are numbered 1–5, 6–10, etc. row-wise from the left to right. Each of the six histograms in each pattern represents the EEG-E levels of a team member. The order of team members is shown below the figure. The surrounding state squares with the 25 boxes show the most frequent associations of particular NS_E patterns with each NS_E state. QMOW = quartermaster on watch; NAV = navigator; ODD = officer on deck; ANAV = assistant navigator; CC = contact coordinator; RAD = radar operator.

Autocorrelation studies have suggested that there may be a temporal component to NS pattern expression, a hypothesis supported by adding an additional modeling step using hidden Markov modeling (HMM; Stevens, Galloway, Berka, & Behenman, 2010c). This process models temporal associations between different symbols that are the 25 NS patterns resulting from the ANN modeling. Like the ANN training process, HMM requires a training step. The input data for this training are 120 epoch-long segments of NS patterns obtained by segmenting the NS pattern data stream. HMM generally requires an estimate of the number of states to model into, and we have used five based on prior work and pilot studies (Soller & Stevens, 2007). The outputs of this HMM modeling are termed NS states, and the mapping of the different NS_E patterns to NS_E states is shown in Figure 5.

What do the different NS_E states represent? Unfortunately there is no simple answer. We have conducted HMM modeling dozens of times but have failed to develop simple patterns such as “team is fully engaged” or “fully unengaged.” There is always heterogeneity of NS_E patterns in each state, which is indicative of an underlying dynamics of NS data streams (Stevens & Gorman, 2011).

Hypothesis 1: NS models can be created that are sensitive to long-term (minutes/hours) and short-term (seconds/minutes) changes in the task

NS_E patterns and states can be visualized on a second-by-second basis or binned over multiple epochs for statistical analysis. Figure 6 illustrates a second-by-second mapping of the NS_E patterns and states for SPAN team T1S1. A consistent feature observed in all SPAN training sessions is the shift in NS_E expression at the scenario–debrief junction (indicated by the arrow). Here team T1S1 showed an increased expression of NS_E Patterns 1–5 and 17–21, which were previously expressed at low levels. The shift was more obvious for the HMM-derived NS_E states where most of the debriefing was dominated by NS_E State 3. Similar shifts at the brief–scenario junction are less pronounced, which is not surprising, as one briefing component is the determination of the ship’s starting position which is similar to rounds. Similar NS_E state shifts at the task segment junctions are also shown for three other teams. These shifts illustrate that NS_E expression is sensitive to major task changes.

OPEN IN VIEWER

Figure 6.

Dynamic expression of NS_E patterns and states during Submarine Piloting and Navigation (SPAN) training sessions. The top figure shows the second-by-second expression of each of the NS_E patterns expressed by SPAN team T1S1, and immediately below it is the expression of the five NS_E states. The expressions of NS_E states for three other teams are shown for comparison. The briefing, scenario, and debriefing segment junctions are shown by the arrows.

Another important feature illustrated in Figure 6 is that NS pattern and state expressions are not homogeneous but often punctuated by recurrent blocks that span shorter time scales. These 30- to 60-s state recurrences provide useful landmarks for relating NS expression with short-term simulation events, team responses, or team speech. Figure 7 shows two experienced navigation teams where the boat’s skippers paused the simulation midscenario to address the crews. For the first team (E1S2, Panel A) prior to Epoch 2890 the team was mainly expressing NS_E States 1, 2, and 3. Within seconds of the CAPT beginning to speak they transitioned through NS_E State 4 to NS_E State 5. The team remained in this state until the scenario resumed when they returned to prior NS_E state expressions. For the second team (E3S2, Panel B) the team was mainly expressing NS_E State 2, which then switched to NS_E States 3 and 4 while the skipper was speaking.

OPEN IN VIEWER

Figure 7.

Changes in NS_E expression during pauses in two simulations. These figures focus on two ∼10 min segments during which the skippers of two boats paused the simulation to address their navigation teams. Panel A is team E2S1, and Panel B is team E3S2. The gray areas highlight the periods of the pauses.

Hypothesis 2: NS models can be used for comparing NS dynamics across teams and training sessions

The next study compared the frequency distributions of the five NS_E states across four teams that each performed two SPAN sessions (Figure 8). As expected from the shifts in NS_E pattern expression at task junctions as shown in Figure 5, there was a differential expression of NS_E states during the scenario and debriefing with NS_E State 2 being overexpressed in the scenario and NS_E State 5 underexpressed (χ² = 1326, df = 4, p < 0.01).

OPEN IN VIEWER

Figure 8.

NS_E state frequencies for Submarine Piloting and Navigation (SPAN) sessions segments. The frequency percentage of each of the five NS_E states was calculated for the scenario (top) briefing (middle) and debriefing (bottom) segments for eight SPAN sessions.

For most teams NS_E States 1 and 2 were most frequent during the scenario. These represented periods when many of the team members were highly engaged (refer back to the NS pattern/state classification diagram in Figure 5). The exception was Team 1, where NS_E State 4 predominated; this state represents periods when many of the team members had below average engagement. There were no significant state frequency differences between SPAN Sessions 1 and 2 for any of the teams. There was more across-team and across-session heterogeneity during the briefing and debriefing segments, and as expected from the data in Figure 6, the distributions in the briefing were more similar to those in the scenario than were the distributions in the debriefing.

Hypothesis 3: The dynamics of NS expression relate to some established aspects of teamwork, yet contribute something new

The content and flow of communication are often data sources for studying teamwork, and messages are generally regarded as a fundamental unit for analysis. The debrief sections of SPAN simulations were initially chosen to explore linkages between NS_E expression and speech, as there is a regular pattern of discussion where only one person speaks at a time. The dialog during the debriefing from one SPAN session was transcribed, time coded, and aligned with NS_E expression (Figure 9). The gray bands emphasize the reports of different team members.

OPEN IN VIEWER

Figure 9.

Mapping discussion units to NS_E state expression. Initially the NAV summarized the simulation and illustrated and discussed areas for improvement. Next, the CC provided an alternative navigation solution to an issue that was raised by the NAV (“My personal comfort level would have been to shoot in between the two inbound merchants and have them be our front and back line backers,” etc.). In the next section (Epochs 3776–3922) the CAPT contributed general comments (“My main point is don’t wait until the last minute and then call that guy after you just messed with his own radar picture,” etc.). The debriefing continued along similar lines changing topics with each NS transition. OOD = officer on deck; MOB =man overboard.

From this example it appears that the major NS state transitions occurred around blocks of ideas or discussion units, that is, when closure of a topic was achieved and a new topic began. The NS_E state shifts were not linked to a specific speaker as multiple team members contributed to each discussion unit. Also, although the team was predominately in NS_E State 5 when the CAPT spoke, there was a long period at the end of the debrief where the CAPT was speaking and the team was in NS_E State 1.

Although the preceding approach may provide useful information regarding associations between NS_E expression and speech, the process was not optimal for studying SPAN teamwork. First, most speech is highly asynchronous during the scenario, with intermixed conversations of the different members of the navigation team. Second, this approach required a time-consuming analysis, decreasing its usefulness as a rapid response monitor of the team to changing SPAN events.

The next studies addressed these challenges by combining speech and neurophysiologic analyses with entropy calculations and expert performance analysis.

The markers for speech frequencies and the NS_E patterns are both symbolic data, and for quantitative comparisons it would be useful to have numeric metrics. One transformation is to calculate the Shannon entropy of the symbolic speech and NS_E data streams (Shannon, 1951). This metric is derived from information science and measures the level of uncertainty or “amount of mix” in a symbol stream. The idea was that as teams organized themselves around significant task events, there may be changes in the entropies of NS_E patterns or speech reflecting this cognitive reorganization.

Entropy is expressed in terms of bits calculated from a data stream. Entropy was calculated at each epoch using a 90-s sliding window of the prior history. As 25 NS_E patterns are available for modeling, the maximum entropy that we could expect from the 25 NS_E patterns would be log2 (25), or 4.64. For comparison, if a 90-s sequence of the data stream contained only 12 of the 25 patterns, then the entropy would drop to 3.6, and if the data stream contained only a single NS_E pattern, the entropy would be 0. Similar calculations were applied to speech. The entropy for speech was calculated by first assigning a numeric code to each of the 16 speakers in the SPAN; the extra 5 speaker symbols added to the 11-man team included a second instructor and/or evaluator, the technician as well as other speakers who could not be clearly identified from the audio stream. These symbols were substituted into the speech log, and then the entropy was calculated as described for NS_E.

Throughout the simulation there were significant speaker and NS_E entropy fluctuations, only some of which were associated with changes in the task boundaries. During the scenario there were periods when the team was relatively quiet, interspersed with periods when there was extensive speech by multiple team members. The speech of the REC provided evidence of when the team was not functioning smoothly as indicated by periods when the round was missed or marked multiple times. To complement these data, a lieutenant who was an instructor at the Submarine Learning Center summarized the teams’ stress level and performance from an audio recording of the session. From these data, a summary of the teams’ performance and dynamics was created (Figure 11), with the event numbers relating to Panel B in Figure 10.

OPEN IN VIEWER

Figure 10.

Linking NS_E entropy with team speech and performance events. Panel A uses bar symbols to mark the second-by-second speech of the different members of a Submarine Piloting and Navigation (SPAN) team (T4S2). Panel B shows the rounds sequence for this team as described in Figure 1, and above it are markers referring to different events that are further described in Figure 11. Panel C shows the entropy profile for the speakers during the simulation, and Panel D shows a similar entropy profile for the NS_E.

OPEN IN VIEWER

Figure 11.

Summary of team events and stress levels of team T4S2 in Figure 10. The event column refers to the steps in Panel B of Figure 10. The lieutenant (LT) evaluation column is the LT’s assessment of the stress level of the team, and the summary column provides an overview of the events occurring at different times. OOD = officer on deck; CC = contact coordinator)