Measuring Cognition in Teams

Team Mental Models

Mental models at the individual level can be described as “mental representations of objects, actions, situations or people” (Langan-Fox, Anglim, & Wilson, 2004, p. 333). As mental representations, mental models include not only the knowledge of objects, actions, situations, or people but also knowledge of the relationships between the concepts and how the individual organizes the information (Johnson-Laird, 1983, as cited in Edwards, Day, Arthur, & Bell, 2006). Team mental models are conceptualized as being relatively stable (i.e., more stable than situation models that change from moment to moment) but are still considered somewhat dynamic in that they exist in dynamic environments and are developed over time (e.g., Cannon-Bowers, Salas, & Converse, 1993; Klimoski & Mohammed, 1994).

Team mental models are by far the most commonly studied construct within this stream of literature with 92 of the recorded instances of team cognition measurement being some form of team mental models. Individuals and teams can hold mental models about a variety of information, but the majority of the reviewed research has examined team mental models regarding task-related knowledge or teamwork-related knowledge. Team mental models are generally operationalized in terms of the similarity of mental models between team members (i.e., the extent to which individual team member mental models are overlapping or shared) and the accuracy of mental models (i.e., the extent to which the team’s mental model is correct as determined by comparison to an expert model). Both, methodologically, are analyzed in similar ways given that accuracy is essentially the similarity between a given mental model and an expert version of that same mental model.

Traditionally, the term shared, when used to describe team mental models, referred to the overlap or similarity between individual team member mental models, but more recent theorizing has started to explore an alternative understanding of the word shared. Specifically, shared can refer to the idea of overlap or commonality (e.g., we share values), but shared can also refer to dividing something up (e.g., we share the pie; Klimoski & Mohammed, 1994). In this second conceptualization in which a shared mental model is the division of knowledge across team members, it seems much more similar to the next team cognition construct to be discussed, transactive memory systems.

Transactive Memory Systems

Transactive memory systems (TMS) can be defined as a shared system used for encoding, storing, and retrieving information about different domains (Ren & Argote, 2011). The concept can be seen as having two primary components: (1) the structured or organized knowledge held by the individuals of a group, and (2) the processes that occur in order to encode, store, and retrieve that knowledge. Historically, transactive memory systems have been viewed as the “other side of the coin” to shared mental models, as shared mental model research originally focused on the overlap of knowledge, whereas transactive memory system research focuses on dispersion, or specialization, of knowledge. As mentioned previously, however, the literature on team mental models has begun to recognize the idea that mental models can indeed be distributed. Therefore, these two constructs have become more similar than different.

Transactive memory systems, when first conceptualized in the context of romantic relationships, referred to the division of labor within relationships in terms of encoding, storing, and retrieving information (Wegner, 1987). As it has been applied to work teams, the conceptualization has changed slightly as new ideas and measures have been developed. On one hand, Austin (2003) defines transactive memory as a four-dimensional construct consisting of (1) group knowledge stock (i.e., the total combination of knowledge across team members), (2) consensus about knowledge sources (i.e., the similarity of mental models regarding who knows what within the team), (3) specialization of expertise (i.e., the extent to which expertise is distributed), and (4) accuracy of knowledge identification (i.e., the accuracy of the mental models regarding who knows what within the team). This particular conceptualization is very clearly linked to the mental model literature in that it essentially focuses on the similarity, accuracy, and structure of knowledge within the team. Lewis (2003), on the other hand, defines transactive memory systems as a three-dimension construct consisting of (1) knowledge specialization, (2) credibility, and (3) coordination. This particular conceptualization includes a focus on both knowledge structure and the coordination processes engaged in by the team to access that knowledge.

Team Situation Awareness

Endsley (1988) defined situational awareness (SA) as “the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future” (p. 97, as cited in Muniz, Stout, & Salas, 1996). This construct differs from team mental models because it represents a dynamic understanding of a specific situation that changes quickly as environmental changes are incorporated into a new understanding, rather than a structured and stable knowledge of the “big picture.” However, it is similar to team mental models in that situation awareness is essentially a team’s dynamic mental model regarding the environmental situation surrounding the team. In fact, some team knowledge research has examined what is known as team situation models, or the “team’s understanding of the situation at one point in time” (Cooke, Kiekel, & Helm, 2001, p. 299). Arguably, the concept of team situation models, drawn from the team mental model literature, is nearly identical to the concept of team situation awareness, and therefore, measurement approaches used to measure one concept could be applied for measuring the other. It should be noted, however, that situation awareness is generally considered to be more dynamic than team mental models, and therefore, some research has focused on developing more dynamic approaches to measuring situation awareness (e.g., Gorman, Cooke, & Winner, 2006).

Strategic Consensus

Strategic consensus is a team cognition construct that has emerged from the management literature and is generally studied within top management teams (e.g., Floyd & Wooldridge, 1992; Kellermanns, Walter, Lechner, & Floyd, 2005). Strategic consensus can be defined as a team’s shared understanding regarding the high-level strategic goals of the team or organization. Studies almost exclusively use field samples of managers and other high-level executives within organizations. The most common measures of strategic consensus either ask individuals to respond to a set of questions regarding a particular strategic approach that organizations can take (e.g., our competitive priority is the development of new products; Camelo-Ordaz, Hernández-Lara, & Valle-Cabrera, 2005) or directly asks respondents to report how much agreement there is within their organization or team on strategic goals (e.g., To what extent is there agreement within the team on the long-term strategic goals of the company; Vissa & Chacar, 2009). Despite the fact that there has been a large amount of disagreement regarding the exact definition of strategic consensus, most research and measures reflect the underlying assumption that consensus refers to the level of agreement or sharedness between individuals (Kellermanns et al., 2005), clearly tying this concept back to the original shared mental model paradigm.

Team Cognition as Interaction

Within a smaller and newer stream of literature that examines team cognition as dynamic interactions or processes, cognition is most often conceptualized as the communication exchanges that occur between team members. Team cognition is seen as an activity, and not a property or a product of that activity (Cooke et al., 2013). It is important to emphasize that within this theoretical perspective, team cognition is not just represented within the communications of the team, but rather the act of communicating is in fact conceptualized as the enactment of team cognition in and of itself. This is distinct from the approach of some research in which communication records are used simply as a data source from which to pull information about conceptually static team cognition structure such as team mental models or transactive memory systems. Studies that truly take a dynamic process perspective to team cognition generally observe team interactions and code particular behaviors and communications as instances of various cognitive processes such as sensemaking or knowledge building.

Integration Across Domains

The research on team mental models tends to refer to the overarching idea that teams hold knowledge, and this knowledge is structured (to some extent shared, to some extent distributed) across team members. Transactive memory systems research has also focused on the structure of knowledge, specifically distribution, but also looks at knowledge-of-knowledge (awareness of expertise location) and coordination of knowledge (cognitive interaction). Strategic consensus can essentially be understood as a shared (in this case, overlapping) mental model of strategy. Situation awareness is also essentially a mental model, but a relatively more dynamic one regarding the changing environment or situation rather than teamwork or taskwork. Finally, team cognition as interaction examines team-level processes, such as coordination or knowledge building, which are conceptually similar to the coordination concept described within the Lewis (2003) conceptualization of transactive memory. In sum, all of the seemingly disparate research domains described above essentially examine different ways to understand the distribution, structure, and interactive manipulation of knowledge within a team. Unfortunately, measurement development has up to this point been relatively restricted to one domain or another, without considering possible options from other closely related team cognition domains despite the fact that measures used in one domain are very likely to be useful in another, given the conceptual similarities. In the following section, we further develop several potential cross-domain applications for various methods.

Interview Transcripts

Some research has studied team cognition using qualitative textual data as a source. Usually, individual interviews or focus group interviews with the members of teams are transcribed or open-ended surveys are used to collect textual data regarding past performance (e.g., Bourbousson, Poizat, Saury, & Seve, 2011; Espinosa, Slaughter, Kraut, & Herbsleb, 2007; Hare & O’Neill, 2000; Rowe & Cooke, 1995). There are several advantages to interviews for eliciting team cognition content. First, because interviews are conducted outside of performance episodes, they are less obtrusive in terms of performance interruption compared to self-report measures. Teams are not required to divert their attention away from the task at hand for any reason. Second, because they are conducted outside of performance episodes, they are temporally flexible and therefore easier to schedule and conduct from a practical perspective. The researchers and participants are not constrained to any particular performance episode to complete the data collection. Third, because interviews can be structured or designed to elicit any self-reportable information, they can be used to measure all of the commonly studied team cognition constructs including team mental models, transactive memory, strategic consensus, and situation awareness. All that is necessary to use interviews within each of these domains is to carefully develop a set of interview questions designed to elicit the corresponding information. Interview questions can be designed to elicit information regarding static or dynamic knowledge structures (e.g., What is your understanding of the task?), perceptions of knowledge (e.g., How similar are your team members to one another regarding their understanding of the task?), or interactive cognitive processes (e.g., describe the activities your team engages in when new information enters a situation).

One of the primary disadvantages of interviews compared to the other textual approach, communication data, is that interviews that are inherently self-report in nature are therefore less objective. When actual team communications are recorded and transcribed, the textual data that result are an exact representation of the interactions and processes as they occurred within the team. When the team members are interviewed after the fact, however, they can only report to the best of their ability their possibly inaccurate perceptions regarding what happened within the team in the past. This means interview data are susceptible to all of the same human cognitive biases that other self-report data are susceptible to, such as remembering only recent events, selective memory, or general halo bias (see Borman, 1991, for a review of potential rater errors).

Communication Transcripts

A sizeable portion of team cognition research has used communication recording as a method of data collection (e.g., Bierhals, Schuster, Kohler, & Badke-Schaub, 2007; Cooke et al., 2009; Ellis, 2006; Tschan et al., 2009). In most studies that use this method, the verbal interactions between team members in a co-located team (e.g., student teams in a laboratory) are recorded via audio-recording technologies, and then the audio recordings are transcribed into written scripts that represent the word-for-word interactions of team members often along with information regarding who said what, to whom, and at what time. Those scripts are then usually coded for team cognition constructs using some sort of content classification scheme, as will be described later. Theoretically, communication records are in line with the underlying assumption that team cognition is an interactive process, and not a product of that process. However, the majority of past research has actually coded communication data for more static knowledge structures (e.g., shared mental models; Bierhals et al., 2007) and only a small number of studies have coded communication for interactive processes (e.g., Gorman & Cooke, 2011).

In terms of advantages, communication recording is one of the least intrusive methods for collecting team cognition data, as it does not require the team members to stop their activities or shift their attention for any reason during team performance episodes, and it also does not require the presence of a physical human observer during real-time team activities. Audio-recording technology is also widely available and relatively inexpensive. Communication records can be replayed, allowing the actual coding and rating of data to happen under less time pressure after data collection is complete, and ensuring that any raters engaged in coding can revisit missed information. This is one of the primary advantages of communication and video recording as data collection methods that do not require the presence of human observers and also result in a permanent, and therefore reviewable, source of data.

The primary disadvantage of communication records as a source of data actually lies within the analysis rather than the data collection. Specifically, almost all current analysis approaches require that communication transcripts are transcribed into written scripts, and then highly trained experts must spend hours reading and categorizing the information within those scripts. A general rule of thumb, drawn from the experience of the authors, is that the transcribing of 1 hour of audio recording will take 3 or more hours to complete. This proportion may vary depending on the number of team members speaking that must be distinguished from one another, the clarity of the voices, recording quality, and experience of the transcriber. However, assuming this proportion is approximately correct, this means that transcribing the 300 hours of verbal communication recorded in Albolino, Cook, and O’Connor, 2007, for example, would take somewhere around 900 hours of labor.

The second stage of content categorization can often take a similar amount of time as well, putting the estimated total time investment to measure team cognition via communication recording in Albolino et al. (2007) at near 1,800 hours of labor. Communication records are a rich, but time-consuming, source of data. An alternative option to using one’s own time or labor resources to complete these transcribing activities would be to send the communication transcripts to a professional transcribing company that has the experience necessary to get the work done much more quickly. This option, however, requires the monetary resources to pay for transcribing rather than spending time to transcribe, so there is a nearly direct resource tradeoff. Clearly, measuring team cognition in this way can result in some of the most in-depth and informative data on team cognition, but it also requires a large amount of upfront investment either in terms of time or money.

Video Records of Behavior

The use of video recording is the most common, and least intrusive, of the behavioral observation approaches (e.g., Patrick, James, Ahmed, & Halliday, 2006; Prince, Ellisa, Brannick, & Salas, 2007; Reid & Reed, 2000). Video recording equipment is used to capture the interactions of team members, and then trained raters code those videotapes for team cognition-related behaviors and events after the fact. This approach is less intrusive than direct observation because it does not require the presence of a human observer. The known presence of human observers can change the behavior of team members, especially if team members are cognitively aware that they are being evaluated or judged. Research on maximum and typical performance suggests that individuals exert more effort toward performing well when they are being observed by researchers or supervisors (Klehe & Anderson, 2007). Discreet videotaping of interactions and after-the-fact rating can reduce the possible unintended impact of observers on team behaviors. Video records also provide another advantage over direct observation in that the videotapes can be watched multiple times, slowed down, and can be reversed/fast-forwarded in order to replay or pinpoint particular behaviors or events. This reduces the possibility that raters will miss important events.

The disadvantages of video records are generally practical in nature. It is possible that when studying certain high-risk or high-secrecy types of teams, video recording will not be permitted and therefore will not be an option for observing behavior. Video recording equipment is also more expensive than some of the other data collection methods such as self-report measures or communication recording. Video recording also requires a large amount of digital data storage space compared to most other methods. These practical considerations may make video recording a less appealing option for team cognition measurement in particularly restricted situations or when monetary or data resources are limited.

Real-Time Observations of Behavior

Sometimes, for practical or other reasons, video recording is not possible, but direct, real-time observation is possible (e.g., Kaber et al., 2006). This is likely in military or government situations in which video recording would be prohibited but observation can be arranged. In this case, trained raters are required to physically be on location during a team’s task performance and to code and rate behaviors as they occur. This essentially results in the same type of data as video records but presents a unique set of challenges. First, the presence of a human observer may have unintended consequences on the behavior of the team, as mentioned previously (Klehe & Anderson, 2007). This influence can be reduced by training raters to remain as inconspicuous as possible and to stay out of the way of the team while observing. As time passes and the team members become accustomed to the presence of the observer, the potential for contamination should be reduced.

Real-time observation also increases the probability that raters may miss particular behaviors or events as the action cannot be paused or reversed as in videos. There is also a potential for unintended influences on the rater’s ability to accurately rate events, especially if the team context is a highly stressful or dangerous one. Real-time observation can also be costly either in terms of time or money, depending on who does the observing. If the researchers do the observation themselves, this will require a large amount of time spent in observation sessions. If the researchers instead pay for trained observers, they may save themselves time but spend much more money. Direct, real-time observation does have one potential advantage over video recording in that observers have the ability to adjust and respond to the situation at hand, for example, if the action moves into a new location. A fixed video camera, however, would likely not have this capability and may not accurately observe behaviors if they move out of the camera’s visual and audio range.

Self-Reported Perceptions of Team Cognition

Self-reported perceptions of team cognition are a very common approach for eliciting team cognition. This approach asks team members to self-report their personal perceptions of particular team cognition constructs as they exist or occur within their team. Research has examined perceptions of knowledge similarity (i.e., how similar do you perceive the knowledge to be across team members), perceptions of knowledge quality (i.e., how “good” do you perceive the team’s transactive memory to be), and perceptions of cognitive processes (i.e., do you perceive that your team engages in activities such as sense-making or knowledge sharing). For example, Gevers and Rutte (2006) measured the perceived sharedness of temporal cognitions using four self-reported items tapping the perception of each individual team member that their team had shared cognitions such as “in my group, we have the same opinions about meeting deadlines.”

Self-reports of perceptions are very common within the transactive memory literature. The most common measure used to capture transactive memory systems in research is the Lewis (2003) self-report measure. This measure asks individuals to respond to a set of questions regarding their individual perceptions regarding the specialization (e.g., each team member has specialized knowledge of some aspect of our project), credibility (e.g., I trusted that other members’ knowledge about the project was credible), and coordination of knowledge (e.g., our team worked together in a well-coordinated fashion) within the team from a scale of (1) strongly disagree to (5) strongly agree. This measure essentially is a combined self-report measure regarding the team members’ perceptions of (un)similarity of knowledge (i.e., specialization), the quality of knowledge (i.e., credibility), and perceptions of process (i.e., coordination). Another commonly used measure, due to the reduced number of items, is the self-reported measure of perception of expertise location (e.g., the team has a good “map” of each other’s talents and skills) used by Faraj and Sproull (2000) rated on a scale from (1) strongly disagree to (5) strongly agree. This scale is uni-dimensional and only taps into the extent to which team members are aware of the distribution of specialized expertise and knowledge.

In terms of advantages, using written or computerized self-report measures of perceptions requires significantly less time and resources to analyze compared to communication or observational data. In fact, if designed correctly, self-report measures can be collected and analyzed automatically using computer programs, making this a much smaller investment in terms of time and, potentially, money. There has been some advancement in voice-to-text technology that could be used to try to automatically transcribe team communications in real time, but there is some level of error in such computer programs that can negatively affect results. At the very least, researchers would need to double-check the accuracy of automatic voice-to-text transcripts, which can take nearly as much time as transcribing them in the first place. Organizations and individuals interested in measuring team cognition need to carefully consider exactly how much time and money they are willing to invest.

In terms of disadvantages, it is important to note that these measures capture the extent to which team members perceive the cognitions within the team to have a certain content or structure (e.g., they are similar), and not necessarily the actual structure or content of the cognition within the team. It is quite feasible, in fact, that the team members could believe they have perfectly shared perceptions, when in fact they actually do not. This is important to note, not because it makes the findings of studies using self-reported perceptions of team cognition invalid, but because it changes the interpretation of the findings to represent perceptions of team cognition rather than the actuality of team cognition. Several other scholars have discussed the distinction between so-called “perceptual” and “objective” measures as one of the critical issues within team cognition measurement (Mohammed, Ferzandi, & Hamilton, 2010; Mohammed & Hamilton, 2012; Smith-Jentsch, 2009), and it is worth repeating given that scholars outside of the traditional team mental model framework are still discussing results obtained with self-reported perceptions of team cognition as if they can be assumed to represent knowledge structure (e.g., Zheng, 2012). Future researchers should make a concerted attempt to more carefully select measures based on the match to the measurement purpose, and to more clearly and accurately label and interpret the findings associated with different measurement techniques.

Self-Reported Individual Knowledge

The final data collection approach used in the reviewed literature is self-report of individual knowledge within teams. This is the most commonly used data collection method. Self-reported knowledge measures take several forms including knowledge tests, relatedness ratings, card sorts, concept maps, and in-task probes (each described in detail below), but the general idea is that the measure captures in some form the actual knowledge or cognitions of individual team members with the intent of aggregating this information to represent knowledge structure at the team level. These measures are considered somewhat more objective than self-reported perceptions of team cognition because they require the respondents to directly report their own understanding of the relatedness between pre-established concepts rather than simply asking the respondents if they perceive that their team understands concepts in a similar way. This method is more appropriate for measuring the actual content or structure of team cognition rather than the perceptions of team cognition, but it is also relatively more cumbersome and time-consuming. Most of these measures take up a large amount of team member time and are usually completed near or during team performance episodes, and therefore represent some of the most obtrusive measures in that the team members must actually halt performance for relatively long periods of time and focus their attention on the measure instead of the task.

Relatedness ratings

Relatedness ratings have been one of the most common, and most validated, approaches for measuring objective knowledge via self-report. This approach assumes that knowledge structure can be derived from understanding how various concepts are related to one another. Generally, a set of task-related or team-related concepts are given to the participants and they are asked to rate the relatedness of the concepts to one another. This method is similar to knowledge tests in that participants rate items on a Likert-type scale. The difference is that they are rating the relatedness of a pair of concepts rather than rating their agreement to an individual item. For example, Fisher, Bell, Dierdorff, and Belohlav (2012) measured teamwork-related mental models by asking the members of military teams to rate the extent to which a set of teamwork concepts (Table 2) were related to one another on a scale from (–4) unrelated to (4) related. The responses to these pairwise relatedness ratings were then aggregated into one scale to represent the extent to which team members shared an understanding about teamwork overall. In a different example, Edwards et al. (2006) measured the accuracy of task-related mental models by asking team members to rate the relatedness of all possible pairs of task concepts developed based on the team task called Space Fortress, and then used Pathfinder software to calculate how similar the team members’ mental models were to a predetermined expert (i.e., correct) model.

OPEN IN VIEWER

Table 2:

Teamwork Concepts for Relatedness Ratings From Fisher et al. (2012, p. 17)

1.	Working well together
2.	Often disagreeing with each other on issues faced by the team
3.	Trusting each other
4.	Communicating openly with each other
5.	Agreeing on decisions made in the team
6.	Backing each other up in carrying out team tasks
7.	Being similar to each other (for example, in personality and ability)
8.	Being aware of other team members’ abilities
9.	Treating each other as friends
10.	Being a highly effective team

The primary advantages of relatedness ratings are the fact that they are well-established and validated measures and that they can be used to provide relatively rich data regarding the relatedness between concepts without being overwhelming complex to complete. Past research has primarily used relatedness ratings to examine teamwork mental models and taskwork mental models, but they could potentially be used to study other team cognition constructs as well. For example, situation awareness could be measured using relatedness ratings involving in-the-moment concepts similar to the concepts currently included in situation awareness measures. Rather than simply having the team members provide answers to situation-based questions, they could be asked to rate the relatedness of various environmental stimuli at that particular moment. This may provide a more nuanced understanding of situation awareness not only by examining the team member’s awareness of particular stimuli but by also examining the team member’s awareness of the current relationships between stimuli.

Relatedness ratings could also be used as a technique to gauge the “awareness of expertise location” dimension of transactive memory systems by adjusting the approach slightly from traditional methods. Rather than rating the relatedness of pairs of task concepts, the team members could be asked to rate the extent to which various team members are related to expertise concepts. This could be done both from a self-report standpoint (i.e., I rate my own relatedness to expertise concepts) and from a perceptions of others standpoint (i.e., I provide my perceptions regarding the relatedness of my teammates to expertise) and therefore could be used to ultimately analyze concepts including team knowledge stock, transactive memory consensus, and transactive memory accuracy. This is similar to, but slightly different from, the ranking approach taken in Austin (2003) in that the rank-based measures measure limited responses to the one team member who is most associated with a skill, which may not always accurately represent real-world team composition. It is likely that more complex patterns of expertise exist in most teams. For example, in Table 3, this hypothetical pattern of relatedness ratings suggests that Team Member 3 is an expert only in Skill 1, Team Member 1 is an expert only in Skill 3, and Team Member 2 is somewhere between an expert and a novice across all three skills. In other words, this team appears to be made up of two specialists and one generalist. This measurement approach therefore allows for a more nuanced understanding of a wide variety of expertise distribution.

OPEN IN VIEWER

Table 3:

Example of Relatedness Ratings to Measure Transactive Memory System

	Skill 1	Skill 2	Skill 3
Team Member 1	1	3	5
Team Member 2	2	3	1
Team Member 3	5	3	1

Note. Participants could rate the relatedness of team members to each skill on any Likert-type scale, such as (1) not related at all to (5) highly related.

Content Analysis

As we have briefly alluded to, the primary approach for analyzing textual data, collected via interviews or communication records, is content analysis. This method requires the measurement administrator to either select or develop a theoretical classification scheme into which to code instances of textual data. Content analysis focuses on the linguistic and semantic content of the communication data, such as the topics of discussions, the frequency of certain words, or the types of language used. For example, Banks and Millward (2000) used this approach to code all verbal interactions of teams into nine categories developed to represent various distributed cognitive processes: (1) request, (2) offer, (3) question, (4) propose, (5) hone, (6) support, (7) widen, (8) other, and (9) not relevant. In another example, Bierhals et al. (2007) recorded and transcribed the verbal interactions of teams and then coded the scripts using the KATKOMP categorization system. This system is meant to capture the interactions within the team that deal with the task itself (content), management of intergroup process (process), and interpersonal relationships. Muniz et al. (1996) analyzed communication data for indicators of situation awareness, and categorized utterances as “situationally relevant” and “less relevant” statements with the assumption that more situationally relevant speech would indicate a team with higher levels of situation awareness. In a similar fashion, Ellis (2006) coded speech units into the categories of directory updating, information allocation, and retrieval coordination in order to tap into the team’s TMS.

One specific type of content analysis, latent semantic analysis (LSA; Cooke & Gorman, 2009; Landauer & Dumais, 1997; Landauer, Foltz, & Laham, 1998), is an automated method of assessing discourse content. LSA takes the raw text of communication and separates words by their unique strings of characters. Word meaning is determined by examining how often words occur together in similar contexts (i.e., passages of text) by identifying the latent space of semantic factors, similar to the approach underlying factor analysis. Semantic relatedness is determined by plotting utterances (e.g., words, sentences) in the semantic space and computing the normalized cosine of the angle between those utterances in the space. For an example, LSA can determine if the terms human and user are related by looking at the normalized cosine of the angle between those words in the latent space of semantic factors. One of the advantages of using this method is that the measures are highly related to performance-based scores (some have been as high as R² = .71). Additionally, there has been some recent success in automating communication analysis using LSA (Foltz & Martin, 2009). One disadvantage of this, and other analysis techniques that require textual data, is the requirement for speech-to-text conversion to produce written transcripts. The transcription process is not very automatable at the present time and can hamper real-time analysis. However, LSA indices have been found to be very highly correlated with the simpler-to-obtain word count metrics, which can be coded and analyzed much more quickly.

Pattern Analysis

Pattern analysis is a technique that has been used for several decades to study interpersonal processes, but has only recently been applied to the study of team cognition in particular (e.g., Gorman, Cooke, Amazeen, & Fouse, 2012; Miller, Scheinkestel, & Joseph, 2009; Xiao, Seagull, Mackenzie, Klein, & Ziegert, 2008). In contrast to examining the semantic content of textual data such as interview or communication transcripts (i.e., what is being said) like in content analysis, pattern analysis is focused on examining the sequence and/or pattern of directed interactions between team members within a team regardless of the content of those interactions. One example of this is communication flow analysis (Cooke & Gorman, 2009). This approach uses communication data regarding who has been talking to whom and for how long, but ignores the semantic content of those communications. These data can be analyzed using a variety of indices such as the dominance statistic (i.e., cross-correlations between team members’ speech quantities) to determine who has more influence on the team’s cognition or simple flow quantity (i.e., the amount of speech to and from particular team members).

Sequential analysis is a pattern analysis technique that is focused on examining the temporal sequencing of communications over time. This method has been used to analyze communication flow by examining the pattern of specific communication behaviors and found that poorly performing teams were more likely to follow action statements with other action statements compared to less poorly performing teams (Bakeman & Gottman, 1997; Bowers, Jentsch, Salas, & Braun, 1998). Sequential methods specifically look for patterns across time (e.g., what follows what). This is in contrast with static analyses that aggregate a team’s communication across time using indices such as mean number of words spoken or mean duration of utterances (Kiekel, 2004). The major benefit of looking at communication in a sequential fashion is that it allows for the study of the interactive processes driving team communication, collaboration, and problem solving (Kiekel, 2004).

One disadvantage of sequential pattern analysis is that it can be extremely data rich (Cooke, Neville, & Rowe, 1996) and can be difficult to interpret without automated methods or methods for reducing the data. Several methods have been developed to increase interpretability and lower the effort cost of the analyses. One method analyzes the flow of communication by building procedural networks (ProNets), via the Pathfinder network scaling technique (Schvaneveldt, 1990), that highlight transitions among the events (in many cases, this would be communication turns, with a turn being a single person’s statements or utterances) being studied (Cooke et al., 1996). Pathfinder is an algorithm that looks at data as pairwise, interconnected “nodes” of information. Pathfinder, essentially, calculates the distance between each pair of nodes based on Euclidean distance measurement (Kiekel, 2004). Kiekel (2004) suggests that Pathfinder can be used to analyze the physical flow of communication over time.

It should be noted that because behavioral observation generally results in auditory (i.e., verbal communication) information similar to communication records, it can also be analyzed using any of the techniques used for coding communication data. The observed or video-recorded communications can be coded for knowledge-based or interaction-based team cognition constructs. However, observation data not only provide auditory information but also provide non-verbal visual information that can be analyzed as well. This information can also be coded using content classification schemes that assign particular behaviors to particular categories.

One pattern analysis method that has been used to analyze observational data is what is known as event data analysis (Cooke & Gorman, 2009). Essentially, event data analysis conceptualizes observed behavior as a sequence of events. One previously utilized event-based analysis tool is Targeted Acceptable Responses to Generated Tasks and Events, otherwise known as the TARGETs method (Fowlkes, Lane, Salas, Franz, & Oser, 1994). One of the key concepts behind event-based observation is that particular team member behaviors of interest must occur, often in a particular order, within an operationally relevant task scenario. In the TARGETs method, SMEs are used to develop realistic and relevant task scenarios as well as identify acceptable responses within these scenarios. The identified responses must be specific, observable behaviors so that the observers can easily record when responses occur. Behaviors are then recorded by observers on a “hit or miss” checklist consisting of the identified responses. Observation has been used to study several cognitive constructs, such as situation awareness (Patrick et al., 2006) and transactive memory (Liang, Moreland, & Argote, 1995; Moreland & Myaskovsky, 2000).

As previously mentioned, Pathfinder can be used for Procedural Networks (ProNets; Cooke et al., 1996) analysis, an “integrated exploratory sequential data analysis technique” (Cooke et al., 1996, p. 32). Pathfinder is particularly suited to the analysis of sequential event data because it is more flexible and unconstrained by hierarchical configurations. Pathfinder can be used to analyze sequences of events coded from communication or observational data. The probability matrices of transition among a set of nodes (i.e., events) are input into the Pathfinder Algorithm, and then a network representation of pairwise connections among the events is generated.

Statistical Aggregation

Statistical aggregation is the most commonly used analysis technique, given that nearly all self-report measures must be aggregated in some way to represent the team. The simplest, and most common, type of statistical aggregation is the use of the arithmetic mean of the team member’s self-reported responses. For example, team members are asked to complete a four-item measure capturing their perceptions regarding whether or not mental models are shared within their team (e.g., in my team, we are on the same page), and these individuals’ responses are averaged to represent the team’s score on “similarity.”

Unfortunately, within the literature, approaches like this are often misleadingly interpreted or labeled as representing the extent to which mental models are actually shared within a team (e.g., mutually shared cognition; Van den Bossche et al., 2006), but it should be emphasized that in reality, this aggregate score only represents the average perception or belief within the team regarding the team cognition construct in question. It does not, in fact, measure the content or structure of mental models and calculate the extent to which they are actually similar or different. Beliefs regarding similarity and actual similarity do not always coincide in teams, and this rule applies to beliefs regarding team cognition among other topics. Other statistical aggregation approaches (see Table 4 for summary) generally assess the similarity or difference between team members such as distance scores (e.g., Clariana & Wallace, 2007), agreement scores (e.g., Blickensderfer, Reynolds, Salas, & Cannon-Bowers, 2010), coefficient of variation (e.g., Kilduff, Angelmar, & Mehra, 2000), r_wg (Boies & Howell, 2009), mean Euclidean distance (e.g., Chou, Wang, Wang, Huang, & Cheng, 2008), and standard deviation (Johnson et al., 2007).

OPEN IN VIEWER

Table 4:

Example Statistical Aggregation Approaches

Statistic	Description	Exemplar Studies
Average/mean	Individual-level scores are averaged to determine the team-level score	Chou, Lin, & Chou (2012); Pearce & Ensley (2004)
Standard deviation	The variation or dispersion of individual scores from the average	Camelo-Ordaz et al. (2005); Johnson et al. (2007)
Coefficient of variation	Standard deviations divided by the mean	Kilduff et al. (2000)
Similarity score	The distances between the ratings of each pair of team members are subtracted from the largest possible distance of the scale	Bierhals et al. (2007)
Within-group correlation	A measure of within-group similarity, calculated by comparing observed group variance with an expected random variance	Levesque, Wilson, & Wholey (2001); Thomas (2006)
Higgins discrepancy formula	Discrepancy = (synonymous mismatches + [2*antonymous mismatches]) – synonymous matches	Bushe & Coetzer (2007)
Consensus formula	A complex formula that utilizes a pairwise comparison to reduce the effect of outliers and is normalized for the range of maximum disagreement on any construct as well as the number of constructs considered	Chiravuri, Nazareth, & Ramamurthy (2011)
Mean Euclidean distance	The root mean squared distance between one member and all other member’s scores, averaged to the team level	Chou et al. (2008)
Phi coefficient	The Pearson correlation coefficient between two dichotomous variables	Smith-Jentsch et al. (2001)
Absolute value of differences	The absolute value of the difference between two scores (e.g., an executive’s score and a team member’s score)	Jones (2006); Hambrick (1986)
Sum	Sum of number of own correct responses plus other correct responses	Rentsch, Delise, Salas, & Letsky (2010); Schilpzand (2010)
Cohen’s Kappa	Coefficient of interrater agreement for categorical data	Rau (2006)

Future research should begin to compare the predictive validity of the large variety of statistical aggregation techniques that exist. Meta-analytic findings have suggested that Pathfinder C statistics derived from pairwise relatedness ratings are more predictive than several other measures. However, it is unknown within, say, measures of agreement if r_wg is more or less predictive than standard deviation or vice versa. Because the comparison of various statistical aggregation approaches does not require any extra data collection efforts because the same self-report data can simply be aggregated in multiple ways, the studies could very easily be developed without any extra time or effort necessary. In fact, past research that has already collected self-report measures could be reanalyzed using other statistical aggregation approaches.

Holistic Approach

A newer, and less established, analysis approach that can be used is the holistic approach. Cooke et al. (2007) used what they referred to as a holistic approach to measure the accuracy of team mental models. The team was given the same set of concepts (e.g., altitude, focus, zoom) to rate in pairwise relatedness ratings that they completed as individuals, but were asked to discuss the pairs and determine their relatedness ratings as a group. The concepts rated describe key elements of the unmanned aerial vehicle task undertaken by the team. This means the team ends up with only one set of responses that represent the team’s consensus decisions. Those responses were then submitted to Pathfinder network scaling, so this scoring approach can actually be a step prior to statistical aggregation if there is still a need to calculate accuracy. Cooke et al. (2007) is an example of self-reported knowledge being analyzed using the holistic approach.

Computational Scaling

Another way in which self-report data can be analyzed is via network scaling programs such as Pathfinder and UCINET. Pathfinder, the same program that can also be used for sequential pattern analysis of communication or observation data, has been used more extensively than UCINET for analyzing self-report data, but both should be mentioned as they have both been used in the team cognition literature. Pathfinder has most often been used to develop network structures using randomly ordered relatedness ratings (e.g., Cooke et al., 2003; Marks et al., 2002). In general, Pathfinder takes pairwise estimates of relatedness for a set of concepts and then generates a graphical representation of those concepts and their relations. The results of meta-analytic research has demonstrated that effect sizes between shared mental models and performance are highest when shared mental models are measured using pairwise relatedness ratings analyzed using Pathfinder compared to other methods such as Euclidean distance scores, within-group agreement scores, and other metrics (DeChurch & Mesmer-Magnus, 2010).

UCINET (Borgatti, Everett, & Freeman, 1992) is a similar network analysis program that shows the convergence between two matrices (Quadratic Assignment Procedure correlation). Mathieu, Heffner, Goodwin, Cannon-Bowers, and Salas (2005) used UCINET to create indices of mental model centrality and sharedness. UCINET yielded a centrality index for compared attributes. Mathieu et al. (2005) define this centrality index as “a measure of importance, or interlinkages, of an attribute to the overall network of attributes” (p. 43). This results in revealing network relationships similarly to how multidimensional scaling, Pathfinder, and other related algorithms work. A high centrality index suggests that the attribute is highly related to many or all of the other attributes in the network (e.g., the hub of a spider web). Similarly, the index of sharedness was also derived from the quadratic assignment proportion (QAP) correlation, which was also derived through UCINET. Mathieu suggests that QAP correlations are the equivalent to Pearson’s correlations but between two mental model matrices instead of individual items. This can capture the extent to which team member mental models show similar patterns of relationships. DeChurch and Mesmer-Magnus’s (2010) meta-analysis indicates that this is the only study that used UCINET to represent shared mental models, and the effect size relating shared mental models to performance was smallest for UCINET compared to other structural metrics.

Emerging Methods

Two interesting, but relatively nascent, methods worth mentioning were recorded in our review of the literature: social network analysis (SNA) and eye tracking. Although it is a pattern analysis methodology, researchers have recently begun to utilize SNA and related network analysis paradigms to study team cognition constructs. SNA is based on the perspective that individual behavior can be understood through the context of that individual’s social network (e.g., who they are linked with through communication, behavior, and family or friend circles). Commonly used for sociological purposes, this methodology is a useful strategy for studying social structures. SNA examines the structural factors of these relational ties such as (1) diameter (e.g., how many agents must be included for information to be communicated from one individual to another), (2) the density of the network (e.g., the proportion of relational links within the network and a proxy for speed of information communication within the team), and (3) the centrality of each individual within the network (e.g., how close each individual is to all others within the network). This appears to be both conceptually and computationally similar to communication flow analysis, which we have discussed previously.

SNA’s use in the team cognition domain is still emerging, but it shows potential to be a useful tool for team cognition research. SNA has already been used to study a variety of team cognition constructs including team SA (Sorensen & Stanton, 2011; Stanton et al., 2006), team mental models (Schneider, Graham, Bauer, Bessiere, & Gonzalez, 2004), and knowledge heterogeneity within teams and organizations (El Louadi, 2008). SNA has several benefits as a measure of team cognition: (1) it could allow for a greater understanding of the directionality and shape of transactive memory systems, knowledge structures, and shared SA within a team; (2) it can be used to identify central figures of great importance to team functioning and cognition; and (3) it allows for a holistic view of the studied contributors to team cognition. Although SNA has been used to study several team cognition constructs, it appears to be exceptionally well suited for studying transactive memory systems. Given that TMS relies on the flow of communication about who knows what information, there is great conceptual overlap between TMS and the network perspective. Indeed, the network perspective has been related to transactive memory systems and “social cognition” already (Borgatti & Foster, 2003).

Eye tracking is a common methodology used in human–computer interaction (HCI) and usability studies. Eye tracking, as the name suggests, tracks the eye’s movement and focus on various visual patterns (Hauland, 2008; Underwood & Everatt, 1992). Eye tracking appears to offer great benefits for team cognition researchers: it can monitor exactly what visual cues an individual is attending to throughout a task in a way that can allow a greater understanding of individual (and potentially team) cognitive processes over time. By tracking the eye’s movement and identifying the location of several cues at that time period, what an individual and what the team attends to over the course of a task can be monitored with a minimum of obtrusiveness within a laboratory setting. This added information could enhance the validity and richness of team cognition measures.

Despite its potential, few researchers appear to have used eye tracking as a team cognition measurement method in studies. Hauland (2008) used eye tracking methodology as part of a validation effort on methods for measuring both individual and team situation awareness. This study investigated the individual and team SA of student air traffic control teams in a simulated scenario. In addition to individual SA, Hauland found that team SA could be measured using eye tracking based on the co-occurrence of visual information seeking and acquisition (p. 301). Hauland’s findings suggest that objective cognitive measurements like eye tracking may be useful in simultaneously identifying both individual-level cognition as well as team cognition. For instance, might similarity in eye movements and focus on visual stimuli over the course of a task relate to shared task mental models? There is, however, at least one drawback to the use of eye tracking for measuring team cognition. Although eye tracking can measure the movements of an eye during a task (Smolensky, 1993), it is an indirect measure. For instance, it may be difficult to identify that the mind actually perceived the visual stimulus even though the eye might fixate on it. Future research should investigate the feasibility of including eye tracking and other objective cognitive measures (e.g., EEG, skin conductivity) into team cognition research and what the potential utility of these objective cognitive measurements might be in understanding and validating team cognition constructs.