Leveraging historiometry to better understand teams in context

Challenge #1: Access to real teams…so you mean I can’t go to space?

The literature on teams has repeatedly acknowledged the importance of examining teams in their natural settings (e.g., Salas et al., 2008). However, one of the first hurdles that must be overcome in studying teams in-situ is gaining access, which is often no small feat. In attempting to study teams in context, researchers are competing against real operational demands. For example, Salas, Grossman et al. (2015) highlight a set of practical and logistical issues for measuring team cohesion in military and surgical teams, suggesting that organizations may be apprehensive to grant access to these teams due to the inconvenient nature of administering self-report examinations. Therefore, resources are often constrained not only in terms of the time allotted to study such teams, but also with respect to sample sizes. This, in turn, has implications for the length and timing of surveys as well as the power afforded due to the presence of what is often small sample sizes. Simply put, gaining access is often a barrier to moving forward with the recommendations in the literature to study teams in natural settings or to pay attention to temporal dynamics due to the difficulty for many to gain access. Having a mandate from someone within the organization to study the phenomenon of interest helps, but access is still a challenge.

Given the tremendous amount of historical data now archived on the internet and social media platforms, historiometry affords the possibility for team researchers to gain access to raw data from intact teams embedded within real systems—serving to move beyond the confines of laboratory studies, while limiting the burden on the teams which might otherwise comprise the investigation. The benefits of historiometry are even more apparent when the investigated target is a team that is either rare in occurrence or who operates in mission critical environments where the base rate of failure is low. In these conditions, researchers might wait for long periods of time before they obtain data with the variation needed to perform statistical analysis and inform the research question(s) of interest. Similarly, access is an issue for researchers seeking to investigate team phenomena within extreme contexts due to the dangerous nature of the environments. Extreme contexts have been defined as those in which “one or more extreme events are occurring or are likely to occur that may exceed the organization’s capacity to prevent and result in an extensive and intolerable magnitude of physical, psychological, or material consequences to- or in close physical or psycho-social proximity to- organization members” (Hannah et al., 2009, p. 898). Similarly, Bell et al. (2018), define extreme teams as those which (a) complete their tasks in performance environments with one or more contextual features that are atypical in level (e.g., extreme time pressure) or kind (e.g., confinement, danger) and (b) for which ineffective performance has serious consequences (e.g., compromised health or well-being of the team or the team’s clients).” (p. 2741).

While historiometry affords the opportunity to study teams in real contexts, not all research questions nor all archived data are appropriate. To assist in determination of the appropriateness of the historiometric method and the corresponding archival data source(s), having clearly defined constructs and research questions is essential. Team taxonomies can help to clearly delineate boundary conditions (e.g., Devine, 2002; Sundstrom et al., 1990; Wildman et al., 2011). These boundaries and corresponding exemplar teams not only guide the determination of the appropriateness of using historiometric methods, but will help drive discussions regarding potential historical events that may contain teams with the characteristics of interest and, therefore, guide the search for raw data.

With regard to the types of research questions most appropriately examined utilizing this methodology, Barnes et al. (2018) have argued that archival data is probably best utilized to examine behaviors, outcomes, and dimensions of the context. Research questions pertaining to psychological states may be more difficult to examine using the historiometric method. As historiometry is a special case of archival data analysis the guidance put forth in Barnes et al. (2018) can facilitate determination of the appropriateness based on the research questions of interest. While research questions revolving around behaviors and outcomes may be the most commonly targeted, the contextual richness and narrative nature of many of the source documents typically utilized in historiometric studies have also provided evidence that questions related to affect can at times be examined. Additionally, as the predominant amount of historiometric studies are correlational in nature, due to rare case when there is a natural control group, research questions targeted at determining causality may be best investigated through other mechanisms. However, one of the conditions for causality, temporal precedence, can often be established due to the event or time-based nature of many historical accounts.

In terms of the types of archival data most appropriately used for historiometric studies, archived data that represent a single viewpoint or perspective is less preferred within this method. While demand characteristics are often less given that the source documents were not written to answer research questions but to document a historical event, biased accounts are possible. This concern becomes greater when the team being documented is high profile or from within a small community (e.g., special forces, elite athletic teams, spaceflight crews) and the archived data is being extracted from publicly available sources (e.g., social media, Heng et al., 2018).

Challenge #2: Moving beyond small sample sizes

Related to notions of access is the problem of small sample sizes when studying teams in-situ. Historiometry has the potential to allow the researcher to move beyond the small sample sizes typically seen when examining teams in their natural settings. Simonton (2009) argues that historiometry offers an opportunity to move beyond the normal limitations found within laboratory studies, wherein sample sizes are often limited by the number of students available or the time that can be allotted to complete the study. With respect to team research in general, no matter whether data is being collected the laboratory or in the field, small sample sizes often plague team research. Within the laboratory, team studies often run into problems in terms of scheduling multiple participants to show up at the same time; no shows are a common phenomenon. While team members showing up is not inherently a problem in studying teams “in the wild,” gaining access to large sample sizes is often difficult due to prior team demands and organizational constraints. Given the nature of historiometry, the researcher is not constrained by organizational limitations or the time constraints of the participants. Moreover, as historical records are not constrained to a single time or place historiometry can facilitate cross-cultural invariance and, in turn, open up a larger potential sample size (Simonton, 1998). Another potential benefit for team researchers that may be less prevalent in other archival methods is the presence of multiple viewpoints on a single historical event.

For the reasons cited above, historiometry can facilitate the attainment of larger sample sizes than what are often available when examining teams in-situ as the researcher is only limited by the prevalence of historical accounts of teams operating in the conditions of interest. While most teams research will not reach the level of hundreds or thousands of cases cited by Simonton (1992), it is possible to move beyond what is often seen as akin to case studies when examining teams in-situ. This is especially true if the researcher is able to find events within which teams are embedded that occur on a fairly regular basis (e.g., sporting events, mountaineering, product development, military, firefighting). The challenge lies in identifying the types of teams that one is interested in and then broadening to consider events those teams are most likely to be embedded within that would be historically documented. This documentation may take the form of journaling, biographies, autobiographies, and other similar sources. While it may be possible to find this form of documentation with a variety of team types, this method may perhaps be most easily applied to teams that fall within the frame of action teams (see Sundstrom et al., 2000).

Historiometry can not only facilitate larger team sample sizes as noted above, but also through the flexibility of the sample (Simonton, 2009). In particular, another way to gain power and increase sample size lies within the decision the researcher makes in how to process the raw historical data. For example, DeChurch et al. (2011) and Burke et al. (2018) used raw historical data that was gathered with respect to teams operating in a small subset of events (i.e., hurricane response teams/military provincial reconstruction teams and space exploration/polar exploration/long duration sailboat racing, respectively) to extract critical incidents depicting the processes of interest. By taking an event-based view, the sample sizes were not the number of teams involved, per se, but the number of critical incidents extracted from the raw data. This, in turn, served to increase the sample size. The traditional longitudinal nature of descriptions contained within historical contexts also provides an opportunity to increase power and may facilitate the ability to look at research questions from both a within and between design perspective. In essence, this method facilitates comparison within a single team over time, as well as between teams.

Challenge #3: The study of teams over time

McGrath (1991) lamented on the lack of attention to the temporal aspects of teams in group research and this criticism remains true today, almost 30 years later (Mathieu et al., 2008). One of the main drivers for studying teams in-situ is the potential to examine the temporal aspects of team dynamics as truly investigating temporal dynamics over any real meaningful time period is difficult within a laboratory setting. While the potential exists to capture the impact of time, research is still plagued by single unconnected snapshots of processes and states. Constructs are often captured at a fairly global level, due to operational constraints, which ensure that the research being conducted does not interfere with the teams’ operational tasks. The bottom line is that the study of teams over time remains a large challenge in team research, whether in laboratory environments or in-situ (Cronin, 2015).

Emerging technologies such as wearable sensors, sociometric badges, and data logging capabilities are making strides in the ability to collect data over time, and often in a manner which is less obtrusive than traditional self-reports. However, this is another area where historiometry as applied to archived historical textual data adds to the methodological toolbox of team researchers. Parry et al. (2014) note that the collection of longitudinal data is an area where historiometry is particularly useful. While many forms of archived data are longitudinal in nature, by its nature, historical data is often presented not only over time, but the temporal periods are often explicitly delineated making analysis easier. It has been argued that “historical narratives are likely to provide detailed descriptions of phenomena of interest as they occur through a target’s history” (Crayne & Hunter, 2018, p. 11). Often the longitudinal nature of such data is only bounded by the natural length of the event (e.g., team’s life span, length of the event that teamwork is embedded within). This is in contrast to the bounds often placed on longitudinal data collected in context by other methods, due to the constraints surrounding sampling with teams operating in-situ. The collected longitudinal data can then be subjected to statistical analyses commonly utilized with longitudinal data (e.g., time series analysis, see Velicer & Plummer, 1998).

A further advantage to utilizing this method in collecting longitudinal data is the fact that, due to the archived historical nature of the data, there is no artificial interruption of the workflow for the researcher to collect the data (Simonton, 2009). The debate as to whether to interrupt team workflow to capture data “in the moment” versus retrospective data collection is an ongoing debate within the literature. However, historical data can provide the best of both worlds as the archived records often have a mixture of both retrospective data (e.g., biographies, autobiographies, interviews) as well as near real-time data (e.g., journaling, blogs). While some of the retrospective data may be done long after the event (e.g., biographies, autobiographies) other forms are more proximal (e.g., interviews, newspaper accounts). While all accounts may have bias, the further in time from the actual event the more likely that recollections may vary from the ground truth in the moment. While common in many forms of archival data analysis, not just historiometry, this hindsight bias can skew the recording of historical and important events (and therefore the interpretations extracted). This points to the importance of not only collecting data from multiple sources (e.g., people and formats), but attempting to balance the temporal lag between the event and its reporting. The fact that historiometry can provide a mixture of accounts that vary in their recency is a general benefit of this method that applies to, but extends beyond, the collection of longitudinal data on team process and states.

Challenge #4: Moving beyond survey data

Similar to the state of affairs with regards to longitudinal research, there have been repeated calls for team researchers to expand their methodological toolbox beyond the use of survey data using Likert scales. The relative ease with which self-report data is collected and the difficulty of obtaining other types of measures has caused little movement in this regard. While moving beyond the primary use of self-report surveys applies to studying teams both in the lab as well as in-situ, it becomes even more challenging in the latter case. Particularly when collecting data in the field, researchers are ever conscious of balancing trade-offs between the psychometric properties of the surveys with the time allotted to collect data. Often researchers must choose a satisficing strategy such that data can be obtained, albeit perhaps not in the detail that is desired, by using shorter scales and/or assessing fewer constructs. While survey fatigue is always an issue, fatigue and annoyance are never more prevalent than when collecting data on teams embedded within organizations where survey completion is taking time away from task completion. This is in contrast to individuals who are expecting to be surveyed as they have signed up for a laboratory study outside the context of their day-to-day job.

Facilitating the expansion of the methodological toolbox for team researchers are recent advances in both technology and statistical analysis that have begun to open up the range of possibilities available to those studying teams. For example, several researchers have begun to investigate the use of sociometric badges to capture aspects of team dynamics (e.g., Kim et al., 2012; Santoro et al., 2015). The use of historiometry can also contribute to move the field toward novel and technologically-driven data collection strategies. Recent technological advances have opened up a wide variety of data collection sources that did not exist previously—allowing the possibility of a more robust data set, due to multiple types of data sources. For example, data sources for historiometric efforts have traditionally relied on archival documentation written by historians, biographers, or archivists (Simonton, 2003). Recent advances in the use of social media have opened up more avenues through which data can be collected unobtrusively (e.g., blogs, electronic journals/diaries, social media posts). Additionally, advances in machine learning can facilitate the processing of journals and other more traditional historiometric sources (e.g., Driskell et al., 2018; Pennebaker et al., 2015). Finally, sources such as journals and blogs are increasingly easily accessed and publicly available due to the prevalent use of technology and the internet. In some cases, this is true not only of the qualitative aspects of historical texts, but the quantitative aspects that may speak to effectiveness or performance criteria (e.g., statistics available on sports teams)

While the raw data sources used within historiometry are often still self-report or first-hand accounts of an event, these accounts are often less suspect to potential bias’ with regard to the research questions being investigated, in the case of teams these questions most often revolve around team dynamics. This is largely due to the fact that the textual documentation typically used as the raw data source(s) within historiometry is not captured by the researcher, but by a third party who did not document team interactions for the purposes of research, but rather for the sake of recording the nature of the events at a point in time deemed important. Additional guards against bias, especially in the application to team research, can be found by comparing accounts made by different individuals experiencing the same event.

While the fact that the textual, archival documentation is not written for the researcher’s specific question in mind is a potential strength of this method, it may also pose a challenge. Specifically, the researcher may suffer through false positives in the search for documentation that contains information to the level of depth and detail to answer the research question of interest. For example, in examining team leadership in the context of polar exploration teams, researchers might use abstracts to select for a biography in the hopes that it may contain information concerning team interaction, but in reality, find that the predominant amount of textual information within the document may pertain to the design of the vessels used on the voyage or the individual explorer’s journey without reference to team interaction. However, this is not that different from the process of conducting meta-analytic work, where at a first pass an article looks like it may meet the criterion for inclusion, but upon closer analysis, once the article is in hand, the statistics reported are not amenable to meta-analytic work or it fails based on other rejection criteria.

Challenge #5: Building contextualized theory

Another area that has been repeatedly recognized within the team literature is the importance of the sociotechnical system within which a team is embedded in driving an understanding how team dynamics unfold. This points to a need to not only understand the principles which are generalizable across teams, but the places where the team’s embeddedness may drive nuances in how team processes and states are operationalized and the impact they have on team performance. While collecting data on teams in operational settings is one way to build contextualized theory, capturing this information in real time is often difficult. As events unfold, it is often difficult to fully understand the impact that the sociotechnical system in which the team is embedded has on team dynamics. However, similar to watching an instant replay in sporting events, important cues are often easier to define once an individual is slightly removed from the immediate situation. In this vein, not only does historiometry provide another means to study team interaction as it naturally occurs, thereby providing context-relevant perspectives, but it does so in a manner by which related data on features within the sociotechnical system may be collected, coded, and quantified.

Given that one of the defining characteristics of historiometry is that events and the individuals involved have been deemed important enough to be recorded for the purposes of preserving the experience, rather than for the purposes of scientific analysis, information regarding contextual features is often embedded within the archived textual documents. For example, in examining research questions regarding leadership and team roles within spaceflight crews, we were able to identify many ways in which the sociotechnical system the crews were embedded within interacted with team (e.g., crew size, crew tenure, crew composition) and mission characteristics (e.g., mission duration) to influence team dynamics. In other instances, contextual details, such as degree of membership change, member familiarity, stressors facing the team, as well as many individual and compositional characteristics were able to be extracted.

Similarly, Simonton (2009) argues that one of the methodological advantages that historiometry provides is the ability to identify and subsequently code situational variables that provide further understanding of the phenomena under study. This advantage is primarily due to the rich level of detail provided in the types of qualitative sources used in historiometry. Given the richness of this data, situational variables are often picked up within the larger narrative that may not be apparent when looking at processed data from these sources. The ability to pick up this type of data is a general strength of many qualitative investigations over those which are purely quantitative in nature. Historiometry provides a mechanism through which both quantitative and qualitative approaches can be combined and supplement one another to provide a richer source of interpretation.

Define constructs and research questions

As with most research endeavors, one of the earliest action steps entails defining the constructs of interest and corresponding research questions. The process of defining the research question and corresponding constructs of interest is no different here than in any other domain. In essence the choice of research question is informed based on knowledge of the literature and corresponding gaps. For example, in previous research we were interested in the degree to which specific team leadership behaviors, as argued for by Morgeson et al. (2010), were aligned with action and transition cycles and the form of leadership through which they were enacted. Furthermore, we were interested in how the above aspects of team leadership would occur within the context of teams operating in isolated, confined environments (Burke et al., 2018). Once we had our research question, we defined our constructs of interest as guided by existing literature. Later the exact operationalization of each construct was refined based on the context within which the team was operating.

Once an initial research question was determined, we used the guidance presented earlier in the manuscript (see information regarding applicable research questions discussed within Challenge 1) to determine the degree to which we felt historiometric methods were a viable approach. For example, did we believe that there were historical texts regarding teams operating in isolated, confined environments that would provide insight into team dynamics and leadership? Or in later work where we were interested in team resilience, were there archived documents that would provide the temporal durations that would facilitate examining team resilience as a process?

On a related note, there have been recent calls to pre-register hypotheses, study design, and analytic techniques prior to engaging in archival research methodologies such as historiometry to ensure scientific rigor and facilitate replicability (see Heng et al., 2018 for more detail).

Investigative piloting

Investigative piloting is similar to piloting in more traditional experimental research, as essentially the researcher is engaging in a “proof of concept.” At this point, the researcher, driven by the research question(s) and construct definitions, begins to gather potential material. For example, in conducting our study on team leadership in extreme teams, it was at this point that we began to explore databases to identify possible sources (Burke et al., 2018). Prior to this point, we had decided, based on our focus on extreme teams, that contexts such as spaceflight, polar exploration, and long duration sailboat racing would be our primary focus. Thereby, databases such as Amazon and Google were searched with keywords such as: space exploration, spaceflight, as well as more specific instances of space exploration (e.g., International Space Station, Mir, Skylab); polar exploration was also searched, along with specific instances of polar explorers and polar expeditions. A similar process was utilized for long duration sailboat racing.

Once a small set of sources has been identified, the next step is to begin to examine them to ensure the phenomenon of interest is likely to be contained therein with enough contextual detail as to be coded. In our research, this has been a two-tiered process, whereby the abstracts or descriptions on the jacket covers of the book were first examined. Sources which made it past this first level of screening were then obtained, and the actual historical text (e.g., biography, autobiography, journal, blog, memoir) was examined at a high level to ensure that within the larger narrative there were: (1) descriptions of team interaction and (2) enough detail regarding the team interaction that the particular constructs of interest were able to be identified. Many of our initial sources did not meet the criterion for inclusion once examined at this level of detail—most commonly due to a lack of detail (e.g., predominantly image-based books regarding the expeditions) or based on contextual detail regarding the wrong targeted phenomena (e.g., describing the environment, or in the case of polar exploration, focusing on describing the dogs or ship, but lacked detail regarding team interaction). During this process the researcher not only gets a rough idea of the probability that the research question of interest can be examined through historiometry, but will also gain insight into the types of sources that will provide the richest data. In essence, trends in data sources will be seen that will help to guide future source identification.

The importance of this step cannot be emphasized enough as the historical texts were not written for research purposes in mind. So, while investigative piloting is common in many forms of research including laboratory experimentation and traditional qualitative research, within historiometry the historical nature of the archived documents makes this step a little different. At this step, the researcher also begins to identify data sources that may describe the same event or interaction from different perspectives, which, in turn, will later assist in the triangulation process. Most likely at this stage, potential triangulation points will be at the event level, until the researchers get into the archived texts in more detail later in the coding process.

Decisions on data structure

Decisions regarding data structure tend to involve specification of how the data will be captured. Crayne and Hunter (2018) argue that data captured most traditionally takes the form of either event-based or chapter-based perspectives. While event based perspective are organized around specific events and their surrounding context, chapter-based perspectives look at the naturally occurring structure to determine critical time periods most relevant to the research question. In our previous research, we have primarily structured data around events, as it not only provides the potential for a larger sample, but often provides a finer grained analysis than taking a chapter perspective. We have also found that taking an event-based perspective is useful for examining team interaction in particular, as it serves to ground the interaction and events may often be compared across time. For example, in recent work that we have been conducting on team resilience, our data is structured around trigger events for team resilience (as guided by the work of Alliger et al., 2015).

It is also at this stage where the researcher should think about the approach that will be taken in regard to the data and research design. Based on the research question(s), the researcher will focus on structuring data collection such that a within-subjects, between-subjects, or mixed design is created. This is important as it will guide the sampling plan. For example, if one is more concerned with investigating patterns of particular teams over time (i.e., within-subjects design), then the sampling approach should put more weight on equally sampling teams throughout their lifespan. Conversely, the sampling approach for one interested in discerning differences between teams (i.e., between-subjects design) will focus on a greater breadth of sampling during a particular period of time.

Prototyping and codebook drafting

This action step pertains to the development of the codebook which includes the structure of the data to be extracted from the larger raw source material as well as the operationalization of all the key variables of interest. At this point, the research team who has begun to examine the potential material begins to establish operational benchmarks that illustrate different levels of key constructs (e.g., high, medium, low). Conversely, the researcher can identify potential established measures to be used for the coding of extracted data. Either way, once the codebook is drafted, it should also be piloted to ensure that it is clear and that it leads to the data being captured in the level of detail and format expected.

Within some of the earlier work that members of our research team conducted employing historiometry, we extracted critical incidents regarding the targeted construct(s) of interest (e.g., team leadership, team roles). On these projects, the codebook and consequent piloting was guided by the work of Flanagan (1954). More recent work on resilience has required us to adapt the critical incident methodology based on our research questions, which are a bit more detailed. In essence, our codebook specifies the format of each of our extended critical incidents. Specifically, critical incidents for team resilience are grounded by the resilience trigger. As such, the extended critical incident contains information on the context that the team is embedded within when the triggering event occurs, description of the triggering event [“the what”], the behavioral and/or attitudinal response to the trigger [“the how”], and the consequence of the action.

Event/chapter selection and dissemination

Woods (1911) considered the historical records at the break of the 20th century to be considerably vast, yet one could argue that it may pale in comparison to the amount of information available to researchers today. Hence, it should come as no surprise that the filtering of potential sources is emphasized. Below, we briefly highlight strategies that assist in the screening and dissemination of potential sources. In doing so, first we highlight those strategies that may be most suited when employing historiometry with respect to teams and teamwork. We then conclude with a few other strategies that are viable to all types of research, including teams.

With respect to the selection and dissemination of potential source material, the following guidance is offered. For one, materials should be examined to ensure that they focus on teams and teamwork processes. Depending on the research question, historical narratives focusing purely on the context within which the teams operated, but lacking description of the actions and consequences of teamwork need to be filtered out. For example, in our attempt to understand the nature of resilience in extreme environments, we gathered sources that we believed would describe teamwork efforts. One of these sources were narratives describing the events that unfolded for wildland firefighting teams (e.g., Mann Gulch fire). While these sources painted a vivid picture of the challenges these teams faced, as well as some of the actions they took against insurmountable odds, the narratives did not describe the actions that discernable teams took. Instead, they described actions that particular individuals took and, consequently, information concerning resilient teams could not be extracted. Additionally, the descriptions of action were of short temporal duration and, as such, were not the ideal for work on team resilience.

Relatedly, another filter may be applied to historical records coming solely from the perspective of the leader. As alluded to above, historical narratives often focus on important individuals and their perception of important events. With respect to teams, these historical narratives often focus on the perspective of the leader and their interpretation team experiences. Extant literature from the team domain suggests that one be cautious in interpreting information provided solely from the leader, as team members may hold different opinions on the events that have unfolded (e.g., Cogliser et al., 2009). Depending on the research question, leader-sourced material may simply be a limitation of the efforts, as opposed to an exclusionary criterion. For instance, Burke et al. (2018) gleaned information about leadership functions from narrative descriptions of the Ernest Shackleton’s exploration efforts. In this case, information gleaned from the Shackleton-focused sources were supplemented by additional data that did not come from the team’s leader.

While the selection of sources for team research may have special considerations, there are also recommendations related to how to select these sources which generalize across a variety of levels (e.g., individual, team, organization) and research questions. We next discuss these general recommendations. In this vein, within historiometry, it is argued that researchers should have multiple team members examine potential sources to ensure that the content is relevant to the research question; this may also assist in reducing selection bias. Given the contextual richness and variety of sources used in historiometry, it is crucial that coders approach source selection (and coding) from the same frame of reference. Indeed, these coders should be trained to the point that they are considered subject matter experts for the research and source in question. That is, these coders should be able to understand phenomena under investigation as well as the jargon of the context in which the narrative took place. For instance, in coding ocean sailboat racing narratives, a coder should be able to understand that a tack refers to a collective effort by the team to change the direction of a boat. Essentially, coders should go through training, developed in conjunction with subject matter experts, that is tantamount to frame-of-reference training (e.g., Gorman & Rentsch, 2009). This type of training helps to eliminate various cognitive biases that may lead to errors and discrepancies (e.g., Aguinis et al., 2009). When discrepancies enviably arise, they should be thoroughly discussed until a consensus is reached.

Thus far, we have discussed steps with respect to evaluation of the content. However, it is important to also evaluate how sources are obtained, maintained, organized, and disseminated. It is crucial that researchers devise a plan with respect to each of these factors as failures in any will produce, at minimum, decreases in efficiency. We believe, whether physical or digital, sources should be stored in a single digital repository to which the entire team has access. In our case, we often name the source files based upon the event they are describing, as there are multiple sources providing different perspectives on the same/similar event. With respect to the filtration process, it should be absolutely clear who is coding which material. We often accomplish this by directly articulating to our coding team who is coding what, providing a document with this same information, and instructing coders to put their initials on the files that they are filtering.

Coder training and protocol execution

With respect to the training of coders, the process is no different than is commonly seen in the coding of team communication within laboratory studies or the coding of qualitative data outside the context of the historiometric method. That is, the rating team is familiarized with the goals of the research and the codebook, frame of reference training and practice sessions are conducted, and feedback provided. Within this stage, the researcher is working to develop shared mental models in terms of the constructs of interest, their operationalization within the current study, and the use of the codebook. In the research that we have conducted using historiometry, we tend to have two separate sets of coders, and therefore, conduct two different sets of training. The first set of coders serve to extract the information that is related to the research question(s) from the larger corpus of raw materials. For our purposes, this set of coders receives the above-mentioned training, but also receives training in the critical incident method of data extraction (see Flanagan, 1954).

With regard to protocol execution, much of this revolves around ensuring that protocols are in place and timelines are established such that the conditions minimize the likelihood of coder fatigue. As part of this process, regular checks on interrater reliability and consensus meetings serve as potential markers as to when coders may begin to experience fatigue. We have also conducted periodic consensus meetings whereby we ensure that everyone on the coding team is on the “same page” to minimize rater drift over time.

Data analysis

Within this step the final dataset is organized to permit the qualitative and quantitative analyses as dictated by the hypotheses and your research plan. The historiometric method can produce various types of data, the format of which will be dictated by the hypotheses researchers generate. Although influenced by the researcher’s hypotheses, the scope of the data analysis techniques that can be applied to historiometric data is boundless. For example, data analysis techniques commonly used within historiometric studies have included: linear regression, ANOVA/ANCOVA, and hierarchical linear modeling. In addition to these common analyses, researchers can also leverage the use of time series/longitudinal data analysis methods due to the temporal nature of the archival historical documents that are traditionally utilized as raw material within this method (Velicer & Plummer, 1998). This is indeed a potential strength of this methodology as time series data that covers a meaningful timeframe is often difficult to obtain from teams operating in context. Further, the flexibility of the data also affords the possibility of using text-based analyses/language processes techniques (e.g., Pennebaker et al., 2015). We recommend using these techniques when the data come from direct quotes of the sources.

It should be noted that the use of various analytic techniques is expanding more so with the advent of certain technological developments. For instance, we are currently in the process of gathering videos from long duration sail boating races, which we will analyze in various ways. These videos directly capture crew interactions at various periods of time. Once gathered, we will then code them for behavioral and affective team actions and utilize this to supplement other data used in our historiometric analyses (e.g., crew blogs, skipper reports). This multimodal data collection/analysis strategy allows us to produce a more comprehensive picture of the phenomenon in question.

Integrating quantitative analysis with qualitative data

The final action step once the quantitative results are obtained is working to use associated qualitative information to engage in storytelling with the reader. In this case, the qualitative data can be used to help explain the practical meaning of the statistical analyses and provide practical examples to practitioners. A number of excellent texts exist which specify the aspects of good qualitative data and corresponding reports (e.g., Guest & MacQueen, 2008; Wilhelmy, 2016; Townsend et al., 2016).