Historiometry in Organizational Science

Overview of the Historiometric Method

Pioneered by the work of Dean Simonton (1988, 1990, 1999, 2003, 2009), studies referred to as HMA have been most prominent in the social psychology literature, in which Simonton remains a vocal advocate for the method’s usefulness (1999). In social psychology, HMA has been used to demonstrate the conditions under which authoritarianism is likely to emerge (Doty, Peterson, & Winter, 1991; McCann, 1999), the role of mass media in violent crime (Phillips, 1986), and the influence of leader charisma (Deluga, 1997; Simonton, 1988). The success of this method within a closely related discipline suggests that similar results would be likely in organizational science; however, HMA is rarely seen outside of isolated studies or niche research areas. Leadership research has acted as a trailblazer for HMA in organizational science, led by the efforts of Mumford (e.g., Mumford, 2002; Mumford et al., 2005), Yammarino (e.g., Yammarino, Mumford, Serban, & Shirreffs, 2013), and their students (e.g., Bedell, Hunter, Angie, & Vert, 2006; Hunter, Cushenbery, Thoroughgood, Johnson, & Ligon, 2011). Aligning with Simonton’s (1999) view that HMA is useful for the study of “eminent individuals,” leadership researchers have found the method applicable to a wide variety of research questions. For example, HMA has been applied to studies of destructive leader behavior (O’Connor, Mumford, Clifton, Gessner, & Connelly, 1995), leader assassination (Yammarino et al., 2013), team leadership through crises (DeChurch et al., 2011), Machiavellianism (Bedell et al., 2006), and tests of emerging leadership theories (Hunter et al., 2011).

Functionally, HMA requires the acquisition, assessment, and coding of narrative historical sources (e.g., academic biographies, periodicals, aggregated corporate histories) in relation to phenomena of interest. The goal of the method is to allow the information within these texts to “speak,” in the words of Bailyn (1970; Walsh et al., 2015), and provide the researcher with insights that can be quantified and analyzed statistically. The approach to narrative interpretation in HMA relies on a holistic exploration of the text, where deep analysis of context is used to develop an understanding of phenomena of interest. A team of highly trained coders is employed to assist in the development of a comprehensive coding schema, acquisition of research materials, and content review. Resultant numeric data, in the form of both continuous and categorical codes, are then subsequently cleaned and organized for analysis. Historiometricians test these data in accordance with theoretically grounded models, developed a priori, most often in a confirmatory analytical style. However, the flexibility inherent in both HMA sources and data quantification processes makes HMA useful for more exploratory research approaches as well.

HMA as a Content Analysis Package

Despite a rich history and named application in published social science, the research approaches and activities that compose HMA can no longer be described as definitively unique to that method. The precipitous growth of text-based research methods has given rise to the more global categorization of these techniques, and others related to them, as content analysis. The term content analysis is broadly applied to describe “any methodological measurement applied to text (or other symbolic materials) for social science purposes” (Shapiro & Markoff, 1997, p. 14). However, the breadth of this definition for content analysis and its applications may give prospective researchers pause, despite the presence of robust instructional guidelines (e.g., Krippendorf, 1980; Neuendorf, 2002; Weber, 1985) and an established literature on how to conduct content analytic research. In order to minimize this issue, researchers may be well served by guidance on the implementation of research techniques that are most useful for specific purposes or under specific conditions.

Such guidance may come by way of “packaging” content analytic techniques together based on the goals of their application, allowing researchers to more readily adopt an established group of techniques for use in a study and to more easily compare studies employing the same set of methods to one another. With these considerations in mind, we position HMA as one such content analysis “package”—a series of established techniques and procedures that differ in both application and execution from others under the methodological umbrella of content analysis. The manner by which the procedural details of HMA depart from alternative approaches to content analysis are important for prospective researchers to understand, as the nuance and rationale behind those details work to define how, when, and why HMA may be the most appropriate set of methods to employ.

For example, content analysis often relies on source content that was developed by the organizations or individuals of interest directly, serving as a primary-source account of the entity in question (see Duriau, Reger, & Pfarrer, 2007, for a discussion). This trend is likely due to the prevalence of a text-as-data analytic strategy, common in content analysis (see Langley, 1999; Sonpar & Golden-Biddle, 2012), which implicitly requires a closeness between the author of the source content and the event or phenomenon under study. The approach of HMA relies instead on sourced, aggregated narratives, most frequently developed by external third parties (e.g., academic biographies, research reports, historical journalism). This methodological distinction is due to HMA’s specific focus on inferring the presence and prominence of constructs (e.g., personality traits) from a holistic interpretation of the narrative rather than the use of the text itself as data. Such an approach requires that historiometricians be particularly attuned to the possibility of author biases, which are likely to color the perspective provided by internally produced sources. In their study of 98 content analysis papers in organizational science, Duriau and colleagues (2007) warned that internally produced source materials were likely to contain “self-serving attributions and attempts by managers to influence the impressions of external stakeholders” (p. 14). Although completely eliminating the possibility of impartiality in these sources is not realistic, a reliance on well-sourced, contextually rich qualitative data from reputable third parties and an attention to controlling for biases has been demonstrated to mitigate its influence over HMA results (Ligon, Harris, & Hunter, 2012; Mumford, 2002).

In addition, the treatment and analysis of qualitative information seen in HMA studies differs from that of more commonly employed content analysis techniques. As referenced previously, HMA calls for an approach to converting qualitative information into numerical data via a holistic and context-driven interpretation of a text-based data source. Relying on clear, detailed, and theoretically sound operationalizations of variables of interest, researchers conducting HMA train coders to capture estimates of variables from the story told by the text as opposed to the text itself (Simonton, 1990). Research has demonstrated the efficacy of this method in measuring a variety of psychometric constructs, such as power (O’Connor et al., 1995), Machiavellianism (Bedell et al., 2006), narcissism (Deluga, 1997), and intelligence (Simonton, 2009).

This “judgmental approach” (Parry, Mumford, Bower, & Watts, 2014) to qualitative data is not new in the content analysis literature or necessarily unique to HMA (see Krippendorf, 1980). However, within modern organizational science, the majority of studies employing content analysis do so through alternative, text-as-data quantification strategies. Of these, the most commonly employed method is frequency counting, in which the researcher identifies and counts the incidence of words in the text that indicate a variable of interest. Although frequency counting has been used in some HMA studies (e.g., Spangler, Gupta, Kim, & Nazarian, 2012), this application is exceedingly uncommon. In contrast, nearly 85% of content analysis studies identified by Duriau and colleagues (2007) employed frequency counting as their primary method of relationship testing. Such methods have a number of analytical advantages (e.g., expediency, coding consistency, simplified data management); however, they may make incorporating the influence of context or applying theoretical operationalizations of constructs to measurement more challenging. Indeed, Langley (1999) viewed such quantification strategies as ironic, as researchers take rich qualitative data and “rush to transform it…into a much thinner data set that can be managed in traditional ways” (p. 698).

The package of analytic techniques that compose HMA may be more appropriate than alternative methods, such as frequency counting, depending on the researcher’s goals and the relationships of interest in the study. However, a researcher’s ability to identify such techniques and their application to a research project is limited by the individual’s understanding of the breadth of content analytic methods and the presence of such methods in the literature. For this reason, we believe it is beneficial to highlight combinations of analytic techniques that have demonstrated usefulness when employed concomitantly and to extrapolate on the particulars that govern the successful application of those methods. In this way, with regard to HMA, we acknowledge the literature that positions HMA as a unique methodology (e.g., Mumford, 2002; Simonton, 2009; Woods, 1909) and well as the more encompassing content analysis literature (e.g., Krippendorf, 1980; Neuendorf, 2002) in an effort to provide specific guidance to future scholars.

When Is Historiometric Research Most Useful?

Although HMA is an applicable methodological option for nearly all research questions that can be addressed with text-based or otherwise qualitative data sources, there exists a small subset of those questions which are particularly advantaged by the use of HMA methods. Below, we elaborate on three research scenarios for which HMA is particularly well suited.

Scenario 1—Analysis of unique or rare samples

Originating from early studies of unique samples (e.g., Galton, 1869), HMA is especially adept in the study of interesting and rare populations (e.g., leaders, artists, innovators). This has been noted by Simonton (1999), who viewed HMA as the ideal method for studying “eminent individuals” such as American presidents (see Simonton, 1988). This unique applicability of HMA to rare populations has likely been the impetus for its presence in leadership research (e.g., Hunter et al., 2011; Ligon et al., 2012; Simonton, 1988). Leadership research is often methodologically hindered by an inability to access and directly observe the population of interest (Mintzberg, 1975), a limitation that is difficult to overcome in the field or the laboratory. Because detailed life histories of leaders across a variety of fields are widely available through academic biographies and other writings, leadership scholars who employ HMA are provided relatively unfettered access to representatives of the population they are interested in.

The sampling of unique populations is not, however, restricted to the study of leadership. To the contrary, a number of organizationally relevant research questions are likely to be best addressed when deliberately framed through and tested with a highly specific sample. For example, Mumford and colleagues (2005) were interested in understanding whether or not discernable patterns existed in the career experiences of high-performers. Developing a perspective on this research question required the use of a fairly unique sample—noted high performers who had reached the end of their careers. Although the researchers might have had success in conducting interviews with these individuals or analyses of their self-reported data, identifying such a specific population of participants would be exceedingly difficult. The researchers instead decided to conduct an HMA on the detailed obituaries of prominent scientists who were deceased, as these narratives would provide information about the individual’s life, major experiences, and ultimate career achievements (Mumford et al., 2005).

One potential argument against this technique is that inferences developed from the analysis of these specific samples may not generalize to the broader population. Although historiometricians have acknowledged this possible limitation (see Parry et al., 2014), we propose two reasons for why this concern may be overstated. First, research has demonstrated that conclusions drawn from HMA studies employing such populations are similar to those from studies using more traditional samples (Simonton, 2009). Second, the validity of a research effort is not necessarily defined by its generalizability to the entire population, but by its efficacy in achieving its own objectives. To that end, the objective of some research efforts may not to be generalizable but to draw conclusions about a research question in relation to a specific population or context (Mook, 1983; Parry et al., 2014).

In addition, studies of unique or rare populations may be more likely to suffer from a restriction of range problem than studies of more accessible populations. Conceptually, as a population becomes more unique in its characteristics, it is also likely to become more homogeneous. This may restrict a researcher’s ability to find results for phenomena of interest statistically, as entities within the sample are too similar to one another and there is little variation between them. In order to combat this, researchers should attempt to maximize between-subjects variance when exploring highly specific populations.

This effort begins primarily with awareness that such issues can arise, and specific attention to the theoretical and practical rationale that governs a researcher’s search for and selection of data sources. Researchers should make concerted efforts to acquire sources describing a population that is reasonably expected to exhibit a construct or phenomenon of interest rather than those that are understood to be examples of that construct or phenomenon. For example, a researcher interested in the relationship between narcissism and decision riskiness might choose to assess this relationship via a sample of historical leaders. In order to maximize between-subjects variance, the researcher should develop a sample from the large and diverse population of well-documented historical leaders, perhaps deriving leader choices through a list established in previous research (e.g., Bedell et al., 2006), rather than looking for a specific population that is likely to exhibit narcissism (e.g., politicians).

A second strategy is to assess any possible restrictions of variance statistically. As materials are being coded by the research team, it is recommended to conduct periodic checks of interrater agreement and consistency. Researchers can use these checks as opportunities to examine the descriptive statistics for coded variables, and identify any patterns that may indicate a possible restriction of range or variance. Should such an indicator be present in the data, the researcher must revisit and amend the sampling strategy accordingly.

Scenario 2—Measurement of context and situational specifics

A potential limitation of many studies in organizational science is a lack of attention to contextual or situational influences on phenomena of interest. This is due in large part to difficulty in accounting for elements of context in real time, such as during a study of participants in the field. In a laboratory setting, where environmental conditions are predetermined and controlled, context may be easier to quantify in some ways. However, any extant contextual differences in these cases would be the result of researcher manipulation and are likely to be useful only for group comparison tests.

A core methodological advantage of content analysis is the ability to readily identify and code for such situational variables and elements of context (see Krippendorf, 1980, p. 59; Tracy, 2010). This is due to the descriptive richness inherent in narratives and other qualitative sources, which provides researchers with more overt examples of such constructs than might be readily apparent in self report data or observations in the field. For example, researchers interested in understanding success in organizational change initiatives may naturally gravitate toward conducting research in the field and observing change as it occurs. However, capturing elements of change in real time limits the researchers’ ability to understand the change within the broader context of the organization. This is important, as certain contextual or situational variables may be theoretically or functionally likely to influence the success of a change program. Moreover, indicators of change success may not manifest for long periods of time, or manifest in ways that are not directly tangible. However, the text-as-data approach that currently pervades much of the qualitative research in organizational science places limitations on how such variables can be assessed (Sonpar & Golden-Biddle, 2012). Alternatively, the techniques that constitute HMA endeavor to draw inferences from systematic interpretation of an entire narrative rather than using the text itself as data. The result is potentially richer perspective of the phenomena of interest than could be captured in the field or through more commonly employed content analysis strategies.

Assessments of context can be useful for categorizing the research sample and creating comparison groups (e.g., O’Connor et al., 1995), as well as for identifying variables to be included as controls or moderators to a hypothesized model. However, the difficulties inherent in capturing and accounting for context effectively have led to a dearth of such research in organizational science (see Johns, 2006). The inclusion of context is relevant to the development of both highly applicable theory and generalizable hypotheses, both of which are central to the continued advancement of organizational science as a field. Thus, HMA provides a way to explore nuances even within the highly specific samples discussed previously.

Scenario 3—Longitudinal research

Organizational science has seen a definitive rise in attention to complex research designs and statistical modeling techniques in recent years. In particular, longitudinal research has become increasingly common in top-tier organizational science outlets as researchers continue to shift their focus away from cross-sectional accounts of phenomena and toward demonstrating growth, adjustment, and stability of effects over time. This can present a considerable challenge to researchers, as hypothesis development, research design, and data collection are all further complicated by the addition of a longitudinal element.

The use of historical data sources can mitigate some of these challenges, as historical content is often presented in a naturally longitudinal format. Moreover, historical narratives are likely to provide detailed descriptions of phenomena of interest (e.g., events, behaviors, contexts) as they occur throughout a target entity’s history. By employing the HMA method, researchers are able to leverage this longitudinal presentation of information and draw comparisons between events occurring at multiple point in time more easily than they could with alternative methods. For example, an academic corporate history of the Ford Motor Company would be likely to account for the entirety of the firm’s development over the course of more than a century (e.g., Brinkley, 2004). A scholar interested in studying the influence of top management team (TMT) turnover on firm performance would be able to use HMA to identify instances of TMT turnover throughout the company’s history, and code for indicators of firm performance. This process could be executed for each instance of TMT turnover captured within the firm’s corporate history, resulting in a rich longitudinal account of the relationship of interest out of one static data source.

The use of HMA may further enhance the conclusions drawn from longitudinal analyses as the time between measurements is essentially unbounded. One potential limitation of longitudinal studies, particularly those conducted with field samples, is that the time between measurements is often relatively short. An argument can be made that, depending on the construct or process of interest, the temporal gaps in these studies may not be long enough to allow interventions to have their desired effect or to reduce the likelihood of common method biases (Podsakoff, MacKenzie, Lee, and Podsakoff, 2003). When using historical sources, however, the temporal distance between events may be on the order of months or years rather than days or weeks. Moreover, common biases in measurement and research design are largely avoided when conducting HMA, as the historical content that is being coded occurred naturally and free of researcher influences (Simonton, 2009). These taken together, HMA provides a series of methodological advantages that scholars engaged in longitudinal research should consider.

It should again be noted that the individual techniques employed in HMA are established in the qualitative methods literature, with several substantial resources available to provide guidance on their execution (e.g., Krippendorf, 1980; Neuendorf, 2002). However, the breadth of such texts may serve as a barrier to entry to some researchers, who are more inclined to find use in highly targeted references rather than broader treatments of the qualitative methods literature. This is in keeping with the perspective that some scholars, particularly those who are not overly familiar with content analysis, may benefit from the concise description of methodological “packages.” Such was the impetus for the pervious treatment of HMA, which continues to below.

Using HMA: 10 Steps

The methodological considerations for proper HMA are extensive, and can be overwhelming without the use of a sufficient reference. A small number of method references specific to HMA do exist; however, their utility in introducing the broader organizational science community to HMA is limited by their extensive and potentially confusing structure (e.g., Simonton, 1990) or their specific focus on applications to a niche research area (e.g., Ligon et al., 2012; Parry et al., 2014). To our knowledge there exists no easily interpretable, prescriptive reference for general HMA within the broader organizational science literature, limiting the exposure of these techniques to scholars who may find them interesting and beneficial to their research. Below, and summarized in Table 1, we provide such a guideline and discuss the steps future scholars should take when employing HMA techniques in their own research.

OPEN IN VIEWER

Table 1.

Action Steps for Historiometric Analysis.

Process Step	Actions	Challenges	Proposed Solutions
1. Define Constructs and Research Questions	Develop theory and hypotheses through extensive literature review	Risk of scope expansion in order to maximize use of narratives. Increased likelihood of confusion and coding inconsistency	Establish clear boundaries for the research project, guided by theory and core questions of interest.
	Operationalize constructs of interest through theory and / or the use of previously validated construct measures	Time and effort requirements by overly complex research programs may make physical and cognitive burnout more likely, increasing likelihood of coding errors	Remain considerate of the physical and cognitive limitations of the research team, and how exhausting such limitations may jeopardize the larger research project.
2. Investigative Piloting	Primary investigator engages in a “proof of concept” exploration of the research landscape via case analysis	Researcher can fall victim to confirmation bias, where an understanding of study goals make selective interpretation of sources and their content more likely	Engage in data source ‘triangulation’. Seek out multiple cases or sources of information which can be compared to each other for consistency in both data and interpretation.
	Acquire a small set of potential narratives and use them to demonstrate that phenomena of interest are likely to be identified in such content.		Establish criteria, rules, and expectations to use as a standardized guideline during material review.
3. Decisions on Data Structure	Determine how the how the data captured through content analysis should be structured	Data structure choices must be informed by assessment of data source availability and their narrative composition.	Develop a thorough and attentive piloting process to provide an assessment of the prospective sampling landscape.
	Establish how constructs or relationships of interest are best captured. Choose an “event-based” or “chapter-based” perspective		Use this analysis of potential source availability and structure will provide context that is informative for structural and strategic decisions.
	Determine whether the study is to approach data from a within-subjects or between-subjects design, or some combination.
	Identify a method for capturing criterion variables.
4. Prototyping and Codebook Drafting	Develop a codebook that reflects the intended data structure and identifies predictor, control, and outcome variables	Coders may express expectation or judgment biases (e.g., halo error, leniency error) during codebook development as a result of looking for the “right” answer to support proposed arguments	Establish and enforce benchmarks, construct prototypes, and common coding schemas through frame-of-reference training.
	Engage the research team in piloting to establish operational benchmarks. These benchmarks should provide examples for low, medium, and high levels of target constructs and phenomena.
5. Data Source Collection and Refinement	Use the coding strategy established during codebook development and information collected during piloting to identify and acquire data sources	Author biases present the most significant challenge to data source acquisition.	Engage in “source” criticism and look for tell-tale signs of bias
	Review the selected content and make assessments of potential author biases, information completeness, sourcing, and consistency	Attempt to reconcile the possibility of differing perspectives between historical accounts of the same individual, entity, event, or phenomenon	Employ only sources that demonstrate thorough and proper sourcing, composed by individuals who are understood to have skill in presenting fact rather than conjecture.
		Attend to the possibility of language and interpretive challenges, particularly when using data sources developed by non-native speakers of the research team’s vernacular language	Reframe the general research question to be less specific and more inclusive of alternative sources.
6. Event / Chapter Selection and Dissemination	Research team reviews the research materials and identifies content that is likely to be relevant to the research effort. Eliminate content which is irrelevant	Content identification and selection may be hindered by confirmation bias. Coders may make dubious assessments of passages within narratives based on a desire to find content that fits predictions	Utilize multiple coders for identifying and reviewing relevant content. Engage in frame-of-reference training and group discussion regarding discrepancies and differences of perspective on the content.
	Develop a plan for disseminating the selected content to the research team, and for storing any research materials
7. Coder Training	Familiarize the rating team with the goals and objectives of the research, as well as the codebook	Common judgement biases are likely to occur within the coding team. Take the time to instruct coders on these biases and how they can impact the research project. Empower the research team to self-police	Educate the coding team on common rater biases and errors. Provide clear examples that familiarize coders with the common characteristics and pitfalls associated with idiosyncratic rating.
	Conduct pilot tests to develop socialized mental models within the team, whereby team members engage in discussion over materials to come to mutual agreement and consistency		Engage coders in frame-of-reference training to establish congruent rating schemas within the team and mitigate the influence of judgmental biases
	Engage in meetings with the rating team in order to maintain consistent perspectives on the materials and processes, as well as address specific questions or concerns		Conduct periodic interrater reliability assessments using established metrics (e.g. Cohen’s kappa, ICC)
8. Protocol Execution and Managing Coder Fatigue	Establish a coding framework so that coders know how to progress through research materials and record data efficiently	Confirmation bias and judgmental errors are common at this stage	Assign multiple coders to each material coding assignment and aggregate their respective scores into composite measures, provided estimates of interrater agreement are sufficient.
	Organize materials such that coders are not constantly grouped together		Establish expectations and procedures for bias assessment and revisit them throughout the project lifecycle.
	Remain vigilant for signals of coder fatigue or judgmental lapses. Check in with coding team directly, as well as look for statistical indicators of inconsistency.		Provide “booster shots” of frame-of-reference training periodically to reacquaint the coding team with previously established frameworks and schemas
9. Data Analysis	Organize final dataset. Conduct statistical analyses necessary for hypothesis testing	Low statistical power may jeopardize the veracity of the conclusions drawn through analysis. Attention to this issue is necessary in the planning and organization stages of the research effort	Consider the possibility of low statistical power early in the research process and design the data collection strategy accordingly. Consider novel ways by which larger samples of targeted phenomena might be gleaned from data sources.
10. Integrating Quantitative Values with Qualitative Data	Assess the results of statistical analyses as they relate to both predictions and existing theory and research	The permanence of narrative sources opens the possibility of unscrupulous research practices (e.g., p-hacking)	Document key assumptions and predictions early in the research effort and refer to such documentation frequently.
	Look for opportunities to illustrate findings or support theory through the story-telling ability of the narratives		Consider the appropriateness and opportunity for study pre-registration with scholarly publication outlets.
	Consider revisiting data sources in search of theoretically sound moderators, informing future research

In addition, we use this section to address the issue of bias as it pertains to HMA. The intimate use of human beings in HMA, as primary investigators, coders, and even authors of source materials, coincides with the ever-present issue of biases. In particular, confirmation bias (Nickerson, 1998), expectation biases or judgment errors, and biases among source authors are likely to have some impact on HMA results. During our discussion of steps in executing the HMA method, we identify several instances in which these biases are most likely to exert some influence. We provide a treatment of how such biases might manifest, the effect they are likely to have, and suggestions for mitigating their overall impact on the research effort.

Finally, given that qualitative data analysis is often criticized as being less rigorous and, by proxy, less trustworthy (Lincoln & Guba, 1985; Sinkovics & Alfoldi, 2012; Tracy, 2010) we aim to provide a series of clear steps with the express intent of improving overall quality of research outcomes. More precisely, as Sinkovis, Penz, and Ghauri (2008) note, to make qualitative research “a viable source of knowledge generation and dissemination, researchers are encouraged to systematize, regularize, and coordinate the work of observation, recording and analysis” (p. 6). Thus, these steps serve not only as a guide and as “how to” with regard to HMA, but also to ensure a level of formalization that can help improve the quality (Tracy, 2010) and trustworthiness (Sinkovics & Alfoldi, 2012) of findings derived from the technique.

Step 1: Define Constructs and Research Questions

One advantage of HMA is that the method and its general procedures often place few limitations on the research questions that can be addressed. Indeed, Simonton (1990) argued that the most useful feature of HMA is the ability to test any model that could be put forth by another research method, regardless of the structure or complexity of the variables. Moreover, previous research employing HMA has generated effect sizes similar to those seen in tests of the same relationships using other research methods (Simonton, 2009). However, the use of HMA is limited by the ability of the researcher to develop clear and theoretically grounded operationalizations of variables of interest, particularly when measuring attitudinal, emotional, cultural, or otherwise psychometric constructs. An additional consideration is the type of content that is required for analysis, and whether the appropriate data sources are likely to be accessible. This is a complicated challenge, as the researcher will not have a complete perspective on what research questions are feasible in HMA research until he or she has looked through the available data sources directly. Therefore, it is recommended that prospective historiometricians engage in a process of investigative piloting in the early stages of the research process.

Challenges and biases

One potential challenge is that of scope expansion. That is, researchers engaging in HMA may be tempted to establish a program of research that is broad and sweeping in scope in order to maximize the value of the narrative sources that are used. As a result, prospective historiometricians may approach their piloting, sourcing, and data collection efforts with a set of research questions identified more by convenience than by scholarly necessity. Scope expansion, normally followed by the addition of additional variables and coding rules, may increase the likelihood of confusion and inconsistency in the content identification and coding processes. In addition is the more pragmatic concern of time and effort. The HMA method is labor intensive and time consuming; both the cognitive and physical resources of the research team must be conserved as much as possible to avoid burnout and the likelihood of coding errors. By expanding the research program unnecessarily the primary investigator may make these two issues more likely. Prospective historiometricians should be cognizant of the true goals of their research program and resist the temptation to expand the scope of the work beyond what is absolutely necessary.

Step 2: Investigative Piloting

In order to confirm that a research question or hypothesis of interest is testable using HMA, it is recommended that the primary investigator take on an exploratory investigation of a smaller scale. This small-scale piloting is, in effect, a “proof of concept” method via case analysis. Having previously established the research questions and constructs of interest, the primary investigator can use both intuition and previous research to determine which sources may be likely to contain information pertinent to the goals of the study.

For example, Mumford (2002) was interested in the concept of social innovation and wished to understand what tactics were necessary in order to succeed in a socially innovative endeavor. Mumford looked to 10 narrative descriptions of social innovation on the part of Benjamin Franklin in order to establish whether or not such criteria could be captured via historical sources. This small study demonstrated that evidence for such phenomena was present and measurable within these narratives, and would likely be found in similar sources should the study be expanded. Although Mumford’s endeavor was intended to only use the 10 examples as a total sample, the work informed the development of a larger study in which pragmatic leaders were compared to charismatic and ideological leaders (Mumford, Gaddis, Strange, & Scott, 2006). This exploratory strategy employed is similar to what one would expect during the planning stages of a larger HMA study.

Challenges and biases

During the piloting stage, the researcher should be wary of confirmation bias (see Nickerson, 1998). At this stage, the researcher is using his or her perspective on the objectives of the study to selectively identify cases that demonstrate the phenomenon of interest. This understanding of the study’s purpose, and the researcher’s desire to develop establish the efficacy of the research process, may make him or her more likely to engage in piloting with cases that are known to fit the project’s goals. In doing so, the researcher may be omitting evidence that would be important for the development of sampling plans and general decisions about the research project.

This issue can be partially mitigated through a “triangulation” of data sources (Golafshani, 2003). Triangulation refers to the use of multiple cases or sources of information that can be compared to one another for consistency in both data and interpretation. This approach has been noted as an effective method for mitigating the influence of researcher and rater biases in qualitative research (see Mathison, 1988, p. 13). In addition, the researcher can establish a set of clear criteria before engaging in piloting and use them as a standardized guideline through the review of case materials. Grounding the research in a series of rules and expectations, guided by theory, will help to alleviate any tendency toward varying interpretation between cases and adapting strategies in order to fit goals.

Step 3: Decisions on Data Structure

Data structure decisions inform how the research team approaches codebook development and strategy for reviewing content, both of which are central to developing effective and reliable estimates of the constructs of interest. These decisions also delineate what qualifies as the sample for the research effort, an important feature that will help researchers determine what type of narrative sources are needed and in what amount. First, the researcher must determine how the constructs or relationships of interest are best captured in the text. This decision will most often decompose to two perspectives on the data: event-based and chapter-based.

In event-based structures, the researcher intends to capture variables of interest through the identification and analysis of specific events described in the text and their surrounding context (see DeChurch et al., 2011, for an example). Alternatively, a chapter-based perspective seeks to use the naturally longitudinal structure of narrative sources to identify critical time periods that are most relevant to the research question. The appropriate “chapters”—an allusion to the chapter format of biographies and the like—are identified through careful review of the source content. For example, a study of leader Machiavellianism and performance sought to identify chapters in biographies that described each leader’s “pinnacle of power” (Bedell et al., 2006, p. 58). The researchers determined that these periods in leader lives, and the context surrounding them, would provide the strongest indications of an individual’s Machiavellian tendencies. In addition, Bedell and colleagues (2006) found that summary chapters were useful for identifying criterion measures of leader performance.

Once a general perspective has been established, the researcher must determine whether the study is intended to have a within-subjects or between-subjects design, or a combination of the two. The central point for this strategic decision is whether the goal of the research is to study the progression of a specific phenomenon over time, or to draw direct comparisons between groups. For example, a researcher interested in studying creativity may wish to use narrative corporate histories of firms that have had historical success in creativity and innovation. The investigator must determine whether to study events that define the development of those processes in a certain set of firms (within-subjects perspective), or to draw comparisons between the firms directly (between-subjects perspective). In either case, the end goal of the research and coding effort is to engage in complex statistical procedures. This requires adequate statistical power, calling for large sample sizes (Simonton, 2009). Thus, discussions of single cases or small-group comparisons are not adequate for the HMA method.

Challenges and biases

Coming to comfortable decisions on data structure in HMA can be challenging, as the consistency of information between data sources is not completely reliable. Although a researcher can assume that narratives regarding similar entities (e.g., leaders) will have commonalities, there is in fact no way to be certain about their individual fit to a chosen data structure until they have each been thoroughly analyzed. Because the intended structure of the data set has significant influence over the development of the codebook, an incomplete assessment of this obstacle may result in a need for repeated organizational and design work, or potentially a reevaluation of the research effort’s viability. It is for this reason that a thorough and attentive piloting process is necessary. Piloting not only establishes the research question as viable for the method, but also provides an assessment of the prospective sampling landscape that will be informative for structural and strategic decisions.

Step 4: Prototyping and Codebook Drafting

Once the model has been established and data structure has been specified, the researcher can develop a codebook to use during data collection. In a manner similar to Step 2, the researcher should take on the assistance of a small (3-5 people) research team to engage in material review and codebook development. The codebook itself should identify predictor, control, and outcome variables (Ligon et al., 2012), and should be developed using an iterative process of item writing and refinement (e.g., Mumford et al., 2006). The style in which these items are created is dependent on the structure of the research; Ligon and colleagues (2012), for example, advocate for the heavy influence of biodata when studying individual leaders and leadership phenomena.

Items should be written so that they are clear and easily interpretable, as well as anchored in a manner that is appropriate given the construct or variable in question (Osterlind, 1998). It is also recommended that multiple items be used to measure each construct; Ligon and colleagues (2012) advocate for grouping items by examining either Likert-type item intercorrelations or the result of Q-Sort tasks (McKeown & Thomas, 1988). In either case, the researcher would need to acquire feedback from groups of independent coders in order to confirm which items should be included in the finished codebook.

Once the codebook has been preliminarily developed, the research team should engage in piloting to establish operational benchmarks. The use of benchmarks to solidify the operationalization of variables is a core procedure in many HMA studies (Ligon et al., 2012; O’Connor et al., 1995). In this instance, the term benchmark refers to detailed examples that can be used as comparators to determine to what extent a construct of interest is expressed. Moreover, these benchmarks allow the researcher to score each variable along a meaningful continuum, as opposed to simply identifying whether or not a variable might be present. Ligon and colleagues (2012) suggest deliberately oversampling so that 10% of the sampled population can be removed for benchmark development. This sample is then reviewed thoroughly by content experts who attempt to identify the phenomena of interest in the text. These excerpts are then aggregated and consensus is developed over which examples represent low, medium, and high values for the construct, to then be used as comparative benchmarks when coding the rest of the sample (for examples, see Bedell et al., 2006; O’Connor et al., 1995).

Challenges and biases

During codebook development, coders are more likely to express any extant expectation or judgment biases that color the way they approach narrative interpretation. For example, leniency bias describes a systematic tendency to provide ratings that are more positive or “lenient” than would be expected given the ratings provided by others (Guilford, 1954). Research has demonstrated that rating leniency is a stable cognitive bias, wherein individuals who are likely to provide overly lenient ratings are likely to do so across rating subjects (Kane, Bernardin, Villanova, & Peyrefitte, 1995). At the codebook development stage of an HMA project such biases are problematic, particularly when developing benchmarks, as coders may take interpretive liberties when trying to identify examples of important phenomena. This can result in inaccurate or skewed examples to be used as benchmarks, which may harm the interpretation of narratives and the estimation of variables in the larger study.

The issue of judgmental biases and their influence on rater accuracy have been thoroughly treated in the organizational science literature, particularly within the domain of performance appraisal (e.g., Woehr & Huffcutt, 1994). One robustly supported approach to mitigating the influence of coder biases is the use of frame-of-reference (FOR) training (Bernardin & Buckley, 1981; Gorman & Rentsch, 2009; Uggerslev & Sulsky, 2009; Woehr & Huffcutt, 1994). In keeping with the benchmarking approach discussed previously, FOR training emphasizes educating coders on the complexity and multidimensionality of variables of interest (e.g., performance), often including examples to help develop a coder’s perspective. Moreover, coders engage in coding practice and receive feedback from subject matter experts as they begin to develop both individual and group coding efficacy. Indeed, the “goal of frame-of-reference training is to train raters to share and use common conceptualizations…when making evaluations” (Woehr & Huffcutt, 1994, p. 192). As such, this training method is also useful in mitigating the influence of judgment biases or inconsistency during the content coding process.

Step 5: Data Source Collection and Refinement

After finalizing the codebook, the coding strategy outlined therein will serve to inform the final identification and collection of narrative sources. The identification of source material is likely the greatest challenge to HMA execution, and also the factor that has most limited the technique’s expansion into organizational science. However, researchers have demonstrated creativity in their ability to identify source materials for successful HMA studies. As Mumford and colleagues (2005) demonstrated with their use of obituaries, and DeChurch and colleagues (2011) with their aggregation of various accounts of historical crises, sources need not only be retrieved from libraries or organizational archives to be relevant. In any case, however, it is imperative that the resources acquired are accurate in both their content and the application of that content to the research questions at hand. Engaging in a critical assessment of source documents is necessary in any endeavor employing historical data, particularly the narrative histories that are most commonly used in HMA. As such, HMA research teams must engage in a systematic review of all materials gathered as potential data sources and remove those that do not meet established criteria for relevance and accuracy.

Challenges and biases

The sources of information used in HMA (e.g., biographies, newspaper articles, periodicals) have one primary disadvantage that is common to most ideographic methods: the bias of the source author. Because narrative accounts are written with regard to past events, the potential exists for the author to introduce biased interpretations of those events and the individuals (or teams, or firms) that are involved. These biases have been demonstrated to have nontrivial impacts on study conclusions (see Mumford et al., 2005, for an example). However, with diligence and attention to detail, a “source criticism” (Alvesson & Skoldberg, 2009), such biases can be identified and weeded out before they influence the research at hand.

As a rule, HMA studies should employ narratives that are well-sourced and were developed for scholarly purposes. Academic biographies and histories, along with well-sourced periodicals and other media, are more likely to present a balanced and complete perspective of historical facts than narratives that are not constructed for those purposes. Before engaging in source acquisition, researchers should assess what the most likely sources of data are for the research question (e.g., biographies, news reports) and develop an understanding of leading authors, reporters, and other narrative sources in that space. Relying on those individuals and their contemporaries will make acquiring relatively unbiased materials an easier task.

Researchers should also consider the role that language and cultural differences may play in the availability of and access to narrative sources. In some cases, rich and well-documented sources may not be available in the vernacular language of the research team. In the case that this issue cannot be resolved by replacing the target entity with another that has documentation in the native language of the research team, this issue can present a considerable limitation. We would recommend, in that case, that the researcher attempt to add a native speaker to the coding team. If that is not possible, reframing the general research question to be less specific and more inclusive of potential sources would likely be the best course of action. However, because HMA relies on large samples of both entities and phenomena in order to test hypotheses, it is unlikely that such a challenge would become a fatal obstacle to the execution of an HMA study.

In addition to biases on the part of the author, the researcher must also be aware of his or her own biases that may influence the selection of source materials. In particular, researchers may inadvertently allow personal beliefs, opinions, or points of view to influence the types of source materials that are pursued. For example, a researcher studying the role of business acumen in the effectiveness of political leaders may be more likely to seek out historical narratives composed by those known to advocate for his or her personal political beliefs. This kind of bias can restrict the source data in a systematic manner, wherein the perspectives presented are unbalanced politically. However, adherence to the guidelines described above, particularly the deliberate use of sources that have been developed for scientific or scholarly work, should provide some protection against this issue.

Step 6: Event/Chapter Selection and Dissemination

Historical documents, be they narratives or other sources, often contain a large quantity of content that is unnecessary to the research project. Before beginning the coding process, researchers should critically assess the materials they have collected for specific subsections that are likely to contain the content that is needed. This may entail the selection of clear, easily identifiable content such as quotes or references to specific issues (e.g., Simonton, 1999), or the removal of content that is understood to be irrelevant to the research at hand. These assessments should be made by subject matter experts and individuals who are familiar with both the objectives of the research and the content of the sources in question, in an effort to limit unnecessary work and maintain organization during the data collection process.

Once the events and/or chapters of interest have been identified, a plan should be devised for disseminating the content to the research team. In many cases the narrative sources that are procured are physical rather than digital, in the form of books or other print media. Using a digital scanner and creating central repository for in-scope content can be an effective way to maintain organization within the project team and minimize back-end work (e.g., tracking physical materials). However, researchers undertaking this strategy must ensure that there is sufficient narrative material covered in each scanned document to provide coders with context for understanding what they are analyzing. This is particular important when working in an event-based data structure, as events of interest may be described only in short passages with significant context held in the surrounding pages. Once the content has been organized and made available to the research team, the researcher can advance to training content coders.

Challenges and biases

The selection of content to be coded within the source narratives is most likely to be challenged by the presence of confirmation bias. Coders who are aware of what they are looking for within a narrative may make dubious assessments of passages, whereby they view the content as indicative of constructs or phenomena of interest when they, in reality, are not (see the discussion on information-processing bases of confirmation bias in Nickerson, 1998, p. 198). Conversely, coders may also be likely to omit otherwise relevant content because it does not clearly fit the predictions of the researcher, thus artificially restricting the variability within the sample. Such a situation is where the use of multiple coders for each passage is particularly useful. Each source should be thoroughly reviewed for relevant content by multiple coders. As each coder develops a list of passages that should be included in the research effort, group discussion over discrepancies, omissions, and other issues should take place. The goal of this process is to mitigate biases through consensus, using the broader coding team as a quality control check to ensure that relevant content is not missed and dubious content is called into question before analysis. In this way, the recommended process for content selection is similar to that of FOR training (see the discussion under Step 4), in which the development of coding team consensus that is grounded by subject matter expert feedback is used as a quality-control mechanism in the research.

Step 7: Coder Training

The coding of qualitative sources is an established process, with a variety of texts providing guidelines for both coder training and analysis of common obstacles (e.g., Katz & Sharrock, 1976; Neuendorf, 2002). In keeping with other methods, the primary goal of coder training in HMA is to familiarize the research team with the goals of the study and the elements of the codebook. In particular, it is important that coders develop a thorough understanding of the constructs that will be coded and how they are operationalized. This is where the use of benchmarks is especially useful, as benchmarks provide contextually rich examples that coders can use to frame their approach to identifying a particular construct within a narrative. Coders can use the benchmarks to develop conceptual frameworks and refer back to them as points of reference when the content they are working on appears to be ambiguous or convoluted.

However, coders are also at risk of relying too heavily on the examples outlined by benchmarks and other defining references. Because benchmarks describe specific and identifiable representations of constructs, coders can easily fall into a pattern of viewing such examples as prescriptive rather than descriptive. That is, coders who have framed their understanding of the construct of interest through benchmarks may begin to approach the content they are coding in a “checklist” style, looking for examples that are similar to those described in benchmarking materials. This bias shifts the focus of the coding effort away from identifying the construct through a thorough analysis of the entire narrative, and toward a search for a narrow set of criteria that may not be present or relevant across disparate contexts. Coders must be made aware of this potential pitfall and trained to avoid it to the best of their ability.

One strategy that may prove effective is mental modeling within the research team. The development of mental models reduces the likelihood of what Hak and Bernts (1996) labeled “ad hocing”—the process of applying unstandardized coding rules to confusing or ambiguous content (Katz & Sharrock, 1976). By developing a shared understanding of the research effort, the included constructs, and how to identify them, the research team develops some natural insulation to both inconsistency and bias. To do so, coders should engage in a process of group piloting and familiarization with the constructs included in the study. Coders are familiarized with operationalizations of codebook variables and established construct benchmarks. The coding team then engages in a uniform coding task, from which numerical data are collected from each coder and analyzed for reliability, with discrepancies identified. The research team then engages in discussion, debate, and group analysis of the content, working toward a consensus regarding the variables of interest and alignment on how any adjustments in thinking about these constructs should inform future coding. This method then repeats, with a new coding assignment granted to the entire research team and new group analyses to follow if necessary. Ligon and colleagues (2012) suggest that a typical benchmarking and training period is likely to last at least two weeks and consist of both group and independent work; we concur with this position as a baseline, but recognize that in many cases the process may require further time commitment.

The result of these efforts is a team of trained coders who share both a functional understanding of the task and mental models regarding the interpretation of content. However, as coders become more skilled and experienced in their roles they may be likely to revert to old habits of interpretation (Hak & Bernts, 1996). For this reason, periodic checks of interrater reliability are necessary throughout the process. We recommend conducting these checks both qualitatively and quantitatively. Meetings, similar to the ones discussed previously, should remain a consistent and valued part of the research process and be used to uncover any challenges or misalignments that arise during the coding process. In addition, the primary investigator should check real coder output for interrater reliability via established metrics such as Cohen’s kappa (Cohen, 1968), the intraclass correlation coefficient (Lebreton & Senter, 2008), and rWG(j) (James, Demaree, & Wolf, 1984). Should these estimates fall below the normative thresholds established in the literature, the group should revisit the items in question and work to develop consensus around the discovered discrepancy. Indeed, the training and coordination of coders does not ever truly end and diligence is required to ensure the most accurate research results.

Challenges and biases

Although biases do not have much influence over the coder training process itself, how coders are trained is likely to influence their propensity to exhibit biases in their work. Some scholars have suggested that training coders on the specific details of common rating biases, using clear examples that familiarize coders with error characteristics and pitfalls, may be effective in mitigating the influence of idiosyncratic rating tendencies (e.g., Bernardin & Buckley, 1981; Latham, 1986). We have found that clear instruction in and discussion of common biases and judgment errors among the research team enables coders to engage in self-policing and better understand the importance of remaining as scientifically objective as possible during the coding process. However, despite meta-analytic support for such programs, the opinion of organizational researchers has been mixed (see Woehr & Huffcutt, 1994).

Far better supported for both reducing biases and improving rating accuracy is the FOR method discussed in Step 4 (Gorman & Rentsch, 2009; Roch, Woehr, Mishra, & Kieszczynska, 2012; Woehr & Huffcutt, 1994). Research has demonstrated that FOR training results in more congruent coding among members of a rating team. For example, Gorman and Rentsch (2009) found that raters who were trained to assess performance under FOR possessed rating schemas that were more similar to the schema that was used as their point of reference than were coders in a control group. To paraphrase Roch and colleagues (2012), FOR training influences group rating accuracy by clarifying what behaviors or other variables indicate levels of a phenomenon of interest, and by developing prototypes that can be used as benchmarks. Using a FOR training framework to develop the coding team mental models described in Step 7, clarified by benchmarks developed during the codebook construction process, will likely substantially limit the influence of individual biases. When combined with statistical checks of interrater agreement and interrater reliability during the coding process, the historiometrician can proceed with confidence that the necessary steps have been taken to arrive at reliable and comparable data.

Step 8: Protocol Execution and Managing Coder Fatigue

After the rating team has been sufficiently trained and interrater reliability is deemed to meet accepted standards, the data collection process can begin. A framework for coding should be established so that coders understand how to progress through research materials and record data in a manner in keeping with the goals of the project. Repositories for finished work must be created so that coding output can be collected, organized, and periodically checked for completeness and consistency. In order to calculate interrater reliability estimates, and mitigate the influence of individual biases, we agree with past historiometricians and qualitative researchers in recommending that each material source be rated by at least three distinct coders (see Carey, Morgan, & Oxtoby, 1996; Ligon et al., 2012; MacQueen, McLellan, Kay, & Milstein, 1998). Furthermore, we recommend that these coders be assigned materials in a manner that limits the frequency of cross-ratings with the same individuals. The creation of distinct coding teams or clustering of analyses among the same group of individuals may belie any well-hidden inconsistencies that may exist within the coding outputs. By deliberately varying the composition of groups assigned to assess particular materials, a clearer picture of overall rating reliability is achieved.

In addition to general execution, the researcher must be on guard for signals of coder fatigue and judgmental lapses. Coders who are fatigued may display decreases in rating reliability (Hoskens & Wilson, 2001) and jeopardize the results of the study overall. They may also be more likely to rely on idiosyncratic rating schemas and judgmental biases like those noted in prior steps. This may be more likely to occur in the initial stages of an HMA study, when coders are not accustomed to the intensity of the research process and the physical and mental demands that come along with it. We recommend formal check-ins with coders, particularly at the midpoint of the study, on how they feel about workload and their ability to produce quality data. In addition, the researcher may use quantitative analysis to uncover significant changes within individual coders (e.g., ratings disparities, inclusion of biases, and deviation from norms). In the instance that a decline in coding quality is detected, the researcher should cease all research tasks across all coders and regroup to assess the root issues. Additional training should be provided, including refresher courses in detecting and avoiding the influence of personal biases and a reevaluation of how critical variables are defined and operationalized.

Challenges and biases

It is during this step that biases and judgmental errors are likely to be most impactful, as at this stage the research team has been tasked with coding the data that will eventually be the basis for statistical analysis and hypothesis testing. The assignment of multiple coders to each task is a first step toward limiting the influence of bias in the final data set. The aggregation of ratings into composite scores will account for some of the variation between coders, creating variable estimates that are more stable and less likely to express the influence of rating biases. However, a necessary precursor to this aggregation is a statistical demonstration of interrater agreement. In a manner similar to that which is seen during training, the primary researcher should periodically collect in-progress data sets from the research team and assess interrater agreement among the coders. Should agreement estimates fail to meet an acceptable threshold, the entire coding team should undertake discussion of the extant discrepancies and additional practice.

Of critical importance to researchers is that these procedures, and the expectations around them, are established well before reaching this stage of the HMA process. The stop-and-go method of protecting the data from bias is likely to frustrate those members of the research team that are not bought in to the rationale behind those procedures. The use of an FOR training approach (Roch et al., 2012) may be particularly useful to this end, as the method requires the clear explication of construct operationalizations and benchmarks that can be referred back to at any time. Researchers should consider providing their teams with periodic “booster shots” of FOR training to reinforce standards as well as realign the coding team to a common, central schema. This can mitigate future fatigue and frustration with the research process, ultimately making it more likely that the HMA project will be successful.

Step 9: Data Analysis

Once data have been collected and the research team is satisfied that materials have been sufficiently reviewed, the data may be prepared for statistical analysis. The analytical approach of most researchers conducting HMA is indistinguishable from that of studies employing more traditional quantitative data (Simonton, 2009). That is, the nature of the research questions explored in HMA and the structure of the data therein lend themselves to hypothesis testing approaches that are common in the broader organizational science literature (e.g., linear regression, hierarchical linear modeling, ANOVA, ANCOVA). However, the use of text as a data source in HMA allows for opportunities to employ novel and emergent data analysis processes that are not applicable in nonqualitative forms of research, such as computer-aided textual analysis (Pollach, 2012).

The use of software to collect, structure, and analyze data from text-based sources has been present in the content analysis space for several decades (see Krippendorf, 2004; Popping, 2000). These studies generally rely on the use of content dictionaries, whether previously established (e.g., LIWC: Pennebaker, Mehl, & Niederhoffer, 2003; General Inquirer: Stone et al., 2000), or developed for the research project specifically (e.g., Gibson & Zellmer-Bruhn, 2001; Palmer, Kabanoff, & Dunford, 1997). Relying on said dictionaries, computer-aided textual analysis is generally used for the same word count and sentiment analyses that are common across qualitative research (Pollach, 2012). Thus, the deep interpretation of a narrative’s context in order to capture variables of interest, which is the goal of HMA studies, is not necessarily aided by such methods. Software has been developed with the aim of conducting such “interpretive” textual analysis, but still relies to date on the use of frequency counting and the presence of key words in context in order to generate data (Pollach, 2012). As a result, researchers engaged in HMA are not likely to find a panacea for their analytical needs in text analysis software just yet, and should continue to rely on well-trained coders and the input of subject matter experts. Continued growth in this area, however, bodes well for the role that computers may play in the future of this research approach.

Challenges and biases

Because HMA is not hindered by any specific analytical limitations, researchers should assume that all challenges common to statistical analysis and interpretation are applicable to HMA. Of particular interest should be that of low statistical power, an issue that has pervaded social science research for decades (see Maxwell, 2004). Fortunately, scholars can make these assessments before data collection begins though the application of mathematical formulas and computer software developed to provide guidance on achieving adequate statistical power in their research. These assessments must be made in congress with data structure decisions in order to avoid accidental undersampling and the need for added work after the study has already begun. Given the unique approach to sampling and data structure employed in HMA, and text-based content analytic methods in general, researchers must give special consideration to how likely their final sample is to have adequate statistical power.

Step 10: Integrating Quantitative Values With Qualitative Data

The end product of an HMA study is a series of statistical results with regard to relationships of interest, as would be expected when using many other methods driven by numerical data. However, the use of rich narratives as a data source in HMA allows the researcher to use these results in several unique ways. These are largely based on the researcher’s ability to revisit both the data source and the coding strategy to draw inductive inferences from the research effort in an attempt to explain or provide support for the analytic results. For example, researchers identifying (or failing to identify) a relationship between two variables in an HMA study can inductively explore the narrative source for explanations for the relationship. This is not to advocate for “p-hacking” or other manipulations of research. Rather, this process provides the researcher with an advantage in explaining the existence or absence of a relationship in order to inform future research or theory. In terms outlined by Tracy (2010), this step aims to provide “meaningful coherence” to the data and findings. The contextual details within narrative accounts are more likely to provide such insight directly than, say, the results of survey respondents, which do not directly provide answers to the question of “why.” As Tracy (2010) notes, these in-depth illustrations available via HMA help provide context and background for quantitative findings.

Similarly, such data sources provide ready-made case illustrations of quantitative findings. Whereas the results of a more traditional study using numeric data may be difficult to translate to a “real-world” scenario, those gleaned from HMA studies do so through those examples directly. This can provide context for explaining how a relationship of interest might manifest, as well as details to inform future research strategies. For example, an HMA study demonstrating that engineering firms produced more creative output when subjected to certain temporal and physical conditions might inform the design of future laboratory research in the creativity space.

Finally, the ability to revisit the narrative sources grants researchers the ability to inform future research through a search for moderators. During the process of interpreting research findings and developing illustrations described above, historiometricians may also identify previously unconsidered moderator variables within the narrative content. Having not considered their inclusion previously these variables would likely have been missed in the initial coding process, but systematic review of the content within the context of the research results has the potential to reveal them. This is a methodologically dangerous area, where deliberate or inadvertent manipulation of research protocols is possible. However, the permanence of both the codebook and the data sources themselves allows for replication of HMA studies with little to no risk of sampling error (Simonton, 2009). Diligent historiometricians who identify potential previously unconsidered moderators may be able to iterate on their own research in a meaningful way, or inform the future research of others. It is of note, moreover, that the flexibility and ability to return to qualitative data sets in iterative fashion is a hallmark of the richness that defines qualitative data, provided such adaptability is treated with the care required to ensure quality and trustworthiness (Sinkovics & Alfoldi, 2012).

Challenges and biases

As was alluded to above, the permanence of the narrative sources used in HMA does open the door to the possibility of unscrupulous behavior on the part of the researcher. Moreover, such behavior may occur incidentally rather than deliberately, as the result of the iterative process of research refinement and interpretation. Within a scholarly climate that is concerned with the ethics and intentions of researchers (see Head, Holman, Lanfear, Kahn, & Jennions, 2015), as well as the reproducibility of research results (see Pashler & Wagenmakers, 2012), historiometricians must be meticulously attentive to how their approach to the research process may be ethically compromised. Researchers can keep themselves and their research teams honest by documenting their assumptions and predictions in the early stages of the research effort, and referring to that documentation if and when the direction of analyses or other research protocols seem questionable. However, a more concrete method for ensuring ethical practices within this highly flexible research framework may be to take advantage of the growing trend toward study preregistration. Although the scientific community has yet to reach consensus on the efficacy of preregistration, a number of academic outlets have begun accepting preregistration submissions in the hope that it will keep scholars focused on a particular line of inquiry and remove some incentives for unethical research practices. In any case, the historiometrician must be acutely aware of this issue and take care to avoid it entirely.

Concluding Remarks

The present discussion has sought to highlight the ways by which the procedures and techniques that constitute HMA can positively influence organizational science. Apart from its limited use in the leadership literature (e.g., Deluga, 1997; Hunter et al., 2011; O’Connor et al., 1995), HMA has had little exposure in the world of organizational research. We feel that this is likely due to both the breadth and complexity of the vast content analysis literature (Shapiro & Markoff, 1997), which make certain techniques and combinations thereof difficult for neophyte qualitative researchers to access, as well as an absence of a discussion on HMA techniques in outlets whose goals are to reach the broader organizational science community. Thus, our aim has been to provide future scholars with a clear, pointed resource that is both prescriptive regarding the procedures for successful HMA and descriptive in both its benefits and its challenges as a program of research methods. It is our belief that HMA may have broad applications across the organizational research domain, and will provide scholars with an additional methodological tool to be used in our continuing quest to uncover truths about the world of work.