Citation Context Analysis as a Method for Conducting Rigorous and Impactful Literature Reviews

Question 1: What Content from the Focal Works Have Subsequent Authors Cited?

The primary research question that citation context analyses typically examine is what specific content or knowledge claims citing authors have mentioned from the cited work or works. Articles and books typically contain a variety of knowledge claims that subsequent works may cite (Anderson, 2006; Anderson & Lemken, 2019; Golden-Biddle et al., 2006; McCain & Salvucci, 2006). These range from the introduction of new concepts, the positing of theoretical relationships between concepts in the form of propositions and hypotheses, and especially empirical research findings. As a general rule, seminal books like March and Simon’s (1958) Organizations or Weick’s (1979) The Social Psychology of Organizing contain vastly more knowledge claims that could be cited compared with a typical article. Only by examining what citing authors say is it possible to rigorously identify the content from major works that citing authors are drawing on, and thus what their realized contribution to the literature has been.

One useful concept in citation context analysis is a work’s “uniformity of usage,” which refers to the extent to which citations to a work are all for the same point versus a variety of points (Small, 1978, p. 330). For a typical work, Small (1978, p. 338) noted that “perhaps the most important result of the social selection of knowledge through citation is the narrowing of meaning which occurs.” This narrowing has significant implications for theory development and testing because it frequently eliminates important qualifications and nuances in the original work that may be essential. One benefit of a citation context analysis is that it charts and codifies the actual use of a seminal work in later scholarship and can help identify specific instances of distortions, omissions, and misappropriations of key knowledge claims from important works.

Question 2: To What Extent Has the Usage of the Work Been Peripheral Versus Substantive to the Citing Author’s Main Arguments?

In reviewing the impact of a seminal work, scholars frequently want to know how extensive the influence of the work has been on subsequent scholarship. The use of citation counts to measure this influence has a widely recognized limitation. As Zhang et al. (2013, p. 1490) point out, “Traditionally, a citation is interpreted as author A being influenced by the work of author B, though without any attempt to specify the strength or direction of that influence. Additionally, it is assumed that each reference has made an equal contribution to the citing article.” In reality, there is a great deal of variance in the extent to which citations are necessary and crucial to a citing author’s argument. Citing works often cite a paper for peripheral knowledge claims (Cozzens, 1985), rather than for its primary knowledge claims, such as citing it for something mentioned in a single sentence or for a particular phrase or buzzword. An example is when articles ritualistically cite March and Simon (1958) for the widely accepted and established notion that people are only “boundedly rational,” particularly if the citing work does not say anything else about the more nuanced view of the “boundaries of rationality” that these authors actually discussed or how subsequent work has developed this idea (Anderson & Lemken, 2019). Indeed, citation context studies frequently find that a large number of citations to a seminal work are peripheral. As an example, Kacmar and Whitfield (2000, p. 392) found that “the vast majority of the citing articles simply listed the focal articles as references instead of using them as an integral component…of the study.” An example of a peripheral or superficial citation would be to make a statement such as “scholars have discussed the importance of literature reviews” and then give one or a few citations. Since dozens of references could be cited for this trivial point, citing those references in this statement does not show that the sources were central to the citing article. In contrast, if a citing author uses a direct quote from the article that discusses multiple reasons why literature reviews are important, as we did when we cited Baumeister and Leary (1997) in our introduction, then it is much clearer that this source was more important to the citing paper.

Normative guidelines for proper citation and reference use have argued that “references are not needed to support obvious or well accepted assertions” (Campion, 1997, p. 166), and yet studies have found that many citations are arguably unnecessary (e.g., Anderson, 2006; Anderson & Lemken, 2019). This is especially a problem when authors cite classic works—the type most likely to be the focus of citation context analyses—because scholars have suggested that authors sometimes cite seminal works merely because it is convenient and difficult for reviewers to verify whether the work does or does not in fact support the assertion being made. For example, Sutton and Staw (1995, p. 373) noted that: “This obfuscation can unfortunately be successful when references are made to widely known and cited works like Kanter (1977), Katz and Kahn (1978), March and Simon (1958), Thompson (1967), and Williamson (1975)…. Papers for organizational research journals typically include a set of such throw-away references.”

For these reasons, one primary approach to categorizing citations is in terms of the importance of the cited work to the citing work (Hanney et al., 2005). In principle, one could distinguish any number of categories of importance, but research frequently divides citations into two categories: those citations that are peripheral and those that are more substantial (Krampen et al., 2007; see also Moravcsik & Murugesan, 1975, who labeled these categories as “perfunctory” and “organic”). Classification schemes for types of citations frequently “distinguish citations of a relatively ‘inconsequential’ kind from those making more substantial reference to a cited work” (Lee, 2010, p. 118). Moravcsik and Murugesan (1975) also discussed “redundant” references, in which several references are made for the same point (which Danell, 2012, called “block citations”). If a work is cited only for trivial reasons, and in situations where many other works could have been equivalently cited to make the same point, then the importance of that citation is much less than when a scholar cites the full knowledge claim of a work (Golden-Biddle et al., 2006). The key issue underlying this research question is that certain sources are essential to the reasoning in a citing paper, and citations to such sources are necessary for giving credit to the originator of the ideas and findings. Other works are cited for tangential or unrelated reasons and make general points such that many other equivalent sources could have been cited instead. These peripheral citations do not indicate significant usage of the cited work.

This question of whether citations are peripheral or substantive has implications for theory development because peripheral citations are much less likely to signify that a citing work is actually building on and extending the theory in the citing article. Indeed, we suggest that there is a danger that researchers display a form of the “input bias” (Chinander & Schweitzer, 2003) when evaluating scholarly works, such that they use the quantity of citations that a work receives as a proxy for evaluating the quality of the theorizing and empirical support that the work has received. But if those citations are peripheral and do not, in fact, truly build upon the theoretical assertions in a substantial way, the overall level of subsequent theoretical development is likely to be far less than it may appear. This potential input bias is exacerbated when search engines use citation counts to determine the order of articles to display in searches.

Question 3: How Has the Content Cited from the Works Changed Over Time?

In order to assess the overall contribution of a work, it can be illuminating to examine whether and how the reasons why the work is being cited are changing over time. Knowledge in a field changes continuously, and new understandings of a phenomenon can render older understandings outdated. In addition, there are management fads and fashions regarding what topics are considered relevant and theoretically interesting (Abrahamson, 1996; Spell, 2001). As knowledge and tastes develop, works that were once seminal may become less relevant, and only a few classics continue to garner citations at high rates (Bornmann & Daniel, 2006). Examining changes in the content that a work is cited for over time therefore highlights “the possibility of using citation context analysis as an historical tool” (Cozzens, 1985, p. 134). Understanding how the citations of a given work change over time is important for theory development, because it helps to identify the parts of a cited work that citing scholars believe have continuing relevance. When aspects of a source work are no longer cited as frequently, it may signify that those parts are discredited or unsupported by subsequent work. It may also represent an opportunity to revisit those to determine if seldomly cited or uncited ideas may have been ahead of their time, foreshadowing current scholarship. In these ways, citation context analyses can reveal progress in a field, and suggest new avenues for future work.

Several citation context analyses have examined whether the content being cited in a focal work has changed over time (e.g., Anderson, 2006; Anderson & Lemken, 2019; Chang, 2013a; McCain & Salvucci, 2006; Richardson & Pysek, 2008), with the implication that the realized contribution of the work can change. Both Cozzens (1985) and Hargens (2000) found that earlier citations to a work were more likely to discuss methods and findings explicitly while later citations made more cursory mentions of knowledge claims or noted how works initiated new fields. Citation context analyses that examine the use of citations from a work over time can reveal important patterns of usage that might otherwise be missed. In addition, by tracking the evolution of theories from their origins in seminal works to their subsequent uses, citation context analysis can help detect whether a field is advancing and developing new knowledge claims or stagnating by relying on a core of largely unchanging theories or untested assertions.

Question 4: How Many of the Citations to the Works Refer to Empirical Evidence Regarding Their Knowledge Claims?

A distinguishing feature of science as an institution is that its body of knowledge is built upon empirical tests. Although some scholars have challenged this view, claiming that empirical work cannot really determine whether assertions are true or not (e.g., Astley, 1985), it is clear that the majority of work in organization studies is empirical. However, except when it is aggregated in meta-analyses, very little scholarship has examined how subsequent work draws on the empirical findings in prior work (Anderson, 2006; Anderson & Lemken, 2019).

An important issue that citation context analyses can address is the extent to which citations refer to either empirical evidence from the cited work, or specific empirical tests of the assertions in the cited work. That is, how many citations mention a knowledge claim in a work and then discuss subsequent empirical tests bearing on that claim, whether in the citing work itself or in other work? Does a citing work adequately assess the empirical evidence, or merely superficially and uncritically report empirical findings? As Cozzens (1985, p. 132) asked, “When a claim is cited, are the criteria for judging its solidity mentioned along with it in the text?…Or is it merely taken at face value?” A look at virtually any published article will show many cases where authors cite empirical findings without saying anything about the level of rigor present in the research that produced that evidence. Anderson (2006) suggested that scholars citing Weick (1979) primarily accepted his assertions without question. Our work (Anderson & Lemken, 2019) found that very few citations to March and Simon’s (1958) Organizations mentioned empirical tests of their assertions, which was surprising given the repeated and explicit emphasis these authors placed on the need for empirical work to test their ideas.

In organization studies, there is an increasing recognition that scholars need to assess the quality of evidence regarding important theories and assertions (e.g., the evidence-based practice literature; Rousseau et al., 2008). The growing concern over the paucity of replications in social science (Tsang & Kwan, 1999) and the lack of replicability of many findings are other indicators of the need for scholars to more rigorously examine the quality of the evidence that authors assert in their studies. This is a phenomenon in the natural sciences as well (e.g., Thomaz et al., 2010). While meta-analyses are enormously useful for aggregating research when there are a sufficient number of empirical tests on relationships, the vast majority of claims cited by authors are not based on meta-analytic reviews. Citation context analysis can help reveal whether assertions have been tested or not, which can point out relationships that need further empirical testing.

Question 5: How Many of the Citations to the Works Are Critical of Their Knowledge Claims, and What Are Those Criticisms?

Another defining feature of science is the belief that its knowledge claims are supposed to be subjected to refutation attempts. Indeed, the philosopher of science Karl Popper (1934) even held that attempts at refutation should be the primary goal of scientific work, because falsification produces knowledge, while attempts at verification can never be decisive in proving a theory is correct. In their discussion of literature reviews, Baumeister and Leary (1997, p. 318) argued that “providing a critique of the evidence is an integral, even a central part of the job of reviewing literature.” Discussions of the goals of literature reviews have stressed the importance of critical analysis, including identifying “any deficiencies, omissions, inaccuracies, and other problematic aspects of the literature” (Torraco, 2016, p. 420).

Beyond assessing the extent to which citations to a work mention empirical findings, another important issue is how often citing work is critical of the works cited. Despite this centrality of critique to the institution of science and to literature reviews in particular, there is evidence that scientists critique each other’s work far less than some might believe. Citations that challenge the validity of a cited work are called “negational” or “refutational” citations (MacRoberts & MacRoberts, 1984; Moravcsik & Murugesan, 1975). Prior work has found that negational citations are quite rare. Studies of work in numerous fields (including management, communications, immunology, and physics) have revealed that only a very small percentage of citations are negational (Anderson, 2006; Burton-Jones, 2014; Case & Higgins, 2000, p. 640; Catalini et al., 2015; Moravcsik & Murugesan, 1975; Stremersch et al., 2015). Indeed, the frequency of negational citations reported by Stremersch et al. (2015) was only 0.6%, for example. Our citation context analysis of March and Simon’s (1958) Organizations found only 10 refutational contexts among the 2,407 contexts examined—less than 1% (Anderson & Lemken, 2019).

Precisely what constitutes a negative citation is disputed in the literature (White, 2004). There are a variety of rhetorical forms that negational citation contexts can take, ranging from vague suggestions that a cited knowledge claim is unsubstantiated to more explicit criticisms that a cited knowledge claim is erroneous (see Danell, 2012, pp. 318-319). Importantly, as White (2004, p. 101) noted, “Negative citations do not necessarily indicate worthlessness; they often simply show controversy. The negative citers may themselves be wrong and the citees right.” For the purposes of a literature review, of course, the key issue is not what percentage of citations are negational, but rather what specific criticisms citing authors have made about the claims in the focal work. Because these negational citations are scattered throughout the citing documents, collating them in a citation context analysis can serve a very valuable function for theory development. Reviewing the full range of negational citations can point to qualifications of the knowledge claims of a source work that readers might otherwise be unlikely to encounter. Critical citation contexts can also highlight aspects of a work that need further theorizing and empirical testing, which can significantly alter the interpretation of a given work or even illustrate why the assertion is incorrect.

Question 6: Are Important Concepts from the Focal Work Neglected or Distorted by Citing Authors?

Citation context analyses can also examine whether citing work has neglected certain knowledge claims in a work, and the extent to which knowledge claims have been distorted by citing authors. Small (1978, p. 338) “stressed the importance of viewing citations as interpretations of cited works,” and noted how in some cases, “the process of becoming public property has transformed the document into something the author may not have intended.” There are a variety of citation context analyses that have looked at these questions. For example, Burton-Jones (2014) argued that subsequent work in information systems had not fully appreciated the theoretical richness of Zuboff’s (1988) classic book. Anderson (2006) mentioned a variety of interesting concepts in Weick’s (1979) book that had not received many citations. Coleman and Salamon’s (1988) analysis of citations in psychology to Kuhn (1962) found that few authors had incorporated Kuhn’s later clarifications about the assertions he made in his primary work, with the implication that citing authors were misrepresenting his views and perhaps caricaturing his arguments. Hansen et al. (2006) examined citations to an influential article that Markus (1983) wrote in the field of information systems, and argued that citing authors had simplified her arguments in ways that distorted them. Finding that authors distort the ideas in cited works has obvious implications for theory development, because misguided interpretations of prior theory mean that new theories may have spurious foundations and assumptions or make improper use of empirical evidence provided by cited work. Any of these issues stemming from the distortion of cited works can hinder theoretical progress.

Question 7: Do Citations from Diverse Fields Cite Different Concepts from a Focal Work or Interpret and Use the Cited Material Differently?

A variety of citation context studies have examined the extent to which seminal works have had interdisciplinary influence as one indicator of their overall breadth of influence, frequently finding that works are cited by articles from a great diversity of fields (Chang, 2016b; Lee, 2010). Some research has also compared whether work in different fields cites a work differently (Chang, 2013b; Cozzens, 1985; McCain & Salvucci, 2006), such as by emphasizing different knowledge claims. A variant of this question is whether different geographic audiences cite the knowledge claims differently. Anderson (2006) examined whether citations to Weick varied in American versus European journals and noted several differences.

Most works are likely to be cited primarily in one or two disciplines. For example, Chang (2016b) found that outside disciplines rarely cited Zipf’s (1949) work on human ecology and the number of citations from those disciplines summed to be less than 4%. This means that while a work may be cited in other disciplines, its actual impact on those disciplines may be small. But some works do have an extensive influence across many diverse fields. Writing about the history of science, Garfield (1985) found that the work he examined, “Little Science, Big Science” by D. J. S. Price (1963), was cited by more than 80 disciplines and specialties, while McCain and Salvucci (2006) found that authors from more than a dozen disciplines cited the work in information science they examined, “The Mythical Man-Month” by J. P. Brooks, Jr. (1975). This type of analysis can show the breadth of influence of a cited work. Understanding the extent of interdisciplinary impact for a work and how a work is cited differently across fields can aid theory development by identifying possible bridges between knowledge claims in different areas of scholarship, as well as boundaries and differences in assumptions across fields.

Question 8: What Works Are Cited in Conjunction with the Focal Work?

For almost any given point that a researcher wants to justify or support with a citation, there will be a range of potential works that could be cited. An eighth research question that citation context analyses can answer concerns identifying the other works that are frequently co-cited with the focal work, with the implication that these works presumably have some degree of similarity with the focal work, at least in terms of the content or reasons why the citing author uses them together. Small and Greenlee (1980, p. 294) go so far as to label works that are frequently co-cited as “redundant documents,” and note that “this redundancy results in most cases from authors’ citing two or more works at the same point in their texts suggesting to the reader that the cited works are equivalent in some sense.” While works cited together may represent a degree of redundancy in some cases, they can also identify the set of works that constitute the network of important scholarship regarding a topic. This can aid future theory development by pointing out the body of relevant works in an area.

For a given work, the “list of the authors most frequently co-cited with it (and who might therefore be assumed to work on problems that are perceived to be most closely or strongly related to those treated” by the work) is called the “citation image” of the work (Furner, 2003, p. 192). For example, Burton-Jones (2014, p. 83) contains a table with the five works most frequently co-cited with Zuboff’s (1988) book. Documents that are co-cited represent one way a researcher can rigorously and quantitatively identify works that belong together and constitute a field. Although many researchers have constructed co-citation networks at the level of the reference sections of articles, a more illuminating and defensible approach might be to examine networks of documents that are co-cited in citation contexts, as these works are cited for the same purposes within the citing documents.

In presenting the above list of research questions that citation context analysis can address, we do not claim to be exhaustive in identifying all potential questions. Instead, our purpose is to identify questions that have been addressed in existing studies and that we believe are particularly important in contributing to a literature review. We encourage researchers to develop new questions that this method can answer. Current research using citation context analysis has merely scratched the surface in exploring the “genealogy” and “evolutionary” progress of seminal theories and concepts in organizational studies. Future work could help to reconstruct the genetic make-up of theoretical building blocks in management and other disciplines, and explore how ideas spread and change over time and across generations.

Step 1: Determine the Domain and Research Questions to Address

As with most types of literature reviews, the first step is to decide on the domain of the review (Siddaway et al., 2019). The purpose of this step is to define the source work or works considered and referenced in citing works. In the case of a citation context analysis, the domain will typically take the form of a seminal work that the scholar wants to examine in terms of its influence on subsequent scholarship. The domain could also be multiple works by a single author (usually a luminary in the field), or work on a particular concept. One needs to justify the importance of the work(s) or author(s) to be investigated through the examination of citations.

The researcher must then select the specific research questions to address in the review. These questions will generally be a subset of those listed in Table 1. The questions selected before the analysis form the evidentiary framework for the main arguments of the literature review. It is important to frame the primary insights to be gleaned from the analysis and thereby help to specify its scope. Naturally, as with other types of reviews, the researcher may want to begin by examining a larger set of questions and then emphasize a subset of questions in the final paper, depending on which results are most interesting and provide the most insight regarding the realized impact of the work. The goal of this step is achieved when the researcher has settled on a cohesive set of questions to be addressed with a clear rationale for each question.

Step 2: Establish the Boundaries for the Review

As in all literature reviews, the researchers need to decide how to bound the review. It is important to remember an issue mentioned by Cooper (1988, p. 114): “whether a literature review can ever by truly exhaustive. All authors of reviews must necessarily exclude a multitude of work that lies near the boundary of their problem domain, works that other reviewers might choose to include.” For certain works or topics, comprehensiveness might be a reasonable and feasible goal. Our experience with this method suggests a comprehensive approach is feasible if the number of citations to the focal work is less than 1,000. For other topics, where the total body of citing works is extensive (e.g., classic books that have been cited tens of thousands of times), it is necessary to narrow the domain in some way. In existing studies that have examined highly cited works, for example, researchers have limited the scope of their analyses to specific sets of prestigious journals from a particular field of study (e.g., Anderson, 2006 looked at citations in three organization studies journals; Anderson & Lemken, 2019 in eight organization studies journals; Burton-Jones, 2014 in nine management information science journals). For less-cited books and journal articles, it may be feasible to examine the entire corpus of citing works, or at least all citing literature available on searchable databases like Web of Science, Google Scholar, or similar repositories of scholarly works (e.g., Anderson & Sun, 2010). In such cases, the researcher may still want to limit their search to citation contexts in published works as opposed to all available works (as in systematic reviews that require comprehensive searches to identify evidence). Another possible approach would be to use a sampling method if the researcher’s goal is to create a representative selection of citations as opposed to focusing on all citations within a discipline. The purpose of this step is to provide a realistic set of boundaries for a citation context analysis to guide the creation of a rigorous but practical research design.

Typically, researchers examine only citations in research articles (as opposed to those in books, editorials, letters, book reviews, textbooks, practitioner articles, etc.). Authors have often excluded works that are not in English (Anderson & Sun, 2010; Chang, 2013a, 2013b; Cozzens, 1985; McCain & Salvucci, 2006), which is reasonable given that most scientific literature is written in English. ² If a study analyzes a book that has gone through multiple editions, then it is important to deliberately include either all editions (typically) or only certain ones (which may be reasonable if the book has changed dramatically; e.g., Anderson, 2006; Chang, 2016b).

If the analysis goal is to consider impact in a particular field, then identifying the top journals in that field is necessary (e.g., using the Journal Citation Reports or similar lists and databases). Beyond citing impact factors and appearance on lists of top business journals such as the Financial Times 50 list of premier business journals, there is little guidance regarding a standard for how many and what journals to examine. One approach is to see how many citations have appeared in various relevant journals, and then decide upon a reasonable standard for inclusion, such as all the journals containing more than 100 citations to the focal work (e.g., Anderson & Lemken, 2019).

Completing a series of simple searches using large, broad-based citation databases like Google Scholar, Web of Science, or ABI/INFORM allows researchers to understand the overall population of citations and then refine the scope for citation samples. It is worth stressing that there is nothing inherently right or wrong about deciding to bound the review to seminal journals if the researcher’s intent is to examine the dominant influence of a cited work, or to specific fields (e.g., management) or timeframes. Research designs that call for widely cited or multiple focal publications may require limiting the universe of citing sources to provide focus for the citation context analysis. For example, March and Simon’s Organizations has more than 25,000 citations according to Google Scholar—an untenable number for completing a comprehensive analysis of citation contexts. Comprehensiveness in literature reviews has typically been discussed in terms of conducting “systematic” reviews (Denyer & Tranfield, 2009; Siddaway et al., 2019; Tranfield et al., 2003), which at the extreme would also include “gray” literature that lies outside of academic peer-reviewed publications (Adams et al., 2017). In management and organizational studies, the most frequent “gray” sources are conference papers, doctoral theses, and working papers (Adams et al., 2017). Researchers should remain aware of the trade-offs between increasing the scope of a study and the required time and effort. We suggest that researchers use thoughtful criteria that lead to purposive samples rather than aiming for the widest coverage. Researchers should also consider other types of exclusions, like unpublished works—which would be biased in terms of those included on Google Scholar versus the complete universe of working manuscripts not included—or excluding books or work published in other languages that might require translation to be included in the analysis.

Step 3: Gather the Citation Contexts and Create a Citation Context Database

Large searchable databases like Google Scholar, ProQuest, EBSCO, JSTOR, Web of Science, and ABI/INFORM are excellent resources as a primary means of locating and collecting citing articles. However, they may not contain all articles in the intended sample frame, or they may list articles that do not in fact contain citations or contexts (or only include citations in tables, for example, which may or may not be deemed suitable for analysis—these typically need to be considered on a case-by-case basis). More specialized secondary sources like publishers’ databases can be useful for checking whether important citations from influential sources have been overlooked. The purpose of this step is to identify and assemble all the focal source documents and the citation contexts that satisfy the requirements identified in Step 2. The collection process in this step must be fully documented, noting unexpected barriers encountered or practical exceptions needed to fulfil the intended purpose and scope of the analysis.

Once the set of citing works has been identified, it is necessary to search the full-text electronic documents containing the entire article to systematically identify each individual citation context. In practice, usually around 20% or so of articles contain multiple citation contexts (Chang, 2013b, found that this number ranged up to 14 contexts in one article). The assumption is that more contexts imply a greater importance of the cited work to the articles. Zhang et al. (2013, p. 1500) wrote that “Voos and Dagaev (1976) suggested that the number of times a reference is cited in an article provides some indication of its relevance to the citing article’s subject.” Burton-Jones (2014, p. 79) found that articles citing a work in a cursory way had fewer citation contexts than those citing it in a detailed way.

Once the full sample of citing articles is confirmed, the same search function can be used to locate contexts within each document. Another option is to load all the full-text documents as sources into computer-aided qualitative data analysis software (CAQDAS) like NVivo or Atlas.ti to both locate all citation contexts and access functions that aid analysis. A simpler and more manual alternative is to search documents using “find” functions and copy and paste citations contexts into a more commonly used program like Excel that can be searched and sorted.

Selecting the appropriate blocks of text from the citing articles is not a mindless task that can be easily automated, but rather one that requires some degree of researcher judgment. Once a citation is located in the body of the article, the researcher needs to determine the appropriate citation context text block—that is, how much of the text surrounding the citation itself is needed to adequately code its meaning and content. How many adjacent sentences should be captured in a citation context depends on the nature of the citation. The key is to gather sufficient surrounding textual material to be able to adequately identify the content of the citation. CAQDAS tools can automate the text selection process but the results of automated selection will inevitably require adjustments. If a citation appears at the end of a paragraph, the relevant text block may require including the sentence and one or two preceding sentences. If the citation is in the middle of the paragraph, the appropriate block may require multiple preceding and following sentences. In many cases, particularly for peripheral citations, a single sentence may be all that is required. In other cases, especially for substantial citations, the researcher may want to capture an entire paragraph or more to fully understand the context in subsequent coding. Some authors have considered multiple citation contexts that are located in adjacent sentences or only one sentence apart as representing one citation context (e.g., Anderson, 2006; Anderson & Lemken, 2019; Anderson & Sun, 2010).

One final consideration is whether there is any reason to exclude particular works. For example, researchers may deem it advisable to exclude works that contain an excessive number of citation contexts, such as our (Anderson & Lemken, 2019) decision to exclude Kilduff’s (1993) extensive deconstruction of March and Simon’s (1958) classic book Organizations. Such sources may be better to discuss individually in a literature review as their sheer number of contexts will make them outliers that bias the results of a numerical analysis of citations. The researcher should be transparent in terms of decisions about how to treat such sources and the process for gathering and coding all citation contexts to improve the replicability of the analysis.

Step 4: Identify and Agree on Codes (or Categories)

Assigning citation contexts to descriptive codes requires multiple evaluators to read the citation contexts and discern the meaning of each citation within the context of the citing author’s argument. Codes are developed and assigned through the process of evaluators reading and making critical interpretations of the content of each citation to discern the differences and similarities between citations. Categories (represented by codes) are created with descriptive names that capture the essence of a group of contexts and highlight the differences between citations contexts. The differences can stem from varying uses of citations, the concepts referenced in them, the specific terms they contain, or other aspects of citations salient to the researcher’s goals for the analysis. Prior to developing the coding scheme, scholars should independently read all the citation contexts and take general notes about how citations are applied as well as the themes and knowledge claims that tend to recur (Anderson, 2006; Anderson & Lemken, 2019; Burton-Jones, 2014). This first reading is not to complete the task of categorization per se but will help evaluators avoid biases from the sequence of citation contexts or from the evaluator’s own prior work. After a first reading to reduce reflexivity and order biases, researchers can begin to assign initial codes, using either an a priori or emergent approach.

The first round of coding should be done independently using an open judging process to ensure objectivity of the citation context analysis (though, as with any coding process, there will always be some inherent degree of subjectivity). Coding with two or more independent evaluators requires researchers to use a combination of language taken from the contexts themselves and their own interpretive language to assign labels to each citation context. In the emergent approach, this step creates a separate list of initial codes for each evaluator. After reviewing each other’s lists of open codes, evaluators should meet to discuss and evaluate all the codes generated. Through discussion and debate, evaluators then combine and refine the lists of open codes, discussing any differences in individual judgments until they reach consensus about the emergent codes to use. The resulting single list of initial codes will identify the common themes and areas of content within the source articles. With an a priori approach, researchers begin with a common list of codes, though they may need to create new codes or modify existing ones during a collaborative coding process. As in the emergent approach, this stage should be followed by a round of open discussion where initial coding results are shared to identify and reconcile differences in how codes are applied or what codes need to be added or dropped. The purpose of this step is to create a final list of open codes to act as a classification scheme of content categories. This may be an iterative process requiring more than a single round of coding to attain a consensus concerning a final list of categories (McCain & Salvucci, 2006).

Step 5: Assign Codes to All Citation Contexts

Evaluators then complete a new round of coding to act as independent judges who apply the single list of refined initial codes created in Step 4 to all the contexts. Once recoding of all contexts is complete and agreed to, each evaluator can use the descriptions of the initial codes and examples of the contexts they represent to create a smaller set of broad-based categories referred to as summary codes (Saldaña, 2015). This next level of coding also requires one or more rounds of comparison, discussion, reconciliation, and recoding to reach consensus among evaluators. Since the purpose of the citation context analysis is to provide an objective assessment of how the focal work is used in citing documents, the summary codes should reflect both the textual commonalities within the citation contexts from different initial codes as well as the thematic similarities of the initial codes themselves. For example, in the analysis performed for our study of March and Simon’s Organizations (Anderson & Lemken, 2019), the initial list of 24 open codes was reduced to 8 summary codes.

If the analysis addresses additional questions (see Table 1), like an assessment of the level of importance of contexts to the citing articles’ arguments, or whether contexts refer to empirical evidence, the supplemental assessments can be performed by evaluators during the initial coding stage or during the recoding of contexts using the final open-code list. Some categories will be more overtly homogeneous than others (e.g., one category might be for a particular quote which is often repeated; McCain & Salvucci, 2006), and researchers may want to have several subcategories underneath major categories, particularly for those categories of content that are cited most frequently. Researchers will often need to include a category for a small number of citation contexts that are not classifiable (e.g., Burton-Jones, 2014).

We suggest the most important things to assess in a literature review will typically be the content drawn upon from the cited work, the importance of the citation context to the citing work (i.e., whether the citation context is peripheral or substantial), whether it discusses empirical evidence, and whether it is critical of the focal work. But there are other possibilities that could be interesting depending on the researchers’ aims. Zhang et al. (2013) provide a brief review of citation categories to consider. The purpose of this step is to build an analytical framework for the analysis. Evaluators then use the framework to interpret the meaning of all citation contexts under consideration and formulate fact-based inferences to address the key research questions of the study.

Step 6: Conduct Intercoder Reliability Checks

Researchers should ideally perform interrater reliability tests after each independent coding step. One useful test for interrater reliability is Cohen’s kappa (Cantor, 1996), which is a calculation similar to chi-square tests of independence in cross-tabulations, and which measures the reliability of each coding step. When more than two evaluators are used in a coding step, Fleiss’s kappa may be a more appropriate measure of interrater reliability. This metric measures the reliability for a fixed number of evaluators using categorical assignments and determines the difference between random (expected) agreement and actual agreement in the analysis (Fleiss, 1981). NVivo 12 provides functions for calculating Cohen’s kappa and other agreement measures. There are also supplemental analytical packages available for Excel that automate agreement measures. The purpose of this step is to check for bias and error in the analysis. Presenting the results of objective measures of reliability increases the confidence of readers as they judge whether a qualitative analysis provides a fair representation of the base of evidence.

Step 7: Report the Citation Context Analysis Results

Authors will face numerous decisions when formulating their report of results, and it is important to document and justify these decisions. For example, examining changes over time requires defining appropriate time intervals to compare. McCain and Salvucci (2006) used 5-year time periods, Chang (2013a) used 3-year periods, and Burton-Jones (2014) split his sample into two 12-year time periods. While these may seem like arbitrary decisions, each was documented and explained given the scope and aims of the study. When specific research questions about the nature of citation contexts are part of the research design—like determining if contexts are peripheral or substantive to the citing author’s arguments or if empirical support is offered—authors should provide examples of citation contexts that exemplify the categories used. Such examples give the reader a clearer understanding of the decisions made by the researchers, and thus more confidence in the adequacy of their findings.

The purpose of this step is to clearly summarize and bring credible meaning to what may be a wide-ranging analysis that covers hundreds of source documents and thousands of citation contexts. In answering multiple research questions, researchers should use the results of the analysis to assemble a practical, cohesive, and compelling view of the implications of the study. Citation context studies should strive to go beyond merely counting the key concepts from a work that are cited by other works, or tabulating how frequently key knowledge claims are cited in different disciplines (as a few studies we reviewed merely did), and use citation context analysis to answer broader questions that aid theory testing and theory development (see Lee, 2010, and Burton-Jones, 2014, for nice examples). As William Bruce Cameron (1963) said, “Not everything that can be counted counts, and not everything that counts can be counted.” Journals prefer reviews that go beyond summarization of the literature to synthesize and add unique and practical insight (Jones & Gatrell, 2014). Table 2 summaries these seven steps.

OPEN IN VIEWER

Table 2.

Summary of Guidelines for Conducting a Citation Context Analysis.

Step 1: Identify the Domain of the Review and Determine the Research Questions to Address	▪ Select one or more focal books, articles, or authors. Alternatively, the domain could be defined by a theory, concept, or research method. ▪ Decide the specific research questions that the study will examine (see Table 1 for eight possible questions).
Step 2: Establish the Boundaries for the Review	▪ Determine the degree of comprehensiveness of the analysis. If the number of citations to the work is large, consider limiting the citing sources that will be examined, such as to those appearing in top-tier journals, or by limiting the timeframe to be considered. ▪ Sample sizes for citation context analyses vary widely. The largest we found in the literature is Chang (2016b), with 2,419 citing articles and 3,017 contexts, while the smallest is Mizruchi and Fein (1999), with only 26 citing articles.
Step 3: Gather the Citation Contexts and Create a Citation Context Database	▪ Identify all the citing articles that fall within the established boundaries. ▪ Search these citing articles to locate each independent citation context (the passage containing the citation), and decide how much surrounding text is needed to adequately code the meaning of the citation context (typically 1-2 sentences before and after the sentence containing the citation).
Step 4: Identify and Agree on Codes (and Categories)	▪ Before beginning the coding of the citation contexts, each researcher should independently read all the focal citation contexts to identify recurring themes or knowledge claims. ▪ Independently code each of the citation contexts using either an a priori approach based on predetermined categories or an emergent approach that uses a combination of the actual language in the citation contexts and the researcher’s own interpretive language to create initial codes (or descriptive labels) for all contexts. ▪ The initial coding step creates two or more lists of codes with associated contexts. After reviewing each other’s lists of open codes, confer to combine and refine the lists of codes and create a single authoritative list that provides the basic descriptors of the contexts. This is an iterative process.
Step 5: Assign Codes to All Citation Contexts	▪ Have each evaluator independently assign the refined list of initial codes to all the contexts again, which will facilitate subsequent interrater reliability checks. ▪ If the analysis will include additional assessments (Questions 2-8 in Table 1) beyond evaluating the content from focal works used in citation contexts (Question 1 in Table 1), these supplemental assessments can be performed by all evaluators during the initial coding stage or during the recoding of contexts using the final open-code list. ▪ Using the finalized initial codes, their definitions, and examples of contexts that fit each code, evaluators then independently review the initial codes to summarize them and create a smaller set of broad-based categories (also called summary codes).
Step 6: Conduct Intercoder Reliability Checks	▪ Perform interrater reliability tests after each independent coding step, such as by calculating Cohen’s kappa or Fleiss’s kappa.
Step 7: Report the Citation Context Analysis Results	▪ Decide how best to report the results of the analyses, typically in the form of tables. ▪ Strive to provide insight that goes beyond mechanical descriptions of what was found, and to articulate implications for future research, especially implications for theory testing and development.

The Potential Role of Citation Context Analyses in Theory Development

Citation context analysis is a method for reviewing literature that can contribute to the essential tasks of theory building and theory testing. In the preface to the Post-Italianate edition of his book On the Shoulders of Giants—A Shandean Postscript, Robert K. Merton wrote, “The difficult trick in the art and craft of science is to exercise discipline while still obeying one’s daimon” (Merton, 1993, p. xxiii). Merton calls for the balance of disciplined scientific inquiry and creative energy (i.e., one’s daimon). As a tool in theory building, citation context analysis can enforce the discipline of faithfully tracing the path by which key theories have (or have not) evolved. Only by systematically evaluating how key theories are applied can we determine their adequacy as reliable foundations for new theories. Citation context analysis can uncover the lack of objective evidence needed to undergird important theories and reveal instances where prior theories are haphazardly or even incorrectly applied to new theory building efforts. To genuinely “stand on the shoulders of giants,” scholars need methods for objectively confirming both how well important ideas have been substantiated and how accurately they have been applied. Otherwise, we risk losing important threads of great ideas and the ability to verify the provenance of new theories that are meant to expand and extend our knowledge.

Weick (1989) conceptualized theory building as the generation of conjectures about ways to address problem statements. He stressed that “a theorizing process characterized by a greater number of diverse conjectures produces better theory than a smaller number of homogenous conjectures. The key property is heterogeneity among thought trials” (Weick, 1989, p. 522). By discovering and characterizing the genealogy of foundational theories, citation context analysis can provide the variety and independence of thought trials that Weick advocates, and reduce the risk of being constrained by habituated ways of thinking. Since new descendant theories are frequently built upon foundational theories, an unbiased and broader array of examples where a foundational theory has been applied in diverse arguments can provide the theory-builder with an important advantage.

Improving Citation Practices

Two key results of existing citation context studies are to demonstrate how our collective citation practices are far from optimal (Small, 1982) and, more importantly, to offer concrete ways to improve them. For example, traditional literature review methods do not rigorously identify all the criticisms of assertions made in cited works, and a citation context analysis can do this. Furthermore, as a means of locating empirical work, citation context analysis also has potential advantages over traditional literature reviews. Without a comprehensive approach to locating all the published research that addresses a specific hypothesis, it is very difficult to ascertain whether a traditional literature review incorporates the complete body of available empirical evidence that supports or refutes it. As Burton-Jones (2014, p. 90) noted, “A lack of studies that cite the Smart Machine for a given idea does not necessarily mean that the idea has not been studied elsewhere. After all, perhaps other researchers have done so quite independently and had no need to cite Zuboff.” In the case of systematic reviews that require rigid specification for searches, scholars have warned that “researchers can employ a non-standard array of terminology to refer to the same underlying constructs” (Rousseau et al., 2008, p. 503). We advocate that authors be more systematic in their production of knowledge as well, by clearly linking empirical findings to the works that first advanced the hypotheses they test (or similar ones). This specification would permit future analyses to more rigorously aggregate and examine the complete set of relevant findings and thus facilitate the testing and refinement of theories.

Hallmarks of a Strong Citation Content Analysis and Future Possibilities

In our review of 34 published articles using citation context analysis to review literature (see the appendix), we considered what distinguished citation context analysis articles that were more highly cited. We found that they presented a clear and detailed criteria for the sample used; included context-based descriptions of their methods and highlighted how the methods addressed the goals of the study; addressed a wider array (three or more) of relevant research questions; and studied a larger sample of citation contexts (300 or more). We believe these four characteristics are hallmarks of rigorous citation context analysis studies that both editors and reviewers should consider as guidelines for judging their quality.

In addition to the types of questions addressed above and in past studies, we envision at least two additional ways citation context analysis can be applied in future studies. One application would be to review all the citations regarding a specific theory or domain (e.g., Greenberg, 2009; Siontis et al., 2009), particularly if that theory is embodied in a small number of texts frequently cited in discussions of the theory. This form of comprehensive literature review would be valuable in debating and resolving equivocal interpretations of evolving theories. Another potential application of citation context analysis is to find and address persistent methodological issues. For example, researchers could examine the body of citations related to a specific research method or methodological rule-of-thumb to identify instances where research practices are misapplied, arbitrary, or overly simplistic (such as Cronbach’s alpha needing to be greater than .70; Meuer & Rupietta, 2017). Lee (2010, p. 118) noted that citation context analyses “of methodological works are notable for their absence.” He examined the diffusion of Robert K. Merton’s “focused interview” method, which was the precursor to the widely used focus group method. Beyond this example, few studies have addressed how scholars cite methodological works. Perhaps one reason is that methodological works also tend to have the most consistent meanings as concept symbols—that is, their “uniformity of usage” is higher (Small, 1982). Future work is needed to examine this possibility, particularly since “there is some evidence that the scientific writings with the highest citation counts and the most longevity over the years tend to convey not theory or empirical results but methods” (White, 2004, p. 104).

It is worth noting that the widespread accessibility of increasingly powerful computers, and the development of new technology focused on the processing and analysis of text-strings, promises to make using citation context analysis as a method of conducting rigorous literature reviews faster and easier. A variety of efforts are underway in the fields of scientometrics and computer science to automate the identification of citation contexts (e.g., Bertin et al., 2016; Jha et al., 2016), and these will undoubtedly continue to progress.

In closing, citations are key indicators of the building blocks used in the construction and interpretation of new scientific insight. We argue that citation context analyses represent a novel form of literature review that has enormous promise for advancing our understanding of the impact of seminal works and for rigorously examining how their ideas and findings have been understood and built upon by subsequent scholarship. The use of citations in calculating impact factors has thus far had an enormous impact on scholarship (Zupic & Čater, 2015), as well as on the social practices of scientists (e.g., in the form of evaluating the reputations of journals and the contributions of scholars for evaluation purposes). We believe that a more complete, thoughtful, and rigorous analysis of the citations themselves—through the use of citation context analysis—has a similar potential to significantly and positively impact theory development and how social scientists conduct their work.