Abstract
The Web of Science citation indexes were originally intended to serve as research aids, to provide easy-to-use bibliographic aids for authors, help authors identify colleagues who have cited their work, and assist librarians in making selections among journals. But they were soon carried by the tidal waves of scientism and data-ism, first in business management and governance, and then also in scholarly research, to near-monopolistic control of the business of journals evaluation in the United States (though increasingly challenged in some areas by the more recent but similar Scopus citation indexes). With that dominance, earlier tentative generalizations based on limited research gradually became more and more rigidified “laws” that have been strictly enforced: that quality can be scientifically measured by the number of articles that cite the article or a particular journal, and, by extension, that the importance and contribution of a scholar’s article, like that of a journal, can be determined by its “impact factor” measured by counting the number of articles citing it. Those “laws” came to be applied first to the natural sciences, extended to the social sciences, and finally also to major spheres of the arts and humanities. Today, they have come to dominate the entire continuum of disciplines and fields ranging from the most universalist of the natural sciences, in which truths may be established by reproducible experiments, to the more particularist social sciences, and still more particularist arts and humanities, in which theories, even facts, are far more contested and tentative. As we move across the spectrum from the more universalist end of natural sciences toward the more particularist end of the social sciences and arts and humanities, such methods have tended to violate ever more the fundamental nature and realities of scholarly research. However, once entrenched, the citation indexes business has shown the same tendencies as any monopolistic entity toward resisting change and transparency. Where those tendencies have been adopted by a centralized government for bureaucratized control, as in China, the misuses and abuses of citation indexes have been further magnified. This article ends by calling for developing more substantive, genuinely peer-review-based methods of evaluation; for relying more on alternative nonprofit bibliographic and data services; and for greater inclusivity, especially with regard to scholarship in languages other than English.
Keywords
Historical Overview
Eugene Garfield, Thomson Reuters, and Clarivate Analytics
The person most responsible for the development of the Web of Science citation indexes is Eugene Garfield (1925–2017), an imaginative, innovative, and entrepreneurial “information scientist.” 1 Not a professional scholar, Garfield is known principally for founding the Institute of Scientific Information in 1955, 2 which first developed the Science Citation Index in the 1960s. The index turned out to be quite a business success. On October 17, 1978, at the dedication of the new company building (which cost a reported $6.5 million at the time) in Philadelphia for its international headquarters, Garfield announced that his enterprise, at that time with 470 employees and offices in nine countries, would expand until the new building housed a thousand employees. Anchored mainly on the Science Citation Index at the time, the company already enjoyed gross sales of $15 million (with Garfield himself owning 65% of the company). Those facts prompted William J. Broad (later two-time Pulitzer prize winner as a New York Times reporter), writing for the prestigious journal Science (of the American Association for the Advancement of Science), to title his article on Garfield and his company “Librarian Turned Entrepreneur Makes Millions Off Mere Footnotes” (Broad, 1978).
Subsequently, in 1992, the institute was sold to the Thomson Corporation (that had been engaged mainly in newspaper publishing), which in 2008 combined with Reuters (that had begun mainly in the business of transmitting stock market quotes) to form Thomson Reuters, with Garfield’s Institute of Scientific Information forming the company’s Intellectual Property and Science Division. In 2016, that division was acquired by the Onex Corporation (a private equity fund, with $23 billion under management) and Baring Private Equity Asia (which mainly invested in Asian real estate and other “alternative assets”), to form the new Clarivate Analytics (CA) company, with “more than 4,000 employees, and offices in more than 100 countries throughout the world” (“Acquisition,” 2016). Clarivate Analytics today owns the Web of Science (WoS), including its three “flagship Citation Indexes”: the Science Citation Index Expanded (SCIE), the Social Science Citation Index (SSCI), and the Arts and Humanities Citation Index (AHCI).
As is well known, those indexes today dominate the business of evaluating academic journals in the United States, by rank-ordering them by their “impact factor” based on counting the numbers of citations they garner and are widely relied on by librarians in selecting journals for their institutions’ collections. The CA company’s dominance of that market, which is crucial to the financial viability of any journal, 3 in turn has made it extraordinarily influential throughout the academic world. Its citation indexes determine which journals of any given field are considered the best and most important “core journals,” and its measurement of the impact factor of the journals in which authors have published has come to be widely used by universities even in the evaluations of faculty for promotions and hiring.
WoS has in recent years been challenged by competition from Scopus (launched by the Dutch Elsevier Company in 2004), which has come to be widely used in Europe, the Middle East, and Africa, because of its greater coverage of scholarship in those areas and greater inclusiveness of non-English-language journals, especially in the arts and humanities, though Scopus remains essentially similar to the WoS in its exclusive emphasis on impact factor measurement according to the number of articles that cite a particular article and a particular journal’s articles. 4 There is one great difference between the two, however: Elsevier, the company that owns Scopus, is itself also a major publisher of journals (about 2,500 of them). That creates a conflict of interest that makes Scopus perhaps even more prone to monopolistic behavior than Clarivate Analytics, which does not engage in journals publishing itself, only in the business of evaluating them. Regardless, WoS remains the major company, especially in the United States, which occupies the largest part of the market of institutional subscriptions.
The Main Organizing Ideas of WoS Citation Indexes
A review of Garfield’s seminal and influential book, Citation Indexing—Its Theory and Application in Science, Technology, and Humanities (Garfield, 1979), parts of which are autobiographical, shows the author’s main concerns and ideas. There was the ideal of service, to help “scientific” research with “scientific information” by the systematic gathering of citation data. Garfield argued that those would facilitate bibliographic searches by researchers, help advance the cause of scientific research, even help map out patterns in the advancement and transmission of science and, of course, also help guide librarians in selecting journals.
We can also see from that original book a keen entrepreneurial mind at work. Garfield early on paid special attention to the “cost-effectiveness” of citation indexing: since in those days every item needed to be processed and entered by hand, ways had to be found to make production of the indexes financially viable and profitable. Those considerations led him to the formulation of what he (and his followers after him) would term “Garfield’s law of concentration”: namely, that the great majority of all citations are to a relatively small number of journals (75% of all citations, Garfield asserted, could be found in a core of a thousand-odd journals, and 84% in just two thousand journals, citing the Science Citation Index data for 1969 compiled by his company—Garfield, 1979: 21). That means that citation indexing can and should be highly selective and focus on just the most important journals as determined and ranked by numbers of citations, and that little would be lost by excluding the large numbers of journals that have only a low impact factor (22–24). That notion would form the core principle of the business of citation indexing to this day. It was at first applied only to the natural sciences (formally starting in 1965), then also to the social sciences (in 1973) and, finally, also to the arts and humanities (in 1978) (16–17).
Always upbeat, Garfield expressed great optimism with regard to the future of citation indexing, an exuberance that is conveyed throughout the book. But even he may never have imagined the extent and power of the information technology revolution to come, and the tidal wave of the rise of the age of application of ostensibly scientific methods and digital data to management (most concretely shown in the rise of the master’s degree in business administration, the MBA) that would sweep across many spheres of human endeavor, not only in business but also in governance, and in the end, also in university and research management.
Currently Operative Principles of the Web of Science
Currently, the official CA policy for the WoS is explicated in the essay “Journal Selection Process” by James Testa, vice president of Editorial Development and Publisher Relations (Testa, 2016). Quoting “Garfield’s Law of Concentration,” the document asserts centrally that “the core literature for all scholarly disciplines may be concentrated in a relatively small number of journals.” What CA’s “citation indexes” and the Journal Citation Reports do is provide precise and objective data (“scientific information”) for discriminating among the increasingly large numbers of journals.
The essay outlines the present-day requirements set by CA for admission to its three flagship indexes. CA apparently processes 3,500 journals a year, of which it selects only 10% for inclusion in its Web of Science indexes (Testa, 2016: 4–5). The basis for selection is, we have already seen, above all Garfield’s law of concentration; for inclusion in the indexes, a journal must demonstrate its worth by the number of citations it garners—in the journals already indexed by CA’s WoS.
While the document sets forth numerous requirements that are commonsensical and noncontroversial, such as regularity (“timeliness”) over the course of several years of publication, use of a peer review process, open and transparent guidelines, and so on, it also makes the striking assertion that “English is the universal language of science,” an assertion clearly intended to apply not just to the natural sciences but also the social sciences, and even the arts and humanities. It makes clear that for all those disciplines and fields of study, English is considered the most desirable when it comes to inclusion in the indexes. Some allowances are made for exceptions: not all journals must have full-text English. But even so, “all journals must have cited references in the Roman alphabet,” and all must “have English language bibliographic information” (Testa, 2016: 7) to be included. Those last requirements, it will be seen below, turn out in practice to be highly exclusivist, especially when it comes to “regional journals” that focus on particular non-English-speaking countries, regions, or localities, and their arts and humanities. Before turning to that question, we need first to consider just how scientific and rigorous the citation indexes and the impact factors they measure really are.
Critiques of the Citation Indexes and Impact Factor Measurement
Qualitative Criticisms
As William J. Broad writing for the renowned journal Science in 1978 already pointed out, the problem with such a logic and method is that they tend to favor the well-established conventional wisdom of a discipline or field over the unconventional and innovative. Broad quoted at length from sociologist Jon Weiner, who wrote: “Among the 3200 indexed journals (listed in 1974), selective coverage is given to such unlikely titles as Mosquito News, Soap Cosmetics, Digestion, and the Tasmanian Journal of Agriculture, but there is no coverage at all of journals like the Review of Radical Political Economy, Radical America, Socialist Revolution, Telos, Insurgent Sociologist, Working Papers for a New Society, or Monthly Review, to name a few” (as cited in Broad, 1978: 855–56).
New discoveries, we know, often take time to become accepted. But in the present scheme of things, those may well be rejected by the most established journals that follow the dominant paradigms in the natural sciences, the most influential models or theories in the social sciences, and the prevailing interpretations in the arts and humanities. Alternative views and journals may well be unable to gain acceptance. In addition, in the social sciences and arts and humanities, the most outrageous interpretations will sometimes draw the largest number of citations, not because the researchers believe they are true or well done, but rather because they often serve as useful foils to clarify or stage the author’s own analysis. That is a phenomenon that Garfield termed “negative citations,” which he acknowledged to be pervasive, but which he never really attempted to deal with, perhaps for reasons of “cost effectiveness” (Garfield, 1979: 244–45). That neglect remains true of the citation indexes today.
Multiple authors is another problem that Garfield acknowledged and discussed (Garfield, 1979: 242–43), but which, again, he and his company never seriously dealt with—again perhaps in part because of “cost-effectiveness” considerations. The WoS citation indexes to this day count only “the lead author,” a practice that has been adopted with a vengeance by academic administrators in China in personnel reviews and appointments. Yet we know that in scholarly practice, researchers often list authors in alphabetical order, and multiple authors are more often than not genuine coauthors who contribute equally to the article. But the WoS has not made any serious effort to deal with this problem. Its net effect, given CA’s predominance, is actually to discourage scholarly collaboration, since the product will only be counted for “the lead author.” Of course, it can also lead to the misuse of multiple authorship as mere form, something widely evident in China today.
More importantly, as noted above, the spheres of natural science, social science, and arts and humanities in fact range across a spectrum from the “harder sciences,” with established paradigms, to the “softer” social sciences and the arts and humanities, inevitably with different conceptual orientations, or “theories” or political ideologies, which are far more contested. Human knowledge spans a continuum that ranges across the more incontrovertible to the more contested, the more universalist to the more particular, the more generalized to the less. High-quality scholarship, especially in the social sciences and arts and humanities, comes in many different perspectives and approaches. Genuine knowledge in fact requires attention to the twin dimensions of the theoretical and the empirical, the universal and the particular, “globalized” as well as “local knowledge.” 5 And good research is above all about the convincing linking of relatively generalized (but empirically delimited) analytical concepts to specific and solid empirical evidence. It is simply foolhardy to imagine a single world of scientism in which there is only one standard and in which all disputes can be settled by laboratory experiments that can prove incontrovertibly whether a theory or fact is correct or incorrect. 6
Yet the WoS indexes operate precisely by such a model, based on the most unified and uncontested of the natural sciences (such as physics). That scientistic attitude is conveyed by the name chosen for Garfield’s company, the Institute of Scientific Information, and also by the order in which the three indexes of the WoS were created: beginning with the Science Citation Index, followed by the Social Science Citation Index, and then the Arts and Humanities Index, based on the same principles and methods. The very name “Web of Science” to encompass all three indexes conveys the same message.
The central idea of “scientific information,” “citation index,” and “impact factor” is to produce numerical guides, with the objectivity, absoluteness, and precision of mathematics, for the evaluation of all of scholarly research. It is something that has been propelled by the widespread popular acceptance and belief in the universality and absoluteness of “science”—in reality a crude scientism in its presumption that science can provide answers to everything. That being the case, it is not surprising that such an approach would threaten, even violate, the kinds of innovative creativity required in high-quality research, not just in the arts and humanities, but very much so also in the social sciences, and even in the natural sciences.
Criticisms from Information Scientists
The Web of Science citation indexes have been shown by information scientists to be far from rigorously scientific. As Larivière and Sugimoto have shown convincingly in their weighty new study, there is great variance among articles of the same journal in terms of the numbers of citations they engender. Examining several journals in depth and then extending their analysis across all the journals listed in CA’s 2016 Journal Citation Reports, they demonstrate how in the great majority of cases a small proportion of articles in any journal account for most of its citations, and that only about 30% of all articles in a journal have numbers of citations equal to or greater than the journal’s impact factor. Which is to say, the mere fact of an article being in a particular journal is no indication that it will draw the number of citations that is suggested by that journal’s impact factor, and hence of the presumed quality and importance measured by that number (Larivière and Sugimoto, 2018: 12–14). That is a false presumption that is commonly made when academic administrators leap from the impact factor of the journals a scholar has published in to judgments about that scholar’s quality and importance.
In addition, Larivière and Sugimoto (L & S) demonstrate great variance among disciplines in terms of the rapidity with which new works come to be cited. CA’s widely influential Journal Citation Reports, the major tools used by librarians in selecting journals, provide citation information on the impact factors of journals for the preceding two years. 7 However, L & S demonstrate that while in some disciplines there is a relatively short time lag between publication and citation, in other disciplines there tends to be a much longer time lag. Glänzel and Moed (2002) had shown, by comparing the science journal Lancet with the American Sociological Review, how the former, rated for its impact factor on a two-year window, had a far higher impact factor in the short term than the American Sociological Review (mean citation rate of 47.83 in 2016, compared with just 4.4, as L & S point out), but across a longer time period, the latter had a much higher average citation rate. Guided by that finding, L & S compared journals across a range of disciplines to demonstrate that, while biomedical research and physics citations tend to draw many of their citations within two years following publication, psychology and the social sciences citations do not, but instead tend to remain relatively stable across a longer period. L & S therefore undertook to examine citations across a 30-year period to show how a much longer time window was required for articles in psychology and the social sciences than for articles in physics and biomedical research to reach the point of 50% of their cumulative total citations in the 30-year time period. Clearly, the meaning of impact factors in the “harder” physical-biological sciences journals is quite different from that in the “softer” psychology and social sciences. It is very misleading to measure all disciplinary journals by a two-year time window (Larivière and Sugimoto, 2018: 11–12). But CA has persisted in such a practice to this day in its widely influential Journal Citation Reports that so many librarians rely on to make journal selections for their institutions.
Finally, L & S demonstrate, as others have also done, that CA calculates its journal impact factor by using as the numerator the total number of citations the journal receives, including not only citations to the “citable items” of research articles and book reviews, but also citations to the “non-citable items” such as news announcements, editorials, letters to the editor, and obituaries, which together account for an average of about 23% of the content of journals. But for the denominator, WoS uses only the citable items. The result is the tendency to seriously inflate the impact factor of journals whose contents include a relatively high proportion of “non-citable items”—an inflation by about 10–25% if the non-equivalent numerator and denominator were equivalent or “symmetrical” (rather than “asymmetrical”), as L & S demonstrate. They call this practice of the WoS citation indexes the “numerator/denominator asymmetry” in impact factor calculation (Larivière and Sugimoto, 2018: 6–8). CA has given no indication of plans to alter this practice.
Along the way of their detailed analysis, L & S recount a particularly notable example of organized opposition to the uses to which CA’s impact factor has been put: at the 2012 annual meeting of the American Society for Cell Biology, a group drew up a Declaration on Research Assessment, calling for “eliminating the use of JIFs [journal impact factor] for assessment of individual scholars and articles.” As of July 2017, that declaration had been signed by thirteen thousand individuals and nine hundred organizations (San Francisco Declaration on Research Assessment, 2012; see also Larivière and Sugimoto, 2018: 22). 8 The opposition to such uses of citation indexes, in other words, has been substantial and considerable, even if not sufficiently powerful and effective yet to compel CA to reform, or to turn the tide of using impact factor data to evaluate a scholar and his or her work.
Particularist Fields and Disciplines of Study
Garfield and the Use of “Scientific Information” for the Arts and Humanities
We have not, up to this point, focused specifically on the challenges to the uses of impact factor and citation indexes in the third major sphere of scholarship, the arts and humanities. In his 1980 article “Is Information Retrieval in the Arts and Humanities Inherently Different from That in Science? The Effect That ISI’s Citation Index for the Arts and Humanities Is Expected to Have on Future Scholarship,” Garfield defined the rationales for his company’s newly launched Arts and Humanities Citation Index. To his credit, he acknowledged the great differences between the two groups of disciplines, natural sciences versus arts and humanities: one is built on reproducible experiments, the other more on subjective judgments; one is populated mainly by articles, the other by books; one leans heavily on the most recent research and advances, the other on timeless pieces and works. His newly developed index, he claimed, would take these differences into account by providing greater time-depth in coverage and by including books. It would also take pains to standardize some idiosyncratic practices in those disciplines’ citations, including the provision of titles of the cited classical works in their original languages. It would aim to provide a genuinely useful tool for researchers (Garfield, 1980).
All well and good, but Garfield does not go on to address the principal problem: that most researchers in such fields do not, would not, trust in the impact factor to make judgments about the secondary literature they use. Almost all would rather rely on their own substantive judgments. Even the WoS itself has rejected the simplistic use of impact factor measurement to rank arts and humanities journals. For years now, the company’s Journal Citation Reports (which rank-order journals by their impact factor) have included only the natural sciences and social sciences, not the arts and humanities. CA itself officially acknowledged, in a statement on August 8, 2017, “The primary reason that there is no Journal Citation Reports edition for Arts & Humanities is because the key metric used in Journal Citation Reports, the Journal Impact Factor, is not an appropriate measure for Arts & Humanities publications.” More particularly, the document notes that books play a much larger role in arts and humanities than in the sciences, and that much longer time frames are needed to assess citations (Clarivate Analytics, 2017). Today, there is actually no consensus on how arts and humanities journals might best be evaluated, but such evaluations in fact go on, even if used only by librarians selecting journals, whether based on the WoS’s AHCI database, or the SCImago, based on Scopus’ database.
This area of “scientific measurement” of journal quality becomes grayer still if we consider also that WoS has now included significant numbers of “regional” journals under its SSCI citation index, many of which can be considered not only social science journals, but also arts and humanities journals (e.g., history journals). Thus, evaluations of many journals that include substantial arts and humanities content in fact continue on the basis of the impact factor measurement employed by WoS’s SSCI. And, even in CA’s own explanation of why it has excluded the arts and humanities from its Journal Citation Reports, it is evident that the company clearly still intends to develop “impact factor” measurements of journals in those spheres: “As the metrics offerings and Clarivate Analytics expand, we’ll have more data and analyses to bring to bear on the question of performance assessment in Arts & Humanities.” In other words, CA will look to longer time windows and more inclusion of books to make evaluations of journals in the arts and humanities, but still by counting citations (Clarivate Analytics, 2017).
Few if any arts and humanities scholars could have foreseen that things would develop in such a way as to give Garfield’s company virtually unchallengeable powers to grant inclusion or exclusion to a journal in almost all disciplines and fields, just by counting the number of citations it garners. Few could have foreseen that those citation indexes can determine the viability or not of almost all new journals, whatever their actual scholarly quality, by denying them inclusion in the indexes, and hence also their marketability—something that is largely determined by librarians who have come to rely so greatly on CA’s Journal Citation Reports in making selections. This issue of exclusivism becomes clearer still when we consider the posture that WoS has taken with regard to their demands for using English for “scientific research.”
English-Language Exclusivism
Garfield first developed the idea that English is the language for scientific research in his 1976 article based on data on 129 science journals in his Science Citation Index. He argued that the data demonstrated that French scientists in general cited English-language articles more than French, and that French science articles were cited mostly by French and not international scholars. While those assertions may well be true, and may well demonstrate that English was by then becoming the more important international language for science, what Garfield asserted categorically from the data is far more controversial: he once again resorted to the leap from “higher” or “lower” relative percentages to generalized evaluative judgments, as he had done in formulating his “Garfield’s law”: that “today French science appears to be in decline,” that “French science is too provincial,” that “the French scientific literature is generally of low impact,” and so on. The impact data of journals, he argued, show that “the United States is most highly cited by the outside world” (by percentages of total citations), and “the U.K. ranks next.” On that basis, he declared, “At the very least, all French journals should require the publication of summaries or abstracts as well as contents pages in English” (Garfield, 1976: 94). Today, those ideas have been operationalized by the WoS as “English is the universal language of science.”
Some readers may find such generalizations based on the counting of citations acceptable, and perhaps even agree with Garfield’s prescriptions for the French based on them, but few would agree with what has evolved into the WoS’s policies for acceptance into its three flagship indexes, not just in science but also in the social sciences and the arts and humanities. Most might be dismayed by the present-day reality that journals in any of those disciplines and fields that do not follow such prescriptions, even regional/national journals concerned mainly with studies of French (or Chinese or other non-English) society, history, and arts and letters, are faced with the absolutist dictate of exclusion from being indexed by the WoS—which today can mean exclusion from the market of library subscriptions that are crucial to the financial viability of a journal. No matter how respected the journal might be among the scholars of the particular subject area or discipline, it would have to fight for survival against the market pressures imposed by CA’s near-monopoly of the business of journals evaluation in the United States.
Let us consider once more CA’s latest statement of policy in its “Journal Selection Process.” The statement asserts flatly: “English is the universal language of science,” an assertion that would be repeated twice more in the 14-page document, and is applied not only to science, where the assertion might be closer to the truth, but also to the more particularist “social sciences,” and even very particularist area studies and the arts and humanities, where it becomes a preposterous kind of English-centrism. To be sure, CA’s official document, in addition to asserting the universality of English, also pays attention to what it calls “regional studies” of “local, rather than global, interest.” That would seem to be an important concession to particularist and substantivist disciplines, and the policy statement concedes that for those areas of study, “English-language full text is also not always required in some areas of arts and humanities where the national focus of the study precludes the need for it. An example of this is studies in regional or national literatures” (Testa, 2016: 10). But, even so, the same statement goes on to declare that “English language bibliographic information is required for all journals seeking coverage” and that “Cited references must be in the Roman alphabet” (11).
Strictly applied to a subject like rural China (or Chinese thought or Chinese literature or Chinese linguistics), this would mean that even those writing in Chinese for Chinese audiences about Chinese realities or texts or the Chinese language, who of course form the great majority of researchers in those fields of study, must nevertheless provide “English language bibliographic information” and “citations in the Roman alphabet” in order to be included in the CA citation indexes.
Now, the provision of English-language bibliographic information (i.e., author’s transliterated name, title of the article in English, English abstract, and English keywords), while an onerous burden for Chinese (or other non-English) authors (and editors), can serve a certain useful purpose: it provides readers who read Chinese with difficulty easy access to a synopsis of the contents before deciding whether to attack the original text. But the provision of romanized references is another matter. All transliterated Chinese words have multiple possibilities of different homonymic characters. The transliterations are simply inadequate for guiding a researcher to the original text by way of, for example, a full-text database such as Chinese National Knowledge Internet (CNKI). One needs the original characters. To give transliterations only for references is therefore of little use; to provide both transliterations and characters is cumbersome and redundant and, most of all, pointless.
What this means in reality is the insistence that all fields of knowledge must be incorporated into the world of English-language scholarship, even if the latter accounts for only a very small percentage of scholars working in the subject area (dozens or hundreds, as opposed to thousands or more)—as is the case with research on rural China or Chinese literature. Its practical consequence amounts to an unrealistic dictate for Chinese scholars writing for Chinese audiences and relying entirely or overwhelmingly on Chinese-language documentation. Imagine the burdens on those scholars to meet the requirement of providing “English language bibliographic information” and “citations in the Roman alphabet.” The latter, especially, is at best an onerous yet useless burden on Chinese (and other non-English) researchers (because foreign scholars of such subjects will and must know Chinese) and, at worst, precludes the legitimacy and survival of non-English language scholarship in the global journals market of today. It is a policy that is very much along the same lines as the flat assertion that “English is the universal language of science,” not just now, but for all time, and with the unavoidable implication that, if it isn’t, it ought to be.
As if these barriers were not enough, the WoS’s final exclusivist barrier is to require that such regional journals demonstrate their value by the “impact factor” they garner from journals already accepted and indexed by the WoS, which are almost all in English. If we take a regional field, like Chinese literature or rural China, in which there might be only dozens or hundreds of serious English-language researchers, as opposed to thousands or tens of thousands of serious Chinese researchers, even the journals cited by most serious researchers in the field might never gain acceptance by the WoS, because the great majority of their citations will be in Chinese not English articles, and will not appear in journals already indexed by the WoS.
If the WoS were to truly take seriously “regional journals,” it must include citations in the major regional journals in regional languages to truly take into account their relative impact in the field of study in question. The indexing of merely a few dozen English-language scholars’ citations simply will not do. To use them as the main standard by which to include non-English (or bilingual) regional journals is simply to go against the very cosmopolitan ideal that motivates regional studies.
At this point, it might be useful to turn to two “national journals” with which this author has unusual familiarity from having founded and edited them over the years. They can serve as concrete examples of what has been argued above, and also bring up other relevant considerations.
Two Empirical Case Illustrations of Longitudinal Change
Modern China: An International Journal of History and Social Science
“Orientalism” was very much present in American scholarship on China in the 1970s, given the predominance of “modernization theory” in social sciences research on China, and also the still powerful influence of the Committee of One Million against the Admission of Communist China to the United Nations. 9 The presumption was, Anglo-American style modernization, predicated on free market economy and liberal democracy, must or should be the universal path to modern development; developing countries, like China, served mainly as an illustration of “the other,” to validate the Anglo-American experience, a set of assumptions that has been pointed out by numerous scholars, and not just those of postmodernist persuasions. (For detailed discussion and documentation, see Huang, 2016a.)
However, such Orientalist concerns were, ironically, mitigated to some degree by the United Sates’ national security concerns and policies going back to the 1950s. Educational institutions had not yet come under the dominant influence of scientism—the belief that imitative “science” should govern all spheres of human endeavor. And, in constructing the new U.S.-led world order, there was the conviction that one should know one’s enemy—hence the multiple congressional National Defense Education Act measures that aimed to promote language and other training of American specialists in “foreign area studies,” and the promotion and funding of multidisciplinary area studies centers and graduate programs. Last but not least, there still prevailed a certain common sense about reality—that knowledge should attend not just to the universal but also the particular, not just encompassing universalist theory but also specific empirical knowledge, not just formalism but also substantivism, not just the separate disciplines but also integrated foreign area knowledge, and not just science but also social sciences and the arts and humanities.
Free enterprise capitalism and market competition also played an important role in mitigating exclusiveness. There were to be sure the established main journals of the disciplines, usually founded by the major national-level academic societies and organizations, but there were also more “alternative” journals of specialized interests or approaches, even of progressive or radical perspectives, that could find room in the multiplicity of small publishing businesses that could take on a journal so long as it had a sustainable market. Sage Publications is a good example of such a journal publisher—founded to make a business of academic journals, on the keen perception that academic journals have an almost unique quality in that payments for the product are usually made with the subscription, generally a year or more before the product is actually delivered, a trade practice that opens up a special kind of cash-flow situation not available to most other businesses: one did not need a great deal of capital to start a journal publishing business. The presence of such journal publishers, without the monopolistic domination of their evaluation by the CA company (or Elsevier), made for a more inclusive environment of market competition.
It was in that larger environment that this author and colleagues managed to found the journal Modern China: An International Journal of History and Social Science. The intent was to establish the journal to balance and challenge what was then the leading journal of the China studies field, China Quarterly, known to have been founded (in 1960) by the CIA through its front organizations, the Congress for Cultural Freedom and the Farfield Foundation (for more details, see Huang, 2016a: 126ff). It should be pointed out here that, even if the impetus behind China Quarterly were in the beginning mainly national security, it did over time acquire a solid and respected academic reputation among scholars studying contemporary China. This author and academic friends who helped to establish Modern China, by contrast, were generally more progressive types interested in a forum that would see beyond so-called national security concerns focused almost entirely on contemporary “Communist China,” to emphasize historical more than contemporary subjects, studies of the common Chinese people more than of the Chinese government, of society and economy more than of foreign policy, of history-literature-anthropology-sociology more than political science. Sage’s leaders, neighboring UCLA in Beverly Hills, found it a marketable (financially sustainable) idea. Thus was the journal launched in 1975. It was seen as “progressive” or “liberal” but neither “establishmentarian” like China Quarterly nor “radical” like the Bulletin of Concerned Asian Scholars (later Critical Asian Studies). Thus was the three-way division of the China field in the United States reflected in its three major journals of the time.
The measurement of the impact factor of journals was not yet prevalent, far from the dominant position it occupies today. Librarians did not yet rely mainly on those to decide whether to start an institutional subscription, but rather called on something akin to common sense: what they knew about the interests and strengths and opinions of their institution’s faculty in Chinese studies, what they saw and learned about the journal by browsing it, and so on. What came to be something of an operative rule of thumb was: has the journal shown its viability by following standard scholarly practices and standards and publishing on schedule for five years?
On those common sense standards, Modern China gradually came to be an “established” journal that found an important place among China scholars. Thus did the journal, now in its 44th year, come to thrive—even if not seen as “the leading” journal of the field, certainly a high-quality and well-established one, and one of a handful in “the field.” It had emerged, to some extent, through the open spaces left by modernizationism, and in a field of study that was still relatively tolerant and inclusive. Still edited by this author and his colleague (and spouse) Professor Kathryn Bernhardt, the journal has shown great continuity in its editorial staff and key editorial board members. In 2009, the journal expanded from four issues a year to six a year.
Rural China: An International Journal of History and Social Science
This author’s later experience with Rural China: An International Journal of History and Social Science, intended to be a sister journal of Modern China, however, provides a sharp contrast with the earlier period, and helps to illustrate the much more rigid academic journals environment that has come with the near-monopolistic domination of journal evaluation by CA’s WoS, in the United States if not in Europe. Zhongguo xiangcun yanjiu 中国乡村研究 (ZXY), the original Chinese version of Rural China, was launched in 2003, with the simple idea that rural China is distinctive because its small-peasant economy (just 20 mu, or 3.3 acres per family farm today) has persisted well into a time of fairly advanced industrial development as well as information technology development, and still accounts for close to 900 million people who are (officially registered as) peasants. 10 The idea was to launch a journal that would become the avenue for scholarly publication on the subject. While there are just a few dozen Western specialists who write mainly on rural China, perhaps a few hundred if we include those who study migrant peasant-workers, there are literally thousands upon thousands of Chinese scholars whose principal research and concern are with villages and peasants and migrant peasant-workers—because of the enormity of the group, the numbers of present and future scholars of peasant origins, and the obvious central importance of peasants to China as a whole. The journal aimed to join the two groups of researchers, one international and the other Chinese, in order to, on the one hand, help internationalize rural China research and, on the other hand, lend the weight and complexity of Chinese research to what is a relatively small field in the West. Each should benefit the other.
Indeed, the journal was established before long in China as the forum for the rapidly emergent field of serious scholarly research on rural China, as was argued and documented in detail by two substantial articles by senior scholars, a rather surprising occurrence for us, and rarely seen by any journal. The first of those, by Nankai University’s Professor Wang Xianming 王先明, was even published in the official People’s Daily, and the second, by Wuhan University’s Professor Ren Fang 任放, was published in the influential journal 史学月刊 (Journal of Historical Science) (Ren, 2011). Wang in his article “Research on Rural China is Gradually Becoming a Hot Point” wrote: “Zhongguo xiangcun yanjiu, edited by Huang Zongzhi 黄宗智 [Philip C. C. Huang], especially, has helped make the modern history of rural China a hot subject of research, becoming the principal venue for the publication of such research” (Wang, 2010). And Ren’s thoroughly documented research article “Thirty Years of Research on Rural China” took for its theme that (editor) Huang Zongzhi has been the principal “landmark” 界标 in the 30-year history of that research. With such kudos, including the praise in the official People’s Daily, ZXY seemed fairly well established in the Chinese academic world.
That was the background against which E. J. Brill contacted us about the possibility for Brill to publish an international edition of the journal, to include also articles in English. Since we from the start had hoped to establish rural China research as a cross-national and bilingual field of study, agreement was quickly reached to publish Rural China. The first two issues appeared in April and October 2013, coinciding with the tenth annual issue of ZXY, now explicitly identified as the Chinese edition of Rural China. (The two markets, China and outside China, are highly segmented; there is no serious effort to market ZXY outside China, by either the publisher or any electronic database—more below.) The two journals in fact overlap considerably, sharing mostly the same articles, with Rural China publishing when possible and available the English versions of the articles (about four to six a year, becoming thereby a major avenue for Chinese scholars of this field to enter into the international world of scholarship, with us providing expert editing help in English), and ZXY the Chinese versions.
The combination of the two seemed to all of us the best of possible worlds. It encourages our English-language contributors to look to translating/writing (or have translated) their articles into Chinese, and our Chinese-language contributors to look to translating/writing their articles into English. A perfect way, we thought, to promote and sustain a field that would truly join the English and Chinese scholarly worlds, encouraging transnational interaction and learning. It would also serve to illustrate how to make such a major area of study something that is bilingual and bicultural, transcending national, cultural, and language boundaries, and exemplary of completely non–Anglo-American-centric approaches to scholarly research.
Seen another way, the Chinese side of researchers would lend the English side more substance and information, as well as the large numbers of researchers and readers that it lacks. The English side, on the other hand, would help internationalize the perspective of Chinese researchers—indeed, the distinctive characteristics of rural China can only be fully grasped by comparison with other nations and geographical areas, especially of the West, where little in the way of a peasant economy remains. We the editors were eager to have the counterintuitive and paradoxical (in terms of the expectations of mainstream Western theories) truths about rural China studied and told.
All seemed well, until we began to encounter unexpected obstacles that Rural China has had to confront from the exclusivism, and even unintended Orientalism, that came with the excessively rigid data-based journals evaluation system, and ZXY has had to deal with those same obstacles not just outside of China but inside China as well (more below). Those obstacles have stemmed above all from the near-monopolistic influence and control of the U.S. journals market by CA and its WoS, and the blind following of CA practices not just in the English-speaking world but also inside China. Today, fifteen years after the Chinese (edition) of the journal was first published, and five years since the publication of its bilingual English-Chinese edition, the journal has yet to be accepted by CA for inclusion in its WoS. (On July 8, 2018, our editorial office was formally notified by Scopus that Rural China has been accepted for inclusion in its citation indexes, but our review by WoS is still in progress.)
Both ZXY and Rural China have met WoS’s demand for “English-language bibliographic information”—that is, romanized names of authors, English-language titles, abstracts, and keywords—because those serve the purpose of a convenient bibliographic aid for researchers who might then wish to examine the original text. But we have not met the even more onerous demand for romanized references, because those would not only be an impossibly unreasonable burden on our Chinese authors but also utterly pointless: original Chinese characters are essential for tracking down the full text and, once those are provided, it is simply redundant and pointless to add romanizations for them. Yet, so long as the WoS continues to insist on romanized references, Rural China may never be included. In our view, it is incumbent on the WoS to adapt its practices to include Chinese-language research for such a field of study—something which can be done readily with today’s technologies—because it is and will most likely remain the source of the great majority of citations in serious scholarly research on the subject. To continue to insist on romanized references in such a field of study is narrow-minded, exclusivist, and untenable.
Misuses of Citation Indexes and Impact Factor Measurement in China
If the rise in recent decades of a monopolistic business that has come to dominate scholarly research by controlling the admission of its major outlet—scholarly journals—is unexpected, what is even more surprising is the extent to which academic administration in China has completely adopted, and even carried to a still more exaggerated extent, such misuses of citation indexes.
First is the merging of scientistic journal evaluations with scientistic management by a centralized bureaucracy to result in a system even more exclusive and monopolistic than what exists in the United States. Chinese academic administrators have to date adopted wholeheartedly the scientistic rankings of the WoS’s three flagship indexes and enforced them in bureaucratized ways that might be difficult for even the present generation of American scholars to imagine. Major universities, under the dictate “to link up with the international (standards and practices)” 与国际接轨 are vigorously enforcing the WoS impact factor–based rankings of journals. All have come to be preoccupied with numbers of publications in the ostensibly “best” “international” journals as ranked by CA. All major universities have adopted highly standardized WoS ranked lists by the categories A+, A, A−, and B lists of “core journals,” across all disciplines and fields. Researchers in the sciences, the social sciences, and the arts and humanities, now labor under a uniform policy that requires a certain number of publications in “core journals,” of which the WoS core journals are seen as the best and most prestigious. The government even ranks the different universities and their departments and schools according to the numbers of articles their faculty have published in international core journals of the SCI, SSCI, and AHCI. All, willingly or not, must engage in this competition to outrank other comparison institutions, lest their institutional ranking and funding be affected.
The hierarchy drawn from the WoS citation indexes has been extended into and imitated by Chinese equivalents, like the Chinese equivalent of the SSCI and AHCI in CSSCI, Chinese Social Science Citation Index 中文社会科学引文索引, which includes the arts and humanities. 11 It is a government-funded entity/“project” managed by Nanjing University, in a deliberate effort to copy the SSCI model. CSSCI now includes more than 500 Chinese journals thus selected (CSSCI, 2018a, b).
Although the rankings are made by citation index data in ways that imitate the WoS, they also bear close correspondence to their administrative status within the Chinese bureaucratic hierarchy. The rank-order generally corresponds also to whether a journal is a central-level (national) publication, a “directly under the center” municipality (like Beijing and Shanghai), or a province-level publication, as opposed to the lower levels of provincial cities and counties, and county towns. Also important is whether the journals are operated by the 116 universities under the 211 Project (referring to the plan to develop in the twenty-first century a hundred world class universities, hence “211”), launched in 1995, and, even more so, the 39 among them that were included under the 985 Project (referring to the year 1998 and the month of May when the project was launched, hence “985”) to develop a top tier group of universities among the 116 “211 universities.” The two groups form tier 1 (39 universities) and tier 2 (73 universities) in China today, distinguished from 1,124 ordinary universities (Quan, Chen, and Shu, 2017: figures 2 and 3). As Quan et al.’s detailed research shows, in 2002–2015, the universities in the top two tiers received from the state average annual budgets that were twelve times those of the ordinary universities. The “core journals” usually come from them. The rest, lower bureaucratic level journals and lower-level (non-985 and -211) universities’ journals, without regard at all as to whether those are or are not in fact seen by professional researchers in the field as high-quality journals, generally do not even count in personnel reviews in the tier 1 and 2 universities. That applies also to irregular “book journals” without officially approved journal (“serial”) numbers 刊号 (more below).
The Chinese journals at the top of the bureaucratic-academic hierarchy are, in theory, ranked as equal to the most prominent of “international” (i.e., non-Chinese) WoS journals, but they are actually far from equal in terms of actual administrative practice. As Quan et al.’s article demonstrates in detail, the amounts and the differences between the rewards can be large. As is well known, scholars publishing in the top-ranked Chinese social sciences journal, Zhongguo shehuikexue (Social Sciences in China), typically receive bonuses from their universities of ten thousand yuan per article, or more, some for three successive years. But the bonuses for publication in the two top-ranked WoS journals, Nature and Science (with an impact factor of 41.5 and 34.9, respectively), without consideration of the actual quality of the article at issue in terms of its content or status among professional scientists, can be many times more (Quan, Chen, and Shu, 2017: table 4). According to Quan et al.’s sampling of the financial rewards policies of a hundred universities (including those of all three tiers), the financial reward for a paper in Science and Nature averaged in 2016 US$43,783 (tables 2 and 5)—more than a quarter million yuan in RMB, about five times the average annual salary of a university professor (US$8,600—see under the penultimate section “Discussion” of the article), and many times that rewarded for a top-ranked national-level core Chinese journal like Social Sciences in China. To colleagues of postmodern sensibilities, this might serve as a striking example of what can be termed “indigenous Orientalism” (as well as scientism) in academic administration in China today.
In recent years, the Chinese Social Science Citation Index has opened its doors some, to include selected “book journals” (in which journals take the form of published books, termed the 来源集刊, as opposed to the officially sanctioned “serials” 期刊). These have emerged because strict limits are set on the officially approved “serial” numbers (employing the international ISSN numbers), which are simply no longer obtainable. The result is that would-be new journals have had to operate in a gray area: by obtaining a Chinese book number (of the international ISBN numbers) and publishing the journal by that number, hence the expression “using a book [(number] to substitute for a journal [serial number]” 以书代刊. From those, CSSCI has selected, by impact factor, a relatively small number for inclusion. In 2012–2013, there were an announced total of 120 such journals accepted for inclusion; for 2017–2018, there were 155 (CSSCI, 2018a, 2018b). Zhongguo xiangcun yanjiu, the domestic Chinese edition of Rural China, for example, was not accepted as a CSSCI journal until 2014, after twelve annual issues, even though it had already been established by 2010 among peers in the field as the most important journal for the subject of rural China. As one younger scholar wrote to us, the editors of Rural China, to explain: “Rural China’s articles are models for us, but under the evaluation system of our university, Rural China is ranked in the lowest category, completely counter to the scholarly quality of the journal, which is surprising, and yet we are in no position to change the situation.” “We have expended a great deal of effort to research and write this article. . . . Would it be possible for us to wait until Rural China has been reviewed [by SSCI] before we publish the article?”
The distinction between the bureaucratically accepted and not accepted is reinforced by not just CSSCI, but also the government-funded (managed by Tsinghua University) CNKI, 中国知网 “Chinese National Knowledge Internet,” which is the main database for full-text scholarly articles in all disciplines and fields. CNKI and CSSCI are two separate entities, but are in fact mutually reinforcing. Even after a journal has been accepted by CSSCI, it still needs to be included into the full-text databases in order to be accessible electronically to researchers. Scholars would not be able to access easily the CSSCI journals if they were not included also in the key CNKI full-text database (barring, of course, the article being uploaded to a public website). It is as if, in the U.S. context, the CA company’s SSCI were to be reinforced with JSTOR (which is a separate and not-for-profit entity) to form an even more overwhelming and unchallengeable monopoly.
One can be the beneficiary as well as the victim of such a system, without regard to the actual quality of the journal or of the individual scholar. Thus, Modern China has been something of an unwitting beneficiary of the system: because of the journal’s long history and established record in the WoS’s SSCI citation index, it is ranked a category A “international” journal by Chinese universities, while Rural China has been excluded from the list of acceptable core journals—because Rural China has not yet been accepted by SSCI, and because ZXY has only semi-legitimate, which is to say also semi-illegitimate, bureaucratic status as a “book journal.” Despite our high evaluation standards (more below) and our preeminence in the field of rural China studies, the journal remains very much a victim of the system. While Modern China is for Chinese scholars a much sought-after place to publish, because of the status it enjoys in the bureaucratized hierarchy of journals, Rural China-ZXY has had to battle the lack of recognition from the WoS, and for some time also the Chinese citation indexes, plus its only semi-legitimate status in the eyes of the Chinese academic bureaucracy. 12
Today, after fifteen years of publication (of ZXY and five years of its international edition Rural China), the journal is in actuality without doubt the major outlet in the field. In the five years 2013–2017, we received a total of 309 submissions, of which we accepted 34 (plus an additional 19 solicited for the two special symposia we organized to spotlight key issues in the field, and also to enhance further the academic quality and influence of the journal). Even so, many scholars, especially those pressured by their institutions, have elected to publish elsewhere in journals that are on the administratively determined “core journals” list, as the letter quoted above from one such scholar explains. That continues to be an issue that we have to battle with. In the meantime, we are faced with the obstacles that the Chinese bureaucratic system has placed before us, when one might expect that scholars, and even administrators, would be the first to support the idea that, in studies of rural China, Chinese scholarship is or can be, and certainly should be, the best in the world. What they should be doing is actively supporting this journal, not intentionally or unintentionally strangling it.
What the history of the two journals tells about is the transition of citation indexing data from a useful tool of scholarship to a misused tool of scientistic evaluations and monopolistic management. Because of that change, Modern China today receives considerable numbers of submissions from savvy scholars, especially Chinese scholars under bureaucratic management pressures, who are more interested in its high ranking than its actual scholarly quality. They have about doubled Modern China’s submission rate to an average of 200 a year, out of which we publish twenty to twenty-two articles. Those types of submissions are seldom of high academic quality and are rarely accepted by us. Rural China/ZXY, by contrast, despite its comparable selectivity has had to struggle under the burdens of obstructive discrimination from both the CA company’s SSCI and the Chinese bureaucratic administration. The contrast provides a concrete illustration of the transformation from use to misuse of citation indexes in the past half century.
How Scholars in Particularist Fields and Disciplines Actually Use the Citation Indexes
After more than 50 years of active engagement in scholarly research and as long-time (founding) editor of the journals Modern China and Rural China/ZXY, this author can say without hesitation that, in our real research work, citation indexes have actually figured only in limited ways. 13 For bibliographic searches, in the 1960s and 1970s, when our generation of scholars first began doing serious research, most of us in modern Chinese history tended to rely on bibliographic guides such as Fairbank and Liu (1961) for our initial searches of Chinese titles, and Fairbank and Banno (1955) for Japanese titles (we all had to learn Japanese because of the sophistication of its research on China), the three-volume Modern Chinese Society: An Analytical Bibliography edited by G. William Skinner (1973), with volume 1 covering publications in Western languages, volume 2 publications in Chinese, and volume 3 publications in Japanese, plus the annual Bibliography of Asian Studies of the Association for Asian Studies, from 1971 to 1991, for studies in Western languages. Today, we use databases like CNKI for both bibliographic searches and full-text access to Chinese titles, and JSTOR for English-language titles. Many of us also use Google Scholar, because its data lag publication by just one year or less, compared to about four years for JSTOR; also because of its helpful displays of links for accessing the full-text of articles; and because it includes books, which still figure greatly in Chinese studies research. Some of us might supplement the above with searches also in WoS and/or Scopus. 14
Perhaps even more important for China studies is Google Scholar’s steadily expanding coverage of Chinese-language titles, which are crucial in a field such as ours. To take this author’s own experience as an example, Google Scholar has for some time been able to generate two separate citation lists under “Philip C. C. Huang” and “黄宗智,” one for scholarship in English, and one for scholarship in Chinese. Recently, Google Scholar appears to have begun to be able to generate a unified, bilingual citation index for an author’s work, in both Chinese and English scholarship, suggesting the possibility in the future of more fully integrated bilingual or even multilingual citation data for individual authors. 15
As for journals rather than individual scholars, very few if any of us would go on to the WoS to check out the impact factor of any of the English-language articles or journals we draw upon, because most of us know which journals are the most important in our field. Few if any of us would bother to check in addition the impact factor ratings of an author or an article in making our judgment, because almost all of us believe that professionals within a given field are the best qualified to judge. This is of course an attitude that lies at the heart of the persistence of peer review practices which had long figured centrally in scholarly evaluations, and which still have considerable influence today. In the United States, even in this very scientistic era, when it comes to hiring and promotions, (confidential) letters from “leaders in the field” still count for much more than simple impact factor measurement data. For most of us, the impact factor data available through WoS (or Scopus) is simply not seen as a terribly useful or relevant evaluative tool for our actual research work. Excessive and extreme reliance on quantified impact factor data tends to be found mostly under the more authoritarian administrative systems in some European countries and areas, and China, Hong Kong, Singapore, and the like. In the United States, for appointments and promotions, the long-standing reliance on peer review in the form of ad hoc committees of one’s colleagues and peers remains pervasive.
What most of us do is refer occasionally to citation information about our own work, not for the purpose of studying a particular subject, but mainly out of a self-centered interest in finding out to what extent and by whom our work is being read and cited. We would take some pleasure in noting the large numbers of hits that a certain piece of our own has drawn (such data are provided by CNKI though not by JSTOR or WoS or Google Scholar), even when we know that a hit does not necessarily mean actual reading of the article. We also take some note of who and how many have cited our work, for that is one useful indication of how much attention, for better or for worse, our works have garnered from peers. Yet most of us know that those data often tell us more about whether our article (or book) has struck chords in the profession—often for reasons other than how important, solid, or careful our research is, but rather for reasons of timeliness, theoretical fads, or ideological concerns. Impact factors, we know, are often not so much about the actual quality of our article (or book) and the scholarly contribution it makes; most of us believe that we ourselves know best the actual weight of our own and our peers’ work. Citation indexes, at their best and on rare occasions, might lead us to connect with colleagues whose approaches and understandings of our subject might seem to us particularly interesting or agreeable but whom we might not otherwise have known about (and even lead us to connect with such colleagues). Overall, however, citation indexes are of limited use to us in our actual research work.
Beyond that, citation indexes and impact factors enter into our lives really only when we are faced with appointment or promotion evaluations (of ourselves, our students, or younger colleagues), that because our institutions have come in recent decades to place more and more emphasis on quantifiable data, some requiring faculty under review to furnish citation data for evaluation. It used to be just a matter of whether the journals we had published in were generally known among our fellow scholars to be the most important, but that was never considered more significant than peer review in the true sense: by ad committee members’ own reading of the material (now rare) and the outside letters from leaders in the field. And, for those of us in disciplines that emphasize books more than articles, it used to matter somewhat whether our books had been published by prestigious university presses (for Chinese studies, presses like Stanford University Press, University of California Press, Harvard University Press, and so on), but that indicator has come to matter less and less, in part because most university presses have since come under severe financial pressures to cut costs—they have not been able to attract and retain top-quality editors, nor have they been able to maintain the editing standards that permanent, full-time salaried copy editors (as opposed to ad hoc copy editors paid by the page, which has become the norm today), had provided. The broad trend is toward less and less substantive evaluation, more and more quantified “scientific” evaluation.
There is thus a widening gap between how most practicing scholars think about scholarly quality and how WoS (and Scopus too), and increasingly also the universities, go about measuring it. Numbers of citations do tell about how widely referred to a journal or an article might be, but that attention might as likely be negative as positive, especially in the social sciences and arts and humanities. Articles with arguments that seem faulty, or facile and exaggerated, are as likely to be cited in our articles as foils to clarify our own argument than as building blocks of our research. We also know that theoretical fads that strike chords can often draw the most citations from the like-minded, often more than carefully documented and constructed tomes. We know too that scholars often cite an item just for some small, particular detail useful to our own piece and not because of our special approval for its overall scholarly contribution. And then there are scholars who routinely use citations as a tool of public relations rather than substance. To turn the complex issue of judgment of substantive quality into the single factor of counting of numbers of articles that cite the article/journal in footnotes is something that actually violates what most of us truly value.
How then do we understand the almost overwhelming influence that the CA’s profit-making business of WoS (and Elsevier’s Scopus) has come to wield in evaluating scholarly quality? This article has suggested that we need to look finally to the gigantic tidal waves of scientism and data-ism—of the application of supposedly scientific methods and quantified data to business administration and governance, followed by imitative usage of such by academic administrators—to understand the rise to predominance of the CA and its WoS citation indexes for evaluating journals and, by extension, also authors.
Such scientistic pretenses depart from reality most glaringly when it comes to the more particularist fields and disciplines, in national/regional studies and the arts and humanities. Ideological/theoretical divides are much sharper in those disciplines and areas than in the natural sciences; disputes over “truths” are much more frequent, without any possibility of complete resolution through controlled experiments. Those basic characteristics impact directly the reasons and motivations for citations, and hence greatly complicate any effort at quantified evaluations. In those fields and disciplines, rigid adherence to impact factors as the measure of quality easily becomes in effect a rigid advocacy of scientism, a defense of existing conventional wisdom, and resistance to change; it obstructs rather than promotes creativity and innovation, which is what genuine scholarly inquiry ought to be about.
Under the regime of impact factor measurement, new journals face immense obstacles to acceptance as part of the standard literature, as we have seen in the case of Rural China, just as unconventional scholars face immense obstacles to gain publication in well-established top journals when their articles challenge in fundamental ways the prevailing wisdom. And, last but not least, non–English-language journals face nearly impossible-to-overcome barriers of access to the ranks of top-ranked journals, given the WoS’s strong and exclusivist emphasis on English as “the universal language of science,” especially its exclusivist requirement that citations be “in the Roman alphabet.” What’s more, the impact factor measurement so crucial for acceptance is predicated entirely on citations in English-centric journals already indexed by the WoS.
An anti-mainstream journal like Modern China of the 1970s and 1980s might well find it almost impossible to get established today, given its non- or anti-establishmentarian orientation. And all non-English, or bilingual and multilingual journals like Rural China, must face the possibility of permanent exclusion by the WoS, and hence also from financial viability. Regional journals not in English, finally, face almost impossible to overcome obstacles due to the English exclusivism of the journals already indexed by the WoS. It is incumbent upon the WoS to find ways to overcome these exclusivist tendencies.
Some Concluding Thoughts
How then might such a state of affairs be improved? One obvious answer is for university administrators, librarians, and academics to become more aware of the realities of citation indexing and impact factor measurement. The misuses of citation indexing stem finally both from the larger historical tides of scientism and data-ism, as well as many researchers’ innocence of how things actually work with the business of citation indexes. Relatively speaking, impact factor measurements work somewhat less poorly for the natural sciences, but very poorly indeed for more particularistic and substantivist regional/national studies and the arts and humanities. Those are not characterized by unified paradigms to nearly the same extent and, even with the natural sciences, all of us are aware of the historical phenomena of paradigmatic revolutions, as Thomas Kuhn had shown so powerfully (Kuhn, 1970 [1962]). We urgently need more systematic substantivist measures of journals (and by extension also of individual scholars) to replace the present unidimensional impact factor measures, especially in the much more contested social sciences and arts and humanities. To be sure, academic researchers have been the victims and not the perpetrators of the oppression of citation indexes, and, by comparison with the academic administrators and near-monopolistic businesses, relatively powerless.
The development of more substantive and precise evaluations of journals—such as quantifications based on systematic sampling surveys of scholars’ opinions in a given field or discipline, in reality a form of substantive peer review—would be one positive direction of change. An example that has been shown to work reasonably well is the rankings of disciplines and fields of different universities by surveys of department chairs and faculty. The same can be done with journal rankings through the sampling of professionals in the given field. A system akin to the World Justice Project’s measurement of the quality of different world justice systems according to relatively well-defined multidimensional standards (eight “factors”), through surveys of specialists (and also of popular opinions, though weighted far less than specialists’ opinions), despite its multiple shortcomings, would also be an improvement over the present system of only counting citations. 16
We would also be helped greatly by more entities that are motivated by concerns for advancing scholarship and understanding more than profit, such as the more inclusive nonprofit database JSTOR (though necessarily limited today by the difficulties new journals face in surviving under the WoS or Scopus regimes), or even China’s government-funded CNKI, to bring reform and change to the present situation. Those offer the potential for more inclusive coverage. Google Scholar has established its value by greater inclusivity and its ability to function both as a search engine and a citations database (by the helpful displaying of links to full texts). It is also showing exciting prospects for unified bilingual or even multilingual citation indexes and databases.
Open access journals also open up promising possibilities, though they continue to suffer under the pressures imposed by the predominance of WoS (and Scopus) over the journal evaluation business, and many have to charge authors fees for publication in order to make ends meet. Cooperative consortia of libraries that have joined together to purchase electronic editions of books and journals at discounted rates and for open access, like Knowledge Unlatched, can also serve as a serious alternative to the existing system.
Monopolies in business have generally shown a strong tendency to seek to perpetuate their control and dominance if allowed to do so; they tend to be resistant to transparency, inclusivity, and change, lest their privileged position, and profits, be affected. We have seen how Garfield’s initially tentative ideas and researches have become reified into operative “laws.” Centralized bureaucracies, we have also seen, have an even stronger tendency to be scientistic and exclusivist in order to preserve and enhance their power and control.
Companies enjoying the present monopoly should be challenged in even more sustained and systematic ways—to persuade them to adopt more open and transparent policies, substantive evaluations rather than singular dependence on the mere counting of citations, and less exclusivist standards and practices. Garfield’s simplistic “law of concentration,” his English-language exclusivism, and his simple use of the natural sciences model for the social sciences and for the arts and humanities, were all outlined and developed in the 1970s, almost half a century ago. They are much too crude for the world of scholarship and are badly outdated as well, considering new possibilities with new technologies, yet they continue to be applied, and ever more rigidly. Those methods and principles have in recent years been wholeheartedly embraced by educational bureaucrats in China and applied in even more exaggerated ways, as we have seen. Those misuses and excesses serve as good warnings to all scholars about the tendencies toward ever greater monopolistic abuses.
To counter those tendencies, a more pluralistic structure with genuine market competition from companies operating by alternative principles and methods would help. Another improvement would be greater influence of entities that are service- rather than profit-oriented, such as JSTOR and Google Scholar. Even state-managed databases, like China’s CSSCI and CNKI, have the potential for attaining results comparable to nonprofit entities like JSTOR and Google Scholar, to balance out the singular dependence on monopolistic businesses.
Finally, the current state of information technology can in fact allow for cost-effective bilingual and even multilingual inclusivity, far more than was feasible in Garfield’s manual-input and “machine translation” days. There need not be English-language exclusivism, especially in this ever-accelerating globalizing world of today, and especially not for area studies and the arts and humanities, for which such exclusivism amounts to obstruction of the cosmopolitanism that lies at the heart of foreign area studies in the social sciences as well as arts and humanities. Perhaps most important, we scholars, librarians, and our universities should not and need not continue to tolerate simple-minded citations counting and monopolistic domination by a single profit-seeking entity. In the end, nothing encourages good and innovative scholarship more than the free and inclusive pursuit of knowledge and understanding. And, in the social sciences and arts and humanities especially, research from multiple perspectives, orientations, and languages is simply a must.
Footnotes
Acknowledgements
My thanks to Perry Anderson, Jenny Chan, Gao Jin, Qin Jiang Higley, Mark Selden, Zhang Jiayan and, most of all, Kathryn Bernhardt, for helpful comments and suggestions.
Declaration of Conflicting Interests
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author received no financial support for the research, authorship, and/or publication of this article.