Minding the gap: Bibliometric equity in theology and religious studies

1. Introduction

Bibliometrics is widely utilized across diverse fields to gain insights into the research landscape and understand the scholarly and societal impacts of research. However, most theology and religious studies scholars remain skeptical about the applicability of quantitative bibliometric techniques, data, and indicators, particularly in bibliometrics for evaluative purposes.

The skepticism or even rejection is complicated and caused by many implicit and explicit factors, many of which are rooted in the scholarly communication practices influenced by the epistemic cultures [1] and intellectual adhocracy [2] in theology and religious studies, including research practices, publication typologies, referencing practices, and the variety of languages and publication venues [3–6].

Meanwhile, the lack of an integrated information infrastructure has created additional barriers [5,6], including the low proportion of born-digital or digitized publications, as well as the variety and fragmentation of supporting information systems and databases. The fact that the bibliographic databases do not sufficiently and accurately represent scholarly activities being studied has rendered the questionable validity and effectiveness of the bibliometric analyses derived from them.

Furthermore, employing bibliometrics for research evaluation purposes, if not used with caution, can oversimplify the complex nature of research, overlook qualitative aspects and interdisciplinary contributions, and incentivize certain behaviors, such as excessive self-citations that can potentially compromise the integrity and diversity of research pursuits. The disciplinary norms and the deficiency in the information infrastructure in theology and religious studies mentioned above further complicates the application of evaluative bibliometrics in this field.

This paper aims to delve into some of the intrinsic and extrinsic characteristics of the theology and religious studies scholarly communication ecosystem and discuss potential strategies to address them. Our discussions are based on both empirical data and field experiences. Empirically, the findings we are publicizing of our analyses of publication data in theology and religious studies are derived from a multi-year and multi-pronged project at Vanderbilt University striving to promote access, reach, and impact of faculty work. Our paper also features a practitioner’s perspective that draws on experience working with scholars and administrators in the Center of Theological Inquiry to adopt bibliometrics to capture and understand the scholarly and public impact of sponsored research.

2. Challenges of institutional evaluative bibliometrics

Theological and Divinity Schools constitute a specialized subset within the broader academic realm. The distinct nature of their educational objectives, missions, and the diverse audiences with which they engage inevitably engenders specific inquiries and perspectives in bibliometrics [7]. In addition to the challenges mentioned above in bibliometrics at the publication and individual levels, institutional evaluative bibliometrics faces some unique challenges in theology and religious studies. We briefly discuss some shared and unique challenges at two different research institutions in the field to provide a more comprehensive background of the following sections and share some experiences.

Vanderbilt University is a private, Carnegie-classified R1 university in Nashville, Tennessee. Its Divinity School is “one of only six graduate schools of religion in the United States without a denominational affiliation that services primarily mainline Protestantism [8]”. The Center of Theological Inquiry (CTI) in Princeton, New Jersey, is a research organization that fosters interdisciplinary inquiries in theology and the sciences [9]. CTI brings domain experts from the natural and social sciences to engage in dialogue with so-called “science-informed theologians”. These engagements aspire to provide theologians with sufficient disciplinary understanding, technical knowledge, and other skills to identify areas of shared inquiry and research interest.

Integrated tools commonly used for institutional evaluative bibliometrics in other disciplines, such as Elsevier’s SciVal or Clarivate’s InCites, have limited utility in theology and religious studies due to their skewness in literature coverage and mining performance, which we will discuss in more detail in the following sections. According to Vanderbilt scholars, their works in SciVal/Scopus do not capture their primary impact on the field or the public. Meanwhile, the heterogeneous nature of research and variations in the perception of quality impedes the development and adoption of inclusive and comprehensive indicators and metrics. In addition, these tools are costly and proprietary, creating an additional barrier for institutions such as CTI, which does not have a permanent faculty and has a small professional staff.

Aligned with the recommended approaches to assess research performance [3], CTI has evaluated its research impact using a mixture of quantitative and qualitative methods. On the quantitative side, in addition to tracking scholarly publications where CTI’s support is acknowledged in member scholars’ publications, several other measures include counts of downloads of its white papers, web traffic at its website, discussion of its sponsored research on social media, and performance of its webinars, podcasts, and public lectures. On the qualitative side, CTI values feedback from its members, who serve as informal ambassadors. They share their experiences as inquiry participants and encourage emerging and established scholars to apply for new inquiries.

3. Building a public institutional infrastructure

In 2014, recognizing the need for an integrated metadata infrastructure, the Vanderbilt University Divinity Library [10] started working with the Digital Scholarship and Communications office (now the Digital Lab) [11] on collecting and archiving Divinity School faculty publications. We compiled a bibliography for each faculty member in Zotero and managed data in a relational database. To date, three thousand four hundred and eighty-two publications have been captured in our database. This comprehensive dataset comprises works published from 1966 to 2022 by forty-one faculty, including both currently (twenty-eight) and previously affiliated (thirteen) scholars (nine retired, three left, and one passed away) by summer 2022.

Unlike the approaches adopted by mainstream bibliographic databases that index publications based on the sources (e.g., journals), we collected all works listed on faculty curriculum vitae (CV) at the item level. This more inclusive approach has generated not only a more comprehensive list of formally published works, particularly books, book chapters, journal articles, and conference proceedings, but also a wide range of works such as blogs, podcasts, encyclopedia articles, and dictionary entries, among others that are not indexed by the most prevalent citation databases or in some cases not even considered formal publications. However, our reliance on faculty CVs for publication data is endangered by some established scholars losing interest and initiative momentum to update their CVs. To ensure the comprehensiveness of our dataset, the data were cross-examined against the Web of Science (WOS), Scopus, and the Atla Religion Database, the largest bibliographic database in theology and religious studies. As a result, over two hundred additional publications were acquired.

To build a publicly available institutional infrastructure, we endeavor to make all bibliographic metadata and open-access versions of publications available on our institutional repository and the campus research information management system currently under construction. We are researching copyright policies and seeking permissions from publishers item by item and making efforts to obtain agreement from the authors to deposit their works.

The CTI now takes a similar approach to collecting data from its members. Starting with the 2023-2025 scholarly cohort, we are collecting bibliographic data from members’ CVs and entering it into a Zotero database as well as creating items on Wikidata. Since scholars who apply to become members of the CTI need to evince evidence of a track record of scholarly publications, we assume that their CVs are reasonably complete. Unlike Vanderbilt, however, CTI does not maintain an institutional repository of journal publications, though it keeps a small physical library of members’ books.

4. Evaluating coverages

We conducted a series of analyses to gain a baseline understanding of publications in our database. One of the first goals was to better understand the representation of publications in our dataset versus other scholarly databases. Figure 1 shows the underrepresentation of these works in the mainstream databases. The Atla Religion database, the largest bibliographic database in theology and religious studies, only covers forty-eight percent of all publications we have collected based on faculty CVs. Web of Science (WoS) covers about seventeen percent of the publications, while Scopus only covers ten percent. While these results are somewhat comparable to findings from previous studies on the coverages of humanities research [12,13], it has once again alerted the reliance on these databases for bibliometric analysis.

A previous review of the top one hundred religious studies journals in the SCImago Journal Rank, based on Scopus data, has indicated that Atla contained eighty-seven percent of them [14]. However, despite Atla’s relatively high coverage of theology and religious journals, a more granular examination at the publications in our database has revealed that works of cross-disciplinary scholars are better indexed in Scopus and WoS, which index works from all disciplines. Specifically, the Atla Religion database has the highest coverage among these three databases for all faculty with only three exceptions: one switched her area of study to social sciences and joined another department on campus a few years ago, and Scopus has the highest coverage of her works; the other two have joint appointments with other humanities departments, and WoS has their highest coverage. This finding has confirmed the results of the interdisciplinarity of theology and religious studies, as indicated in previous research [15]. Additionally, for organizations such as CTI, which aims at fostering interdisciplinary research, relying on discipline-specific databases such as Atla would fail to document the broadness of its scholars’ engagement with the social and natural sciences.

Even though CVs in some cases might include incomplete, outdated, biased, and even falsified and fabricated information [16], our case study has provided empirical evidence on the importance of CVs compared to external sources for bibliometrics. This finding aligns with established guidelines such as the Leiden Manifesto [17], emphasizing the importance of maintaining openness and transparency in data collection and analysis and enabling scholars to verify data used in bibliometrics and research evaluation. Our findings also offer insights into the reasons behind the predominantly adverse reactions from scholars in theology and religious studies regarding their acceptance of bibliometrics. As more discussions are on the horizon to invite the endeavor to create a more open and inclusive academic bibliometric system for the humanities, scholars would rather have these issues in their own hands than adopt top-down installed bibliometric products and often over-simplified and misused metrics created by profit-driven vendors with their proprietary services.

5. A closer look at books

Figure 1 provides a detailed perspective of the publication types in our database. Breaking down by publication type, book sections (twenty-six percent), alongside books (eleven percent), collectively constitute approximately thirty-seven percent of the total publications. Including encyclopedia articles and dictionary entries, these book-typed publications comprise forty-four percent of all publications, making a comparable portion as journal articles (forty-eight percent).

OPEN IN VIEWER

Fig. 1.

Vanderbilt University Divinity School Publications by publication types.

Many theology and religious studies schools have conventionally counted on publishing and citing books as a staple of scholarly communication. The percentage of references to books and edited books in religion (eighty-eight percent) has been the highest among the nine humanities disciplines [18]. However, the coverage of Scopus and Web of Science admittedly only features a small fraction of the publications in theology and religious studies, let alone many denominational publications. This practice continues to predominate in the field. The emphasis on monographs or edited books as a vehicle for scholarly communication has complicated the bibliometric analysis of theological and religious scholarship [19]. This has caused the most prevailing citation indexes, such as Scopus and Web of Science, to have minimal utility in bibliometric analysis because they focus on fields where the prevalent means of scholarly communication is via journal articles.

To further compound the challenge, the Atla Religion database, as the largest bibliographic database in theology and religious studies, undeniably has a limited capacity for indexing books. Atla mostly indexes articles, including book reviews and book chapters in edited volumes. Books are not typically indexed in Atla unless they are associated with an edited book volume or until they are associated with a book review article. The findings that Atla covers less than half of the publications listed on our faculty CVs (Fig. 2) and it has even more significant limitations in capturing cross- and inter-disciplinary research discussed above calls for caution when considering it as the gold standard for constructing publication profiles of scholars in the field of theology and religious studies.

OPEN IN VIEWER

Fig. 2.

Coverages of Vanderbilt University Divinity School Publications in Atla, Scopus and Web of Science.

In addition to the unparalleled significance of books in theology and religious studies, our findings have revealed a unique spectrum of non-conventional publications. In our analysis, we considered book-type publications, journal articles, theses, and conference papers as conventional publications and the other types of publications as non-conventional ones, including audio recordings, blog posts, documents, interviews, magazine articles, newspaper articles, podcasts, presentations, reports, video recordings, and webpages (refer to Fig. 1). We should note that considering encyclopedia articles and dictionary entries as conventional publications is unique in theology and religious studies. Several scholars have many well-received publications that fall into the non-conventional category, for instance, a highly utilized website with primary research data, a highly read and shared blog article, or a favorably viewed homiletics video.

These non-conventional publications are mostly born-digital and born-open. While it might be rare to see them formally cited in conventional publications, it is possible to capture a variety of digital traces around them under the umbrella term of altmetrics [20,21], including views, downloads, saves, bookmarks, mentions, and recommendations in blog articles, on Wikipedia, expert review or Q&A platforms, social media platforms and reuses on GitHub, etc. These will help create a richer picture of their reach and usage so that they can be recognized and measured more comprehensively and fairly, as recommended by guidelines such as the San Francisco Declaration on Research Assessment (DORA) [22].

Playing out the backdrop against the dominating practices, our findings illuminate the value of actively engaging scholars in research assessment and combining qualitative and quantitative approaches to assess research impact. An example is the adoption of narrative CVs such as the Résumé for Research and Innovation (R4RI) [23], which can empower scholars to establish a comprehensive account of their research trajectory, highlight their unique experiences and contributions, and facilitate a more holistic evaluation process that instills a deeper understanding of their overall research impact.

7. Persistent identifiers and metadata quality

Unique identifiers serve as essential markers that enable precise identification and tracking of specific entities across different contexts and systems in a more extensive information ecosystem. In the case of the scholarly communication ecosystem, unique identifiers (e.g., the person identifier Open Researcher and Contributor Identifier (ORCID) and the publication identifier Digital Object Identifier (DOI)) effectively capture and represent relationships between objects and stakeholders such as scholars, publications, publication venues, institutions, etc. They play a pivotal role in ensuring the integrity of bibliometrics.

Regrettably, within our sample, a minuscule fraction of book sections and books (four percent) exhibit the presence of DOIs. Journal articles display the most substantial proportion (thirty-three percent) of DOIs. Despite a discernible upward trajectory over time, the prevalence of DOIs in these works still needs to be improved. Meanwhile, ORCID has been adopted by twelve out of twenty-eight current faculty members by summer 2022.

The lack of unique identifiers makes it highly challenging for bibliometrics at all levels, especially on a large scale. More work remains to be done to reap the boon of using persistent identifiers, and multiple stakeholders need to be involved, including publishing and information professionals and scholars. The striving goals from an academic library’s perspective comprise familiarizing scholars with the scholarly communications ecosystem, equipping them with standard identifiers such as ORCIDs, helping to enhance the quality of metadata, and integrating metadata into an open infrastructure such as Wikidata.

8. Wikidata as a bibliographic alternative

Aspiring to increase access to Vanderbilt’s theology and religious studies publications data globally on an open knowledge graph with high-quality and up-to-date metadata, the VandyCite project migrated these data to Wikidata [24], a hub for open-linked data for all the Wikimedia projects [25]. Since Wikidata is built on linked data principles, it is possible to connect authors with their publications and publication with their publishers, etc. Data in Wikidata follows the RDF semantic triple model: subject-predicate-object. Each subject on Wikidata has an identifier (a Q-item) that is linked via properties (P-names) to other Q-items or to numbers, strings, dates, etc. This simple information model has allowed Wikidata to scale to more than a hundred million items since its launch in 2012.

With the automated VanderBot [26,27], as well as manual editing, rectification, and enhancement efforts, Vanderbilt Libraries staff members have completed the upload of the bibliographic metadata of all conventional publications mentioned above for all Divinity faculty. Now, books, book chapters, encyclopedia articles, and dictionary entries often absent in other citation indexes can be viewed on Wikidata and queried by the RDF query language SPARQL [28,29]. Divinity faculty’s scholarly contributions can also be viewed via Scholia [30], a tool developed in the framework of WikiCite [31] that aggregates, analyzes, and visualizes bibliographic metadata on Wikidata. Besides listing publications, Scholia offers scholarly profiles on authors’ and institutions’ information, topic analysis, venue statistics, review statistics, citation statistics, and more [30]. See the various metrics presented, for example, in Professor Fernando Segovia’s profile on Scholia [32].

Open access sources of bibliographic data are skewed toward the STEM fields because data from those fields are generally easier to come by. Vanderbilt’s efforts to improve bibliographic data about theology and religious studies in Wikidata would pay dividends for other research institutions with the potential to remedy the limitations of proprietary tools and break disciplinary barriers.

9. Towards a linked open future for theology and religious studies

Generally, a dual-layered assessment involves an understanding of how well a publication is covered in a citation index: whether the publication itself is accurately indexed; and whether its references and citations are thoroughly indexed. In theology and religious studies, publications are underrepresented when indexed at the starting point because books are less digital, less findable, and less freely-available. Second, the challenge of references and citations needing to be tracked in books exacerbates further underrepresentation in the citation networks. Even in the current possibly most extensive open citation index, the OpenAlex [33,34], only 2.4% (122,409 out of 5,162,215, retrieved on May 30, 2023) of books have references [35]. Up to this point, despite our efforts to make Vanderbilt publications accurately and comprehensively indexed on Wikidata, additional works are warranted to further mine the references and citations, thereby establishing more extensive citation networks.

Although Wikidata as a source of bibliographic data remains unbalanced and “lumpy”, WikiCite is expanding quickly. Since its inception in 2014, WikiCite encompasses around forty-two million publications and two hundred and eighty-eight million citations [36]. With the aid of treating monographs, edited volumes, and periodical publications on an equal footing and removing strictures about what counts as formal publications, Wikidata proves more hospitable to scholars of theology and religious studies than many commercial alternatives. If WikiCite continues to grow at its current pace, other research institutes may consider it an acceptable and, perhaps, preferred option to commercial databases. Over time, information about Wikidata items grows more complete and more interconnected with other items, expanding into a richly layered bibliographic data source.

OpenAlex is another promising resource if it crawls more references and creates entries by linking to databases such as Open Library [37] and WorldCat [38] to enhance the robustness of its information architecture. Other relatively more minor but equally important projects are also emerging in the humanities attempting to mine citations of all kinds, such as the Cited Loci and Linked Books [6]. Theology and religious studies could also benefit from recommendations proposed for the Humanities Citation Index (HuCI): comprehensive coverage including books and citations to primary sources; chronological depth; thematic collection-driven content; rich in context to qualify each citation [39].

With a more interconnected knowledge graph, quantitative studies based on citation relationships will be able to better depict the intellectual structure of fields and subfields related to theology and religious studies. For instance, for CTI, citation analysis can provide a potential method of measuring the degree of interdisciplinary dialogue to better understand the extent to which CTI’s research inquiries have advanced the state of the conversation between theology and the sciences. Bibliographic coupling networks, which focus on shared references among articles, can highlight the research’s intellectual proximity and interdisciplinarity. Co-citation networks, on the other hand, can identify articles frequently cited together and reveal influential works and clusters of related research. Such analyses enable scholars to understand the dynamics of knowledge creation, map research trends, and identify key contributors and critical areas of investigation within a research field.

Ideally, access to open bibliographic data would be coupled with open access to the content itself. The theology and religious studies field has historically proved uneven in its approach to the open-access movement. In some cases, authors and institutions have embraced open access as a digital corollary of their religious mission. In other cases, advocates of open-access policies have met resistance from authors who worry about the loss of trade publishing contracts. As funders in our discipline begin to adopt open access mandates [40], sharing and promoting scholarship will be more feasible, and we should expect the impact of that scholarship, particularly among the public, to increase.