A new method of calculating the disruption index based on open citation data

Abstract

This article discusses the method of calculating the disruption index (D index) based on COCI (the OpenCitations Index of Crossref open DOI-to-DOI citations), breaks through the difficulties brought by the acquisition of massive citation data and verifies the reliability of the method of calculating the disruption index based on open citation data based on empirical research. Through empirical research, we found that (1) there is little difference in the number of citation data of focus papers in Web of Science (WoS) and COCI; (2) the levels of disruptive innovation of the papers calculated based on the WoS and COCI are significantly strongly correlated; (3) among the D index and related extended indicators calculated based on COCI, ${DI}_{t}^{nor}$ has the strongest correlation with peer-review indicators, which is consistent with the calculation results based on WoS. Given the broad disciplinary coverage of COCI, although it has not yet been able to fully replace the function of commercial citation databases in research assessment, it can undoubtedly serve as an important source of citation data for further the disruption index and other scientometrics research thereafter.

Keywords

Innovative evaluation open citation data open science reproducible research the disruption index

1. Introduction

In business theory, disruptive innovations are those that create new markets and value networks or enter the bottom of existing markets and eventually displace established market-leading firms, products and alliances. Since Christensen and Rosenbloom [1] of Harvard Business School proposed this theory in 1995, many research cases have proved that seizing the important opportunity of disruptive innovation is crucial to getting a head start in the wave of global technological development. Therefore, in recent years, research related to disruptive innovation has been favoured by a large number of scholars in related fields who have conducted a series of studies on the connotation, characteristics, prediction, identification, evaluation and management of disruptive innovation.

In the current context of the urgent need for disruptive innovation in the field of basic research, the first thing that must be addressed is the inclusiveness of the structure of the science and technology evaluation system and the evaluation paradigm, thus forming a system and culture that supports disruptive innovation. This plays an important role in promoting the national innovation-driven development strategy on the ground and improving national scientific and technological strength, but the evaluation of disruptive innovation in basic research has been a common challenge for the international academic community [2]. How to evaluate research results in terms of scientifically sound disruptive innovations is a common challenge faced by the international academic community.

As the main form of output of current scientific and technological research results, the innovative evaluation of patents and academic papers is a top priority. Currently, for the disruptive innovation evaluation of these two types of scientific research results, Funk and Owen-Smith [3] and Wu et al. [4] have carried out preliminary exploration and proposed two indicators consolidation-disruption (CD) index and disruption index (D index), as shown in equations (1) and (2)) to measure the level of disruptive innovation of patents and academic papers, respectively. Related scholars have also carried out a series of works to refine and expand them. Among them, Leydesdorff and Bornmann [5] proposed an automated computational program based on Web of Science (WoS) citation data. Although this study solved the problem of batch computation of D index, the data needed for the automated computation program proposed in Leydesdorff’s study still needed to be obtained manually by researchers. As an example, in the evaluation study of five scientific and technical talents in the segmented field by Song et al. [6] which selected from 193 focus papers between 2006 and 2018, there were 8465 citation of focus papers and 3,012,930 citations of the reference of focus papers that needed to be acquired. This implies that even if a very small-scale measurement of disruptive innovation is conducted, the citation relationship data collection effort required is quite large. In this regard, Leydesdorff’s study avoids this problem using metadata provided by the Max Planck Digital Library (MPDL), which essentially still relies on commercial citation databases. Although commercial citation databases often have the advantages of rapid data updates, operational sustainability and rich functionality, their high prices are not affordable for all countries and academic institutions. In addition, the current mainstream commercial citation databases usually do not allow researchers to make accessible and in-depth use of the data, nor do they allow researchers to make secondary distribution of the data obtained from the databases. This poses tremendous difficulties for scholars concerned with relevant research to conduct results validation, multi-perspective studies and baseline comparisons. However, when researchers take informal data access channels such as web crawlers, they may pose a greater risk to the access to other databases purchased by the institution, while using open citation data will effectively avoid this problem

C D_{T} = \frac{1}{n_{t}} \sum_{i = 1}^{n} \frac{- 2 f_{it} b_{it} + f_{it}}{w_{it}}

(1)

D = \frac{N_{F} - N_{B}}{N_{F} + N_{B} + N_{R}}

(2)

In this article, the sources and acquisition methods of citation data used in the current calculation of the disruption index and its limitations on research in practice are proposed, and then the calculation method of the disruption index based on open citation data is proposed. Finally, the reliability of calculating the disruption index based on open citation data is verified by empirical research.

2. Overview of data sources for calculating the disruption index

In this article, we collected a total of 48 relevant papers using D index and related extended indices based on WoS and CNKI (China National Knowledge Infrastructure) databases and investigated the citation data sources used and the acquisition methods. Of the 48 research papers, 20 used commercial databases and 30 used open data (as shown in Figure 1). Among the 20 papers that used commercial databases for their research, the main commercial database was WoS, but the access methods were different. Among them, four research teams, Wu et al. [4], Li and colleagues [7 –10], Bornmann and colleagues [5,11 –14] and Liu colleagues [15 –18], mainly relied on the citation data resources provided by Clarivate (directly from Clarivate, MPDL, Indiana University and NSLC, respectively) and Song obtained them through web crawlers. Among the 30 papers using open data for research, a wide variety of open data sets/repositories were involved, including USPTO Open Data, ORCID, APS Open Data, Third-party open data set, MAG, PubMed and OpenCitations Index of Crossref open DOI-to-DOI citations (COCI).

Figure 1.

Data sources, access and frequency of use in studies related to the disruption index.

In terms of the nature of research, both groundbreaking [3] and disruptive [4] studies in this field have used open data sets, and Wu et al.’s [4] study also used commercial databases. However, in terms of study size, most of the studies using larger-scale data still rely on WoS. Third-party data and open data sets from domain institutions are usually applied in the initial exploration of research branches, validation and cross-sectional studies. Among all the data sources involved, those with a wide range of subject areas include WoS, USPTO, MAG, PubMed and COCI with USPTO and PubMed focusing on patents and life sciences research papers, respectively, and MAG being out of service. According to a recent independent analysis by Martín-Martín et al. [19] the coverage of OpenCitations is now close to that of the two major proprietary citation indexes, WoS and Scopus. Thus, compared with the commercial database WoS, COCI has achieved a certain balance between accessibility and domain coverage.

3. Calculation method of the disruption index based on open citation data

3.1. Data source

OpenCitations is an open scholarly infrastructure dedicated to publishing open citation data as linked open data using semantic web technologies, thus providing a disruptive alternative to traditional proprietary citation indexes [20]. OpenCitations was born out of the Initiative of Open Citations (I4OC) was proposed in 2017. The aim of the initiative is to promote structured, separable and open citation data. Structured means that the data representing each publication and citation instance is represented in a common machine-readable format which can be accessed programmatically. Separable means that citation instances can be accessed and analysed without access to the original documents in which the citations were created. Open means that the data can be freely accessed and reused.

As David Shotton [21], one of the two directors of OpenCitations who is also at the Centre for Electronic Research at the University of Oxford, argues, ‘Bibliographic metadata is a key component of the open scholarly ecosystem and should not be placed behind the subscription paywall’. However, there is currently a sharp conflict between the intellectual property rights and data disposal rights of database vendors and the right to know and autonomy about data of the scholarly community and it is often difficult for researchers to have a voice. And achieving this goal could bring many benefits to researchers and related institutions in the field of scientometrics, including but not limited to: increased discoverability of published content which would particularly benefit individuals who are not members of academic institutions that subscribe to commercial citation databases; the ability to build new services from publicly available citation data that benefit publishers, researchers, funding agencies, academic institutions and the public and enhance existing services; create public citation mapping to explore connections between fields of knowledge and track the evolution of ideas and scholarly disciplines [22].

Based on the relevant information disclosed on the OpenCitations website and the article Software review: COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations published by Heibi et al. [23] in Scientometrics, this article summarises the current resources and services provided by OpenCitations (as shown in Figure 2), which include four main parts: data sets, query functions, tools and software. In this article, we focus on the data sets provided by OpenCitations:

OpenCitations Indexes and OpenCitations Meta. OpenCitations Indexes contain information about the citations where citations are not considered as simple links but as data entities. This means that it is allowed to assign descriptive properties to each citation, such as the date of creation, time span and type. The advantage of this alternative approach is that it allows giving descriptive attributes to citations and citations become easier to describe, distinguish, count and process. OpenCitations Meta, on the contrary, is responsible for storing and providing bibliographic metadata for all publications covered in OpenCitations Indexes. The master data set of OpenCitations Index, COCI, provides scholarly citation information between publications covering all scholarly subject areas, containing over 77 million bibliographic resources and over 1.463 billion citation links (as of December 2022 [24]). As a public digital infrastructure [25], OpenCitations has become one of the primary sources of scholarly data for publishers, authors, librarians, funders and researchers [26]. The metadata it contains is expanding at an average rate of 11% per year, and the functionality and support it provides for application programming interfaces (APIs) have been enhanced and expanded [27].

OpenCitations Corpus. OpenCitations Corpus contains publicly downloadable RDF data sets of bibliographic and citation data and is publicly available under the Creative Commons 0 (CC0) protocol. The collection of articles is based on the Europe PubMed Central REST API.

OpenCitations in Context Corpus. OpenCitations in Context Corpus is also an RDF data set that includes bibliographic and citation data mined from the full text of articles, such as text citations, structural elements, rhetorical elements and sequential numbering of structural elements, including text citation pointers [28]. Like OpenCitations Corpus, OpenCitations in Context Corpus content is derived from the open access subset provided by PubMed Central and is also collected using the Europe PubMed Central REST API and publicly distributed under the CC0 protocol.

Figure 2.

Current composition of OpenCitations’ resources and services.

3.2. Data acquisition

OpenCitations provides a four-fold form of access to all data in COCI. First, OpenCitations provides SPARQL endpoints for all published data sets. Second, all data in any OpenCitations published data set can also be retrieved using the HTTP REST API. This provides web developers and users without specialised semantic skills easy access to the data contained in OpenCitations data sets. OpenCitations has also developed user-friendly browsing interfaces that can be used to search data in all OpenCitations data sets. Again, OpenCitations offers files in format of CSV, N-Triples and Scholix on Figshare. The availability of multiple access methods can meet the needs of various types of users in different usage scenarios, effectively relieving the network load on the service provider. The most suitable for large-scale data computing is to use the data dumped in Figshare and download it to the local researchers for further study. Figshare is an online open access repository, where researchers can upload all their research results and make them publicly available. Users can upload files in any format and projects will be given a DOI for free.

3.3. Data processing

Take, for example, the processing of the data set stored on Figshare in CSV format which is a plain text file containing a list of data and usually used to exchange data between different applications. The special value of this format is to solve the problem of storing, transferring and sharing data between different application environments, rather than utilising it directly. After obtaining the dump data from Figshare, it needs to be processed accordingly in order to make it available for efficient use by researchers. Considering the size of resources in COCI data sets, converting data resources in this format into local databases is more beneficial for researchers to perform calculation of the disruption index. Due to Figshare’s file storage setting, the file of COCI data set on Figshare is sliced into multiple CSV format files with annotated time series. The transformation of the data set from multiple fragmented files to a single database can then be done using the data import tool of a visual database management tool such as Navicat. If the hardware environment used for the study does not support visualisation, the same purpose can be achieved by directly using native database software such as SQLite or using a programming language with database manipulation capabilities such as Python (based on sqlite3 lib).

After completing the basic data format conversion, multi-dimensional and multi-level data slicing of the entire database according to attributes such as citation creation time, journal source, authors, etc. and reasonable indexing of the data tables can save time spent in the subsequent calculation of the disruption index of the focus papers. Considering the size of the data and the performance requirements of the actual research, the above model is more suitable for a single researcher or a small team to carry out the relevant practice. If the research institution can provide high-performance equipment, Pandas or an in-memory database such as Redis would be a better choice for processing and would offer the possibility of interactive operations [29].

3.4. Calculation of the disruption index

After Wu proposed the D index, many scholars improved it and carried out further applications and studies according to specific application scenarios (see Table 1). However, we can see from this that no matter how the index is expanded, the index actually depends on three parameters, namely N_F, N_B and N_R. Therefore, it is only necessary to obtain the three relevant parameters of the focus papers to complete the calculation of the disruption index of the focus papers. It just so happens that seven fields are provided in the COCI data set (see Table 2). Once the researcher has selected the focus papers, the three parameters $N_{F}, N_{B} and N_{R}$ of the focus papers can be calculated based on the DOI number of the focus papers and the corresponding fields in the COCI. After obtaining the results of the three parameters, further calculations and analyses are performed based on the specific indices selected by the researchers. The above three-part process is shown in Figure 3.

Table 1.

The disruption index and various extension indicators.

Reference citation	Object	Publication date	Indicators and calculation formula
Funk and Owen-Smith [3]	Patent	2017	$C D_{T} = \frac{1}{n_{t}} \sum_{i = 1}^{n} \frac{- 2 f_{it} b_{it} + f_{it}}{w_{it}}$
Wu et al. [4]	Paper	2019	$D = \frac{N_{F} - N_{B}}{N_{F} + N_{B} + N_{R}}$
Bornmann et al. [13]	Paper	2020	$D I_{t} = \frac{N_{F} - {N_{B}}^{t}}{N_{F} + {N_{B}}^{t} + N_{R}}$
			${DI}_{t}^{nor} = \frac{N_{F} - {N_{B}}^{t}}{N_{F} + {N_{B}}^{t}}$
			$Deln = \frac{N_{b \cdot cited}}{N_{F} + N_{B}}$
Liu et al. [17]	Paper	2021	$D_{Z} = \frac{2 {N_{F}}^{2}}{2 N_{F} + 2 N_{B} + N_{R}}$
			$Rela_D_{Z} = \frac{8 {N_{F}}^{2}}{2 {(N_{F} + N_{B})}^{2} + (N_{F} + N_{B}) \cdot N_{R}}$
Leydesdorff et al. [30]	Paper	2021	$D I^{*} = \frac{N_{F}}{N_{F} + N_{B} + N_{R}}$
			$D I^{#} = \frac{N_{B}}{N_{F} + N_{B} + N_{R}}$
			${DI}_{t}^{b} = \frac{N_{F} - {N_{B}}^{t}}{N_{F} + N_{B} + N_{R}}$
Bu et al. [31]	Paper	2021	$DEP = \frac{N_{B}}{N_{F} + N_{B}}$
Song et al. [6]	Scholar	2021	$newD = Rela_D_{Z} \ln (\sum_{i}^{n} \ln (cit e_{i} + 2))$
Wang et al. [32]	Paper	2022	$mE D_{t} = m_{t} * (α * E D_{s} + (1 - α) * E D_{p, t})$
Zhou et al. [33]	Paper	2022	$DPran k_{i} = c \sum_{j = 1}^{N} (\frac{A_{ji} ({(p_{j} - p_{i})}^{- γ (1 - d_{i}) / 2})}{k_{j}^{out}} (1 - δ_{k_{j} w_{j}, 0}) + \frac{1}{N} δ_{kj w_{j}, 0}) s_{j} + \frac{1 - c}{N},$
Wang et al. [34]	Scholar	2023	$MDRW = \frac{\sum_{i}^{n} \frac{N_{F} - N_{B}}{N_{F} + N_{B} + N_{R}}}{n}$
Ruan et al. [10]	Paper	2023	$MD = citations * \frac{N_{F} - N_{B}}{N_{F} + N_{B} + N_{R}}$
Jiang and Liu [35]	Journal	2023	$JDI = \frac{\sum_{i}^{n} \ln (D z_{i} + 1)}{n}$
Deng and Zeng [36]	Paper	2023	$newD = \frac{\bar{N_{F}} - \bar{N_{B}}}{\bar{N_{F}} + \bar{N_{B}} + N_{R}}$

Table 2.

The meaning of each field of COCI.

Field	Meaning
OCI	Open citation identifier for the citation
Citing	DOI of the referring entity
Cited	DOI of the referenced entity
Creation	Date of Publication of Citing Unit
Timespan	The interval between the publication date of the cited entity and citing entity
Journal_Sc	Records whether the citation is journal self-citation
Author_Sc	Record whether the citation is author self-citation

Figure 3.

Process of calculating the disruption index based on COCI open citation data.

4. Empirical study

4.1. Research object

In order to better reflect the effect of disruptive index calculation based on open citation data, this study chooses 11 papers identified as Landmark in Faculty Opinions 2020 (see Table 3), as well as a Nobel Prize paper and a group of highly cited papers with the same topic and publication time as the research object (see Table 4). Based on these papers, the reliability of the method of calculating the disruptive index based on open citation data is verified.

Table 3.

Landmark papers in Faculty Opinions of 2020.

No.	First author	Title	Source journal
1	Bastard P	Autoantibodies against type I IFNs in patients with life-threatening COVID-19	Science
2	Imachi H	Isolation of an archaeon at the prokaryote-eukaryote interface	Nature
3	Olivieri M	A genetic map of the response to DNA damage in human cells	Cell
4	Reversat A	Cellular locomotion using environmental topography	Nature
5	Merritt JR	A supergene-linked estrogen receptor drives alternative phenotypes in a polymorphic songbird	PNAS
6	Nakane T	Single-particle cryo-EM at atomic resolution	Nature
7	Suzuki K	A synthetic synaptic organizer protein restores glutamatergic neuronal circuits	Science
8	Miga KH	Telomere-to-telomere assembly of a complete human X chromosome	Nature
9	Yip KM	Atomic-resolution protein structure determination by cryo-EM	Nature
10	Gouwens NW	Integrated Morphoelectric and Transcriptomic Classification of Cortical GABAergic Cells	Cell
11	O’Reilly FJ	In-cell architecture of an actively transcribing-translating expressome	Science

Table 4.

Nobel Prize papers and highly cited papers of the same year on the same topic.

No.	First author	Title	Source journal
1	Jinek, M	A Programmable Dual-RNA-Guided DNA Endonuclease in Adaptive Bacterial Immunity	Science
2	Gasiunas, G	Cas9-crRNA ribonucleoprotein complex mediates specific DNA cleavage for adaptive immunity in bacteria	PNAS
3	Zhang, J	Structure and Mechanism of the CMR Complex for CRISPR-Mediated Antiviral Immunity	Molecular Cell
4	Yosef, I	Proteins and DNA elements essential for the CRISPR adaptation process in Escherichia coli	Nucleic Acids Research
5	Westra, ER	CRISPR Immunity Relies on the Consecutive Binding and Degradation of Negatively Supercoiled Invader DNA by Cascade and Cas3	Molecular Cell
6	Hale, CR	Essential Features and Rational Design of CRISPR RNAs that Function with the Cas RAMP Module Complex to Cleave RNAs	Molecular Cell
7	Datsenko, K	Molecular memory of prior infections activates the CRISPR/Cas adaptive bacterial immunity system	Nature Communications
8	Cady, KC	The CRISPR/Cas Adaptive Immune System of Pseudomonas aeruginosa Mediates Resistance to Naturally Occurring and Engineered Phages	Journal Of Bacteriology
9	Swarts, DC	CRISPR Interference Directs Strand Specific Spacer Acquisition	PLOS ONE
10	Gasiunas, G	Mechanism of Foreign DNA Selection in a Bacterial Adaptive Immune System	Molecular Cell

Faculty Opinions, the most authoritative peer-reviewed database in the global biomedical field for nearly 20 years, incorporates the combined efforts of more than 8000 leading international experts worldwide and is a knowledge discovery tool for evaluating published research. Faculty Opinions’ reviewers are leading experts in the life sciences and medicine and provide the quality and rigour of the reviewers is such that they are able to provide comments, opinions and validation of key papers in their fields. The quality and rigour of the reviewers means that researchers can be assured of the quality of the papers they recommend and Faculty Opinions brings these recommendations together to recommend high-quality research to a wider audience.

4.2. Research indicators

The evaluation indices in this study include the following three types of indices: impact indices, innovation indices and peer review indices. Among them, Cumulative Citations for 3 years (CC3) are selected as the impact indicator; $D, D_{Z}, D I_{t}, {DI}_{t}^{nor}, D I^{*},$ ${DI}_{t}^{b}$ , $Rela_D_{Z}$ are selected as the innovation indicator (all the parameters t in the extended indicators were taken as 5) and Faculty Opinions Score (FScore), Faculty Opinions Rating (FStar), Faculty Opinions Evaluation Times (FTime), Weighted Faculty Opinions Rating (FStar_w) and Weighted Faculty Opinions Evaluation Times (FTime_w) were selected as the peer review indicators. The weighted index here refers to the weighting of the ratings and the number of evaluations by the number of participants in the evaluation when the evaluation is completed by multiple reviewers.

4.3. Research results

4.3.1. Comparison of citation relationship data coverage

The number of citation relationship data of the 11 focus papers in the WoS core set (SCIE, SSCI and AHCI) and COCI is shown in Tables 5 and 6 and Figures 4 and 5.

Table 5.

Reference/citation relationship data of Landmark papers in WOS core set and COCI.

No.	References WoS (RW)	Citations WoS (CW)	References COCI (RC)	Citations COCI (CC)	RC/RW (%)	CC/CW (%)
1	36	1211	43	1261	119.44	104.13
2	89	233	93	280	104.49	120.17
3	98	141	60	166	61.22	117.73
4	27	71	31	82	114.81	115.49
5	56	17	62	19	110.71	111.76
6	40	237	38	316	95.00	133.33
7	89	35	89	43	100.00	122.86
8	52	268	63	356	121.15	132.84
9	22	219	26	245	118.18	111.87
10	82	105	40	117	48.78	111.43
11	85	87	86	101	101.18	116.09

Table 6.

Reference/citation relationship data of Nobel Price and relevant papers in WOS core set and COCI.

No.	References WoS (RW)	Citations WoS (CW)	References COCI (RC)	Citations COCI (CC)	RC/RW (%)	CC/CW (%)
1	38	541	45	660	118.42	122.00
2	26	143	27	180	103.85	125.87
3	43	103	43	143	100.00	138.83
4	26	93	27	155	103.85	166.67
5	38	91	40	134	105.26	147.25
6	42	89	41	134	97.62	150.56
7	27	87	28	142	103.70	163.22
8	28	44	36	119	128.57	270.45
9	32	74	45	83	140.63	112.16
10	46	58	34	59	73.91	101.72

Figure 4.

Reference/citation relationship data of Landmark papers in WOS core set and COCI.

Figure 5.

Reference/citation relationship data of Nobel Price and relevant papers in WOS core set and COCI.

From them, we can find that (1) the difference in citation relationship data between most of the focus papers in the WoS core set and COCI is very small, (2) the number of citations in COCI is higher than the number of citations in the WoS core set for all focus papers and (3) only few focus papers showed a large difference in the number of references in the WoS core set and in the COCI.

Phenomenon 1 is due to the fact that the WoS core set contains the vast majority of high-quality journals and therefore also covers the majority of cited literature sources. Phenomenon 2 is due to the fact that COCI also contains a portion of journals that are not included in the WoS core set and this portion of journals also contributes a portion of the citation count. Phenomenon 3 may be due to the fact that the focus papers cite a portion of earlier published literature (without DOI registration) or the journals and publishers (especially some large for-profit publishers [37]) that do not open citation data to Crossref [38].

4.3.2. Comparison and correlation analysis of the disruption index calculation results

Taking the $D_{Z}$ index as an example, the disruptive innovation levels of the selected 11 Landmark papers were calculated based on the WoS core set and the COCI citation data, respectively, and ranked from highest to lowest based on the WoS core set (see Table 5 and Figure 6). We can find that the ranking of the 11 Landmark papers based on the WoS core set and COCI citation data is basically the same and there is a significant strong correlation between the two sets of calculated values (0.9779). The main reason for the differences in $D_{Z}$ values calculated for different focus papers based on the WoS core set and COCI citation data is the different coverage of the two data sources (Table 7).

Figure 6.

The logarithmic calculation result based on WoS and COCI.

Table 7.

The calculation result based on WoS and COCI.

No.	WoS $D_{Z}$	WoS $D_{Z}$ rank	COCI $D_{Z}$	COCI $D_{Z}$ rank
1	24.63	1	43.35	1
8	3.94	2	2.00	4
9	2.71	3	6.88	3
6	1.85	4	9.98	2
4	0.38	5	1.04	5
3	0.28	6	0.73	7
11	0.13	7	0.06	9
2	0.05	8	0.28	8
10	0.04	9	1.04	6
7	0.03	10	0.05	10
5	0.00	11	0.04	11

From the calculation results of Nobel Prize paper and the comparative highly cited papers in two different citation data sources, the level of disruptive innovation of Nobel Prize papers is much higher than that of the comparative highly cited papers. Moreover, the evaluation of disruptive innovation based on COCI can compare the differences of disruptive innovation level of different papers in more detail (Table 8).

Table 8.

The calculation result of Nobel Prize papers and comparative highly cited papers based on WoS and COCI.

	No.	WoS $D_{Z}$	COCI $D_{Z}$
Nobel Price paper	1	9.0851	2.0664
	2	0.5391	0.0469
	3	0.0000	0.0528
	4	0.0000	0.0995
	5	0.0048	0.0443
Comparative group papers	6	0.0020	0.0512
	7	0.0028	0.0910
	8	0.0000	0.0527
	9	0.0000	0.0126
	10	0.0036	0.0058

4.3.3. Comparison of the disruption index extended metrics with peer-reviewed results

The results of different extension indicators of the disruption index calculated for the selected 11 focus paper based on COCI citation data are shown in Table 9. The differences between the ranking of the focus papers and the sum of absolute ranking differences with the results of peer review are shown in Table 10. It can be found that all the extended indices are closer to the results of expert peer review than the original D index, among which the evaluation results of ${DI}_{t}^{no r}$ are closest to the peer review results. This finding validates Wu and Wu’s [39] concerns about the effectiveness of the original D index and is consistent with Bornmann et al.’s [13] prior findings on the effectiveness of different extended disruption indices.

Table 9.

Innovation level of papers based on COCI citation data using different extended disruption indices.

No.	$D$	$D_{Z}$	$D I_{t}$	${DI}_{t}^{nor}$	$D I^{*}$	${DI}_{t}^{b}$	$Rela_D_{Z}$
1	–0.0205	43.3524	0.0487	0.9509	0.0467	0.0455	0.1375
2	–0.0018	0.2768	0.0004	0.1410	0.0016	0.0004	0.0040
3	0.0001	0.7341	0.0038	0.7500	0.0044	0.0038	0.0177
4	0.0000	1.0422	0.0116	0.7826	0.0130	0.0115	0.0508
5	–0.0023	0.0439	0.0005	0.0769	0.0032	0.0005	0.0092
6	0.0071	9.9800	0.0208	0.5451	0.0291	0.0206	0.1263
7	0.0003	0.0503	0.0007	0.6250	0.0010	0.0007	0.0047
8	–0.0027	1.9973	0.0052	0.6108	0.0068	0.0052	0.0224
9	0.0100	6.8756	0.0191	0.7158	0.0227	0.0189	0.1123
10	–0.0008	1.0388	0.0070	0.5775	0.0095	0.0069	0.0355
11	–0.0018	0.0553	–0.0002	–0.0847	0.0010	–0.0002	0.0022

Table 10.

Ranking of innovation level of papers based on COCI citation data different extended disruption indices.

No.	$D$	$D_{Z}$	$D I_{t}$	${DI}_{t}^{nor}$	$D I^{*}$	${DI}_{t}^{b}$	$Rela_D_{Z}$
1	10	0	0	0	0	0	0
2	6	6	8	7	7	8	8
3	1	4	4	0	4	4	4
4	1	1	0	2	0	0	0
5	4	6	4	5	3	4	3
6	4	4	4	2	4	4	4
7	4	3	1	2	4	1	2
8	2	4	2	2	2	2	2
9	8	6	6	5	6	6	6
10	4	4	5	3	5	5	5
11	4	2	0	0	1	0	0
The sum of the absolute ranking differences between the peer-reviewed results	48	40	34	28	36	34	34

The peer review indicators of the selected 11 focus papers in Faculty Opinions are shown in Table 11. The results of correlation analysis between different disruption indices and peer review indices are shown in Table 12. The correlation between ${DI}_{t}^{nor}$ and peer-reviewed indices was the strongest and the D index was negatively correlated with peer-reviewed indices. The other extended indices were weakly correlated with the peer review indices. The strength of the correlations between different extensions disruption indices and peer-reviewed indicators is similar to the previous results of Bornmann et al. [13].

Table 11.

Peer review results of 11 landmark papers.

No.	FScore	Landmark (_w)	Exception (_w)	Very good (_w)	Good (_w)	FTime (_w)	FStar (_w)
1	173.1	1(4)	5(6)	2(3)	0(0)	8(13)	32(62)
2	139.4	1(4)	4(4)	1(2)	0(0)	6(10)	26(52)
3	120.4	1(4)	1(1)	5(7)	0(0)	7(12)	22(50)
4	110.8	1(4)	3(3)	0(0)	0(0)	4(7)	20(44)
5	101.4	1(4)	2(2)	1(2)	0(0)	4(8)	18(44)
6	93.7	1(4)	2(2)	0(0)	0(0)	3(6)	16(40)
7	83.1	1(4)	1(1)	1(2)	0(0)	3(7)	14(40)
8	75.1	1(4)	1(1)	0(0)	0(0)	2(5)	12(36)
9	74.9	1(4)	0(0)	2(2)	0(0)	3(6)	12(36)
10	74.2	1(4)	1(1)	0(0)	0(0)	2(5)	12(36)
11	74.2	1(4)	1(1)	0(0)	0(0)	2(5)	12(36)

Table 12.

Correlation analysis between different extended disruptive indices and peer-reviewed indicators.

Spearman’s correlation	$D$	$D_{Z}$	$D I_{t}$	${DI}_{t}^{nor}$	$D I^{*}$	${DI}_{t}^{b}$	$Rela_D_{Z}$
FScore	–0.267	0.141	0.164	0.419	0.212	0.164	0.182
FStar	–0.254	0.074	0.116	0.363	0.170	0.116	0.135
FTime	–0.135	0.093	0.172	0.473	0.219	0.172	0.195
FStar_w	–0.279	0.009	0.065	0.341	0.112	0.065	0.084
FTime_w	–0.157	–0.028	0.078	0.429	0.095	0.078	0.092

5. Conclusion

5.1. COCI and WoS have considerable citation coverage

The number of citation relationship data of the focus papers in the WoS core set and COCI has not significantly different. Nine of the 11 focus papers can obtain more citation/reference data in COCI than in WoS core set. Only two focus papers have less citation/reference data from COCI than from WoS core set.

5.2. The calculation results based on COCI are consistent with those based on WoS

The levels of disruptive innovation of the papers calculated based on the WoS core set and COCI open citation data, respectively, are generally consistent and the correlation between them is 0.9779 which showing a significant correlation.

5.3. The expanded indices show the same superiority in COCI as in WoS

Among the selected disruption indices and related extension indicators, the correlation between ${DI}_{t}^{nor}$ and peer review indicators is the strongest based on COCI citation data which is consistent with the results of the calculation based on the WoS core set.

5.4. The method based on open citation has its advantage and meaning

The use of the COCI data set for calculation of the disruption index avoids the direct retrieval of citation data from commercial databases and the resulting costly and time-consuming problems, effectively reducing the difficulty of obtaining large-scale citation data which improves the accuracy of scientometric metrics while ensuring the reproducibility of research. Considering the broad disciplinary coverage of COCI [40], although it has not yet fully replaced the function of commercial citation databases represented by WoS in research evaluation [41], it can undoubtedly serve as an important source of citation data for further disruptive index research in the future.

In essence, the use of open citation data represented by COCI for scientometrics research is not confined to the single topic of the disruption index calculation but can be applied to the whole field of scientometrics. Research based on open citation data is conducive to promoting the organic integration of scientology and open science, solving the long-standing problems in academic publishing and scientific communication such as academic plagiarism, data falsification, difficult to verify results and difficult to reuse data, thus promoting the further development of scientometrics.

5.5. The method based on open citation and this study still have some limitations

From the perspective of the method itself, we collected citation relation data of focus papers from COCI in this study. However, the earlier papers usually did not register DOI and some large for-profit publishers that do not open citation data to Crossref. Therefore, this method also has the problem of missing references and citations without DOI.

From the research point of view, we assumed that landmark results must have a high level of disruptive innovation and therefore chose the results of peer review by domain experts as the ‘gold standard’ for disruptive innovation evaluation in this study. However, Faculty Opinions has a limited number of reviewers, there may be bias in peer assessment [42] and peer review tends to find high potential impact results rather than disruptive results [43]. In addition, Faculty Opinions provides a less differentiated scoring mechanism for reviewers and does not provide a good reminder of the degree of variation in the quality of the literature. The practice of Brezis and Birukou [44] illustrates that the correlation between the review results and the quality of the papers will be significantly improved if the number of reviewers is increased to about 10. However, it is very fortunate that the findings obtained in this study based on open citation data are all largely consistent with those obtained from related studies conducted using commercial citation data. In future studies, we will systematically evaluate the effects of calculation of the disruption index based on open citation data and the differences in the evaluation effects of different extended disruption indices based on open peer review data from more different subject areas.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This research received the financial support from the National Social Science Foundation of China (23BTQ085).

ORCID iD

Yuyan Jiang

Availability of data

All reference relationship data can be obtained from opencitations.net, and all peer review information can be obtained from facultyopinions.com.

References

Christensen

Rosenbloom

. Explaining the attacker’s advantage: technological paradigms, organizational dynamics, and the value network. Res Policy 1995; 24: 233–257.

Yang

Zheng

Dong

. How to review basic research with a disruptive innovation nature. Bull Natl Nat Sci Found China 2017; 31: 313–315.

Funk

Owen-Smith

. A dynamic network measure of technological change. Manage Sci 2017; 63: 791–817.

Wang

Evans

. Large teams develop and small teams disrupt science and technology. Nature 2019; 566: 378–382.

Leydesdorff

Bornmann

. Disruption indices and their calculation using web-of-science data: indicators of historical developments or evolutionary dynamics? J Informetr 2021; 15: 101219.

Song

Feng

Long

, et al. Study on discovery of outstanding scientific and technological talents in specific domains based on optimized disruptive index. J Intell 2022; 41: 61–65.

Lyu

Ruan

, et al. Research on the relationship between the interdisciplinarity and radicalness of papers from the perspective of knowledge integration. J China Soc Sci Tech Inf 2022; 41: 263–274.

Lyu

Gong

Ruan

, et al. Does research collaboration influence the ‘disruption’ of articles? Evidence from neurosciences. Scientometrics 2021; 126: 287–303.

Ruan

Lyu

Gong

, et al. Rethinking the disruption index as a measure of scientific and technological advances. Technol Forecast Soc 2021; 172: 121071.

10.

Ruan

Lyu

, et al. Effect of the topic-combination novelty on the disruption and impact of scientific articles: evidence from PubMed. J Inf Sci. Epub ahead of print 15 March 2023. DOI: 10.1177/01655515231161133.

11.

Bornmann

Tekles

. Disruption index depends on length of citation window. Prof Inform. Epub ahead of print 25 March 2019. DOI: 10.3145/epi.2019.mar.07.

12.

Bornmann

Devarakonda

Tekles

, et al. Disruptive papers published in Scientometrics: meaningful results by using an improved variant of the disruption index originally proposed by Wu, Wang, and Evans (2019). Scientometrics 2020; 123: 1149–1155.

13.

Bornmann

Devarakonda

Tekles

, et al. Are disruption index indicators convergently valid? The comparison of several indicator variants with assessments by peers. Quant Sci Stud 2020; 1: 1242–1259.

14.

Osório

Bornmann

. On the disruptive power of small-teams research. Scientometrics 2021; 126: 117–133.

15.

Liu

Shen

Liao

, et al. The research about the improved disruption index and its influencing factors. Libr Inform Serv 2020; 64: 84–91.

16.

Liu

Liao

Zhu

. A preliminary study on the application of disruption index to scientific research evaluation. Inform Stud: Theor Appl 2021; 44: 34–40.

17.

Liu

Shen

Liao

, et al. Research on the stable time window of disruption index. Libr Inform Serv 2021; 65: 49–57.

18.

Liu

Zhu

. Research on the relationship between the disruption of scientific papers and the age of references. Inform Stud: Theor Appl 2023; 46: 60–66.

19.

Martín-Martín

Thelwall

Orduna-Malea

, et al. Correction to: Google Scholar, Microsoft Academic, Scopus, Dimensions, Web of Science, and OpenCitations’ COCI: a multidisciplinary comparison of coverage via citations. Scientometrics 2021; 126: 907–908.

20.

Peroni

Shotton

. OpenCitations, an infrastructure organization for open scholarship. Quant Sci Stud 2020; 1: 428–444.

21.

Shotton

. Funders should mandate open citations. Nature 2018; 553: 129.

22.

Teixeira da Silva

Huang

C-K(K)

Ortega

. Open citations as a tool for bibliometric verification and transparency and for correcting erroneous references. J Scholarly Publ 2023; 54: 60–79.

23.

Heibi

Peroni

Shotton

. Software review: COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations. Scientometrics 2019; 121: 1213–1228.

24.

OpenCitations – COCI, https://opencitations.net/index/coci (accessed 3 December 2023).

25.

Krüger

Petersohn

. ‘I want to be able to do what I know the tools will allow us to do’: practicing evaluative bibliometrics through digital infrastructure. Res Evaluat 2022; 31: 475–485.

26.

Hendricks

Tkaczyk

Lin

, et al. Crossref: the sustainable source of community-owned scholarly metadata. Quant Sci Stud 2020; 1: 414–427.

27.

Lammey

. Using the Crossref Metadata API to explore publisher content. Sci Ed 2016; 3: 109–111.

28.

Peroni

Dutton

Gray Jones

, et al. Setting our bibliographic references free: towards open citation data. J Doc 2015; 71: 253–277.

29.

Light

Polley

Börner

. Open data and open code for big science of science studies. Scientometrics 2014; 101: 1535–1551.

30.

Leydesdorff

Tekles

Bornmann

. A proposal to revise the disruption index. Prof Inform 2021; 30: e300121.

31.

Waltman

Huang

. A multidimensional framework for characterizing the citation impact of scientific publications. Quant Sci Stud 2021; 2: 155–183.

32.

Wang

Mao

, et al. Quantifying scientific breakthroughs by a novel disruption indicator based on knowledge entities. J Assoc Inf Sci Tech 2023; 74: 150–167.

33.

Zhou

X-L

Yang

X-H

, et al. The influence of disruption on evaluating the scientific significance of papers. Scientometrics 2022; 127: 5931–5945.

34.

Wang

Zhou

Zeng

. Evaluating scientists by citation and disruption of their representative works. Scientometrics 2023; 128: 1689–1710.

35.

Jiang

Liu

. A construction and empirical research of the journal disruption index based on open citation data. Scientometrics 2023; 128: 3935–3958.

36.

Deng

Zeng

. Enhancing the robustness of the disruption metric against noise. Scientometrics 2023; 128: 2419–2428.

37.

Lammey

. Crossref at 20 years: what do the community need? Sci Ed 2020; 7: 125–129.

38.

Lammey

. CrossRef developments and initiatives: an update on services for the scholarly publishing community from CrossRef. Sci Ed 2014; 1: 13–18.

39.

. A confusing definition of disruption. SocArXiv. Epub ahead of print 18 April 2019. DOI: 10.31235/osf.io/d3wpk.

40.

Visser

Van Eck

Waltman

. Large-scale comparison of bibliographic data sources: Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic. Quant Sci Stud 2021; 2: 20–41.

41.

Chudlarský

Dvořák

. Can crossref citations replace web of science for research evaluation? The share of open citations. J Data Inf Sci 2020; 5: 35–42.

42.

Bornmann

. Scientific peer review. Annu Rev Inform Sci 2011; 45: 197–245.

43.

Jiang

Liu

. The relationship between absolute disruption index, peer review index and CNCI: a study based on virology papers. Libr Inform Serv 2023; 67: 96–105.

44.

Brezis

Birukou

. Arbitrariness in the peer review process. Scientometrics 2020; 123: 393–411.