ICRM: An intelligent citation recommendation mechanism based on BERT and weighted BoW models

2.1 Context-based recommendation

Many studies [8–10] used traditional product recommendations for uses. These approaches adopted the content-based, user-based or collaborative filter mechanisms and considered the papers as the product. Therefore, these mechanisms were applied to extract the relationship between authors (user) and papers (product). Then a set of papers can be recommended as the candidates of the citations to a certain author. The process of relationship extraction between authors and papers mainly applied the Bag-of-Words model to generate a vector for each paper or for each author. Then the cosine similarity mechanism was applied to find a set of candidates as the citation recommendation.

Lee et al. [8] presented a recommendation mechanism that recommended articles relevant to the research field of the users based on the traditional product recommendation mechanisms. Finally, the K-Nearest Neighbors was applied to estimate the preference of a target user and then recommended the most preferred papers. Nogueira et al. [9] proposed a two-stage approach: candidate generation followed by reranking. Initially, they assigned a vector to each paper based on the Bag of Word model. Then the abstract and title of the target paper are treated as a query. By taking the query and each vector of document as a pair of inputs, the BERT model will generate a score to represent the relation between the target paper and the input document. According to the scores, the papers with higher scores were selected as the candidates for citation recommendation. However, the Bag-of-Words techniques didn’t consider the context of the words, which represented the semantic similarity between the articles. Dai et al. [10] proposed a novel neural network model called ASL (Attentive Stacked Denoising AutoEncoders (ASDAE) with Bi-LSTM) for context-aware citation recommendation. The proposed ASDAE was used to capture the global attention from citation context when encoding cited paper. They also extended Bi-LSTM with a local attention layer to learn the hidden vectors for citation context, which could capture the key information for effective embedding and benefit for extracting suitable citation context. Personalization also embedded valuable author information to produce personalized citation recommendations. To overcome this limitation, this paper uses the Feature-based Ruler to exploit the context relation between the target article and the considered articles. The Feature-based Ruler extracts the features based on the word2vec model which assigns a vector based on n-skip gram or CBOW. Since the n-skip gram and CBOW establishing a vector to each word is based on a large number of contexts, the vector embeds the semantic similarity among articles. This paper also utilizes the Voronoi Diagram to partition the overlapped citations, aiming to remove the redundant citations and balance the citations.

2.2 BERT-based recommendation

Many studies [13–15] proposed embedding-based approaches to capture the resemblance between the query and the considered article according to the cosine distance or the Euclidean distance between their embeddings. Gökçe et al. [13] derived a subset of papers from the databases according to a user-defined keyword-based filter. They used the document embedding model, called Sent2Vec, to rank the filtered papers. Jeong et al. [14] proposed a BERT-GCN model for context-aware paper citation recommendation. It computed for each paper embeddings of the citation graph and the query context. The learning presentation of context sentences, through pre-trained BERT, achieved high performance. The GCN model was used to represent the citation relationship between papers and to extract a learning representation of them. The BERT-GCN model was evaluated on small datasets of only thousands of papers, partly due to the high cost of computing the GCN, which limited its scalability for recommending citations from a large paper database. In addition, similar to the disadvantage of the previous category, the recommendation only considered the global context of the target article. To recommend citations for the bracket of the target article, the local context near by the bracket is still needed to be further considered.

Gu et al. [15] proposed a novel two-stage local citation recommendation system, which contained prefetching stage and reranking stage. The prefetching stage used an embedding-based paper retrieval system, in which a Siamese text encoder first pre-computes a vector-based embedding for each paper in the database. The query text was then mapped into the same embedding space to retrieve the K nearest neighbors of the query vector. To encode queries and papers of various lengths in a memory efficient way, it designed a two-layer Hierarchical Attention-based text encoder (HAtten) that first computed paragraph embedding and then computed from the paragraph embedding the query and document embedding using a self-attention mechanism. After that, the reranking stage fine-tuned the SciBERT to rerank the candidates retrieved by the HAtten prefetching model. However, the functions of the proposed two stages were similar, which major relied on the attention scheme to extract the relation between citations of previous referenced papers and suffered with overlapping references. This paper proposes Feature-based Ruler and attention-based mechanisms that recommend candidates in two stages, the coarse grain and fine-grain. The two mechanisms can extract the features of each article based on different aspects. In addition, the proposed ICRM adopts the Voronoi Diagram to tackle the overlapping brackets. An Improper-Size Adjustment (ISA) Mechanism is proposed to further balance the number of citations for brackets.

Table 1 presents a detailed comparison between the proposed ICRM and prior related work in various aspects, including context extraction, embedding method, scalability, overlapping citations and citation balance. It highlights the limitations of existing research methods and illustrates how ICRM addresses these limitations with its innovative solutions and advantages.

For context extraction, previous research [8–10] depended heavily on the Bag-of-Words model and lacked a comprehensive understanding of context. The proposed ICRM utilizes a combination of TF-IDF and WCBW to extract more diverse features, enhancing the context and relevance of citations. In terms of the embedding method, ICRM introduces SCI-BERT with features derived from both TF-IDF and WCBW, which contrasts with the homogeneous feature extraction via embedding [13–15] used in previous studies, offering more accurate and complementary feature extraction. In addressing scalability, models like BERT-GCN [14] demonstrated limited scalability. The proposed ICRM uses Voronoi Diagrams for overlapping citations, allowing for more efficient handling of large databases and providing unique categorization. For the challenge of overlapping citations, prior research [13–15] suffered from overlapping references. The ICRM effectively utilizes Voronoi Diagrams for better management and differentiation of such citations. Lastly, in the aspect of citation balance, the ICRM introduces an Improper-Size Adjustment Mechanism, addressing the absence of a balancing mechanism in previous studies [9, 10] and ensuring a balanced distribution of citations across various brackets.

3.1 Problem statement

Assume that there is a set D = {D₁,D₂, . . .  ,D _m } which consists of m papers D _i , for 1≤i≤m. Let U _i ,A _i ,T _i ,K _i ,C _i denote the authors, abstract, title, keywords, and conclusion of the paper D _i , respectively. The paper D _i can be denoted by Exp. (1). D i = { U i , A i , T i , K i , C i } ( 1 ⩽ i ⩽ m )(1)

Let D ^t denote the target paper that needs to find citations. Let U ^t ,A ^t ,T ^t ,K ^t ,C ^t , and L ^t denote the authors, abstract, title, keywords, conclusion, and citations of the paper D ^t , respectively. The paper D ^t can be denoted by Exp. (2). D t = { U t , A t , T t , K t , C t , L t }(2)

Assume that there is an ordered list of citations, considered as the labels, which are cited in n positions of D ^t . The citations L ^t can be denoted by Exp. (3). L t = ( L 1 t , L 2 t , … , L n t )(3)

Let M be a mechanism that predicts the citations of D ^t . Let P j M , t denote the prediction of the j-th position of citations L j t using mechanism M. Assume that there are α prediction cited references P j t = ( p j , 1 t , p j , 2 t , . . , p j , α t ) . Let L j t denote a list of actually cited references of j-th position in the paper D ^t . Assume that there are β actual cited references L j t = ( l j , 1 t , l j , 2 t , . . , l j , β t ) .

Let Boolean variable λ i , j M denote whether the paper D _i is in prediction P j M , t . It can be derived by Exp. (4). λ i , j M = { 1 , D i ∈ P j M , t 0 , otherwise(4)

Let Boolean variable μ _i,j denote whether the paper D _i is in label L j t . It can be derived by Exp. (5). μ i , j = { 1 , D i ∈ L j t 0 , otherwise(5)

According to the values of P i , j M and L_i,j, the confusion matrix can be calculated. Let TP i , j M , FP i , j M , and FN i , j M denote the true positive, true negative, false positive and false negative of the confusion matrix whether D _i is cited in the j-th position L j t , respectively TP i , j M denotes correct prediction that D _i is cited in the j-th position L j t . FP i , j M denotes incorrect prediction that D _i is not actually cited in the j-th position L j t . FN i , j M denotes incorrect prediction that D _i is cited in the j-th position L j t .

The TP i , j M , FP i , j M , and FN i , j M can be derived by Exps. (6)-(8), respectively. TP i , j M = { 1 , λ i , j M * μ i , j = 1 0 , otherwise(6) FP i , j M = { 1 , λ i , j M - μ i , j = 1 0 , otherwise(7) FN i , j M = { 1 , μ i , j - λ i , j M = 1 0 , otherwise(8)

Let TP j M , FP j M , and FN j M denote the true positive, true negative, false positive and false negative of the confusion matrix for all citations of the j-th position, respectively. TP j M denotes correct prediction that all citations are cited in the j-th position L j t . FP j M denotes incorrect prediction that all citations are not actually cited in the j-th position L j t . And FN j M denotes incorrect prediction that all citations are cited in the j-th position L j t .

The TP j M , FP j M , and FN j M can be derived by Exps. (9)–(11). TP j M = ∑ i = 1 m TP i , j M(9) FP j M = ∑ i = 1 m FP i , j M(10) FN j M = ∑ i = 1 m FN i , j M(11)

Let TP ^M , FP ^M , and FN ^M denote the true positive, true negative, false positive and false negative of the confusion matrix for n citations of the paper D ^t , respectively. TP ^M denotes correct prediction that n citations are cited in all brackets. FP ^M denotes incorrect prediction that n citations are not actually cited in all brackets. FN ^M denotes incorrect prediction that n citations are cited in all brackets.

The TP ^M , FP ^M , and FN ^M can be derived by Exps. (12)–(14). T P M = ∑ j = 1 n TP j M = ∑ j = 1 n ∑ i = 1 m TP i , j M(12) F P M = ∑ j = 1 n FP j M = ∑ j = 1 n ∑ i = 1 m FP i , j M(13) F N M = ∑ j = 1 n FN j M = ∑ j = 1 n ∑ i = 1 m FN i , j M(14)

Let PC ^M denote the precision of prediction. It can be calculated by Exp. (15).

P C M = T P M T P M + F P M = ∑ j = 1 n ∑ i = 1 m TP i , j M / - ( ∑ j = 1 n ∑ i = 1 m TP i , j M + ∑ j = 1 n ∑ i = 1 m FP i , j M )(15)

Let RC ^M denote the recall of prediction. It can be calculated by Exp. (16).

R C M = T P M T P M + F N M = ∑ j = 1 n ∑ i = 1 m TP i , j M / - ( ∑ j = 1 n ∑ i = 1 m TP i , j M + ∑ j = 1 n ∑ i = 1 m FN i , j M )(16)

Let F1 ^M denote the F1–score which can be calculated by Exp. (17). F 1 M = 2 * P C M * R C M P C M + R C M(17)

Let Φ denote the set of all possible citation mechanisms, M ∈ Φ, that predict the citations for the target document D ^t .

4.1 Coarse-grained weighted bag of word (WCBW) phase

This phase aims to find the top η appropriate candidates from D for each bracket of the target document. The η candidates are selected based on Weighted Bag of Words and TF-IDF schemes.

Recall that there are m articles in database D. Recall that notations U _i ,A _i ,T _i ,K _i and C _i denote authors, abstract, title, keywords, and conclusion of the article D _i , respectively. That is D _i  = {U _i ,A _i ,T _i ,K _i ,C _i }. And D ^t is the target paper. Let Φ ^D denote the set of words in all documents D _i , for 1≤i≤m, as shown in Fig. 1. That is,

OPEN IN VIEWER

Let notations U ^t ,A ^t ,T ^t ,K ^t and C ^t denote authors, abstract, title, keywords, and conclusion of the target paper D ^t , respectively. Let Φ _i denote the union of A _i and C _i . That is, Φ i = A i + C i(20)

Let Φ _t denote the union of A ^t and C ^t . That is, Φ t = A t + C t(21)

The next step aims to extract the most important q keywords to organize a Weighted Bag of Words from Φ _t and Φ _i for 1≤i≤m. Let f^TF - 1DF (w, D _i ,Φ ^D ) denote the TF-IDF function applied on word w in D _i . Let v w , Φ i TF - IDF denote the TF-IDF value of each word w∈Φ _i . That is, v w , Φ i TF - IDF = f TF - IDF ( w , Φ i , Φ D )(22)

The value of v w , Φ i TF - IDF indicates the importance of the word w which did not frequently appear in other documents but appeared in document Φ _i frequently. Let max^q(V) be the maximum function that returns the q largest values in set V. Let F i A , C denote the set of the q most important words in Φ _i , where notations A and C denote the words in Abstract and Conclusions. The F i A , C can be obtained by applying Exp. (23). F i A , C = arg max q w ∈ Φ i ( v w , Φ i TF - IDF )(23)

Similarly, we may extract the most important p words to organize another Bag of words from Φ _t . Let v w , Φ t TF - IDF denote the TF-IDF value of each word w∈Φ _t . That is, v w , Φ i TF - IDF = f TF - IDF ( w , Φ t , Φ D )(24)

Let F^t,A,C denote the set of the p most important words in D ^t , where A and C denote the words in Abstract and Conclusions. The F^t,A,C can be obtained by applying the following Exp. (25). F t , A , C = arg max p w ∈ Φ t ( v w , Φ i TF - IDF )(25)

Till now, we have obtained the p and q most important words from Φ _t and Φ _i , respectively.

Next, we will illustrate how to present the document vector using the features in F. Let f _j denote the j-th feature in F^t,A,C. That is, F t , A , C = { f i t | 1 ⩽ i ⩽ p }(26)

Till now, the set of F^t,A,C has collected the p most important words which are extracted from the abstract and conclusion of the target document D ^t . The next step is to include the authors, keywords and title in F^t,A,C. Let F ^t be the set extended by including authors, keywords and title of D ^t to F^t,A,C. That is, F t = F t , A , C + U t + T t + T t(27)

Let β be the number of words in F ^t . The β words will be used as the features to measure the similarity between the target document D ^t and each document D _i .

Similarly, each D _i should extract the important features to compare the similarity between itself and D ^t . Recall that F i A , C denotes the set of the most important q words extracted from the abstract and conclusion of the document D _i . Since the words in authors, title and keywords are very important, they should be reserved to represent the features of each D _i . Let F _i be the set extended by including authors, keywords and title of D _i to F i A , C . That is, F i = F i A , C + U i + T i + K i(28)

The words in F ^t will be organized as a Bag of Words to construct a vector for each D _i . However, the Bag of Words uses the words in the Bag to represent a vector of a sentence, a paragraph or a document. One disadvantage of the Bag of Words is that the vector only presents the property of appearance of those words in the Bag. That is, the relationship of the words that appeared in a sentence has been ignored. Since the word2vec adopts the n-skip gram or CBOW to generate the vector for each word w by looking at the relationship of the word w and the neighboring words that appeared in the same sentence of a large number of documents in WiKi, the vector can represent a certain level of semantic.

To cope with this problem, the Weighted Bag of Words is proposed. Each word in F ^t should be transferred to a vector by applying the Word2Vec. Let V ^t denote the ordered list each element is a vector transferred from each word in F ^t . The V ^t will be treated as the Bag of Words. When the Word2Vec generates the vector for one word, it has been trained to extract the relationship of those words in the same sentence. Therefore, the embedded vector of each word has the advantage of exploiting the relationship among words in the same sentence. To achieve this, each feature word in F ^t and F _i should be transferred to an embedded vector. Let f ^{w 2v} be the function of word2vec which returns the embedding vector of a given input word. Let w_i,j∈F _i denote the j-th word in F _i . That is, w_i,j is an important keyword of the article Φ _i . Let v_i,j be the embedding vector generated by f ^{w 2v} for a given input word w_i,j∈F _i . Let V _i denote the set of all v_i,j. That is, v i , j = f w 2 v ( w i , j ) for each w i , j ∈ F i V i = { v i , j | v i , j = f w 2 v ( w i , j ) for each w i , j ∈ F i }(29)

Similarly, let w i t ∈ F t denote the i-th word in F ^t . That is, w i t is an important keyword of the target article D ^t . Let v k t be the embedding vector generated by f ^{w 2v} for a given input word w i t ∈ F t . Let V ^t denote the set of all v i t . That is, v i t = f w 2 v ( w i t ) for each w i t ∈ F i V t = { v i t | v i t = f w 2 v ( w i t ) for each w i t ∈ F t }(30)

Figure 2 gives a conceptual explanation of the abovementioned operations. As shown in Fig. 2, F _i collects α important words from each document D _i . That is, it uses TF-IDF to initially identify the top q important words from the abstract and conclusions of D _i . It then selects α-q important words from authors, title, and keywords to create F _i . Afterward, these words are converted into word vectors using word2vec. Similarity, F ^t collects β important words from D ^t . The V _i and V ^t and be derived through the Exps. (29) and (30), respectively.

OPEN IN VIEWER

Fig. 2

The conceptual explanation of WCBW.

Till now, we have transformed each important word of the article D _i and D ^t to a vector by applying the word2vec scheme.

The next step is to represent each document V _i as a vector DV _i =(ρ _{i ,1},ρ _{i ,2}, . . .  ,ρ _{i, β}) with length β. Then the cos similarity scheme can be applied to select the η most related documents as the candidates for the target document D ^t . Let v k t be the k-th vector in V ^t . Let v k best ∈ V i be the vector with the smallest distance with v k t . That is,

v k best = arg max d ( v k t , v i ) for each v k t ∈ V t(31) where d ( v k t , v i ) denotes the distance of the vector v k t ∈ V t and the vector v _i ∈V _i . Since v k t and v _i are two vectors, the distance function d ( v k t , v i ) can be implemented by applying cos similarity as follows. d ( v k t , v i ) = v k t , v i ∥ v k t ∥ × ∥ v i ∥(32)

Let Pi , k = v k best . That is, put d ( v k t , v i ) as the value of the k-th position of the vector DV _i . Then each document vector can be obtained. As shown in Fig. 3, each document D _i uses a Weighted Bag of Words DV _i to presents its features. The DV _i is derived based on V ⁱ and V ^t . for the document vector DV _i =(ρ _{i ,1},ρ _{i ,2}, . . .  ,ρ _{i, β}) with β tuples, the value of each tuple ρ _i,j is a weight, denoting the distance of the whole document and the j-th keyword v j t ∈ V t of document D ^t . The value of ρ _i,j is a weight value in a range (0, 1), rather than 0 or 1 as defined in the traditional Bag of Word. Therefore, this mechanism is called Weighted Bag of Word, denoted as WBOW in short.

OPEN IN VIEWER

Fig. 3

Weighted Bag of Words (WBOW).

Next, we will select the η candidates that are mostly related to target paper D ^t using the document vectors. As shown in Fig. 4, let D ^I denote the candidates found by the first phase of the proposed mechanism. Let δ_i,t = f ^cos (DV _i ,V ^t ) denote the cosine similarity function which compares the documents vectors V _i and V _t and outputs a similarity value δ_i,t. That is,

δ i , t = v i · V t ∥ v i ∥ × ∥ v t ∥(33)

The set D ^I of k candidates can be derived by applying Exp. (34) as shown in the following. D I = arg max D ˜ i ∈ D n ( δ i , t )(34)

OPEN IN VIEWER

Fig. 4

The most related candidate documents are selected according to their WBOWs.

During this phase, the SciBERT model is utilized to assess the degree of alignment between the candidate documents and the contextual information surrounding the brackets within the target document.

Let D i = { D ˜ 1 , D ˜ 2 , … , D ˜ η } denote the set of η candidates selected by Phase A. Let A ˜ i , T ˜ i , K ˜ i and C ˜ i denote the abstract, title, keywords and conclusions of the i-th candidate, respectively. Let L _j denote the local context of the bracket b _j . The L _j could be the closest r words before and after the bracket b _j . Let Φ t , j Local denote the union of Φ _t and L _j . That is, Φ j Local = A t + T t + K t + C t + L j(35)

Let function f ^sciBERT take Φ t , j Local and Φ _i as inputs and returns value v D t , D i sci - BERT as its output. That is, dv D t , D ˜ i sci - BERT = f sciBERT ( Φ t , j Local , Φ i )(36)

A large value of v D t , D i sci - BERT indicates that document D _i is more related to the local context of the bracket b _j of D ^t .

Let n _j denote the number of required citations for the bracket b _j . The next step is to select the n _j documents that are most related to the local context of the bracket b _j from k candidates. Let C _j be the set of recommended citations for bracket b _j . The value of C _j can be derived by Exp. (37). C j = arg max D ˜ i ∈ D t n j ( v D t , D ˜ i sci - BERT )(37)

This phase aims to adjust the recommendation results obtained from the previous two phases. Each bracket b _j of D ^t has been recommended with a set C _j consisting of n _j recommended citations for b _j . However, this result still has two problems: overlapping and improper size problems. Firstly, the overlapping problem refers to the overlapping citations existing in two or more brackets. This occurs because our model recommends the same citations in different brackets, especially for the neighboring brackets. The other problem is the improper size that some certain brackets might have too many or not enough citation recommendations.

4.3.1 Overlapping adjustment mechanism

This paper introduces the “distance-based mechanism,” which employs Voronoi diagrams to partition citation points, effectively addressing the issue of overlapping citations by considering the distances between all possible pairs of brackets and citations.

Let R ^citation denote the set of all recommended citations. That is, R citation = ∪ j = 1 n C j(38)

Let O ^citation denote the set of citations appearing in more than one bracket. Let O ^bracket denote the set of brackets with overlapping citation recommendations. That is, O citation = { c i | c i ∈ C j ∩ C k for j ≠ k } O bracket = { b j | c i ∈ C j ∩ C k for j ≠ k }(39)

The next step is to assign a vector to each element in O ^citation and O ^bracket . The vector therefore can be treated as a point in pace, aiming to define the distances among these points. The distances among these points are useful to decide the bracket to which each overlapped citation should be finally belonging.

Herein, we will apply the function f ^BERT which takes Φ _i as its input. Let v c i BERT denote the output of f ^BERT . That is,

v c i BERT = f BERT ( Φ i )(40)

Therefore, each c _i ∈O ^citation already has a vector which is the vector corresponding to the vector V _i of document D _i ∈D∪D ^t . However, each bracket still needs a vector. The next step is to assign a vector to each bracket. Recall that notation L _j denotes the closest r words before and after the bracket b _j . The r words in L _j represent the local features of the bracket b _j . Let v bj BERT denote the vector of the bracket b _j . The value of v bj BERT can be derived by applying Exp. (41).

v bj BERT = f BERT ( L j )(41)

Till now, each element in O ^citation and O ^bracket has its own vector and can be mapped as a point in the vector space. The next step is to define the distance d(a, b) between any pair of two points (a, b) as d ( a , b ) = 1 − f c o s ( → ⇀ a , → ⇀ b ) = 1 − → ⇀ a → ⇀ b ‖ → ⇀ a ‖ × ‖ → ⇀ b ‖ ,(42) where  ⇀  → a and  ⇀  → b could be a citation vector or bracket vector.

The following introduces the distance-based mechanism. First, we should construct a Voronoi diagram for all points b _j ∈O ^bracket . Let Ω _j be the voronoi cell of bracket point b _j . One principle of Voronoi diagram is that all points in a Voronoi cell Ω _j are closest to the bracket point b _j , as compared with other bracket point b _k for all j≠k. This implies all citations points falling in voronoi cell Ω _j are closest to bracket point b _j . As a result, each overlapped citation c _i ∈ O ^citation can be assigned to one bracket b _j ∈O ^bracket if the following condition is satisfied. b j = arg min b k ∈ O bracket d ( b k , c i )(43)

Assume that the citation point c _i falling in Ω _j . This indicates that c _i and b _j satisfy Exp. (43). Therefore, the overlapped citation point c _i will be included in bracket b _j .

Figure 5 gives an example where all elements in O ^citation and O ^bracket have been mapped to the two-dimensional vector space. There are three bracket vectors b₁ = [c₁,c₂,c₃,c₄,c₅], b₂ = [c₃,c₄,c₅,c₆,c₇] and b₃ = [c₆,c₈,c₉] where b₁ and b₂ are overlapped and b₂ and b₃ are overlapped. The following presents how to decide that each overlapped citation finally belong to a certain bracket. The set O ^citation of overlapped citations can be derived as c₃,c₄,c₅,c₆. Then the mechanism prepares Φ₃, Φ₄, Φ₅, Φ₆ according to c₃,c₄,c₅,c₆. Then the L₁,L₂,L₃ can be derived from the closest r words before and after the bracket b₁, b₂, and b₃, respectively. Each element in sets Φ₃, Φ₄, Φ₅, Φ₆ and L₁,L₂,L₃ serves as an input of BERT model. The v c 3 BERT , v c 4 BERT , v c 5 BERT , v c 6 BERT and v b 1 BERT , v b 2 BERT , v b 3 BERT can be derived by applying Exps. (40) and (41). These vectors can be mapped onto the space where the voronoi diagram is applied to partition the space into three disjoint areas. According to the falling areas of v c 3 BERT , v c 4 BERT , v c 5 BERT , v c 6 BERT , the overlapped citations can be partitioned into different brackets based on distance-based mechanism, resulting in the following brackets. b 1 = [ c 1 , c 2 , c 3 ] b 2 = [ c 4 , c 5 , c 7 ] b 3 = [ c 6 , c 8 , c 9 ]

OPEN IN VIEWER

Fig. 5

The set of overlapped citations Ocitation = {c₃, c₃, c₃, c₃} of brackets b₁, b₂ and b₃ can be partitioned into different brackets.

5.1 Data sets

The datasets including PubMed, ACL (Association for Computational Linguistics) and arXiv are used for measuring the performances of the compared studies. The PubMed is a biomedical literature database maintained by the U.S. The National Library of Medicine contains over 37 million abstracts and full-text articles. The ACL Anthology currently hosts 83313 articles on the study of computational linguistics and natural language processing. The arXiv is an online preprint platform, covering papers in fields such as physics, computer science, mathematics, statistics, and biology. This dataset contains a large number of academic papers that can be used for research in areas such as machine learning and natural language processing. It contains over 1.7 million articles.

5.2 Simulation results

Table 3 compares the WCBW, HAtten-SciBERT and ICRM in terms of precision, recall and F1-score. The number of articles in dataset varies ranging from 100 to 500. All the compared mechanisms have the common trend that the precision, recall, and F1 are increased with the number of articles in dataset. More articles in dataset help the model distinguish between relevant and irrelevant citations with greater precision. Consequently, the values of precision, recall and F1-score are increased. The proposed ICRM outperforms the WCBW and HAtten-SciBERT mechanisms in all cases. The main reason is that the proposed ICRM incorporates the WCBW and TF-IDF schemes to recommend the most related papers from D as the potential candidates. Additionally, it further examines the detailed relationship between each candidate paper and the given sentences to enhance the Precision, Recall and F1-score of citation recommendation.

Figure 6 (a), (b) and (c) compare the PubMed, arXiv and ACL in terms of Precision, Recall and F1-Score, respectively. The number of citations in dataset varies ranging from 10 to 50. It is obvious that the performance of ACL dataset is generally better than the other datasets.

OPEN IN VIEWER

Fig. 6

Comparison of WCBW, HAtten-SciBERT and ICRM with different datasets.

Table 4 further compares WCBW and ICRM with the PubMed, arXiv and ACL datasets in terms of MRR (Mean Reciprocal Rank). The value of MRR can be calculated by the Exp. (45).

MRR = 1 m ∑ j = 1 m 1 r j(45) where r _j denotes the rank of document D _j among the predicted papers and document D _i is the target document of this citation bracket. As shown in Table 4, the number of articles in dataset varies ranging from 1000 to 10000. It is obvious that the MRR is decreased with the number of articles in datasets. The reason is that it is difficult for the lower-ranked papers to be accurately recommended in the top positions when the number of articles increases. The proposed ICRM outperforms the WCBW and HAtten-SciBERT in all cases. The reason is that the proposed ICRM adopts FSB phase to select the documents that are most related to the local context and citation adjustment phase to adjust the remediation results obtained from the WCBW phase.

Recall that notations α and β denote the user-defined lower and upper bounds for the number of citations in each bracket, respectively. The values of α and β will highly impact the number of citations for each bracket and also affect the number of overlapping citations.

Figure 7 compares the overlapping of different α and β (β≥α). The values of α and β vary ranging from 1 to 6. The overlapping refers to the commonality or shared citations among different brackets under various α and β settings. It is calculated by Exp. (46). overlapping = max ∀ b j , b k ∈ O bracket b j ∩ b k b j ∪ b k ,(46) where b _j denotes the set of citations at j ^th bracket. As shown in Fig. 7, with the increasing of α and β, the overlapping seems to follow an increasing trend, especially in the higher range of α and β. This implies that higher α and β settings may result in more overlapping between citations.

OPEN IN VIEWER

Fig. 7

The relationship between Overlapping and bounds (α, β).

Figure 8 (a) and (b) compare the HAtten-SciBERT and ICRM in terms of overlapping in each bracket, respectively. The top-k documents will be recommended, where 2≤k < 6. The similarity can be calculated by applying Exp. (42). The similarity threshold varies from 0 to 0.7, which is used to allow the maximal similarity for any pair of brackets. A small similarity threshold will restrict the overlapping citations in different brackets. This constraint can reduce the overlapped citations. However, the HAtten-SciBERT did not cope with the overlapping problem. As a result, as shown in Fig. 8(a), the similarity threshold did not impact the overlapping ratio. On the contrary, as shown in Fig. 8(b), the proposed ICRM significantly reduces the overlapping ratio for a smaller similarity threshold value. When the similarity threshold is zero, the ICRM mechanism utilizes Voronoi diagram to assign each citation to a distinct branch, ensuring that there is no duplication as shown in Fig. 8(b).

OPEN IN VIEWER

Fig. 8

Overlapping.

Figure 9 further investigates the impact of (α, β) on the Voronoi average distance. The range of (α, β) varies from (1,1) to (1,6). We intend to observe two brackets with overlapping citation, and the variation in distance between the overlapping citation and the Voronoi edge formed by the two brackets. If the distance is greater, it indicates that the overlapping citation is more likely to be assigned to a specific bracket based on semantics. In this experiment, we attempt to perform the above-mentioned analysis by introducing overlapping citations into two brackets at different distances within the article. To control the variation in distance, we select these two brackets from different sentences within the same paragraph, or from different paragraphs within the same chapter, or from different chapters within the same article for the analysis and investigation. In the experiment, the distributions of citations in different locations, including inter-sentence, inter-paragraph, and inter-chapter, are represented by the proportions (0.5,0.25,0.25), (0.333,0.333,0.333), (0.25,0.5,0.25), and (0.25,0.25,0.5), respectively. As shown in Fig. 10, we use different colors to denote different distances of the two brackets. The number of citations varies from 10 to 25. As shown in Fig. 10, the Voronoi average distance is shortest for inter-sentence citations and farthest for citations placed between chapters. This occurs because that the semantic context of different brackets will exhibit more noticeable differences as the distance between the brackets within the article increases. This makes it easier to distinguish, based on semantics, which bracket an overlapping citation belongs to. Consequently, the distance from the Voronoi edge will also be greater. In addition, an increasing number of citations leads to a rising trend in the Voronoi average distance. This indicates that with a greater number of citations, the similarity between citations decreases, resulting in an increased average distance in the Voronoi diagram.

OPEN IN VIEWER

Fig. 9

The relationship between Voronoi average distance and bounds (α, β).

The experimental results indicate that the proposed ICRM can effectively recommend the appropriate number of citations for individual brackets within a document. However, this paper has several limitations, which are presented in the following.