Summarization for online reviews based on hierarchical attention network

Abstract

With the rapid growth of online reviews, the rapid and effective access to summary information is a hot topic of research. In this paper, an abstract sentence extraction model based on hierarchical attention network is proposed. The model has an encoder-decoder structure, which introduces a two-layer attention mechanism. The sentence encoder uses attention mechanism to obtain the vectorized representation of the sentence by inputting the aspect words. The review document encoder uses the attention mechanism to realize the context association of the sentences before and after. When decoding, a sentence output component composed of the GRU network makes the choice whether the sentence should be selected as the candidate sentence. Greedy algorithm is used to eliminate the redundantly result, and then obtains the final summary based on the sentence sorting method. The experiment results show that the proposed model is better than the benchmark model.

Keywords

Document summarization attention network review analysis

1 Introduction

According to the 42nd China Internet Network Information Center (CNNIC) released the “Statistical Report on the Development of China’s Internet Network” [4], China’s Internet industry is booming. As of June 2018, China’s Internet users reached 8.02 Billion, the Internet penetration rate reached 57.7%, and the number of new netizens increased by 29.68 million in the first half of the year. The development trend is shown in Fig. 1. Among these netizens, the proportion of online shopping users and users using online payment is 71.0%, and online shopping and Internet payment have become the application of higher netizens.

Fig. 1

Summary model.

As the user base continues to expand, a large number of Internet reviews are generated daily, and these reviews are of great value to consumers, businesses, and governments. For potential consumers, users usually pay attention to the user’s reviews on the intended products before purchasing. Checking the reviews in advance is an important way to understand the real situation of a product or service. By browsing more subjective reviews, you can basically understand to the opinion of a product or service that you like, and for the enterprise, the review is the first channel to understand customer feedback, understand product advantages and potential problems, and the company analyzes reviews with opinions. Being able to understand and master the preferences and needs of most users; through continuous monitoring of online public opinion, government departments can grasp people’s attitudes and tendencies towards hot issues in life.

However, the amount of data submitted by these users is huge, and it is increasing. It is impossible to read each review carefully to obtain an overall cognition of a product. However, reading a small part is easy to be partial and cannot be accurately judged. Therefore, how to enable potential consumer users to quickly find valuable information from massive reviews, so that enterprise users and relevant government departments to effectively use the massive reviews generated has become an urgent problem.

2 Related work

Text automatic summarization technology is a research hotspot that continues to be widely concerned. The automatic summary technology can be divided into two major categories, one is a decimation summary, and the other is a production summary.

The abstract is extracted by extracting important sentences in the text. The sentences extracted from the text are smoothed and organized into abstracts. For example, Rada [17] uses each sentence in the text as a node in the graph, using sentences. The similarity construction diagram is based on the PageRank algorithm to iteratively calculate the weights of the nodes in the graph, and according to the weight value as the basis for evaluating the importance of the sentences, the extraction of the text summary sentences is realized. Wong [15] used the support vector machine and naive Bayesian classification to combine the relevance of words, extract multiple features and design the weight rules of the features, and get the importance order of the sentences in the text. In DUC2001 (Document Understanding Conference, 2001) The data set provided showed good results.

With the development of deep learning, the generated text abstracts based on neural networks have developed rapidly, especially the neural network model based on seq2seq (Sequence to Sequence) has many technical methods to implement text abstracts [3 , 24]. However, such methods have the problem of easily repeating the contents of their own texts. See et al. [1] proposed a model of fusion seq2seq and a pointer-generator network, on the one hand, the ability to maintain abstraction through the seq2seq model, and on the one hand, the word generation network directly extracts words from the original text, which improves the inaccurate information generation in the text abstract, the lack of processing ability for unregistered words, and the high repetition rate of abstracts.

In order to facilitate users to understand the main content of the massive reviews, it is necessary to further simplify the review information and summarize the subset of the review sentences. In recent years, the rapid development of deep neural networks has pushed forward the limits of machine intelligence in the fields of image classification and machine translation due to its powerful representation ability. With the help of deep neural networks, automatic text abstracts have also achieved remarkable development. The problem of long-distance dependence is that the network loses a considerable amount of information when it is output, resulting in a summary that is not accurate enough. Therefore, the attention mechanism is applied, which helps the model to acquire important document content and improve the effect of the abstract [11, 21]. In this paper, a summary model Bi-HAN (Binary-HAN) based on Hierarchical Attention Networks (HAN) [5] is proposed for the summary of review information. This model introduces the attention mechanism at two levels. In the sentence layer and the review document layer, the GRU is extended, and the external knowledge from LDA model is used to introduce the aspect word identified in the previous chapter into the input word vector, so that the model has different attention to sentences of different importance in the review document. The ability to get summary information that is valid for the current aspect.

3 Model

Automatic text summaries can generally be divided into two categories: extractive and abstractive. The extractive summary judges the important sentences in the original text and extracts them into a summary.

Definition: The extractive summary task, set S to represent sentences in the review dataset, it is made up of many sentences [s₁, ⋯ , s_N], the model’s goal is to get a sequence of sentences that can represent the summary f, where N means the number of sentence, it can be expressed as f = [f₁, ⋯ , f_N], the element value in the sequence is 1 or 0, 1 means the sentence is selected in the summary, and 0 is the opposite [8]. The task of the summary is to calculate the sequence of tokens with the highest probability.

The model composition based on the hierarchical attention network proposed to solve the above tasks, it includes sentence encoder, document encoder, sentence outputter, and a sentence sorting component, and their relationship in the model is as shown in Fig. 1.

Sentence Encoder: use the word vector to get a vectorized representation of the sentence;

Document Encoder: consists of a bidirectional GRU, which is combined with sentence information to obtain a representation of the comment document;

Sentence output component: the selected sentence according to the sentence information and the encoder that fuses the aspect feature information;

Sentence Sorting Component: sorts the weights of the sentences in the summary based on importance.

In the model, the sentence encoder is represented by a word vector as a fixed-length embedded vector, and the sentence vector is used to describe the document vector [19, 20]. The attention mechanism is introduced at the sentence and document levels respectively. The weight of each time series is calculated first, then the weighted sum of all time series vectors is used as the feature vector, and the output of the last time series is used as the feature vector to perform softmax classification to calculate whether the sentence should be selected in the summary.

3.1 Sentence encoder

The bidirectional cyclic neural network is used to encoding the sentence, and the attention mechanism is used to get the important words in the sentence. The overall structure is shown in Fig. 2.

Fig. 2

Sentence Encoder structure diagram.

First, let s_i to be the i-th sentence, it is made up of many words, words can be represented as [x₁, ⋯ , x_T] where x_t means the t-th word. Each word is replaced with a word embedded representation e_i by a pre-trained word vector which is trained based on the Google News data set by Mikolov [21] before being sent to the neural network.

Since a review is composed of multiple sentences and a sentence is composed of multiple words, a hierarchical structure from the word layer, sentence layer to document layer can be used to model the review. In order to get a better document representation, a bi-directional GRU (gated recurrent unit) is used to model the context information of words and sentences in the document. Embed each word into a hidden state using GRU network h_t = GRU (h_t-1, x_t). The gating mechanism is used to control how information enters the next time node from the last time node, the gate mechanism is expected to be able to preserve the information flow over a longer time series. The update gate is shown as Equation (1), the reset gate is shown as Equation (2). New memory is shown as Equation (3), final memory is shown as Equation (4). $u_{t} = sigmoid (W^{u} x_{t} + U^{u} h_{t - 1} + b^{u})$ (1) $r_{t} = sigmoid (W^{r} x_{t} + U^{r} h_{t - 1} + b^{r})$ (2) ${\tilde{h}}_{t} = tanh (W \cdot [r_{t} * h_{t - 1}, x_{t}])$ (3) $h_{t} = (1 - z_{t}) * h_{t - 1} + z_{t} * {\tilde{h}}_{t}$ (4)

A bi-directional GRU model is used, $\vec{GRU} (•)$ and $\overset{\leftarrow}{GRU} (•)$ represent the GRU process of forward and backward respectively as Equation (5) and Equation (6). $\vec{h_{t}}$ contains semantic information from sentences entered into the current state; $\overset{\leftarrow}{h_{t}}$ encodes information from the current node to the final state of the sentence. Then concatenating $\vec{h_{t}}$ and $\overset{\leftarrow}{h_{t}}$ to get $h_{t} = [\vec{h_{t}}, \overset{\leftarrow}{h_{t}}]$ , which generate a word representation of the encoded sentence. ${\vec{h}}_{t} = \vec{GRU} (\vec{h_{t - 1}}, x_{t})$ (5) $\begin{matrix} \overset{\leftarrow}{h_{t}} = \overset{\leftarrow}{GRU} (\overset{\leftarrow}{h_{t - 1}}, x_{t}) \end{matrix}$ (6) $h_{t} = \vec{h_{t}} \oplus \overset{\leftarrow}{h_{t}}$ (7)

Because not each word has the same contribution to the sentence representation, we should find out the words that have more weight in expressing the meaning of the sentence, use the attention mechanism of aspect to distinguish different vocabulary, construct sentence representation related to aspect, attention mechanism construction structure similar to literature [12].

Introduce the aspect word extracted by LDA model, top 30 words is selected, encode them as a continuous real-value vector u_w, and add it to the input word vector to make the input information richer and improve the ability to capture important features of the word. u_w and h_t was mapped to a transform space. The importance of the t-th word is converted into a normalized weight matrix α_t by the softmax function. $u_{t} = tanh (W^{a} h_{t} + W^{w} u_{w} + b^{a})$ (8) $a_{t} = \frac{exp (u_{s}^{T} u_{t})}{\sum_{t} exp (u_{s}^{T} u_{t})}$ (9) $d_{k} = \sum_{t} α_{t} h_{t}^{k}$ (10)

W^a, W^w, b^a and u_s are parameters within the attention mechanism. After the normalized attention weight matrix is obtained, the hidden state h_t is weighted and summed with the corresponding encoder to obtain a representation vector of the time sentence Equation (10), which measuring the importance of the i-th sentence.

3.2 Document encoder

The document encoder uses the sentence vector sequence s = [s₁, ⋯ , s_N] encoded by the above sentence encoder as an input. The same network structure as the sentence encoder is used, such as Equation (11) and Equation (12). In the review text, there are many possibilities for a sentence to review on a content. The order of the sentences will affect the importance of the sentence as a description of the content, and the splicing $\vec{h_{i}}$ and $\overset{\leftarrow}{h_{i}}$ will be performed to obtain the state vector of the time t, as in Equation (13), the representation contains context information in two directions. In order to facilitate subsequent calculations, the n hidden states H = (h₁, h₂, ⋯ h_n) are combined and recorded. $\vec{h_{i}} = GRU (\vec{h_{i - 1}}, s_{i})$ (11) $\overset{\leftarrow}{h_{i}} = GRU (\overset{\leftarrow}{h_{i - 1}, s_{i}})$ (12) $h_{i} = [\vec{h_{i}}, \overset{\leftarrow}{h_{i}}]$ (13)

The above process uses a two-way GRU model to incorporate the context information of the sentence into its hidden layer representation h_i. In the representation of the document, the aspect factor is also considered, and the attention mechanism is introduced. As mentioned in the previous section, the aspect extracted by LDA [7] model is introduced. The word which is encoded as a continuous real-value vector u_w, a_i is a scalar value representing the importance of the i-th sentence, thereby obtaining a review document vector w of the sentence vector weighted sum. $u_{s} = tanh π (W^{b} h_{i} + W^{a} u_{w} + b^{b})$ (14) $a_{i} = \frac{exp (u_{i}^{T} u_{s})}{\sum_{i} exp (u_{i}^{T} u_{s})}$ (15) $w = \sum_{i} a_{i} h_{i}$ (16)

3.3 Sentence output component

Each sentence of the review document uses a concatenated representation of the corresponding sentence and vector, taking into account the sentence-level and document-level context when predicting whether the sentence is in the digest [2]. The sentence exporter consists of an GRU network that makes a selection of the mark by detecting the sentences in the review corpus.

Given the review document representation and the implicit state H of the review document encoder, the sentence outputter will combine the hidden state of the decoding with the implicit state of its corresponding position encoding to predict whether the representation of the i-th sentence is a 1 or 0 flag: $p (f_{i} | s_{i} w) softmax (sigmoid (h_{i}, \bar{h_{i}}))$ (17) The calculation method of $sigmoid (h_{t}, \bar{h_{t}})$ is as follows: $sigmoid (h_{t}, \bar{h_{t}}) = V tanh (U_{1} h_{t} + U_{2} \bar{h_{t}})$ (18)

In order to estimate the degree of inconsistency between the predicted value of the model and the true value, the loss function used in the model training process is as follows: $f_{loss} = - \frac{1}{n} \prod_{t = 1}^{n} p (f_{t} = \bar{f_{t}} | s_{t}, w)$ (19)

Determine whether a sentence can be filtered as a summary, and its expression is calculated: $\hat{f_{t}} = arg \begin{matrix} max \\ f \in {0, 1) \end{matrix} p (f_{t} | s_{t}, w)$ (20)

In this model, the common sentence filtering method is greedy algorithm and ILP (Integer Linear Programming) for the phenomenon of excessive redundant sentences that often appear in the digest process. Generally speaking, the greedy score is not as good as ILP because it Greed to maximize, while ILP is precisely localized to a limited extent. However, the greedy algorithm provides a good balance between performance and computational cost. This section uses greedy algorithms to filter sentences to achieve the elimination of redundant information. Highly relevant sentences are iteratively added to the abstract, and only when one sentence s_t satisfies $\frac{bigram - overlap (s_{t}, S)}{Length (s_{t})} ⩽ 1 - λ$ When it is placed in the summary. This method gives verification data in the following experiments by whether or not to join the process.

bigram - overlap (s_t, S) representing the sentence s_t and the current summary S The method of selecting the duplicate content on the binary-gram is based on Ren [16], which is set to 0.65. The value of the data set is also discussed experimentally.

4 Experiment

4.1 Experiment DataSet

The public datasets, from TripAdvisor and Amazon, it is used by Wang [10]. Table 1 shows some general statistics of the dataset.

Table 1
Properties of datasets

TripAdvisor Amazon

Number of reviews 192,997 24,259

Number of sentences 2,055,033 268,079

Number of words 37,103,804 4,428,861

Number of unique word tokens 14,709 36,061

Average Std. Dev. Average Std. Dev.

Word length (characters) 8.07 2.84 7.39 2.57

Sentence length (words) 18.06 15.37 16.52 11.42

Document length (words) 192.25 212.06 182.57 200.32

	TripAdvisor	Amazon
Number of reviews	192,997	24,259
Number of sentences	2,055,033	268,079
Number of words	37,103,804	4,428,861
Number of unique word tokens	14,709	36,061
	Average	Std. Dev.	Average	Std. Dev.
Word length (characters)	8.07	2.84	7.39	2.57
Sentence length (words)	18.06	15.37	16.52	11.42
Document length (words)	192.25	212.06	182.57	200.32

4.2 Experimental setup

Four models were selected as baseline comparison models, 1) NNSE model [12], which based on neural network model to achieve abstract extraction, 2) LEAD [21], the method is relatively simple, the first 3 sentences of fixed selection documents are abstract 3) LexRank [9], a graph structure summary method based on feature vector centers; 4) Logistic Regression, hereinafter referred to as LR, used to predict whether a sentence is listed in the abstract.

Since 90% of the sentences in the data set do not exceed 15 words, 90% of the review documents do not exceed 30 sentences, so the sentence length is set to 15, and the review document length is set to 30, for the aforementioned review document encoder and sentence output. Optimize network parameters using the RMSProp optimizer [22] during training. Set the GRU unit size to 600, dropout [23] to 0.2, and learn rate to 0.01.

4.3 Evaluation method of summary

The ROUGE [6] evaluation model is used to calculate the quality of the abstract. ROUGE-N (N = 1, 2, 3, 4) measures the degree of overlap of the n-gram phrases between the candidate summary and the reference summary, and measures the amount of information of the summary. In the experiment, ROUGE-1, ROUGE-2 and ROUGE-L were obtained. Because Owczarzak [14] mentioned the efficiency of ROUGE-2 in automatic summary evaluation, it was used as the main evaluation indicator.

4.4 Results

4.4.1 Summary content

The Bi-HAN model was compared with NNSE, LEAD, LR and LexRank. The verification results of all models are shown in Table 2.

Table 2
The ROUGE Tested in two Dataset

DataSet Model ROUGE-1 (%) ROUGE-2 (%) ROUGE-L (%)

TripAdvisor NNSE 39.33 6.85 17.85

LEAD 33.43 6.93 14.02

LR 30.28 5.43 13.35

LexRank 28.37 6.92 14.74

Bi-HAN 38.85 7.95 18.39

Bi-HAN^r 38.93 7.95 18.51

Amazon NNSE 35.67 7.86 14.48

LEAD 36.70 7.43 12.94

LR 29.74 6.31 13.11

LexRank 35.36 6.01 12.01

Bi-HAN 37.25 7.94 18.02

Bi-HAN^r 38.73 8.61 18.07

DataSet	Model	ROUGE-1 (%)	ROUGE-2 (%)	ROUGE-L (%)
TripAdvisor	NNSE	39.33	6.85	17.85
	LEAD	33.43	6.93	14.02
	LR	30.28	5.43	13.35
	LexRank	28.37	6.92	14.74
	Bi-HAN	38.85	7.95	18.39
	Bi-HAN^r	38.93	7.95	18.51
Amazon	NNSE	35.67	7.86	14.48
	LEAD	36.70	7.43	12.94
	LR	29.74	6.31	13.11
	LexRank	35.36	6.01	12.01
	Bi-HAN	37.25	7.94	18.02
	Bi-HAN^r	38.73	8.61	18.07

Based on the verification of two data sets, it is found that both the benchmark model and the Bi-HAN model are more stable in the score bias, that is, if the algorithm scores lower in one data set and scores in another data set, the result is low, because the two review data sets have little deviation in the statistical characteristics of the text, and the stability of the algorithm is also reflected.

As seen from Table 1, for the three evaluation indicators, such as the black body display, although the NNSE scored the highest ROUGE -1 score in the validation of the TripAdvisor data set, its ROUGE -2 score is lower than Bi-HAN. 1.1%, ROUGE-L is about 2.9% lower. In comparison, the Bi-HAN model is more stable. Regardless of TripAdvisor or Amazon datasets, the Bi-HAN model performed the highest under the ROUGE-2 metric, reflecting better results than several other baseline methods.

In particular, Bi-HAN after performing the redundant culling method, has a slightly improved effect and is underlined in the table. It can be seen that the Bi-HAN model uses the attention mechanism in sentence coding and document coding to improve the acquisition ability of the document theme, and the selected sentences are more representative.

4.4.2 The effect of the attention mechanism

In order to further test the effect of this model, and analyze the role played by each component in the model, Table 3 shows the evaluation indicators of the model described in this chapter after deleting different levels of attention mechanism. Taking the ROUGE-2 indicator as an example, the experiment based on the TripAdvisor data set does not use the attention mechanism of the sentence layer. The score is reduced by 0.54, and the decrease is 6.79%. Without the attention mechanism of the document layer, the score decreases by 0.47, and the decrease is 5.91%; based on the same experiment with Amazon dataset, the scores were reduced by 0.68 and 0.59, respectively, with a decrease of 8.56% and 7.43%, respectively. In the experimental comparison of the same data set, the use of the sentence layer attention mechanism caused a larger drop in the evaluation score. It can be speculated that under the model of the double-layer attention mechanism, the sentence attention mechanism played a relatively speaking role. The more important role, the external input of this mechanism is derived from the aspect information extracted in the previous chapter, which verifies that it is beneficial for abstract extraction.

Table 3
Performance comparison after model reduction

Dataset Model ROUGE-1 (%) ROUGE-2 (%) Comparison in ROUGE-2 with Bi-HAN

TripAdvisor No attention in Sentence 34.36 7.41 6.79% ↓

No attention in Review 36.43 7.48 5.91% ↓

Bi-HAN 38.85 7.95

Amazon No attention in Sentence 33.38 7.26 8.56% ↓

No attention in Review 36.26 7.35 7.43% ↓

Bi-HAN 37.25 7.94

Dataset	Model	ROUGE-1 (%)	ROUGE-2 (%)	Comparison in ROUGE-2 with Bi-HAN
TripAdvisor	No attention in Sentence	34.36	7.41	6.79% ↓
No attention in Review	36.43	7.48	5.91% ↓
Bi-HAN	38.85	7.95
Amazon	No attention in Sentence	33.38	7.26	8.56% ↓
No attention in Review	36.26	7.35	7.43% ↓
Bi-HAN	37.25	7.94

The value of λ mentioned in Section 3.3, which affects whether the sentence will be selected as a summary. If the value is too small, the redundancy elimination effect will be weakened. As many sentences as possible are selected as abstracts, and the value is too large. The number of candidates satisfying the requirements is less. From the condition of redundant screening, the value is related to the length of the sentence in the corpus. The general news corpus and the online review corpus have different statistical characteristics in the sentence, in order to analyze the value. The relationship with the evaluation value was verified on the basis of the experimental data. Figure 3 shows the relationship, the abscissa is the value of 0.05 in steps, the ordinate is the value of ROUGE-1 and ROUGE-2 (in %), the upper curve shows the value of ROUGE-1, below the curve shows the value of ROUGE-2. It can be seen that ROUGE-1 reaches the highest score when it takes about 0.6, and ROUGE-2 reaches the highest score when it is between 0.6 and 0.7. Therefore, it is set to be 0.65.

Fig. 3

The relation of lamda and ROUGE-1/2.

5 Conclusion

In this paper, based on the hierarchical attention network design summary model for the review document data, the GRU is extended, the aspect knowledge is introduced, the aspect vocabulary is introduced, the structure of the GRU is enhanced, and the attention weight mechanism is used to identify words with higher weights in the sentence. These words form a sentence representation, and then form a representation for the text based on the attention mechanism. The model is compared with a series of unsupervised and supervised summary baselines. The experimental results show that the model performs better than most of the baselines in the text, and can use the loop structure to capture context information and introduce attention in two encoders. The attention mechanism effectively improves the saliency sentence extraction of the abstract. Finally, in the experiment, whether or not to add the redundant culling step into the model test, the results show that the redundant culling is beneficial to the summary result of the review.

Next, the evaluation method ROUGE is based on the word correspondence, regardless of the semantic level. If the words are different, even if the semantics are closely, the calculated evaluation value will not be high. As the abstract technology becomes higher and higher, the phenomenon of synonymous different texts will increase. Therefore, there are still some shortcomings in the evaluation method of the abstract, and further consideration for authentication method is needed.

Footnotes

Acknowledgments

This research is supported by Scientific Research Project of Beijing Educational Committee (No. KM201810017005).

References

Marstawi

, Sharef

N.M.

, Aris

T.N.M.

, et al., Ontology-based Aspect Extraction for an Improved Sentiment Analysis in Summarization of Product Reviews, Proceedings of the 8th International Conference on Computer Modeling and Simulation, ICCMS (2017), 100–104.

Yang

, Yang

, Liu

, Han

and Zhang

, Research on 3D positioning of handheld terminal based on particle swarm optimization, Journal of Internet Technology20(2) (2019), 563–572.

Rush

A.M.

, Chopra

and Weston

, A Neural Attention Model for Abstractive Sentence Summarization, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, (2015), 379–389.

C CINI. The 42nd “Statistical Report on the Development of China’s Internet” [EB/OL]. [August 20th in 2018]. http://www.cnnic.net.cn/hlwfzyj/.

Kiddon

, Zettlemoyer

and Choi

, Globally Coherent Text Generation with Neural Checklist Models, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP2016), (2016), 329–339.

Lin

C.Y.

, ROUGE: a package for automatic evaluation of summaries, Text Summarization Branches Out (2004), 74–81.

Blei

D.M.

, Ng

A.Y.

and Jordan

M.I.

, Latent dirichlet allocation, J Machine Learning Research Archive3 (2003), 993–1022.

Nan

, Li

, Jia

, Dong

and Chen

, Application of improved SOM network in gene data cluster analysis, Measurement145 (2019), 370–378.

Erkan

and Radev

D.R.

, LexRank: graph-based lexical centrality as salience in text summarization, Journal of Artificial Intelligence Research22 (2004), 457–479.

10.

Wang

, Lu

and Zhai

, Latent Aspect Rating Analysis on Review Text Data: A Rating Regression Approach, Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, (2010), 783–792.

11.

, Zhang

, Feng

, Jia

and Kong

, Feature extraction and area identification of wireless channel in mobile communication, Journal of Internet Technology20(2) (2019), 544–553.

12.

Cheng

J.P.

and Lapata

, Neural Summarization by Extracting Sentences and Words, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics: ACL 2016, Berlin, (2016).

13.

Lopyrev

, Generating news headlines with recurrent neural networks, (2015).

14.

Owczarzak

, Conroy

J.M.

, Dang

H.T.

, et al., An Assessment of the Accuracy of Automatic Evaluation in Summarization, Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization. Association for Computational Linguistics, (2012).

15.

Wong

K.F.

, Wu

and Li

, Extractive Summarization Using Supervised and Semi-Supervised Learning, COLING ‘08 Proceedings of the 22nd International Conference on Computational Linguistics, (2008), 985–992.

16.

Ren

, Chen

and Ren

, Leveraging Contextual Sentence Relations for Extractive Summarization Using a Neural Attention Model, Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, (2017), 95–104.

17.

Mihalcea

, Language Independent Extractive Summarization, ACL, (2005), 49–52.

18.

Chopra

, Auli

and Rush

A.M.

, Abstractive Sentence Summarization with Attentive Recurrent Neural Networks, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (2016), 93–98.

19.

Goyal

, Garg

and Mishra

V.N.

, New composition of graphs and their Wiener Indices, Applied Mathematics and Nonlinear Sciences4(1) (2019), 163–168.

20.

Yadav

S.K.

, Some results on D-homothetic deformation of (LCS)2n+1-manifolds, Applied Mathematics and Nonlinear Sciences4(1) (2019), 113–128.

21.

Mikolov

, Chen

, Corrado

, et al., Efficient estimation of word representations in vector space, (2013).

22.

Tieleman

and Hinton.

, Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude, COURSERA: Neural Networks for Machine Learning, (2012).

23.

Zaremba

, Sutskever

and Vinyals

, Recurrent neural network regularization, (2014).

24.

Dauphin

Y.N.

, Fan

, Auli

, et al., Language modeling with gated convolutional networks, arXiv preprint, (2016).