An augmented semantic search tool for multilingual news analytics

1 Introduction

News analytics deal with measuring various qualitative and quantitative attributes of unstructured textual news articles. Some of these attributes include information pertaining to sentiments, relevance and novelty aspects. Expressing news articles in the form of numbers and metadata gives ability to manipulate everyday information in mathematical and statistical ways. News analytics are generally obtained through automated text analysis and applied to digital texts using elements from Natural Language Processing(NLP) and Machine Learning(ML) such as latent semantic analysis, support vector machines, "bag of words" among other techniques. Text analytics consist of phases like NER, Text Similarity, and Latent Dirichlet Allocation(LDA)[1, 2].

The quality of news published by several online news agencies remains questionable as amount of crisp content in an article is very low as compared to the size of article and it is quite time consuming for user to get the context out of the article. There can be biased information about a specific person/entity from particular news publishers. Solely relying on such type of biased information might not be good for the user. There can be contradicting statements by different online news publishers about the same news. In few scenarios there is possibility that news articles’ content might not match the headline stated for that article. These are some of the reasons why news analytics has become a mandate in news industry.

Various challenges exist in the front of news analytics. Information gathering, is one of the prominent challenges since huge volume of data is produced every minute. Gathering appropriate information from different sources itself is a challenging task and it becomes even more harder if the articles are in different languages. Web scraping from various news sites is another challenge since each site follows a different strategy for updating the daily news in their respective sites. The problems related to accumulation, representation, and storage of such information in an organised manner for semantic and quick access are still not solved in its entirety [2, 3]. Though various open source libraries and tools are available for text analytics such as predicting stock trends, they are not productive and are still used at experiment level [4]. Google news labs analyse audience trend, and media analytics to help new publishers to publish better quality news articles [5]. Thus, humongous information is available in various news sites. Nevertheless, there is a dearth of generic solution to semantically retrieve information from multilingual news articles based on specific keywords. Machine learning in combination with NLP helps data analysts turn unstructured text into usable data and gain insights from the data. Text data requires a special ML approach due to high dimensionality as a consequence of hundreds of thousands of words and phrases.

This paper proposes a tool to provide user a solution to explore various news from varied sources based on augmented search using specific keywords from different sites in different languages.

The contributions of the paper are:

Design a tool to extract semantic details from news articles published through various internet sources in various languages.

2 Related Work

News analytics involve automated text analysis by using NLP and ML techniques for semantic analysis and classification [6–8]. A thorough analysis of qualitative dimensions of event types in scientific research has been ignored by scholars previously. But, the repeated longitudinal analysis and comparison of organizational behaviors has been done [9]. Within business, news analytics informs event-driven trading strategies and business intelligence applications [10]. Search mechanisms to retrieve semantic information from documents are available that are based on database indexing, neural networks, and matrix decomposition strategies [11–13].

A similar work was identified in [14], which focuses on presenting condensed news data to the user. The data extracted from rss feeds, are readily available in xml format. However, in our proposed model, the data is extracted directly from news websites and is also extended to non-English articles. Our work has implementation of various information retrieval methods to extract data, which is not done in the referred work. Further, the proposed work focuses on retrieving data for a given keyword and sorting the results based on relevance to the keyword as opposed to the referred work that only focuses on a given category to retrieve the news articles. Currently available systems are mostly processing English news articles [15, 16]. Functionalities for sentiment analysis [17, 18] are available but support for non-English articles such as Chinese, Spanish, German, Indian languages etc. are yet to develop [17, 19]. Thus, semantic retrieval from Hindi, Malayalam, articles are not directly available.

This paper provides enhancements to the work carried out in [20]. The proposed tool extends its functionalities for news articles from non-English websites (in this case Hindi and Malayalam). The non-English articles are initially translated to English and then processed for storage and retrieval in the later stages. The information retrieval is executed as a set of multiple parallel processes (8 processes in this case) instead of one process, thereby reducing the time required to complete extracting the necessary information. A text based clustering is applied to obtain common set of keywords, for the given set of articles for a particular date. The search results are sorted in order of importance of the search word in the article. Even the user interface has been enhanced to filter news articles based on the news websites and the source language.

The computational research aiming at automatically identifying Named Entities in texts forms a vast and heterogeneous pool of strategies, methods and representations [19, 22]. A multimodal approach to quantify the entity coherence between image and text in real-world news is adopted in [22]. A system to find important facts from relevant articles and graph based visualization is framed in [23]. One of the first research papers in the field was presented by Lisa F. Rau that describes a system to “extract and recognize [company] names”[24]. It relies on heuristics and handcrafted rules. Feature space for Named Entity Recognition comprises of descriptors or characteristic attributes of words designed for algorithmic consumption. Feature vector representation is an abstraction over text where typically each word is represented by one or many Boolean, numeric and nominal values. For example, a hypothetical NER system may represent each word of a text with 3 attributes: a Boolean attribute with the value true if the word is capitalized and false otherwise; 2) a numeric attribute corresponding to the length, in characters, of the word; 3) a nominal attribute corresponding to the lowercase version of the word. But here the semantics of the documents in terms of topics or how each word in the document can attribute to a topic of the document is not reflected. Hence its necessary to model the topics of each news article to capture its meaning in a succinct form.

Topic modeling is an unsupervised learning technique that aims at discovering the abstract topics from a set of documents. One of the most powerful techniques based on Dirichlet distributions is Latent Dirichlet Allocation(LDA) [1, 20] that was first introduced by Blei, Ng and Jordan in 2003. Researchers have published many articles in the field of topic modelling and applied in various fields such as software engineering, political science, medical and linguistic science [2, 26]. LDA is a generative probabilistic model in which the documents are represented as random mixtures over latent topics. Here, a topic is characterized by a distribution over words. LDA represents topics by word probabilities. The words with highest probabilities in each topic usually give a good idea of what the topic is. Each latent topic in the LDA model is also represented as a probabilistic distribution over words and the word distributions of topics share a common Dirichlet prior as well. Given a corpus D consisting of M documents, with document d having w words, LDA models D based on a generative process as shown in Algorithm 1.

Algorithm 1 The LDA algorithm

1:State Choose a multinomial distribution φ (t) for topic t ∈ 1, . . . , T from a Dirichlet distribution with parameter β

2:Choose a multinomial distribution θ (d) for document d ∈ 1, . . . , M from a Dirichlet distribution with parameter α  

3:for w _i  ∈ d and i ∈ (1, 2, . . . . . , N _d ) do

4:Select a topic z _i from d  

5:Select a word w _i from z _i  

6:end for

In Algorithm 1, words in documents are the only observed variables while others are latent variables φ, θ and hyper parameters (α and β). Thus, the primary objective of this work is to identify latent semantic topics from the news articles in different languages in order to provide the stakeholders summarized and semantic information based on date or keyword search. The content is organized by news scraping and information is extracted from the scrapped articles to be stored in RDBMS. Other details like URL of the articles or URL of similar articles is also shown to view the complete article. In this paper, the terms scraping and crawling are used interchangeably.

3 Proposed Model

A tool has been developed that facilitates storage, management, and search of multilingual news articles in English, Hindi and Malayalam languages. An efficient retrieval of semantic information from multilingual news articles is proposed. This is an extension of the work [20] where only English newspapers were considered for data retrieval. The current system has developed Crawlers for Hindi and Malayalam news articles as well to extract relevant information along with news URLs and date of article. A generic code for scraping from news articles using Scrapy together with BeautifulSoup packages in Python, mitigates the need for separate code for data extraction from different news websites. The crawled details are stored in the database. Periodically, the article content is extracted from the URL stored in the database and preprocessed, which can later be used for retrieving the information. The details extracted by the tool are entities (location, person, organisation) mentioned in news, news summary, keywords mentioned, and other similar news articles. These information are stored in database, which are then displayed to the user, based on the user query. The tool facilitates the retrieval of content from the database, based on user keyword with decreased search time. The automated crawling of news helps in easy maintenance of different news sources. A search engine is built that helps user to find appropriate data that he is looking for. Keyword extraction for non English feeds, Clustering the documents to explore the common keywords in the articles, and LDA based search mechanism that sorts the search result in the order of relevance are the enhancements in the tool. The user interface of the tool is also augmented with provision of search in non-English language. The concept of multi-threading has been used to improvise the retrieval time of the documents based on keyword and date.

The schema layout used for database storage is as shown in Figure 1. The text in grey indicates the name of table. The items listed are the attributes that describe the content stored in the table. The set of underlined attributes form the primary key, that are unique to each data. The primary key for the table news acts as a marker for each of the news article. The primary key in news-source is used for identifying the website for an article present in news. The arrows indicate foreign key constraints. The foreign key is useful as reference to the original news article.

OPEN IN VIEWER

Fig. 1

Schema Diagram of the tables present in the Database, along with the attributes, to store various details of the news. The underlined attributes are the primary key and arrowhead to the primary key represent foreign keys

OPEN IN VIEWER

Fig. 2

The sequence of modules implemented in the tool. The dashed lines indicate the independence of modules of one another

The various operations in the tool are depicted in Figure 2. The dashed lines in Figure 2 denote the modules’ independence from one another. This means that the modules can be easily extended without any complications by the person maintaining the tool and faults can be diagnosed quickly. The tool must be executed in the order shown in the diagram. The code is written in Python, and the libraries used are open source.

The DBMS used is PostgreSQL-10.19 for storage of various information as shown in the schema diagram 1. Once the text is stored in the tables, we use psycopg2 module as an interface between Postgresql and the Python code to perform analytics on text. The outputs obtained from various analytics such as LDA and Clustering are stored back in the tables such as news_lda,news_cluster,news_keywords,news_cluster. The users can query the database with the keywords and the relevant information can be retrieved from the concerned tables. The advantage of the proposed solution is priority based search and filters to filter articles by language and news sites. As the primary focus is to pull all the data related to news articles based on required keywords, we need to have structured format to arrange the data for efficient retrieval of appropriate and relevant articles. So we chose PostgreSQL compared to NoSQL as it has enormous number of extensions with good indexing mechanisms and provide good scalability with inherent concurrency control. However, this approach is not suitable if mere analytics is to be done on multiple entities and does not work if sentences are given for information retrieval instead of a keyword. The tables News_source and News_domain are populated prior to information retrieval, by scraping the required news articles.

3.2 Text Analytics

After the summary and keywords extraction are stored in the database, text analytics aims at Semantic analysis of the articles for efficient Search engine. This stage incorporates Named Entity Recognition, Text Similarity and Keywords/Topic modelling using LDA (Latent Drichlet Allocation), and Clustering.

LDA is a generative probabilistic model of a corpus [25]. It represents documents as random mixtures over latent topics, where a topic is characterized by a distribution over words. LDA represents topics by word probabilities. The words with highest probabilities in each topic usually give a good idea of what the topic is about. These words can act as keywords for the given article. The working of LDA for an article is given in Figure  3. For the given sum of word scores obtained as output of LDA, keywords A,B,C are obtained. Word A has the highest score of 0.56, followed by B with 0.28 and C with lowest score of 0.08. The word, along with the scores is inserted into news_lda table and is later used in keyword based searching.

OPEN IN VIEWER

Fig. 3

Example of how important keywords along with the score are retrieved from the obtained sum of word scores, as a result of performing LDA. In this case A, B, C are the most important keywords, with A having the highest probability and C having the lowest.

Named entity recognition generates tokens from a given article where each token if identified as a Person, a Location, or an Organization is added to the database. To retrieve these details, a NER tagger, using Stanford NER package is used. There is a caveat here. For instance, if in an article, ’New Delhi’ is to be recognized as location, the tokenizing step will split it into different set of tags as two Locations, ’New’, and ’Delhi’. In order to overcome this issue, the current token is compared with previously scanned token. If tags are not same, the previous token is added to the database else the current token is appended to the previous token and stored as an entity. To avoid storage of duplicate entities, a dictionary of scanned entities is made to scan the the current entity and to the database only if it is not available in the dictionary.

4 Experimentation and Results

5 Conclusion

This paper has proposed a tool that facilitates analysis based on unsupervised learning, summarization and semantic information retrieval of news articles from various sources. The tool not only supports news articles in English but also in Indian languages Hindi and Malayalam. It comprises of web crawling, keyword extraction based on LDA, clustering of the articles, storage and management of the articles in an organized form within DBMS with proper indexing structures. Experiments on selective and real news websites in English, Hindi, and Malayalam languages show the feasibility of the proposed system. It has been observed that the retrieval time of the relevant articles is linear with the total size of the articles stored, given the keywords as input. If keywords are not explicitly mentioned, then clustering of the articles is sought to identify the apt set of articles from each clusters based on the common keywords in each of them. Thus, the proposed tool integrates the crawling of articles in different languages, compiling and analysing the news articles from different sources in different languages, and search engine in single platform obviating the need for separate APIs or tools for different functionalities.

In future, this tool can be equipped with crawlers for foreign languages. The search engine can be extended to allow querying the database for multiple sentences and phrases and not just set of keywords. Currently, only unsupervised learning is supported. Supervised learning for news analytics can also be included to understand the performance and accuracy of the system better.

OPEN IN VIEWER

Fig. 4

Graph of average retrieval search time against no of articles. The blue line indicates that there is a linear relation between the no of articles and the time of retrieval of the articles when searched for a given keyword