News events prediction using Markov logic networks

Abstract

Predicting future events from text data has been a controversial and much disputed topic in the field of text analytics. However, far too little attention has been paid to efficient prediction in textual environments. This study has aimed to develop a novel and efficient method for news event prediction. The proposed method is based on Markov logic networks (MLNs) framework, which enables us to concisely represent complex events by full expressivity of first-order logic (FOL), as well as to reason uncertain event with probabilities. In our framework, we first extract text news events via an event representation model at a semantic level and then transform them into web ontology language (OWL) as a posteriori knowledge. A set of domain-specific causal rules in FOL associated with weights were also fed into the system as a priori (common-sense) knowledge. Additionally, several large-scale ontologies including DBpedia, VerbNet and WordNet were used to model common-sense logic rules as contextual knowledge. Finally, all types of such knowledge were integrated into OWL for performing causal inference. The resulted OWL knowledge base is augmented by MLN, which uses weighted first-order formulas to represent probabilistic knowledge. Empirical evaluation of real news showed that our method of news event prediction was better than the baselines in terms of precision, coverage and diversity.

Keywords

Causality inference event representation Markov logic news prediction ontology construction text analytics

1. Introduction

Recent developments in the field of natural language processing (NLP) have led to a renewed interest in textual inference and analytics. One of the most significant ongoing discussions in the field involves predicting future events from text. It is becoming increasingly difficult to ignore the role of event prediction for decision-making in areas such as politics, economics and society. To predict future events, people generally use common-sense knowledge about how the world behaves. This knowledge that expresses their beliefs has already been obtained under specific conditions of causal relationships. Much of this causal knowledge is expressed in unstructured texts produced by them on traditional/social media such as news archives. The process of drawing a conclusion about a causal relationship based on the conditions of the occurrence of an effect is called causal inference.¹ The goal of this research is to provide a model that performs causal inference in textually represented unrestricted and uncertain environments such as online newspapers.

Our investigation did not attempt to extract named entity and relation [1] including causal relation (textual entailment) [2], specifically temporal information (TDT) [3] and future-related information [4] from text. Rather, we performed open information extraction (OIE) [5] in order to extract the semantic event content of the news. Since our goal was to find all the potentially useful facts/events from news corpora on the web, we could not assume any specific target relation type.

While some related works focused on predicting news importance (news impact/volume) [6 –8] and news interests (news filtering/recommendation) [9 –11], other works dealt with predicting dynamic content on the web, often using time-series [12] and state-space models [13]. Several works [14 –16] are also proposed based on numerous comments provided by social networks. Users on networking sites such as Facebook and Twitter usually reflect social changes in their writings simultaneously. This content can be used for predicting future events. A notable characteristic of such approaches is the use of high volume of short texts about social issues. Similar problems are also faced when it comes to event prediction, but this issue has been taken up by many researchers. Stock market prediction [17] poses a similar problem. Here, the problem involves predicting the occurrence of a certain event. Yet another issue related to this area is the use of web search history to predict which documents a user is likely to request next [18] and which top terms will appear in the news of tomorrow [19].

Some studies also focused on computational history [20] in their attempt to harness computational power to support history analysis. From this perspective, history could help us understand the present and predict the future to some extent. A number of studies explored the use of text mining techniques over a large corpus of news articles to extract collective memories [21] and collective expectations [22]. The authors in Michel et al. [23] developed ‘culturomics’, an approach to studying how culture develops in societies, by applying text mining techniques to millions of digitalised books. While these works focused just on the perception of history through studies of societal views of different countries, several attempts [24, 25] have been made for identifying and modelling the broad social forces. Goldstone et al. [24] developed a model for identifying countries that experienced instability, by examining onsets of political instability in countries. They offered a predictive model based on both violent civil wars and nonviolent democratic reversals, suggesting common factors in both types of change. While some historians believed that computational techniques cannot be used to illuminate the past, advocates of ‘cliodynamics’, which was first founded by Turchin and colleagues [26, 27], maintain that the occurrence of various events such as political instability [28] is not a coincidence. The approach to studying cliodynamics is part of a groundswell of efforts to apply scientific methods to history by identifying and modelling broad social forces [25]. One of the limitations of this explanation is that cliodynamics is useful only for looking at broad trends, and it cannot predict specific historical events. However, the patterns and causal relations exposed by it can enlighten politicians and decision-makers.

Our work stands out in many ways from those mentioned above. We present a semantic and language-independent model to predict news events in textually represented unrestricted and uncertain environments. Our approach is based on the semantic analysis and understanding of natural language news texts, as well representing in standard web ontology language (OWL) and reasoning via Markov logic. We use state-of-the-art NLP techniques to extract a large amount of primary propositions (OWL triples) from news text corpora. This study aims to build a predictive model of news events that leverages all types of knowledge generating and representing in standard ontology language OWL and reasoning with a set of causal rules in a probabilistic logic schema.

Recent literature on the topic offers findings about explicit or implicit causal relations between phenomena in the real world that are used for predicting special events. For example, it is observed that when a flood occurs in some countries, cholera also breaks out. So when a flood occurs in a country – considering this country’s geographical and climatic conditions are similar to those countries’– we can expect that cholera will outbreak; hence, we should take urgent action to deal with it. However, modelling these phenomena is very difficult and expensive for many reasons. Foremost among these difficulties and challenges are the following: (1) lack of appropriate sources of data: existing computational models often require appropriate resources with a considerable amount of data for training. Usually, the sources of data are not available with this specification. These resources are very costly and time-consuming to produce. Moreover, before producing it, the design of the data resources, which includes production method, determining the type of data and so on, also necessitates in-depth research. (2) Lack of appropriate computational models: modelling these complex and uncertain phenomena requires computational models that receive a heterogeneous and large data source as input and provide their outcome while determining the weight and effect of each.

To the best of our knowledge, previous studies have not dealt with these challenges. The only outstanding work in this field has been conducted by Radinsky et al. [29 –31], which was used as a logical approach to dealing with a large-scale artificial intelligence (AI) problem of textual events prediction. However, their work still suffers from the problems raised above. One major criticism of Radinsky’s work is that in his research the causal relations were extracted just from the news titles rather than the entire text. Consequently, many of these relations were not discovered by the algorithm and remained overlooked. Another problem with this approach is that it requires a high-precision NLP system for extracting these relations. Besides, it uses the inappropriate approach to extract causal event pairs. Since prior works have dealt with causal relation extraction for causality rule induction, we posit that using logical rule base (a priori knowledge) represented in the first-order logic (FOL) formalism would be efficient for predicting the effect events. Indeed, the aim is to use logical rules that, if any person or machine follows all of these rules, then, for each input sentence (fact/event), it will derive facts (or predict events) close to the output of the other person or machine. The advantage of the formulation and using these logical rules is that first it is a solution to the problem, and second, it can be considered a semi-labelled collection by reforming the outputs, which can be then used in machine-learning methods. One question that needs to be asked, however, is how we can deploy these logical causality rules in the various events for machine common-sense understanding and employ this uncertain knowledge to perform news event prediction. Moreover, dealing with the uncertainty – which is one of the main characteristics of the research problem – has also been ignored in the abovementioned studies, which is perhaps the most serious disadvantage of those methods. In this problem domain, it is not possible to create complete, consistent models of the world. Therefore, the system must act in uncertain worlds, which the real world is. When implementing some uncertainty scheme, we must be concerned with three issues: (1) How to represent an uncertain event? (2) How to generalise an uncertain event? and (3) How to draw inference using an uncertain event?

The authors in another study [32] using a probabilistic approach had made two assumptions for future event prediction. The first assumption is that all events in the real world are generated by a probabilistic model, so causality occurs only within storylines in which events are correlated and interlinked. The other assumption is that all events within the storyline are independent. For the first assumption, it is clear that a correlation between two events does not necessarily imply that one causes the other, as has been emphasised in statistics. The latter also is obviously wrong because events are not necessarily independent, and it is very likely that some events may depend on each other. A serious weakness of this argument, however, is that it requires very complex models for representing a wide variety of knowledge and that it was difficult to incorporate a wide range of domain knowledge. This leads to our second research question. How can we incorporate both types of static (e.g. a priori and contextual) and dynamic (a posteriori) knowledge extracted from multiple and heterogeneous sources of information and base our inference on imperfection and contradictory knowledge? Furthermore, due to the complexity of the problem, the size of the system and its many components, errors are unavoidable [29]. Another challenge resulting from such a complex task is the accumulation of errors between the stages of the pipeline [33] that considerably affects the value of the results. To fix the problem, joint inference [34] was proposed as a key solution. However, we have ever assumed that there exist sufficient training examples for predicting news events, but what if we don’t have any examples? In this case, we need to leverage indirect supervision. To tackle these limitations, a more sophisticated approach for representation and reasoning about future events is necessary in textual environment.

Prior research into textual event prediction has almost exclusively relied on lexical, factual and shallow semantic representations of large amounts of natural language text. While these approaches have proven useful for the task we faced, we feel a deeper semantic understanding of news texts is required to predict future events, leading us to our next research question. Which combination of semantic text analytics techniques is most valuable to represent language-independent and semantically rich event model for textual event prediction?

We used state-of-the-art NLP techniques to extract the semantic event content of news texts. We perceive that it is efficient to extract news events from texts in a semantically deep manner, thus representing it into a machine-readable ontology language OWL model for predicting future events. Ontology models are able to represent a large amount of information using a small number of individuals and relationships, while also developing semantically rich applications such as the one presented in this work. We also feel that enrichment of textual representation schemes by adding semantic roles to OWL triples may contribute to better splitting the sentence (fact/event) into its clause constituents and provide better predictive ability.

Yet a major problem with causal inference in text is supporting both complexity and uncertainty in a much richer semantic way. Complexity can be handled with FOL [35]. It enables us to succinctly represent a wide variety of knowledge. A problem solution for efficiently handling uncertainty is also a probabilistic approach [36], which certainly is very useful for doing this. Given that prior research on textual event prediction has focused solely on the purely logical [29 –31] or purely probabilistic [32] methods, we ask whether combining logic and probability to use the advantages of both in a semantic representation is effective to predict news events. This seeks to address the following question: How effective is the combination of logic and probability in handling both complexity and uncertainty and leveraging both direct and indirect supervision?

This study proposes solutions to fill the gaps in previous research on event prediction. The goal of our investigation is to provide approaches for causal inference to predict future events in textual environments. We intend to develop a new prediction method that can predict effect events it can cause based on a given event represented in natural language. To this end, as an addition to the text news corpus (dynamic knowledge), a set of domain-specific causal rules are fed into the machine to define the probabilistic logical rules in FOL formalism with their weights as a priori knowledge or common-sense knowledge (a part of static knowledge). These rules were injected into the machine independently of natural language by a human expert in line with the formalism of Markov logic. Then, several large-scale ontologies, including general ontologies such as Wikipedia and DBpedia [37], word ontologies such as WordNet [38] and FarsNet [39] and verb ontologies such as VerbNet [40], are employed as contextual knowledge (the other part of static knowledge) to generalise events and generate the predictions. This model is represented in standard OWL to obtain the most appropriate generalisation of the given cause event and then use the probabilistic logical rules to predict output effect events.

The main contributions of this research are as follows. The first contribution is a novel and scalable method that uses a semantic and language-independent technique to predict news events. Second, we propose an event and causality representation model to handle both complexity and uncertainty. The proposed model enables us to define and deploy causal rules to facilitate user intervention for machine common-sense understanding, so the user can comprehend and modify the rules of the model. This common-sense knowledge is formulated by a human expert in a probabilistic logic formalism called the Markov logic network (MLN). The model also leverages indirect supervision through employing small amount of domain knowledge (FOL formulas) plus large-scale joint inference. Third, we propose a framework to fuse different types of knowledge, consisting of both static (e.g. a priori and contextual) and dynamic (a posteriori), in ontology language OWL and support for uncertainty to perform prediction. Finally, we propose to automate our experiments without human involvement in order to measure precision and coverage of predictions, which has not been observed in previous studies and will allow providing richer metrics of performance such as diversity.

The rest of this article is organised as follows. Owing to importance and extent of ontology construction and utilisation to predict future events, this issue will be discussed separately in section 2. In section 3, the problem definition and solution are described in detail. The evaluation metric and experimental results are discussed in section 4. Finally, the conclusion and future work are presented in section 5.

2. Domain ontology

One of the most important aspects of this research is to implement and manipulate a variety of ontologies. Notably, all the knowledge acquired and extracted from multiple heterogeneous sources must be stored and represented in the formalism of ontology language. An ontology language is a formal language used to construct an ontology. It represents the encoding of knowledge about specific domains and commonly includes logical rules for reasoning of the knowledge.

Ontology construction is generally an important step of ontology-based information extraction (OBIE) [41], which is to say that it follows the paradigm of OIE [5]. An OBIE is a system that processes unstructured or semi-structured natural language text through a mechanism guided by ontologies to extract certain types of information and presents the output using ontologies [42]. We can extract information related to an ontology for representing them as instances of the ontology and reasoning of future events. We describe the ontology representation and reasoning tasks in subsections 2.1 and 2.2, respectively, and investigate the prerequisites of an appropriate ontology construction for probabilistic schema in subsection 2.3.

2.1. Ontology representation

A prerequisite of ontology utilisation in the Semantic Web is developing a standard for definition and interchange of ontologies. Knowledge representation languages enable us to define and communicate among ontologies and to achieve passivity and compatibility with each other. The most important current standards for ontology representation express information as linked data [43]. Linked data is a method for representing and publishing huge amount of related data on the web in a standard, machine-readable format for drawing inferences. For this purpose, the World Wide Web Consortium (W3C) has designed Resource Description Framework (RDF) as a standard model for data interchange on the web. RDF is a general framework for conceptual description or representation of knowledge performed in the World Wide Web, by extending the linking structure of the web to use uniform resource identifiers (URIs) to name the relationship between things and the two ends of the link, together referred to as a ‘triple’. An extension language based on RDF was the Defense Advanced Research Projects Agency (DARPA) Agent Markup Language (DAML). This language was then followed by an extension called DAML + OIL (Ontology Inference Layer or Ontology Interchange Language). Later, it was the starting point for the OWL.

OWL is a computational logic-based language designed to represent rich and complex knowledge for authoring ontologies. There are three variants of OWL – OWL Lite, OWL DL and OWL Full (ordered by increasing their expressiveness). Out of these three, only OWL Full is compatible with RDF schema. The current version of OWL is also referred to as ‘OWL 2’. Ontologies in OWL format provide for a standardised means of representing, querying and reasoning over large-scale knowledge bases. They comprise a syntax for illustrating and incorporating ontologies and formal semantics that give them meaning.

The classes, subclasses and instances are used to semantically classify things in OWL. OWL semantics are based on description logics (DLs) [44]. DL is a family of logic-based formal knowledge representation languages that can be used to represent the knowledge of an application domain in a structured and formally well-understood way [45]. Most DLs can be considered as (decidable) fragments of FOL and less expressive than it, although some provide operators such as transitive closure of roles that requires second-order logic [46]. In this article, we have developed a framework that augments and reinforces the ontology language OWL for representing and reasoning with uncertainty based on the combination of the expressive power of FOL and the probabilistic uncertainty management of Markov random fields (aka Markov networks). A Markov network is a set of random variables having a Markov property described by an undirected graph. Using Markov networks associated with FOL allows us to support uncertainty reasoning.

2.2. Ontology reasoning

Ontology reasoning is the process of inferring logical consequences using an inference engine to derive facts that are not expressed in ontology explicitly. The inference rules are generally determined by means of an ontology language and often a DL language. Reasoning in most DLs is sound and complete, which is ensured to conclude every consequence of the knowledge in the ontology. Baader et al. [45] showed that DL interpretations can be viewed as FOL interpretations and vice versa. This is done through two translation functions, which preserve the semantics. Accordingly, both DL and FOL semantics are convertible to each other, so that all concepts and roles in DLs can be depicted as unary and binary predicates (relations) in FOL, respectively, as well as corresponding classes and properties in OWL.

Using only FOL in ontology language OWL depends on the assumption that all facts/events are true, though not in the real world. To handle the uncertainty arising from this crisp logic, we extend the ontology language OWL to obtain the weighting of the concepts, properties and instances in the domain and to perform reasoning with imperfection and contradictory knowledge. We propose to incorporate Markov networks and ontology language OWL, which together form a MLN. More details are provided in subsection 3.3.

2.3. Investigated ontologies

Nowadays, there are numerous ontologies for research projects. One of the largest existing ontologies containing more than 1 billion facts has been used in Radinsky et al. [29]. To use these large-scale ontologies, it is required to use high-performance servers and parallel processing infrastructure. For operational usage of ontologies, we should consider their scale (number of triples), size of dataset and required physical memory. Table 1 shows specifications of the investigated ontologies from this perspective. Since we need general knowledge in this study, the domain-specific ontologies were not considered in this comparison.

Table 1.

Specifications of the investigated general ontologies

Ontology	Size of dataset (number of triples)	Required memory (GB)
BTC Dataset 2014 (http://km.aifb.kit.edu/projects/btc-2014/)	~4 billion	1000+
BTC Dataset 2012 (https://km.aifb.kit.edu/projects/btc-2012/)	~1.5 billion	500+
BTC Dataset 2011 (https://km.aifb.kit.edu/projects/btc-2011/)	~2 billion	500+
BTC Dataset 2010 (https://km.aifb.kit.edu/projects/btc-2010/)	~3.2 billion	1000+
BTC Dataset 2009 (https://km.aifb.kit.edu/projects/btc-2009/)	~1.14 billion	500+
Data-gov Wiki (https://data-gov.tw.rpi.edu/wiki)	5+ billion	1000+
DBpedia (http://wiki.dbpedia.org/)	8+ billion	1000+
Freebase (https://www.rdf.freebase.com/)	~337 million	500+
OpenCyc (http://sw.opencyc.org/)	~1.6 million	10
YAGO (http://mpii.de/yago)	~120 million	1000+

BTC: Billion Triples Challenge.

In this article, we focused on entities available on DBpedia [37], the most important open web source of data that essentially makes the content of Wikipedia available in RDF using Semantic Web and linked data technologies. The DBpedia 2015–10² release contains 8.8 billion facts (RDF triples), out of which 1.1 billion were extracted from the English edition of Wikipedia, 4.4 billion were extracted from other language editions and 3.2 billion came from DBpedia Commons and Wikidata. This release consists of 127 different language editions of Wikipedia, currently describing 6.2 million instances. These were extracted only from the English edition of Wikipedia.

DBpedia is interlinked with various other data sources following linked data on the web, for example, with Freebase, Flicker, GeoNames, LinkedGeoData, OpenCyc, Wikidata, Wordnet and YAGO. By incorporating facts from those extra links (in terms of RDF triples), we may obtain further and more accurate knowledge to draw inference and subsequently predict news events. In brief, the main reasons for using DBpedia in this research are as follows:

It is a large-scale ontology that covers a wide range of topics and many domains.

It serves as multilingual knowledge base facilitating knowledge transfer into other languages.

It includes Wikipedia data which automatically evolves as Wikipedia changes.

It represents real community agreement.

It incorporates links to other Semantic Web source of data, which can be used to further enrich the content of the knowledge base.

According to the experiments conducted by previous researchers [29 –31], manipulation of large-scale ontologies usually is not possible on a single computer and needs parallel or distributed computing platforms. On the one hand, DBpedia is a large-scale, multilingual, cross-domain ontology comprising more than 1 billion RDF triples. On the other hand, all this information is not necessary for reasoning and predicting future events. To this end, we suggested two techniques for generalisation and summarisation of instances in DBpedia ontology, which are described, respectively, in subsections 2.3.1 and 2.3.2. The implementation of a parallel and distributed platform for processing and reasoning ontology can be considered in future work.

2.3.1. Generalisation of instances in ontology

The DBpedia dataset was widely used in many research communities, and so far numerous applications, algorithms and tools have been developed to manipulate or apply the ontology. Due to its large schema, size of the dataset and the diversity of topics, manipulation of DBpedia poses a major challenge in this domain. Especially, development of powerful systems using DBpedia in this field based solely on simple patterns or domain-specific dictionaries is difficult due to its size and broad coverage. Therefore, a prediction system that provides reliable answers by drawing inference from DBpedia can be considered as an intelligent system. So, in order to increase the performance of using DBpedia, we have processed the DBpedia instances and properties in comma-separated values (CSV) file format. The goal of the processing performed in this subsection is illustrated with an example in the following:

Example 1. Suppose that there is a triple in the CSV file of DBpedia properties as follows:

<http://dbpedia.org/resource/Tehran> <http://dbpedia.org/ontology/capital> <http://dbpedia.org/resource/Iran>

First, the system should search the first entries of DBpedia instances according to each entry in the given triple and then take the most frequent ‘type’ in DBpedia instances for that entry. For example, assume that the most frequent ‘type’ for the first entry of the above triple is as follows:

<http://dbpedia.org/resource/Tehran> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://dbpedia.org/ontology/City>

And also for the third entry, the most frequent ‘type’ is as follows:

<http://dbpedia.org/resource/Iran> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://dbpedia.org/ontology/Country>

Eventually, a rule can be added to the system as follows:

<http://dbpedia.org/resource/City> <http://dbpedia.org/ontology/capital> <http://dbpedia.org/resource/Country>

This result indicates that the ‘capital’ relation is established between the two entities of City and Country. This process was repeated for each triple entry in the CSV file of DBpedia properties and so the need for powerful servers and parallel processing infrastructures is decreased significantly.

2.3.2. Summarisation of instances in ontology

One of the most useful aspects of the proposed framework is providing facilities for ontology summarisation. General ontologies such as DBpedia include a lot of entities. The large amount of data in these ontologies can slow down the process of loading and inference in ontology. It is noteworthy that most data in general ontology are not related to the local language news and entities that are required for prediction. Besides, many of them are not in the domain of the prediction system. Hence, summarisation of instances in the ontology does not hamper system performance. Accordingly, we used the summarisation of both classes and properties to reduce the number of ontology instances. For this purpose, we can first select the domain-specific classes from the list of DBpedia classes that are used to predict in specific domain. We then only take the instances containing those classes. Similarly, we can also select interest properties from the list of DBpedia properties and then take the instances associated with those specific properties. Using these two approaches significantly reduced the number of ontology instances.

For example, if we want to predict events in a domain in which there are concepts such as city and country, we should select City and Country classes and ‘capital’ property while taking the instances of selected classes and properties in DBpedia ontology.

3. Event prediction

In this section, we aim to develop a general framework for news event prediction. We call it general because the proposed framework does not consider any assumptions about common goals of prediction systems in textual environments. However, despite the generality of the framework, it is specific enough to obtain stable and reliable results, as will be discussed in the following. The proposed framework uses contextual knowledge available in ontology and the arguments extracted from online news sources and satisfies the rules that had been fed into the system to predict new events. Figure 1 shows the overall operation of the proposed framework for predicting news events.

Figure 1.

Overall architecture of the proposed framework for news event prediction.

The contextual knowledge was added to the system through a large-scale OWL ontology, which was the Persian ontology extracted from DBpedia. As mentioned in subsection 2.3.2, an approach is considered for summarising general otology into the framework; it summarises contextual knowledge according to the classes and properties of the given ontology. The summarised OWL knowledge base is stored and loaded on the main (merged) ontology of the system. Subsequently, the domain-specific rules acquired by human experts in weighted FOL formulas are transformed into OWL-encoded syntaxes in order to be considered for reasoning. While pre-processing the corpus of news collected from online sources (the working memory), events are extracted from the main textual content of each article using a deeper processing technique by semantic role labelling (SRL). They are then post-processed through the automatic generation of a semantically meaningful structure (OWL format).

All the identified, extracted and obtained knowledge is then transformed into an expedient machine-readable OWL format for the news. An inferential model is then built on these OWL triples using probabilistic logical scheme via the MLN. Afterwards, the user is able to draw inferences via the user interface. Notably, this interface receives only events from working memory that matches the ‘if’ part of the rules. Hence, by filtering the news events, additional and unnecessary processing is avoided and inference is accelerated. The outputs of inference are facts/events that can be converted into triple arguments in OWL format. After drawing inferences, the user can add and incorporate the inference outputs into the main ontology in an OWL-supported file format. The following subsections describe these steps in detail.

3.1. Representation of events extracted from news archives in OWL format

According to the recent events extracted from news archives and considering the FOL causality rules written by a human expert, the prediction system could predict future events. The events extracted from news (the working memory) was added to the system and transformed into an OWL structure for generating semantic graphs as a representation of the meaning of a text. OWL triples are inherently usable for reading, reasoning and eventually prediction by the machine. These triple arguments are effective to semantically understand a text without the need for reading it in detail. Moreover, it is possible to incorporate it into other forms of knowledge represented in the same format. Furthermore, it is easier to search, retrieve and perform logical inference on the large-scale text corpora. These were the motivations that guided us to represent the facts/events extracted from news archives in the OWL format.

In this work, events are considered predicates in news sentences that describe a situation or condition in which something establishes or changes at a certain time. Accordingly, each sentence available in news articles is stored in the OWL ontology in the form of triple arguments. To read sentences in news articles and extract a large set of (OWL) verb-based triples from each article, we used OIE. An OIE takes all types of (verbal) binary relations found in each sentence, comprising a verb relation and two arguments. The core OIE is accomplished by linguistic pre-processing including tokeniser, sentence splitter, parts of speech (POS) tagger and lemmatiser, followed by named-entity recognition (NER) and co-reference resolution components. For a more semantic understanding of texts, temporal semantic analysis and SRL were used to represent semantically rich event model.

Before any language processing beyond the lexical level can be performed, it is necessary to break the news text up into words, phrases, punctuation marks or other meaningful units called tokens. Sentence splitting assembles the tokenised text into sentences. Texts without sentence mark-ups require pre-processing to add it without disturbing the existing mark-ups. This enables further processing of the texts, especially POS tagging. The POS tagger marks up words (and other tokens) in the text and tags them as POS such as nouns, verbs, adjectives and adverbs. After tagging the part of speech of a word, different normalisation rules were applied for each part of speech to determine the lemma for each given word. The goal of the lemmatiser is to reduce inflectional and sometimes derivationally related forms of a word to the canonical form.

NER is a subtask of information extraction that seeks to label sequences of words as named entities in a text such as the names of things, people, organisations and products. When two or more expressions in the text refer to the same person or thing, the co-reference resolution is used to find all expressions that refer to the same entity. A temporal tagger for recognising and normalising temporal expressions (SUTIME [47]) was also used to annotate documents with temporal information. It is a deterministic rule-based system designed for extensibility as part of the Stanford CoreNLP [48] pipeline.

Our approach to news event prediction is based on a semantic understanding of news, one that goes beyond lexical, factual and shallow syntactic representations. Such levels of processing could be evolved from SRL, which extracts the events contained in news, the entities included in the events and the roles that each entity plays according to the events. SRL is an important step towards deep semantic processing of sentences (facts/events) in order to make sense of its meaning. This deep processing comprises detecting semantic arguments associated with a verb in a sentence and their classification into (semantic) thematic roles.

We investigate the applicability of SRL for the task of OIE, which extracts any type of relation from unrestricted text without requiring a pre-specified domain. All the verbs and thematic roles extracted from SRL-based OIE are depicted in corresponding classes and properties of the main ontology in the OWL structure.

To extract semantic relations between words in a sentence, they first need to be established at the syntactic level. For this purpose, we used a dependency-based method [49] that defines how the words in a sentence are connected to each other. Dependency grammar provides verbal relations between the words of a text, in which the main verb of the sentence is usually independent and other words are dependent on some others. Specially, we used the Persian dependency treebank [50] comprising 30,000 annotated Persian sentences and nearly 4800 distinct verb lemmas in its sentences based on dependency grammar. We then used the Malt parser [51] to generate a dependency parser model for Persian language.

After parsing each sentence of the news article using the generated dependency parser, we found out the verb clauses that it constituted. We then proceeded to identify the thematic roles corresponding to each clause. Finally, we applied a set of rules on the clause constituents to extract OWL triples.

In this model, each sentence contains a verb and several thematic roles, in which the verb clauses and their thematic roles are, respectively, represented as classes and properties of the OWL ontology. Thematic roles connected a verb with the instances of other classes. The number and type of these roles are predefined. This model bears points of comparison with the Kim’s [52] extended model for representing the events. Kim’s [52] extended model assumes that a specific set of entities is available. For example, these entities can be humans, things or abstract concepts. In this method, the events are represented by an ordered triples in notation [X_n, P, T], where entities X_n = (x₁, x₂, …, x_n) exemplify the n-adic empirical property P at time T.

Based on this approach, we propose to represent the events in each sentence as <X₁, X₂, …, X₁₇, P, T>, in which variables X₁ to X₁₁ contain action labels and variables X₁₂ to X₁₇ contain description labels used in thematic roles. Table 2 shows the list of the proposed (semantic) thematic roles corresponding to each variable presented in Kim’s extended model.

Table 2.

The proposed thematic roles for corresponding variables of the Kim’s extended model

Extended variables	Semantic roles	Description
P	Action	Verb of action
O ₁	Agent	Actor of action
O ₂	Patient	Object of action
O ₃	Theme	Subject of action
O ₄	Manner	Mode of action
O ₅	Instrument	Tools for action
O ₆	Location	Place of action
O ₇	Start_location	Where the action starts
O ₈	End_location	Where the action ends
O ₉	Start_time	Since the start of action
O ₁₀	End_time	Since the end of action
O ₁₁	Cause	Reason of action
O ₁₂	Attribute	Quality/qualification
O ₁₃	Value	Quantity/number
O ₁₄	Measurement	Unit
O ₁₅	Has-a	Part of
O ₁₆	Is-a	Kind of
O ₁₇	Instance-of	Sample of
T	Time	Time of action

Example 2. Consider the following news sentence: ‘Iran’s production of uranium increased by at least 20% due to centrifugation at Isfahan in April 2014’. The variables of the extended model in this event are represented in the form of: <P = product, O₁ = Iran, O₂ = null, O₃ = uranium, O₄ = increased, O₅ = null, O₆ = Isfahan, O₇ = null, O₈ = null, O₉ = null, O₁₀ = null, O₁₁ = centrifugation, O₁₂ = at least, O₁₃ = 20, O₁₄ = percent, O₁₅ = null, O₁₆ = null, O₁₇ = null and T = Apr 2014>.

Each of these roles in the representation model shows the name of a relation with two arguments, and this relation is stored in OWL or other standards. So, the event extracted for this sentence including 10 arguments will be as follows:

<…/product_001> <…/Type> <…/product>

<…/product_001> <…/Agent> < …/Iran>

<…/product_001> <…/Theme> <…/uranium>

<…/product_001> <…/Manner> <…/increased>

<…/product_001> <…/Location> <…/Isfahan>

<…/product_001> <…/Cause> <…/centrifugation>

<…/product_001> <…/Attribute> <…/at least>

<…/product_001> <…/Value> <…/20>

<…/product_001> <…/Measurement> <…/percent>

<…/product_001> <…/Time> <…/Apr 2014>.

Note that product_001 is an instance of the class product. The dynamic ontology can have countless instances of this type of action. The corresponding number to this verb (i.e. 001) is also considered as a sentence or news identifier. Figure 2 shows how Example 2 is represented in ontology.

Figure 2.

Graph representation of an event mapped on the ontology language OWL.

The proposed verb-based approach is effective to handle complex sentence structures and extract semantically deep events from news text, as well as to represent them into a machine-readable OWL structure. This deep structure leads to a more distinctive and elaborate semantic representation. Making sentences distinctive and elaborate means their representations will comprise a more coherent set of other things, which will make each sentence correspond to one and only one sentence. As a result, such deep semantic processing should lead to better understanding of texts.

3.2. Representation of rules

We employed the probabilistic first-order rules of MLNs to represent common-sense knowledge for making inference about imperfection and contradictory knowledge. These rules are represented in the form of OWL to be used for reasoning and eventually for prediction. Each rule consists of a number of nodes. These nodes can be added to the ‘if’ or ‘then’ part of the rules. Indeed, with the satisfaction of the ‘if’ part nodes, all of the ‘then’ part nodes are satisfied. Both the ‘if’ and ‘then’ parts of the nodes consist of one of the four types, namely, class, property, verb and thematic role. Example 3 shows how to create rules.

Example 3. Suppose we want to add a rule to the rule set that ‘If two countries negotiated on nuclear issues, then the one which is a member of P5+1 would sign a nuclear agreement’. The rule will be added to rule set as follows:

[rule i:

(?A RDF:Type negotiate)//verb

(?A Theme nuclear_issues)//thematic role

(?A Agent?B)//thematic role

(?B RDF:Type country)//class

(?A Patient?C)//thematic role

(?C RDF:Type country)//class

(?B member-of P5+1)//property

→

(?D RDF:Type sign)//verb

(?D Theme nuclear_agreement)//thematic role

(?D Agent?B)//thematic role

]

Each line of the rule i contains a node that expresses one of the four types of proposition, for example, suppose the user wants to create the node (?A Theme nuclear_issues) and add it to the ‘if’ part of the rule. First, the user should select the thematic role Theme for the type of node and ?A as a variable, and then choose nuclear_issues from general ontology as a value of the thematic role. This process can be used for adding other nodes to the given rule. After creating all the desired rules, the rule set is saved in the OWL file format to be used for reasoning and eventually for prediction.

3.3. MLN

We seek to deal with the complex and uncertain knowledge and to enrich textual representation of an event model by combining logic and probability. Statistical relational learning [53] is one of the most practical ways of uncertain inference to unify logical and probabilistic models. Especially, Markov logic [54] is considered as a unifying framework for statistical relational learning.

The MLN is a probabilistic logic that utilises Markov networks to represent first-order formulas in which one formula is worth a thousand labels. MLNs are appropriate for solving joint inference problems [34] that model complex interdependencies and propagate information from more certain decisions to resolve ambiguities in others. Additionally, they provide the principal way to leverage both direct and indirect supervision and are an easy way to acquire knowledge provided by a domain expert. These characteristics compensate for its lack of training examples.

So far, MLNs have been applied to applications such as information extraction [55], entity resolution [56], text mining [57], social recommender systems [58], opinion mining [59] and sentiment analysis [60]. However, the present work highlights the first attempt at semantically deep processing of complex and uncertain events using MLN, which has made it possible to predict news events. Example 4 shows how MLNs work for this purpose.

Example 4. Suppose we have two rules with their corresponding weightings: ‘If two countries negotiated on nuclear issues, then the probability is high (1.5) that either would sign the nuclear agreement or both would not’ and ‘If a country signs a nuclear agreement, then the chance for lifting sanction is high (1.1)’. First, we translate them into FOL with associated weightings as shown in equations (1) and (2)

1.5 \forall x \forall y : N e g o t i a t e (x, y) \Rightarrow (S i g n_A g r e e m e n t (x) \Leftrightarrow S i g n_A g r e e m e n t (y))

(1)

1.1 \forall x (S i g n_A g r e e m e n t (x) \Rightarrow L i f t_S a n c t i o n (x))

(2)

Each sentence or formula in FOL should be converted into the clausal form, as shown in equations (3)–(5), which can be then used to perform first-order resolution.

Formulas are constructed using four types of symbols – constants, variables, functions and predicates. For a more detailed overview, the reader is recommended to refer to Domingos and colleagues [35, 54]

\neg N e g o t i a t e (x, y) \lor S i g n_A g r e e m e n t (x) \lor \neg S i g n_A g r e e m e n t (y)

(3)

\neg N e g o t i a t e (x, y) \lor \neg S i g n_A g r e e m e n t (x) \lor S i g n_A g r e e m e n t (y)

(4)

\neg S i g n_A g r e e m e n t (x) \lor L i f t_S a n c t i o n (x))

(5)

We then construct an MLN template, so that each formula owns a weighting and matches one clique. Given the different sets of constants, different networks will be constructed, known as ground Markov networks. The probability of a particular truth assignment x to the variables in a ground Markov network is defined as equation (6)

P (X = x) = \frac{1}{Z} \underset{i}{Π} ϕ_{i} {(x_{{i}})}^{n_{i} (x)} = \frac{1}{Z} \exp (\sum_{i} ω_{i} n_{i} (x))

(6)

where w_i is the weighting of the formula i, n_i(x) is the number of true groundings of formula in x and $ϕ_{i} (x_{{i}}) = e^{w_{i}}$ . Figure 3 shows the graph of the ground Markov network defined by formulas in equations (3)–(5) and the constants Iran (IR) and the United States (US).

Figure 3.

Ground Markov network obtained by applying the formulas in equations (3) and (4) to the constants IR and US.

Notably, the relations Country (Iran) and Country (US) can be implicitly extracted from DBpedia ontology as follows:

<http://dbpedia.org/resource/Iran> <http://www.w3.org/rdf-syntax-ns#type> <http://dbpedia.org/ontology/Country>

<http://dbpedia.org/resource/US> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://dbpedia.org/ontology/Country>

Finally, we can compute whatever we want to predict. For example, suppose we want to know the probability that Iran and the US have negotiated, both have signed a nuclear agreement, but none have lifted the sanction. The probability value is calculated as equation (7)

P (X = x (U S, I R)) = \frac{1}{Z} \exp (\sum_{i = 1}^{3} ω_{i} n_{i} (x (U S, I R))) = \frac{1}{Z} \exp (1.1 * 4 + 1.1 * 4 + 1.5 * 0) = \frac{1}{Z} \exp (8.8)

(7)

4. Prediction evaluation

4.1. Dataset

The required data for evaluation included static knowledge (information extracted from DBpedia), dynamic ontology (events extracted from news corpora) and common-sense knowledge (rules extracted from domain expert), all of these transformed into an OWL format.

The information extracted from news texts and inference rules provided by the human expert are very important for evaluating the system. This is because the power of inference depends on both of these. The evaluation procedure is described in subsection 4.3.

4.2. Evaluation criteria

In previous works, most evaluation approaches focused only on the accuracy of the generated predictions based on human involvement. However, users are more likely to explore and provide a list of specific events. Accordingly, accuracy-based measures such as mean absolute error (MAE), mean square error (MSE) and root mean square error (RMSE) are not efficient to evaluate the prediction system. To this end, we use precision and coverage metrics to evaluate the proposed method and compare it with the other methods. However, these metrics alone are not indicative of the better performance of the news prediction system because users usually tend to have a more diverse prediction in addition to high precision and coverage. Therefore, in this article we used precision, coverage and diversity metrics to evaluate our experiments.

Precision and coverage are, respectively, defined as equations (8) and (9)

p r e c i s i o n = \frac{| {E x t r a c t e d e v e n t s} \cap {P r e d i c t e d e v e n t s} |}{| {P r e d i c t e d e v e n t s} |}

(8)

c o v e r a g e = \frac{| {E x t r a c t e d e v e n t s} \cap {P r e d i c t e d e v e n t s} |}{| {E x t r a c t e d e v e n t s} |}

(9)

Diversity refers to how the generated predictions vary with regard to each other. Inter-diversity (D_Inter) of the generated predictions for two news topics T and T^′ can be measured using the Hamming distance [61]. The Hamming distance is calculated by the sum of the absolute values of positions at which the corresponding events are different, which is defined as equation (10)

D_{I n t e r} (L) = 1 - \frac{| C_{T T'} (L) |}{| L |}

(10)

where C_TT′(L) is the number of common events in the top L places of the predicted lists of T and T^′. The higher its value, the more diverse are the events predicted for topics.

Intra-diversity (D_Intra) is defined by considering the prediction lists of the variety of events. By denoting the predicted events for a specific domain as {e₁, e₂, …, e_L}, the similarity of these events s(e_i, e_j) can be used to compute the intra-diversity as shown in equation (11). The lower its value, the more diverse is the prediction given to the user

D_{I n t r a} (L) = \frac{1}{L (L - 1)} \sum_{i \neq j} s (e_{i}, e_{j})

(11)

We define the similarity of two events e_i and e_j, s(e_i, e_j), according to Radinsky et al. [29]. We considered an event as ordered set e = <X₁, X₂, …, X₁₇, P, T>, which is described in subsection 3.1. We also used FarsNet [39], a lexical ontology for Persian language, and localised VerbNet extracted from verbs’ argument structures instead of WordNet [38] and VerbNet [40] ontologies.

4.3. Evaluation procedure

There are no benchmarks to evaluate the task we tackled in this article, and as with many other AI problems, we also encountered the lack of training data with certain labels. To this end, we collected the Persian news articles about foreign policy (specifically Iran’s nuclear programme) contained in FarsNews,³ AsrIran⁴ and Tabnak⁵ archives from June 2015 to September 2015. The news corpus was collected in three topics – nuclear deal (T₁), sanctions relief (T₂) and development of economic relations (T₃). Two human groups informed about domestic and foreign policy news were considered to generate cause-effect data and causal rules for evaluation. The first group was asked to annotate the cause and effect events extracted from 200 news articles that were randomly selected from the overall gathered data, where each pair of cause and effect consists of a fact and at least two possible effects. It is noteworthy that the cause events (E_c) annotated by this human group should be necessarily extracted from the texts of given news articles, but for the extraction of effect events (E_e) the group could additionally employ other source of news published in specific domains. The second group was also responsible for creating the causal rules in a predefined form of FOL associated with their weighting according to the given news texts and E_c as the premise of the rules. This group was not aware of the effect events (E_e) specified by the first group, so their opinion did not exactly tally with the other one.

All of the human-labelled events were manually marked with semantic tags, which include thematic roles listed in Table 2. To provide such information for the system, we initially pre-processed and parsed all documents as mentioned in subsection 3.1. There were 300 sentences (facts/events), with 1190 semantic tags for cause events and 496 semantic tags for causal rules. Figure 4 shows the distributions of training and test data for various domains of foreign policy news.

Figure 4.

Distributions of training and test data for various domains of foreign policy news dataset.

First, all cause events of training data were given to the experts for generalisation and production of the rules. Thereafter, all cause events of test data (E_c_test) and production rules were fed into the system that generated new effect events as prediction outputs. Finally, the generated predictions are automatically compared with the desired outputs (E_e_test), so the precision, coverage and diversity of generated predictions are evaluated using equations (7)–(9). The experimental results are presented in the following subsection.

4.4. Experimental results

To evaluate the proposed method, which uses logic and probability for inference, we compared its performance with baseline methods, which use only logic or only probability. The main research question we purposed to answer with these experiments is whether adding probability to the logical rule base improves its prediction of the news events. To this end, we observe the results of the generated prediction using only inference with FOL rules. We use the Jena⁶ rule engine to derive consequences from RDF models to provide OWL reasoning.

Another question that must be answered with the experiments is whether available probabilistic models are now strong enough to be used in problems domain without requiring a deeper structure proposed by MLN. To answer this question, we used the Naive Bayes (NB) algorithm as a propositional learner [54] in which the domain first requires to be propositionalised. In this article, we employed the MLN interface engine Tuffy [62], which leverages time-tested relational database management system (RDBMS) infrastructure for joins to achieve high performance at scale.

We were also interested in considering the influence of adding semantic tags and different sources of world knowledge to the performance of predictions. Notably, the work presented by Radinsky et al. [29 –31] added only 4+1 semantic tags (including agent, theme, instrument, location and time) for event representation schema and used only logical approach (CBR⁷ algorithm) to infer future events. Therefore, we will also be able to compare the performance of the proposed method with this baseline algorithm. We employed a small number of FOL formulas as domain knowledge instead of performing large-scale learning [29] over the large data corpus, with almost the same performance.

For this purpose, we consider predictors solely based on using the 4+1 semantic roles (news + 4s), on all the 17+1 semantic roles (news + 17s) and on both the semantic roles and the ontological hierarchies (news + ontology). The experimental results in precision, coverage and intra-diversity of generated predictions are shown in Tables 3 –5, respectively, for three domains T₁, T₂ and T₃.

Table 3.

Comparison of the precision of prediction for various domains of the foreign policy news dataset

Method	Topic
Method	Nuclear deal (%)	Sanctions relief (%)	Development of economic relations (%)
MLN (news + ontology)	89.74	81.96	83.47
MLN (news + 17s)	71.26	66.73	74.56
MLN (news + 4s)	68.44	60.18	63.37
FOL (news + ontology)	72.65	67.47	70.56
FOL (news + 17s)	59.38	52.24	53.45
FOL (news + 4s)	46.68	39.08	41.23
NB	57.14	48.64	54.76

MLN: Markov logic network; FOL: first-order logic; NB: Naive Bayes.

Table 4.

Comparison of the coverage of generated predictions for various domains of the foreign policy news dataset

Method	Topic
Method	Nuclear deal (%)	Sanctions relief (%)	Development of economic relations (%)
MLN (news + ontology)	92.26	87.32	88.43
MLN (news + 17s)	83.49	79.33	73.04
MLN (news + 4s)	77.09	73.45	64.48
FOL (news + ontology)	86.13	76.62	69.34
FOL (news + 17s)	71.85	68.27	65.37
FOL (news + 4s)	64.47	61.38	54.89
NB	68.26	49.17	57.19

MLN: Markov logic network; FOL: first-order logic; NB: Naive Bayes.

Table 5.

Comparison of the intra-diversity of generated predictions for various domains of the foreign policy news dataset

Method	Topic
Method	Nuclear deal (%)	Sanctions relief (%)	Development of economic relations (%)
MLN (news + ontology)	84.47	78.43	82.06
MLN (news + 17s)	76.53	71.56	73.85
MLN (news + 4s)	73.60	68.76	70.18
FOL (news + ontology)	79.46	70.39	76.78
FOL (news + 17s)	59.71	57.40	57.54
FOL (news + 4s)	42.67	36.13	38.41
NB	66.48	64.82	69.33

MLN: Markov logic network; FOL: first-order logic; NB: Naive Bayes.

As shown in the above tables, the proposed method is significantly more precise, has more coverage and is more diverse than other methods, showing the promise of this approach. It is probably that these results are due to a combination of the expressive power of logic and the probability of handling both complexity and uncertainty. The purely logical and purely probabilistic methods mostly stand when intercurrent relations should be derived, while MLN is greatly free-standing. The NB algorithm performs poorly compared with the other methods. A possible explanation for this might be the lack of appropriate training data and the nature of the algorithm. FOL works well on using both semantic roles and ontological hierarchies but poorly on ignoring ontological triples and reducing semantic roles. On the whole, out of the purely logical and purely probabilistic methods, the best-performing method is FOL, but its results fall well short of the best MLN.

Figure 5 shows the inter-diversity of the generated predictions for various domains of foreign policy T₁, T₂ and T₃. As has been shown, the results of news prediction for T₃ and T₂, respectively, have the most and the least diversity compared with other similar domains.

Figure 5.

Comparison of the inter-diversity of generated predictions for various domains of foreign policy news dataset.

Furthermore, we conducted another experiment to evaluate the matching of every semantic role extracted from the news corpus to the localised (Persian) ontology extracted from DBpedia. The purpose of this experiment is to show whether the labelled semantic roles from the news corpus was mapped precisely to a concept in the ontology. The results are summarised in Table 6.

Table 6.

Precision of matching semantic roles extracted from the news corpus to concepts in the ontology

Semantic roles	Precision (%)
Action	93
Agent	89
Patient	84
Theme	78
Manner	65
Instrument	74
Location	85
Cause	69
Attribute	46
Value	41
Measurement	46
Has-a	54
Is-a	56
Instance-of	39

We find that adding knowledge, either these semantic tags or the ontological triples, improves the performance of predictions. The highest performance is achieved when applying both refinements by MLN. Returning to the question raised at the beginning of this study, it is now possible to state that using the combination of these semantic roles is efficient to extract news events from texts in a semantically deep manner.

5. Conclusion

The purpose of this study was to develop an efficient and generalisable knowledge framework for predicting future events in textual environments. We have proposed a method to realise this framework using MLN, which simply combines probability and FOL. Within this framework, different types of knowledge, such as static (e.g. a priori and contextual) and dynamic (a posteriori) knowledge extracted from multiple, heterogeneous and uncertain sources of information can be incorporated together in standard OWL to be used for common-sense reasoning and eventually prediction. We have consequently suggested a testing methodology for evaluating news prediction algorithms. Experimental results with respect to real news datasets showed that the proposed framework achieved significant improvements in prediction compared with the baselines, which used purely logical and purely probabilistic models. Moreover, enrichment of textual event representation by a deeper semantic structure lent better predictive ability to all the methods.

The most obvious finding from this study is that our method provides a promising framework for representing general-purpose, language-independent and semantically rich event model that can combine semantic analysis with different types of knowledge resources. Another finding of the study is reasoning under uncertainty to predict news events. These findings further support the contribution to machine common-sense understanding and a deeper semantic natural language understanding. In future research, we intend to extend our framework to include parallel and distributed environments. We also plan to investigate our framework in other related tasks such as textual entailment and question answering. Besides, there is a great potential for adapting the proposed method for applications such as stock event analysis and health-care event monitoring by utilising their specialised domain ontologies.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

This paper was extracted from the dissertation prepared by Sina Dami to fulfil the requirements required for earning the degree of Doctor of philosophy which was accomplished with the support of the Department of Information and Communication Technology (ICT), Malek-Ashtar University of Technology (MUT), funded by Research and Development Centre at MUT.

Notes

References

Sarawagi

. Information extraction. Found Trends Database 2008; 1: 261–377.

Blanco

Castell

Moldovan

. Causal relation extraction. In: Proceedings of the international conference on language resources and evaluation (LREC), Marrakesh, Morocco, 26 May–1 June 2008.

Ling

Weld

. Temporal information extraction. In: Proceedings of the twenty fifth national conference on artificial intelligence, San Francisco, CA, 7–11 August 2011.

Jatowt

Kanazawa

Oyama

et al . Supporting analysis of future-related information in news archives and the web. In: Proceedings of the 9th ACM/IEEE-CS joint conference on digital libraries, Austin, TX, 14–19 June 2009, pp. 115–124. New York: ACM.

Banko

Cafarella

Soderl

et al . Open information extraction from the web. In: Proceedings of the international joint conferences on artificial intelligence (IJCAI), Hyderabad, India, 6–12 January 2007.

Krestel

Mehta

. Predicting news story importance using language features. In: Proceedings of the IEEE/WIC/ACM international conference on web intelligence and intelligent agent technology (WI-IAT’08), Sydney, NSW, Australia, 9–12 December 2008, pp. 683–689. New York: IEEE.

Gaugaz

Siehndel

Demartini

et al . Predicting the future impact of news events. In: Hanbury

Kazai

Rauber

et al . (eds) Advances in information retrieval. Berlin, Heidelberg: Springer, 2012, pp. 50–62.

Tsagkias

Weerkamp

De Rijke

. Predicting the volume of comments on online news stories. In: Proceedings of the 18th ACM conference on information and knowledge management, Hong Kong, 2–6 November 2009, pp. 1765–1768. New York: ACM.

Dou

Lian

et al . Content-based collaborative filtering for news topic recommendation. AAAI 2015, pp. 217–223, http://ihome.ust.hk/~zluab/docs/mypubs/Content-basedCFforNewsTopicRecommendation.pdf

10.

Liu

Dolan

Pedersen

. Personalized news recommendation based on click behavior. In: Proceedings of the 15th international conference on intelligent user interfaces, Hong Kong, 7–10 February 2010, pp. 31–40. New York: ACM.

11.

Minkov

Charrow

Ledlie

et al . Collaborative future event recommendation. In: Proceedings of the 19th ACM international conference on information and knowledge management, Toronto, ON, Canada, 26–30 October 2010, pp. 819–828. New York: ACM.

12.

Amodeo

Blanco

Brefeld

. Hybrid models for future event prediction. In: Proceedings of the 20th ACM international conference on information and knowledge management (CIKM), Glasgow, 24–28 October 2011, pp. 1981–1984. New York: ACM.

13.

Radinsky

Svore

Dumais

et al . Modeling and predicting behavioral dynamics on the web. In: Proceedings of the 21st international conference on World Wide Web (WWW), Lyon, 16–20 April 2012, pp. 599–608. New York: ACM.

14.

Bothos

Apostolou

Mentzas

. Using social media to predict future events with agent-based markets. IEEE Intell Syst 2010; 25(6): 50–58.

15.

Asur

Huberman

. Predicting the future with social media. In: Proceedings of the 2010 IEEE/WIC/ACM international conference on web intelligence and intelligent agent technology (WI-IAT’10), Toronto, ON, Canada, 31 August–3 September 2010.

16.

Kak

. A survey of prediction using social media, 2012, https://arxiv.org/ftp/arxiv/papers/1203/1203.1647.pdf

17.

Schumaker

Chen

. Textual analysis of stock market prediction using breaking financial news: the AZFin text system. ACM T Inform Syst 2009; 27(2): 12.

18.

Zukerman

Albrecht

Nicholson

. Predicting users’ requests on the WWW. In: Proceedings of the 7th international conference on user modeling (UM99), Banff, AB, Canada, 1999, pp. 275–284. New York: Springer-Verlag.

19.

Radinsky

Davidovich

Markovitch

. Predicting the news of tomorrow using patterns in web search queries. In: Proceedings of the IEEE/WIC/ACM international conference on web intelligence and intelligent agent technology (WI-IAT’08), Sydney, NSW, Australia, 9–12 December 2008, pp. 363–367. New York: IEEE.

20.

Yeung

Jatowt

. Studying how the past is remembered: towards computational history through large scale text mining. In: Proceedings of the ACM conference on information and knowledge management (CIKM), Glasgow, 24–28 October 2011.

21.

Halbwachs

Coser

LA.

On collective memory. Chicago, IL: University of Chicago Press, 1992.

22.

Jatowt

Yeung

C-m Au

. Extracting collective expectations about the future from large text collections. In: Proceedings of the 20th ACM international conference on Information and knowledge management, Glasgow, 24–28 October 2011, pp. 1259–1264. New York: ACM.

23.

Michel

Shen

Aiden

et al . Quantitative analysis of culture using millions of digitized books. Science 2011; 331: 176–182.

24.

Goldstone

Bates

Epstein

et al . A global model for forecasting political instability. Am J Polit Sci 2010; 54: 190–208.

25.

Spinney

. Human cycles: history as science. Nature 2012; 488: 24–26.

26.

Turchin

Historical dynamics: why states rise and fall. Princeton, NJ: Princeton University Press, 2003.

27.

Turchin

Nefedov

SA.

Secular cycles. Princeton, NJ: Princeton University Press, 2009.

28.

Turchin

. Dynamics of political instability in the United States, 1780–2010. J Peace Res 2012; 49: 577–591.

29.

Radinsky

Davidovich

Markovitch

. Learning to predict from textual data. J Artif Intell Res 2012; 45: 641–684.

30.

Radinsky

Davidovich

Markovitch

. Learning causality for news events prediction. In: Proceedings of the 21st international conference on World Wide Web, Lyon, 16–20 April 2012, pp. 909–918. New York: ACM.

31.

Radinsky

Davidovich

Markovitch

. Learning causality from textual data. In: Proceedings of learning by reading for intelligent question answering conference, Barcelona, 16–18 July 2011.

32.

Radinsky

Horvitz

. Mining the web to predict future events. In: Proceedings of the sixth ACM international conference on Web search and data mining, Rome, 4–8 February 2013, pp. 255–264. New York: ACM.

33.

Razvan

. Learning with probabilistic features for improved pipeline models. In: Proceedings of the 2008 conference on empirical methods in natural language processing, Honolulu, HI, 25–27 October 2008, pp. 670–679. Stroudsburg, PA: Association for Computational Linguistics.

34.

Poon

Domingos

. Joint inference in information extraction. In: Proceedings of the 22nd national conference on Artificial intelligence (AAAI’07), Vancouver, BC, Canada, 22–26 July 2007, pp. 913–918. New York: ACM.

35.

Richardson

Domingos

. Markov logic networks. Mach Learn 2006; 62: 107–136.

36.

Abrial

J-R

. Faultless systems: yes we can! Computer 2009; 42: 30–36.

37.

Lehmann

Isele

Jakob

et al . DBpedia – a large-scale, multilingual knowledge base extracted from Wikipedia. Semantic Web 2012; 1: 1–5.

38.

Miller

. WordNet: a lexical database for English. Commun ACM 1995; 38: 39–41.

39.

Shamsfard

Hesabi

Fadaei

et al . Semi automatic development of farsnet; the persian wordnet. In: Proceedings of 5th global WordNet conference, Mumbai, India, 31 January–4 February 2010.

40.

Kipper

Korhonen

Ryant

et al . Extending verbNet with novel verb classes. In: Proceedings of the international conference on language resources and evaluation (LREC), Genoa, 22–28 May 2006, p. 1, http://www.lrec-conf.org/proceedings/lrec2006/pdf/468_pdf.pdf

41.

Gutierrez

Dou

Fickas

et al . A hybrid ontology-based information extraction system. J Inf Sci. Epub ahead of print 26 October 2015. DOI: 10.1177/0165551515610989.

42.

Wimalasuriya

Dou

. Ontology-based information extraction: an introduction and a survey of current approaches. J Inf Sci 2010; 36(3): 306–323.

43.

Bizer

Heath

Berners-Lee

. Linked data – the story so far. Int J Semant Web Inf 2009; 5: 1–26.

44.

Horrocks

Patel-Schneider

. Reducing OWL entailment to description logic satisfiability. In: Fensel

Sycara

Mylopoulos

(eds) The semantic web-ISWC 2003. Berlin, Heidelberg: Springer, 2003, pp. 17–29.

45.

Baader

Horrocks

Sattler

. Description logics. In: Van Harmelen

Lifschitz

Porter

(eds) Handbook of knowledge representation, vol. 3. Amsterdam: Elsevier, 2008, pp. 135–179.

46.

Borgida

. On the relative expressiveness of description logics and predicate logics. Artif Int 1996; 82: 353–367.

47.

Chang

Manning

. SUTime: a library for recognizing and normalizing time expressions. In: Proceedings of the 8th international conference on language resources and evaluation (LREC), Istanbul, Turkey, 21–27 May 2012, pp. 3735–3740. European Language Resources Association (ELRA).

48.

Manning

Surdeanu

Bauer

et al . The Stanford CoreNLP natural language processing toolkit. In: Proceedings of the 52nd annual meeting of the Association for Computational Linguistics (ACL): system demonstrations, Baltimore, MD, 22–27 June 2014, pp. 55–60. The Association for Computer Linguistics.

49.

Rasooli

Tetreault

. Yara parser: a fast and accurate dependency parser, 2015, https://arxiv.org/abs/1503.06733

50.

Rasooli

Kouhestani

Moloodi

. Development of a Persian syntactic dependency treebank. In: Proceedings of the 2013 conference of the North American chapter of the Association for Computational Linguistics: human language technologies (NAACL HLT), Atlanta, GA, 9–14 June 2013.

51.

Nivre

Hall

Nilsson

. MaltParser: a data-driven parser-generator for dependency parsing. In: Proceedings of the international conference on language resources and evaluation (LREC), Genoa, 22–28 May 2006.

52.

Kim

Supervenience and mind: Selected philosophical essays. Cambridge: Cambridge University Press, 1993.

53.

Getoor

Taskar

Introduction to statistical relational learning. London: MIT Press, 2007.

54.

Domingos

Lowd

. Markov logic: an interface layer for artificial intelligence. In: Brachman

Dietterich

(eds) Synthesis lectures on artificial intelligence and machine learning, San Rafael, CA: Morgan & Claypool, 2009, pp. 1–145.

55.

Satpal

Bhadra

Sellamanickam

et al . Web information extraction using Markov logic networks. In: Proceedings of the 17th ACM SIGKDD international conference on knowledge discovery and data mining, San Diego, CA, 21–24 August 2011, pp. 1406–1414. New York: ACM.

56.

Fisher

Christen

Wang

. Active learning based entity resolution using Markov logic. In: Proceedings, Part III, 20th Pacific-Asia conference on knowledge discovery and data mining, Auckland, New Zealand, 19–22 April 2016, pp. 338–349. Springer.

57.

Natarajan

Bangera

Khot

et al . Markov logic networks for adverse drug event extraction from text. Knowl Inf Syst. Epub ahead of print 8 August 2016. DOI: 10.1007/s10115-016-0980-6.

58.

Choi

Lee

et al . Markov logic network based social relation inference for personalized social search. In: Camacho

Kim

S-W

Trawiński

(eds) New trends in computational collective intelligence. Cham: Springer, 2015, pp. 195–202.

59.

Mojica

. Fine-grained opinion extraction with Markov logic networks. In: Proceedings of the 2015 IEEE 14th international conference on machine learning and applications (ICMLA), Miami, FL, 9–11 December 2015, pp. 271–276. New York: IEEE.

60.

Yao

et al . Sentiment classification technology based on Markov logic networks. New Rev Hypermed Multimed. Epub ahead of print 4 April 2016. DOI: 10.1080/13614568.2016.1152317.

61.

Hamming

. Error detecting and error correcting codes. Bell Syst Tech J 1950; 29: 147–160.

62.

Niu

Ré

Doan

et al . Tuffy: scaling up statistical inference in Markov logic networks using an RDBMS. Proc VLDB Endow 2011; 4: 373–384.