N-ary relation extraction for simultaneous T-Box and A-Box knowledge base augmentation

In this paper, we endeavor to answer the following research questions, in descending order of specificity:

How can we populate general-purpose KBs like DBpedia and Wikidata, maximizing the use of automatic techniques, while keeping their implementation at a reasonable cost?

In this paper, we focus on Wikipedia as the source corpus and on DBpedia as the target KB. We propose to apply NLP techniques to Wikipedia text in order to harvest structured facts that can be used to automatically add novel statements to DBpedia. Our Fact Extractor is set apart from related state of the art thanks to the combination of the following contributions:

The remainder of this paper is structured as follows. We introduce a use case in Section 2, which will drive the implementation of our system. Its high-level architecture is then described in Section 3, and devises the core modules, which we detail in Section 4, 5, 6, 7, 8, and 9. A baseline system is reported in Section 10: this enables the comparative evaluation presented in Section 11, among with an assessment of the T-Box and A-Box enrichment capabilities. In Section 12, we gather a list of research and technical considerations to pave the way for future work. The state of the art is reviewed in Section 13, before our conclusions are drawn in Section 14.

Given the use case corpus, we first extract the complete set of verbs through a standard NLP pipeline: tokenization, lemmatization and POS tagging. POS information is required to identify verbs, while lemmas are needed to build the ranking. TreeTagger16

The unordered set of extracted verbs needs to undergo a further analysis, which aims at discovering the most representative verbs with respect to the corpus. As a matter of fact, lexicon (LUs) in text is typically distributed according to the Zipf’s law,17

Two measures are leveraged to generate a score for each verb lemma. We first compute the term frequency-inverse document frequency (TF-IDF) of each verb lexicalization t belonging to the set of occurring tokens T over each document d in the corpus C: this weighting measure α t , d is intended to capture the lexicographical relevance of a given t, namely how important it is with respect to other tokens in C. Next, we calculate the standard deviation value over the set A t of α t , d values for a given t: this statistical measure β t is meant to catch heterogeneously distributed verb lexicalizations. A high β t value is desired, since it indicates that t situates far from the average usage in C. Hence, we view it as an evidence of peculiarity: the higher β t is, the more variably t is used. Ultimately, for each verb lemma l belonging to the set of occurring lemmas L in C, we determine the average over B l , i.e., the set of β t values stemming from the set T l of verb lexicalizations that correspond to the given l. In this way, we can produce the final score s l and bind it to l accordingly. To clarify how the two measures are combined, we formalize the LU selection problem as follows. ∀ t ∈ T , ∀ d ∈ C let α t , d = t f id f ( t , d ) ; A t = ⋃ d ∈ C { α t , d } ; β t = st dev ( A t ) ; ∀ l ∈ L , let B l = ⋃ t ∈ T l { β t } ; s l = avg ( B l )

The ranking is publicly available in the code repository.18

The sentence selection procedure allows to harvest meaningful sentences from the input corpus, and to feed the classifier. Therefore, its outcome is two-fold: to build a representative training set and to extract relevant sentences for classification. We experimented multiple strategies as follows. They all share the same base constraint, i.e., each sentence must contain a LU lexicalization.

First, we note that all the strategies but the baseline necessitate an evident cost overhead in terms of language resources availability and engineering. Furthermore, given the soccer use case input corpus of 52 , 000 articles circa, all strategies but the syntactic one dramatically reduce the number of sentences, while the baseline performed an extraction with a 0.95 article/sentence ratio (despite some noise). Compared to the sentence splitter strategy, the syntactic one brought an increase of roughly 4x in the number of sentences, at a cost of 375x in processing time, which we deemed not worth. These numbers arise from an experiment carried out for Wikidata, with a larger corpus composed of 500 , 000 documents circa from heterogeneous Web sources, and are illustrated in Table 2.

Consequently, we decided to leverage the baseline for the sake of simplicity and for the compliance to our contribution claims. We set the interval to 5 < w < 25 , where w is the number of words. The selection of relatively concise sentences is motivated by empirical and conceptual reasons:

We still foresee further investigation of the other strategies for scaling besides the use case. Specifically, we believe that the refinement of the chunker grammar would be the most beneficial approach: POS tagging is already involved into the system architecture, thus allowing to concentrate the engineering costs on the grammar only.

We apply a one-step, bottom-up approach to let the crowd perform a full frame annotation over a set of training sentences. In frame semantics, lexical ambiguity is represented by the number of frames that a LU may trigger. For instance, vincere (to win) conveys Trofeo (trophy) and Vittoria (victory), thus having an ambiguity value of 2. The idea is to directly elicit the detection of core FEs, which are the essential items allowing to discriminate between frames. In this way, we are able to both annotate the FEs and let the correct frame emerge, thus also disambiguating the LU. The goal is achieved as follows: given a sentence s holding a LU with frame set F and set cardinality (i.e., ambiguity value) n, we solicit n annotations of s, and associate each one to the core FEs of each frame f ∈ F . We allow workers to select the None answer, and infer the correct frame based on the amount of None.

The training set is randomly sampled from the input corpus and contains 3 , 055 items. The outcome is the same amount of frame examples and 55 , 385 FE examples. The task is sent to the CrowdFlower platform.

OPEN IN VIEWER

Fig. 3.

Worker interface example.

6.2.2. Task design

We ask the crowd to (a) read the given sentence, (b) focus on the “topic” (i.e., the potential frame that disambiguates the LU) written above it, and (c) assign the correct “label” (i.e., the FE) to each “word” (i.e., unigram) or “group of words” (i.e., n-grams) from the multiple choices provided below each n-gram. Figure 3 displays the front-end interface of a sample sentence, with Fig. 4 being its English translation.

OPEN IN VIEWER

Fig. 4.

Worker interface example translated in English.

During the preparation phase of the task input data, the main challenge is to automatically provide the crowd with relevant candidate FE text chunks, while minimizing the production of noisy ones. To tackle this, we experimented with the following chunking strategies:

third-party full-stack NLP pipeline, namely TextPro [45] for Italian, by extracting nominal chunks with the ChunkPro module;23

²³

http://textpro.fbk.eu/

custom noun phrase chunker via a context-free grammar;

EL surface forms;

We surprisingly observed that the full-stack pipeline outputs a large amount of noisy chunks, besides being the slowest strategy. On the other hand, the custom chunker was the fastest one, but still too noisy to be crowdsourced. EL resulted in the best trade-off, and we adopted it for the final task.

The task parameters are as follows:

we set 3 judgments per sentence to enable the computation of an agreement based on majority vote;

the pay sums to 5$ cents per page, where one page contains 5 sentences;

we limit the task to Italian native speakers only by targeting the Italian country and setting the required language skills to Italian;

the minimum worker accuracy is set to 70 % in quiz mode (i.e., the warm-up phase where workers are only shown test questions and are recruited according to their accuracy) and relaxed to 65 % in work mode (i.e., the actual annotation phase) to avoid extra cost in terms of time and expenses to collect judgments;

on account of a personal calibration, the minimum time per page threshold is set to 30 seconds, which allows to automatically discard a contributor when triggered;

we set the maximum number of judgments per contributor to 280, in order to prevent each contributor from answering more than once on a given sentence, while avoiding to remove proficient contributors from the task.

The outcomes are resumed in Table 3.

Finally, the crowdsourced annotation results are processed and translated into a suitable format to serve as input training data for the classifier.

OPEN IN VIEWER

Table 3

Training set crowdsourcing task outcomes. Cf. Section 6.2.1 for explanations of CrowdFlower-specific terms

Sentences	3,111
Test questions	56
Trusted judgments	9,198
Untrusted judgments	972
Total cost	152.46$

We train our classifiers with the following linguistic features, in the form of bag-of-features vectors:

both classifiers: for each input word token, both the token itself (bag of terms) and the lemma (bag of lemmas). Input n-grams recognized as entities via EL earn an additional feature;

11.1. Classification performance

We assess the overall performance of the baseline and the supervised systems over a gold standard dataset. We randomly sampled 500 sentences containing at least one occurrence of our use case LU set from the input corpus. We first outsourced the annotation to the crowd as per the training set construction and the results were further manually validated twice by the authors. CrowdFlower provides a report including an agreement score for each answer, computed via majority vote weighted by worker trust: we calculated the average among the whole evaluation set, obtaining a value of 0.916.

With respect to the FEs classification task, we proceed with 2 evaluation settings, depending on how FE text chunks are treated, namely:

lenient, where the predicted ones at least partially match the expected ones;

The frame classification task does not need to undergo chunk assessment, since it copes with the whole input sentence. Therefore, the lenient and strict settings are not applicable, and we proceed with a standard evaluation. The results are reported in Table 5.

Supervised Classification Performance Breakdown.  Figures 5 and 6 respectively display the FE and frame classification confusion matrices: they are normalized such that the sum of elements in the same row is 1. Since we highlight the cells through a color scale, the normalization is needed to avoid too similar color nuances that would originate from absolute results.

FEs.  Besides regular FE labels, the classifier also assigns the LU tag to the token considered as the trigger LU. We observe that Competizione is frequently mistaken for Premio and Entità, while rarely for Tempo and Durata, or just missed. On the other hand, Tempo is mistaken for Competizione: our hypothesis is that competition mentions, such as World Cup 2014, are disambiguated as a whole entity by the linker, since a specific target Wikipedia article exists. However, it overlaps with a temporal expression, thus confusing the classifier. Both Entità and Stato have performance values of 0, since the gold examples are always classified as false positives. However, this does not seem to affect the overall performance, due to the low quantity of gold examples holding those FE labels. Agente is often mistaken for Entità, due to their equivalent semantic type, which is always a person.

OPEN IN VIEWER

Fig. 5.

Supervised FE classification normalized confusion matrix, lenient evaluation setting. The color scale corresponds to the ratio of predicted versus actual classes. Normalization means that the sum of elements in the same row must be 1.0.

OPEN IN VIEWER

Fig. 6.

Supervised frame classification normalized confusion matrix. The color scale corresponds to the ratio of predicted versus actual classes. Normalization means that the sum of elements in the same row must be 1.0.

OPEN IN VIEWER

Fig. 7.

Supervised FE classification precision and recall breakdown, lenient evaluation setting.

OPEN IN VIEWER

Fig. 8.

Supervised frame classification precision and recall breakdown.

Frames.  We note that Attività is often mistaken for Stato or not classified at all: in fact, the difference between these two frames is quite subtle with respect to their sense. The former is more generic and could also be labeled as Career: if we viewed it in a frame hierarchy, it would serve as a super-frame of the latter. The latter instead encodes the development modality of a soccer player’s career, e.g., when he remains unbound from some team due to contracting issues. Hence, we may conclude that distinguishing between these frames is a challenge even for humans.

Furthermore, frames with no FEs are classified as “O”, thus considered wrong despite the correct prediction. Vittoria is almost never mistaken for Trofeo: this is positively surprising, since the FE Competizione (frame Vittoria) is often mistaken for Premio (frame Trofeo), but those FEs do not seem to affect the frame classification. Again, such FE distinction must take into account a delicate sense nuance, which is hard for humans as well.

Figures 7 and 8 respectively plot the FE and frame classification performance, broken down to each label.

One of our main goals is to extend the target KB ontology with new properties on existing classes. We focus on the use case and argue that our approach will have a remarkable impact if we manage to identify non-existing properties. This would serve as a proof of concept which can ideally scale up to all kinds of input. In order to assess such potential impact in discovering new relations, we need to address the following question: “which extractable relations are not already mapped in DBPO or do not even exist in the raw infobox properties datasets?”. Table 6 illustrates an empirical lexicographical study gathered from the Italian Wikipedia soccer player sub-corpus (circa 52 , 000 articles). It contains occurrence frequency percentages of word stems (in descending order) that are likely to trigger domain-relevant frames, thus providing a rough overview of the extraction potential.

The corpus analysis phase (cf. Section 4) yielded a ranking of LUs evoking the frames Activity, Defeat, Match, Trophy, Status, and Victory: these frames would serve as ontology property candidates, together with their embedded FEs. DBPO already has most of the classes that are needed to represent the main entities involved in the use case: SoccerPlayer, SoccerClub, SoccerManager, SoccerLeague, SoccerTournament, SoccerClubSeason, SoccerLeagueSeason, although some of them lack an exhaustive description (cf. SoccerClubSeason31

For each of the 7 aforementioned DBPO classes, we computed the amount and frequency of ontology and raw infobox properties by querying the Italian DBpedia endpoint. Results (in ascending order of frequency) are publicly available,33

OPEN IN VIEWER

Fig. 9.

Italian DBpedia soccer property statistics.

First, we observe a lack of ontology property usage in 4 out of 7 DBPO classes, probably due to missing mappings between Wikipedia template attributes and DBPO. On the other hand, the ontology properties have a more homogenous distribution compared to the raw ones: this serves as an expected proof of concept, since the main purpose of DBPO and the ontology mappings is to merge heterogenous and multilingual Wikipedia template attributes into a unique representation. On average, most raw properties are concentrated below coverage and frequency threshold values of 0.8 and 4 respectively: this means that roughly 80 % are rarely used, and the log scale further highlights the evidence. While ontology properties are better distributed, most still do not reach a high coverage/frequency trade-off, except for SoccerPlayer, which benefits from both rich data (cf. Section 2) and mappings.34

In light of the two analyses discussed above, it is clear that our approach would result in a larger variety and finer granularity of facts than those encoded into Wikipedia infoboxes and DBPO classes. Moreover, we believe the lack of dependence on infoboxes would enable more flexibility for future generalization to sources beyond Wikipedia.

Subsequent to the use case implementation, we manually identified the following mappings from frames and FEs to DBPO properties:

Frames: (Activity, careerStation), (Award, award), (Status, playerStatus);

Our methodology enables a simultaneous T-Box and A-Box augmentation: while frames and FEs serve as T-Box properties, the extracted facts feed the A-Box part. Out of 49 , 063 input sentences, we generated a total of 213 , 479 and 216 , 451 triples (i.e., with a 4.35 and 4.41 ratio per sentence) from the supervised and the baseline classifiers respectively. 52 % and 55 % circa are considered confident, namely facts with confidence scores (cf. Section 9 and 10) above the dataset average threshold.

To assess the domain coverage gain, we can exploit two signals: (a) the amount of produced novel data with respect to pre-existing T-Box properties and (b) the overlap with already extracted assertions, regardless of their origin (i.e., whether they stem from the raw infobox or the ontology-based extractors). Given the same Italian Wikipedia dump input dating 21 January 2015, we ran both the baseline and the supervised fact extraction, as well as the DBpedia extraction framework to produce an Italian DBpedia chapter release, thus enabling the coverage comparison.

Table 7 describes the analysis of signal (a) over the 3 frames that are mapped to DBPO properties. For each property and dataset, we computed the amount of available assertions and reported the gain relative to the fact extraction datasets. Although we considered the whole Italian DBpedia KB in these calculations, we observe that it has a generally low coverage with respect to the analyzed properties, probably due to missing ontology mappings. For instance, the amount of assertions is always zero if we analyze the use case subset only, as no specific relevant mappings (e.g., Carriera_sportivo35

Table 8 shows the results for signal (b). To obtain them, we proceed as follows.

slice the use case DBpedia subset;

We estimate the overall correctness of the generated statements via an empirical evaluation over a sample of the output dataset. In this way, we are able to conduct a more comprehensive error analysis, thus isolating the performance of those components that play a key role in the extraction of facts: the frame semantics classifier, the numerical expression normalizer, and an external yet crucial element, i.e., the entity linker.

To achieve so, we randomly selected 10 instances for each frame from the supervised dataset and retrieve all the related triples. We excluded instance type triples (cf. Section 8), which are directly derived from the reified frame ones. Then, we manually assessed the validity of each triple element and assigned it to the component responsible for its generation. Finally, we checked the correctness of the whole triple.

More formally, given the evaluation set of triples E, the frame predicates set F, the non-numerical FE predicates set N ¯ , and the numerical FE predicates set N (cf. Section 5), relevant triple elements are added to the classifier C, the normalizer N, the linker L, and to the set of all facts A as follows. E ⊆ S × P × O ; P = F ∪ N ¯ ∪ N ; F ∩ N ¯ ∩ N = ∅ ; p c ∈ F ∪ N ¯ ; p n ∈ N ; O = O c ∪ O n ; O c ∩ O n = ∅ ; o c ∈ O c ; o n ∈ O n ; ∀ ( s , p , o ) ∈ E let C ← C ∪ { ( p c , o c ) } ; N ← N ∪ { ( p n , o n ) } ; L ← L ∪ { o c } ; A ← A ∪ { ( s , p , o ) } Table 9 summarizes the outcomes.

Discussion.  First, we observe that all the results but the linker are in line with our classification performance assessments detailed in Section 11.1. Accordingly, we notice that most of the errors involve the linker. More specifically, we summarize below an informal error analysis:

Unfortunately, the linker tends to assign a fairly high confidence to these matches and so does the classifier, which assumes correct linking of entities. This leads to many assertions with undeserved high scores and underlines how important EL is in our pipeline.

12.1. Lu ambiguity

We acknowledge that the number of frames per LU in our use case repository may not be exhaustive to cover the potentially higher LU ambiguity. For instance, giocare (to play) may trigger an additional frame depending on the context (as in the sentence to play as a defender); esordire (to start out) may also trigger the frame Partita (match). Nevertheless, our one-step annotation approach is agnostic to the frame repository. Consequently, we expect that the LU ambiguity would not be an issue. Of course, the more a LU is ambiguous, the more expensive becomes the crowdsourcing job (cf. Section 6.2).

12.2. Manual intervention costs

Despite its low cost, we admit that crowdsourcing does not conceptually bypass the manual effort needed to create the training set: workers are indeed human annotators. However, we argue that the price can decrease even further by virtue of an automatic communication with the CrowdFlower API. This is already accomplished in the ongoing StrepHit project (cf. Section 14), where we programmatically create jobs, post them, and pull their results. Hence, we may regard crowdsourcing as an activity that does not imply any direct manual intervention by whoever runs the pipeline, if we exclude a minor quantity of test annotations, which are essential to reject cheaters.

Even though we recognize that the use case frame repository is hand-curated, we would like to emphasize that (a) it is intended as a test bed to assess the validity of our approach, and (b) its generalization should instead maximize the reuse of available resources. This is currently implemented in StrepHit, where we fully leverage FrameNet to look up relevant frames given a set of LUs.

12.3. NLP pipeline design

On account of our initial claim on the use of a shallow NLP machinery, we motivate below the choice of stopping to the POS layer. The decision essentially emanates from (1) the sentence selection phase, where we investigated several strategies, and (2) the construction of the crowdsourcing jobs, where we concurrently (2a) maximized the simplicity to smooth the way for the laymen workers, and (2b) automatically generated the candidate annotation chunks.

Chunking is substituted by EL, as explored in Section 6.2.2;

12.4. Simultaneous T-Box and A-Box augmentation

The Fact Extractor is conceived to extract factual information from text: as such, its primary output is a set of assertions that naturally feed the target KB A-Box. The T-Box enrichment is an intrinsic consequence of the A-Box one, since the latter provides evidence of new properties for the former. In other words, we adopt a data-driven method, which implies a bottom-up direction for populating the target KB. It is the duty of the corpus analysis module (Section 4) to understand the most meaningful relations between entities from the very bottom, i.e., the corpus. After that, the system proceeds upwards and translates the classification results into A-Box statements. These are already structured to ultimately carry the properties into the top layer of the KB, i.e., the T-Box.

Table 10 presents the cumulative (i.e., all FEs and frames aggregated) statistical distribution of confidence scores as observed in the gold standard. If we dig into single scores, we notice that the classifier usually outputs very high values for “O” and LU chunks, while average scores for other FEs range from 0.821 for Competition to 0.594 for Winner, down to 0.488 for Loser. On the other hand, EL scores have a relatively high average and a standard deviation of 0.273. In other words, the EL component is prone to set rather optimistic values, which are likely to have an impact on the global score. Hence, we believe that the choice of a suitable confidence score threshold for the linker may be a way to tune the final fact score as well.

Overall, due to the high presence of “O” chunks (circa 80 % of the total), the EL and the classifier scores roughly match for each FE, and so do the final ones computed with the strategies introduced in Section 9. Assigning different weights to core and extra FEs has little impact on the global scores as well, varying their value by only 1 or 2 % in both the weighted and the harmonic means. The arithmetic and weighted means yield the most optimistic global scores, averaging at 0.83 over the output dataset, while the harmonic mean settles at 0.75.

12.6. Scaling up

Our approach has been tested on the Italian language, a specific domain, and with a small frame repository. Hence, we may consider the use case implementation as a monolingual closed-domain information extraction system. We outline below the points that need to be addressed for scaling up to multilingual open information extraction. With respect to the language, we rely on training data availability for POS tagging and lemmatization. Moreover, the LUs automatically extracted through the corpus analysis phase should be projected to a suitable frame repository. Concerning the domain, the baseline system requires a mapping between FEs and target KB ontology classes. The supervised classifier needs financial resources for the crowdsourced training set construction, on average 4.79$ cents per annotated sentence; furthermore, it necessitates an adaptation of the query to generate the gazetteer.

12.7. Crowdsourcing generalization

With the Wikidata commitment in mind (cf. Section 1), we aim at expanding our approach towards a corpus of non-Wikimedia Web sources and a broader domain. This entails the generalization of the crowdsourcing step. Overall, it has been proven that the laymen execute natural language tasks with reasonable performances [54]. Specifically, crowdsourcing frame semantics annotation has been recently shown to be feasible by [32]. Furthermore, [4] stressed the importance of eliciting non-expert annotators to avoid the high recruitment cost of linguistics experts. In [23], we further validated the results obtained by [32], and reported satisfactory accuracy as well. Finally, [11] proposed an approach to successfully scale up frame disambiguation.

In light of the above references, we argue that the requirement can be indeed satisfied: as a proof of concept, we are working in this direction with StrepHit, where we have switched to a more extensive and heterogeneous input corpus. Here, we focus on a larger set L of LUs, thus | L | × n frames, where n is the average LU ambiguity.

13.1. Information extraction

Although the borders are blurred, nowadays we can distinguish two information extraction procedures that focus on the discovery of relations holding between entities: relation extraction (RE) and open information extraction (OIE). While they both share the same purpose, their difference relies in the relations set size, either fixed or potentially infinite. In other words, the former is based on a pre-defined schema, the latter is instead schema-agnostic. It is commonly argued that the main OIE drawback is the generation of noisy data [15,57], while RE is usually more accurate, but requires expensive supervision in terms of language resources [2,55,57].

13.2. Knowledge base construction

DBpedia [36], Freebase [9] and YAGO [31] represent the most mature approaches for automatically building KBs from Wikipedia. Despite its crowdsourced nature (i.e., mostly manual), Wikidata [56] benefits from a rapidly growing community of active users, who have developed several robots for automatic imports of Wikipedia and third-party data. The Knowledge Vault [15] is an example of KB construction combining Web-scale textual corpora, as well as additional semi-structured Web data such as HTML tables. Although our system may potentially create a KB from scratch from an input corpus, we prefer to improve the quality of existing resources and integrate into them, rather than developing a standalone one.

Under a different perspective, [41] builds on [12] and illustrate a general-purpose methodology to translate FrameNet into a fully compliant Linked Open Data KB via the Semion tool [42]. The scope of such work diverges from ours, since we do not target a complete conversion of the frame repository we leverage. On the other hand, we share some transformation patterns in the dataset generation step (cf. Section 8), namely we both link FEs to their frame by means of RDF predicates.

Likewise, FrameBase [51,52] is a data integration effort, proposing a single model based on frame semantics to assemble heterogenous KB schemata. This would overcome the knowledge soup issue [25], i.e., the blend of disparate ways in which structured datasets are published. Similarly to us, it utilizes Neo-Davidsonian representations to encode n-ary relations in RDF. Further options are reviewed but discarded by the authors, including singleton properties [40] and schema.org roles.37

OIE output can indeed be considered structured data compared to free text, but it still lacks of a disambiguation facility: extracted facts generally do not employ unique identifiers (i.e., URIs), thus suffering from intrinsic natural language polysemy (e.g., Jaguar may correspond to the animal or a known car brand).

To tackle the issue, [16] propose a framework that clusters OIE facts and maps them to elements of a target KB. Similarly to us, they leverage EL techniques for disambiguation and choose DBpedia as the target KB. Nevertheless, the authors focus on A-Box population, while we also cater for the T-Box part. Moreover, OIE systems are used as a black boxes, in contrast to our full implementation of the extraction pipeline. Finally, relations are still binary, instead of our n-ary ones.

The main intuition behind Legalo [47,49] resides in the exploitation of hyperlinks, serving as pragmatic traces of relations between entities, which are finally induced via NLP. The first version [47] focuses on Wikipedia articles, like we do. In addition, it leverages page links that are manually curated by editors, while we consume EL output. Ultimately, its property matcher module can be leveraged for KB enrichment purposes. Most recently, a new release [49] expands the approach by (a) taking into account hyperlinks from EL tools, and (b) handling generic free-text input. On account of such features, both Legalo and the Fact Extractor are proceeding towards closely related directions. This paves the way to a novel paradigm called open knowledge extraction by the authors, which is naturally bound to the open information semantification one introduced in [16]. The only difference again relies on the binary nature of Legalo’s extracted relations, which are generated upon FRED [26,48].

FRED is a machine reader that harnesses several NLP techniques to produce RDF graphs out of free text. It is conceived as a domain-independent middleware enabling the implementation of specific applications. As such, its scope diverges from ours: we instead deliver datasets that are directly integrated into a target KB. In a fashion similar to our work, it encodes knowledge based on frame semantics and employs EL to mint unambiguous URIs for entities and properties. Furthermore, it relies on the same design pattern for expressing n-ary relations in RDF [30]. As opposed to us, it also encodes NLP tools output via standard formats, i.e., Earmark [44] and NIF [28]. Additionally, it uses a different natural language representation (i.e., discourse representation structures), which requires a deeper layer of NLP technology, namely syntactic parsing, while we stop to shallow processing via POS tagging.

13.4. Semantic role labeling

In broad terms, the semantic role labeling (SRL) NLP task targets the identification of arguments attached to a given predicate in natural language utterances. From a frame semantics perspective, such activity translates into the assignment of FEs. This applies to efforts such as [34], and tools like MATE [8], while we perform full frame classification. On the other hand, systems like SEMAFOR [13,35] also serve the frame disambiguation part, uniformly to our method. Hence, SEMAFOR could be regarded as a baseline system. Nonetheless, it was not possible to actually perform a comparative evaluation of our use case in Italian, since the parser exclusively supports the English language.

All the work mentioned above (and SRL in general) builds upon preceding layers of NLP machinery, i.e., POS-tagging and syntactic parsing: the importance of the latter is especially stressed in [50], thus being in strong contrast to our approach, where we propose a full bypass of the expensive syntactic step.