Special issue on Semantic Deep Learning

1. Introduction

Semantic Web technologies and deep learning share the goal of creating intelligent artifacts that emulate human capacities such as reasoning, validating, and predicting. Both fields have been impacting data and knowledge analysis considerably as well as their associated abstract representations. The term deep learning is used to refer to deep neural network algorithms that learn data representations by means of transformations with multiple processing layers. Today, such architectures are well studied in the field of Natural Language Processing (NLP), where they have been successfully applied to numerous research challenges. These include low-level tasks, such as part-of-speech [51] and morphological tagging [17], as well as higher-level linguistic problems, such as language modeling [19,43,48,53,61], named entity recognition [38], machine translation [7,37], or direct speech to speech translation [34]. Semantic Web technologies and knowledge representation boost the re-use and sharing of knowledge in a structured and machine-readable fashion. Semantic resources such as WikiData [66], Yago [60], BabelNet [45] or DBpedia [5], as well as knowledge base construction and completion methods [9,10], have been successfully applied to improve systems addressing semantically intensive tasks (e.g. Question Answering as in [32]).

There are notable examples showcasing the influence of neural approaches to knowledge acquisition and representation learning on the broad area of Semantic Web technologies. These include, among others, ontology learning [40,49,65], learning structured query languages from natural language [69], ontology alignment [20,28,35,52], ontology annotation [15,58], joined relational and multi-modal knowledge representations [62], and relation prediction [1,59]. Ontologies, on the other hand, have been repeatedly utilized as background knowledge for machine learning tasks. As an example, there is a myriad of hybrid approaches for learning linguistic representations by jointly incorporating corpus-based evidence and semantic resources [13,25,27,33,50]. This interplay between structured knowledge and corpus-based approaches has given way to knowledge graph embeddings, which in turn have proven useful for tasks such as hypernym discovery [21], collocation discovery and classification [22], word sense disambiguation [12,54], joined relational and multi-modal knowledge representations [62] and many others.

In this context, this special issue aims to provide a playground for exploring the interaction between neural NLP and representation learning, on the one hand, and symbolic representation of knowledge and data-driven approaches to pattern recognition, on the other. Specifically, we invited submissions illustrating how Semantic Web resources and technologies benefit from interacting with neural networks. At the same time, we also encouraged submissions showing how knowledge representation would assist in neural NLP tasks, and how knowledge representation systems can build on top of deep learning. The timeliness of this special issue becomes apparent, also, in the potential of symbolic representations of knowledge in the form of ontologies, knowledge graphs, and rules to contribute to the long standing goal of explainable and interpretable Artificial Intelligence [39], for example, for “keep-a-human-in-the-loop” approaches [31] or directly for reasoning about neural network decisions [14].

This special issue builds on and complements a series of workshops dedicated to Semantic Deep Learning (SemDeep), co-organized by the editors of this issue. The first workshop, co-chaired by Dagmar Gromann, Thierry Declerck and Georg Heigold, took place as a satellite event to the 14th Extended Semantic Web Conference (ESWC 2017). Based on the success of this first edition of SemDeep, a new edition was submitted to the 12th International Conference on Computational Semantics (IWCS 2017), where SemDeep-2 could reach the computational semantics community. SemDeep-3, now co-chaired by Dagmar Gromann, Thierry Declerck and Luis Espinosa Anke, was co-located with the 27th International Conference on Computational Linguistics (COLING 2018). The fourth edition of SemDeep came back to the Semantic Web community and was co-located with the 17th International Semantic Web Conference (ISWC 2018). SemDeep-5 is a workshop of the 28th International Joint Conference on Artificial Intelligence (IJCAI 2019). For this last workshop, the Organizing Committee was glad to welcome José Camacho Collados and Mohammad Taher Pilehvar, as this new edition of the successful SemDeep workshop series was augmented by a challenge on evaluating contextualized word representations called Word-in-Context (WiC).

2. Recent Semantic Deep Learning approaches

Neural-symbolic approaches (see e.g. [6,8,30] for an overview) represent a relatively young field of research, having only attracted considerable attention within the last few years. The SemDeep series in general, and this special issue in particular, have offered a forum where such methods, from a proof-of-concept stage to a more advanced and robust stage of development, could be presented and discussed.

Specifically, SemDeep has seen contributions on the explicit modeling of lexical and semantic relations stemming from joint neural-symbolic methods [23,44,55]. Additionally, well-defined NLP tasks have also been the focus of several SemDeep papers over the years, covering event detection [11], part-of-speech tagging [67], co-reference resolution [63], sentiment analysis [47], named entity recognition [41] or question answering [32]. Interestingly, another area that has been prominently covered in SemDeep is (formal) knowledge representation, such as the tasks of link prediction in generic knowledge bases as well as domain-specific use cases [3,4,71]. Fewer works focused on more technical aspects of a knowledge-enhanced deep learning pipeline, for example, exploring disjointness in loss functions for classification tasks [57], end-to-end memory networks [36], image-based neural user profiling [68] or Siamese Long Short Term Memory (LSTM) networks [29].

The topic that has attracted most interest in the SemDeep workshop series has been representation learning, and a plethora of submissions were accepted for publication where vector representation of linguistic items, as well as meta-embeddings, were discussed. The concrete topics covered included word and document embeddings [56,70], knowledge graph embeddings [26], joint knowledge graph and text embeddings [16,42], multi-modal approaches [64,68], leveraging external information such as lexical resources [46], embeddings for low resource languages like Igbo [24], and learning structured knowledge [2,18].

3. Overview of this special issue

The paper Deep learning for noise-tolerant RDFS reasoning by Bassem Makni and James Hendler presents a noise-tolerant RDFS reasoning approach building on neural machine translation. To this end, they present an embedding approach where RDF graphs are layered and encoded in 3D adjacency matrices where each layer layout forms a graph word. Both input graph and its entailments are represented as a sequence of graph words and RDFS inferences can then be formulated as a machine translation of these graph word sequences. As such the approach seeks to bridge the neural-symbolic graph, adapting the idea of knowledge graph embeddings to RDF graphs, the differences of which are analyzed in detail. Evaluations demonstrate the ability of the approach to learn RDFS rules from a synthetic dataset as well as DBpedia subset and a noise-tolerance not observed in rule-based reasoners.

In Semantic Referee: A Neural-Symbolic Framework for Enhancing Geospatial Semantic Segmentation Marjan Alirezaie, Martin Längkvist, Michael Sioutis, and Amy Loutfi propose a neural-symbolic framework in which an ontological reasoner characterizes output errors of a deep learning framework, providing corresponding feedback to improve its performance. In contrast to approaches that seek to integrate neural and symbolic aspects, this approach focuses on the interaction between these two. The reasoner functions as supervisor, a so-called semantic referee, in the training process of a variation of a Convolutional Autoencoder utilized to perform semantic segmentation of satellite images. In order to feed concepts inferred by the reasoner for misclassified regions back to the neural network, such concepts are encoded as image channels and concatenated with the original RGB channels. Additional information from the reasoner in the proposed approach relates to shadow estimation, elevation estimation, and inconsistencies with respect to the ontology constraints. On two real-world datasets and the OntoCity ontology the approach could demonstrate its capacity to reduce classification errors.

The paper Vecsigrafo: Corpus-based Word-Concept Embeddings – Bridging the Statistic-Symbolic Representational Gap in Natural Language Processing by José Manuel Gómez-Pérez and Ronald Denaux proposes to jointly learn word and concept embeddings from large semantically annotated corpora. Words are tokenized and disambiguated, where three disambiguation techniques are tested, and associated with concepts from an existing knowledge graph. In an extensive evaluation comparing to word and knowledge graph embeddings on the tasks of word similarity, word prediction, and relation prediction, the merits of the proposed approach are demonstrated. It could show that the joint learning of word and concept embeddings improves the quality over individual word and knowledge graph embeddings and different aspects of such joint vector spaces are discussed in detail in the paper. Furthermore, an extensive ablation study provides interesting insights into variants of Vecsigrafo, such as lemmatization having positive effects on the resulting joint embeddings, filtering improving the coverage of concepts, and disambiguation strategies only marginally impacting lexical embeddings.

In Studying the Impact of the Full-Network Embedding on Multimodal Pipelines by Armand Vilalta, Dario Garcia-Gasulla, Ferran Parés, Eduard Ayguadé, Jesus Labarta, E Ulises Moya-Sánchez, and Ulises Cortés, a Full-Network Embedding architecture is evaluated on the task of image annotation and retrieval. This architecture takes an image and its corresponding caption as input and produces a single vector representation that combines the output of a Convolutional Neural Network (CNN) applied to the image with the output of a Gated Recurrent Unit (GRU) applied to the caption. The authors propose, in this empirical study, to determine the fitness of such model as opposed to the one-layer image embeddings typically used in the literature. They report experimental results in three publicly available datasets, namely Flickr8K, Flickr30K and MSCOCO, and discuss different settings, involving hyperparemeter configurations, quality of training data and type of source CNN models.

The paper Hate Speech Detection: A Solved Problem? The Challenging Case of Long Tail on Twitter by Ziqi Zhang and Lei Luo first describes a data analysis to quantify and qualify the linguistic characteristics of hate speech in the social media. As a result, the authors show that it is much harder to detect hateful content than non-hate speech in social media, as hateful speech (in Twitter) lacks of unique, discriminative linguistic features. In a second part, the authors propose neural network structures for identifying specific types of hate speech, with a focus on two neural models. One is simulating a skip-gram like feature extraction based on modified a CNN, while the other extracts orderly information between features using a GRU. An evaluation on an English Twitter datasets shows that the described approach can outperform state-of-the-art methods by up to five percentage points.

In A Convolutional Neural Network-based Model for Knowledge Base Completion and Its Application to Search Personalization by Dai Quoc Nguyen, Dat Quoc Nguyen, Tu Dinh Nguyen, and Dinh Phung, an embedding model of entities and relationships for knowledge base completion is introduced. This model, named ConvKB, generalizes transitional characteristics in transition-based embedding models. ConvKB has been evaluated on two benchmark datasets, WN18RR and FB15k-237, and better link prediction performance than state-of-the-art embedding models can be reported. ConvKB has been additionally evaluated for triple classification on two benchmark datasets, WN11 and FB13, in order to check if a given triple is valid or not. Here ConvKB also gets better results than state-of-the-art models. Finally, the authors describe the adaptation of ConvKB for search personalization. This application of ConvKB has been positively verified on the query logs of a commercial web search engine.

4. Conclusion and future directions

Contributions to this special issue have focused on utilizing deep learning in connection with reasoning – either making the network itself a reasoner or enabling interaction between deep learning and a reasoner – multi-modal embeddings, feature extraction from natural language, and knowledge base completion. While this enumeration already hints at the large variety of central approaches to Semantic Deep Learning, further advances are needed, especially in the area of deep reasoning and inferences.

Combining Semantic Web technologies and deep learning holds the potential to crucially contribute to the recent hype of Explainable Artificial Intelligence (XAI). This might, for instance, take the form of injecting knowledge into training procedures to estimate changes of behaviors depending on utilized knowledge. Another important future direction is further systematic investigations into multi-modal approaches connecting linguistic, visual, and sensory inputs.