Abstract
Human behavior is impacted by emotion, mood, personality, needs and subjective well-being. Emotion and mood are human affective states while personality, needs and subjective well-being are influences on those affective states. Ontologies are a method of representing real-world knowledge, such as human affective states and their influences, in a format that a computer can process. They allow researchers to build systems that harness affective states. By unifying terms and meanings, ontologies enable these systems to communicate and share knowledge with each other. In this paper, we survey existing ontologies on affective states and their influences. We also provide the psychological background of affective states, their influences and representational models. The paper discusses a total of 20 ontologies on emotion, one ontology on mood, one ontology on needs, and 11 general purpose ontologies and lexicons. Based on the analysis of existing ontologies, we summarize and discuss the current state of the art in the field.
Introduction
Ontologies have become more and more popular in fields such as web technologies or data integration and extraction [68]. An ontology can be seen as a catalog that shows entities in a specific field and the relationships between them. It represents structural knowledge for any domain and defines a common vocabulary to be shared. In addition, it defines data and data structures to be used in applications in the same field [57]. An ontology is defined as “an explicit specification of a conceptualization” [35]. An ontology consists of classes, properties and individuals that define a particular domain [54].
Classes are the focal point of ontologies, and they describe the concepts in a domain. A class represents a group of different individuals that share similar characteristics. An instance of a class is called an individual. Object properties describe the semantic relationship between individuals [57]. People and systems communicate with each other from different backgrounds and contexts, using varying words and concepts [83]. A well-designed ontology provides standardized definitions and vocabularies in a particular domain, allowing a flow of communication [89]. Figure 1 shows a representation of an ontology and its components. For instance, person is a class, Jon is an individual, and studiesIn is an object property.
Representation of ontology components. Oval shapes represent classes, rounded rectangles represent individuals, and arrows indicate object properties.
Ontologies are created using a machine-processable language such as the Web Ontology Language (OWL), an international standard for the design and exchange of ontologies. The Web Ontology Language uses a set of classes, sub-classes and properties which are organized into a hierarchical structure by property axioms [39]. New ontologies may be developed from the foundation of pre-existing ones and potentially be designed with future re-use in mind. Ontologies can be designed at the top level (a general ontology) and then customized according to the domain or application. They can also be designed for a particular application or system, in which case they are called application ontologies. Ontologies can be re-used as a whole or partially, depending on the project needs [10,37]. By not having to create an entire ontology from scratch, time is saved, and the quality and maintainability of the new ontology is improved. Moreover, by reusing existing work, knowledge can be mapped from one domain to the domain of another ontology [27].
Not only does using an ontology allow for human affective states and their influences to be represented in an understandable computer format, it also improves the understanding and communication between people. The structure of the ontology reveals the definition of human affective states, influences and the relationships between them. It enables the sharing of human knowledge in a digital format.
Human behavior is formed by affective states and their influences. While the human interpretation of the relationship between affective states and influences is far from perfect, it is superior to the interpretation by a computer. Therefore, representing human affective states and their influences in a semantic way enables the communication between humans and systems. Moreover, it inspires the development of applications that automatically detect and predict behaviors and meanings. Ontologies provide a unified vocabulary for each concept in a domain, so that the interpretation of messages shared between computer applications is universal.
For example, an ontology that defines emotion, causes, and events to predict student emotion in an e-learning session can be used to predict the emotions of students with regard to answering test questions [25]. Another example is an avatar that is capable of showing appropriate facial expressions and gestures [32]. The expressions and gestures were selected based on an ontology that represents emotions associated with facial expressions and gestures.
The left side shows ways in which humans express their emotions (Domain). The context impacts the expressed emotions. On the right side, psychological models of emotion are shown (Representation Model).
In [18] a survey was carried out about ontologies for human behavior recognition. The emphasis of the survey is on context ontologies to track human activities. It presents general and domain-specific ontologies. In our paper, however, we aim to give an overview of existing ontologies on human affective states and their influences.
The remainder of this paper is organized as follows. In Section 2, we introduce the psychological theories used to build the existing ontologies for human affective states and their influences. Section 3 describes the lexicons used in the existing ontologies. Section 4 surveys current ontologies for human affective states and their influences as well as other related ontologies. Our conclusions and an outlook are provided in Section 5.
We argue that psychological theories represent the primary point of ontology design in the domain of human affective states and their influences. These theories form the basis for the existing ontologies that are discussed in Section 4. In Section 2.1, affective states (emotion and mood) are presented. The influences, which are personality, subjective well-being and needs are introduced in Section 2.2. Finally, Section 2.3 describes the relationship between affective states and their influences.
Affective states
In the
Another discrete emotion classification was proposed by Douglas-Cowie et al., who listed 48 emotion categories and arranged them into 10 groups. They include negative forceful, negative/positive thoughts, caring, positive lively, re-active, agitation, negative not in control, negative passive and positive quiet [21].
Plutchik grouped eight basic emotions in a wheel, placing similar emotions together and opposing ones at 180 degrees apart. The model is called Plutchik’s wheel of emotions. The contrasting pairs consist of joy versus sadness; anger versus fear; acceptance versus disgust; and surprise versus expectancy. The model also includes advanced emotions made up of combined basic ones. In addition, each emotion in the model represents a basic level of intensity [62].
Furthermore, Drummond used a vocabulary of 10 emotions: happiness, caring, depression, inadequateness, fear, confusion, hurt, anger, loneliness, and remorse. They were divided into the three categories: strong, medium, and light. For example, the emotion of happiness consists of being thrilled in the strong category, cheerful in the medium category, and cool in the light category.1
In the
A graphical representation of the Circumplex Model of affect. The horizontal axis represents the valence dimension, and the vertical axis represents arousal.
Mehrabian created the Pleasure–Arousal–Dominance (PAD) model of emotional states where dominance was added as a third dimension. It is the feeling of being in control of a situation versus the feeling of being controlled [53]. Osgood et al. use the names evaluation, activity and potency [60]. Cowie et al. use evaluation, activation and power [16]. A fourth dimension named unpredictability was added by Fontaine [28]. It denotes a person’s reaction to a stimulus based on their familiarity with the situation. Other choices of dimensions can be seen for example in the model of Watson and Tellegen who proposed the dimensions of negative affect (NA) and positive affect (PA) [84], or in the model of Feidakis et al. who use intensity, frequency and duration as emotion dimensions [26].
The basis of
The OCC (Ortony, Clore, and Collins) appraisal model reasons about agents, beliefs, objects and events. This model is popular in computer science systems that draw conclusions from emotions [59]. The OCC model defines a finite set that allows for the characterization of emotions. Moreover, it delivers a semi-formal descriptive language of emotion types. The model classifies 22 emotions into three main categories: consequences of events (e.g., joy and pity), actions of agents (e.g., pride and reproach), and aspects of objects (e.g., love and hate). These three main categories are further classified into subgroups. For instance, if the evoked emotion differs whether the consequence of an event is focused on the person or on others.
As the name implies, the Big Five theory represents personality in five dimensions. An outgoing, energetic person is described by high Extroversion. A friendly and cooperative person is described by the Agreeableness trait. Conscientiousness means that someone is responsible, dependable and organized. A sensitive and nervous person has Neurotic traits, and a social, intellectual person has a large value in the Openness dimension [52].
In the MBTI, each personality fits into only one of 16 types. These types are based on four features of personality, each one combined with its opposite: Extroversion (E) vs Introversion (I), Sensing (S) vs Intuition (N), Thinking (T) vs Feeling (F) and Judgment (J) vs Perception (P) [86]. Because there are two features within each of the four dimensions, there are 16 possible combinations. It is noteworthy that although the MBTI is a very widespread test of personality, many psychologists do not support it and claim that no significant conclusions can be drawn from it. There is no evidence that every individual can be described with its 16 categories.
In Human Motivation Theory, Abraham Maslow presents a pyramid of five need categories arranged in hierarchical levels based on their importance to human beings. The five categories by decreasing importance are survival, security and safety, social, self-esteem and self-actualization. This model has been updated to adapt two new dimensions under the self-actualization category: cognition and aesthetic needs. Also, the theory explored the self-transcendent needs as the need to help others as a further category on the top of the pyramid [48].
In the Human Scale Development Model proposed by Max-Neef, the fundamental need categories for individuals and communities are formulated in a universal and interactional structure [50]. The model distinguishes between universal needs and the satisfiers, or strategies to meet these needs. The needs are finite and constant across all human cultures, while the satisfiers are changeable over time and differ between cultures. The model defines the needs and satisfiers in a matrix with two dimensions; the need dimension in axiological categories consist of: subsistence, protection, affection, understanding, participation, idleness, identity, creation and freedom. The satisfiers in existential categories are represented in the form of being, having, doing and interacting.
The relationships between the affective states and their influences
Emotion and mood can impact and influence each other. Moods influence which emotions will be experienced and repeated experience of emotions contributes to mood. For example, a negative mood can be triggered when a person interacts with an object or situation that elicits frustration [12]. An individual’s Big Five personality traits have an impact on their emotions. People can show different emotional responses to the same situation, and in some cases, personality is responsible for this difference. As an example, when a person with an extroverted personality is offered help by a stranger, the person may be happy about the help. On the other hand, if the person had an introverted personality, they might react with fear instead [23]. Subjective well-being can also be influenced by personality. Extroversion is the most significant predictor of positive affect, while Neuroticism is the most significant predictor of negative affect and life satisfaction [33].
An interesting point is that subjective well-being influences a person’s mood and emotion. When people make a positive judgment about their life, they will experience good emotions and moods. When a person experiences bad emotions or moods, then this is because they feels unhappy about their life expectations [19]. Feelings and emotions indicate the state of satisfaction of a person’s needs [66].
Emotion related lexicons and language
This section describes the emotion dictionaries that are used later in Section 4.2. Emotion dictionaries classify words into emotional dimensions, emotional categories, or both. In addition, they group emotion words into sets of synonyms.
EmotionML syntax in an emotional text with annotations encoded in XML.
Since the data is annotated in a standard way, the interpretation of the message between systems is the same. EmotionML uses Ekman’s discrete basic emotions and the PAD dimensional model to represent emotions and their features. The language can be applied in different contexts, such as data annotation and emotion recognition. The annotation can be applied to text, static images, speech recordings and video. Figure 5 demonstrates a case where an emotion is recognized from face and voice.
EmotionML syntax for an emotion that was detected from face and voice.
To include concepts of affect,
In the field of the Semantic Web, there are many lexicons available that represent data with different formats. For example, the amount of parts of speech can differ between lexicons. Moreover, it is difficult to link them with existing ontologies. The
This section presents the existing affective state ontologies and their influences. In Section 4.1, we discuss general purpose ontologies that were re-used by more specific emotion ontologies. Re-Use can be a starting point for the creation of a new ontology and it can increase domain knowledge [57]. In Section 4.2, existing emotion ontologies are presented. There exist more ontologies that target emotion than there are ontologies for mood and other influences. In Section 4.3 then presents a mood ontology and Section 4.4 introduces a need ontology.
Re-used ontologies
This section introduces general ontologies that are being re-used for the creation of the emotion ontologies that are discussed in Section 4.2.
Example from FOAF Ontology.
The
Daily human communication carries many emotions [3] which can be expressed through text, facial expression, voice and body language. Emotion can also be influenced by the contextual environment. A person expresses the same emotion in different situations (contexts), but with different intensity. Emotion ontologies can also describe general concepts. We categorize existing emotion ontologies into the five domains as they are shown in Fig. 2.
Text domain
A lot of work has been put into building ontologies that analyze and detect emotion from text. People express their emotions with words in formal and informal ways. Text in social media has different characteristics: users often use slang terms and abbreviations. Additionally, users of textual media may express their emotions via emoticons.
Summary of emotion ontologies in the text domain. In addition to the reference, the names of the ontologies are listed if available
Summary of emotion ontologies in the text domain. In addition to the reference, the names of the ontologies are listed if available
Many ontologies were built to analyze text in social media. Some of the ontologies in the text domain were created for a particular purpose, focusing on international languages like English, Chinese, Japanese, French and Italian. Table 1 summarizes emotion ontologies in the text domain. The table contains columns displaying the ontology name or prefix, the goal, the emotion model that is used, other ontologies that were re-used, and the lexicons that provided the terminology. The used emotion models can be discrete, dimensional, or based on the OCC model. It should be noted that some ontologies were built from scratch and are not based on existing ontologies or lexicons. This leads to some cells being empty in Table 1.
A system was built to analyze the unstructured informal text inside posts about electronic products to understand online consumer behavior in the market [71]. The aim of the system was text-mining in social media. In order to be able to analyze consumer behavior on social media, the Emotions Ontology was created. One of the main classes in the ontology is Sentiment, with two subclasses Happiness and sadness. For example, under happiness there exist the subclasses enjoy, fun, eager and smiling, while under sadness, there are dislike, disappointed, bad and worst. Additionally, the ontology contains classes about products such as computer, and household. The system is comprised of four modules: the ontology management module, which contains the created ontology, a user query processing module, an information foundation module, and a query analysis engine module. The ontology models the product with the associated emotions based on the social media posts. Then a user can utilize the system to make a query about the product with a specific emotion.
The Emotive Ontology [80] was built to detect and analyze emotion in informal text from social media as well. The approach detects a range of eight high-level emotions: anger, confusion, disgust, fear, happiness, sadness, shame and surprise. Each emotion in the ontology is named according to prior work (Izard, Ekman, Plutchik, Drummond), and commonly encountered emotions within Twitter messages are added. The Emotive Ontology is also capable of expressing the intensity of emotions. During the creation of the Emotive Ontology, many dictionaries and word dataset were looked at, such as WordNet, Dictionary.com, Thesaurus.com, the Oxford English online dictionary and the Merriam-Webster online dictionary. Since its goal is to detect emotion from informal slang text, websites that contain slang expressions were also examined. Examples are the Leicestershire Slang Page, the Dictionary of Slang, and the Online Slang Dictionary. To make sure that an emotion set that is as large as possible is covered, existing emotional lexicons were integrated into the Emotive Ontology. Natural Language Processing (NLP) and part of speech tagging were used as pre-processing steps for emotion detection. The ontology was tested and evaluated on a dataset taken from Twitter.
An ontology that helps to give students appropriate feedback in e-learning sessions was proposed by Arguedas et al. [1] The ontology is divided into the two main classes Emotion Awareness and Affective Feedback. In emotion awareness, the emotion is analyzed, and in affective feedback, the appropriate feedback that a teacher would give to a student is determined. The emotion awareness class includes the different types of emotions in a categorical model, moods (bored, concentrated, motivated, unsafe) and behaviors that students experience in e-learning environments. The emotion is detected during collaborative virtual learning processes, including textual conversations, debates and wikis.
In [49] the Ontology of Emotions and Feelings that automatically annotates emotion in texts and determines their intensity is introduced. This French ontology classifies 950 words (600 are verbs and 350 are nouns) into 38 semantic classes according to their meanings. Words in the lexicon are emotionally labeled as positive, negative and neutral. NaviTexte, a software designed for text navigation, was used to apply the ontology. It understands and applies knowledge to a specific text [15]. The goal of the ontology is to automatically annotate emotions in texts and to automatically navigate through the text.
An ontology of emotion objects is introduced in [64]. Emotion objects are collected from a large, Japanese blog corpus. An emotive expression lexicon for Japanese language is used to distinguish emotion words. The ontology is created using an EmotionML annotation scheme, that was modified to meet the needs of the Japanese language. The ontology classes represent emotion according to Nakamura’s classification which is “a collection of over two thousand expressions describing emotional states collected manually from a wide range of literature” [63]. The emotion in the ontology is represented in a dimensional model. The ontology also contains classes for number of characters, part of speech, and semantic categories. In the latter class, emotion objects are categorized into groups such as human activities and abstract objects.
To define emotion words and their intensity in Japanese, a Japanese emotion ontology was proposed [44]. Emotion words were taken from websites such as Twitter. The intensity calculation is based on how many times an emotion word appears in a document. The words are categorized into ten emotions: joy, anger, sadness, fear, shame, like, disgust, exciting, comforted and surprise. The authors adapt their ontology into other emotion classifications which are positive, negative and neutral. The Pleasure–Arousal–Dominance theory is adapted as well. The authors used OWL and EmotionML to describe the ontology. One of the proposed applications for the ontology is a character generator. The system can receive voice inputs. The audio is then translated into text and analyzed by the emotion ontology. The output is a character with facial animations.
To analyze Chinese text, a Chinese emotion ontology was created [87]. It was semi-automatically created using HowNet. The ontology contains 113 emotion categories and was created by first extracting affective events from the dictionary. Then, emotions are manually assigned to the semantic role of the events, producing the Emotion Prediction Hierarchy. Finally, the Emotion Prediction Hierarchy is transformed into the emotion ontology. This step involves assigning verbs extracted from the dictionary to the Emotion Prediction Hierarchy.
For advanced emotion analysis, the Onyx Ontology was presented [72]. It offers a comprehensive set of tools for any kind of emotion analysis. Onyx re-uses the Provenance Ontology and the LEMON model. In the Provenance Ontology, the activity is emotion analysis, which means turning plain data into semantic emotion information. The Onyx ontology thus has a class called Emotion Analysis that is responsible for representing the information source, e.g., a website, the algorithm, used as well as the emotion model. The Emotion Set class contains information about a group of emotions found in a text, including the person expressing the emotion, the domain, information about the original text and the sentences that contain the emotion. The Emotion class of the ontology contains information about the emotion model, appraisal, action tendency and emotion intensity. To support the annotation process, WorNet-Affect and EmotionML were used. Two different testing scenarios were created in order to evaluate the ontology. The first is to make queries against the ontology, and second is translating EmotionML resources to Onyx and vice versa.
Users of social media express their emotions using emoticons. The Smiley Ontology (SO) represents the structure and the semantic meaning of emoticons [65], which allows an application to understand and utilize emoticons. Moreover, it allows applications to exchange emoticons with the correct interpretation. The ontology design is based on Smiley Layer Cake.7
The Visual Sentiment Ontology (VSO) [11] was proposed to detect sentiment from visual content. The psychological foundation for the ontology is Plutchik’s wheel of emotion. The initial step to build the ontology is to retrieve images and videos from Flickr and YouTube respectively. The tags associated with the images and videos were analyzed. Then the top 100 tags for different emotions were obtained. For each tag, the sentiment value is computed according to the two linguistic models SentiWordNet and SentiStrength [82]. Due to this use of linguistic models on the image tags, we decided to categorize the VSO into the text domain. The assigned sentiment value for each emotion word ranges from
It has been shown that emotional facial expressions make up 55% of our communication [3]. Emotion is expressed in humans by facial movement. For example, when a person is surprised, they open their mouth and raise the eyebrows. Table 2 summarizes emotion ontologies in the facial expressions domain.
Summary of emotion ontologies in the facial expression domain
Summary of emotion ontologies in the facial expression domain
An emotion ontology was created to support the modeling of emotional facial animation expression in virtual humans within MPEG-4 [32]. Human actions are translated into a virtual world with avatars by using an ontology. A virtual world (environment) is a computer graphic-based environment that generates the impression that users are in a different place than their actual location [24]. The ontology allows storing, indexing and retrieving the right information about facial animation for a given emotion. The ontology define the relationship between facial animation concepts standardized in MPEG-4 and emotions. It contains the classes Face, Face Animation, Face Expression, Emotion and Emotion Model (Ekman and Plutchik). In addition, a class for parameters of facial animations is present. To use the ontology and extract the correct information for a specific emotion, the Racer Query Language interface for OWL is used [5]. This query language is close to natural language. For example, we can query using following question: What is the facial animation that expresses a depressed emotion?
Another emotions ontology was proposed within a framework called Nonverbal Toolkit for the cooperation of heterogeneous modules that gather, analyze and present non-verbal communication cues [38]. The aim of this framework is to gather non-verbal behavior in the real world and represent it in a virtual environment, such as an avatar in second life.8
Summary of individual ontologies that do not fit into the domains of the other tables
Summary of emotion ontologies in the contextual environment domain
In the e-learning domain, the Ontology for Predicting Students Learners’ Affect (OLA) [25] was introduced. The ontology is based on the OCC model of emotions. An interactive application was designed to estimate student emotion when interacting with a quiz about Java programming. The application monitors and records student action when answering questions and saves them in the student’s log file. When a student does not answer a question correctly, the event status becomes confirmed and appreciation is set to disliking. On the other hand, when the student answers a question correctly, appraisal become desirable. So, the log file data is the input for the ontology to compute the OCC model variables that predict student emotion by using ontology inference. Students may express different emotions while answering a single question. Therefore, the students’ emotions were studied in three different situations: When the students see the question for the first time, when the students choose the answer, and when the correct answer is displayed.
Emotion can be detected from voice by analyzing the change in voice tone, volume, rate, pitch, and the pauses between words [29]. In voice emotion extraction, the EmoSpeech system was built to convert unmarked input text to emotional voice. The developed emotion ontology (OntoEmotion) is organized in a taxonomy that covers the basic emotions to the most specific emotional categories [29]. OntoEmotion is presented in English and Spanish. The emotion class in the ontology uses the categorical model. The ontology represents the specific words each language offers for denoting emotion in its class named Word. To classify words in English and Spanish, the class Word has two subclasses named English Word and Spanish Word. Another class in the ontology defines emotion synonyms. The emotion concepts are linked to the three emotional dimensions of Evaluation, Activation and Power. This is in accordance with Osgood et al. as mentioned in Section 2. The EmoSpeech system uses EmoTag, which is a tool for automated mark-up of texts with emotional labels. These values are the input for the ontology. The ontology then classifies the input into an emotional concept. Next, the text is read aloud with the emotion assigned by EmoTag. For instance, EmoTag was applied on fairy tale stories in English and Spanish to annotate emotions [30].
Another application was designed to extract rich emotional semantics of tagged Italian artistic resources through an ontology method [6]. To select the tags that contain emotional content, several Semantic Web and natural language processing tools were incorporated such as multilingual lexicons (MultiWordNet) and affective lexicons (WordNet-Affect, and SentiWordNet). The software uses OntoEmotion because it has a taxonomic structure that reflects psychological models of emotions and is implemented by using Semantic Web technologies. However, the ontology was enhanced by adding a new subclass named Italian Word to the root concept Word.
Table 3 summarizes individual ontologies from the following domains: Emotion Voice Domain (OntoEmotion), Emotion Body Expression Domain, mood (COMUS), and need (FHN). Since there is only a small number of ontologies in these domains, we decided to create a table for miscellaneous domains.
Body expressions domain
Gestures and expressions of the human body can convey emotion. In [31] an ontology of body expressions to represent body gestures in virtual humans within MPEG-4 is presented. Animations are annotated with emotional information by using Whissel’s wheel of emotion. Because of the complexity of bodily expression, gestures were associated with emotions. In the ontology, seven gestures were considered. For example, hand clapping is associated with joy and excitement. To use the ontology, a query with natural language was used.
Contextual environment domain
Analyzing emotions within a given context gives insights to the relationship between an emotion and its cause. Table 4 summarizes emotion ontologies in the contextual environment domain.
Summary of emotion ontologies in the general domain
Summary of emotion ontologies in the general domain
An ontology to represent the affective states in context-aware applications was generated in [9]. It expresses the relationship between affective states and other contextual elements such as time and location. The ontology is built based on the existing ontology CONON. Its Emotion class (state) uses Ekman’s basic emotions. In addition, the class named Secondary contains emotion that is related to the targeted scenario. The ontology was applied to a visit to an art museum, where a person moves from one room to another while their emotions were monitored. The secondary emotions are thus relaxed and stressed. The ontology represents the most powerful emotion (dominant). Object properties are used to express the relation between emotions and context elements.
The ontology in [8] was built based on the previous ontology [9]. However, emotion was defined by three possible data type properties: positive, negative and neutral.
The BIO_EMOTION ontology recognizes emotion based on the user’s electroencephalographic (EEG) and bio-signal features, as well as the situation and environmental factors [88]. It supports reasoning about the user’s emotional state. The focus of the ontology is the mapping between low-level biometric features and high-level human emotion. It defines inference rules by using corresponding relationships between EEG and emotion. The BIO_EMOTION ontology consists of 84 classes and 38 properties. The Emotion class defines user affective states. Emotions are represented by Ekman’s discrete model and the dimensional circumplex model. User context is represented by the Situation class through location, time and event. Demographic information like name, age, and gender is integrated into the ontology. Additionally, a class to represent bio signals is provided. The machine learning software WEKA was used to generate IF-THEN statements in the reasoning process.9
General ontologies, also called upper level ontologies, have been proposed to recognize emotion. They define general concepts that are the common in a domain. Such ontologies can be extended according to the developer’s purpose by defining domain-specific classes, or they can be linked to an existing domain-specific ontology. Table 5 summarizes emotion ontologies in the general domain.
A high-level ontology named the Human Emotions Ontology (HEO) was developed in order to annotate emotion in multimedia data [34]. The main class in the ontology is Emotion which is expressed in dimensional and categorical models. An emotion has an intensity, appraisals and action tendencies, and it can be expressed through face, text, voice and gesture. Additionally, the ontology contains classes for the multimedia content and the annotator of the media. The Annotator class has two subclasses: Human or Machine (automatically annotated). Since the emotion is expressed by a person, HEO re-uses the Friend Of A Friend (FOAF) ontology. A subclass Observed Person of class person was created in FOAF and connected to the Emotion class of HEO. Moreover, some object properties were added in FOAF that are relevant to emotion such as age, culture, language and education.
The Semantic Human Emotion Ontology (SHEO) [2] was built based on HEO to identify complex emotions that are composed of two or more simple emotions. For instance, the complex emotion of contempt is a combination of the two basic emotions anger and disgust. Software was designed to use the ontology for analyzing simple and complex emotions in text as well as emotion in images.
In the general context, the Ontology of Emotional Cues to describe emotional cues at different levels of abstraction was proposed [58]. The concepts are gathered into three modules to detect emotion. Emotion is represented by a categorical and a dimensional model. An emotional cue can be simple or complex. This is represented in the emotional cue module. A simple cue can be expressed through facial expressions, gesture and speech. Complex emotional cues are a combination of two or more simple cues. The media module describes the properties of the emotional cues.
Mood ontologies
Mood can affect a person’s daily life choices. Building ontologies that can match a person’s mood with their desires allows for greater satisfaction. Oftentimes, people choose the music they listen to based on their current mood.
A recommendation-based music system was built with the Context-Based Music Recommendation Ontology (COMUS) to retrieve music in a semantic way [77]. The ontology reasons about the user’s mood, situation and preferences. It consists of classes concerning the Person, their Mood and the Music. The COMUS Ontology is connected to many ontologies such as FOAF. Some classes in COMUS, such as the person class that includes personal information, are similar to the FOAF ontology. The Person class defines general personal properties such as name, age, gender and hobby. Additionally, the ontology represents the user contexts event, time and location. The Mood class use a discrete model. Each mood class has a sub-class of mood similarity. For example, aggressive is similar to hostile, angry, etc. The other classes are domain-specific and related to music. Users can ask the music recommender system to find songs based on their mood. The system then delivers the appropriate music to them with the help of the ontology.
Need ontologies
Understanding and conceptualizing human needs helps to achieve human satisfaction. Building ontologies that define human needs through vocabulary and relationships allows to build systems that can automatically interpret and serve human needs.
The Fundamental Human Needs Ontology (FHN) was introduced to express the relationship between various needs and their satisfiers [17]. The ontology is based on the need model by Max-Neef. It represents the Agent and his or her Role, their Needs and Satisfiers. A person can play different roles, and each role requires different satisfiers. For example, when a person is at home their needs are satisfied differently from when they are at work.
Conclusions
This paper surveys existing ontologies in human affective states (emotion and mood) and their influences (personality, needs and subjective well-being). Emotion can be expressed in many ways such as in text, voice, facial expressions, and gestures. Great efforts have been made to build ontologies to detect and annotate emotions, while there are only few that target mood and human influence. We believe that investigating ontologies for mood, needs and personality in a similar way as it has been done for emotion is an interesting avenue for future research.
Subjective well-being corresponds to human life satisfaction, which in turn leads to positive emotion and good moods. It is thus an important influence factor on human affective states. After exploring existing ontologies, and to the best of our knowledge, we did not find ontologies regarding subjective well-being. Regarding personality, the Friend Of A Friend ontology includes MBTI personality traits. Even though MBTI is a popular personality theory, it is only used in the training world for businesses, while the Big Five personality model is preferred in academic research. It thus seems surprising that the Big Five model has not been considered in the design of any of the surveyed ontologies. We believe that creating ontologies for subjective well-being and the Big Five personality traits are interesting future fields of research.
Existing emotion ontologies display many similarities regarding the language, their classes and the Psychological theories that have been adapted. The OWL language is used in many ontologies such as in [44], and [32]. Also, it can be seen that the Ekman theory was adapted in many ontologies such as [32,80], and [38]. In contrast to Ekman’s emotion categories, the OCC model is also a popular model in the computer science area. It takes events and their causes into consideration. Hence, it is a valuable model to track the shift of human emotion when an event takes place. The dimensional emotion model, which is also employed frequently, plots emotions with multiple dimensions. As a result, it distinguishes variances in these dimensions for different people with similar emotions. For example, two persons with the same emotion “happy”, can have different numbers for each dimensions.
The existing ontologies were built to serve a variety of different languages like English, Chinese, Spanish, Japanese, French and Italian. An interesting observation is the existence of multilingual ontologies such as OntoEmotion. Multilingual ontologies can be built by mapping ontologies in different languages onto each other. Ontology mapping can be seen as a re-using process, which is one of the benefits of building ontologies. Mappings can be managed by using a multilingual dictionary such as WordNet. Several projects have adapted WordNet for the use with different languages. In the mapping process, the main step is finding semantic relationships between concepts in different ontologies. Creating a multilingual ontology allows for the building of multilingual applications that overcome the language barrier. This will open up an application to a wider range of users [22,78]. Areas like the medical domain and many others can benefit from multilingual ontologies. Physicians can make sure that they are referring to the same concept in multiple languages. Social media applications can also benefit from a multilingual ontology, because their users speak different languages.