Dialogue Systems and Conversational Agents for Patients with Dementia: The Human

Abstract

This study aimed to identify and describe the fundamental characteristics of spoken dialogue systems, and their role in supporting human–robot interaction and enabling the communication between socially assistive robots and patients with dementia. First, this work provides an overview of spoken dialogue systems by considering the underlying technologies, approaches, methods, and general issues. Then, the analysis focuses on studies, systems, and approaches that have investigated the role of dialogue systems and conversational agents in the interaction with elderly people with dementia by presenting the results of a literature review. While the overview of spoken dialogue systems relies on existing surveys and reviews, a research was conducted to identify existing works in the literature that have investigated the role of conversational agents and dialogue systems in the elderly and people with cognitive impairments. Inclusion criteria were as follows: (1) use of conversational agents, dialogue systems, or language processing tools for people with cognitive impairments; (2) age ≥60 years; (3) diagnosis of dementia according to National Institute on Aging-Alzheimer's Association (NIAAA) criteria; (4) presence of tests or experiments with qualitative or quantitative results. Initially 125 studies published between 2000 and 2017 were identified, of which 12 met the inclusion criteria. The review identifies the issues and challenges that are reported when conversational agents and speech-based interfaces have been used for interacting with people with cognitive impairments. In addition, the review led to the identification of studies that have investigated speech processing and natural language processing capabilities to assess the cognitive status of people with dementia.

Introduction

Worldwide, ∼50 million people have dementia, with ∼60% living in low- and middle-income countries.¹ The estimated proportion of the general population aged 60 and over with dementia at a given time is between 5 and 8 per 100 people.¹ The total number of people with dementia is projected to reach 82 million in 2030 and 152 million in 2050.¹

The difficulty with communication is an initial sign in the dementia onset, causing isolation and loneliness. In light of this assumption, the social robotic solutions could be a real and effective help for elderly people with dementia in terms of support and prevention.

The ability to communicate using natural language is a fundamental requirement for a social robot that aims at providing support for patients with dementia. Spoken dialogue is generally considered as the most natural way for social human–robot interaction.² Research efforts in the field of human–robot interaction increasingly focus on the development of robots equipped with intelligent communicative abilities, in particular speech-based natural language conversational abilities.³ These efforts directly relate to the research area of computational linguistics, generally defined as “the subfield of computer science concerned with using computational techniques to learn, understand, and produce human language content.”⁴

The advances and results in computational linguistics provide a foundational background for the development of so called spoken dialogue systems; that is, computer systems designed to interact with humans using spoken natural language.⁵ The strong interest in dialogue systems and conversational agents derives from their great potential in several domains. Significant R&D investments from major companies (such as Google, Apple, Microsoft, and Nuance) resulted in the integration of speech-based technologies in mobile platforms, with “digital personal assistants” entering the market in the form of well-known applications such as Apple Siri, Google Now, or Microsoft Cortana.

Therefore, for some real-world scenarios dialogue systems are mature enough to be commercially deployed, and provide support for specific tasks that require access to information and services, as in the case of applications for mobile platforms or in-car navigation and control systems.⁶ Hirschberg and Manning⁴ identify four main factors that enabled the significant progresses in computational linguistics and dialogue systems witnessed in the last few years, namely (1) the availability of increased computational power; (2) the availability of very large amounts of linguistic data in digital form; (3) the development of effective machine learning techniques; and (4) the increased understanding of the structure and role of natural language, in particular in social contexts.

Despite these factors, the availability of dialogue systems for supporting complex scenarios is limited to research prototypes, including the integration in service robots and assistive technologies requiring human–robot interaction to provide cognitive and physical assistance for disabled or elderly persons. Simulating human-to-human communication to enhance and ease human-to-machine communication is still a very challenging research task, in particular when the goal is to enable a natural, context-aware, adaptive, and intelligent interaction. Achieving such a complex goal requires a multidisciplinary approach with a contribution from different scientific disciplines, including linguistics and cognitive sciences, artificial intelligence and software engineering.

The aim of this work is to identify and describe the fundamental characteristics of spoken dialogue systems, by considering their main components and functionalities and outlining the methods and approaches that have been proposed and adopted. While performing a detailed and comprehensive review of the increasing number of existing dialogue systems is out of the scope of our analysis, we mainly focus on providing an overview of the underlying technologies, approaches, methods, and general issues. We then focus on studies, systems, and approaches that investigated the role of dialogue systems and conversational agents in the interaction with elderly people with dementia.

Conversational Agents: Chatterbots and Dialogue Systems

Early spoken language systems have been mainly developed for telephone-based conversational interfaces (Fig. 1).^7

–11 Modern conversational interfaces are more sophisticated, in that they perform a deeper analysis of utterances, make use of internal and external knowledge, and maintain a conversational status. Conversational agents can be classified in chatterbots and dialogue systems.^12,13 The main difference between them is that chatterbots aim at keeping a conversation going (e.g., for entertainment purposes), without an explicit attempt at understanding the meaning of the utterances, while dialogue systems try to establish an effective communication channel between the system and the user, for sharing information, responding to commands and building a common knowledge.

FIG. 1.

General architecture of a dialogue system.

More in detail, chatterbots and dialogue systems usually differ in the following features: Text processing: dialogue systems perform a more sophisticated processing (semantic parsing of the sentences). Conversational state: dialogue systems can manage conversational plans and maintain a conversation state (e.g., by keeping a dialogue history). Internal knowledge: dialogue systems generally make use of internal knowledge to reply appropriately. Clearly, the classification in chatterbots and dialogue systems is blurred since good chatterbots need to perform some kind of understanding of the language to be able to respond appropriately, while existing dialogue systems are not able to fully understand (as a human would) the meaning of sentences.

Chatterbots usually have limited language understanding capabilities, and they mainly focus on replying in a natural way, rather than really understanding the language. For this reason, they are usually more robust to errors, but they are quite limited in all tasks that require a real understanding of the utterances (e.g., learning from speech and provide useful answers). Modern chatterbots, however, are quite complex and open up to more advanced tasks that are usually carried out by complex dialogue systems. One of the first chatterbots was ELIZA,¹⁴ developed in 1966, which simulated a psychotherapist. After the success of ELIZA, hundreds of chatterbots have been developed. For instance, PARRY¹⁵ simulates a paranoid person. Chatterbot¹⁶ was one of the first chatterbots to pass a restricted version of the Turing test.

More recently, ALICE,¹⁷ a novel chatterbot based on the AIML language,¹⁸ was developed and won the Loebner Prize (a competition based on the Turing test). Other modern chatterbots, such as Cleverbot,¹⁹ learn conversational patterns from interaction corpora. Chatterbots have also been proposed for therapeutic purposes; for example, monitoring and reducing adolescents' stress²⁰ and treating people with Parkinson's disease.²¹ Other chatterbots include Facade⁴ and The Personality Forge.⁵ ChatScript²² is a recently developed chatterbot engine with advanced capabilities. The first chatterbot developed with ChatScript was Suzette,⁶ which won the 2010 Loebner Competition. Several languages, frameworks, and tool kits for the development of dialogue systems have been recently proposed.

The project TRINDI,^23,24 for instance, developed the framework TrindiKit for simplifying the implementation of dialogue processing theories, making the development of dialogue systems easier. Pamini²⁵ is another framework for defining human–robot interaction strategies based on generic interaction patterns. A recent work on spoken dialogue systems was done by Berg²⁶ who designed and developed NADIA, a framework for the creation of natural dialogue systems. Both chatterbots and dialogue systems have their advantages and disadvantages. Chatterbots are usually more tolerant to errors and noise, but they are often too simplistic for complex applications.

On the contrary, dialogue systems are harder to build, less robust to noise and their conversation domains are often more restricted, but they can handle more complex conversations. Some approaches have been recently proposed to combine chatterbots and dialogue systems into hybrid systems to take advantage of both paradigms, as discussed, for example, in previous studies.^12,13.

Dialogue Systems: Architectural Model, Components, and Approaches

Dialogue systems integrate multiple components, each providing a specific functionality, according to the reference conceptual architecture shown in Figure 2, where the modular structure of such systems and the information flow between the components are illustrated. A dialogue system consists of five main components providing the following functionalities: automatic speech recognition (ASR), natural language understanding (NLU), dialogue management (DM), natural language generation (NLG), and text-to-speech synthesis (TTS). Basically, user's utterances are converted into a textual representation by a speech recognizer, and then processed by a language understanding unit.

FIG. 2.

Functional architecture of a spoken dialogue system.

A dialogue manager interprets the semantic information, manages the conversation flow, and relies on a language generator for producing textual messages that are then rendered by a speech synthesizer. Simple dialogue systems and chatterbots may not have an internal knowledge, and hence they are stateless: they reply to every single request without contextualization. More complex systems maintain a simple status of the conversation (e.g., memorize which answers have been already given), and hence their conversation appears more natural. Other systems (known bots) use a huge amount of knowledge (typically gathered from the Web) to improve the interaction and answer to questions.

The internal knowledge can also be expanded with the knowledge gathered from speech analysis. In the following, we detail the role of each component, primarily focusing on NLU and DM, mainly in line with the analyses available in previous studies.^27
–29

Automatic speech recognition

ASR capabilities are provided by a speech recognizer or speech-to-text component. The speech recognizer is in charge of converting an input speech utterance into a textual representation. It receives as input the user's speech (typically captured by a microphone or provided as an audio file) and produces as its output a recognition hypothesis, as a sequence of words that most likely corresponds to what the user said. The recognized strings are often associated with some kind of confidence scores, where low values (typically in the range [0–1]) represent low confidence in the correct word recognition. Confidence scores can be also associated with entire word sequences, according to a N-best recognition approach where a list of N recognition hypotheses ranked in terms of likelihood is generated as output.

Depending on several factors, the textual result of the speech-to-text conversion may contain errors; for example, in terms of inserted, substituted, or missing words. López-Cózar et al. generally relate possible errors to environmental conditions (such as noise), acoustic similarity between words, and specific phenomena concerned with spontaneous speech, such as false starts, filled pauses, and hesitations.²⁷

These factors are systematically considered in a previous study,²⁸ where the authors relate possible recognition errors to the variability in the speech signal, which in turn depends on (1) linguistic variables, due to linguistic phenomena such as coarticulation (i.e., the acoustic realization and sound of a given phoneme may change depending on the context given by the preceding or following phonemes); (2) speaker variability, covering differences between and within speakers; and (3) channel variability, including the impact of noise and the properties of the transmission channel or input device (e.g., a microphone). The speech recognition process relies on the availability of a set of models to be matched with user's utterances encoded in the incoming speech signal.

The vast majority of speech recognizers adopt a stochastic/probabilistic approach by combining an acoustic model and a language model for a given language (e.g., English, Italian, French, etc.). In a nutshell, the acoustic model represents the basic speech units (phonemes or triphones) and their relationships. Acoustic models are typically represented as Hidden Markov models (HMMs) and derived from a previous training process.³⁰ The language model provides knowledge about words' likelihood in a given sequence, to determine the words or sentences that are expected from the user. Language models may be manually encoded in the form of finite state networks or grammars for a specific domain. These approaches provide good recognition accuracy in the case of well-defined sequences but lack in flexibility, as they assume that possible speech inputs are specified in advance and may force the user to comply with a static controlled vocabulary or restricted set of commands.³¹ Advanced approaches aim at deriving probabilistic language models by applying a learning process over language corpora. Statistical language models estimated from data, such as N-gram models, provide significant flexibility but may have higher error rates. Recent approaches, as implemented in the Google speech API, are relying on knowledge graphs able to provide vocabularies with millions of entities, while advanced machine learning techniques, such as deep learning approaches, are being investigated.^32,33

Natural language understanding

NLU component operates over the output of the speech recognizer, with the goal of deriving a semantic, meaningful representation of user's utterances. Language understanding is a complex task that may involve both syntactic and semantic analyses, taking into account, on the one side, the inherent complexity of natural language processing (e.g., resulting from the presence of ambiguities, anaphora, ellipsis, etc.) and, on the other side, the quality of the output of the speech recognizer. The NLU component thus combines different modules, including error correction tools, semantic parsers, and reference resolution and contextual interpretation engines. Different solutions have been proposed and adopted for addressing the task of producing semantic representations from textual representations of user's utterances, as provided by the speech recognition component.

Natural language parsers aim at building a representation of the input text taking into account its grammatical structure. According to the “principle of compositionality” identified in a study,²⁸ sentences are analyzed on the basis of their constituent structure under the assumption that “the meaning of a sentence is a function of the meanings of its parts.” Grammar thus represents the fundamental NLP approaches for text parsing and analysis to derive a semantic interpretation of user's utterances.³¹ Basic shallow parsing approaches, widely used in dialogue systems, are based on the (manual) definition of task- or domain-specific patterns that are matched against the input text for identifying and extract specific parts. The processing aims at recognizing or spotting specific information-bearing phrases or constituents (e.g., locative phrases and temporal expressions), ignoring the rest.

Shallow parsing is often coupled or even directly integrated with a speech recognition approach based on a controlled vocabulary or restricted grammar. These techniques are generally fast and efficient, and have the advantage of being easy to define and develop. Semantic shallow parsing approaches aim at identifying syntactical phrases, or chunks corresponding to basic semantic concepts, in natural language sentences to be labeled with semantic roles. In addition, a direct mapping of the understanding patterns to domain-specific concepts and semantics is possible. However, this domain specificity, together with development efforts, poses limitations in terms of language coverage and robustness.

Nevertheless, simple methods using words recognition, part-of-speech (POS) sequences (with the identification of nouns, verbs, preposition, etc.), or simple templates can often lead to notable results in domain-specific application scenarios.³⁴ Advanced approaches for NLU rely on domain-independent grammars and models, and aim at performing a grammatical and semantic analysis of the utterances with the goal of getting a rich understanding of their linguistic structure. These approaches rely on machine learning techniques, and are grounded in formal languages (which enable automated reasoning) and computational linguistic theories. NLU tools, such as Stanford CoreNLP³⁵ and FRED, that are able to identify syntactic and semantic information, taking also into account the discourse context, have been developed.

They rely on complex processing workflows that include POS tagging, named entities resolution, semantic role labeling, coreference resolution, and word-sense disambiguation. Multiple background theories are leveraged, such as Discourse Representation Theory, Combinatory Categorical Grammar and Frame Semantics. Linguistic frames are increasingly used for representing the output of the NLU process. Frame Semantics is a formal theory of meaning, whose basic idea is that humans can better understand the meaning of a single word by knowing the contextual knowledge related to that word.³⁶ Frame semantics allows real-world knowledge to be captured by semantic frames, which describe particular types of situations, actions, objects, or events, and their semantic arguments or participants, characterized by specific semantic roles with respect to the situation described by the frame.

NLU can thus be mapped to the frame detection or frame recognition task, which has the goal of recognizing complex relations in natural language text and build a machine-readable semantic representation. These approaches and techniques are strongly linked to machine reading, where interpretable structured knowledge is built from written text. However, written and spoken language differ in many ways. Although speech recognition provides a textual representation of user's speech, the textual encoding of user's utterances may have distinctive characteristics with respect to written language. In addition to possible errors introduced by the speech recognizer, user's utterances may not represent grammatically well-formed strings, and can include linguistic constructions and irregularities that are typical of spontaneous speech, such as self-corrections, hesitations, repetitions, and sentence fragments.

These factors, which are particularly relevant in the utterances of elderly people and patients with dementia,^37,38 pose challenges to the parsing capabilities of language understanding systems and require specific techniques for robust processing that range from syntactic/semantic input preprocessing (e.g., with the introduction of additional grammar rules able to recognize and handle some of these phenomena occurring sufficiently regularly) to partial parsing (similar to shallow parsing approaches) when a full linguistic analysis on each input term fails.

Dialogue management

DM capabilities are provided by a dialogue manager. The dialogue manager represents the core of a dialogue system: it keeps the conversational state of a dialogue and manages the overall human–machine interaction flow, taking into account user's utterances, the dialogue history, contextual information, and background knowledge. Typical activities performed by the dialogue manager relate to decision making and include the following: trigger a specific task or action (e.g., in the presence of a command issued by the user); provide information to the user (e.g., in response to a question); ask the user to provide information (e.g., through the generation of a question); undertake verification and/or confirmation steps in the case of uncertainty, recognition errors, and incomplete information.

An important task that crosses the boundaries between NLU and DM is the recognition and identification of speech or dialogue acts expressing the function of an utterance. Speech act analysis goes beyond syntactic and semantic analysis, as it may rely on external information, such as the discourse context and the user model. As the function of an utterance relates to user's goals, it plays a significant role in the decision-making process of a dialogue manager. Bunt et al. provide a taxonomic classification of domain-independent, general-purpose functions for classifying utterances.³⁹ On the one side, information-transfer functions distinguish between information-seeking acts (i.e., the speaker asks for information) and information providing acts (i.e., the speaker provides information).

On the other side, action discussion functions distinguish between commissive acts (i.e., the speaker commits to something, such as accepting/declining suggestions) and directive acts (i.e., the speaker suggests or instructs the interlocutor to do something). The different approaches that have been proposed and can be found in the literature for DM can be generally classified according to two related dimensions: the dialogue control strategy and the dialogue control model.

Dialogue control strategies

Dialogue control strategies consider the extent to which the user and/or the system maintains the initiative in a dialogue. Three possible strategies can be implemented. In a dialogue based on system initiative, the dialogue strategy is system led, and the system asks a sequence of questions to gather information from the user. The user can only perform question answering in response to system-initiated speech acts.

In a dialogue based on user initiative, the dialogue strategy is user led, and the user can ask questions to the system, which provides question answering capabilities, or issue commands. In a dialogue based on mixed initiative, the previous strategies are combined, and both the system and the user can take the initiative: the user can ask questions at any time, but the system can also take control of the dialogue to gather information from the user.

Dialogue control models

System-initiative and mixed-initiative strategies require the system to rely on a dialogue control model to manage the dialogue flow. Main dialogue control and management approaches include finite state-based models, frame-based approaches, information state models, and agent-based approaches.^27
–29

Finite state-based models

In a finite state-based dialogue model, a dialogue is represented as a finite state transition system, where the nodes represent the dialogue state and correspond to system's utterances or actions, while the transitions correspond to user's utterances. Possible interactions are thus defined by possible paths through the graph. Small-sized finite-state automata for dialogue control are easy to define, and the corresponding dialogues result in a predictable system behavior. These approaches are thus well suited for representing dialogue flows that correspond to well-structured, predefinable task-oriented interactions. However, state-based models lack in flexibility and naturalness, and the overall dialogue strategy is mainly driven by system initiative. The dialogue is basically defined as a scripted interaction, with limited support for deviations.

Frame-based models

In frame-based dialogue control strategies, the dialogue state is represented as a frame with slots that have to be filled by the user's answers. The basic idea of frames with slots to be filled is analogous to the concept of a form with fields to be filled. Basically, for each slot in a frame the system asks a question to gather information from the user and fill the corresponding slot, although the information provided by the user with an utterance may contribute to filling more than one slot, as in the case of overinformative answers. The goal of the system is to gather all the information required to fill the slots of a frame. A frame thus acts as a template that drives the interaction flow. The sequence of questions issued by the system is not predetermined and does not follow a rigid question–answer scheme.

Possible dependencies between frame slots can be declaratively expressed with precondition rules or priorities. Depending on the information gathered for each slot, the system may undertake verification or confirmation actions. Frame-based models increase the degree of flexibility with respect to finite state-based models, and a mixed-initiative strategy can be supported. Frame-based approaches are well suited for information retrieval tasks, where the system aims at gathering a fixed set of information from the user. Specific dialogue behaviors, such as information verification or confirmation steps, still need to be hard coded into the dialogue's control structure.

Agent-based approaches

Agent-based approaches include different techniques that rely on artificial intelligence and machine learning techniques for modeling and managing a dialogue as an interactive collaborative process between intelligent agents (basically, the system and the user). Plan-based approaches, for example, adopt planning techniques for DM. The basic assumption is that a dialogue is undertaken to achieve a specific goal, and utterances modeled as speech acts can be considered as action operators in a planning domain, in terms of constraints, preconditions, and effects. A dialogue strategy can thus be devised by solving a planning problem. Some agent-based approaches for plan-based DM are grounded in the belief–desire–intention model.

Basically, on the basis of its current beliefs about the domain (which include nested beliefs about shared knowledge) and the discourse obligations it has, the agent selects communicative goals, decides the speech acts to be performed, generates an utterance, analyzes the response, and updates its beliefs about the discourse state and its own discourse obligations. Recent research efforts relate DM with partially observable Markov decision processes (POMDPs).⁴⁰ These approaches aim at identifying an optimal or approximated DM policy by relying on a probability distribution over possible dialogue states, and updating the distribution as a result of the observed dialogue behavior (i.e., possible interpretations of linguistic utterances), in an action space given by the set of possible dialogue moves.

POMDP-based approaches allow for directly incorporating state uncertainty in the DM strategy and exploit reinforcement learning techniques for devising a suitable policy. Agent-based approaches allow implementing robust dialogue control strategies for flexible mixed-initiative dialogues. However, the planning and reasoning stages required for DM assume a complex representation of the dialogue state, introduce an additional source of complexity, and represent computationally intensive tasks that may contrast with the real-time performance requirements of dialogue systems.

Information state model

The information state approach²³ defines a general model for DM on the basis of information state-based theory.

The general theory defines the following main components: a description and formal representation of the informational components that collectively define the information state, including both static and dynamic components such as dialogue participants and user models, background knowledge, beliefs, intentions, etc.; a set of dialogue moves that produce updates over the information state; a set of update rules that formalize and control the way the information state is updated and dialogue moves are chosen; update rules basically consist of applicability conditions over the information state and effects that represent changes on the information state when the rule is applied; an update strategy that defines how to select and apply update rules among the applicable rules at each stage of the dialogue.

The general model allows for a rich representation of the dialogue state, and does not impose specific formalisms or algorithms to be used for the formalization of dialogue moves, update rules and update strategies. Rather than a specific DM approach, it represents a framework that allows dialogue theories to be formalized, implemented, and evaluated.

Natural language generation and text-to-speech synthesis

NLG capabilities are provided by a natural language generator component. It has the goal of transforming the information or decision produced by the dialogue manager into a grammatically and semantically correct textual format to be sent to the speech synthesizer to be spoken to the user. The textual message has to be coherent with the current dialogue status, and its construction requires to identify (1) the information that should be included; (2) how this information is structured; and (3) the linguistic realization of the message, that is, the formulation of the message, including the choices of lexical items and syntactic structures, as well as politeness and formality aspects. Basic approaches for language generation are template based; that is, they rely on templates that allow generating different sentence types.

A template typically includes fixed parts/words with predefined slots that must be filled/instantiated with data provided by the dialogue manager. More advanced NLG approaches dynamically produce textual messages using explicit domain and language models, such as generation grammars or statistical models.^41,42 The textual sentences generated by the NLG process represent the input for the TTS component, in charge of transforming the sentences into speech. In the case of speech capabilities based on a restricted dictionary, prerecorded words or sentences can be concatenated to produce an output utterance. However, modern speech synthesizers allow transforming arbitrary text into speech.

Although out of the scope of this discussion, speech synthesis is still a complex task, where speech generation is preceded by a text analysis stage that includes text segmentation and normalization, morphological analysis, and the modeling of continuous speech effects, with the final goal of producing natural-sounding speech.

In the following paragraphs, a literature review was made about the studies, systems, and approaches that investigated the role of dialogue systems and conversational agents in the interaction with elderly people with dementia.

Methods

The literature review has been performed according to the general guidelines defined in a previous study.⁴³ A keyword-based search strategy was defined to identify relevant studies through Google Scholar and accessing the SpringerLink digital library, the IEEE408 Xplore Digital Library, the ACM Digital Library, and PubMed. The search focused on articles published from 2000 to 2017.

The articles were evaluated according to the following inclusion criteria: (1) use of conversational agents, dialogue systems, or language processing tools for people with cognitive impairments; (2) age ≥60 years; (3) diagnosis of dementia according to the criteria of the National Institute on Aging-Alzheimer's Association (NIAAA)⁴⁴ and the Diagnostic and Statistical Manual of Mental Disorders-Fifth Edition (DMS-5)⁴⁵; and (4) presence of tests or experiments with qualitative or quantitative results. Articles presenting work on socially assistive robots where speech-based interaction is not explicitly considered were excluded.

Similarly, we excluded the articles presenting conversational agents or dialogue systems not specifically targeted at people with cognitive impairments. As shown in Figure 3, initially 125 articles were identified by the search strategy, and 67 articles were then selected after removing duplicates and a preliminary analysis of their titles. We then considered the abstract, introduction, and conclusions of the selected articles, and 44 of them were excluded accordingly as they did not meet the inclusion criteria. Twenty-three articles were further assessed, and 11 of them were excluded after an in-depth examination. Twelve articles were finally selected for the review (Table 1). During the analysis, the bibliographies of the selected articles were considered to identify additional works potentially excluded by the search strategy.

FIG. 3.

Flow diagram outlining the selection procedure to identify articles that were included in the analysis of the state of the art in speech-based robot socialization.

Table 1.

Conversational Agents, Dialogue Systems and Natural Language Processing Techniques for Patients with Dementia

Studies	Methods	Outcomes
Heerink et al.⁴⁶	CRI	Acceptance of the robot as a conversational partner increases if the robot exhibits socially communicative abilities.
Yasuda et al.⁴⁷	VCA	Conversational agent practical and valuable alternative to humans for reminiscence; issues in speech recognition for nonspeech utterances.
Yaghoubzadeh et al.⁴⁸	VCA	Importance of explicit confirmation dialogues as part of the dialogue strategy; better results achieved when limited amount of information is presented for confirmation.
Bruno et al.⁴⁹	SAR	Need to provide a socially assistive robot with the ability to perceive, interpret, and convey emotions through speech.
Granata et al.⁵⁰	SAR-DS	Take into account the difference between the vocabulary and syntactic structures used by elderly and nonelderly adults.
Roy et al.⁵¹	SAR-DS	Keyword-spotting techniques considered sufficient to provide support for simple tasks for people with mild forms of dementia.
Montemerlo et al.⁵²	SAR-DS	Partially observable Markov decision processes as a robust solution to cope with uncertainty and poor speech recognition.
Nonaka et al.⁵³ Huang et al.⁵⁴ Sakai et al.⁵⁵	VCA-SA	Pause duration, utterance duration, pitch, and frequency of head nods related to patient's responsiveness in the conversation, as indicators of patient's cognitive status.
Roark et al.⁵⁶	NLP-SA	NLP and speech processing techniques able to automatically derive measures and diagnostic markers for discriminating between healthy subjects and those with mild cognitive impairment.
Sadohara et al.⁵⁷	DS-DAC	Machine learning techniques for a sublexical classifier for dialogue acts using sequences of phonemes.

CRI, communicative robotic interface; DAC, dialogue act classification; DS, dialogue system; NLP, natural language processing; SA, speech analysis; SAR, socially assistive robot; VCA, virtual conversational agent.

The results section was divided as follows:

Conversational agents

Socially assistive robot

Speech analysis and classification

Results

Conversational agents

Heerink et al. present the results of a field experiment conducted with a Wizard-of-Oz approach to investigate the role of perceived social abilities in the acceptance of a communicative robotic interface by elderly users.⁴⁶ The study was performed in 2005 with the Philips iCat robot with simulated conversational capabilities. Conversational scripts were developed for the robot to simulate two conditions: a more socially communicative interface (with the robot able to remember and use the interlocutor's name, and configured to look at the user, nod, and smile) and a less socially communicative interface. Forty elderly inhabitants (13 male, 27 female) of an eldercare institution in The Netherlands were observed during interaction sessions lasting 5 minutes, and were then interviewed with questions on the perceived social abilities of the robot and technology acceptance.

Specifically, the participants were invited to have a conversation with the robot to ask it to perform the three simple tasks (setting an alarm, giving directions to the nearest supermarket, and giving the weather forecast for the next day). From a general perspective, the experiment was not able to highlight any significant correlation between perceived social abilities and technology acceptance. However, significant differences were observed in the results regarding the acceptance of the robot as a conversational partner, asking the participants whether they felt uncomfortable when talking to a robot. All participants interacting with the robot configured with more socially communicative abilities reported to feel comfortable, while nearly half of the participants interacting under less socially communicative conditions reported to feel a little or very uncomfortable.

In addition, behavior analysis over recorded interactions showed that elders tend to be more expressive when interacting with a more sociable robot. The authors also report that, for several participants, the conversation with the robot went beyond the possible tasks and the presented possible functionalities of the robot. The authors describe the development of an agent system able to serve as a conversation partner for individuals with dementia.⁴⁷ The agent is an animated face of a child shown on a computer screen, and it was used to conduct an evaluation experiment with eight subjects with mild Alzheimer's disease (two male and six female with an average age of 78.5 years and a mean mini-mental state examination score of 22.2).

Participants were asked to reply to the same 15 reminiscent questions under two conditions; that is, with questions asked by the virtual agent and by a human speech therapist. Each question was preceded by introductory comments and also shown in textual form on the screen. The analysis focused on evaluating the effectiveness of the conversational agent in eliciting information from the subjects, by comparing the length of the utterances (using the syllable number) under the two conditions. Reported results show that the overall number of syllables used by all the participants to reply to the agent was lower (74%) than that used when replying to the human partner. Although the methodology and results of the study are questionable, the authors claim that conversational agents may represent a practical and valuable alternative when no human conversation partner exists.

Relevant observed phenomena include (1) the fact that some subjects replied to the introductory comment by the agents before the actual question was asked; (2) the difficulties faced by the speech recognition system to deal with nonspeech utterances (such as coughing or sighing) and background noise; and (3) the comment made by one of the participants about the possibility to “talk freely without any hesitation or anxiety” when interacting with the agent. Yaghoubzadeh et al. investigated the role of virtual agents as daily assistants for elderly or cognitively impaired people, also focusing on how to structure and present the dialogue-based conversational behavior of the agent to support a robust and effective interaction with the user.⁴⁸

To this end, a prototype of a virtual agent (acting as an assistant in managing appointments on the calendar) was built and evaluated. Speech recognition capabilities are provided by Windows Speech Recognition or Nuance Dragon NaturallySpeaking systems, and NLU capabilities are based on the ability to identify keywords (i.e., keywords spotting) and simple grammatical structures. DM was implemented on the basis of the information state model described in the previous section. However, the evaluation was conducted according to a Wizard-of-Oz approach, to evaluate users' interaction with a conversational agent and their reaction to system misunderstandings introduced on purpose. The study was conducted with two user groups: 6 elderly users and 11 cognitively impaired users.

Participants were instructed to verbally interact with the system to insert appointments in the calendar, and were then interviewed about system usability and acceptability and about desirable features of the virtual agent. During the dialogues, two data confirmation and verification strategies were experimented, with the agent either summarizing an appointment in a single utterance to be confirmed or presenting the information slots one by one (date, time, topic, etc.) with sequential confirmations. On the basis of their results, the authors conclude that both user groups were able to handle interaction problems that may occur when interacting with speech-based systems (such as errors in the recognition and understanding process), but the corresponding dialogue strategies were simulated and not really implemented in the system.

In addition, the analysis suggests that explicit confirmation dialogues are an important aspect of the dialogue strategy, and better results are achieved when limited amount of information is presented for confirmation.

Socially assistive robot

An analysis of the functional requirements and design principles for socially assistive robots for elderly people with mild cognitive impairments is carried out in the study of Bruno et al.⁴⁹ Starting from the observation that dialogue is one of the most important social interaction abilities for socially assistive robots, the authors identify the ability to convey information in spoken natural language (i.e., the ability to produce speech) as a key requirement for the robot's interaction system. The analysis considers the role of emotions in social robots, and identifies the need to provide the robot with the ability to perceive, interpret, and convey emotions. The authors argue that it is possible to design a robot able to perceive and convey emotions within speech. Their claims draw on existing results that, on the one side, have investigated the role of speech as a valid method to convey emotions and, on the other side, have shown that emotions produce vocal effects, which are consistent between speakers.

The combination of a voice-based and graphical interface for supporting the interaction between an assistive robot and elderly people with mild cognitive impairments is discussed in a previous study.⁵⁰ Starting from the observation that memory deficits affect language comprehension and production, the authors identify the need to design an interface able to deal with potential problems of human speech, including sentence fragments, interruptions, and false starts. To identify the required features for the vocal and graphical interface, two tests were conducted with a Kompai mobile robot. The implemented dialogue system relies on speech recognition, DM, and text-to-speech capabilities augmented with a multimodal approach that allows combining voice and graphical input/output. Specific tests were conducted with six patients from a geriatric hospital in Paris to identify the vocabulary and syntactic structures that characterize the utterances of people with mild cognitive impairments. The dialogue system was configured with a vocabulary of ∼170 French words and different syntactic models related to the specific tasks supported by the robot.

Speech capabilities of the robot were based on the production of short and simple sentences, as a simple syntactic structure avoids a cognitive overload for the user. The patients were asked to perform selected tasks with the robot (such as make an appointment, add/remove items from a shopping list, or asking for information about, for example, date and time or appointments) using a Wizard-of-Oz approach. The sentences uttered by the patients were then manually annotated and compared with the preconfigured vocabulary and syntactic models. The authors report a significant difference between the vocabulary used by elderly and those used by nonelderly adults (and used to configure the dialogue system). Some differences were also observed regarding the syntactic structure of the utterances.

The authors highlight the importance of taking into account, in the design process, the peculiarities of the vocabulary and syntactic structures of the target users, as age and cognitive impairments may introduce differences with respect to reference models used by nonelderly adults. In early 2000, the Nursebot project started investigating the development of a personal service robot, Flo, with social interaction abilities, specifically targeted at people with mild forms of dementia and cognitive impairment.⁵¹ Spoken interaction capabilities were considered as an essential requirement to enable a natural interaction between the users and the robot. Flo was equipped with a speech recognition interface configured with a vocabulary of ∼100 words and controlled by a dialogue system.

To deal with simple domains, language understanding was based on keyword-spotting techniques. Although advanced NLP techniques were considered as not necessary, the dialogue manager was able to exploit external information sources and connect to the World Wide Web. Flo was superseded by the Pearl robot, and some experiments were conducted in an assisted living facility, as reported in a previous study.⁵² In Pearl software components were implemented relying on probabilistic techniques to take into account uncertainty factors. Regarding speech capabilities, this was required due to the difficulties observed in people with cognitive deficiencies in articulating computer-understandable responses.

Speech analysis and classification

Techniques based on POMDPs were used for robot control and DM. Among the experiments, the speech interface was compared with a system unable to deal with uncertainty. The average time to task completion, the average number of errors, and the average user-assigned reward were measured in the presence of good, average, and poor speech recognition conditions. Reported results show that, especially in the case of poor speech recognition, the POMDP-based approach exhibits a greater time to task completion due to the generation of clarification questions, but the corresponding error rate is much lower. The authors generally identify probabilistic models and techniques that cope with uncertainty as a viable solution for building robust assistive robotic solutions for people with mild forms of dementia and cognitive impairment.

The authors investigate the possibility to assess the cognitive status of people with dementia through conversations, focusing on patients' communication responsiveness.^53
–55 To this end, a prototype of a virtual listener agent (shown on a computer screen) was developed, and conversations between the agent and 10 people with dementia (2 male and 8 female) were recorded and evaluated. The conversational agent is able to sequentially ask the user a set of questions regarding, for example, the patient's physical conditions or related to reminiscence and patient's memories. Understanding capabilities are based on a combination of keyword spotting and a similarity-based method to compare keywords with a list of possible utterances to be recognized.

While interacting with the user, the agent records the number of utterances, their duration, and the pitch. User's behavior in providing each response was then manually classified as high responsive and low responsive. For these two classification groups, the following metrics were evaluated: (1) the pause between the end of the agent's question and the start of the user's answer, (2) the pitch of the user's response, (3) the average utterance duration, and (4) the average frequency of head nods per response. The authors report that higher responsiveness correlates with shorter pause, higher pitch, longer utterance, and more frequent head nods. They thus claim that patient's responsiveness in the conversation with an agent, as an indicator of patient's cognitive status, can be assessed using the aforementioned metrics.

Roark et al. have recently investigated the application of natural language processing techniques to analyze spoken responses provided by subjects during neuropsychological examinations.⁵⁶ The study aims at identifying diagnostic markers in spoken language that allow discriminating between healthy elderly subjects and subjects with mild cognitive impairment. To this end, the authors evaluate the role of different linguistic complexity measures and pause statistics to derive early markers of Alzheimer's disease. Audio recordings were collected from 74 neuropsychological examinations, partitioned into 37 healthy subjects and 37 subjects with mild cognitive impairment.

The recordings were manually transcribed and annotated with syntactic annotations. In a first stage, multiple linguistic complexity measures and other spoken language-derived measures were evaluated, covering both measures derived from the syntactic structure of the utterances and measures derived from temporal aspects of the speech samples (e.g., pauses and their duration). In a second stage, automated NLP techniques were used for parsing and processing transcripts and audio recording. The study highlights a high correlation between the measures derived from automatic processing and the manually derived measures.

According to the authors, this correlation indicates that the automatically derived measures are able to preserve the discriminative utility of manually derived measures in differentiating between healthy subjects and subjects with mild cognitive impairment. Presented results show the potential of applying NLP and speech processing techniques to spoken language samples, to automatically derive measures for discriminating between healthy subjects and those with mild cognitive impairment. The authors propose an approach for classifying dialogue acts in spontaneous speech of the elderly with mild dementia.⁵⁷

The goal is to provide a dialogue system with the ability to classify dialogue acts and assign functional tags to user's utterances, to enable the understanding of the communicative intentions of the utterances and produce an appropriate response in the context of a conversation. Rather than undertaking a language modeling approach, considered by the authors as too expensive for spontaneous speech, the proposed solution introduces a sublexical dialogue act classifier that processes sequences of phonemes provided by a phoneme recognizer. The study builds on the hypothesis that subsequences of phonemes should be effective indicators for identifying dialogue acts, in the same way as word n-grams can be used for this task. The analysis and classification are performed by a sublexical classifier based on a support vector machine (SVM).

An empirical study was conducted to compare the sublexical dialogue act classifier with a classifier based on a HMM. The approaches were compared on a dialogue corpus containing 4080 user utterances derived from the interactions between 20 participants (3 male and 17 female, with an age ranging from 67 to 97 and a mini-mental state examination score ranging from 9 to 30) and the system. In total, 12 dialogue acts were identified for representing the communicative intention of the utterances (e.g., questions, confirmations, affirmative answers, etc.). Both the classification approaches were evaluated and compared in terms of accuracy. The sublexical SVM-based classifier was also compared with a SVM-based classifier operating on lexical features. Presented results show that the sublexical classifier is robust against the poor language modeling of language, and it performed better than the HMM-based lexical classifier whose performance decreases when the mismatch between the language model and the actual corpus increases.

Conclusion

Speech is largely considered as the most powerful and effective communication mode for an assistive social robot to interact with its users. Recent technological developments and research results are contributing to solving the challenges that characterize the design and implementation of spoken dialogue systems for human–robot interaction. In particular, the increasing accuracy of speech recognizers and the advances in the field of natural language processing provide viable solutions for the adoption of dialogue systems in different domains. In addition, probabilistic models and machine learning techniques are increasingly used to deal with the uncertainty factors that characterize language processing and DM.

We have provided an overview of the technologies, approaches, methods, and general issues that come into play in the complex process of building a dialogue system. Each functional component in the processing chain plays a fundamental role in the overall system performance and has to be carefully designed. However, the approach and techniques to be adopted for each stage and their effectiveness largely depend on the target domain, the intended users, and the specific tasks that the system has to support. For example, the usage of a restricted vocabulary, keyword-spotting techniques, and simple scripted DM strategies can be effective in supporting predefined tasks and dialogues where the interaction is mainly driven by the system (e.g., for eliciting specific information from the user through a set of questions). While advanced stochastic techniques and deep NLP would be an overkill in these settings, they improve the ability of the system to deal with uncertainty and open-ended dialogues with a mixed-initiative interaction pattern. The presented analysis supports the communication between the elderly with cognitive impairments and a social robot, and introduces additional issues. The ability of a dialogue system to respond to user needs affects the perceived social abilities of the robot, and has thus an impact on the acceptance by the users. Speech disfluencies, such as repetitions and incomplete words or sentences observed from the analysis of conversational dialogues, pose challenges to speech recognition and language processing capabilities.

In addition, some authors have observed that age and cognitive impairments produce modifications in the vocabulary and syntactic structures used by the elderly or people with dementia, resulting in the need for adaptive techniques able to take these changes into account. Similarly, it has been observed that the communication abilities of the system have to be adapted to the target users to improve the understanding and reduce the cognitive load. This includes, for example, a relatively slow speech rate, closed-ended questions, repetitions and small verification questions, a reduced syntactic complexity and sentences with few clauses.

Most of the experiments considered in the analysis were conducted using a Wizard-of-Oz approach, and thus aimed at deriving requirements and evaluating the potential feasibility and acceptability of vocal interfaces and conversational agents, rather than at evaluating the actual capabilities of dialogue systems in real-world scenarios. Other authors have investigated speech processing and NLP capabilities from a different perspective. They focused on the possibility to assess the cognitive status of people with dementia by identifying and analyzing metrics and diagnostic markers able to serve as indicators for evaluating patient's cognitive status and discriminating between healthy subjects and people with cognitive impairment.

Reported results show that these approaches have a potential and their integration in a dialogue system deserves further investigation, along with emerging NLP techniques that focus on sentiment analysis and the identification of the emotional state of the speaker. Similarly, the ability of a social robot to convey emotions through the produced speech is still an open research challenge, where further efforts are necessary with multidisciplinary contributions toward socially assistive robots.

Footnotes

Acknowledgments

The research leading to these results has received funding from the European Union Horizon 2020—the Framework Programme for Research and Innovation (2014–2020) under grant agreement 643808 Project MARIO “Managing active and healthy aging with use of caring service robots.”

Author Disclosure Statement

No competing financial interests exist.

References

World Health Organization. Dementia: Updated December. 2017. www.who.int/mediacentre/factsheets/fs362/en/ Last accessed February 06, 2018

Fong

, Nourbakhsh

, Dautenhahn

. A survey of socially interactive robots. Rob Auton Syst, 2003; 42:143–166.

Mavridis

. A review of verbal and non-verbal human-robot interactive communication. Rob Auton Syst, 2015; 63(P1):22–35.

Hirschberg

, Manning

. Advances in natural language processing. Science, 2015; 349:261–266.

McTear

. Spoken Dialogue Technology: Towards the Conversational User Interface. London: Springer, 2004.

Mariani

, Rosset

, Garnier-Rizet

, et al. Natural Interaction with Robots, Knowbots and Smartphones: Putting Spoken Dialog Systems into Practice. New York: Springer, 2013.

Möller

. Quality of Telephone-Based Spoken Dialogue Systems, 1st Ed. New York: Springer, 2010.

Zue

, Glass

. Conversational interfaces: Advances and challenges. Proc IEEE, 2000; 88:1166–1180.

Gorin

, Riccardi

, Wright

. How may I help you?. Speech Commun, 1997; 23:113–127.

10.

Zue

, Sene

, Glass

, et al. JUPITER: A telephone-based conversational interface for weather information. IEEE Trans Speech Audio Process, 2000; 8:85–96.

11.

, Rudnicky

. Task-based dialog management using an agenda. In: Proceedings of the 2000 ANLP/NAACL Workshop on Conversational Systems, vol. 3. Seattle, WA: Association for Computational Linguistics, 2000, pp. 42–47.

12.

Kluwer

. From chatbots to dialog systems. In: Perez-Marin

, Pascual-Nieto

(eds.): Conversational Agents and Natural Language Interaction: Techniques and Effective Practices. Hershey, PA: IGI Global, 2011, pp. 1–22.

13.

Dingli

, Scerri

. Building a hybrid: Chatterbot—Dialog system. In: Habernal

, Matousek

(eds.): Text, Speech, and Dialogue. Berlin Heidelberg: Springer, 2013, pp. 145–152.

14.

Weizenbaum

. ELIZA—A computer program for the study of natural language communication between man and machine. Commun ACM, 1966; 9:36–45.

15.

Colby

, Weber

, Hilf

. Artificial paranoia. Artif Intell, 1971; 2:1–25.

16.

Mauldin

. ChatterBots, TinyMuds, and the Turing Test: Entering the Loebner Prize competition. In: Proceedings of the Twelfth National Conference on Artificial Intelligence, vol. 1. Menlo Park, CA: AAAI, 1994, pp. 16–21.

17.

Wallace

. The Anatomy of A.L.I.C.E. In: Epstein

, Roberts

, Beber

. (eds.): Parsing the Turing Test: Philosophical and Methodological Issues in the Quest for the Thinking Computer. Dordrecht: Springer, 2009, pp. 181–210.

18.

AIML: Artificial Intelligence Markup Language. 2001. http://xlm.coverpages.org/aiml-ALICE.html Last accessed on February 3, 2018 .

19.

Cleverbot. www.cleverbot.com Last accessed on January 22, 2018 .

20.

Huang

, Li

, Xue

, et al. TeenChat: A chatterbot system for sensing and releasing adolescents' stress. In: Proceedings of the 4th International Conference on Health Information Science. Cham: Springer, 2015, pp. 133–145.

21.

Ireland

, Liddle

, McBride

, et al. Chat-Bots for people with Parkinson's disease: Science fiction or reality?. Stud Health Technol Inform, 2015; 214:128–133.

22.

GitHub. https://github.com/bwilcox-1234/ChatScript Last accessed on January 22, 2018 .

23.

Traum

, Larsson

. The information state approach to dialogue management. In: van

Kuppevelt J

, Smith

(eds.): Current and New Directions in Discourse and Dialogue. Dordrecht: Springer, 2003, pp. 325–353.

24.

Larsson

, Traum

. Information state and dialogue management in the TRINDI dialogue move engine toolkit. Nat Lang Eng, 2000; 6:323–340.

25.

Peltason

, Wrede

. Pamini: A framework for assembling mixed-initiative human-robot interaction from generic interaction patterns. In: Proceedings of the SIGDIAL 2010 Conference. Stroudsburg, PA: Association for Computational Linguistics, 2010.

26.

Berg

. NADIA: A simplied approach towards the development of natural dialogue systems. In: Natural Language Processing and Information Systems. Cham, Switzerland: Springer, 2015, pp. 144–150.

27.

López-Cózar

, Callejas

, Griol

, et al. Review of spoken dialogue systems. Loquens, 2014; 1:e012.

28.

McTear

. Spoken dialogue technology: Enabling the conversational user interface. ACM Comput Surv, 2002; 34:90–169.

29.

Zue

, Sene

. Spoken dialogue systems. In: Benesty

, Sondhi

, Huang

(eds.): Springer Handbook of Speech Processing. Berlin Heidelberg: Springer, 2008, pp. 705–722.

30.

Gales

, Young

. The application of hidden Markov models in speech recognition. Found Trends Signal Process, 2007; 1:195–304.

31.

Bastianelli

, Castellucci

, Croce

, et al. Effective and robust natural language understanding for human-robot interaction. In: Proceedings of the Twenty-first European Conference on Artificial Intelligence. Amsterdam, The Netherlands: IOS Press, 2014, pp. 57–62.

32.

Deng

, Li

. Machine learning paradigms for speech recognition: An overview. Trans Audio Speech Lang Proc, 2013; 21:1060–1089.

33.

, Deng

. Automatic Speech Recognition: A Deep Learning Approach. New York: Springer, 2014.

34.

Roukos

. Natural language understanding. In: Benesty

, Sondhi

, Huang

(eds.): Springer Handbook of Speech Processing. Berlin Heidelberg: Springer, 2008, pp. 617–626.

35.

CoreNLP. https://stanfordnlp.github.io/CoreNLP Last accessed on January 22, 2018 .

36.

Fillmore

. Frame semantics and the nature of language. Ann N Y Acad Sci, 1976; 280:20–32.

37.

Vacher

, Aman

, Rossato

, et al. Development of automatic speech recognition techniques for elderly home support: Applications and challenges. In: Proceedings of the First International Conference on Human Aspects of IT for the Aged Population. Cham, Switzerland: Springer, 2015, pp. 341–353.

38.

Young

, Mihailidis

. Difficulties in automatic speech recognition of dysarthric speakers and implications for speech-based applications used by the elderly: A literature review. Assist Technol, 2010; 22:99–112.

39.

Bunt

, Petukhova

, Traum

, et al. Dialogue Act Annotation with the ISO 24617-2 Standard. In: Dahl

(ed.): Multimodal Interaction with W3C Standards: Toward Natural User Interfaces to Everything. Cham, Switzerland: Springer, 2017, pp. 109–135.

40.

Young

, Gasic

, Thomson

, et al. POMDP-based statistical spoken dialog systems: A review. Proc IEEE, 2013; 101:1160–1179.

41.

Rieser

, Lemon

, Keizer

. Natural language generation as incremental planning under uncertainty: Adaptive information presentation for statistical dialogue systems. IEEE/ACM Trans Audio Speech Lang Proc, 2014; 22:979–994.

42.

Shannon

, Zen

, Byrne

. Autoregressive models for statistical parametric speech synthesis. IEEE/ACM Trans Audio Speech Lang Proc, 2013; 21:587–597.

43.

Kitchenham

, Charters

. Guidelines for performing systematic literature reviews in software engineering. Technical report, EBSE, 2007.

44.

McKhann

, Knopman

, Chertkow

, et al. The diagnosis of dementia due to Alzheimer's disease: Recommendations from the National Institute on Aging-Alzheimer's Association workgroups on diagnostic guide-lines for Alzheimer's disease. Alzheimers Dement, 2011; 7:263–269.

45.

American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, 5th Ed. Washington: American Psychiatric Association, 2013.

46.

Heerink

, Krose

, Evers

, et al. The influence of a robot's social abilities on acceptance by elderly users. In: Proceedings of the 15th IEEE International Symposium on Robot and Human Interactive Communication. 2006, pp. 521–526.

47.

Yasuda

, Aoe

, Fuketa

. Development of an agent system for conversing with individuals with dementia. In: Proceedings of the 27th Annual Conference of JSAI. Tokyo, Japan: The Japanese Society for Artificial Intelligence, 2013, vol. JSAI2013 3C1-IOS-1b-2.

48.

Yaghoubzadeh

, Kramer

, Pitsch

, et al. Virtual agents as daily assistants for elderly or cognitively impaired people. In: Aylett

, Krenn

, Pelachaud

, Shimodaira

(eds.): In: Proceedings of the 13th International Conference on Intelligent Virtual Agents . Berlin Heidelberg: Springer, 2013, pp. 79–91.

49.

Bruno

, Mastrogiovanni

, Sgorbissa

. Functional requirements and design issues for a socially assistive robot for elderly people with mild cognitive impairments. In: 2013 IEEE RO-MAN. 2013, pp. 768–773.

50.

Granata

, Chetouani

, Tapus

, et al. Voice and graphical-based interfaces for interaction with a robot dedicated to elderly and people with cognitive disorders. In: 19th International Symposium in Robot and Human Interactive Communication. Viareggio, Italy: IEEE, 2010, pp. 785–790.

51.

Roy

, Baltus

, Fox

, et al. Towards personal service robots for the elderly. In: Workshop on Interactive Robots and Entertainment, vol. 25. Pittsburg, PA: WIRE 2000, pp. 184.

52.

Montemerlo

, Pineau

, Roy

, et al. Experiences with a mobile robotic guide for the elderly. In: Eighteenth National Conference on Artificial Intelligence. Menlo Park, CA: American Association for Artificial Intelligence, 2002, pp. 587–592.

53.

Nonaka

, Sakai

, Yasuda

, et al. Towards assessing the communication responsiveness of people with dementia. In: Proceedings of the 12th International Conference on Intelligent Virtual Agents. Berlin Heidelberg: Springer, 2012, pp. 496–498.

54.

Huang

, Matsushita

, Kawagoe

, et al. Toward a memory assistant companion for the individuals with mild memory impairment. In: 2012 IEEE 11th International Conference on Cognitive Informatics and Cognitive Computing. 2012, pp. 295–299.

55.

Sakai

, Nonaka

, Yasuda

, et al. Listener agent for elderly people with dementia. In: 2012 7th ACM/IEEE International Conference on Human-Robot Interaction. New York, NY: ACM, 2012, pp. 199–200.

56.

Roark

, Mitchell

, Hosom

, et al. Spoken Language Derived Measures for Detecting Mild Cognitive Impairment. IEEE Trans Audio Speech Lang Processing, 2011; 19:2081–2090.

57.

Sadohara

, Kojima

, Narita

, et al. Sub-lexical dialogue act classification in a spoken dialogue system support for the elderly with cognitive disabilities. In: 4th Workshop on Speech and Language Processing for Assistive Technologies. Stroudsburg, PA: ACL, 2013, pp. 93–98.

Dialogue Systems and Conversational Agents for Patients with Dementia: The Human–Robot Interaction

Abstract

Introduction

Conversational Agents: Chatterbots and Dialogue Systems

Dialogue Systems: Architectural Model, Components, and Approaches

Automatic speech recognition

Natural language understanding

Dialogue management

Dialogue control strategies

Dialogue control models

Finite state-based models

Frame-based models

Agent-based approaches

Information state model

Natural language generation and text-to-speech synthesis

Methods

Results

Conversational agents

Socially assistive robot

Speech analysis and classification

Conclusion

Footnotes

Acknowledgments

Author Disclosure Statement

References