TTS-Driven Synthetic Behaviour-Generation Model for Artificial Bodies

4.1 The lexical affiliates

The lexical affiliates have been described by Schegloff as the “word of words deemed to correspond most closely to a gesture in meaning” [23]. The affiliates are used in order to establish explicit semantic relationships between speech and gestures. The semantic relation used in the proposed system is limited to the identification of meaningful words (e.g., the words and the gesture were observed to co-align with). In addition, the affiliates used within our system also symbolically describe the gesture prototype in the form of semiotic classification (e.g., intention, morphology). The most natural way to establish these relations is by observing natural conversational behaviour (gesture analysis by the annotation of a multi-modal corpus). In particular, the corpus-based nonverbal behaviour models are, in general, the closest possible approximations of natural behaviour. The annotation schema proposed in [36] is already oriented towards text-centric (text-driven) synthesis for more “natural” human-like co-verbal gestures. The proposed topology and the formal model of the annotation as shown in Figure 2 suggest that each expression be encoded as a series of consecutive prototypical shapes encoded with additional linguistic information (e.g., segmentation hypothesis [24]). The formal model of the annotation schema captures semantic and semiotic features separately for each body-part and in the form of movement phases and movement phrases [22]. Although there are some significant differences between speaker-styles (e.g., intensity, frequency, particular shapes used, etc.), some general trends can be identified [27]. For instance, specific gesture prototypes have the tendency to co-occur within the same semiotic dimensions and over certain meaningful words, phrases or sequences. If not done so semantically, the gesture prototypes may be linked based on similar morphological patterns representing a semiotic dimension.

During the annotation procedure, the gesture prototypes were identified in the Slovenian multi-modal database and classified into the following semiotic dimensions:

4.2 Describing shapes of co-verbal expressions

The concept of ‘gesture’ as proposed in [36] describes the shape and propagation of movement in very high detail. In the context of shape, the used annotation schema shown in Figure 2 is pose-oriented, and captures the spatial configurations of key-poses in the form of 3D configurations of the articulators manifesting the shape. This is also very important for the representation of the gestures on humanoid robots. In particular, and as in the case of skeleton-based articulated conversational agents, the humanoid robots can also be driven by a kinematic model [38] (Figure 1). Therefore, similar movement controllers (e.g., joints) can be used to drive the desired movement. The general shape of the movement refers mostly to the shape of the hand, manifested and observed within the communicative expression. As a guideline, and as a base, we defined prototypical (referent) hand-shapes based on the Ham-no-sys [39] and SignPhon [40] databases. The general position of the arm is described in the form of abstract spatial dimensions: the radial-orientation of the elbow (e.g., far-out, side, front, inward, etc.), the height of the palm position (e.g., head, shoulders, chest, abdomen, belt), the distance of the palm from the torso (e.g., fully-extended, normal, close, touching, etc.), and the elbow inclination (e.g., orthogonal, normal, etc.). The annotation also allows for spatial configurations of movement controllers to be stored under each abstract dimension. The shape prototypes, therefore, describe the kinematic model that will manifest them in either the physical or the virtual domain. Since we wish for the physical, virtual and real-life movement to represent the same meaning, the gesture repositories use the notion of similar shapes and identification based on the Hamming-distance [41].

OPEN IN VIEWER

Figure 2.

Form-oriented annotation of communicative sequences as constructed in ANVIL [37] (Slovenian multi-modal database).

In the following section, we will present the proposed behaviour generation algorithm that co-aligns verbal and non-verbal behaviour. The algorithm also performs resynchronization that arises as a result of behaviour adaption due to restrictions provided by physical agents (humanoid robots).

5.1 TTS-driven non-verbal behaviour generation stream

In order to recreate/perform different co-verbal gestures within either virtual or physical interfaces, three tasks must be taken into account. The first task is to select and prepare the visual content. Based on the understanding of the verbal information, this task must identify the intent, the meaningful words (or phrases), and also the suitable gesture prototypes representing the meaningful words. The second task then addresses the content presentation. Based on the identified words and additional prosodic features, this task must identify exactly where the meaningful part will be utilized and what visual cues (if any) will surround it. Finally, and for the third task, by using elimination techniques (e.g., communicative functions, redundant gestures, etc.), the produced density of the visual cues must be reduced and only those visual cues that actually contribute to the meaning of the communication are to be kept. In the proposed system, the first task – also called ‘semantic synchronization’ - is performed in the phase-tagger deque. The second task (pragmatic synchronization) and the third task (temporal synchronization) are performed in the gesture-planner deque (Figure 1).

5.2 Data structures

In the system, a common HRG data structure is used that is accessible by all the modules. In this way, they are all able to access, change, store or enrich the information while performing several algorithm steps on general input text. The HRG data structure is not static and can dynamically change its structure through the queuing mechanism (e.g., non-verbal modules are able to create new relation structures while processing the input sentence) [55]. Relation structures can be in the form of linear lists or else in the form of trees, depending on the type of information that is stored in the HRG structure by the specific module. The HRG structure is ultimately (in the acoustic-processing deque) used for the generation of the speech signal and in the expression-processing deque for the EVA-Script-based behaviour specification. Figure 3 presents the complete structure of the HRG (when all the processing steps of the proposed algorithm are already performed). The relation structures in the form of linear lists are Segment, Syllable, Word, Phrase, IntEvent, SynUnits, ContentUnits and MovementPhases. In addition, those in the form of trees are SyllableStructure, PhraseStructure, IntonationStructure, SynUnitsStructure, CEStructure, CoverbalExpressionStructure and MPDynamicsStructure. For example, SynUnitsStructure relates items in the SynUnits relation layer and items in the Segment relation layer. Next, CEStructure relates items in the MovementPhases relation layer and items in the ContentUnits relation layer. SyllableStructure relates items in the Segment relation layer and items in the Syllable relation layer. As can be seen, it also relates items in the Syllable relation layer and items in the Word relation layer. IntonationStructure relates items in the IntEvent relation layer and items in the Syllable relation layer. MPDynamicsStructure relates items in the MovementPhases relation layer and items in the Syllable relation layer. CoverbalExpressionStructure relates items in the ContentUnits relation layer and items in the Word relation layer. Finally, PhraseStructure relates items in the Phrase relation layer and items in the Word relation layer. TTS-driven Synthetic Behaviour-generation Model for Artificial Bodies www.intechopen.com TTS-driven non-verbal behaviour generation stream consists of phase-tagger deque, gesture-planner deque, and expression-processing deque (Figure 1). Each deque performs several processing steps, as specified by the proposed algorithm, and described in detail in the following subsections.

OPEN IN VIEWER

Figure 3.

A common HRG structure as a storage for verbal and non-verbal data in the algorithm.

5.3 Phase-tagger deque

The decision as to whether a specific text sequence represents a possible gesture trigger is performed based on semiotic grammar, as deduced by the annotation of the conversational corpora [36]. The semiotic rules (needed in order to identify gesture triggers) are based on relations established by the linguistic properties of semiotic gestures, as proposed by Pierce [21]. Some of these properties are language-dependent. However, language dependencies are resolved through relations/rules maintained within the grammar. The algorithm's processing steps performed in the phase-tagger deque are presented in Figure 4. As can be seen, the deque has to perform semiotic tagging, semiotic processing and matching.

The input represents already-POS-tagged word sequences (stored in the HRG structure) that are matched against the semiotic grammar's rules and relations as defined in the gesture dictionary. Firstly, the semiotic tagging step tags those words that could trigger co-verbal movements (mandatory word-type-units). At the morphology level (based on the semiotic grammar's rules), the procedure searches for nouns, verbs, pronouns, fillers and determiners. The mandatory word-type-units and their neighbours are then grouped into word-phrases that are compared against the classification of semiotic dimensions. The classification into a semiotic dimension defines the intent of the phrase, the form of visual representation, and it also suggests the structure of the co-verbal gestures. As shown in Figure 4, the semiotic processing step traverses through the HRG's Word relation layer and, by considering the preceding POS tags and successive POS tags, tries to detect those patterns that match one of the word-type-orders defined by semiotic classification. Finally, during the matching step, the lexical affiliates are chosen for those words identified as carriers of meaning. This is done by performing a semantic lookup in the gesture lexicon and selecting an appropriate physical representation of the word(s). In the external gesture dictionary, a suitable physical shape is selected based on the following rules: a) the shape that has been observed to precede the meaningful manifestation, and b) the shape that has been observed to precede the meaningful manifestation co-aligned with similar semiotic patterns. The properties of the selected co-verbal movement are then stored as additional attributes to the items in the Word relation layer, as seen in Figure 5. Additionally, the matching step creates a new ContentUnits relation layer in order to store generated content units (CUs). The phase-tagger deque may identify multiple semiotic patterns and multiple possibilities for representation. This means that each CU may be described by multiple word items and that each speech phrase may be defined by multiple CUs (e.g., they can be identified by multiple intents). The final selection is performed in gesture-planner deque, when all the prosodic features are already available.

OPEN IN VIEWER

Figure 4.

The processing steps in the phase-tagger deque.

As seen in Figure 5, the information stored within the CUs is also in the form of attribute-name/value specifications, where each attribute has a predefined set of values that subsequent deques can select from. In particular, the set of values for the attribute semi is based on the semiotic types the system can classify (e.g., iconic, adaptor, symbolic, etc.). The attribute mwords holds those key-words that may, in the observed sequence, represent meaning. For example, a word SL: iste (EN: the same). In addition, the attribute pwords indicates those words that the preparation movement phases may be triggered by. For example, a word SL: vedno (EN: always). Word items are enriched at this level with new attributes, describing the symbolical co-verbal alignment and movement-structure (propagation) of shapes of non-verbal expressions with regard to the verbal content. These attributes are: movement phase, rhshape (right-hand-shape), rashape (right-arm-shape), lhshape (left-hand-shape) and lashape (left-arm-shape). These attributes have their values defined based on the selected gesture prototypes available in the external gesture dictionary.

OPEN IN VIEWER

Figure 5.

Assigning co-verbal movements to the text sentence in the phase-tagger deque.

In the Word relation layer, the processed word sequence SL: vedno iste in samo iste obraze (EN: always the same and only the same faces) is tagged as: R, P, C, Q, P and N (adverb, pronoun, conjunction, particle, and noun). The semiotic tagging step recognizes three mandatory word-types: two pronouns and a noun. Next, the semiotic processing step indicates two semiotic patterns: adverb-pronoun and particle-pronoun-noun that may trigger a co-verbal expression. Therefore, two iconic CUs are inserted into the HRG structure. Within both semiotic patterns, the pronoun carries the meaning that might be reflected by the physical manifestation. The pronouns also indicate the position of the stroke movement-phase. Further, the adverb in the first pattern and the particle in the second pattern indicate the preparation movement-phase. Moreover, the noun in the second pattern indicates the hold movement-phase. By considering semiotic relations, implicit rules and lexical affiliation, the matching step gives us the co-verbal movement for each semiotic pattern, which is identified as 1-handed, the lexical affiliates manifested over the word-carrying meaning as K7 (for the hand shape), and Fr|Ab|Cl|O (front, abdomen, close, orthogonal) for the position of the right arm. The phase-tagger deque in this way identifies what non-verbal content can be represented on a specific input sentence and the starting and ending points in the Word relation layer that drive the manifestation of the physical shapes. Within the proposed algorithm, it takes care in identifying those words that carry meaning (or a functional role) and the physical manifestation that the meaning may be represented by.

5.4 Gesture-planner deque

As seen in Figure 1, the phase-tagger deque is followed by the grapheme-to-phoneme conversion deque, which assigns phonetic transcriptions and inserts syllable markers by considering the cross-word contexts and by using phonetic lexica and CART trees [30]. Next, the symbolic prosody deque performs the following three algorithms on the syllable level: the prediction of phrase breaks (position and level), the prediction of prominence labels, and the prediction of Tilt intonation labels, by using CART trees [30] [34]. The phrase break prediction step inserts phrase break labels, the prominence prediction step marks the prominent syllables, and the intonation prediction step assigns symbolic Tilt intonation labels to each syllable, as proposed in [33]. The next acoustic prosody deque then also predicts the duration of segments (phonemes), the duration of pauses at phrase break positions, and the Tilt acoustic parameters for the intonation events assigned by the symbolic prosody module. All these deques generate the prosodic features and temporal information required by the gesture-planner deque. By using the information already stored in the HRG structure for a given input sentence, the gesture-planner deque performs the elimination of multiple possible intents and multiple representations of the intent. The elimination process uses features such as prosodic word phrases, accents and intonation. The gesture-planner deque also performs the temporal alignment of triggers and conversational expressions in the form of movement-phases [22]. Furthermore, it also temporarily co-aligns the consecutive series of conversational expressions into a conversational act, as follows. Firstly, the deque performs the alignment of the propagation of the selected co-verbal gestures and the input-sentence on the syllable level by using information in the following HRG layers: ContentUnits, Phrase, Word and Syllable relations. It also creates a new HRG layer, called ‘MovementPhases’ for describing the relation between the propagation of co-verbal movement and the input sentence on the syllable level, as can be seen in Figure 6. The starting and ending points of the CUs are compared against assigned prosodic word phrases, as predicted by the symbolic prosody deque. In the system, the B3 label is used for labelling major phrase breaks, and the B2 label for minor phrase breaks. The prosodic word phrases indicate those text-sequences that have a complete meaning. In addition, sentences with more than one prosodic word phrase can indicate additional explanation, emphasis, or even the negation of the preceding meaning [30]. Presented in Figure 7 are the processing steps that have to be performed within this deque, according to the proposed algorithm. The first step, called Search for word-phrase breaks, adjusts (extends or even removes) the starting and ending points of the selected CUs. Specifically, it aligns CUs with the predicted prosodic counterparts. In this step, the rule is applied such that each prosodic word phrase can contain only one (or none at all) co-verbal gestures. Further, the movement-propagation of each co-verbal expression must be maintained within the predicted prosodic word phrase. As seen in Figure 6, the sentence SL: vedno iste in samo iste obraze (EN: always the same, and only the same faces) at this point of the algorithm contains two CUs: CU-1 and CU-2. Their starting and ending points are determined by the phase-tagger deque as: SL: vedno iste (CU-1) (EN: always the same), and SL: samo … obraze (CU-2) (EN: and only the same faces). However, and obviously, the predicted prosodic word phrases demand necessary adaptation at this level. As can be seen, the predicted word phrases are SL: vedno iste (WP-1), and SL: in … obraze (WP-2). Clearly, the CU-1 matches the WP-1, whereas the CU-2 has to be extended in order to also include the word SL: in (EN: and). However, if the CUs include more words than the predicted WPs, those words are automatically disregarded.

OPEN IN VIEWER

Figure 6.

The alignment of the propagation of co-verbal expression and the input sentence on the syllable level.

OPEN IN VIEWER

Figure 7.

The processing steps in the gesture-planner deque.

The second step, called Search for emphasized words, identifies the emphasized word, as predicted by the symbolic prosody deque. This is a word that contains a syllable assigned with a PA label (primary accent - the most accentuated syllable within the specific prosodic phrase). The stroke-movement phase is then co-aligned with the word containing the PA syllable. When the CU has no PA syllable (i.e., no emphasized word), the CU is simply disregarded. Those words preceding the emphasized word then represent preparation or pre-stroke-hold movement-phases. Moreover, those words following the emphasized word represent the post-stroke-hold or retraction movement-phases. On all the remaining stressed syllables within the intonation prosodic phrase, the secondary accent (NA) is assigned. The NA labels identify the preparation movement-phase (when the word contains a NA-tagged syllable and precedes the prosodic gesture trigger) or the retraction movement-phase (if this is the last co-verbal expression or if a longer pause is predicted). In addition, when silences (sil) are predicted by the acoustic prosody deque, they indicate the existence of the hold-movement-phases. The third step, called Search for stressed syllables, aligns the stroke-movement phases according to the PA syllable. In this step, the starting and ending points of the stroke phase are determined by the beginning of the emphasized word and the end of the PA syllable. In Figure 6, there are two emphasized words containing a PA syllable. The first syllable ‘is’ in the emphasized word SL: iste (EN: the same) identifies the position of the first meaningful shape defined by CU-1 (e.g., the physical representation of the word SL: iste). Therefore, the first stroke phase will be propagated over the PA syllable. The second syllable ‘mo’ in the emphasized word SL: samo (EN: only) identifies the position of the manifestation of the second meaningful shape defined by CU-2. The fourth step, called Align stroke, aligns the preparation movement-phases. In particular, the CUs already contain the definition of the shape preceding the meaningful shape (e.g., the “initial” physical manifestation from which the body transforms throughout the stroke phase). The fifth step, called Align preparation, aligns the preparation movement-phase. The preparation phase is identified by the first word with the NA syllable that precedes the specific prosodic gesture trigger. The start and end points of the preparation phase are further identified by the position of the NA syllable and the end of the “preparation” word. As seen in Figure 6, the words SL: vedno (EN: always) and SL: in (EN: and) are indicated as preparation words for the first and second prosodic word phrases, respectively. Both words also influence the physical manifestation of shapes within the preparation movement-phases of CU-1 and CU-2. For CU-1, the NA syllable is ‘ve’ and the start and end points of the movement phase are determined by the syllables ‘ve’ and ‘dno’. For CU-2, there is no word preceding the prosodic gesture trigger that would contain the NA syllable. Moreover, and according to the aforementioned rule, this indicates that no preparation movement phase is required. Nevertheless, since CU-1 and CU-2 represent similar meanings (in terms of gesture affiliates), the preparation phase over the word SL: in (EN: and) is artificially inserted into the movement propagation scheme. Finally, the last step, called Align hold/retraction, aligns the hold (both pre- and post-stroke) and retraction movement phases. The retraction movement phase is identified by the last meaningful word-phrase via the word that contains the NA syllable and which precedes the B3 phrase break or else precedes a longer pause. Additionally, the start and end points of the movement-phase are determined by the NA syllable and the end of the “retraction” word. In Figure 6, the starting and ending points are the NA syllables ‘ra’ and ‘ze’ of the word SL: obraze (EN: faces). Moreover, the hold movement phase is determined by the pauses (the predicted sil units, stored in the Syllable relation layer) and the residual (those content syllables not yet used in the preparation, stroke or retraction movement phases) of the meaningful words not yet aligned with the stroke movement-phase. Further, in addition to the sil units, the residual of the word phrase WP-1 is the unused syllable ‘te’, while the residuals of the word phrase WP-2 are the syllables ‘is’, ‘te’ and ‘ob’. As can be seen in the MovementPhases relation layer of the HRG structure in Figure 6, the residual content and the sil units are assigned as hold movement-phases. The movement structure as generated by all six steps is presented in detail in Figure 8. The movement phases such as preparation (P), stroke (S) and retraction (R) indicate a physical manifestation of a shape, and the hold (H) movement phase indicates the maintaining of the closest previous physical manifestation.

OPEN IN VIEWER

Figure 8.

The generated movement structure with the sequence of shapes for conversational expressions.

The conversational expression CE-1 is aligned with the prosodic boundaries (phrase breaks) of the word phrase SL: vedno iste (EN: always the same) marked as WP-1. As seen in Figure 8, during propagating, two shapes will manifest. The shape CU1-P will manifest itself over the syllables ‘ve’ and ‘dno’. Next, the preparation movement phase is followed by a hold (H), indicated by the sil unit. The next stroke shape CU1-S will manifest itself over the syllable ‘is’. The residual syllable ‘te’ then indicates the post-stroke hold phase (H). Obviously, an additional post-stroke hold phase (H) is inserted due to the presence of the predicted sil unit after the word SL: iste (EN: the same). In a similar way, we describe the conversational expression CE-2 aligned with the prosodic boundaries of the word phrase WP-2. The MovementPhases relation layer in this way stores the generated movement structure of the observed sequence (Figure 6). Accordingly, the layer outlines what shape should manifest over specific words, where the movement should be withheld, and where it should be retracted to its neutral (rest) state. However, these steps do not yet specify any temporal boundaries for movement generation. Ultimately, there remain several repeated holds within the movement sequence which should either be filtered or merged. Therefore, in the proposed algorithm the temporal alignment of verbal and co-verbal gestures is also integrated as part of the gesture-planner deque. As presented in Figure 9, these additional processing steps benefit from the temporal properties at the phoneme/viseme level, as predicted by the acoustic prosody deque, and stored within items in the Segments relation layer in the HRG structure. Additionally, the Segments relation layer also stores the temporal information about the predicted pauses (sil units). The temporal information in the Segments relation layer is already used by the TTS modules (verbal stream). However, within the proposed algorithm the duration of each movement phase can also be determined by predicted durations of phonemes/visemes and silences. The temporal processing steps benefit from the following HRG layers: ContentUnits, Segments and MovementPhases. Additionally, new units called ‘phase units’ (PUs) are stored within the MovementPhases relation layer of the HRG structure, while the CUs (stored in the ContentUnits relation layer) are enriched with additional attributes.

OPEN IN VIEWER

Figure 9.

The processing steps for the temporal alignment in the gesture-planner deque.

OPEN IN VIEWER

Figure 10.

Temporal alignment in the HRG structure.

Each sil unit represents a hold movement-phase in the movement structure (MovementPhases relation layer). The first step in Figure 9 is called Filter CEs and searches for those sil units that have a predicted duration of 0 ms, since they are unnecessary and should be removed. In order to filter any corresponding hold-movement phases, the vertical HRG's connections linking each sil unit (in the Segments relation layer) and the corresponding hold movement-phase are deleted, and the corresponding PUs removed from the MovementPhases relation layer. For example, in Figure 10 the sil unit positioned between the syllables ‘dno’ and ‘is’ has a predicted duration of 0 ms. Therefore, the corresponding connected hold movement PU has to be removed from the MovementPhase relation, and new links between the neighbouring preparation and stroke movement-phases are established (dotted links). This step also merges repeated hold-movement phases into a single hold. When this is done, the start and end points of the merged hold are adjusted to the starting point of the first hold and the ending point of the last hold. The second step, called Align CEs, temporally aligns each conversational expression (CE) with the input text. In Figure 10, CE-1 is represented by the co-verbal gesture CU-1. Its attributes tell us that the assigned co-verbal gesture is iconic and that the meaningful shape represents the word SL: iste (EN: the same) (lexical affiliate). Considering the context, the word units contained within CE-1 are the words SL: vedno (EN: always) and SL: iste (EN: the same). Next, the starting and ending point of CE-1 are determined by the syllables ‘ve’ and ‘te’. Considering the vertical link between the syllable and movement-phases relation, we are also able to assign to CE-1 the sequence of the following movement phases: Preparation-Stroke-Hold. In addition, the physical manifestation within the stroke-movement phase represents the word SL: iste (EN: the same). In particular, the physical manifestation within the preparation movement-phase represents the initial shape of the physical representation of the word SL: iste (EN: the same) triggered over those words marked as in the pwords attribute.

The CUs are temporally defined by the following attributes: delay, duration, persistence and duration-down. Moreover, the PUs are temporally defined by the attributes duration-up and persistence. The temporal values of the CUs and PUs are calculated by using the temporal values of their “children” in the HRG structure (phonemes and sil units), as predicted by the acoustic prosody deque. In order to calculate all the attributes' values, Equations (1)–(6) are suggested to be used in this step. The delay attribute represents the absolute duration, measured from the beginning of the input text sequence, for which the co-verbal gesture is withheld prior to execution. When the first conversational expression (CE-1) does not start with the beginning of the input sentence, a default CE-0 is inserted by assigning that duration which is equal to the sum of the segment units preceding the first segment of CE-1. In addition, when CE-1 starts with the first segment in the input sentence, the corresponding delay is always set to 0. For this attribute, the following equation is used:

C E i d e l a y = ( Σ i = 1 i C E i d e l a y + C E i d u r a t i o n ) ,

(eq 1)

where i represents the observed content item. The duration of the last hold-movement phase of each CE represents the persistence attribute's value for the CE. The value is calculated as follows:

C E i p e r s i s t e n t = t ( M P m ) ;

(eq 2)

when the last movement-phase is hold, otherwise its value is zero. The persistence attribute represents the duration for which the shape of the preceding movement-phase is maintained prior to the execution of the next CE. Further, the duration attribute represents the overall duration of the conversational expression. However, it does not contain the duration of the final hold-movement phase. It is calculated as follows:

C E i d u r a t i o n = ( Σ i = 1 n t ( M P i ) − C E i p e r s i s t e n t ,

(eq 3)

where n represents the number of movement phases. The next duration_down attribute represents the duration of the retraction phase, and is calculated as follows:

C E i d u r a t i o n _ d o w n = t ( M P n ) = ( Σ i = k n t ( S i ) ) ;

(eq 4)

where n represents the number of segments and k the first segment in the given CE. The PU stored within the MovementPhases relation layer, in addition to shape manifestation, also stores the temporal values of the preparation/stroke movement-phase and the temporal values of those holds that immediately follow the observed preparation/stroke movement-phase. The corresponding attribute duration_up represents the duration of the preparation/stroke movement-phase - this is how long the transformation between shapes has to last. The attribute's value is calculated as a sum of predicted temporal values on segment units (phonemes/visemes) contained within the specific preparation/stroke movement-phase:

P U i d u r a t i o n _ u p = t ( M P S P ) = ( Σ i = k n t ( S i ) ) ;

(eq 5)

where n represents the number of segments and k represents the first segment in the given PU. Likewise, the attribute persistence is determined by the first hold phase that follows the observed preparation/stroke movement-phase. Its value is calculated as a sum of temporal values for segments contained within the hold phase:

P I i p e r s i s t e n t = t ( M P m ) = ( Σ i = k n t ( S i ) ) ,

(eq 6)

where n represents the number of segments and k the first segment in the given PU. In Figure 11, the generation of conversational expressions is presented in more detail, including the temporal alignment of movement on the level of phonemes/visemes. As seen in Figure 11, the required links can be efficiently and flexibly established between the Segment relation layer, and the MovementPhases relation layer.

OPEN IN VIEWER

Figure 11.

The temporal alignment of movement (on phoneme/viseme level).

In this way, we have available relations between conversational expressions, words, movement-phases, and phase units. The PU fuses the preparation/stroke movement-phases and the hold/retraction movement phases that follow the preparation/stroke. As mentioned before, the exceptions are the hold movement-phases, which are the last in the observed sequence. In Figure 11, the conversational expression CE-1 consists of the phase units PU-1 and PU-2, while the execution of CE-1 is finished by a hold (H) movement-phase. The CE-1 can then be temporally described by equations (1)–(3). Since the CE-1 starts with the first segment in the input text sequence, the delay value is set to zero. Next, the persistence value is determined by the duration of its last hold movement-phase (in this case, the persistence value is 0.068 s). Moreover, the duration value is determined by summing the durations of PU-1 and PU-2. The PUs' durations are determined by the predicted duration of those segments it encapsulates. In the case of CE-1, the duration_up value for PU-1 is determined by summing the durations of t('v'), t('e'), t('d'), t('n') and t('o'). In addition, in the case of PU-2, this is determined by summing the durations of t('i') and t('s'). Since none of the PUs encapsulates the hold movement-phase in this case, the corresponding persistence values are, in both cases, set to zero.

By using equations (1)–(6), any conversational expression and corresponding shape manifestation can be temporally aligned with the speech signal, as synthesized on the general input text. Both processing steps finally generate two types of units that contain all the information necessary for the recreation of the co-verbal gestures. Each CU stores a global temporal structure for corresponding conversational expressions, whereas each PU stores local information regarding the overlaid shapes. In the following section, we discuss the final nonverbal deque that has to transform CUs and corresponding PUs into suitable procedural descriptions of synthetic gestures, as required for the recreation of co-verbal gestures by using synthetic embodied conversational agents and robots.

5.3 Expression-processing deque

ECA-based animation engines generally recreate nonverbal behaviour based on procedural animation description mark-up languages, such as BML [12] and EVA-SCRIPT [18]. These mark-up languages all require at least the temporal specification (e.g., relative position, duration) of behaviour and the description of shape provided in at least abstract notation. These behaviour descriptions are then able to be fed into the animation engine, and can be recreated by a synthetic ECA. In the proposed system, the expression-processing deque is needed in order to transform the HRG data structure into a form understandable to the ECA-EVA animation engine (or into the engine driving the movement of a robotic unit). Therefore, within this deque the HRG structure is transformed into EVA-Script format for the behaviour descriptions, supporting both lip-sync and co-verbal gesture animation processes. Since the HRG structure contains very detailed information about co-verbal gestures, transformation into other mark-up languages is possible and straightforward. In Figure 12, the processing step of this deque is presented. As input, the CUs in the Content relation layer and the PUs in the MovementPhases relation layer are used. The output is a behavioural script, written in EVA-SCRIPT. In order to re-create (animate) the co-verbal gestures as described in the HRG structure, those shape models determined by the PUs' attributes (rhshape, rashape, lhshape and lashape) are selected from the gesture dictionary. The selected prototypes are temporally adjusted according to the temporal specification, stored in the CUs and PUs. The processing step in the deque simply traverses through the CUs, recalls the aligned PUs, and generates the corresponding XML description (EVA-Script). As can be seen (in the EVA-Script), each CU is represented by a bgesture tag. The obligatory attributes for this tag are thus the following attributes: name, type and delay. In the context of recreation, name value is used only for easier access to the selected procedure in the animation graph. The type value is generically set to ‘arm_animation’, since hand and arm gestures are to be animated, as specified by the PUs. The start value defines for how long the animation of the described behaviour has to be withheld prior to its execution. It is equivalent to the delay value stored in the CU. The durationDown value, stored in the CUs as duration-down, describes the duration of the controller's retraction (e.g., its transition from an excited to a neutral (or previous) state).

Each PU in the HRG structure then represents a sequence of body-parts moving in a parallel manner. The configuration of body-parts is maintained within the UNIT tags. The value of the attribute name represents a gesture prototype stored within the gesture dictionary. The temporal specification for each UNIT is provided by specifying the following two obligatory EVA-Script attributes. The durationUp value, stored in the PU as duration_up, represents the absolute temporal value and describes how long it takes for the specified shape to be fully overlaid on the synthetic agent (e.g., the duration of the transition between shapes). Meanwhile, the value, stored in the PU as the persistence value determines persistence for the period after the transition between sequential shapes is completed and in which the overlaid shape (the excited state) has to be maintained. Additionally, this deque also forms a description of facial expressions required for the-lip-sync process, where the visual sequence of visemes/phonemes is described in the form of viseme in EVA-Script's tags, encapsulated by using the speech tag. The corresponding information is stored in the Segment relation layer. In this layer, each segment (including sil) represents one viseme tag within a lip-sync specification, and at the same time a gestural affiliate for a facial expression which, in the mouth region, overlays the pronunciation of the segment. Additionally, the duration of each segment is also available and is equivalent to the duration attribute of the EVA-Script's viseme tag. The appropriate transition between segments is handled internally by the EVA animation engine, as proposed in [18].

OPEN IN VIEWER

Figure 12.

Transformation of the HRG structure into an EVA-script behaviour description.

The expression-processing deque also integrates an optional capabilities-conversion processing step. It ensures that the behaviour specification also becomes compatible with the target robotic platform (in our case, iCub [19]). As already mentioned, there are several similarities between ECA-EVA-based movement-control mechanisms (virtual specification) and iCub movement specifications (physical specification). The similarities range from the movement controllers used to the techniques used to simulate (recreate) the specified movement. In particular, both the concept of ECAs (based on the EVA-Framework [18]) and the concept of robots specify movement controllers as the joints (bones) controlled by the HPR (H-heading, P-pitch and R-roll) mechanism. Although the physical systems mostly use the RPY (P-pitch, R-roll and Y-aw) mechanism, the analogy in terms of 3D space exists. In particular, e.g., the “yaw” is analogous to the “heading” attribute. These joints are a part of skeleton structure approximating the human skeleton structure (a concept of forward-kinematics). A minor difference between the physical and virtual model may be observed regarding the HPR configuration. The virtual model specifies the HPR configuration as a single controller, whereas the physical model configures the HPR values by using three movement controllers (one for each dimension in the HPR space). For instance, the left shoulder joint of the virtual model is represented by the following joints: 0 (shoulder pitch – P), 1 (shoulder roll – R), and 3 (shoulder yaw – Y/H). The dynamics of movement are then defined by the time it takes to perform the transition. The same principles are also shared by the EVA-Script. The behaviour is specified in a context of forward-keying. Major parts of movement are defined as sequential movement phases (sequences). Each such sequence also describes the HPR configuration of the particular joints that are responsible for the formation of shape. In addition to shape and movement articulators, the major disadvantages of robot movement are the dynamics and fluidity of the movement, as well as the gradualness of its trajectory. The dynamics are addressed mostly by the relation between the temporal distribution of the co-verbal expression (movement-phases) and the spatial configuration of the shapes. Virtual movement can be specified as almost instantaneous, whereas the movement controllers of the robotic unit require longer temporal periods. The responsiveness of the servo-motors (the robot's physical movement controllers) is defined by the motors' maximum angular velocities and by their maximum angular acceleration. These values are generally specified by the manufacturers of the servomotors [48]. Since the stroke-phase is the most important phase (the co-verbal shapes carry meaning in the stroke phase), it can, if required, gain additional temporal space. The temporal-adjustment-process shortens (or even eliminates) those movement-phases that surround the stroke-phase. The process takes into account the responsiveness of the robot's physical movement controllers. The surrounding movement-phases are shortened to the minimal requirements for the stroke-phase to adjust to the movement controllers specifications. For instance, the stroke-phase (Figure 13) is extended over the preparation-phase in order to include the whole duration of the trigger word. In order to temporally adjust the movement within the stroke-phase, the minimal time required to perform the physical angular adjustment is calculated and compared against the duration of the stroke. The stroke-phase is then (if necessary) extended into the preparation and/or hold/retraction movement phases. The temporal remainder of those optional phases is then compared again against the responsiveness of the robot. Those phases that cannot be performed in the suggested time are automatically disregarded.

OPEN IN VIEWER

Figure 13.

Recreating communicative acts by a robot and an ECA-EVA.