Research on discourse role recognition in task-oriented collaborative dialogue

2.1 Dialogue mode

The dialogue mode was first proposed in the field of education. In general, informal learning cannot be thoroughly compared to collaborative learning in an educational environment. However, there are similarities in how meaning is formed and recognized, especially when people engage in task-oriented collaborative dialogue. Mercer et al. [1] considered how speakers are currently talking and thinking and proposed three modes of social thinking: argumentative dialogue, cumulative dialogue, and exploratory dialogue. Ferguson et al. [7] explored a method for detecting exploratory dialogue in online synchronous text chat. They manually identified a list of cue phrases indicating the presence of exploratory dialogue. Ferguson et al. [8] used exploratory dialogue to indicate ongoing learning. They developed a self-training framework based on previous work that uses discourse and topic features for classification by integrating cue phrase matching and k-nearest neighbor classification. Ekman et al. [9] developed a framework for encoding exploratory dialogue based on Mercer’s research, including five categories: challenge, evaluation, extension, reasoning, providing information, and community work. The exploratory elements of the whole dialogue were analyzed through qualitative content analysis, focusing on how people use language to think together when solving problems. Based on a coding framework for dialogue modes, this paper defines categories of discourse roles. The focus is on discourse’s role in collaborative dialogues for goal achievement to analyze the participant’s contributions to the dialogue. In addition, discourse roles can reflect changes in the way information is conveyed and the way speakers interact with each other in a dialogue, thus contributing to the understanding of the conversation.

2.2 Speaker role

Most of the existing work enhances the ability of language models to understand dialogues mainly by encoding speaker roles. Chi et al. [4] proposed a role-based contextual model that can independently consider the roles of different speakers based on various speaking patterns in multi-person dialogues. Similarly, Chen et al. [10] proposed an attention-based network that utilizes temporal information and speaker roles to improve spoken comprehension. Zhu et al. [11] trained a vector for each speaker role and represented it as a fixed-length append to the embedding of speaker turns to improve the performance of the summary model. Song et al. [12] Used a single-layer neural network to convert speaker roles into vector representations. They attached them to a word encoder, which helps to recognize patient and doctor discourse better. Unlike methods based on speaker roles, the method proposed in this paper focuses on speaker turn and is independent of the speaker’s identity information. Thus, it is still valid when the speakers in a dialogue do not have formal roles. This allows the model to better adapt to different types of dialogue scenarios.

2.3 Dialogue act

Dialogue Act (DA) is a semantic label associated with each discourse in a dialogue that indicates the speaker’s intention, e.g., question, backchannel, stated opinion, etc. Recently, many studies have approached DA classification as a sequence labeling task to fully use contextual information between discourses. Chen et al. [13] posed the problem of Dialogue Act recognition in terms of capturing hierarchically rich discourse representations and generalized richer structural dependencies of the CRF attentional graphs without giving up end-to-end training. Li et al. [14] proposed a dual-attention hierarchical recurrent neural network for DA classification. Their model can capture information about DAs and topics in response to discourse and information about their interactions. Raheja and Tetreault [15] utilize a context-aware self-attention mechanism and a hierarchical RNN. Shang et al. [16] use a BiLSTM-CRF architecture that considers speaker information for a DA classification task. Colombo et al. [17] introduced a seq2seq model customized for DA classification using a hierarchical encoder, a novel guided attention mechanism, and beam search applied to training and inference. Malhotra et al. [18] proposed a Transformer-based architecture with novel speaker- and time-aware contextual learning for classifying dialogue behavior in mental health counseling dialogues. Similar to the above work, this paper treats the discourse role categorization task as a sequence labeling task. The difference is that the above approaches assist in discourse category identification by proposing more complex and specialized models to model speaker interaction or conversation topic information, which inevitably introduces many parameters for training. In this paper, speaker turn information and dialogue topic information are encoded through two additional embedding layers. This approach requires only negligible modifications to the recurrent model and introduces O(1) space complexity.

3.1 Data collection

Since the object of study in this paper is task-oriented collaborative dialogue, a large corpus of dialogues about meetings is consulted to construct a Discourse Role dataset. Ultimately, the MRDA corpus [19] is chosen as the original dialogue data. This is because MRDA is a multi-party, task-oriented, collaborative conversation corpus with a large number of discussions on a wide range of topics. At the same time, the dataset is tagged with dialogue act, which can be used as auxiliary information for annotating the Discourse Role labels. The MRDA dataset is constructed by annotating the dialogue act on the ICSI Conference Corpus [20]. The ICSI Conference Corpus is captured from real face-to-face conferences covering various specialized areas, such as computer science, linguistics, and so on. This gives authenticity and diversity to the conversations in the corpus. It consists of 75 meetings of approximately one hour each. There are 53 unique speakers in the corpus, with an average of about six speakers per session. In ICSI, there are, on average, 10,189 words and 464 turns in the conference transcripts. Since the ASR system generates the transcripts, the textual error rate of ICSI is 37% respectively [21]. This paper selects eight meetings to be annotated with Discourse Role labels.

3.2 Dataset annotation scheme

Seven discourse role labels were designed by analyzing data from real collaborative dialogues. These labels aim to describe the different roles of discourse for goal achievement. The specific descriptions are as follows:

Initiating New Topics (INT): This label is used for discourse that introduces a new topic related to the task in a dialogue. It serves as a guide to the dialogue.

3.3 Annotation process

The ICSI meeting corpus contains dialogue summaries, topic descriptions, and dialogue act information. Therefore, to improve the annotation quality of the dataset, before annotation, the annotator reads the summaries of the annotated dialogues to understand the content of the dialogues. During the annotation process, the Discourse Role annotation work will be done with the help of conversation topic descriptions and conversation act labels. The annotators are five graduate students from the computer technology major.

At first, the basic unit of Discourse Role annotation was set as turn (i.e., the speaker did not change the words spoken). However, during the annotation process, it was found that some conference speakers spoke too long, and their Discourse Roles changed during the long speech. For example, in the early part of the speech, the Discourse Role is Interpretive Inference(II), and in the later part, the Discourse Role is changed to Seeking Clarification(SC). Therefore, the annotator segments the speaker’s speech according to the semantics of the discourse and takes the speaker’s complete expression as the annotation unit.

To ensure an understanding of the task and the annotation scheme, the dataset was sampled, and each annotator was asked to annotate it according to a prepared set of guidelines. After this, all annotators engaged in a discussion to ensure consistency. After several rounds of annotation and discussion, the entire dataset was available for annotation. After the data annotation, a Kappa score for MRDR was calculated to measure inter-annotator agreement. An inter-annotator consistency score of 0.643 was obtained.

3.4 Data pre-processing

To ensure the dataset’s quality and eliminate interfering factors in the discourse role identification task, the following processing measures were taken: (1) Discourses that were not fully transcribed in the dialogue were deleted. (2) All the discourses of one complete formulation of each speaker were concatenated as the unit of the sample, i.e., discourse. (3) Unique numerical codes were assigned to each discourse role in increasing order, and then unique discourse role labels, i.e., DR, were assigned to each discourse. (4) To facilitate the chunking of dialogues, each meeting is considered a complete dialogue segment, and the dialogues are coded with a numerical serial number in increasing order, i.e., dialogue_id. The dataset is processed accurately and orderly through these steps, providing a reliable basis for subsequent analysis and modeling.

3.5 Statistics

In total, there are 11064 discourses in MRDR. A total of 6021 data were annotated with one complete expression of the speaker as the annotation unit. Compared to other dialogues, the dialogues in MRDR are usually long, with an average dialogue length of about 1500 times. Since we used one complete expression of the speaker as the annotation unit, the average dialogue length was about 660 times, and the average length of the discourse was 15 words.

Table 2 shows the distribution of Discourse Role labels in the MRDR dataset. It can be observed that Interpretive Inference (II) occupies the most significant proportion, followed by Passively Accepting (PA). In contrast, Non-Exploration (NE) had the smallest percentage. This distribution pattern may be because the process of interpretation and inference is crucial for problem-solving and consensus-building in task-oriented dialogue. At the same time, acceptance of others’ viewpoints and information helps to establish an effective communication and cooperative atmosphere. Non-Exploration (NE) is relatively less frequent because Non-Exploratory discussions may reduce the engagement and depth of the whole dialogue. It is worth noting that the distribution pattern of Discourse Roles included in the MRDR dataset is uneven, with specific Discourse Roles being used more frequently than others in task-oriented collaborative conversations. This distribution pattern reflects the natural variation in participants’ use of the Discourse Role in collaborative dialog. At the same time, the skewed nature of this data presents a challenge for Discourse Role identification.

Given corpus D = { ( C n , Y n , T n , S n ) } n = 1 N consists of N dialogues, where C n = 〈 u l n 〉 l = 1 L is a dialogue fragment containing L discourse sequences, and T n = 〈 t b n 〉 b = 1 B are a topic labels containing B dialogue topics, the S n = 〈 s l n 〉 l = 1 L are the speaker turn labels, and Y n = 〈 y l n 〉 l = 1 L are the corresponding Discourse Role labels. Given a dialogue C _k and its corresponding speaker turn labels S _k and topic labels T _k , the goal of the STTATM model is to be able to assign the correct Discourse Role label Y _k to each discourse in the dialogue C _k .

4.2 Hierarchical dialogue encoder

This paper uses a hierarchical structure to model the relationships between discourses in a dialogue. Each discourse is first encoded independently, and then the encoded discourse sequence is fed into a recurrent neural network to generate a context-aware discourse representation.

Finally, using a linear layer to predict the Discourse Role y _k of each discourse.

y k = W e h i + b e(5) where W ^e  ∈ R^E×H and b ^e  ∈ R ^E are trainable parameters and E is the number of Discourse Role categories.

This layer is optimized using cross-entropy loss.

L cls = - 1 ∑ i = 1 C K i ∑ i = 1 C ∑ j = 1 K i log P i , j [ y i , j ](6) where C is the number of dialogues. K _i is the number of discourses in dialogue i. P_i,j is the probability distribution of Discourse Role labels for discourse j in dialogue i. y_i,j is the true Discourse Role label.

5.1 Implementation details

This paper implements the proposed model using the PyTorch framework, and the classification cross-entropy loss is optimized using the Adam optimizer. The complete list of hyperparameters is given in Table 3.

Since the dialogues in MRDR are much longer (up to 1053 times), dialogues are partitioned into fixed-length dialogue blocks to evade overflowing memory during training. As shown in Fig. 2, a dialogue of length 10 is sliced into three dialogue blocks of length 4, where each block denotes a data spot. The chunking operation is not required for validation or testing, as maintaining the computational graph during training consumes more GPU memory.

OPEN IN VIEWER

Fig. 2

Example of dialogue blocking.

Keeping the other hyperparameters constant, Table 4 shows the results using different block sizes on MRDR. As the block size increases from small values, the performance improves, and the RNN can utilize more contextual information. Nevertheless, once a threshold is exceeded, a further increase in the block size leads to a performance reduction. In this case, the RNN suffers from gradient disappearance and explosion problems and forgets about long-term dependencies. Therefore, it can be argued that partitioning long dialogue into smaller blocks as input improves performance in the Discourse Role classification task. Therefore, the block_size parameter is set to 16.

TextRNN: TextRNN is a Recurrent Neural Network (RNN) based text classification model that models variable-length sequences. TextRNN transforms each word of a text into a vector and then inputs all word vectors into the RNN for sequence modeling.

TextRNN_Att [26]: TextRNN_Att is a model that adds an attention mechanism to TextRNN. At each time step, TextRNN_Att obtains the weight of each word by calculating the inner product of each word vector with a weight vector and then sums all word vectors weighted to obtain the output vector for that time step.

Transformers [23]: Transformers is a model based on a self-attentive mechanism. It learns the relationship between different positions in a sequence by performing self-attentive computation on all positions of the input sequence.

FastText [27]: FastText is a text classification model based on the bag-of-words model, which obtains a bag-of-words representation of text by performing word separation and n-gram operations on the text. Then, FastText performs classification using a hierarchical Softmax method to map the text into predefined categories.

TextCNN [28]: TextCNN is a convolutional neural network (CNN)_based text classification model that uses convolutional kernels of different sizes to capture information of different lengths in the text and integrates all the features at the end through a pooling layer to finally output the classification results.

TextRCNN [29]: TextRCNN (Recurrent Convolutional Neural Network) is a deep learning model specifically for text classification tasks. It combines the advantages of RNN and CNN to capture both time series information and spatial information in text data.

Bi_LSTM_CRF [30]: The method constructs a hierarchical bi-directional LSTM as the basic unit and a conditional random field as the top layer to accomplish the task of dialogue act recognition.

LSTM_Softmax [31]: The method applies a deep LSTM structure to classify dialogue acts through softmax operations. The authors claim that word embedding, dropout, weight decay, and several LSTM layers significantly impact the final performance.

CASA [15]: This is a context-aware attention-based system for classifying dialogue act. It uses RNNs at the dialogue and discourse levels and computes context-aware self-attention before final classification.

RoBERTa [6]: We use RoBERTa as a baseline in this work due to its superiority in various benchmark tests. RoBERTa is similar to BERT. it is an encoder-only language model trained in an unsupervised manner on a large amount of unlabeled data using masked language modeling targets.

5.3 Experimental results

In the experiments, the dataset was divided into the ratio of 70:20:10 as training, validation, and test sets, respectively. To measure the performance of the STTATM model and other baseline models, the experiments in this paper use Precision(P), Recall(R), F1-measure(F1), and Accuracy(ACC) as the evaluation metrics of model performance. The speaker turn embedding, and topic embedding proposed in this paper can be used in other embedding-based discourse classification methods. Since these two embeddings are not used in the code of the baseline model, the proposed speaker turn embedding, and dialog topic embedding are not implemented on the baseline. To effectively validate the performance of the STTATM model, this paper uses speaker turn and topic embeddings on the RoBERTa model.

Table 5 shows the results of the comparison experiments. It can be observed that the STTATM model achieved state-of-the-art results on MRDR with an accuracy of 86.99%. Except for the RoBERTa model, none of the other baseline models reached 80% accuracy. The possible reason for this gap is that RoBERTa learns rich contextual information on large-scale textual data through pre-training and can capture the rich semantic representation of discourse. Second, RoBERTa performs well on a small amount of labeled data relative to traditional models because it can migrate pre-trained knowledge. Meanwhile, RoBERTa model’s performance improves significantly with the addition of speaker turn and dialogue topic information, demonstrating the effectiveness of modeling speaker turn and dialogue topic in discourse role recognition task. It is worth noting that the accuracy of the RoBERTa speaker+topic model is still lower than that of the STTATM model. Discourse role recognition relies on remote contextual information.RoBERTa is a Transformer-based model that uses a self-attention mechanism that captures semantic relationships between words. Although it can consider a certain amount of context during pre-training, its self-attention mechanism may be limited when dealing with long sequences. In contrast, BiLSTM is a recurrent neural network that processes sequences in a time-step-by-time-step manner, gradually accumulating and transferring information in the sequences, and thus has advantages in dealing with long-distance dependencies. Thus, the model can understand and model the context at different levels by combining the two, improving the overall understanding of the sequence.

Table 6 shows the performance of the STTATM model in terms of discourse role labels. It can be observed that II, E, SC, PA, and NE are better recognized with accuracies of 89.30%, 71.74%, 89.77%, 93.47%, and 92.86%, respectively. These discourse roles appear more frequently in the dataset, their discourse features are apparent, and the model can capture their discourse features effectively. However, the accuracy rates of PI and INT are 51.52% and 48.94%, respectively. These two types of discourse roles appear less frequently in the dataset, and their discourse forms are diverse, which may lead to difficulties for the model to learn their discourse features thoroughly. In summary, the STTATM model performs well in most discourse roles. However, there is still space to improve the performance of the model for the rare roles with less distinctive discourse features, and more training data or more complex model structures are needed to improve the recognition of these discourse roles.

Figure 3 reports the confusion matrix of STTATM. Two pairs with significant error rates (≥ 40%) can be observed - INT: II (44.7%) and PI: II (42.4%). The STTATM model tends to predict INT and PI as II incorrectly. This error can be attributed to the diversity of discourse, i.e., INT and PI are often presented in different discourse forms. For example, "I think we should......" and "because someone already has...... For example, "I think we should " and "because someone already has " belong to INT and PI, respectively, but they are easily confused with II because of the presence of "I think" and "because," which are keywords of II’s discourse role. Another possible reason is that II is a more significant part of the dataset, and the model may be more inclined to predict the more frequent categories and less effective in predicting the rare categories. For the remaining cases, the error rate can be considered nominal. Therefore, this paper clarifies that STTATM can be further improved with a more balanced dataset.

OPEN IN VIEWER

Fig. 3

Confusion matrix for STTATM.

Ablation experiments are conducted to verify the effectiveness of speaker turn embedding and dialogue topic embedding in the STTATM model. The experimental results are shown in Table 7. “¬speaker” indicates that no speaker turn embedding is added; “¬topic” indicates that no topic embedding is added. It can be observed that there is a significant improvement in the performance of the STTATM model compared to the STTATM_{¬speaker +¬topic} model, indicating the effectiveness of modeling speaker turn and dialogue topic in the discourse role classification task. In multi-round dialogues, discourse roles usually change dynamically, and speaker turn embedding can help the model capture this dynamism. In addition, there is a correlation between dialogue topics and discourse roles, and dialogue topic embedding can help the model capture this correlation. Meanwhile, it is noted that the STTATM_¬speaker model outperforms the STTATM_¬topic model in all metrics. Therefore, it can be concluded that dialogue topic information is more effective for discourse role modeling than speaker turn information. Dialogue topics provide a broader context that helps the model understand the overall context of the current dialogue. Although speaker turn information is essential for understanding the alternation between speakers, it provides only local information that limits the model’s understanding of the entire dialogue. Therefore, the speaker turn provides limited auxiliary information.