Argumentative relation identification in academic texts

Abstract

The argumentation in academic writings is necessary to clearly communicate the ideas of the students. The relations between argumentative components are an essential part since this shows the contrast or support of the presented ideas. In this paper, we present two approaches to relation identification between pairs of components. In the first, we detect initially which components are related, to later classify them in support or attack relation. In the second approach, we identify directly which components have a support relation. For these approaches, we employed machine learning techniques with representations of several lexical, syntactic, semantic, structural and indicator features. Experiments in argumentative sections of academic theses showed that the models achieve encouraging results solving the task, and revealing the argumentative structures prevailing in student writings.

Keywords

Argument component relation argument mining academic writing argumentation studies annotated theses corpus

1 Introduction

Argumentation in academic writings is necessary to clearly communicate the ideas of students. This is a critical skill in academic writing. This ability consists of formulating an argumentation structure that provides numerous arguments to support each claim presented in academic texts. An argument is a set of statements (i.e. premises) that individually or collectively provide mainly support, but some can oppose, to a claim (conclusion). The identification of these relations (support or attack) is an essential part of argument analysis. This task is part of argument mining, which aims to identify argumentative relations of types such as support and attack between arguments in natural language, by classifying pairs of segments of text as an attack, support or neither [15]. The automatic analysis of arguments in theses becomes a necessity in the academic field, to facilitate the analysis of long texts and shorten the times of revision. In previous studies, we approached the task of classification of argumentative components [8] and assessment at the paragraph level [9]. In this paper, we focus on a fine-grain analysis at argumentative relation level.

In recent years, researchers have studied the task of automatic processing of arguments in a variety of fields, for instance in Law, where the aim is to facilitate access to jurisprudence that supports a case [19 , 31]. In political discourses, automatic prediction of relations has been studied for classification of pairs of arguments as support or attack relations [17]. In scientific articles, particularly in biomedicine, automatic processing of arguments can be applied to more rapidly identify arguments for or against a hypothesis under investigation [11]. In social networks, the analysis of an argument is also employed to identify the overall stance (for or against) of comments in a debate [2]. There are also studies oriented to identify support relations between components in persuasive essays [25]. Still, there remains an absence of studies for larger and complex academic works such as academic theses, in particular in the Spanish language.

In this paper, we present models for identification of argumentative relation using machine learning techniques with representations of several lexical, syntactic, semantic, structural and indicator features. We propose two approaches for the identification of the argumentative relations between a pair of components. In the first, we present a model to detect initially which components are related, that is, if the pair of components has an attack or support relation, then afterwards we move to develop a model to classify the type of relation. In the second approach, we identify directly which components have a support relation, since this type is prevalent in students arguments. To evaluate our model, we created a corpus of thesis sections (problem statement, justification, conclusions) with annotated argumentative components and relations.

The paper is structured as follows. In Section 2, we discuss related work for argument relation identification. In Section 3, we present a theoretical background for argumentation structures. Section 4 details the corpus used in the experiment. The proposed features and learning approach are described in Section 5. In Section 6, we report the results of the efficacy of our models for argument relation identification. Finally, we conclude with final remarks and work in progress.

2 Related work

Recently several researchers have addressed the task of argument relation identification. After identifying the types of argumentative components and their relations, the argumentative structure can be elucidated. In this section, we present previous related research.

In the case of the work of [19, 20] for legal texts, the use of a Context-Free Grammar (CFG) allows to obtain a tree expressing the argument structure. The CFGs are not only employed to identify relations but also to classify premises and conclusions in legal texts at sentence level, as detailed in [19].

Moreover, using computational learning to identify relations between components, in [25] the authors tackled the identification of pairs of components with a supporting relation in persuasive essays. To create the training set, all possible pairs between components were assembled, later indicating if there exists a support or non-support relation, obtaining 989 (15.6%) support pairs and 5,341 (84.4%) pairs of non-support. The attributes used to represent the pairs were of four types: structural, including the number of tokens and the number of punctuation marks; lexical attributes such as combinations of word pairs, first words, and modal words; syntactic expressed in terms of production rules; and indicators such as argumentative markers with binary characteristics of appearance, and prediction of the type of components (whether is a premise, conclusion or major conclusion). Subsequently, using a support vector machine algorithm, the authors reported an accuracy of 86.3% and an F1 measurement of 0.519 for the identification of pairs of components with a support relation. In [23], using the same features, relations of attack and support between argument components were identified. They report an F measure of 0.388 using a maximum entropy classifier with the implementation in MALLET (Machine Learning for Language Toolkit).

A similar approach is observed in the work of [4], which also applies computational learning with a set of features to represent pairs of sentences. The authors used Bag of Words and attributes that relate two sentences such as similarity based on WordNet 1 , as well as distance and textual entailment measures. For sentiment features, the authors considered lists of words, discursive markers (for instance, SentiWordNet 2 tool). Employing the Random Forest classifier, the authors identified relations between two sentences as support, attack or none, and reported an accuracy of 77.5% using a corpus with 854 pairs of messages coming from social media. However, the authors do not report a level of agreement for the annotation study, as it was still ongoing.

In [14], a method to identify relations was proposed with semantic similarity between sequential propositions of the same paragraph to determine the presence of a connection (support/conflict). To obtain the similarity between two propositions, WordNet was used to determine the distance of each word of the first proposition with each word of the second proposition. This similarity is inversely proportional to the number of nodes along the shortest path between the synsets of those words. The results in precision were 0.82, with a recall of 0.56, and an F1 measure of 0.67 for the identification of a connection between two propositions. The data set utilized is based on the Araucaria corpus [5]. They extracted only 78 complete arguments, with a total of 404 propositions. The types of argumentation schemes considered were expert opinion and positive consequence.

Another approach to identify the relations between arguments is by textual entailment. An analysis was done on formal texts (debates) of Debatepedia 3 , where 100 pairs were selected to train the textual implication tool EDITS 4 and 100 pairs extracted for test purpose [2]. The implication between 2 sentences was measured and if they found an entailment, it was classified as support; otherwise, it was labeled as an attack. The authors reported a 75% accuracy for this classification task.

The prediction of relations in political discourses in monological form where there is no direct interaction between opponents is analyzed in [17]. A corpus was created, based on the transcription of speeches and official statements issued by US Presidents Nixon and Kennedy, during the 1960 presidential campaign. The pairs of arguments were annotated as support or attack relations. The analysis is developed in two steps. The lexical overlap between fragments, the position of the mention of the subject in the fragment, and the cosine similarity between the fragments are the features considered in the first step. The second step employs features such as the lexical overlap, the number of words with negation, the thematic keywords represented with the embedding of word2vec [18], textual entailment between fragments, and a sentiment score. Both steps apply an SVM classifier with a radial kernel from the LIBSVM implementation. In step one, authors achieved an F1 measure of 0.65 for the classification of related and unrelated pairs, and the score for classification of attack and support was an F1 measure of 0.77. The authors indicate that they achieved a better result by performing the classification in sequence (steps one and two) than by performing a multi-class classification (i.e. with the three classes: support, attack and unrelated).

A deep learning architecture is employed in [6] to identify relations of types such as support, attack or none, in texts of movies, technology, and politics. The corpus has a proportion of 31% attack relations, 32% support relations and 37% unrelated. The authors proposed an architecture which receives as input the words of the two text segments to be encoded with Glove [22] to be later processed by two parallel LSTM recurrent networks which generate a representation for the two texts separately. The representations obtained by the LSTM networks are concatenated to be processed by a softmax classifier. With the proposed model, an F1 measure of 0.89 is reported for the identification of support, attack or unrelated relations.

3 Argumentative structures

An argument consists of several components. These components are a set of assertions, that individually or as a whole support another statement [3]. The assertion supported is a conclusion (claim). The conclusion is the principal component of an argument. There is only one conclusion for each argument; however, each conclusion can be based on a series of contrasting or supporting assertions. Assertions that provide support (or attack) are called premises. A premise is a reason provided by the writer to convince the reader of the conclusion. These components are linked with support or attack relations to model the structure of the argument. Among argumentation theories [7 , 29], the consensus is that the structure of an argument consists of several argumentative components. In this work, we adhere to the conclusion-premise model presented by Freeman [7] since introduces structures employed in argument mining [12].

A graphical representation of an argument structure facilitates the understanding of how argument components are associated. Argument diagramming is the area that supports students in formulating their arguments. Each premise and conclusion is associated to a letter that is further expressed as a node of a graph. Then, directed edges (arrows) are established between nodes to indicate relations between the components. A simple argument has only one premise that is used to support one conclusion [28].

[Today educational institutions have a greater number of computers with Internet.]/P1 [Therefore, more students have access to Internet.]/C2

As we can observe in example (1), the first sentence is a premise (in square brackets /P1) supporting a conclusion in the second sentence (in square brackets /C2). In a simple argument, a premise provides elements to sustain the veracity of the associated conclusion. Figure 1 illustrates a simple argument structure in which the premise P1 supports the conclusion C2 (on the right side).

Fig. 1

A simple argument structure as graph: A premise with a support relation to a conclusion.

As we can notice in the argument example, the word “therefore” plays an essential role in the identification of a possible conclusion. These patterns of words are called argumentative markers and can help with the detection of elements in an argument.

As mentioned previously, arguments can have more than one premise as support. In Figure 2, four types of structures for arguments are depicted. A convergent structure (a) has several supporting premises. In this case, either one of the two premises can be eliminated and the conclusion is still supported [28]. A linked structure (b) has two premises which together are used to provide support for a conclusion. A divergent structure (c) has a unique premise to support several conclusions. A serial structure (d) has arguments deployed in successive order. In this type of structure, the conclusion of one component acts as a premise for another element [7]. These simple structures can be joined to form a more complex graph.

Fig. 2

Types of argument structures.

4 Corpus of argumentative relations

The corpus created to identify the argumentative aspects in undergraduate and graduate academic writings comes from the Coltypi collection [10]. This collection has 468 theses and research proposals from the area of information technology and computing, written in Spanish. The texts come from undergraduate (college and higher technical studies) and graduate (master and doctorate) levels. In particular, our study focuses on the sections of Problem statement, Justification and Conclusions, since these are considered highly argumentative [16].

The corpus was built with 444 sections with at least two annotations per section, after following an annotation guide. The team consisted of seven annotators from fields related to linguistics who worked in different subsets of the corpus. The results of the annotation process are provided below. The analysis of the level of agreement for the argumentative annotations was done considering all possible combinations of pairs of argumentative components per paragraph. Therefore, the paragraphs taken into account have at least one conclusion and one premise. We also considered the direction of the relation, from a source component to a target component, for example, in a given relation, a premise that supports a conclusion and vice versa. We found 663 paragraphs fulfilling these criteria, with a total of 6,370 pairs of relations of support, attack and unrelated types. The level of agreement computed for the annotations of support, attack and unrelated relations was a kappa of 0.569, corresponding to a “moderate” level [13]. The level of agreement for the support relation (with support and without support) was kappa of 0.574. For the attack relation (with attack and without attack), we got a kappa measure of 0.538. We also calculated the level of agreement taking into account the annotation of relations, grouping the attack and support relations (i.e. with relation and without relation), we obtained a kappa measure of 0.568. In Table 1 we present the agreement levels for the annotation of the relations between components, and for all configurations the agreement corresponds to a “moderate” level.

Table 1
Agreement level of annotators for argumentative relations

Annotations % K (kappa)

support, attack, w/ relation, w/o relation 85.5 0.569

w/ support, w/o support 87 0.574

w/ attack, w/o attack 98.2 0.538

w/ relation, w/o relation 85.8 0.568

Annotations	%	K (kappa)
support, attack, w/ relation, w/o relation	85.5	0.569
w/ support, w/o support	87	0.574
w/ attack, w/o attack	98.2	0.538
w/ relation, w/o relation	85.8	0.568

Only matching relations between the two annotators were selected to create the final corpus used in the experiments. A total of 5,531 pairs were obtained, of which 15.5% are support relations; 1.4% attack relations; and 83.1% unrelated pairs. The final corpus is composed of 854 support relations and 79 attack relations, we can notice that most relations are of the support type. Support relations are established, mainly, between a premise and a conclusion; for this particular combination of components, 788 relations were found. In addition, we encountered 47 support relations between premises (P→P) and 19 between conclusions (C→C). Attack relations are established, mainly, from a premise to a conclusion; there are a total of 39 attack relations with this combination. Also, there are several attack relations between premises (P→P), with a total of 36 and, to a lesser extent, there are three attack relations from a conclusion to a premise (C→P) and only an attack relation between conclusions (C→C).

5 Argumentative relation identification

We consider the identification of argumentative relations between argumentative components (one component as a source to another as a target) as a task of classification of text pairs as support, attack or unrelated. We explored two approaches for the task, first to classify the existence of argumentative relations as a binary classification to determine if a pair of components are related or unrelated. Afterwards, a classification of attack and support relations is done, focusing mainly on the identification of attack relations, since they are the scarcest. The second approach performs the classification of support relations, since they represent almost all the relations in our corpus, therefore, the most relevant for the structure of arguments in academic texts. The instances employed consider all possible combinations of pairs of argumentative components per paragraph. The features to represent the instances are lexical, syntactic, semantic, structural, indicators, prediction of components, lexical similarity and semantic similarity, extracted before applying machine learning algorithms.

Lexical features (Lex): a concatenated representation of two vectors of unigrams with TF-IDF weighting of the source and the target components is used with the 500 most frequent tokens [26]. The representation considered words and punctuation marks to obtain the feature vector. We obtained 1000 attributes to represent each instance of relations.

Syntactic features (Synt): the Part-of-Speech (POS) labels of the argumentative components of source and target are taken with TF-IDF weighting. In addition, 355 attributes are extracted to represent each relation instance. An example of a POS tag for the word ’research’ is VMN0000 where the first letter “V” represent the verb category and the second letter “M” indicates that is a main verb.

Semantic features (Sem): for these features, we employed word embeddings, specifically those of Polyglot [1]. Word embeddings were trained with Wikipedia in Spanish to represent both the source and target component. To obtain the representation of each component, the embeddings of words present in each component were added. The final size of the vector to represent each relation instance is 128 attributes which result of the concatenation of the representation with word embeddings of the source component with the target component.

Structural features (Str): the location, context, proportion, and distance between the components are taken to represent this set of features. The first three attributes generate a representation for the source and target component. The location indicates the position of the components within the sentence and the paragraph, whether they are at the beginning, end, or middle. The context captures information from neighboring sentences, that is, the number of references, punctuation, words, and tokens in the previous, current and following sentence. The proportion indicates the number of tokens of the components with respect to the sentence that contains them. Another ratio is the number of tokens in the containing sentence with respect to the paragraph, and a Boolean attribute to indicate whether the components cover the entire sentence. The distance between components is used to indicate whether the two components are in the same sentence and to indicate whether the source component has the target component in the next or previous sentence. Finally, a numeric attribute is used to express the number of sentences between the source and target components.

Indicators (Ind): to represent the two components of the relation, five categories of argumentative markers were considered, counting the frequency of markers in each category: justification, explanation, deduction, refutation, and conditional. In total, ten attributes were employed to represent each instance of the relation; five for the source component, and five additional for the target component.

Prediction of argument component type (Comp): the argumentative type of the source and target components (for example premise, conclusion or none) is obtained from the classification model of argumentative components with lexical, semantic, syntactic, structural and sentiment features using an SVM classifier, the analysis of this model is detailed in [8]. The source component is represented with two Boolean attributes, which consist of an attribute to indicate if the type is a premise and the second to indicate if is a conclusion, in the same way, the target component is represented with two attributes. In total, there are four attributes to represent the prediction of the two argumentative components.

Lexical Similarity (Lexsim): to represent the source and target component, two vectors of 500 unigrams with TF-IDF weighting were extracted. The cosine similarity was computed to measure the similarity between the components. The similarity between the two vectors can be calculated by the angle they form, as the cosine of the angle, considering that the closer the two vectors, the greater the similarity between them. The values of the cosine similarity have a closed interval [-1,1], from which a value of 1 is obtained if the angle between the two vectors is zero; that is, similar texts are close to the value of 1 and quite different texts are close to -1.

Semantic similarity (Semsim): the semantic representation of the components and the words was done with word embeddings. One attribute used was the Jaccard semantic similarity, taking pairs of similar terms between both components. Similar pairs were determined when the cosine similarity between these words is greater than 0.6 as in [24]. The Jaccard semantic similarity is processed by dividing the sum of the cosine distances of the similar terms by the sum of the cosine distance of the similar pairs and the different pairs. Also, we considered the addition of the cosine distances of similar and different word pairs between the source-target components. Another attribute is the number of pairs of similar words, the proportion of similar and different pairs. Finally, the cosine distance between the semantic representation of the entire text of the source and target components was also included.

6 Experimental analysis

In this section, we present the analysis for the argumentative relations for each type of features, that is, to identify whether a pair of argumentative components; one component as a source to another as a target are related. Our first approach includes the detection of related pairs as presented in section 6.1. Subsequently, a classification task of attack and support relations, focusing mainly on the identification of attack relations (more scarce), which required balancing the classes, the model is detailed in section 6.2. The second approach presents a model to identify the support relations since they are the most numerous in the academic texts, this is explained in section 6.3.

6.1 Analysis of the existence of argumentative relations

We performed experiments to detect the presence of argumentative relations; that is, given two components, determine if they have either some type of relation (attack or support) or if they are not related. In the experiments, 460 paragraphs were used with at least one conclusion and one premise both with almost exact adjustment in the text limits. We employed 2,402 instances with 519 related (attack or support) and 1,883 unrelated. We performed a stratified cross-validation with 10 folds training several classifiers, such as: Support Vector Machine (SVM), Naive Bayes, Decision Tree (DT), Random Forest, and Logistic Regression (LR) [6], implemented in the Scikit-learn tool [21]. Table 2 shows the best macro-F measure for each representation as the evaluation metric, since our classes are unbalanced.

Table 2
Results of argumentative relations detection for each feature

F1 F1 F1

Algorithm Accuracy Macro w/relation w/o relation

Lex SVM 0.8002 0.6484 0.4175 0.8794

Synt DT 0.7294 0.5896 0.3500 0.8291

Lexsim SVM 0.7839 0.4394 0.0000 0.8789

Comp NB 0.7839 0.6763 0.4897 0.8630

Str NB 0.6978 0.6087 0.4220 0.7954

Ind NB 0.7515 0.5753 0.3018 0.8488

Semsim SVM 0.7839 0.4394 0.0000 0.8789

Sem LR 0.7881 0.5975 0.3204 0.8745

			F1	F1	F1
Lex	SVM	0.8002	0.6484	0.4175	0.8794
Synt	DT	0.7294	0.5896	0.3500	0.8291
Lexsim	SVM	0.7839	0.4394	0.0000	0.8789
Comp	NB	0.7839	0.6763	0.4897	0.8630
Str	NB	0.6978	0.6087	0.4220	0.7954
Ind	NB	0.7515	0.5753	0.3018	0.8488
Semsim	SVM	0.7839	0.4394	0.0000	0.8789
Sem	LR	0.7881	0.5975	0.3204	0.8745

The representation of component prediction (Comp) obtained the best result with an F macro measure of 0.6763, that came out also as the best to identify the instances of the class “with relation” reaching a 0.489 of F measure. We assume that the relations are possible when the text analyzed has a premise and a conclusion. Therefore, the prediction of components provides crucial information to the model. The second best result was obtained with the lexical representation (Lex) reaching the best accuracy of 80.02% with an F macro measure of 0.648 and the best efficacy to identify the class “without relation” with an F-measure of 0.879. Finally, we can observe that the structural representation (Str) reaches an F measure of 0.608 which better identifies the class “without relation” in comparison with the class “with relation”. We assume that the difference in the values of the F measure is due to the imbalance of the classes, i.e. the class “without relation” has a higher number of instances. The less informative features are lexical similarity (Lexsim) and semantic similarity (Semsim), since both achieved an F measure of 0.439. Both features failed to identify any instance of the classes “with relation”, which indicates that the similarity between components does not provide information to identify a relation.

Afterward, we analyzed the representation with all the features as well as the impact of omitting each one of the feature types at a time in the representation. Table 3 shows the result of the classification using these combinations of features in terms of F-measure. The table is ordered by measure F from lowest to highest, the best values are indicated in bold and the lowest values are underlined. The representation with all features obtains an F measure of 0.7527 with the Logistic Regression algorithm. We compared the value obtained by the representation with all the features against the semantic, syntactic, lexical similarity and semantic approaches, we observed they decreased the efficacy of the model achieving values below 0.7522 of F macro measure. Due to these results, the four features that contribute information to the model were selected to assemble an improved representation; these correspond to those in the upper part of Table 3 (Str + Lex + Comp + Ind).

Table 3

Impact analysis in the detection of argumentative relations

			F1	F1	F1
	Algorithm	Accuracy	Macro	w/relation	w/o relation
w/o Str	SVM	0.7877	0.6616	0.4551	0.8681
w/o Lex	LR	0.8343	0.7389	0.5811	0.8967
w/o Comp	SVM	0.8318	0.7434	0.5927	0.8940
w/o Ind	LR	0.8376	0.7433	0.5877	0.8989
w/o Synt	SVM	0.8393	0.7522	0.6053	0.8991
w/o Semsim	LR	0.8426	0.7524	0.6029	0.9019
w/o Lexsim	LR	0.8422	0.7531	0.6048	0.9014
w/o Sem	LR	0.8551	0.7676	0.6250	0.9102
All features	LR	0.8422	0.7527	0.6040	0.9015

In Table 4 we can observe the efficacy of the improved representation when using the lexical, structural, indicator, and component prediction features with an SVM classifier. The representation obtains an F measure of 0.7716 and a Cohen kappa measure of 0.5439 which corresponds to a “moderate” level.

Table 4

Best representation for detection of argumentative relations

Lexical + Component + Structural +	Indicators
Kappa Cohen	0.5439
Accuracy	0.8535
F1 Macro	0.7716
F1 with relation	0.6349
F1 without relation	0.9083

The structures generated from the best identification model of related components were mostly simple. The paragraphs processed were 460, of which only in 319 some relation was identified. In 228 paragraphs, simple relations were detected, which for the most part, i.e. 121, the source component is a premise and the target component is a conclusion. In addition, 49 simple relations were observed from one premise to another premise as well as 27 relations between conclusions.

Figure 3 depicts the most frequent argument structures. The convergent structures (a) and (b) denote the relation of several premises to a conclusion, in particular, we observed seven structures of type (a) and three of type (b). The convergent structure is that most employed by students since it provides more support to the connected conclusion. In addition, some paragraphs have structures composed of several simple relations such as the type (c) with five structures identified and (d) with seven. Finally, serial structures were observed in which several components are linked that end in a conclusion, such as structures of type (e) and (f). The most commonly appearing serial structures are those of type (f) in which a succession of two premises supports a conclusion.

Fig. 3

Structures of arguments identified in the corpus.

6.2 Classification of attack and support relations

We also explored a model to detect the attack and support relations, particularly, to identify a way to differentiate attack relations, since we only have 49 instances. The experiment was carried out using the 49 instances of attack and 49 of support relations to obtain a balanced set. We performed a 5 folds cross-validation using classifiers such as Support Vector Machine (SVM), Naive Bayes, Decision Tree (DT), Random Forest, and Logistic Regression (LR) [6] again from the Scikit-learn tool [21]. The features extracted to classify argumentative relations were detailed in section 5.

In Table 5, we show the result of the best macro-F measure for each individual representation. Indicators feature with classifier LR is observed as the best model to classify relations of attack and support, obtaining a measure F macro of 0.867 and an accuracy of 86.73%. We suppose that the indicators feature achieves the best result because they contain the categories of justification and refutation which provides information to differentiate the attack relations. On the other hand, the least effective feature is the semantic similarity which could indicate a need for lexical information for the task.

Table 5
Classification result of relations with support / attack

F1 F1 F1

Algorithm Accuracy Macro Support Attack

Ind LR 0.8673 0.8673 0.8660 0.8687

Lex DT 0.7959 0.7959 0.7959 0.7959

Sem RF 0.6939 0.6939 0.6939 0.6939

Synt SVM 0.6531 0.6529 0.6600 0.6458

Comp NB 0.6531 0.6518 0.6731 0.6304

Str SVM 0.6224 0.6224 0.6263 0.6186

Lexsim DT 0.5306 0.5306 0.5306 0.5306

Semsim DT 0.4694 0.4694 0.4694 0.4694

			F1	F1	F1
Ind	LR	0.8673	0.8673	0.8660	0.8687
Lex	DT	0.7959	0.7959	0.7959	0.7959
Sem	RF	0.6939	0.6939	0.6939	0.6939
Synt	SVM	0.6531	0.6529	0.6600	0.6458
Comp	NB	0.6531	0.6518	0.6731	0.6304
Str	SVM	0.6224	0.6224	0.6263	0.6186
Lexsim	DT	0.5306	0.5306	0.5306	0.5306
Semsim	DT	0.4694	0.4694	0.4694	0.4694

We also performed an impact analysis of features in the detection of argumentative support and attack relations. The models in which only one feature of the complete set is omitted are presented in Table 6 showing the efficacy of the classification in terms of F measure. The lowest performance of the model was obtained when omitting the syntactic, lexical similarity, and structural features, with 0.7647, 0.7938 and 0.8059 of F1 macro, respectively. This indicates that they contribute information to the model. The semantic feature again shows to negatively affect the model, since by omitting it, we achieve better efficacy. This analysis allows confirming that the best model takes advantage of the indicators features with an LR classifier, achieving an F macro measure of 0.8673.

Table 6

Impact of features in relations model of support/attack

			F1	F1	F1
	Algorithm	Accuracy	Macro	Support	Attack
w/o Synt	DT	0.7653	0.7647	0.7527	0.7767
w/o Lexsim	RF	0.7959	0.7938	0.8148	0.7727
w/o Str	RF	0.8061	0.8059	0.8119	0.8000
w/o Semsim	DT	0.8163	0.8160	0.8085	0.8235
w/o Comp	DT	0.8163	0.8163	0.8163	0.8163
w/o Ind	DT	0.8163	0.8163	0.8163	0.8163
w/o Lex	DT	0.8265	0.8264	0.8211	0.8317
w/o Sem	SVM	0.8673	0.8670	0.8738	0.8602
All	DT	0.8163	0.8163	0.8163	0.8163

The structures found with the model using indicators features with LR classifier were 82 simple relations. We identify 21 support relations between a premise and a conclusion. The support relations were also encountered between premises with 8 relations and between conclusions with 9 relations identified. Moreover, the attack relations were found from one premise to another premise with a total of 18 relations. We also observed 13 relations generated from a premise attacking a conclusion.

Based on the results presented, we can notice that the relations between two components using the model created with argumentative indicators with LR classifier presents the best option for identifying the attack or support relations.

6.3 Detection of support relations

We also proposed a model to detect support relations, because most of the relations utilized by students are aimed at reinforcing their statements, so once detected these relations, it is possible to offer a more direct feedback to the student. The experiment was carried out using the 470 instances of support relations; in addition, for the ‘without _support’ class, 49 attack relationships were added with 421 pairs of non-related components randomly chosen, in order to reach balanced sets. A cross-validation was performed in ten folds, employing the following classification algorithms: SVM, Naive Bayes, DT, Random Forest, and LR, from the Scikit-learn tool. We extracted the features of argumentative relations such as: lexical, syntactic, semantic, structural, indicators, lexical, and semantic similarity.

In Table 7, we show the results of the models using individual features, indicating the classification algorithm with the best efficacy considering the F macro measure. The lexical feature obtains the best efficacy with an F macro of 0.70 and an accuracy of 70.9%. The model with the best performance for the class “w/ support” is the structural and the best for the class “w/o support” is the prediction of the component type. These results show that a combination of features such as the lexical, prediction of component type and structural would achieve a better efficacy. The features with less contribution are the lexical similarity and semantic similarity.

Table 7
Detection results of support relations for each type of feature

F1 F1 F1

Algorithm Accuracy Macro w/Support w/o Support

Lex RF 0.7096 0.7096 0.7105 0.7086

Comp SVM 0.6926 0.6912 0.6705 0.7119

Str SVM 0.6862 0.6814 0.7204 0.6424

Synt RF 0.6340 0.6340 0.6387 0.6293

Sem LR 0.6266 0.6265 0.6317 0.6214

Ind LR 0.6170 0.6170 0.6218 0.6121

Semsim LR 0.5234 0.5234 0.5234 0.5234

Lexsim LR 0.5085 0.5054 0.5444 0.4665

			F1	F1	F1
Lex	RF	0.7096	0.7096	0.7105	0.7086
Comp	SVM	0.6926	0.6912	0.6705	0.7119
Str	SVM	0.6862	0.6814	0.7204	0.6424
Synt	RF	0.6340	0.6340	0.6387	0.6293
Sem	LR	0.6266	0.6265	0.6317	0.6214
Ind	LR	0.6170	0.6170	0.6218	0.6121
Semsim	LR	0.5234	0.5234	0.5234	0.5234
Lexsim	LR	0.5085	0.5054	0.5444	0.4665

To analyze the contribution of each type of feature to the model, we evaluate different models omitting only one of them. The representation with all the features obtains an F macro of 0.7351 with the SVM classifier, as Table 8 shows. When omitting the structural element, the model is affected reaching only an F macro of 0.7053, which indicates the importance of this feature for the task. In the same way, we observed that, by omitting features as the syntactic, indicators, lexical and the prediction of components, the model is also affected, which indicates that these features provide information to the model. On the other hand, the features that negatively affect the model are semantic, and once again lexical similarity, and semantic similarity.

Table 8

Impact analysis in the detection of support relations

			F1	F1	F1
	Algorithm	Accuracy	Macro	w/Support	w/o Support
w/o Str	RF	0.7053	0.7053	0.7063	0.7044
w/o Synt	SVM	0.7245	0.7244	0.7218	0.7271
w/o Ind	SVM	0.7245	0.7245	0.7253	0.7236
w/o Lex	LR	0.7255	0.7255	0.7255	0.7255
w/o Comp	SVM	0.7319	0.7319	0.7313	0.7325
w/o Lexsim	SVM	0.7362	0.7362	0.7373	0.7350
w/o Semsim	SVM	0.7468	0.7468	0.7479	0.7457
w/o Sem	SVM	0.7809	0.7808	0.7822	0.7794
All	SVM	0.7351	0.7351	0.7343	0.7359

The best configuration for identification of support relations was the selection of lexical, component prediction, structural and indicators features. Table 9 shows the results of the best representation (Lex + Comp + Ext + Ind) applying the SVM algorithm, that reached a measure F macro of 0.7872 with an accuracy of 78.72%. Notice that these figures are close in terms of accuracy and F measures given that this experiment was with a balanced data set.

Table 9

Best representation for detection of support relations

Lexical + Component + Structural +	Indicators
Kappa Cohen	0.5745
Accuracy	0.7872
F1 Macro	0.7872
F1 with relation	0.7890
F1 without relation	0.7854

Finally, the structures identified by the model for detecting support relations were 332 structures, where 219 show simple structures. These structures basically consist of relations between two components and 113 with more complex structures. Simple structures were observed mainly between a source premise and a target conclusion (P → C) with a total of 121 structures with such configuration. In addition, there were 40 support relations from premise to premise (P → P) and 29 support relations from conclusion to a conclusion (C → C). To describe the most complex structures we take as a reference those in Figure 3, where different types of convergent, simple and serial structures are presented. We found eleven convergent structures of type (a) in which two premises support a conclusion. In addition, two structures of type (b) also appeared, with three premises supporting a conclusion. Four simple structures of type (c) (P→C, P→P) and six of type (d) were obtained (P→C, P→C). In addition, we identify serial structures, two of type (e) (P→C→C) and three of type (f) (P→P→C). We can observe that support relations are employed mainly as a premise that supports a conclusion or several premises that support a conclusion such as the convergent structure. We assume that most of the argumentative relations in thesis writings are support relations, since the student mainly presents elements to support his statements, either in his section on the Problem statement, Justification or Conclusion.

7 Conclusion

In this research work, we have presented a series of methods for the identification of relations between argumentative components in academic thesis. We proposed two approaches for the identification of argumentative relations; the first, to detect initially which components are related, then, classify them as support or attack; and the second approach to identify which components have support relations. The first approach obtains a fine grain relation type nevertheless demands to realize two steps to obtain the result, on the other hand, the second approach obtains the support relations in only one step.

A corpus was built for the investigation of arguments in thesis and research proposals in Spanish. The annotation task was carried out with the help of seven annotators from several fields related to linguistics, who worked in different subsets of the corpus. They reached an agreement between annotators for argumentative relation with a “moderate” level of kappa. We discovered that students establish support relations most of the time, mainly between a premise and a conclusion, and barely questioning (attacking) conclusions. This could reveal a deficiency in student argumentation since is convenient to offer pros and cons for a conclusion. The corpus was the basis for the development and evaluation of the proposed methods.

Regarding the two approaches presented to identify argumentative relations, we conclude that for the second (one-step) approach, an SVM classifier applied on features such as lexical, structural, indicators and type prediction of components is the most effective. In this approach, the structural feature is useful for the identification of relations with support, and, on the other hand, the component type prediction feature provides more information for the detection of unsupported relations. Notice that, except of the lexical features, the rest (structural, indicators and type prediction of component) are relatively language independent, so we plan to explore them for the task in other languages.

The argument structures generated by the one-step model were for the most part, simple, that is, a premise that supports a conclusion, and the most frequent complex structures are convergent with two premises supporting a conclusion. However, an additional number of attack relation instances are required to refine the classification of that type of relation.

In future work, we expect to analyze the integration of the models for argument analysis in order to process them jointly. For example, the components classification and relation identification simultaneously, using optimization algorithms.

We have also reached enough knowledge of the task and the resources to move to explore improvements for the proposed models by incorporating deep learning techniques, such as the use of recurrent neural networks (e.g. long short-term memory LSTM) to represent word sequences of argumentative components.

Footnotes

Acknowledgment

We thank the annotators for the assistance in the corpus creation. The first author was partially supported by CONACYT, México, under scholarship 357381.

References

Al-Rfou

, Perozzi

and Skiena

, Polyglot: Distributed word representations for multilingual nlp. In Procs of the 17th Conf on Computational Natural Language Learning, pp. 183–192, Sofia, Bulgaria, August 2013. ACL.

Cabrio

and Villata

, Combining textual entailment and argumentation theory for supporting online debates interactions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers - Volume 2, ACL ’12, pp. 208–212. Association for Computational Linguistics, 2012.

Capaldi

, Cómo ganar una discusión. Gedisa, 1990.

Carstens

and Toni

, Towards relation based argumentation mining. In Procs of the 2nd Workshop on Argumentation Mining, pp. 29–34. ACL, June 2015.

Chris Reed

G.R.

, Palau

Raquel Mochales

and Moens

M.-F.

, Language resources for studying argument. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), pp. 2613–2618, Marrakech, Morocco, may 2008. European Language Resources Association (ELRA).

Cocarascu

and Toni

, Identifying attack and support argumentative relations using deep learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, (2017), pp. 1374–1379. Association for Computational Linguistics.

Freeman

J.B.

, Argument Structure: Representation and Theory. Springer, 2011.

García-Gorrostieta

J.M.

and López-López

, Argument component classification in academic writings, Journal of Intelligent & Fuzzy Systems34(5) (2018), 3037–3047.

García-Gorrostieta

J.M.

, López-López

and González-López

, Automatic argument assessment of final project reports of computer engineering students, Computer Applications in Engineering Education26(5) (2018), 1217–1226.

10.

González-López

and López-López

, Colección de tesis y propuesta de investigación en tics: un recurso para su análisis y estudio. In XIII Congreso Nacional de Investigación Educativa, (2015), pp. 1–15.

11.

Green

, Identifying argumentation schemes in genetics research articles. In Proceedings of the 2nd Workshop on Argumentation Mining, pp. 12–21. Association for Computational Linguistics, June 2015.

12.

Habernal

and Gurevych

, Argumentation mining in user-generated web discourse, Computational Linguistics43(1) (2017), 125–179.

13.

Landis

J.R.

and Koch

G.G.

, The measurement of observer agreement for categorical data, Biometrics33(1) (1977), 159–174.

14.

Lawrence

and Reed

, Combining argument mining techniques. In Proceedings of the 2nd Workshop on Argumentation Mining, pp. 127–136, Denver, CO, June 2015. Association for Computational Linguistics.

15.

Lippi

and Torroni

, Argumentation mining: State of the art and emerging trends, ACM Transactions on Internet Technology (TOIT)16(2) (2016), 10.

16.

López Ferrero

and García Negroni

, La argumentación en los géneros académicos. In Actas del Congreso Internacional La Argumentación, (2003), pp. 1121–1129. Universidad de Buenos Aires, Buenos Aires.

17.

Menini

, Cabrio

, Tonelli

and Villata

, Never retreat, never retract: Argumentation analysis for political speeches. In Thirty-Second AAAI Conference on Artificial Intelligence, (2018), pp. 4889–4896. Association for the Advancement of Artificial Intelligence Press.

18.

Mikolov

, Chen

, Corrado

G.S.

and Dean

, Efficient estimation of word representations in vector space, CoRR, abs/1301.3781, 2013.

19.

Mochales

and Moens

M.-F.

, Study on the structure of argumentation in case law. In Procs of the 2008 Conference on Legal Knowledge and Information Systems, (2008), pp. 11–20. IOS Press.

20.

Mochales

and Moens

M.-F.

, Argumentation mining, Artificial Intelligence and Law19(1) (2011), 1–22.

21.

Pedregosa

, Varoquaux

, Gramfort

, Michel

, Thirion

, Grisel

, Blondel

, Prettenhofer

, Weiss

, Dubourg

, Vanderplas

, Passos

, Cournapeau

, Brucher

, Perrot

and Duchesnay

, Scikit-learn: Machine learning in python, Journal of Machine Learning Research12(Oct) (2011), 2825–2830.

22.

Pennington

, Socher

and Manning

, Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543, Doha, Qatar, Oct. 2014. Association for Computational Linguistics.

23.

Persing

and Ng

, End-to-end argumentation mining in student essays. In Proceedings of NAACL-HLT (2016), pp. 1384–1394.

24.

, Tang

, Li

, Xie

and Wang

, Using semantic similarity to reduce wrong labels in distant supervision for relation extraction, Information Processing & Management54(4) (2018), 593–608.

25.

Stab

and Gurevych

, Identifying argumentative discourse structures in persuasive essays. In Procs. of the Conference on Empirical Methods in Natural Language Processing, pp. 46–56. ACL, October 2014.

26.

Stab

and Gurevych

, Parsing argumentation structures in persuasive essays. Computational Linguistics, 2017.

27.

Toulmin

S.E.

, The uses of argument. Cambridge University Press, 2003.

28.

Walton

, Fundamentals of critical argumentation. Cambridge University Press, 2005.

29.

Walton

, Reed

and Macagno.

, Argumentation schemes. Cambridge University Press, 2008.

30.

Wyner

and Bench-Capon

, Towards an extensible argumentation system. In Proceedings of the Ninth European Conferences on Symbolic and Quantitative Approaches to Reasoning with Uncertainty, (2007), pp. 283–294. Springer.

31.

Wyner

, Mochales-Palau

, Moens

M.-F.

and Milward

, Approaches to Text Mining Arguments from Legal Cases, (2010), pp. 60–79. Springer.

Argumentative relation identification in academic texts

Abstract

Keywords

1 Introduction

2 Related work

3 Argumentative structures

Table 1 Agreement level of annotators for argumentative relations Annotations % K (kappa) support, attack, w/ relation, w/o relation 85.5 0.569 w/ support, w/o support 87 0.574 w/ attack, w/o attack 98.2 0.538 w/ relation, w/o relation 85.8 0.568

6 Experimental analysis

6.1 Analysis of the existence of argumentative relations

Footnotes

Acknowledgment

References

Table 1
Agreement level of annotators for argumentative relations

Annotations % K (kappa)

support, attack, w/ relation, w/o relation 85.5 0.569

w/ support, w/o support 87 0.574

w/ attack, w/o attack 98.2 0.538

w/ relation, w/o relation 85.8 0.568