The interplay of complexity and subjectivity in opinionated discourse

Introduction

The research we present in this paper is situated at the nexus of discourse analysis, text linguistics and theoretical complexity research. Our paper is inspired by the current boom in studies on language complexity which is all about defining, measuring and explaining complexity at various linguistic levels (e.g. Baechler and Seiler, 2016; Baerman et al., 2015; Kortmann and Szmrecsanyi, 2012; Mufwene et al., 2017). The interest in language complexity sprung up about two decades ago in the sociolinguistics-typological community and originally centred around the question of whether some languages are simpler or more complex than others (e.g. McWhorter, 2001). Although most of this research focuses on language complexity from a typological perspective, some publications explore language complexity in different text types. Recently, it has been shown that different text types systematically vary in regard to their complexity, for example, newspaper writing tends to be more complex than conversation or email (Ehret, 2017, 2018; Szmrecsanyi, 2009). Furthermore, complexity variation in different text types is closely linked to the formality of the functional-communicative context and discourse situation (Ehret, 2018). As a matter of fact, in text linguistics, complexity has been known to be a distinguishing characteristic between different text types, most notably academic writing versus conversation, for some time (Biber and Gray, 2010, 2016; Biber et al., 2011). In discourse analysis, however, complexity is a less explored construct – despite the fact that it would work well as a descriptive feature for discourse categorisation.

In contrast, notions of subjectivity have been extensively studied in discourse analysis, corpus linguistics and related fields: Subjectivity features prominently in the study of opinion and sentiment (Wiebe and Riloff, 2005; Wiebe et al., 2004), appraisal (Martin and White, 2005) as well as analyses of stance (Biber and Finegan, 1989; Englebretson, 2007), and is the principal object of study in many corpus-based analyses of evaluation (Bednarek, 2006; Hunston and Thompson, 2000b).

Against this backdrop, we sketch a quantitative, corpus-based approach to discourse analysis by exploring the interplay between text complexity and subjectivity in opinionated discourse. Our paper thus fills an important gap in the literature on subjectivity and evaluation because next to nothing is currently known about how text complexity interacts with lexical markers of subjectivity and argumentation, nor how text complexity characterises distinct types of opinionated discourse vis-à-vis arguably more objective news articles. We thus specifically address questions such as whether increased levels of subjectivity result in increased complexity in texts, and whether lexical markers of subjectivity and text complexity correlate with the perceived subjectivity of different text types (such as the hard news, opinion articles and reader comments we investigate here).

Methodologically, the interplay between text complexity and subjectivity, and their respective impact on the formation of distinct text types is assessed by combining conditional inference trees, a type of classification and regression trees, with the analysis of random forests. In this paper, we draw on an innovative information-theoretic metric of complexity to assess text complexity at an overall, morphological and syntactic level. The measure is based on the notion of Kolmogorov complexity and measures the linguistic complexity of a text in terms of its description length: The shorter the information – theoretic description of a text, the lower is its linguistic complexity. In linguistic terms, Kolmogorov complexity is a holistic measure of structural surface redundancy and regularity. Specifically, an implementation of the measure known as the compression technique which has been adapted for the analysis of naturalistic corpora (Ehret, 2017) is utilised. Our definition of subjectivity is guided by the interest in discovering the relationship between text complexity and subjectivity in a quantitative and usage-based fashion. Therefore, we take a practical view on subjectivity and define it in terms of specific lexico-grammatical items that serve the goal of conveying opinion, evaluation, attitude, stance and subjectivity in a text. To be more precise, the set of relevant linguistic features includes an extensive list of evaluative words (Taboada et al., 2011), stance adverbials (Biber, 1988; Biber et al., 1999), modal verbs and connectives which mark argumentative discourse relations (Prasad et al., 2007).

Overall, we propose a strongly quantitative, corpus-based and large-scale approach to investigate the complex relationship between complexity and subjectivity. Although Discourse Studies readers may be familiar with different approaches to subjectivity, we believe that the approach to complexity presented here, and its application to the study of subjectivity in different text types, presents a novel way of analysing discourse.

The results demonstrate that text complexity is a powerful tool in distinguishing different text types, particularly opinionated discourse from news articles, and that stance adverbials interact with complexity in online news comments. More generally, we find evidence for the fact that subjectivity in language is pervasive even in supposedly objective texts (cf. Alba-Juez and Thompson, 2014).

This paper is structured as follows. In section ‘Complexity meets discourse analysis’ we provide some background on the study of language complexity and approaches to analysing subjectivity. Section ‘Database’ introduces the data source. Section ‘Features and measures’ gives an overview of the features and measures utilised. In section ‘Trees and forests’ the statistical analysis is described and in section ‘The interplay of complexity and subjectivity’ the results are presented and discussed. Section ‘Concluding remarks’ offers some concluding remarks.

Complexity meets discourse analysis

The starting point of the current debate on language complexity was the question of whether all languages were equally complex or not (e.g. Kusters, 2003; McWhorter, 2001; Shosted, 2006). Among sociolinguists and typologists, it had been the common belief throughout much of the twentieth century that, on the whole, all languages are equally complex (e.g. Crystal, 1987; Edwards, 1994). In the meantime, plenty of empirical evidence has shown that the complexity of languages and language varieties can and does differ. Theoretical complexity research is now primarily concerned with the definition and measurement of language complexity. Most of this research analyses the complexity of either single languages (e.g. McWhorter, 2012), cross-linguistic datasets (Lupyan and Dale, 2010) or different geographical varieties of the same language (e.g. Szmrecsanyi and Kortmann, 2009). Of particular interest to the current paper, however, are a few studies investigating the complexity in different text types of English. ¹

Szmrecsanyi (2009) analysed the written and spoken texts sampled in the British National Corpus (BNC) in terms of three complexity measures, which relate to the frequency of specific grammatical markers. The results show that different written and spoken text types differ in their complexity, and that these differences correlate with the well-known dimensions of textual variation established by Biber (1988) (Szmrecsanyi, 2009: 335–336). Specifically, Biber (1988) describes the following six dimensions: involved versus informational, narrative versus non-narrative, explicit versus situation-dependent reference, overt expression of persuasion, abstract versus non-abstract information, and online informational elaboration. A more recent analysis of text types in the BNC relies on Kolmogorov complexity to assess text complexity at the overall, morphological and syntactic level (Ehret, 2018). Finding substantial variation in complexity across the various BNC text types, the study stresses that text type, or more generally, the situational context of production, is a crucial factor for language-internal complexity variation and provides empirical evidence for the fact that language adapts to the situational context of production. Furthermore, Kolmogorov complexity captures variation along Biber’s Involved versus Informational Dimension (Ehret, 2018: 18–20), which distinguishes between highly involved production such as face-to-face conversation on the one hand, and abstract-informational production such as academic writing or newspaper writing on the other hand (Biber, 1988: 107–108). For instance, Ehret (2018) reports that formal registers (e.g. newspapers, biography, academic writing) are overall complex and exhibit a lot of morphological variation while comparatively informal/involved registers (e.g. conversation, public debate) are marked by fixed word order (Ehret, 2018: 16, 18–19). This is in line with prominent literature on text linguistics and register variation, which has long since established that language varies according to formality and situational context (e.g. Biber, 1988).

In text linguistics, complexity, among other things, is also known as a linguistic feature characterising different text types. In particular, the complexity of conversation, as a prototypical spoken register, and academic writing, as a prototypical written register, has been extensively studied (Biber and Gray, 2010, 2016). Both conversation and academic writing turned out to be complex, however, in structurally different ways. Complexity in conversation is related to subordination, while noun phrase structures and phrasal modifiers generate complexity in academic writing (Biber and Gray, 2010: 2–4). Similarly, Biber et al. (2011) analysed the adequacy of measures commonly used in the assessment of grammatical complexity in second language writing. The authors found that common measures like T-unit length and increased subordination are characteristic of spoken language, that is, conversation, rather than of academic writing, which is marked by phrasal complexity and a wide range of lexico-grammatical combinations (Biber et al., 2011: 28–31). In short, the complexity of different text types crucially depends on the use of specific grammatical structures which, in turn, are chosen according to the functional-situational and communicative setting of language production. In other words, complexity is a well-established metric for distinguishing different text types, and therefore well suited for the linguistic characterisation of opinionated discourse. Note, however, that the formulation of descriptive complexity in work by Biber and colleagues is not adopted here. In Biber and Gray (2016), the approach is structural and feature-specific, that is, certain grammatical features are considered to be indicative of more complex writing. For instance, dependent clauses (conditional, non-finite complement clauses), relative clauses and prepositional phrases as noun modifiers are markers of complex academic writing. In our approach, by contrast, complexity is not feature-specific, but holistic, in that it considers the entire structural complexity in texts, and can therefore be applied across text types and languages without the need to adjust which specific grammatical features should be considered.

Thus, our particular interest is in the role of complexity in opinionated discourse, specifically, in how it interacts with expressions of subjectivity and evaluation in opinion articles and reader comments. Taking a practical view on subjectivity, we aim at a narrow definition of subjectivity, mostly concerned with how opinion is conveyed through specific linguistic items. Numerous areas of linguistics and other social sciences study subjectivity, often under different terms, and in some cases, using very different methodology. Studies of stance, evidentiality, attitude, sentiment or appraisal are concerned with overlapping linguistic phenomena. Some of the first definitions of subjectivity describe the concept in terms of point of view, that is, as the speaker’s expression of themselves in an utterance, the definition that Lyons (1981) introduced (probably inspired by Benveniste, 1966), and sometimes from the perspective of a physical point of view (Langacker, 1985). Hunston and Thompson (2000a) refer to subjectivity in terms of evaluation and propose that subjectivity comprises two aspects, namely, modality and a point-of-view aspect, variously called evaluation, appraisal or stance. In some approaches, modality and evaluation are two distinct phenomena (Halliday, 1985; Martin and White, 2005), and in some cases the two expressions of subjectivity are combined under one label, such as stance (Biber and Finegan, 1989; Conrad and Biber, 2000). Bednarek (2006) proposes a tripartite classification of evaluation: perspective (the point-of-view aspect), affect (emotion and expression of the self) and modality (epistemic status of propositions). Biber and Finegan approach stance from a purely textual level and define it as ‘the lexical and grammatical expression of attitudes, feelings, judgments, or commitment concerning the propositional content of a message’ (Biber and Finegan, 1989: 93). This is the approach we take in this paper, coupled with previous work on the subjective nature of some discourse relations (Trnavac et al., 2016). The view of subjectivity adopted here is also informed by sentiment analysis, the computational extraction of subjective and evaluative meaning from text. We choose to use the term ‘subjectivity’ as a more general term, since we do not espouse the more specific approaches evoked by terms such as evaluation or stance.

In addition to Biber and colleagues’ approach to lexical markers of stance, we also consider how certain discourse relations signal subjectivity. In the framework of Rhetorical Structure Theory (Mann and Thompson, 1988), Trnavac and Taboada (2012) and Trnavac et al. (2016) have studied how different types of discourse relations are indicators of subjectivity and evaluation, following well-established distinctions in the literature between semantic and pragmatic relations (van Dijk, 1979), external and internal discourse relations (Martin, 1992) or subjective and objective relations (Sanders et al., 1993). Thus, in this paper, we focus on discourse connectives that are markers of subjective relations, as indicators of subjectivity in text.

Finally, sentiment analysis and opinion mining focus on how subjectivity is linguistically conveyed in texts (Wiebe and Riloff, 2005; Wiebe et al., 2004). These fields do, however, not only encompass the study of evaluative expressions but are also concerned with their direction and strength: subjective expressions can be either positive (love) or negative (hate) and can vary in the degree of subjectivity expressed (dislike vs hate vs loath) (Taboada et al., 2011; Wiebe, 2000). In sentiment analysis, subjectivity is essentially defined as the linguistic expression of opinion and evaluation.

As can be seen from this brief run-through of the literature on subjectivity – which by no means does justice to the vast amount of publications on the topic – subjectivity is a multi-faceted and hard-to-define concept. Maybe a core definition could be derived from the goal of most of the research on subjectivity and evaluation which is ‘[. . .] to explicate the range of lexical, grammatical, textual, and intertextual means by which speakers and writers laminate their language with their attitudes and points-of-view about its content’ (Englebretson, 2007: 16). Given that our research is quantitative and large-scale, we focus on the first of those means, the lexical level, adding some aspects of the grammatical level, in the form of grammatical structures that convey evaluation. Our definition of subjectivity, then, is one where subjectivity is defined as the expression of evaluation and opinion through specific lexical items and grammatical structures, which can include a variety of devices, with modal expressions being one of them. Hunston and Thompson (2000a) use the umbrella term evaluation for this combination. This definition is congruent with the scope of phenomena under the Appraisal Framework (Martin and White, 2005) and is also similar to stance in Conrad and Biber (2000), and Biber and Finegan (1989).

In broad strokes (a full description is provided in section ‘Database’), subjectivity is operationalised as (i) specific evaluative words (adjectives, nouns, verbs and adverbs); (ii) stance adverbs; (iii) modal verbs; and (iv) connectives which mark argumentative discourse relations. Although our take on subjectivity and evaluation is quantitative and large-scale, these expressions occur in the context of naturalistic discourse (Alba-Juez and Thompson, 2014), where they fully develop their meaning. The study of subjectivity and evaluation in context is therefore certainly within the scope of discourse analysis. As Du Bois (2007: 140–141) points out, when researching how stance works in language, ‘we find ourselves faced with a complex web of interconnections linking stance with dialogicality, intersubjectivity, the social actors who jointly enact stance, and the mediating frameworks of linguistic structure and sociocultural value they invoke in doing so’.

Database

We tap the Simon Fraser Opinion and Comments Corpus (SOCC), the up-to-date largest collection of opinionated language. SOCC comprises roughly 10,000 opinion articles and 660,000 reader comments posted in response to these opinion pieces. Thus, a direct comparison between different types of opinionated discourse – the more formal opinion articles versus the comparatively more personal and informal reader comments – is facilitated. The data spans the time period from 2012 to 2016 and was collected from the Canadian online newspaper The Globe and Mail (Kolhatkar et al., 2020). SOCC has been specifically designed to analyse the characteristics of online comments preserving the comments’ thread-structure and providing extensive discourse annotation (e.g. Appraisal and negation annotation). In this paper the plain text version of the articles and comment threads is used.

As a benchmark for our measures and in order to compare the textual properties of opinionated discourse to (arguably) more objective discourse, we compile a complementary corpus of news articles from The Globe and Mail following the sampling guidelines outlined in Kolhatkar et al. (2020).

We furthermore sample ‘page-one stories’ which cover a wide range of general and, at the time of their publication, topical issues to immunise the database against potential lexical bias. In other words, we argue that page-one-stories are more representative in terms of vocabulary and grammatical structures than, for instance, a database comprising a random sample of articles or articles of a particular category (e.g. political). The collected news corpus counts 4600 articles.

We then generate a random sample of SOCC articles and comment threads that approximates the news corpus in size and number of texts to obtain a database with roughly the same size per text type. In general, attention is furthermore restricted to texts with a minimum number of 700 words because the complexity measure utilised requires comparatively large texts to return representative results (see section ‘Complexity’; for a discussion see Ehret, 2017). A statistical overview of the final database is given in Table 1.

OPEN IN VIEWER

Table 1.

Number of texts per text type in the final database.

Text type	Number of texts	Number of words
Opinion articles	3509	2,758,993
Comment threads	3509	22,191,197
News articles	3430	3,332,559
Total	10,448	28,282,749

Features and measures

Subjectivity

Our take on subjectivity is empirically-oriented and defines subjectivity as lexico-grammatical expressions of opinion, evaluation and subjectivity in a text. As described in section ‘Complexity meets discourse analysis’, we employ four types of linguistic items as markers of subjectivity: (i) specific evaluative words (adjectives, nouns, verbs and adverbs); (ii) stance adverbs; (iii) modal verbs; and (iv) connectives which mark argumentative discourse relations. This subsection describes the sources which we tapped to obtain these markers and provides some rationale for why these markers have been included in the analysis. In addition, corpus examples illustrate how these items and structures are used in context.

In order to obtain a comprehensive list of evaluative words, we draw on the Semantic Orientation CALculator (SO-CAL) which has previously been applied in the task of sentiment analysis (Taboada et al., 2011). SO-CAL comprises dictionaries of positive and negative adjectives, nouns, verbs and adverbs, amounting to a total of 5042 evaluative words (2252 adjectives, 1142 nouns, 903 verbs and 745 adverbs). The original dictionaries contain a fine-grained classification of sentiment which distinguishes between five valence categories indicating varying degrees of positive or negative sentiment. ³ Words can take valences ranging from −5 to 5, where higher positive or negative numbers are closer to the extremes of the evaluative continuum and thus indicate stronger evaluation. For example, award-winning has a valence of 5, monstrosity has a valence of −5, informative is 2, and distasteful is −3. In this paper, attention is restricted to evaluative words with a valence of −4/4 and −5/5 as markers of subjectivity because we are primarily interested in words that unambiguously (and strongly) convey evaluation and subjectivity (as in 2). Furthermore, the different word classes are collapsed into a simple binary categorisation of positive and negative evaluative words (regardless of word class) which we consider sufficient to capture subjectivity. Thus, a total of 674 evaluative words is analysed. These positive and negative words may, of course, be used with different polarity in context. For instance, a strongly negative phrase such as not great at all would be captured only in the positive adjective great. Similarly, sarcastic uses of words (that’s just great!) are to be interpreted with the opposite polarity to that of their dictionary entry. Since we are concentrating on a few words, and their most typical usage, such concerns can be disregarded here, although they are of importance in practical applications such as sentiment analysis (Taboada, 2016).

(2) a. Experts say that the post-generation is less polarized over Europe – their parents either adored

or detested continental unification. (GLOB000020120622e86m00009.txt)

 b. Funny, they never seem to find government candidates to be unacceptable. (com- ments2014.17829753.txt)

 c. . . . I doubt anyone on earth would deny the magnificence of Rio or Vancouver or the

awesomeness of Boston. (comments2013.11033911.txt)

Stance adverbs, as illustrated in 3 include adverbs that indicate epistemic, attitudinal or style stance, according to Biber and Finegan (1989) and Conrad and Biber (2000). Our list includes 78 adverbials and prepositional phrases (e.g. undeniable, of course, to my surprise) culled from Biber (1988) and Biber et al. (1999).

Modal verbs (see Example 4) are included as markers of subjectivity, as Hunston and Thompson (2000a) suggest. Our set of modal verbs includes the full list of modals (including abbreviated and negated forms) commonly found in English, that is, will, would, might, shan’t, ’ll. ⁴

(3) a. Frankly, as a US citizen if I were living abroad and earning my living outside the US the IRS could go pound sand for all I would care. (comments2013.6994760.txt)

b. I have zero issue with most immigration, in fact, most in Canada are immigrants. (comments2016.30609024.txt)

(4) a. And who will be burning all that fossil fuel Mike. The wackos won’t care but the law will by then. (comments2016.29805674.txt)

Finally, our list of markers of subjective discourse relations was compiled from the Penn Discourse Treebank annotation manual (Prasad et al., 2008) following the assumption that some discourse relations are subjective and more frequent in argumentative text. By discourse relations we mean those that connect discourse units and can be assigned labels such as Elaboration, Contrast or Summary (Mann and Thompson, 1988; Prasad et al., 2008). Finding such relations automatically is still an open question, but it is well known that connectives and discourse markers are robust indicators of discourse relations. For that reason, we extracted all discourse markers listed as signals for six relations that are typically considered argumentative (Moens et al., 2007; Peldszus and Stede, 2016): Cause, Comparison, Condition, Contrast, Evaluation and Explanation. The list of argumentative markers includes a total of 137 items comprising connectives such as additionally, if or specifically as well as multi-word markers such as on the ground that, except insofar as.

(5)  a. Both Canada and the U.S. are high immigration countries but because Canada has the right model we have far more immigrants, yet far less social dislocation. (opinion2016.29294152.txt)

 b. But the regime they conveniently champion is ultimately self-defeating and constraining as far as the broader goals of these sovereignty movements is concerned. (comments2014.20598876.txt)

The complete set of subjectivity markers (see Table 2 for a tabular overview) analysed in this paper thus comprises 907 items. These markers of subjectivity were automatically extracted from our database using a custom-made python script. Subsequently, the frequencies of all markers were normalised per 1000 word tokens as is customary in corpus linguistic frequency analyses in order to control for differences in text length.

OPEN IN VIEWER

Table 2.

Number of markers per marker category.

Marker category	Number of markers
Evaluative words, positive	348
Evaluative words, negative	326
Stance adverbs	78
Modal verbs	18
Connectives	137
Total	907

Trees and forests

In order to analyse how text complexity and markers of subjectivity interact and contribute to the characterisation of distinct types of opinionated discourse, we utilise conditional inference trees (in the following simply referred to as trees), a type of classification and regression trees, and conditional random forests, an ensemble method for regression and classification based on an aggregate of single trees. ⁵ In general, trees and random forests are powerful tools for analysing and visualising complex interactions between many different predictor variables, as in our analysis with three complexity metrics and five different types of subjectivity markers, and assessing the importance of individual predictors (Strobl et al., 2009b). In particular, the combination of trees and random forests is well suited to explore our dataset as there are a couple of issues that cannot be easily handled by, for example, standard general regression techniques (Tagliamonte and Baayen, 2012: 161–165). First, text type is a categorical dependent variable with three levels (opinion articles vs reader comments vs news writing); second our predictor variables have different scales of measurement (complexity scores vs relative token frequencies); and third, some of the predictors are somewhat correlated. These issues can be conveniently addressed by utilising conditional inference trees and random forests.

Our objective in this paper, however, is neither prediction nor classification. Rather, we aim to describe and characterise opinion and reader comments vis-à-vis arguably more objective newspaper writing and to explore the interplay between text complexity and markers of subjectivity. For this reason, the implementation of trees and forests and the statistical descriptions are kept straightforward. All statistics reported in this paper were conducted with the open-source software R (R Core Team, 2019). For the implementation of trees and random forests we specifically draw on the R package partykit (Hothorn and Zeileis, 2015) and largely follow the recommendations outlined in Strobl et al. (2009a). The R code, additional statistics, as well as the original dataset are available at https://github.com/katehret/compinion.

In this spirit, we construct a feature matrix with the individual texts as rows and each of the variables, that is, text type, complexity measurements and frequency counts of the various subjectivity markers, as columns. The year in which a particular article or comment was posted is also included as predictor variable. In short, we analyse the descriptive power and interactions between nine predictor variables in 10,448 samples of naturalistic discourse.

In a first step, we construct a single tree to explore and visualise the complex interactions between the various predictor variables. In order to avoid overfitting, yet obtain a linguistically interpretable tree, we tune our tree model by splitting the data into training and test data. Subsequently, we compare how well the model performs in both datasets under different parameter settings. The accuracy of each parameter combination in the training and test dataset is reported in Table 4 (Appendix). ⁶ The ‘best’ model is then selected based on the prediction accuracy in the test and training dataset as well as the linguistic interpretability of the tree. This tree has a prediction accuracy of 86% for the training data and 85% for the test data.

In a second step, we grow random forests to assess the importance of each individual predictor variable using the default settings for growing trees in the partykit package, which is recommended for unbiased forests. On a technical note, the data is split into a training and test dataset and random forests of varying sizes are grown to control for variable selection bias as well as to ensure that the results are robust. Specifically, we grow forests with N = 500, and N = 1000 and N = 2000 trees. To evaluate model performance, the prediction accuracy of the training and test set was calculated for each forest. The prediction accuracy across the differently sized forests (see Table 3 is virtually identical and the ranking of the predictors according to variable importance does not vary (for statistical details see https://github.com/katehret/compinion). In this paper, we therefore discuss the simplest model, that is, the forest with N = 500 trees. The prediction accuracy for this forest is 93% for the training data and 92% for the test data.

OPEN IN VIEWER

Table 3.

Training accuracy and test accuracy for differently sized forests.

Tree size	Training accuracy	Test accuracy
500	0.9265	0.9199
1000	0.9271	0.9196
2000	0.9260	0.9183

The importance of individual predictor variables is assessed by calculating the conditional permutation-importance measure (Strobl et al., 2008, 2007) for each predictor because it can reliably assess the importance of predictors which are correlated. As mentioned at the beginning of this section, some of the predictor variables are correlated (Figure 1). To be more precise, connectives and adverbials (Pearson’s correlation coefficient r = 0.37), as well as morphological and overall complexity (Pearson’s correlation coefficient r = 0.38) exhibit a medium positive correlation. In contrast, overall complexity moderately negatively correlates with both connectives and adverbials (Pearson’s correlation coefficient r = −0.37 and r = −0.39, respectively). We also observe the typical trade-off between morphological and syntactic complexity (discussed e.g. in Ehret, 2018; Ehret and Szmrecsanyi, 2016b) which is expressed by a high negative correlation (Pearson’s correlation coefficient r = −0.74).

OPEN IN VIEWER

Figure 1.

Pairwise correlations of predictor variables based on Pearson’s correlation coefficient. Red squares indicate positive correlation; blue squares indicate negative correlation.

The interplay of complexity and subjectivity

In this section, the interactions between text complexity and subjectivity are unravelled and the importance of the individual predictor variables is discussed.

Figure 2 plots the selected tree model with branches ending in terminal nodes. At each node binary splits of the data are performed depending on the value of the splitting variable. These splits result in different branches, which, basically, group the data into consecutively more homogeneous subsets (Strobl et al., 2009b: 326). Interactions occur where a predictor appears only in one of the resulting two branches after a previous split (Strobl et al., 2009b: 239). Interactions in trees are therefore always local, that is, interactions between particular predictors apply to a specific subset of the data rather than the entire dataset. In this model, interactions between only four of the nine predictor variables can be observed: overall complexity and syntactic complexity, overall and morphological complexity and morphological complexity and adverbials.

OPEN IN VIEWER

Figure 2.

Tree model with 19 nodes displaying the complex interactions between overall complexity, syntactic complexity, morphological complexity and adverbials. The values on the edges of the split nodes indicate at which complexity scores or frequencies the data was split. All values were rounded.

In the interpretation of the tree, we will move along the edges (the lines connecting the nodes) from the root node at the top to the terminal nodes at the bottom. For example, we can identify an interaction between morphological complexity and the presence of adverbials only in texts with an overall complexity score of ⩽0.7 (node 1), a syntactic complexity score of > 0.94 (node 2), an overall complexity score of >−1.96 (node 8) and a morphological complexity score of less than or equal to −0.96 (node 10). In other words, the effect of adverbials depends on the values of morphological, overall and syntactic complexity.

It can thus be observed that the data is split into two major branches based on the overall complexity of the texts: a first branch grouping opinionated discourse, that is, comments and articles, together (ending in the terminal nodes 5, 6, 7, 9, 12, 14, 15 and 16), and a second branch of news articles (ending in the terminal nodes 18 and 19). Overall complexity is therefore the most distinguishing predictor in this dataset (see also Figure 3). News articles are overall more complex (>0.7, node 1) than comments and opinion articles (⩽0.7, node 1). This finding is consistent with previous complexity research (Ehret, 2018) in which newspaper writing, in particular broadsheet newspaper, was found to be overall the most complex register among a substantial number of written and spoken registers analysed. Furthermore, news writing is known to be characterised by a high type-token-ratio and information density, which are both indicators of lexical richness (Biber, 1988). This dovetails with the high overall complexity scores observed in news texts, also bearing in mind that lexical richness and overall complexity are somewhat correlated (Ehret, 2018: 14).

OPEN IN VIEWER

Figure 3.

Conditional variable importance of the individual predictors.

Turning to the left part of the plot, it can be seen that syntactic complexity roughly distinguishes between opinion articles and comments (node 2). Opinion articles tend to be less syntactically complex (⩽0.94, node 2) than comments (>0.94, node 2). Syntactic complexity is defined in terms of word order rigidity, that is, more rigid word order is more complex while more varied word order counts as more syntactically simple. In this context, then, opinion articles are marked by less rigid and more varied word order than comments. This result matches with our intuitions, as opinion articles which are carefully edited should naturally contain more varied word order and syntactically elaborate structures (including patterns such as Never was I more astonished) than comments, which tend to be written ‘on the fly’. This interpretation is supported by the split in node 3 which groups the texts further together based on their syntactic complexity. The terminal nodes 5 and 6 sample almost exclusively opinion articles, while in node 7 there are also some comments. Node 4 furthermore distinguishes between opinion articles of differing levels of overall complexity.

At this point, it is important to note that these observations do not mean that news writing is less syntactically complex than opinion articles. Rather, syntactic complexity is only a distinguishing criterion for this specific subsection of the data, that is, for texts with an overall complexity of ⩽0.7, as trees only depict local interactions.

Let us now focus on the centre of the tree. Node 8 splits opinionated discourse – the majority of comments, but also a couple of opinion articles – into texts with an overall complexity score equal to/less than and greater than −1.96, respectively. All of these texts are marked by very low overall complexity as indicated by the negative score. The articles and comments sampled in node 9 are the least overall complex texts in the dataset. At node 10, the data is divided according to morphological complexity: Most comments were categorised in the terminal nodes below. Therefore, morphological complexity is a distinctive characteristic for comments as opposed to opinion articles, such that most comments are less morphologically complex (⩽−0.96, node 10) than opinion articles (>−0.96, node 10). At the next two nodes below (nodes 11 and 13), an interaction between morphological complexity and subjectivity, that is, stance adverbials, can be observed. Stance adverbials are an important characteristic for comments with a level of morphological complexity below −0.96. In other words, only in some comments, namely those of relatively low overall complexity, high syntactic complexity and low morphological complexity, are adverbials an important predictor. On an interpretational plane, these interactions indicate that comments tend to follow a comparatively rigid word order and make frequent use of stance adverbials such as frankly, admittedly, seriously to mark personal opinion and structure discourse.

In summary, the tree visualises how different levels of text complexity locally interact and clearly shows that, in general, overall text complexity is the major characteristic distinguishing between opinionated discourse and news articles. Furthermore, syntactic complexity distinguishes between the two text types opinion articles and comments. The ranking of variable importance presented in Figure 3 largely corroborates the findings described above. Overall complexity is by far the most important predictor, followed, at a considerable distance by stance adverbials and morphological complexity. This reflects the local interaction between morphological complexity and adverbials which are distinctive predictors in the categorisation of comments. It is surprising that syntactic complexity is only on rank five, although it served as splitting variable in the tree at various levels. This might be explained by the strong negative correlation between syntactic and morphological complexity, which can only be accounted for in the calculation of the conditional permutation-importance. The ranking of variable importance, however, clearly shows that subjectivity markers other than stance adverbials barely matter in the classification of opinion articles, comments and news articles. Neither the tree nor the variable importance ranking can illuminate this issue. In search for an explanation, we therefore survey, post hoc, the distribution of subjectivity markers in the three different text types. Figure 4 plots the percentage of each type of subjectivity marker in news articles, opinion articles and comments (see Appendix, Table 5, for raw frequencies). Strikingly, the different types of subjectivity markers, including positive and negative evaluative words, occur with roughly the same percentage in all three text types, irrespective of whether the texts are opinionated or supposedly objective.

OPEN IN VIEWER

Figure 4.

Distribution (in percentage) of the different subjectivity markers per text type in the entire dataset.

In short, the various markers of subjectivity are not a distinctive characteristic in opinion articles, comments and news articles, and therefore do not serve as splitting variables in the tree nor are they ranked as important predictors in the variable importance ranking. Only in interaction with the various levels of text complexity do adverbials of stance make a contribution to the categorisation of the three text types at a local level, that is, only for a small subset of the data.

Concluding remarks

We have offered a quantitative corpus-based analysis of subjectivity and complexity in opinionated discourse and news articles. By comparing three different text types (general news articles, opinion pieces and online comments), we explore the relationship between text complexity and linguistic markers of subjective and evaluative language. To capture text complexity, we use a holistic, information-theoretic measure of complexity which has previously been shown to be a reliable tool in text type analyses (Ehret, 2018). We approach subjectivity from a lexical standpoint, compiling a list of subjectivity markers from the literature (Biber et al., 1999; Taboada et al., 2011), which comprise highly positive and negative words, stance adverbials, modals and argumentative connectives. The analyses include conditional inference trees for visualising complex interactions together with random forests, an established method for investigating the predictive power of individual variables.

Our main finding is that text complexity, and specifically overall complexity is by far the most powerful predictor distinguishing between opinionated discourse on the one hand, and news articles on the other hand: opinionated discourse is less overall complex than news articles. This dovetails with previous research on the complexity of different British English registers which reports that newspaper writing was one of the most overall complex registers (Ehret, 2018). On a methodological plane, this finding highlights that overall text complexity may be a useful tool for text type classification and characterisation, with a potential of further application, for example, in fake news identification and other applications that involve register classification. We emphasise that our approach is lexically-based and aggregate in nature. This might be considered a caveat, in that it does, on the one hand, not capture more subtle structures and patterns (Hunston, 2011) that may be deployed in the expression of subjectivity such as modal-like patterns of the type ‘noun of + verb-ing’ (the advisability of, the likelihood of, the difficulties of), and, on the other hand, not distinguish for example, between different types of stance adverbials. That said, the strength of this approach lies in its applicability to large-scale naturalistic datasets whose (qualitative) analysis would otherwise be extremely time consuming, empirically expensive and therefore outside the scope of traditional discourse analysis.

We further find that stance adverbials are the only markers of subjectivity analysed in this paper which have a distinguishing nature vis-à-vis the complex interactions of text complexity, yet only in a small subset of comments. There are several possible explanations for this. We pointed out earlier that less morphologically complex texts show this interaction, perhaps due to relatively rigid word order and comparatively simpler structures that rely on stance adverbials to convey opinion. Another explanation may be that (stance) adverbials are present overall in opinionated texts, but perhaps more saliently so in comments, suggesting that stance adverbials are a text type specific stylistic device (cf. Ehret and Taboada, 2020).

Our work has some interesting theoretical implications. We analysed a large number of well-established subjectivity markers in naturalistic discourse. Contrary to our expectations, we found practically the same proportion of these markers across the three text types. More to the point, opinionated discourse, represented here by opinion articles and reader comments, does not contain more subjectivity markers than the arguably more objective news texts. This is particularly striking in regard to positive and negative evaluative words which, in our dataset, are the clearest and strongest markers of evaluation and expression of subjectivity. This raises the question of whether ‘objective’ discourse exists at all, or whether any text produced by language users is invariably subjective in the sense of Lyons (1981). Expressing one’s opinion, or point of view, is one of the most fundamental functions of human language, one which language users might not be able to avoid. We thus conclude that naturalistic discourse, be it labelled opinionated or objective, always conveys to some extent the opinion, stance, or evaluation of the language user. As has been pointed out by a reviewer, the opinion, evaluation or stance conveyed in a text might not always be the writer’s own opinion and we acknowledge that subjectivity attribution is outside the scope of our approach which is lexically-based and quantitative. Naturally, it also goes without saying that a different, more fine-grained distinction between positive and negative evaluative words might have led to different results and that the findings presented here should be further corroborated with different data covering more varied text types. We would furthermore like to point out that sarcastic and ironic uses of evaluative words (see section ‘Subjectivity’) are not captured here. Nevertheless, our findings are supported by recent qualitative research investigating the connection between emotion and evaluation in discourse (Mackenzie and Alba-Juez, 2019, see also Alba-Juez and Thompson, 2014). In particular, Alonso Belmonte (2019) describes how journalists use emotional-evaluative language in a major Spanish newspaper. Furthermore, our results dovetail with a substantial body of literature on changes in journalism, or media discourse more generally, which also discuss the use of involved, informal or emotional language (e.g. Fairclough, 1995; Zelizer, 2009). As a result of such linguistic changes, the distinctions between different news types, such as opinion and factual news, are no longer clear-cut (e.g. Peters, 2011: 298).

However, the finding that news texts employ roughly as many lexical markers of subjectivity as opinion articles and reader comments, could also be seen as part of a more general (linguistic) development, namely, informalisation (Fairclough, 1992) or colloquialisation (Hundt and Mair, 1999) which refers to an increasing informal, colloquial language use in written discourse, most evident in newspaper writing (see also Peters, 2011).

Finally, this paper advances the state-of-the-art in discourse analysis by showcasing how quantitative methods can be applied to explore discourse analytic concepts, such as subjectivity, in otherwise inaccessible datasets.