A linguistic approach for determining the topics of Spanish Twitter messages

2.1. Categorization of Twitter messages

Most work on categorizing Twitter messages is based on classical bag-of-words approaches without taking into account linguistic information other than surface forms appearing in texts. In this context, Sriram et al. [15] apply a bag-of-words model for classification of Twitter messages into five categories (News, Events, Opinions, Deals and PrivateMessages), also incorporating author name and seven binary features: the presence of shortened forms of words and slang, time-event phrases, opinionated words, emphasis on words, currency and percentage signs, @username at the beginning of a tweet, and @username within a tweet. They perform experiments on a corpus composed of 5407 tweets from 684 authors and find that the author feature is the most discriminatory one.

Some authors propose applying techniques for dimensionality reduction. Thongsuk et al. [16] propose a method for classifying Twitter messages into three categories (Airline, Food and Computer & Technology) using Latent Dirichlet Allocation (LDA [17]) to cluster words extracted from tweets into a set of 50 topics, which are then used as features of an SVM classifier [18]. They perform some experiments on a purpose-specific corpus, reporting a significant improvement with respect to a baseline provided by a bag-of-words approach. LDA was also used by Hong and Davison [19] for topic modelling on Twitter. They apply the topic models to classify users and corresponding messages into 16 categories (Art&Design, Books, Business, Charity, Entertainment, Family, Fashion, Food & Drink, Funny, Health, Music, News, Politics, Science, Sports and Technology), showing that models trained on aggregate longer texts (i.e. texts resulting from combining all messages generated by the same user or from aggregating all the messages that contain a given term) yield better performance than models trained on single messages. Therefore, their work is in a middle ground between short-text categorization and classical classification of longer texts.

The fact that a classifier can perform better than another on a given topic but worse on a different topic is exploited by Fiaidhi et al. [20] by training four different classifiers on 12 classes of trending topics (Politics, Education, Health, Marketing, Music, News & Media, Recreation & Sports, Computers & Technology, Pets, Food, Family and Other) and defining a method for choosing the best classifier for each class.

In a certain way, Twitter resembles an information network, and in the same way as the classification of news was a classic problem in professional journalism, tweet classification into news categories is a current relevant problem for online content creators. In this context, Bastos et al. [21] propose 11 Twitter topics, on a scale from hard to soft news – Politics, Altruism, Events, Technology, Games, Idioms, Music Personality, Movies, Celebrity, Lifestyle and Sports – and explore the distribution of messages across these topics, without performing linguistic processing. Lee et al. [22] classify Twitter trending topics into 18 categories (Arts & Design, Books, Business, Charity & Deals, Fashion, Food & Drink, Health, Holidays & Dates, Humor, Music, Politics, Religion, Science, Sports, Technology, TV & Movies, Other News and Other) using a bag-of-words model and a decision tree learner, testing it on a corpus of 768 trending topics. It is worth noting that they do not perform classification on individual tweets, instead, they save up to 1000 tweets corresponding to each trending topic in a document and then classify this large document into one of the 18 predefined categories. As a consequence, their work has more similarities with classical text classification than with tweet classification.

Some works on classification of short messages integrate the textual content of each message with information from other sources such as hyperlinks or Wikipedia ⁵ pages. Genc et al. [23] develop a technique for calculating distances between messages based on the distance between their closest Wikipedia pages, regarding Wikipedia as a transform space in which measurements can be made more accurately. They perform experiments on a small corpus of 100 tweets into three categories. Gattani et al. [24] describe a real-time, end-to-end industrial system for Twitter processing involving entity extraction, linking, classification and tagging. In the case of topic classification, they define a set of 23 predefined topics corresponding to the child nodes of the root of the taxonomy of a knowledge base built out of Wikipedia, but also enriched with a variety of structured data sources. Their approach was tested on a small collection of 99 tweets, with a per-topic performance ranging from 100% precision and recall for topics such as Environment or Travel to 0% recall for composite topics such as Food & Agriculture or Home & Leisure. Kinsella et al. [25] investigate topic classification in social media by using textual content and metadata retrieved from external hyperlinks. Their work focuses not on Twitter messages but on longer ones. In particular, they perform experiments on classifying 6626 user-generated posts submitted to a message board into general categories (Photography, Soccer, Musicians, Movies, Politics, Martial Arts, Motors, Poker, Television, Atheism) and 1564 posts in music categories (Rock, Electronic, Alternative, Hip-Hop, Punk) where the category of each post is determined by the forum in which it is published. They find a large variation in classification performance across different categories, for example, classification of Musicians is trivial since almost all posts have a link to MySpace ⁶ but classification of Television posts is challenging because messages in this category cover any topic that is broadcast. Scores for music categories are also lower than for general categories owing to the higher similarity of the topics.

2.2. Categorization of Twitter messages on the TASS corpus

The corpora used in the works cited so far exhibit a great variety in their size and topics, making a fair comparison of results difficult, if not impossible. The TASS corpus [26, p. 6] is a standard corpus for topic categorization of Spanish Twitter messages, where each tweet is assigned at least one topic from a list including Politics, Entertainment, Economy, Music, Soccer, Films, Technology, Sports (other than soccer), Literature and Other. ⁷ Several authors have performed experiments on this corpus using a variety of approaches.

Batista and Ribeiro [27] rely on logistic regression classification models, which correspond to maximum entropy classification for independent events, creating a binary classifier for each topic that estimates the probability of a tweet pertaining to that topic or not. For each tweet, its unigrams and bigrams are submitted to all classifiers and the most probable topic is selected based on these classification probabilities. A similar approach is undertaken by Pla and Hurtado [28]: a cascade of binary SMO classifiers [29] trained for each topic is applied to each tweet, determining if a message belongs to it or not. As a result, they obtain as output the topics assigned by at least one classifier. In the case where no topic is assigned to a tweet, a cascade of libSVM classifiers [18] is used as back-off: as each libSVM classifier provides a weight for each possible assignment of a class to the tweet, the tweet is assigned the topic which provides the highest probability estimate. Martínez-Cámara et al. [30] also apply a SVM classifier to a bag-of-words enriched with selected hashtags and related words obtained from Google AdWords KeyWord Tool. ⁸

Martín-Wanton and Carrillo de Albornoz [31] build for each topic a lexicon of words that best describe it, in terms of Kullback-Leibler Divergence (KLD [7, chapter 9]), thus representing each topic as a ranking of discriminative words. Moreover, a set of events is defined according to an LDA model. To determine which of the topics corresponds to each event, the topic with the highest statistical correlation is obtained by comparing the ranking of words of each topic and the ranking of words most likely to belong to the event. KLD is also used in the construction of language models by Castellano González et al. [32]. They address the challenge posed by topic categorization from an Information Retrieval (IR [7]) perspective, where the training set of tweets is represented and indexed using language models. To determine the category of an unseen tweet, its content is used as a query against the index storing the previously built models. They extend their work in Castellano González et al. [33], where they analyse the type of tweet information to be used in the classification and which process should be followed to take this information into account, proposing different types of modelling as well as different ways of performing the information retrieval process according to the different types of information. The results suggest that named entities only help in the classification of a small number of tweets, so that it is necessary to use the overall bag of words to attain average performance. An IR perspective is also applied by Montejo-Ráez et al. [34] by defining a text processing component to convert tweets into vectors according to the Vector Space Model with tf.idf weighting [7, chapter 2] after a normalization process. Then, an automatic extraction system is used to generate a term-feeling matrix from affective Twitter posts, whose dimension is reduced by means of Latent Semantic Indexing [35, chapter 18]. The approach shows poor performance, owing to the reduced size of the training set, according to its authors.

Fernández Anta et al. [36] apply several machine learning classifiers on a portion of the corpus in order to test the utility of n-grams, stemming, lemmatization, word correction, presence of hashtags, presence of user references and presence of hyperlinks, concluding that none of them make a clear difference when introduced in the classification framework. Trilla and Alías [37] adapt a text classification scheme based on Multinomial Naive Bayes to deal with Twitter messages, using a binary-weighted feature space, which is unable to predict the class with the fewest number of training examples but performs well on other classes. Another variant of Naive Bayes, Naive Bayes Complement (NBC, [38]), is applied by Rufo Mendo [39]. In particular, he presents a co-training version of NBC consisting of four steps: (a) train an NBC classifier with the training dataset; (b) classify the test dataset and take tweets with a confidence value higher than 0.9; (c) train the NBC with the training dataset and the tweets obtained in step (b); and (d) classify messages in the test dataset. The results obtained are not very encouraging. In particular, results obtained by co-training NBC are far worse than the results obtained by NBC without co-training.

Cordobés et al. [40] present a technique based on graph similarity to classify Twitter messages as being related to a specific topic. Each vertex in their graphs is a word stem, with weighted arcs representing the frequency of joint occurrence of two given stems, which can be pondered by the distance between the two stems in parse trees. In the training phase, a representative graph for each topic is built as the union of the graphs for the corresponding tweets. Then, for each tweet we wish to classify in the test phase, its graph is constructed and the most similar reference graph is retrieved using metrics inspired in PageRank [41] and HITS [42]. As a consequence, only one category can be assigned to each given tweet, and thus this method does not perform multitopic classification. An interesting result they have found is that using synonyms in the construction and retrieval of graphs has a negative impact on performance.

It is interesting to point out that, although the problem of topic categorization has been attacked from a great variety of angles on the TASS corpus, no approach has attained the level of performance shown by the system that we analyse in this article.

4.1. Pre-processing

Twitter is characterized by the use of a very informal language, and so we need to pre-process each tweet in order to normalize some lexical text features that are usually omitted or not respected. In particular, we homogenize punctuation marks (period, semicolon, etc.) by adding blanks when required. We must take into account that Twitter users often avoid spaces between a punctuation mark and the next word. This may affect the subsequent processing steps such as segmentation, tokenization or PoS-tagging. For example, following our pre-processing algorithm; the phrase ‘Hi,Jack!’ should be represented as ‘Hi, Jack!’

Some special Twitter characters such as ‘@’ (user names) or ‘#’ (hashtags) may negatively affect the performance of the rest of the natural language processing steps. This kind of symbol is especially confusing for tools such as the tokenizer or the parser. Thus, if the beginning or the end of the tweet contains a hashtag, we delete it completely. We hypothesize that this type of hashtag is only used to label the tweet, without providing any type of syntactic information. ⁹ If the hashtag is at some other position in the tweet, the pre-processor simply deletes the ‘#’ because it considers the hashtag refers to a relevant word or entity in the sentence (e.g. the sentence ‘The #screen is really good’ would be converted into ‘The screen is really good’)

With respect to usernames, we just delete the ‘@’ and capitalize the first letter, so that they will be treated as if they were real proper nouns.

4.2. PoS tagging

Part-of-speech (PoS) tagging is the process of marking up a word in a text as corresponding to a part-of-speech, based on both its definition and its context. Part-of-speech tags can be coarse-grained (when they only represent the grammatical category: noun, verb, adjective, etc.) or fine-grained (when they include additional morpho-syntactic information such as gender, number, tense, etc.). For the purpose of PoS tagging, we rely on a Brill tagger [45] implementation to label tweets. Concretely, we employed the version included in NLTK, ¹⁰ a framework for natural language processing written in Python. ¹¹ To train it, we used the Ancora corpus [46], a collection of tagged texts extracted from newspapers. Following common practice in the natural language processing community, 90% of the corpus was used as the training set and the remaining 10% as the test set.

The informal language used in Twitter messages is a source of confusion for a tagger. In this respect, an important issue in Spanish language is the use of acute accents, which are frequently omitted in web environments. For example, the noun ‘cálculo’ (‘calculus’ ) can be confused with the verb forms ‘calculo’ (‘I figure’ ) or ‘calculó’ (‘She computed’ ) when the acute accent is removed. The success of Twitter has prompted researchers to develop specific part-of-speech taggers for labelling tweets, mainly for English [47], but to the best of our knowledge this kind of tools are not available for the Spanish language. As a consequence, we have decided to consider a novel approach based on applying transformations to the training corpus in order to make it more similar to the kinds of text we found in Twitter. This approach has been recently tested successfully on user-generated web content: user reviews on cars, washing machines, books, cell phones, music, computers, movies and hotels [48].

In particular, we expanded the training set of the corpus in such a way that each text was duplicated without including any accent. We trained two models, the first one considering the original training set, and the second one taking into account the expanded set. The expanded training corpus makes the tagger able to handle words whose diacritical accents have been omitted owing to the characteristics of the language used in web environments. Thus, the new tagger is able to solve the ambiguities created by omitted accents, as it has seen training examples of them, and it can rely on the word context to tell between correctly spelled forms and those where an accent has been omitted. In contrast, the regular tagger has a stronger dependence on the form of the word, which is a weakness when the diacritical accent is not found. For example, the phrase ‘el medio jugo tan bien como siempre’ could be translated literally as ‘the half juice, as good as usual’, completely unrelated to the sports category, if we consider ‘jugo’ as a noun (‘juice’) and ‘medio’ (‘half’) as an adjective. However, the intended meaning corresponds to ‘the midfielder played as good as usual’, since ‘jugo’ is a misspelled form (without diacritical accent) of ‘jugó’, past tense form of the verb ‘jugar’ (‘to play’ sports of games, but not performing arts), and ‘medio’ is the noun for ‘midfielder’.

The evaluation on the test set showed that both taggers perform very similarly on regular texts (with accuracies of 95.86 and 95.71%, respectively), but practical performance on tweets was much better for the tagger built with the expanded training set, as detailed in the literature [48, Section 3].

4.3. Dependency parsing

We obtain the syntactic structure of tweets by means of dependency parsing. Given a sentence S = w₁… w_n, where w_i represents the word at the position i in the sentence, a dependency parser returns a dependency tree, a set of triplets {(w_i, arc_ij, w_j )} where w_i is the head term, w_j is the dependent and arc_ij represents the dependency type, which denotes the syntactic function that relates the head and the dependent. For example, for the simple sentence ‘I eat peanuts’ we obtain the triplets (‘eat’, subject, ‘I’) and (‘eat’, direct object, ‘peanuts’).

We rely on MaltParser [49], a data-driven dependency parser generator, to build our parser. The Ancora corpus was also used to train the model. We again employed 90% of the corpus as the training set, and the remaining 10% as the test set, achieving an LAS ¹² of 83.75% and a UAS ¹³ of 88.16%, which is coherent with the state of the art for Spanish language.

4.4. Psychometric knowledge

We also consider a perspective based on psychological knowledge. For this purpose, we rely on the dictionaries presented by Ramírez-Esparza et al. [50]. This lexicon distinguishes around 70 dimensions of human language. It provides information about psychometric properties of words (cognition mechanisms, anxiety, feelings, inhibition, sexuality, etc.), or even pragmatic information (words referring to oneself, exclamations, questions, etc.). In this way, the verb ‘imagine’ would represent a cognition mechanism and insight. This psychological linguistic resource is found in the LIWC software [51].

4.5. Extraction of linguistic features

The different kinds of linguistic information obtained after the application of the NLP pipeline will be used as features for automatic classification. The basic model, usually called a unigram model, considers each single word as a feature. This often becomes a simple baseline that obtains a decent performance, although it has an important disadvantage: it does not consider enough context. We will use the tweet ‘The fifth bowl. Nadal saluting to the Argentine team. And now to the public. Oe, oe, oe’ ¹⁴ as running example to illustrate the theoretical advantages and disadvantages of some of the proposed set of features, as far as possible. The word ‘team’ is a relevant feature to identify a tweet dealing with sports, but it could also refer to economics or politics if the phrase ‘economic team’ appears. Furthermore, we can easily propose an example where the words ‘team’ and ‘economic’ appear, but referring to a sports tweet: ‘His favourite team does not give any economic benefits’. Therefore, considering each single word as a feature to feed the classifier is not the best strategy to correctly deal with topic classification.

A way to solve this issue is to employ larger n-grams, ¹⁵ such as bigrams, that is, sequences of two contiguous words. The main drawback of these types of features is their sparsity, which may decrease performance if the training set is not representative and large enough. Continuing with our running example, instead of the unigram ‘team’ we would have the bigram ‘Argentine team’, which is probably a better indicator of a sports tweet. The same will be true if we again take the phrase ‘economic team’, where we now obtain a unique and more discriminative bigram, instead of separate features that may confuse the classifier, as we detailed above.

We also take part-of-speech tags as a possible complement to improve the performance of other sets of features. PoS tags are not intuitively proper features, by themselves, to difference between various topics. However, some of them can be helpful when they are used jointly with other attributes. For example, the proper name tag can be more frequent in domains such as movies, where users often refer to the director, the actors or the producer to implicitly refer to the quality of the art, than in other domains such as technology, for example.

Moreover, we consider the extraction of the psychometric properties present in a text. We identify psychological features relying on LIWC dictionaries [51]. These lexicons allow us to significantly reduce the number of features, avoiding sparsity problems and assigning words to very specific areas. Their main drawback is their low recall, limited to the terms considered in the lexicons.

Finally, we propose relating terms by means of dependency parsing. More specifically, we rely on the concept of generalized dependency triplets, originally presented as back-off dependency triplets [52], and later enhanced and applied successfully to polarity classification in Vilares et al. [14], where the authors obtained a significant improvement over a pure bag-of-words approach. Given a triplet (w_i, arc_ij, w_j), generalizing it consists of abstracting the information provided by either the term i or the term j, obtaining a more general dependency triplet of the form (b(w_i, x), arc_ij, b(w_j, x)), where b is a generalization function and x the parameter indicating what to return: a part-of-speech tag, a psychometric property, a lemma, a word form or even nothing (completely eliminating that element). We also consider the option of removing the dependency type arc_ij in order to further reduce sparsity, obtaining a syntactic bigram [52]: two words, not necessarily contiguous, related by a syntactic relation. If we again take our running example, the dependency triplet (team, modifier, Argentine) could be generalized in different ways, as shown in Table 2.

OPEN IN VIEWER

Table 2.

Examples of generalization for the dependency triplet (team, modifier, Argentine)

Desired generalization	Result
(b(team, PoS-tag), delete(modifier), Argentine)	(common name, , Argentine)
(b(team, psychometry), modifier, Argentine)	(leisure, modifier, Argentine)
	(sports, modifier, Argentine)
	…
(b(team, psychometry), modifier, b(Argentine, PoS-tag))	(leisure, modifier, adjective)
	(sports, modifier, adjective)…

5.1. Dataset description

The TASS 2013 General Corpus is a collection of tweets that has been specifically annotated to perform text analytics tasks. It was presented at the Workshop on Sentiment Analysis at SEPLN [6]. It is a collection of tweets in Spanish written by 150 public figures, such as soccer players, politicians or journalists from Spain, Mexico, Colombia, Puerto Rico, USA and other countries. Message dates range from November 2011 to March 2012. The corpus is composed of a training set and a test set which contain 7,219 and 60,798 tweets, respectively. Each tweet is annotated with one or more topics, which involve up to 10 categories: films, soccer, economics, entertainment, literature, music, politics, sports (other than soccer), technology and other. We take the ‘other’ class as the default class (i.e. if our one vs all strategy always discards the topic under study, the system will assign the default topic to the tweet). The gold standard has been generated by a pooling of the submissions, followed by a human review by TASS organization for the thousands of ambiguous cases. The Appendix shows the topic distribution of tweets in the collection, for both training and test sets. The classes of the training set are unbalanced. This may be interesting from a real-world environment and industry point of view, since some topics are often more popular than others, and therefore it may be difficult to build a balanced training set. In this situation, from a performance perspective, supervised methods tend to present biases when there are large differences in the number of training samples for each class. Thus, we decided to apply oversampling to the minority categories.

5.2. Evaluation metrics

We evaluate our approaches by means of the standard metrics for multilabel classification: Hamming loss (HL), label-based accuracy (LBA) and exact match (EM). They are calculated according to equations (1)–(3), where:

Hamming loss = 1 | D | ∑ i = 1 | D | | Y i Z i | | L |

(1)

Label - based accuracy = 1 | D | ∑ i = 1 | D | | Y i ∩ Z i | | Y i ∪ Z i |

(2)

Exact match = no . of instances exactly classified no . of instances

(3)

These metrics reflect different aspects whose relevance should depend on the type of application. We will illustrate the behaviour of these metrics with the following example. Take two tweets, t1 and t2, where:

Hamming loss is a loss function, thus its optimal value is zero. It measures the number of wrong labels with respect to the total number of labels, but does not appropriately reflect the percentage of the correctly predicted labels. Calculating the hamming loss for t1 and t2, we obtain:

H L t 1 | t 1 a Δ t 1 p | | L | = | { sports , politics } | | L | = 2 | L |

(4)

H L t 2 | t 2 a Δ t 2 p | | L | = | { sports , politics } | | L | = 2 | L |

(5)

and so HL_t1 = HL_t2, even though t2 has a larger percentage of successful predicted topics.

Label-based accuracy is a measure able to harmonize the number of not assigned topics with respect to the wrongly selected ones. Taking again t1 and t2 as example, the LBA for each one would be:

LB A t 1 | t 1 a ∩ t 1 p | | t 1 a ∪ t 1 p | = | { economy } | | { politics , sports , economy } | = 1 3

(6)

LB A t 2 | t 2 a ∩ t 2 p | | t 2 a ∪ t 2 p | = | { films , entertaiment , football , economy } | | { sports , politics , films , entertaiment , football , economy } | = 2 3

(7)

concluding that the LBA for t2 is better than for t1.

Finally, a special case of LBA is the Exact match metric, which is a more restrictive metric owing to the fact that it only considers a multilabel classification as successful when Y_i = Z_i, that is Y_i∩ Z_i = Y_i∪ Z_i. If we calculate the exact match for t1 and t2 we will obtain in both cases a value of 0. Note that, if we take an example i where Y_i = Z_i, then the EM, LBA and HL would be 1, 1 and 0, respectively.

Additionally, we will also consider two more metrics, used by TASS organizers, calculated according to equations (8) and (10): At least one scores a classification as valid whenever at least one topic is right, whereas Match all considers a multilabel classification valid when a superset of the actual topic set has been predicted.

At least one = 1 | D | ∑ i = 1 | D | f ( i )

(8)

where

f ( i ) = { 1 if Y i ∩ Z i ≠ ∅ 0 if Y i ∪ Z i ≠ ∅

(9)

Match all = 1 | D | ∑ i = 1 | D | g ( i )

(10)

where

g ( i ) = { 1 if Y i ⊆ Z i 0 if Y i ⊆ Z i

(11)

The main drawback of these two measures is that it is possible to obtain a ‘perfect result’ by assigning all possible categories to each tweet. Therefore, they are less robust with respect to academic misconduct.

5.3. Supervised classifier

To be able to assign several topics to the same tweet, we carried out a one vs all strategy: given n topics, this perspective proposes to train n classifiers where each one makes it possible to differentiate a topic i, where i ∈ n, from the others. Thus, if we plan to discover if a text is talking about a topic X, Y or Z, or several of them at the same time, we would create three classifiers: topic X vs Other, topic Y vs Other and topic Z vs Other.

We use the WEKA data mining software [53] to build machine learning models. As a classifier, we have chosen to work with SMO [29], an implementation of Support Vector Machines.

5.4. Feature filtering

The number of features derived from n-grams, psychometric properties and dependency triplets results in a high multidimensional feature space. To address this issue, we test the use of an information gain filter, which measures the relevance of an attribute with respect to the class. Only features with an information gain greater than zero are selected to then feed a supervised classifier. Given the huge number of attributes and the low value of most of them, we cannot ensure the optimal selection of attributes, but experimental results suggest that applying an information gain filter improves performance with respect to omitting it. We use the WEKA attribute selection tools to apply the information gain filtering.

5.5. Feature models

As a starting point, we consider several basic feature models based on lexical, syntactic, psychometric and semantic knowledge. We will measure their performance when they are used independently, and then we will combine them in order to improve either the Hamming loss, the label-based accuracy or the exact match:

Finally, we will combine the best-performing features with syntactic features: we represent syntactic information by means of generalized dependency-based features. We test different levels of generalization over the head and the dependent word of a dependency triplet, including lemmas, psychometric properties and fine-grained PoS-tags.