A study of Turkish emotion classification with pretrained language models

Abstract

Emotion classification is a research field that aims to detect the emotions in a text using machine learning methods. In traditional machine learning (TML) methods, feature engineering processes cause the loss of some meaningful information, and classification performance is negatively affected. In addition, the success of modelling using deep learning (DL) approaches depends on the sample size. More samples are needed for Turkish due to the unique characteristics of the language. However, emotion classification data sets in Turkish are quite limited. In this study, the pretrained language model approach was used to create a stronger emotion classification model for Turkish. Well-known pretrained language models were fine-tuned for this purpose. The performances of these fine-tuned models for Turkish emotion classification were comprehensively compared with the performances of TML and DL methods in experimental studies. The proposed approach provides state-of-the-art performance for Turkish emotion classification.

Keywords

Emotion analysis emotion extraction fine-tuning pretrained language model Turkish language

1. Introduction

Emotion classification is a subfield of text mining. It classifies a word, sentence or text according to the emotions contained in the discourse. Generally, the basic emotion models of Ekman [1] and Plutchik [2] are used in emotion classification studies. The Ekman [1] emotion model consists of six basic classes of emotions: ‘anger, sadness, joy, disgust, fear and surprise’. Moreover, the Plutchik [2] emotion model is composed of eight basic emotions: ‘anger, sadness, joy, disgust, fear, surprise, trust and anticipation’.

The feature engineering processes in traditional machine learning (TML) methods cause some important information in the text to be lost, which decreases the classification performance [3]. Some emotion classification methods in which the inputs are represented, such as the bag of words [4], cause the loss of the order and integrity of the words in the sentence [5].

Deep learning (DL) methods take the raw text as input and preserve the context of the words that coexist. In this way, it is possible to classify emotions more successfully than when using TML methods. However, the performance of DL methods depends on the size of the data set used. Nevertheless, the number of data sets for emotion classification is very limited in the literature.

Turkish is a morphologically rich language since it is an agglutinative language and has the ability to produce new words with the help of suffixes. Relatively more samples are needed to create a DL model for Turkish text analysis studies. Thus, it is necessary to use as many written sources as possible to cover the whole vocabulary while training a DL model for Turkish.

In recent years, word embeddings and pretrained language model approaches have changed the direction of studies in natural language processing fields. The training process does not start from scratch thanks to the use of these methods in deep networks. Thus, the generalisation of the model and the learning of the problem are accelerated. In this context, there are sentence level–based methods [6, 7] and word embedding methods [8,9,10] providing prior knowledge at the word level. General purpose pretrained language model approaches [11, 12] were developed as a result of studies aiming to transfer more prior knowledge from word meanings. Pretrained language models aim to learn semantic, syntactic and grammar rules and usage habits along with the word meanings in the text. The requirement of a large-sized emotion data set in DL methods is reduced thanks to the pretrained language models.

In this study, emotion classification was performed on Turkish texts using pretrained language models. The pretrained language models, which are the Bidirectional Encoder Representations from Transformers (BERT) [11] and BERTurk [13] developed for Turkish, were adapted to the emotion classification problem for Turkish. TML, DL and the two pretrained models were applied to the problem. The obtained performances of the applied models were investigated comparatively. It was observed that the pretrained language models are superior to the others.

1.1. Related studies

Emotion classification studies are mainly conducted for the English language [14,15,16,17]. Therefore, most of the available data and tools are for English. There are various studies conducted on languages such as Chinese [18], Arabic [19], French [20] and Romanian [21].

There are studies using pretrained word embeddings [22] and studies [16] using pretrained language models for English. As a result of the emotion analysis performed on an English data set using BERT, the success rate measured by the F₁ metric was increased from 58% to 74% [16]. Various studies have been conducted on English emotion analysis using this approach [23,24,25].

The first emotion classification study on Turkish [26] was conducted using TML in 2013. The data set used in this study was derived from the International Survey on Emotion Antecedents and Reactions (ISEAR) [27] data set in English by translating 4265 sample sentences belonging to the four classes of ‘joy, sadness, anger and fear’ into Turkish. Experiments performed using different feature combinations showed that a success rate of up to 81% was achieved in the general accuracy and F₁ metrics.

Another study using TML methods [28] was conducted on Turkish Twitter posts. In the study based on Ekman’s emotion model, a data set consisting of 6000 samples was created by taking 1000 samples for each emotion. In experimental studies in which different ML methods with various feature combinations were tested, the support-vector machine (SVM) [29] achieved the highest performance in the term count metric with 70%.

The Turkish emotion data set (TREMO), which is based on Ekman’s model and consists of 27,350 samples, was created as a result of a questionnaire conducted on 4709 people in the study [30, 31]. In the experiments performed on TREMO, the highest performance was achieved with the SVM with an 86% general accuracy. In the continuation study [32], an emotion lexicon was produced using TREMO, and lexicon-based emotion analysis was performed. The lexicon-based approach proposed in the study showed competitive performance against the TML methods with an accuracy of 91%.

In another study [33], emotion classification was conducted on Turkish using DL methods. The Turkish Twitter emotion data set (TURTED) consisting of 195,000 samples was created using emotion keywords according to Ekman’s model. An artificial neural network, a convolutional neural network (CNN) and long short-term memory methods were applied on TURTED and TREMO in the experimental study. The CNN achieved the highest performance on TURTED with 74% accuracy. The methods achieved significantly better performance in the experiments conducted on the TREMO data set compared with the experiments conducted on TURTED. No classification study with pretrained language models on Turkish has been found in the related literature.

2. The proposed approach

In this study, the transfer learning method was applied to BERT and its variations, which are pretrained language models. The models were fine-tuned using the TREMO data set for emotion classification in Turkish. The scheme of the pretrained language model in Figure 1 describes the common structure of the pretrained language models used in the study. The pretrained language model was trained using the training set so that the model could classify emotions. The weights of the pretrained language model were optimised for Turkish emotion classification according to the samples in the training set. The performance of the customised model was measured using the test set.

Figure 1.

Proposed fine-tuning method.

Transfer learning adapts the knowledge gained from a source task to the target task. Transfer learning can be performed in many types. However, we use pretrained language models adapted to a downstream task in this study. The adaptation process is performed in three steps. In the first step, in order to conduct transfer learning with a pretrained language model, a suitable model is selected for the task. We selected BERT for this study because it is a successful model for natural language understanding tasks. The second step is the selection of an appropriate layer of the model for adaptation. The last layer was selected as the adaptation layer due to its simplicity and effectiveness. In the third step, the transfer strategy between fine-tuning (unfrozen parameters) and feature extraction (frozen parameters) should be determined. Fine-tuning was chosen as the transfer strategy in this study because it is more suitable for classification tasks.

In this study, the prior knowledge in pretrained language models in the BERT architecture was fine-tuned for emotion classification. The [CLS] token represents the entire sentence in the BERT representation, and it is created for classification tasks. The [CLS] token in the last hidden layer of the model was used for fine-tuning in this study. A softmax layer was located next to the [CLS] token to determine the probability that the entry belongs to class c (equation (1))

p (c | s) = softmax (W \cdot h + b)

(1)

where W is the weight matrix and b is the bias. In this way, the weights obtained from pretrained language models were rearranged according to the emotion classification. The softmax function ensures that the values of the output vector elements are in the range of [0,1] and that their sum is 1.

The pretrained language models, the multilingual BERT [11] and BERTurk [13], were adapted to emotion classification in this study. The BERT [11] and DistilBERT [34] language models pretrained in English were used to verify the proposed approach.

The SimpleTransformer¹ library using the transformer [35] infrastructure was used for the fine-tuning operations in the implementation of the proposed approach. The system parameters were set as ‘batch-size:8’, ‘learnin-rate:4e-5’ and ‘num-train-epoch:5’. Calculations were conducted on the graphics processing unit (GPU) in the Colab² infrastructure. Subsequently, the trained models were tested on the test sets, and the evaluation metrics and results were reported.

3. Data sets

TREMO [30], the emotion data set in Turkish created and verified manually for Turkish emotion classification, was used. The Twitter Emotion Corpus (TEC) [36] and Blogs [37] data sets used for English emotion classification were also used for the verification. The data sets were split into three parts as ‘train, dev and test’ for the experiments (Table 1).

Table 1.

Data sets’ statistics.

Data sets	Split	Anger	Disgust	Fear	Happy	Sadness	Surprise	Total
TREMO	Train	2657	2036	2471	2941	2824	1689	14,618
TREMO	Dev	885	679	824	980	942	563	4873
TREMO	Test	1181	905	1098	1308	1255	751	6498
TREMO	Total	4723	3620	4393	5229	5021	3003	25,989
TEC	Train	1007	495	1801	5254	2462	2436	13,455
TEC	Dev	239	115	459	1315	586	652	3366
TEC	Test	308	151	550	1663	775	756	4023
TEC	Total	1554	761	2810	8232	3823	3844	21,024
Blogs	Train	110	105	67	343	105	73	803
Blogs	Dev	26	30	19	85	29	17	206
Blogs	Test	43	37	29	108	39	25	281
Blogs	Total	179	172	115	536	173	115	1290

The TREMO Turkish emotion data set is labelled with Ekman’s six emotions and has a balanced distribution. The data set was created as a result of a questionnaire conducted on 4709 people. In the questionnaire, the volunteers were asked to briefly write a memory or experience for each emotion of Ekman’s model. The 27,350 collected entities were labelled by at least three different people. After labelling, unanimous and majority entries were eliminated.

The TEC data set consists of 21,024 Twitter posts labelled with Ekman’s emotion classes for English emotion classification. The TEC data set was created by selecting posts containing hashtags such as #anger, #happy and #fear without expert evaluation. The Blogs data set consists of 4090 sentences selected from 173 blogs. The Blogs data set has been labelled with the classes of Ekman’s emotions and neutral by some experts who assigned only one label to each sentence. A total of 2800 sentences in the neutral class were eliminated for the experiments in this study.

4. Experimental studies

The experiments were conducted using the TML, DL, and DL with word embedding methods and fine-tuned pretrained language models consecutively. Moreover, all the experiments were repeated by taking both the full word and lemma in order to measure the effect of lemmatisation in Turkish. Turkish Stemmer³ was used for the lemmatisation process.

First, emotion classification on Turkish was performed using the Naive Bayes (NB) [38] and SVM [29] TML methods in order to create a baseline and make some comparisons. Since raw text cannot be input in conventional methods, a feature selection process was performed. A total of 4000 features with the highest representation ability according to term frequency–inverse document frequency (TF–IDF) weights were selected among the words and n-grams (2, 3 and 4) in the training set and used as the inputs in the TML experiments.

In the experiments performed with the GRU [39], LSTM [40] and attention [35] DL algorithms, emotion classifications were conducted using their networks bidirectionally. These DL experiments were repeated after the weights of FastText [10] pretrained word embeddings were appended to the embedding layer of the deep networks. Raw text was used as the input, and the models were trained for 30 epochs in the experiments. Softmax [41] was used as the activation function, Kullback–Leibler divergence [42] was used as the loss function, and Nadam [43] was used as the optimiser.

Finally, the last experiments were conducted using the pretrained language models that were adapted to emotion analysis. The input texts were converted into BERT representations. In addition, all experiments were also performed on the TEC and Blogs English emotion data sets in order to validate the developed model in terms of the same hyperparameters. The obtained results were compared with the results obtained on TREMO.

5. The performance metrics

The performance of the proposed approach was measured with the precision (P), recall (R), F₁ and accuracy (Acc) (equations (2)–(5))

Precision (P) = \frac{TP}{TP + FP}

(2)

Recall (R) = \frac{TP}{TP + FN}

(3)

F_{1} = \frac{2 PR}{P + R}

(4)

Acc = \frac{TP + TN}{TP + TN + FP + FN}

(5)

where TP represents the true positives, TN represents the true negatives, FP represents the false positives, and FN represents the false negatives.

6. The results

The proposed pretrained language model adapted to emotion classification in Turkish achieved a success rate of 92% as measured by the Acc on TREMO (Table 2). This score is the state-of-the-art result in the literature.

Table 2.

Turkish emotion classification results with the fine-tuned BERTurk [13] model.

Emotion	Train			Dev			Test
	P	R	F ₁	P	R	F ₁	P	R	F ₁
Anger	100	100	100	92	93	93	90	92	91
Disgust	100	100	100	94	95	94	95	95	95
Fear	99	99	99	90	93	92	91	94	92
Happy	100	100	100	94	93	94	95	95	95
Sadness	100	100	100	92	92	92	92	90	91
Surprise	100	99	100	92	88	90	91	87	89
Accuracy			100			92			92
Macro-average	100	100	100	92	92	92	92	92	92
Weighted-average	100	100	100	92	92	92	92	92	92

As seen in Table 2, a success rate of 89% was achieved, even for the most unsuccessful class ‘surprise’. Furthermore, a success rate of 95% was achieved for the ‘disgust’ and ‘happy’ classes. The performances achieved in all classes are similar in both the macro- and weighted-average results. This indicates that the performance of the model is independent of the number of samples belonging to the classes.

Table 3 shows the sample numbers of all classes in the TREMO data set and the confusion matrix of the proposed model. The numbers of correctly predicted labels in the classes are quite high. The lowest performance was obtained for the ‘surprise’ class with the smallest sample. The relatively low success rate in this class is related to the number of samples considering that the numbers of samples in the training and dev sets also have similar distributions (Table 1).

Table 3.

Test set confusion matrix.

Emotion		Anger	Disgust	Fear	Happy	Sadness	Surprise	Total	Recall
Real label	Anger	1089	23	12	6	40	11	1181	92%
	Disgust	19	856	20	0	5	5	905	95%
	Fear	19	11	1027	6	28	7	1098	94%
	Happy	13	3	11	1237	16	28	1308	95%
	Sadness	43	8	41	14	1133	16	1255	90%
	Surprise	23	4	12	44	14	654	751	87%
		Predicted label						6498	92%

Four pretrained language models were adapted to the emotion classification in Turkish. The multilingual BERT was developed for languages that have sufficient resources on Wikipedia. The resources and methods specific to Turkish were not used in the development of this model. The DistilBERT approach, which aims to reduce the size of the pretrained language model by distilling it, uses the BERT model as a source model. BERTurk, which is developed using Turkish-specific resources and tools [13], is more successful in Turkish emotion classification. DistilBERTurk is the distilled version of BERTurk. Although the size of DistilBERTurk is quite small, the model yields competitive results (Table 4).

Table 4.

Turkish emotion classification results with the fine-tuned pretrained language model.

Pretrained language models	Train			Dev			Test
	P	R	F ₁	P	R	F ₁	P	R	F ₁
BERT (multilingual)	99	99	99	89	89	89	89	89	89
DistilBERT (multilingual)	99	99	99	89	88	89	89	88	88
BERTurk	100	100	100	92	92	92	92	92	92
DistilBERTurk	100	100	100	91	91	91	90	91	91

The performances of the DL methods, whose training process starts from scratch, depend on the numbers of samples and epochs. The DL methods and the DL methods weighted with FastText embeddings were compared (Table 5). Pretrained word embeddings provided very strong prior knowledge. DL with FastText achieved a success rate that was approximately 7% better than that of DL without FastText according to the results.

Table 5.

Effects of word embeddings.

Method	DL (scratch)			DL + FastText
	P	R	F ₁	P	R	F ₁
GRU	84	83	83	90	89	90
LSTM	85	84	84	90	89	89
Attention	85	85	85	86	86	86

DL: deep learning.

The proposed pretrained emotion model is more successful than the other learning methods in [30], as seen in the comparison in Table 6. The effects of using words and lemmas as input on the performance of the SVM were also investigated. Using lemmas as input does not have a significant effect on the results. When the performances of the SVM and Bi-GRU are compared, there are no significant differences in the results. However, TML methods require more preprocessing, such as feature engineering. However, GRU with FastText has a better performance than SVM.

Table 6.

Proposed model versus others.

	SVM reported in [30]			SVM-word			SVM-lemma			Bi-GRU			Bi-GRU + FastText			BERTurk
	P	R	F ₁	P	R	F ₁	P	R	F ₁	P	R	F ₁	P	R	F ₁	P	R	F ₁
Anger	85	87	86	79	85	82	77	84	80	85	78	81	85	90	87	90	92	91
Disgust	95	91	93	90	85	87	89	85	87	92	85	89	94	92	93	95	95	95
Fear	89	88	88	89	85	87	87	85	86	86	88	87	91	90	91	91	94	92
Happy	81	89	85	83	85	84	82	86	84	77	89	82	90	92	91	95	95	95
Sadness	85	81	83	76	82	79	78	80	79	80	80	80	88	87	88	92	90	91
Surprise	88	82	85	91	76	83	91	76	83	87	79	83	91	85	88	91	87	89
Macro-average	87	86	86	85	83	84	84	83	83	85	83	84	90	89	90	92	92	92
Weighted-average	87	86	86	84	83	83	83	83	83	84	84	84	90	90	90	92	92	92

SVM: support-vector machine.

All experiments were conducted on the TEC and Blogs data sets in English to verify the created emotion classification model. The obtained results for these data sets are similar to the results for TREMO (Figure 2). TML methods have the lowest performance below DL methods, similar to the Turkish methods. The performances of the DL methods increased drastically when word embeddings were transferred to them. When the transferred prior knowledge was enhanced using pretrained language models, the highest performances were obtained.

Figure 2.

Emotion classification results.

The results of the experiments on the English data set have a similar pattern to the experiments on TREMO (Table 7). The emotion classifications on the Turkish data set are more successful than those on English data sets. This occurs because TREMO was created and verified manually while the TEC and Blogs data sets were derived from social media posts and blogs, respectively. The fact that some classes have very few samples in English data sets (Table 1) also causes this difference to be larger. Furthermore, 30 epochs are sufficient to learn the Turkish emotion model but not the English model in DL methods according to the results on the training data set.

Table 7.

Turkish and English results with the macro-average F₁ metric.

Method	TREMO			TEC			Blogs
	Train	Dev	Test	Train	Dev	Test	Train	Dev	Test
SVM	91	84	84	64	44	44	95	42	40
NB	87	83	82	43	34	34	27	10	13
GRU	99	84	83	91	43	43	100	37	44
GRU + FastText	94	90	90	79	50	47	100	63	71
LSTM	100	84	84	53	38	38	100	47	47
LSTM + FastText	92	90	89	60	48	47	99	58	55
Attention	100	85	85	64	42	42	100	42	41
Attention + FastText	100	86	86	75	43	42	100	50	48
BERTurk/BERT	100	92	92	100	54	54	100	76	77
DistilBERTurk/DistilBERT	100	91	91	100	52	52	100	73	77

SVM: support-vector machine; NB: Naive Bayes.

The proposed pretrained emotion classification model for Turkish yields more successful results than the results of other methods and previous studies. State-of-the-art results were obtained as a result of the experiments without using any attribute specific to the emotional context.

7. Conclusion

In this study, we aimed to classify emotion in Turkish texts with a pretrained language approach. Experimental studies show that the proposed pretrained emotion model has the highest success rate compared with the methods used in previous studies. This study is noteworthy due to the unique nature of Turkish and the lack of studies on this language in this field. In addition, the models proposed for multilingual and other languages are inadequate for Turkish texts. The proposed study will enable high-level artificial intelligence tasks such as more accurate public opinion polling, customer relationship management, brand management, cyberbullying detection, the determination of election tendencies and the determination of partisan comments in Turkish media. In addition, it will be a resource for researchers in this field and can be used in Turkish text mining tasks. Future studies should aim to support the proposed models with features specific to the emotional context and thus to perform more successful emotion analysis. Future research should also aim to adapt the proposed model to other text mining subtasks such as emotion analysis, named entity recognition and question answering.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

ORCID iDs

Alaettin Uçan

Murat Dörterler

Notes

References

Ekman

. An argument for basic emotions. Cogn Emot 1992; 6: 169–200.

Plutchik

. The emotions. New York: University Press of America, 1991.

LeCun

Bengio

Hinton

. Deep learning. Nature 2015; 521: 436–444.

Goldberg

. Neural network methods for natural language processing. Synth Lect Hum Lang Technol 2017; 10: 1–309.

Deng

Liu

. Deep learning in natural language processing. Singapore: Springer, 2018.

Mikolov

. Distributed representations of sentences and documents. In: Proceedings of the ICML 2014 31st international conference on international conference on machine learning, Volume 32. Beijing, China, 22–24 June 2014.

Lin

Feng

Santos

et al. A structured self-attentive sentence embedding. arXiv Prepr: 170303130.

Mikolov

Corrado

Chen

et al. Efficient estimation of word representations in vector space. In: Proceedings of the International Conference on Learning Representation (ICLR 2013), Scottsdale, AZ, 2–4 May 2013, pp. 1–12. https://www.researchgate.net/publication/319770439_Efficient_Estimation_of_Word_Representations_in_Vector_Space

Pennington

Socher

Manning

. Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language Processing (EMNLP), Doha, Qatar, 25–29 October 2014, pp. 1532–1543. Stroudsburg, PA: Association for Computational Linguistics.

10.

Bojanowski

Grave

Joulin

et al. Enriching word vectors with subword information. Trans Assoc Comput Linguist 2017; 5: 135–146.

11.

Devlin

Chang

M-W

Lee

et al. BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, Volume 1, Minneapolis, MI, 2–7 June 2018.

12.

Yang

Dai

Yang

et al. XLNet: generalized autoregressive pretraining for language understanding. In: Proceedings of the 33rd conference on neural information processing systems, Vancouver, BC, Canada, 8–14 December 2019, pp. 5753–5763. https://proceedings.neurips.cc/paper/2019/file/dc6a7e655d7e5840e66733e9ee67cc69-Paper.pdf

13.

Schweter

. BERTurk – BERT models for Turkish, https://zenodo.org/record/3770924#.X9zcjFUzbIU (accessed 27 April 2020).

14.

Kratzwald

Ilic

Kraus

et al. Decision support with text-based emotion recognition: deep learning for affective computing, http://arxiv.org/abs/1803.06397 (accessed 16 March 2018).

15.

Majumder

Poria

Hazarika

et al. DialogueRNN: an attentive RNN for emotion detection in conversations, http://arxiv.org/abs/1811.00405 (accessed 1 November 2018).

16.

Al-Omari

Abdullah

Shaikh

. EmoDet2?: Emotion Detection in English Textual Dialogue using BERT and BiLSTM Models. In: Proceedings of the 2020 11th international conference on information and communication systems (ICICS), Irbid, Jordan, 7–9 April 2020, pp. 226–232. New York: IEEE.

17.

Jiang

Liu

Yao

et al. A hybrid recommendation model in social media based on deep emotion analysis and multi-source view fusion. J Cloud Comput 9. Epub ahead of print 7 October 2020. DOI: 10.1186/s13677-020.

18.

Tian

Lai

et al. Deep learning based emotion analysis of microblog texts. Inf Fusion 2020; 64: 1–11.

19.

Abdul-Mageed

Alhuzli

Elhija

et al. DINA: a multidialect dataset for Arabic emotion analysis. In: Proceedings of the 2nd workshop on Arabic corpora and processing tools, 24 May 2016, P. 29. http://www.lrec-conf.org/proceedings/lrec2016/workshops/LREC2016Workshop-OSACT2_Proceedings.pdf

20.

Abdaoui

Azé

Bringay

et al. FEEL: a French expanded emotion lexicon. Lang Resour Eval 2017; 51: 833–855.

21.

Briciu

Lupea

. RoEmoLex – a Romanian emotion lexicon. Stud Univ Babe?-bolyai Inform 2017; 62: 45–56.

22.

Naderalvojoud

Ucan

Akcapinar Sezer

. HUMIR at IEST-2018: Lexicon-sensitive and left-right context-sensitive Bi-LSTM for implicit emotion recognition. In: Proceedings of the 9th workshop on computational approaches to subjectivity, sentiment and social media analysis, Brussels, 31 October 2018, pp. 182–188. Stroudsburg, PA: Association for Computational Linguistics.

23.

Huang

Y-H

Lee

S-R

M-Y

et al. EmotionX-IDEA: emotion BERT – an affectional model for conversation. arXiv Prepr: 190806264.

24.

Huang

Trabelsi

Zaiane

. ANA at SemEval-2019 task 3: contextual emotion detection in conversations through hierarchical LSTMs and BERT. In: Proceedings of the 13th international workshop on semantic evaluation, Minneapolis, MI, 6–7 June 2019.

25.

Tomihira

Otsuka

Yamashita

et al. Multilingual emoji prediction using BERT for sentiment analysis. Int J Web Inf Syst 2020; 16: 265–280.

26.

Boynukalin

Karagoz

. Emotion analysis on Turkish texts. In: Gelenbe

Lent

(eds) Information Sciences and Systems 2013. Cham: Springer, 2013, pp. 159–168.

27.

Wallbott

Scherer

. How universal and specific is emotional experience? Evidence from 27 countries on five continents. Inf Int Soc Sci Counc 1986; 25: 763–795.

28.

Demirci

. Emotion analysis on Turkish tweets. Ankara, Turkey: Middle East Technical University, 2014.

29.

Cortes

Vapnik

. Support-vector networks. Mach Learn 1995; 20: 273–297.

30.

Tocoglu

Alpkocak

. TREMO: a dataset for emotion analysis in Turkish. J Inf Sci 2018; 44: 848–860.

31.

Alpkoçak

Tocoglu

Çelikten

et al. Türkçe metinlerde duygu analizi için farklı makine oğrenmesi yöntemlerinin karşılaştırılması. Deu Muhendis Fak Fen Ve Muhendis 2019; 21: 719–725.

32.

Tocoglu

Alpkocak

. Lexicon-based emotion analysis in Turkish. Turkish J Electr Eng Comput Sci 2019; 27: 1213–1227.

33.

Tocoglu

Ozturkmenoglu

Alpkocak

. Emotion analysis from Turkish tweets using deep neural networks. IEEE Access 2019; 7: 183061–183069.

34.

Sanh

Debut

Chaumond

et al. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv Prepr: 191001108.

35.

Vaswani

Shazeer

Parmar

et al. Attention Is All You Need, http://arxiv.org/abs/1706.03762 (accessed 12 June 2017).

36.

Mohammad

. # Emotional tweets. In: Proceedings of the first joint conference on lexical and computational semantics-volume 1: proceedings of the main conference and the shared task, and volume 2: proceedings of the sixth international workshop on semantic evaluation, Montréal, QC, Canada, 7–8 June 2012, pp. 246–255. Stroudsburg, PA: Association for Computational Linguistics.

37.

Aman

Szpakowicz

. Identifying expressions of emotion in text. In: Matoušek

Mautner

(eds) International conference on text, speech and dialogue. Berlin, Heidelberg: Springer, 2007, pp. 196–205.

38.

Webb

. Encyclopedia of machine learning Naïve Bayes. In: Sammut

Webb

(eds) Boston, MA: Springer US, pp. 713–714.

39.

Chung

Gulcehre

Cho

et al. Empirical evaluation of gated recurrent neural networks on sequence modeling, https://arxiv.org/abs/1412.3555 (accessed 11 December 2014).

40.

Hochreiter

Schmidhuber

. Long short-term memory. Neural Comput 1997; 9: 1–32.

41.

Goodfellow

Bengio

Courville

. Deep learning. Cambridge, MA: MIT press, 2016.

42.

Joyce

. Kullback–Leibler divergence. In: Lovric

(ed.) International encyclopedia of statistical science. Berlin, Heidelberg: Springer Berlin Heidelberg, 2014, pp. 720–722.

43.

Dozat

. Incorporating Nesterov momentum into Adam. 2014. https://openreview.net/pdf?id=OM0jvwB8jIp57ZJjtNEZ