Sentiment and Emotion Network Analysis—SENA: An Analytic Framework and No-Code Software to Integrate Natural Language Processing,Linguistic Analysis,and Data Science with Qualitative and Mixed-Methods Research

Texts as Unstructured Data

Although texts are classified as unstructured data (Nandwani & Verma, 2021), texts are not inherently unstructured. Instead, textual or written data are guided by linguistic and grammatical rules that we as humans have learned to decode (De Beaugrande & Dressler, 1981). From this view, and as briefly noted above, the reason why texts are considered unstructured from a quantitative analytical perspective, is that in their original format, written information is not stored in matrices or rectangular databases with n by m dimensions–i.e., rows and columns, respectively. The main implication of this lack of rectangular structure is that for machine learning, classification, and other statistical analyses of textual data to be possible, specialized pre-processing tools must be applied before we are able to extract the meanings embedded in vast amounts of unstructured, written information.

The analytic process followed to structure textual data is referred to as natural language processing (NLP). NLP is “a collection of computational techniques for automatic analysis and representation of human languages, motivated by theory” (Chowdhary, 2020, p. 604). As briefly stated, this use of computer algorithms is to a great extent justified precisely by the aforementioned vast amount of data currently available and constantly being generated. Essentially, with NLP techniques, we can normalize and store textual data in matrices or databases so that we can start applying computer- and statistical algorithms (Chowdhary, 2020; Kumar & Bhatia, 2013) designed to extract, learn, or detect potentially useful information that highlights the structure embedded in vast amounts of data.

NLP and text analysis may include word level analysis in the form of term frequencies (word clouds or tag clouds) as well as a variety of machine learning text classification tools (Chowdhary, 2020; González Canché, 2023a, 2023b). It may also be used to objectively and automatically determine the distribution of sentiments and emotions contained in textual databases (González Canché, 2024a; Nandwani & Verma, 2021; Sharma et al., 2024). When the resulting outputs indicate negative, neutral, or positive sentiments, this latter approach is referred to as sentiment analyses—and this technique has also been referred as polarity detection (Mäntylä et al., 2018). When the resulting analyses are presented in terms of emotions that go beyond positive or negative experiences or opinions (i.e., beyond polarity detection), the analytic method is called emotion analysis. From this view, emotion analysis represents efforts to move beyond polarity detection to gain more nuanced understandings in written information (Mäntylä et al., 2018).

Considering that polarity detection is a useful technique to aggregate positive and negative sentiments, whereas emotion analyses yield potentially more nuanced understandings, in our study we offer both a sentiment and emotion analytic framework for the resulting outcomes integrate both procedures. However, as further explained below, we also deviate from traditional sentiment and emotion analyses (SEA) approaches by minimizing the risk of aggregation bias (James, 1982), which may lead to losing nuance in our understandings based on failing to discover potential differences across groups of participants. We also depart from the traditional use of SEA by illustrating how our proposed framework may be applied to mixed methods and qualitative academic research—as opposed to being mostly a marketing analysis tool, as we further elaborate below.

Study Contributions and Purpose

In this study we focus on the intersection of linguistics, natural language processing, statistical and relational modeling, and computer science (Chang et al., 2021; Kumar & Bhatia, 2013; Taboada, 2016) to offer an analytic framework designed to (a) detect sentiments and emotions embedded in our participants speeches, discourses, or textual contributions across seventeen languages, ¹ (b) assess for emotional inconsistencies (i.e., emotional entropy) in the content of these texts both individually and by groups, and (c) formally test (i.e., hypothesis testing) whether the prevalence of emotions is similarly distributed across groups of participants. In other words, our proposed framework is saliently different from the traditional use of sentiment and emotion analyses (SEA) because our framework goes beyond aggregate analyses of the distributions of sentiments and emotions in a collection of texts (i.e., corpus) and it integrates measures of speech or content inconsistency at both individual and group levels.

The application of NLP to structure textual data, as well as the subsequent data processing, sentiment and emotion identification, interactive and dynamic visualizations, and hypotheses testing requires advanced technical expertise in data engineering. This expertise continues to serve as an important barrier toward the mainstream use of data science in general, and NLP and text classification in particular (Chang et al., 2021; González Canché, 2023a, 2023b). From this perspective, another contribution of this study is our goal to expand access to data engineering and data science tools by providing a standalone, free to use and free to distribute software that automatically implements all analytic steps discussed below (see Figure 1) without requiring any computer or statistical programming expertise. That is, all the steps, which include the structuring and normalization of textual data, the identification of sentiments and emotions, and the application of relational or network modeling for the visualization of findings and hypothesis testing may be executed by simply uploading the collection of texts to be analyzed.OPEN IN VIEWER

As part of the versatility of this no-code software, the analytic framework is implemented by either uploading a collection of Microsoft word documents or uploading a collection of texts stored in a spreadsheet file. An important attribute of this software is that it was programmed to conduct the sentiment and emotion analyses in seventeen different languages without the need to upload our datasets to any server—hence running all analyses locally. Based on our reliance on relational modeling to identify potential group variation in sentiment and emotion distributions, the resulting analytic framework and its accompanying software is referred to as sentiment and emotion network analysis (SENA). Finally, with the goal of providing qualitative and mixed methods researchers with the opportunity to become fully familiarized with these analytic tools, all databases used to showcase this analytic framework and software tool are being made publicly available so that researchers may interact firsthand with the SENA software to implement the analytic framework discussed in the remainder of this document.

Flow Diagram for the Entire SENA Process

To illustrate the conceptual and analytic process followed by SENA—from database creation to software execution, we present Figure 1. With the purpose of providing as much transparency and clarity of the entire process as possible, every step (or polygon) in the flow diagram presented in Figure 1 may be conceptualized as follows:

(1) Collect stories, essays, narratives, or any written information.

(a) SENA allows loading a collection of word documents or a spreadsheet where one column contains the text to be analyzed.

OPEN IN VIEWER

Figure 4.

Sentiment and Emotion Analyses of Students’ Evaluations/Comments About a Data Science Course (interactive versions: (a) here https://cutt.ly/FeJdsGo0, (b) here https://cutt.ly/leJddzfw).

OPEN IN VIEWER

Figure 5.

Sentiment and Emotion Network Analyses of Students’ Evaluations/Comments About a Data Science Course (interactive versions: (a) here https://cutt.ly/QeJfNb9t, (b) here https://cutt.ly/DeJfNILc).

OPEN IN VIEWER

Figure 6.

QAP testing across groups for small figure (a) and big data figure (b).

OPEN IN VIEWER

Figure 7.

Sentiment and Emotion Analyses of Classified Comments About a Trump’s Assassination Attempt (interactive versions: (a) here https://cutt.ly/lrx53Pej, (b) here https://cutt.ly/Mrx51Osc).

In the following pages, we expand on details of all back-end processes needed to implement the flow diagram described here and that are conducted by the SENA software.

Natural Language Processing and Text Preparation

Once the collection of texts (in *.doc format or a *.csv file) are uploaded, the text natural language process begins with the creation of the corpus or collection of texts. It is during this process that group identification, if it exists, is displayed. This grouping process is achieved if word documents are named based on individual attributes, or if a column in the database contains these attributes. If no attributes are added, the dialogue displayed in SENA just describes the number of word documents (*.doc) uploaded or the number of rows with valid text read in the excel (*.csv) file.

After this initial step is completed, standard text mining procedures are implemented (Feinerer, 2020; Silge & Robinson, 2017). First, all non-standard characters such as: “&,” “Ÿ,” “#,” “\r\n” are removed. Additionally, all characters are transformed to lower case to avoid creating duplicate words (e.g., “Learn,” “learn,” and “LEARN”). SENA also removes terms classified as “stop words” across the seventeenth languages included in SENA. Stop words, like “at,” “by,” “for,” “with,” “on,” “off,” “over,” “under,” are words that while needed to form thoughts and sentences, add no meaning alone.

To exemplify the process of removing stop words, let us consider the following sentence “the mere thought of losing you gives me the horrors.” In this sentence we have highlighted using bold font all stop words (the, of, you, me, the), therefore a sentence that does not contain stop words reads as “mere thought losing gives horrors.” This implies that, once stop words removal is applied, the resulting vocabulary would contain information that could be classified into emotions and sentiments.

In continuing with text mining and normalization, SENA allows users to choose whether they want to implement word lemmatization, a text normalization technique typically employed in natural language processing (Silge & Robinson, 2017). Lemmatization applies a morphological analysis to all words with the goal of reducing variants while preserving the full meaning of a word based on its form (i.e., pronoun, verb, noun, adjective). We selected lemmatization instead of stemming, another normalization technique, because the latter preserves just the root or base of a word, resulting in loss of meaning and usually incomplete words that will not be found in emotion and sentiments dictionaries. For example, in the string “educational,” “educated,” “educate,” “education,” “educative,” applying stemming results in “educ,” “educ,” “educ,” “educ,” “educ.” Applying lemmatization to the same original string, renders “educational,” “educate,” “educate,” “education,” “educative.” From this example, we can infer that lemmatization is more computationally demanding in both the initial transformation and the length/complexity of the resulting vocabulary. That is, in stemming, this example renders one word (i.e., one column in the word index vocabulary length), whereas lemmatization yields four words, which translates into four columns of the resulting document term matrix (see González Canché, 2023a, 2023b).

Like any other normalization technique, lemmatization may potentially change some of our participants’ meanings, which is a particularly salient issue for sentiment and emotion analyses. For clarity, let us build from our previous sentence example which reads “the mere thought of losing you gives me the horrors.” In this case, we highlighted with bold font all the words that can be located in a dictionary of sentiments and emotions (as we describe below, this dictionary is also referred to as a lexicon of emotions and sentiments). In this case, if we decide to apply lemmatization, the resulting sentence becomes “the mere think of lose you give me the horror.” ³ The main implications of these changes are that the lemmatized words changed the assumed meaning as follows:

• thought was changed to think leading to two issues. The first is that think is not part of a lexicon or dictionary of sentiments and emotions. The second is that thought is part of such a lexicon, and it is associated with the emotion anticipation. This means that lemmatization led to losing this emotion.

Even though we are at risk of changing sentiments and emotions when we lemmatize, by not lemmatizing we are at a greater risk of not being able to locate our participants’ words in the lexicon or dictionary. For example, assume we have the following sentence “Manuals are no panaceas.” If we do not lemmatize, we would lose the word manuals and panaceas because most lexicons only contain the word manual and panacea but not their plural form. On a positive note, however, our analyses of hundreds of thousands of texts suggest that the issues we discussed in our “the mere thought of losing you gives me the horrors” example are quite unusual. Contrary to this issue, in most cases lemmatization is recommended to be able to locate more words in the lexicon. With pros and cons of lemmatization in mind, researchers should conduct sensitivity checks by running models where lemmatization is applied and models where it is not and observe whether those decisions yielded different results or whether these results were consistent. The good news is that these checks are as easy as executing SENA twice, once with and once without lemmatization.

Having decided whether to lemmatize, we need to execute the text mining button in the SENA’s user interface. This process will result in a summary of the top 20 most common words. This frequency of word occurrence within texts is captured in a document term matrix. In this matrix, the documents (word documents or cells of a column in a database) configure the rows of the matrix and the terms (i.e., words or vocabulary) configure the columns. All words that were not used in a given document appear with a frequency of zero in the respective intersection of that row and that column of this matrix. Word frequencies are used to calculate sparsity. Sparsity indicates that some words, while may frequently appear in one document, they may not necessarily be part of the vast majority of the remaining documents, thus not being relevant for the overall vocabulary (Silge & Robinson, 2017). Our user interface was programmed to detect texts containing very uncommon words (appearing in less than three percent of all documents) and remove them from the analyses.

Finally, following Schütze et al. (2008) we provide researchers with the option of removing the most “extremely” common words across all texts. This is an important step because in some instances these extremely common words may add little value to the analyses (Schütze et al., 2008). This potential issue may be particularly true when these common words are synonyms of the main research goal or a central topic in an interview protocol, for example. That is, if researchers are interested in students’ opinions about a data science course, the transcriptions or texts may have common words such as “data,” “science,” “student,” and “course.” If researchers believe that keeping these words in the corpus would add little value for the detection of sentiments and emotions, then they should remove these words. Similarly, when dealing with social media data, we may decide to transform emojis with their official description. In cases like these the word “face” may appear too frequently for many emojis use this word.

It is with the goal of identifying potentially “extremely common” words, that SENA’s user interface shows the top 20 most frequent words, along with their frequencies across all documents. It is at this point that we may decide to remove x number of them. This optional removal step completes the text preparation and mining procedures—we illustrate in detail how to remove these words in our applications section.

Sentiment and Emotion Lexicons or Dictionaries

SENA’s flexibility to analyze sentiments and emotions in 17 languages locally, that is, without having to upload data to any server, is based on its use of sentiment and emotion dictionaries provided by the National Research Council Canada (NRC). The NRC Emotion Lexicon is a list of words and their associations with eight basic emotions (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) and two sentiments (negative and positive). As described by Mohammad and Turney (2013a) the classification of sentiments and emotions were manually done by crowdsourcing. That is, like Wikipedia, NRC was created with the collaboration of a large number of people or coders, as a form of peer-production. Specifically, NRC relied on Amazon’s mechanical Turk, “an online crowdsourcing platform that is especially suited for well-defined tasks but that can be done over the internet through a computer or a mobile device” (Mohammad & Turney, 2013a, p. 2).

It is important to note that coders were given tasks to carefully annotate term-emotion association based not only on individual or isolated words but considering their context in sentences and paragraphs. From this view, the training or identification of term-emotion association using crowdsourcing is similar to the training that generative AI follows to assess whether the text produced by ChatGPT (and similar generative AI platforms) is “human-like” enough. That is, via crowdsourcing, NRC has been able to consider context when creating the term-emotion association. This association included nouns, verbs, adjectives, and adverbs.

Based on crowdsourcing, the NRC lexicon was built based on random checks to ensure that no erroneous or misleading annotations are created (Mohammad & Turney, 2013b). SENA’s reliance on NRC is strategic for this lexicon includes both polarity, which is a classification of terms in negative or positive, as well as eight emotions classification of those terms. When words or terms have been associated with more than one emotion based on context, NRC respected those annotations. To ensure coder annotation quality, each coder’s tasks had two steps. In the first step there is a correct response that a coder needs to address, and it is only if this response is correct that the annotation of that coder is considered as a good candidate for further quality assessment. In their 2013 paper, Mohammad and Turney (2013b) mention the following example. First, they ask the coder, what word is closest in meaning to shark? where shark is the target term. To answer this question, the team offered the following options: car, tree, fish, olive. If the coder selected other than fish, the answer is not considered. But if the coder correctly selected fish, then they were asked how they would associate the word in terms of positive and negative sentiments (polarity) or in one or more of the following emotions: anger, fear, joy, sadness, disgust, surprise, trust, and anticipation.

Each of the terms were annotated, classified, or coded by five people. Mohammad and Turney (2013b) reported that in 74.4% of the annotations all five coders agreed on the emotions and that in 16.9% four out of five annotators agreed with one another. These levels of inter-coder agreement are a notable attribute of NRC. However, an arguably more salient attribute of NRC is its availability in 16 more languages, in addition to English. This is important because, as Mohammad et al. (2016) mentioned, sentiment analyses have predominantly been English-centric. From this perspective, the fact that NRC’s lexicon has been applied to the Basque, Catalan, Danish, Dutch, English, Esperanto, Finnish, French, German, Irish, Italian, Latin, Portuguese, Romanian, Spanish, Swedish, and Welsh languages is truly remarkable, and a value added to SENA’s engine and contribution to this line of research.

Minimizing (or Even Avoiding) Aggregation Bias

Aggregation bias applied to natural language processing, consists of classifying large amounts of text under a single element, when such a text may be configured by a myriad of classes or elements (González Canché, 2023a). Applied to sentiment and emotion analyses (SEA), aggregation bias consists of believing that an aggregate representation applied to all participants in the sample is enough to capture the accurate distribution of emotions and sentiments, when in reality important variations of the underlying distribution of SEA may be observed at both group and individual levels.

To illustrate this issue, let us present Figure 5. This figure is divided in (a) and (b), with Figure 5(a) showing the aggregate distribution ignoring participants attributes and Figure 5(b) considering participants’ gender. The data analyzed consists of 99 brief essays or reflections on a data science course with 50 of these essays being provided by men and 49 by women. In the aggregate representation, we can see that most sentiments and emotions were positive, however, Figure 5(b) shows that these positive distributions of sentiments and emotions were led by women for men had more mixed reflections with close to parity in positive and negative sentiments.

Later in the paper we will discuss in more detail these data and findings. For now, please note that by far the most common application of sentiment analysis has relied on aggregated analyses of the content of document term matrices. In some fewer instances, these analyses may have included changes over time. For an example of how time is traditionally incorporated in SEA, let us look at an output presented by determ, one of the for-profit software tools we mentioned earlier in the “Current applications” section. This output, published in January of 2023, can be accessed at https://cutt.ly/fwlZUgdA. There, analysts at determ showed how sentiments (classified in negative, neutral, and positive), changed from late October to late January (note that no year was specified in this determ’s example).

Although a temporal analysis is useful, for it allows us to observe positive and negative peak changes over this time frame, an even more nuanced understanding of these patterns may be attained by incorporating electors’ individual attributes, like those shown in Figure 5(b). That is, it may be useful to know if the negative peaks observed on December 5th in that determ figure, were driven by electors from specific demographics—women voters, for example. From this perspective, we argue that with a more nuanced level of analyses, better decisions or plans of action may be implemented. For example, if spikes in negative sentiments were driven by women, a campaign may be designed to address their concerns, or with this information we may decide to gather more data (i.e., focus groups or interviews) to better understand the concerns of this group. Accordingly, although disaggregated and group SEA analyses have remained underused, they may deepen our understandings not only in marketing settings (Kennedy & Inkpen, 2006; Taboada, 2016; Turney, 2002) but also in academic and education research (Calvo & D’Mello, 2010; Dolianiti et al., 2018; Luo et al., 2015; Yu et al., 2018; Yu et al., 2016). These analyses, however, although important, do require specialized statistical and computer programming knowledge that has prevented their use among qualitative and mixed methods researchers. Accordingly, a notable contribution of our proposed analytic framework and software is that they were designed to purposefully account for participants’ attributes when conducting the sentiment and emotion analytic procedures without requiring any programming experience.

Emotional Entropy or Mixed Messages

Although the identification of sentiments and emotions is reliable based on the lexicon used, there remains the possibility that some texts contain mixed or even contradictory messages. Accordingly, to offer a sense of how prevalent these mixed messaging practices are, all SENA outputs, (i.e., word cloud summaries—see Figure 4—, sentiment and emotion analyses histograms—see Figure 5–, and network analyses—see Figure 6), include a measure of emotional entropy or contradictory messaging, as described next.

Emotional entropy “can be thought of as a measure of unpredictability and surprise based on the consistency or inconsistency of the emotional language contained in a given text. [In other words, a] text with conflicting emotional language may be said to express or contain a mixed message.” (Jockers, 2023, para. 37). For example, if a text reads, “I hate loving you” this would be a mixed message with high entropy. As explained next, SENA normalized this value to range between 0 and 1, with the latter representing mixed or confusing messages and zero indicating that the content of the text was configured by emotions that were all positive or all negative. For example: “I love loving you” and “I hate hurting you” both have emotional entropy values of zero.

Procedurally, emotional entropy is measured as the sum of positive and negative words included in a sentence or text. For example, let us assume we have the following text: “I love hating you and I would be happy if you die.” This sentence has two positive (love and happy) and two negative (hating and die) words. In texts like this example, ties in the number of positive and negative words lead to high emotional entropy values. But as this number of words increases and evenness in the distribution of positive and negative sentiment words decreases, entropy distributions become less extreme. For example, if instead of four words we have 11 and the sentiment distribution is 3 negative and 8 positive, the entropy value would be 0.84. But with the same 11 sentiments, if only one is negative, the entropy is 0.43. When no negative or positive sentiments are mixed with each other, the entropy is zero. These values are obtained from the following two equations

f r e q u e n c i e s = s e n t i m e n t i ∑ s e n t i m e n t

(1)

entropy = − ∑ ( frequencies * log 2 ( frequencies ) )

(2)

Following Jockers (2015), entropy is standardized to range between 0 and 1 using a Log base 2 or binary logarithm log2 transformation. This transformation is useful because values of 0 mean no change, while negative values indicate reductions and positive values indicate increases. The negative symbol ensures that entropy is always positive. Additionally, it works well with small and long databases. In the emotional entropy value implemented in SENA, every word in the text is counted in the final “sentiment” vector even in the presence of repetitions. That is, if one person used the word “love” three times, this count is represented in “sentiment” as opposed to counting it once, as implemented by Jockers (2015). This change is relevant to our goal of preserving the accuracy of sentiments representation as observed in our participants’ words.

SENA computes entropy values at the text level so that each participant’s text is evaluated following equations (1) and (2). To display individual or text level results, SENA relies on its network interactive visualization. As shown in Figure 6 when we click over an individual (i.e., blue node), a message box displays the estimated emotional entropy of the text that such an individual provided. This same dialogue box displays the proportion of words per participant that were associated with positive sentiments as another tool to understand the mechanisms of each participant’s textual contributions.

Additionally, SENA also offers the average emotional entropy by group when presenting the distribution of sentiments and emotions and the summaries of most frequently used words. Specifically, each title of Figure 4 contains the average entropy of the groups represented in each word cloud. Finally, the average entropy values across all texts, irrespective of groups, is shown in the X-axis of the sentiment and emotion histograms, as shown in Figure 5.

For convenience, each of these values, at both individual and group levels, indicate whether the entropy is considered low, mid, or high. Since the entropy ranges between 0 and 1, we relied on tertiles to capture these values. Specifically, the messages displayed are: “High entropy, high inconsistency” with values greater than 0.66. “Mid entropy, lower consistency” with values over 33 and below 67, and “Low entropy, high consistency” with values below 34.

Pragmatically, we offer emotional entropy estimates with the goal of identifying texts that deserve more attention. For example, if NLP missed sarcasm or the content is more complex by nature, involving a variety of sentiments, thus texts should be more carefully analyzed. Accordingly, by identifying these texts we can qualitatively assess whether the sentiment and emotions analyses were able to capture that complexity or sarcasm or whether we should be concerned with the resulting classification.

Network Modeling and Data Transformations

As discussed above, SENA relies on network analysis methods or relational thinking to identify each unit’s connection strength with the sentiments and emotions embedded in their textual contributions. These relationships are conceptually shown in Figure 3 and are operationalized in Figure 6. Procedurally, recall that each text is analyzed to detect sentiments and emotions, and this process may be represented as a matrix, wherein the rows contain units (i.e., text providers) and the columns are the sentiments and emotions identified across those texts. In returning to our working example, since we have 99 essays, after the sentiment and emotion identification we would have a matrix of 99 rows and 10 columns—with eight of these columns accounting for the emotions discussed above and two columns containing positive or negative sentiments (or polarity).

In sentiment and emotion analyses (SEA) the number of columns will always be 10. This is based on our NRC lexicon use containing eight emotions and two sentiments. From this view, if our total number of texts to be analyzed is 99 and we ignore participants’ attributes, the resulting relational matrix will have 99 rows and 10 columns as its dimensions. From now on, we will be representing these dimensions as [99, 10]. Nonetheless, when we construct these matrices considering participants’ attributes, the row dimensions will account for the number of elements in each group. That is, since we have 50 essays written by men and 49 essays written by women, then the resulting dimensions for each group are [50, 10] and [49, 10] for men and women, respectively. In sum, regardless of whether we consider these attributes, all the matrices handled by SENA are configured by two different elements (i.e., texts in the rows and emotions in the columns of the resulting matrices).

In network modeling these matrices are referred to as bipartite or two-mode matrices (Breiger, 1974; González Canché, 2019, 2023, 2024; González Canché et al., 2025b; González Canché & Zhang, 2025a; Kolaczyk & Csárdi, 2014) for they allow us to capture relationships taking place among two types of elements—i.e., the sentiments and emotions embedded in those texts. With these matrices we can add the relational or network attribute to SEA. making it SENA. This is how SENA enables us to build sociograms or network representations like those shown in Figure 6. In these networks the intersection of a row and a column captures the strength of that relationship with true values. That is, if row 88 and column 8 have a value of 24, this means that for this individual’s speech, or text, the emotion contained in column 8 appeared 24 times in her essay or any type of written information. But if this value is zero, this means that such an emotion was not present. The resulting interactive visualizations (see https://cutt.ly/QeJfNb9t and https://cutt.ly/DeJfNILc for Figure 6(a) and (b), respectively) contain this information so that when we place the cursor on top of a connecting line or link, the message box displays the intensity or the number of times that individual mentioned words classified as that emotion. In this network representation, absence of a line indicates that such an individual did not mention words classified under that particular emotion.

As described earlier, the matrices obtained from the NRC lexicon contain both sentiment and emotions. Note further that sentiments reflect polarity with negative and positive categories. That is, in NRC the sentiments are not measured on the same scale as emotions (Jockers, 2015, 2023; Mohammad & Turney, 2013b). Based on this discrepancy in measurement, our network representations along with the quadratic assignment procedures discussed in the following section do not include sentiments but focus on emotions distributions. Having noted this, we are not discarding the distribution of sentiments at the individual level. Instead, we are displaying positive sentiment prevalence in the network visualizations as an attribute of the participants. Specifically, we show an estimate of the proportion of sentiments per actor that were classified as positive—which, as discussed in equation (1), is one of the steps followed to compute the emotional entropy estimates. For example, in Figure 6(a) we can see that a dialogue box shows that for ID8, 80% of his words located in the sentiments lexicon were classified as positive. In Figure 6(b), for participant ID61, 91.7% of her words located in the NRC lexicon were also positive. In addition to this individual level depiction, we also include sentiment distribution in the histograms of sentiments and emotions as shown in Figure 5.

In terms of data transformations, the main implications of focusing on emotions for the visualization and the hypothesis testing via quadratic assignment procedures (QAP) is that the two-mode matrices will be [ n , 8] where n represent the number of texts and 8 is the number of emotions. Moreover, also note that the QAP analyses require matrices that are square, meaning that they contain the same elements in the rows and the columns. As explained in the QAP section discussed below, we needed to project or transform these two-mode matrices into their one-mode form retaining the information contained in the emotions (or the columns) without altering the underlying structure of these two-mode matrices. We achieved these transformations following Breiger (1974) projections as we detail in the QAP section.

Before moving to the next section let us note that our network visualizations weight the size of the emotions via betweenness centrality. Centrality in network modeling serves to highlight the relevance of the actors or units in a network based on their role (Borgatti, 2006; González Canché, 2019, 2023c; Kolaczyk & Csárdi, 2014). Our selection of betweenness centrality is based on our interest in highlighting the emotions that connected more actors in the network or served as bridges. In this centrality, a unit is relevant to the extent such a unit falls in between two other units (Borgatti, 2006; González Canché, 2019; Kolaczyk & Csárdi, 2014). That is, emotions that are connecting or bringing together actors or texts have a bigger size. In the case of men, this emotion was fear (in green) and for women it was trust in purple (see Figure 6(a) and (b), respectively). From this view, our emphasis on these visualizations is to highlight emotions’ relevance while at the same time identifying clusters of cases that are more similar by being connected to the same emotions. That is, units that are located close to each other (blue dots in the figures) tended to mention the same emotions and with a similar intensity than units that are located further away, as we further discuss in our findings section.

Hypothesis Testing via Quadratic Assignment Procedures

When group information is added to texts, SENA automatically tests the hypothesis that some groups’ emotions are more similar than others as a function of the participants’ attributes. For example, in Panel 3 of Figure 3, women seemed to have had more positive emotions than their men counterparts by being ascribed more frequently to surprise and joy, whereas men’s texts were the only ones associated with fear. Although qualitative descriptions like these are important, such descriptions do not allow us to test whether these results are more robust than chance alone. To better understand whether the SENA outputs we observe may present differences by groups, SENA relies on Quadratic Assignment Procedures (QAP), as discussed next.

QAP is a non-parametric analytic procedure that does not rely on normality assumptions and does not assume independence (Krackhardt, 1988). Indeed, QAP “builds into the test statistic the kind of row/column interdependence that is assumed in network data” (Krackhardt, 1988, p. 363). Accordingly, given that QAP enables testing for statistical significance using network data in matrix form (Whitbred, 2011), it is particularly well-suited to help us test “the null hypothesis that two network variables are uncorrelated” (Krackhardt, 1988, p. 362).

As mentioned in the “Network Modeling and Data Transformations” subsection above, this matrix form is obtained from identifying the emotions associated with each of the texts configuring the corpus. We further elaborate on network projections via matrix algebra below. For now, let us mention that QAP takes the form of correlation and regression analyses. In correlation analyses, one can measure the extent to which two networks are statistically associated without any assumption of directionality in this relationship. In regression models directionality exists, and an outcome matrix is explained by a set of network attributes also measured in matrix form to account for all dyadic relationships (Krackhardt, 1988). In both cases, i.e., correlation and regression, QAP translates into measuring whether every corresponding dyad or pair of connections in two (or more) matrices tend to vary in the same direction (positive correlation or association), the opposite direction (negative correlation or association), or whether they are independent.

Procedurally speaking, QAP can be classified under the random permutation test family, which is sometimes referred to as a randomization test (Phipson & Smyth, 2010; Rubin & González Canché, 2019). The random permutation tests begin by recording a given statistic of interest across two mutually exclusive and exhaustive groups. As just discussed above, these statistics can be mean differences, and correlation and regression coefficients, for example. These observed estimates are recorded and then the actual values across these groups are shuffled randomly X number of times. Each time this random shuffle or permutation occurs the statistics obtained from this process is recorded. This procedure enables the recreation of a sampling distribution wherein the coefficients obtained through a random process should have a mean of zero given the lack of inherent structure upon which the data generating process rendering these coefficients was founded. Once this sampling distribution is obtained, analysts can then compare the number of times for which the randomly generated estimates were greater than the estimates obtained with the actual observed data.

In using a correlation coefficient, for example, we could estimate what is the proportion of times that the randomly generated correlation coefficient among two networks was larger than the correlation obtained with the observed data. Once more, the fundamental rationale is that randomly obtained coefficients should be distributed around zero, on average. Hence, if this randomly generated coefficient exceeds the actual observed coefficient, then one would conclude that our observed correlation was not better than what one would expect to see by chance alone. If the proportion of times that randomly generated correlations coefficients are greater in magnitude than our coefficient obtained with actual data approximates zero, then one can conclude that our observed correlations are better than chance by 1 minus the proportion of times randomly generated coefficients were greater. Specifically. assume that .005 is the proportion of times out of 50,000 permutations that the random coefficients surpassed the magnitude of our observed coefficients. Then we can conclude that 99.5% of the times we would expect our observed results to hold true had we had access to other networks of the same size and come from the same population of interest.

The QAP correlation begins by estimating a Pearson correlation coefficient across the corresponding cells of two matrices (Phipson & Smyth, 2010). This requires that the units in the networks are the same and follow the same order across matrices. For example, the following representation has three units. Note that these matrices contain the same actors, and they follow the same order (A1, A2 and A3). The process takes the dyadic correlation among A i and A j in matrices 1 and 2 to assess whether a given dyadic intersection in one matrix, being classified under trust, for example, like shown in Figure 3, is realized with a similar distributional magnitude in the other network. In our example, these networks are the projection of emotions obtained from men and women, following Figure 3 as our working example. The second step is depicted in the shuffled section below of the matrix represented in Table 1, where all the rows and cells of that matrix are randomly permuted (Krackhardt, 1988). Each time this permutation takes place its corresponding Pearson coefficient is recorded.

OPEN IN VIEWER

Table 1.

Illustration of the Analytic Process and Rationale Employed by the Quadratic Assignment Procedure Analyses.

	Observed matrix men			Observed matrix women
	A1	A2	A3	A1	A2	A3
A1	2	1	5	4	2	5
A2	1	4	0	0	5	0
A3	1	0	6	2	0	7
	Observed matrix men			Shuffled matrix women
A1	2	1	5	?	?	?
A2	1	4	0	?	?	?
A3	1	0	6	?	?	?

Matrix Projections

The matrices built from qualitative data analyzed by SENA are the one-mode representation of the actors and their ascription to emotions as represented in Figure 3. This one mode transformation is achieved by network projections as shown in equation (3) (see Breiger, 1974). As described above, the dimensions of the SENA matrices are rows accounting for the number of participants or texts (i.e., 50 rows and 49 rows for men and women, respectively) and the columns are the emotions to which their texts were classified.

In the case of men, the matrix are [50, 8] and in the case of women the matrix is [49, 8]. Following (Breiger, 1974), when the goal is to preserve the information contained in the columns, which renders an emotions matrix, we can transpose the matrix and multiply it by the original (but binarized) matrix, more on this below. Assuming we call this matrix A, we have that

A T * A b i n a r i z e d

(3)

will yield a one-mode projected network or matrix of emotions. Note that the dimensions of this multiplication in the case of men are [8, 50] ∗ [50, 8], which following matrix algebra results in a [8, 8] matrix. In the case of women, the dimensions are [8, 49] ∗ [49, 8], also resulting in a [8, 8] matrix. These matrices have the attribute that the order of the row and column information will be the same across men’s and women’s projected matrices. Based on this description, all projected matrices are square [8, 8] containing the eight emotions described above.

The element A _binarized in equation (3) indicates that the matrix will only contain 0 and 1s when a participant is linked to an emotion. This binarization avoids double counting relationships when transforming weighted two-mode matrices (González Canché, 2023c).

When the presence of attributes is included in the texts uploaded to the user interface, SENA automatically produces a PDF containing all dyadic comparisons. For example, Figure 7 shows the results of comparing the distribution of emotions contained in the responses provided by women and men at the end of course evaluation of a data science course. The output produced by SENA shows the comparisons conducted, which in this case it reads “Comparisons: man and woman.” Then it shows the statistics of interest. Specifically, SENA prints the observed correlation coefficient Rho in a blue dotted line and a density distribution based on 50,000 permutations. The value of zero is the average correlation across these permutations. The red lines are two standard deviations from the mean obtained from 50,000 permutations. Notably, when the blue line falls in between the red lines, this indicates that the distribution of emotions across these groups is different—that is the matrices are not statistically correlated. Therefore, when the blue line falls outside the red lines, the conclusions are that both groups have emotions that are more similar than chance alone, so there are no significant discrepancies in these distributions.

SENA’s User Interface

SENA’s user interface (UI) is available for Mac (access here: https://cutt.ly/QwhYruBr) and Windows (access here https://cutt.ly/YwhJJKvO) operating systems. As shown in Figure 8, this UI has a maximum of seven steps. The eighth step is optional in case users need to reset the application or change the data format.OPEN IN VIEWER

Short Essays Examples

As described above, the 99 essays included in this section are configured by 50 essays wherein students reacted to the prompt: After having spent the last eight weeks learning about data science applications, what are your reactions or your assessment regarding data science uses, including pros and cons you have observed?

As part of this data gathering exercise, we obtained participants gender attributes that resulted in 50 men and 49 women. To illustrate SENA’s performance and flexibility, in the following analyses we present all data formats that SENA can handle, separating the following subsections into Microsoft Word and spreadsheet formats including and not including participants’ attributes.

Application to Machine Learned Classes of Big Data

As briefly described above, the YouTube comments database can be analyzed by video source, that is using each posting account as an attribute and then analyze whether there are sentiments variations these accounts (i.e., the New York Times, The Wall Street Journal, Channel 4 News in the UK, as well two individual accounts). Alternatively, this database can also be analyzed in SENA following a different approach that involves machine learning.

Our analytic strategy consisted of merging SENA with machine learning text classification and then analyzing whether the resulting machine-learned topics or classes vary in their distribution of emotions and sentiments. Specifically, we first determined machine-driven text classes via topic modeling as implemented by MDCOR (González Canché, 2023b) and then analyzed whether there exist variations in the distribution of these sentiments and emotions as a function of the machine-learned classes. This approach enabled us to ask whether machine-learned classes are configured by different distributions of sentiments and emotions as well as the level of emotional entropy those classes contain. From this perspective then, SENA may be easily integrated with existing software and with machine learning text classification applications.

Note that a description of the topic modeling via latent Dirichlet allocation is beyond the scope of this paper. For now, let us simply describe that the result of implementing MDCOR with 5000 MCMC/Gibbs resamplings and 500 burn-in samples yielded the following seven topics as the optimal solution of this text classification learning process: V1. Criticism to the video accuracy, V2. Middle East comments/support, V3. Surprise of failure to stop this attempt, V4. Multiple Conspiracy theories, V5. Belief that it was staged, V6. Love and support for Trump, and V7. Laughing in disbelief. For more details about text classification using MDCOR, please read the methodological paper that provides access to the software in Windows and Mac operating systems (see González Canché, 2023b).

Small(er) Data Analyses

In this section we illustrate how aggregated results may be misleading. Specifically, note that Figure 5 contains the aggregate (Figure 5(a)) and the group analyses (Figure 5(b)) of students’ comments. In this case it is clear that only relying on aggregate analyses would have let us to believe that the vast majority of sentiments (about 70% of them) were positive and that this result may apply to all participants in the sample. This latter issue is referred to as aggregation bias (James, 1982), which as depicted earlier may be minimized by conducting analyses across groups (González Canché, 2023a, 2023b).

Before moving forward, let us further describe the content of Figure 5(a). Each SEA figure presents the outputs separated by negative emotions (left panel), positive emotions (center panel), and sentiment distribution (right panel). This output is normalized so that sentiments distributions (i.e., positive and negative) adds to 1. That is, these distributions are represented in proportions. Similarly, the left and center panels together add to 1 for both account for emotions. In other words, the distribution of the eight emotions represented in this figure adds to 1. The result of the sentiments distribution indicates that there were 264 words across all 99 essays (or 30.9%) that were associated with negative sentiments. In terms of emotions, the most prevalent emotion was trust with a 251 representing 22.8% and the least prevalent was disgust with a frequency of 38 (or 3.5%). The interactive visualization displays these values when posing the cursor on top of each bar. In this example we show the proportions described in this paragraph for the trust and disgust emotions.

Following with our example, Figure 5(b) shows the resulting group analyses output. In that figure we can clearly see that women’s responses were more positive than the comments provided by their men counterparts. The interactive visualization (available here https://cutt.ly/leJddzfw) shows that 86.2% of women’s words identified in the Lexicon were classified as positive sentiments. Subgroup emotion analyses reflected that men tended to have more affiliation with anger and disgust emotions, whereas women’s words were associated with trust, joy, and anticipation. As displayed in this figure we also see that women’s trust almost doubled the trust emotion found in men’s texts (29.2% vs. 17.7%).

Although these differences are notable, we still do not know if such distributions are statistically significantly different. This hypothesis was tested automatically by SENA and the results may be found in Figure 7(a). In this QAP output, we had enough evidence to conclude that emotions distributions of women are different than their men counterparts and that, considering the histogram shown in Figure 5(b) the former tended to be more positive than men’s emotions about this data science course. Specifically, the observed matrix correlation had a magnitude of 0.374, which was not far away from the simulated mean of around 0. That is, the distribution of these matrices is uncorrelated. This findings mean that we can describe the distributions shown in Figure 5(b) as being statistically significantly different.

Figure 6(a) and (b) show individual analyses. These figures are particularly useful in situations where the identification of cases with negative or positive emotions may be of relevance, as described in research where students evaluations were used as predictors of subsequent success (see Calvo & D’Mello, 2010; Dolianiti et al., 2018; Luo et al., 2015; Yu et al., 2018; Yu et al., 2016). As mentioned above, each of these cases also contain emotional entropy estimates that indicate mixed messages in text content. Texts with a mix of positive and negative emotions are classified as confusing, whereas others may be clearer by indicating either negative or positive emotions. Once cases of interest are identified, we could go and read the texts of these cases to gain a deeper understanding of their concerns or praises regarding their experiences. Moreover, following network analyses principles, we can observe how subgroups are formed. For instance, in Figure 6(a) and (b) those participants located in the bottom right sections are more related to a specific set of emotions than participants located in the middle of the graph or in the top right section of this sociogram. For women, bottom right location indicated their affiliation with trust, surprise and joy. For men, participants located in the bottom right were associated with anger, fear, and surprise.

Finally, note that more descriptive analyses may be gained with word clouds. Figure 4(a) and (b) in addition to summarizing content relevance, also show average emotional entropy estimates. These estimates suggests that women have half the emotional entropy values on average than men. That is, men’s texts were less clear regarding their inclusion of positive and negative emotions.

Big Data Analyses of Machine-Learned Topics

As mentioned above, the analyses of YouTube comments involved a classification of these comments and then analyzing whether the emotion distribution of those comments were different or similar. Just as in the short essay case, Figure 11(a) shows that an aggregate analyses of the seven topics identified via machine learning could be understood as having more positive sentiments. However, the disaggregate analyses shown in Figure 11(b) suggest that topics 6 and 7 (Love and support for Trump and Laughing in disbelief or nervously, respectively) were associated with more positive sentiments and emotions. Of the other five topics, only topic 1 (Criticism to the video accuracy) was associated with more negative sentiments and emotions than all the other remaining topics.OPEN IN VIEWER

Access to all word clouds and SENA outputs per topic may be gained by following this link https://cutt.ly/ieZlGeOI. For this example, let us address the question of whether emotions distributions are similarly distributed across topics. To this end we can use the QAP output. However, given that 7 groups or topics exist, the resulting dyadic combinations rendered 21 comparisons. These comparisons are shown in Figure 7(b).

Overall, these results confirmed that the discrepancies observed in Figure 8(b) are important. All comparisons that included topics 6 and 7 (Love and support for Trump and Laughing in disbelief or nervously, respectively) consistently indicated that the positive emotions captured in these topics were different from those captured in topics 1 to 5. Of these latter topics, topic 1 captured the most negative emotions. But topic 1 (Criticism to the video accuracy) was only significantly different from topic 2 that included Middle East comments/support. Note that when an extremely potentially viral topic exists in social media, groups tend to “profit” from that topic by posting their support to issues that they care. In this case, the issues were then concurrent conflicts in the Middle East. Finally, topics 6 and 7 were highly correlated, therefore allowing us to conclude that they are configured by similar distributions of emotions, which as shown in Figure 11(b) are overwhelmingly positive.

In closing this section, let us note that a study focused on the analyses of these datasets would proceed to qualitative analyzing the most relevant texts. This identification can be done using cases shown in the SENA outputs, either individually or by their location in the resulting sociograms. Specifically, as discussed in Figure 6(a) and (b), certain individuals and groups tend to be clustered together by being linked to similar emotions. This identification of cases could help researcher to qualitatively explain why group differences were found, preferably using participants’ own voices, as captured by their original essays or comments (see González Canché, 2023a, 2023b, for examples of the relevance of preserving these voices).

Limitations

SENA currently does not incorporate machine learning and artificial intelligence in the identification of sentiments and emotions. We refrained from this approach because those AI applications cannot yet be computed locally without the use of quite powerful computers. Instead, these AI applications, either need a server connection where our qualitative data needs to be uploaded or super computers to accommodate the large language processing power needed. With respect to the former, given the need to protect participants confidentiality and anonymity in academic research, which is the primary audience for the application of SENA, we preferred to offer access to 17 languages rather than require users to upload their databases to servers. With respect to the computing power costs, even if we develop local servers, costs will still impede the vast majority of qualitative researchers from benefiting from SENA.

Having noted this potential limitation, we also showcase how SENA may be easily incorporated with existing machine learning procedures like those implemented with no-code software like “Latent code identification (LACOID)” and “Machine driven classification of open-ended responses (MDCOR)” (see González Canché, 2023a, 2023b, respectively). In these latter applications, we first need to classify the raw text into codes or topics and then apply SENA to these learned classes to test whether the distribution of emotions is similar or statistically diverges. This analytic flexibility is notable because we can include machine leaning and AI to texts that will then be analyzed via SENA without compromising the integrity of the analyses.