Abstract
The increase of online hate speech, especially against women, has become a big problem in digital communication, especially in low-resource languages like Malayalam and in situations where English is mixed with other languages. This work examines the efficacy of synthetic data augmentation techniques—Machine Translation (MT), Masked Language Modeling (MLM), and Few-Shot Learning (FSL)—in enhancing hate speech identification inside Malayalam-English (Manglish) social media text. We use these three methodologies to improve transformer-based models like mBERT, BERT, and IndicBERT. Our experiments show that classification performance has improved a lot. For example, mBERT got an F1-score of 86.42%, but real data only got 81.24%. LIME's explainability research indicates that contextual clues, not just offending words on their own, are what make detection accurate. Also, synthetic data makes things more fair by cutting down on false positives and false negatives and makes models more broad by exposing them to a larger range of code-mixed expressions. The approach is effective, but it has certain drawbacks. For example, it may be hard to apply to other code-mixed languages or fields, and there are ethical issues with creating synthetic data. The results have practical consequences for implementing fairness-aware, transparent, and resilient hate speech detection algorithms on multilingual social media platforms. This is the first study we know of that looks into synergistic synthetic data augmentation for detecting hate speech that mixes languages, with the goal of reducing online harassment of women.
Keywords
Introduction
Hate speech detection is becoming a crucial task in the realm of social media, where individuals articulate their opinions through various social media platforms. Detecting hate speech is a prominent research area in natural language processing, where researchers employ machine learning and deep learning algorithms to identify hate speech content across various social media platforms. Therefore, the oversight of content on social media is the responsibility of every technology professional. This content may propagate hate against society based on sex, race, caste, politics, etc. We refer to such content as hate speech. Hate speech is regarded as a subset of aggressive and abusive language; nonetheless, all these forms possess an offensive characteristic (Alkomah & Ma, 2022). It is a common observation that websites frequently feature multilingual expressions or writing, which presents an additional obstacle. From where does code mixing originate? Individuals who are members of multilingual communities frequently communicate in multiple regional languages. This type of text is referred to as code-mixed data and is represented using a variety of languages (Shanmugavadivel et al., 2022). Hate speech detection is already a difficult task; however, it is even more difficult to detect hate speech in code mixed data due to the unreliability of the annotated dataset, morphological complexity, and syntax variety.
Research on code-mixed data in Indian languages, particularly Dravidian languages, is notably scarce. Our research examines code-mixed data of Malayalam and English, where Malayalam is a Dravidian language spoken by the Malayali people in the Indian state of Kerala and the union territories of Lakshadweep and Puducherry (Mahé district). It is one of the 22 officially recognized languages of India. In 2013, Malayalam was officially recognized as a “Classical Language of India (Khullar et al., 2024).” As a low-resource language, Malayalam is receiving limited research attention in the field of natural language processing, particularly concerning code-mixed data. In our prior research, we gathered code-mixed comments from social media platforms, particularly YouTube and Facebook, which were subsequently pre-processed and labelled by various annotators for hate speech detection tasks. We are motivated to engage in synthetic data generation. Synthetic data generation denotes the artificial production of data that simulates real-world data, created through computational methods rather than sourced from actual environments (Yadav et al., 2023). This methodology is especially beneficial in low-resource environments, such as for the detection of code-mixed hate speech in Malayalam-English, where the acquisition and annotation of authentic data is costly or time-consuming.
Hate speech detection is becoming a crucial task in the realm of social media, where individuals articulate their opinions through various social media platforms (Alkomah & Ma, 2022). Detecting hate speech is a prominent research area in natural language processing, where researchers employ machine learning and deep learning algorithms to identify hate speech content across various social media platforms. Therefore, the oversight of content on social media is the responsibility of every technology professional. This content may propagate hate against society based on sex, race, caste, politics, etc. We refer to such content as hate speech. Hate speech is regarded as a subset of aggressive and abusive language; nonetheless, all these forms possess an offensive characteristic. It is a common observation that websites frequently feature multilingual expressions or writing, which presents an additional obstacle. From where does code mixing originate? Individuals who are members of multilingual communities frequently communicate in multiple regional languages. This type of text is referred to as code-mixed data and is represented using a variety of languages (Shanmugavadivel et al., 2022). Hate speech detection is already a difficult task; however, it is even more difficult to detect hate speech in code mixed data due to the unreliability of the annotated dataset, morphological complexity, and syntax variety.
Research on code-mixed data in Indian languages, particularly Dravidian languages, is notably scarce. Our research examines code-mixed data of Malayalam and English, where Malayalam is a Dravidian language spoken by the Malayali people in the Indian state of Kerala and the union territories of Lakshadweep and Puducherry (Mahé district). It is one of the 22 officially recognized languages of India. In 2013, Malayalam was officially recognized as a “Classical Language of India (Khullar et al., 2024).” As a low-resource language, Malayalam is receiving limited research attention in the field of natural language processing, particularly concerning code-mixed data. In our prior research, we gathered code-mixed comments from social media platforms, particularly YouTube and Facebook, which were subsequently pre-processed and labelled by various annotators for hate speech detection tasks. We are motivated to engage in synthetic data generation. Synthetic data generation denotes the artificial production of data that simulates real-world data, created through computational methods rather than sourced from actual environments (Yadav et al., 2023). This methodology is especially beneficial in low-resource environments, such as for the detection of code-mixed hate speech in Malayalam-English, where the acquisition and annotation of authentic data is costly or time-consuming.
While many studies have looked at how to find hate speech in multilingual settings, not many have looked at Malayalam-English code-mixed text, especially when it comes to abuse against women. Most current methods either work with tiny datasets, don't look at hate speech that is based on gender, or don't have ways to make predictions clear. It's hard to construct strong detection models for low-resource languages like Malayalam because there isn't much annotated data available. There has been an upsurge in code-mixed abusive content aimed at women on social media platforms in Kerala and among the Malayali diaspora. This makes it even more important to do research that not only finds this type of content but also explains why it was classified that way. This study offers various distinctive contributions to mitigate the discovered deficiencies in Malayalam-English hate speech identification. First, we create a bigger and more varied synthetic dataset to get over the problems that come with small annotated corpora. Second, we focus on hate speech against women, which is a small but important part of online abuse detection. Third, we systematically compare several transformer-based models to create a performance benchmark that is better than previous work. Fourth, we deploy explainable AI tools to show how each word affects the model's predictions, making it easier for both end users and content moderators to understand. Finally, we show that combining synthetic data augmentation with transformer topologies for this important NLP job leads to quantifiable performance increases over baseline models.
This study employed three synthetic data generation methodologies: 1) Translation-based augmentation involved utilizing the Hinglish dataset (Hindi-English) (Periti et al., 2024) from Yadav, A. K. et al. (2023) and executing machine translation via Google Translate after segregating Hindi and English words, which will be elaborated upon in Section 3. 2) Lexical substitution utilizing the XLM Roberta model (Stahlberg & Kumar, 2024), wherein entities have been replaced. 3) Contextual Sentence Generation utilizing the mT5 model (Vidgen & Derczynski, 2020), wherein we formulate prompts to enable the model to produce hateful posts within the specified context upon receiving several examples of hate speech.
Formally, given a corpus The use of synthetic data to improve the detection of hate speech in Malayalam-English code-mixed language: We propose an innovative method to tackle data scarcity in Malayalam-English code-mixed hate speech detection by producing synthetic data through three techniques: Machine Translation (MT), Masked Language Modeling (MLM), and Few-Shot Learning (FSL). Our research is among the initial efforts to comprehensively examine the comparative and synergistic performance of these strategies in a low-resource, code-mixed environment. Comprehensive Comparative Evaluation of Transformer Models: We assess five leading transformer models—BERT, mBERT, XLM-RoBERTa, IndicBERT, and MuRIL—using both real and synthetically enhanced datasets. The results show that multilingual models, notably mBERT, do much better with carefully made synthetic data and always do better than other models in F1-score, accuracy, precision, and recall. Empirical Validation of Multimodal Synthetic Data Fusion: Our comprehensive experiments reveal that the integration of MT, MLM, and FSL-based data yields superior model performance, with mBERT attaining an F1-score of 86.42%, surpassing models trained on authentic or singularly-strategy augmented datasets. This shows how important synthetic variety is for strong generalization in code-mixed hate speech problems. Detailed Error and Explainability Analysis: We also use LIME-based explainability to show that models trained on MLM and FSL data can find both explicit and implicit hate speech. This shows how important it is to pay attention to context when classifying code-mixed text. Insights Based on Language and Culture: Our results provide us a better understanding of how synthetic augmentation approaches keep or change the language and cultural patterns that are present in informal Malayalam-English discourse. We demonstrate that MLM and FSL approaches more effectively preserve context and sociolinguistic attributes compared to MT, rendering them more appropriate for delicate applications such as hate speech detection. Open Research Implications for Low-Resource Code-Mixed NLP: Our study establishes a methodological framework for subsequent investigations into resource-deficient languages, especially within multilingual contexts, by assessing synthetic data techniques in a multilingual and code-mixed environment. The methodology can be applied to additional Indian language pairings and social media analysis areas outside hate speech.
This paper is organized as follows: Section 2 provides a synthesis of the literature. Section 3 delineates our proposed methodology. Section 4 and 5 provides detailed experiments and an in-depth examination of the results, followed by concluding observations in Section 6.
Overview of Key Studies on Hate Speech Detection in Indian Code-Mixed Languages.
Overview of Key Studies on Hate Speech Detection in Indian Code-Mixed Languages.
The rise of multilingual social media platforms over the past ten years has made code-mixed communication a major area of study in Natural Language Processing (NLP) research (Roy & Kumar, 2025). Finding hate speech in code-mixed text is still very hard because of syntax problems, a lack of annotated datasets, and the fact that languages are so different. There has been a lot of success in finding hate and offensive speech in languages that are spoken by a lot of people, such English and Hindi. However, there is a growing need for more research on code-mixed languages that don't have as many resources, like Malayalam-English, Tamil-English, and Telugu-English.
Researchers have investigated many methodologies to address this issue, encompassing standard machine learning algorithms, deep learning models, and more sophisticated transformer-based structures. Table 1 gives a full list of 25 important research that were published between 2018 and 2025 that look at how to find hate speech in different Indian code-mixed language environments. The studies were chosen based on the following criteria: (i) they had to be relevant to finding hate or offensive speech in code-mixed or multilingual settings; (ii) they had to include Indian languages or regional variants; (iii) they had to be published in peer-reviewed journals, shared tasks, or reputable repositories; and (iv) they had to make methodological contributions using machine learning, deep learning, or transformer-based models.
This carefully chosen collection of literature includes shared tasks, new ways to integrate data, ways to employ transfer learning, and ways to use multilingual benchmarks. Altogether, it makes up the basis for finding research gaps and inspiring the current work. A careful examination of the surveyed literature uncovers persistent tendencies and growing issues, notably the insufficient focus on Malayalam-English code-mixed texts. Furthermore, limited research has systematically examined issues pertaining to class imbalance, equity, or gender-based violence, particularly against women—underscoring an urgent necessity for more inclusive and socially conscious methodologies.
In addition, code-mixed hate speech detection has not yet looked at the use of synthetic data generation methods like masked language Modeling (MLM) and few-shot learning (FSL). Also, existing models don't often include ways to explain predictions to check for bias. These gaps show how new and important this research is. It is the first to use adapter-based transformer models trained on synthetically augmented Malayalam-English code-mixed hate speech data, with a strong focus on fairness, explainability, and protecting marginalized voices in online spaces.
Our work has the same main goal as Choudhury et al.'s (Miškolci et al., 2020) effort, which is to find objectionable language in code-mixed Indian languages, but it is very different in terms of scope, methods, and goals. That paper is mostly about comparing transformer-based models utilizing publicly available datasets in a shared work setting. On the other hand, our study is only on finding hate speech against women in Malayalam-English code-mixed social media posts. We put together a dataset of 3,000 Facebook comments on abuse based on gender that we collected by hand. We use synthetic data generation methods like Machine Translation (MT), Masked Language Modeling (MLM), and Few-Shot Learning (FSL) to deal with the problem of not having enough data in low-resource code-mixed environments.
The research that were looked at show that hate speech identification has come a long way for Indian code-mixed languages, especially for prominent pairs like Hindi-English and Tamil-English. This is mostly thanks to transformer-based models and classic machine learning methods. These works utilize many methodologies, including ensemble learning, transfer learning, and contextual modeling, demonstrating robust performance on publically accessible datasets. Nonetheless, there exists a paucity of research focused on Malayalam-English code-mixed texts and an absence of synthetic data augmentation techniques, such masked language modeling and few-shot learning. Furthermore, the majority of current methodologies insufficiently tackle issues of model explainability and justice, particularly in relation to gender-based hate speech and targeted communities. The subsequent research gaps identified from the previous study There is a lack of studies explicitly aimed at detecting hate speech that mixes Malayalam and English. Not many people employ synthetic data augmentation approaches (like MLM and FSL) to deal with a lack of data and an imbalance in classes. Not enough research on model explainability methods to understand predictions and find bias. There isn't enough focus on justice and unbiased detection, especially when it comes to abuse based on gender or misogyny. Mostly using small, manually annotated datasets makes it hard to generalize and make things stronger.
The proposed study directly confronts these constraints. We concentrate on Malayalam-English code-mixed hate speech, particularly abuse directed at women, an area that has been largely overlooked in previous research. To address the shortage and imbalance of annotated data, we investigate various synthetic data augmentation techniques, including Machine Translation (MT), Masked Language Modeling (MLM), and Few-Shot Learning (FSL). We also test a group of transformer-based models (BERT, mBERT, XLM-RoBERTa, IndicBERT, and MuRIL) to see how well they work with both real and augmented datasets. We also use LIME to do explainability analysis, which helps us understand how the model makes decisions and points up fairness issues. This makes the detection process clearer and more reliable than other methods.
Methodology
The proposed way to find hate speech in Malayalam-English code-mixed text is set up as a methodical pipeline, as seen in Figure 1. The first step is to gather real-life social media comments that are nasty and hateful, especially toward women. These comments are often very unstructured, with spelling mistakes, problems with transliteration, and a lot of moving between Malayalam and English. This makes preprocessing an important step. We fix this by normalizing the data, which means getting rid of extra characters, repeated letters, and noise. This makes sure that the input is clearer and better for training the model.
Proposed Methodology.
The pipeline uses three different synthetic data generation methods to make up for the lack of annotated data: Machine Translation (MT), Masked Language Modeling (MLM), and Few-Shot Learning (FSL). These approaches provide more samples to the training set by creating a variety of code-mixed examples. This makes the classes more balanced and helps the model generalize better. Using the enriched dataset, many transformer models, such as BERT, mBERT, XLM-RoBERTa, IndicBERT, and MuRIL, are then fine-tuned. Finally, conventional metrics like Accuracy, Precision, Recall, and F1-score are used to measure how well the model works. LIME-based explanations and error analysis are used to make sure that the model is easy to understand. This step-by-step architecture makes the suggested method strong, clear, and a good fit for dealing with the problems of finding hate speech in low-resource, code-mixed settings.
Here we have covered our strategy for creating synthetic data for hate speech detection in Malayalam-English code-mixed language using three methods:(1) Machine Translation (2) Masked language modelling (3) few shot learning using mT5.
The scarcity of labelled hate speech data often complicates the training of robust models. This study employs machine translation to transform Hinglish (Hindi-English code-mixed text) into Manglish (Malayalam-English code-mixed text), generating synthetic samples for model training. The procedure commences with the transliteration of Hinglish into Hindi utilizing the ITRANS system (Rani et al., 2021), transforming Romanized Hindi text into the Devanagari script. Subsequently, Google Translate is employed to convert the Hindi text into Malayalam (Nayak & Joshi, 2021), ensuring the preservation of the fundamental meaning. The Malayalam script is ultimately transliterated into Manglish (Romanized Malayalam) via ITRANS, resulting in a synthetically generated code-mixed sentence. The entire process is illustrated in Figure 2, and a sample example is provided in Figure 3
Machine Translation Approach for Synthetic Data Generation.
Sample Translation from Hinglish to Manglish.
Figure 2 shows the process that takes Hinglish (Hindi-English code-mixed) inputs and turns them into synthetic Manglish (Malayalam-English code-mixed) words and Figure 3 shows the process with the example. The first step is to write a Hinglish statement like “yeh movie bohot acchi hai, I really enjoyed it.” To start, the Hinglish part is written out in Hindi characters using the ITRANS technique. Then, Google Translate takes the Hindi-English statement and changes the Hindi parts to Malayalam while leaving the English parts alone. Using ITRANS, the Malayalam part is then transliterated into Latin script (Manglish). This makes a final Manglish-English code-mixed statement like “ee cinema valare nallathayirunnu, I really enjoyed it.” This pipeline is especially useful for making a wide range of training data that is appropriate for hate speech recognition in Malayalam-English code-mixed social media posts.
One disadvantage of this method is that during human evaluation of sentence efficiency, we observed that intra-sentential code-mixing sentences were ineffective, whereas inter-sentential code-mixing sentences performed better. Intra-sentential code-mixing occurs when multiple languages are integrated within a single sentence. This frequently entails the incorporation of words, phrases, or clauses from one language into another. Conversely, inter-sentential code-mixing transpires when a speaker alternates languages between sentences or clauses (Mandl et al., 2023).
We have implemented this method using the language model XLM-RoBERTa. The detailed methodology is given in Figure 4. Masked language models are essential as they can analyze the contextual meaning of words more efficiently than conventional models. We have dynamically obscured specific words in a sentence and employed XLM-RoBERTa to predict the masked word in a Manglish sentence. As a multilingual model, our expectations for the output were elevated. However, the actual output produced by the pretrained XLM-RoBERTa was subpar and chaotic.
Methodology for MLM for Synthetic Data Generation.
We have considered fine-tuning XLM-Roberta utilizing the existing Manglish dataset employed in our prior research (Khullar et al., 2024; Rani et al., 2020). Prior to finetuning, we eliminated extraneous noise from the text data, tokenized it, and subsequently employed it for finetuning, utilizing dynamic masking (Singh & Mamidi, 2020). In dynamic masking, randomly introduce [MASK] tokens within the text to facilitate MLM training. This facilitates the model's acquisition of contextual word substitutions. The finetuned model has been saved as
The Facebook AI team launched XLM-RoBERTa (
Sample Output of Proposed MLM Model*.
Sample Output of Proposed MLM Model*.
The necessity for substantial training data in deep neural networks to attain favourable outcomes renders these models inefficient in scenarios where training data is scarce. Hate speech and offensive language are prevalent phenomena on social media that disregard linguistic barriers; consequently, the scarcity of adequately labelled data in certain languages, particularly low-resource ones, renders automatic detection algorithms impractical (Patwa et al., 2020).
Few-shot learning lets large language models (LLMs) create synthetic data without needing a lot of fine-tuning. This makes it an efficient and cost-effective way to add to data. The quality of the hate speech data that is made depends a lot on how well the prompts are designed (Bohra et al., 2018). This is where well-structured examples tell the model what to do based on the context. But validation is still an important step to make sure that the generated text fits the target context and shows hate speech patterns correctly. It is possible to make hate speech detection models much more effective by adding validated synthetic data to the training dataset. This leads to better generalization and robustness in real-world applications.
First, a small group of real hate speech examples in Manglish (Malayalam-English code-mixed) are gathered to prepare the ground for few-shot learning. After that, carefully thought-out prompts are made using these examples in a structured way to help the language model (LLM) make synthetic hate speech. A few examples (usually 3–5) are used in few-shot learning to help the language model learn the style, structure, and context of hate speech in Manglish. The model, mT5, makes up new synthetic hate speech based on the patterns it has learned after it has been trained. The outputs are then post-processed, which means they are looked over and filtered to get rid of text that isn't relevant or isn't good enough. This is done by manually verifying, toxicity filtering, and de-duplicating the text. Lastly, the validated synthetic hate speech dataset is put together and saved so that it can be used to train and improve models that find hate speech. The detailed methodology is depicted in Figure 5.
Few Shot Learning Methodology to Create Synthetic Data.
The mC4 dataset has natural text in 101 languages and is used to train the mT5 model. The model can understand shared representations and common patterns across different languages thanks to this in-depth training (Kamble & Joshi, 2018). There are several good reasons why mT5 (Multilingual T5) is a good choice for making synthetic data for Manglish (Malayalam-English code-mixed) hate speech detection: mT5 is already trained on several languages, such as English and Malayalam, which makes it a good choice for working with code-mixed text like Manglish. mT5 uses the text-to-text paradigm of the T5 model, which makes it very adaptable to tasks like creating text, translating it, and sorting it into groups. This makes it easy to change for making synthetic data with well-thought-out prompts. mT5 can make relevant synthetic hate speech samples with just a few examples (few-shot learning) because it has been trained on different types of text.\
Prompting
It involves making and improving the instructions that are given to a Large Language Model (LLM) so that it can produce accurate and useful results (Mane et al., 2025). It is very important to carefully plan the prompts so that the model can find hate speech in code-mixed Malayalam and English data. In few-shot learning, prompts are provided with 3 to 4 labelled examples, as illustrated in Table 3. Here Every sentence adheres to the abusive patterns outlined in the prompt. It adversely affects women, perpetuating patterns of hate speech. The syntax and style are preserved in Manglish to maintain authenticity.
Sample Output of mT5 After few Shot Learning*.
Sample Output of mT5 After few Shot Learning*.
This section talks about the experimental setup, model settings, dataset splits, and how to look at the findings. We evaluated multiple transformer models, including BERT, mBERT, XLM-RoBERTa, IndicBERT, and MuRIL, using synthetic data generated via different augmentation techniques. We used the Hugging Face Transformers library with a PyTorch backend to fine-tune all of the models. The training and testing were done using a computer with 32GB of RAM, an NVIDIA RTX 3090 GPU, and Ubuntu 20.04 as the operating system. The dataset utilized in all of the studies included real hate speech remarks as well as synthetic comments made with Machine Translation (MT), Masked Language Modeling (MLM), and Few-Shot Learning (FSL).
We used a systematic grid search technique to choose the hyperparameters in Table 4. This made sure that the model worked as well as possible while still being fair among different transformer architectures. We tried different settings for the learning rate (1e-5, 2e-5, 3e-5), batch size (8, 16), and dropout rates (0.1, 0.2) for each model and judged them based on validation loss and F1-score. The last values that were presented are the ones that always led to stable convergence and high generalization across runs. IndicBERT did the best with a learning rate of 3e-5, while mBERT and MuRIL did the best with a learning rate of 2e-5. We utilized the AdamW optimizer and early halting (patience = 5) in all of the models to keep them from overfitting during training. We used grid search to look into how different algorithmic parameters affected the results and find the optimal combinations that balanced training speed and classification accuracy. This methodical adjustment approach made the experiments more rigorous and made it easier to repeat them. In the future, researchers could look into more advanced optimization methods, such Bayesian optimization, or change the hyperparameters on the fly while training. But the current grid search method made sure that both actual and synthetic Malayalam-English code-mixed datasets were fair, open, and had good empirical performance.
The Hyperparameters Used for Each Transformer Model.
The Hyperparameters Used for Each Transformer Model.
The trials demonstrated that the selection of hyperparameters directly impacted both the pace of convergence and the ultimate classification performance. For example, we may lower the learning rate to 1e−5 often made convergence slower and sometimes caused underfitting. Raising it to 3e−5 made IndicBERT more stable, but it made BERT and XLM-RoBERTa's performance a little less stable. Increasing the batch size to 16 made training go a little faster, but it made the models less able to generalize when tested on synthetic data. This suggests that smaller batches helped the models pick up on more subtle code-mixed patterns. Dropout values more than 0.1 typically lowered performance, indicating that excessive regularization impeded the models’ ability to acquire essential language differences in code-mixed text.
The hyperparameter tuning studies showed that transformer models are somewhat sensitive to learning rate and batch size, but their performance trends were the same for both real and synthetic data. The settings shown in Table 4 provide the best compromise between stability, generalization, and speed of computing. This also illustrates that synthetic augmentation didn't need a lot of different adjustment from real data, which supports the strength of our method. These results show how important it is to tune transformers systematically when using them in low-resource, code-mixed situations.
This section discusses about the experimental setup, such as the assumptions, platform, datasets, augmentation methods, model configurations, evaluation criteria, and experimental outcomes. To make it easier to read, we break the information down into the following subsections:
A set of thoroughly thought-out assumptions is used to judge transformer models for finding hate speech in Malayalam-English code-mixed data. These assumptions make sure that experiments with both actual and synthetic datasets are fair, consistent, and can be repeated. During data preparation, model training, and evaluation, the following assumptions were made: Fair Test Conditions for All Models: We trained and tested all of the transformer models, including BERT, mBERT, XLM-RoBERTa, IndicBERT, and MuRIL, in the same way. To make sure the comparisons were fair, each model was tested on the same data (real data) Data splits that are always the same: We divided the real dataset into two parts: training, and testing. We only used synthetic data to add to the training set; therefore, the test set stays the same and is not changed in any of the experiments. The data split is given in Table 5. Synthetic Data Only for Training: We only contributed synthetic samples made by Machine Translation (MT), Masked Language Modeling (MLM), and Few-Shot Learning (FSL) to the training data. No synthetic data was used in the validation or test phases, which kept the evaluation honest. Equal Representation of Enhanced Data: To avoid bias toward any one augmentation strategy, an equal number of synthetic samples from each method were used when combining MT, MLM, and FSL procedures. Evaluation Based on Parts of the Confusion Matrix: We used the same confusion matrix that came from test set predictions to figure out all of the performance metrics: Precision, Recall, Accuracy, and F1-Score. The definitions of True Positives (TP), False Positives (FP), True Negatives (TN), and False Negatives (FN) were used in the same way throughout. F1-Score is the Main Metric: Because hate speech data is not evenly distributed, the F1-score was chosen as the main way to measure how well the classification worked. It strikes a good mix between Precision and Recall, which makes it perfect for this job. Multilingual pre-trained models like mBERT, MuRIL, and IndicBERT are usually thought to work better with code-mixed data than monolingual models like BERT because they can better capture the semantic and syntactic differences between languages. Hardware and computing power: All of the tests were run on computers with similar processing power (they all have GPUs) so that differences in training dynamics wouldn't happen because of a lack of resources.
Distribution of Data for Different Experiments.
Distribution of Data for Different Experiments.
Hardware (GPU, RAM, OS)
Software (PyTorch, Hugging Face, Python version)
Implementation libraries (transformers, tokenizers)
Dataset
Real dataset collection (YouTube scraping, 48 videos, hate & non-hate examples, annotation details, Cohen's Kappa = 0.94)
Synthetic dataset generation (MT, MLM, FSL, combination)
Final splits (Table 5)
Research Questions
RQ1: How well do transformer models (BERT, mBERT, XLM-R, IndicBERT, MuRIL) work for finding hate speech in Malayalam-English?
RQ2: Does synthetic augmentation (MT, MLM, FSL) make things work better than just using real data?
RQ3: Which method of augmentation gives the best results?
RQ4: Does using more than one augmentation approach make generalization better?
Experimental Setup
This section discusses five experiments that used a mix of genuine and fake Malayalam-English (Manglish) hate speech data. There are five transformer models—BERT, mBERT, XLM-RoBERTa, IndicBERT, and MuRIL—that are tested in each experiment using different data augmentation methodologies. The goal is to find out how different ways of adding noise affect how well a model works at finding hate speech in noisy, code-mixed environments.
Experiment on Baseline Model Using Real Data
In Experiment 1, we utilized authentic data collected from 48 YouTube videos sourced from prominent Malayalam channels. Due to confidentiality concerns, we are unable to disclose channel names here. Due to their popularity, these videos garnered a substantial number of comments and reactions. We have concentrated on a channel predominantly led by women, featuring content such as cooking, beauty vlogs, and dieting vlogs. Utilizing the YouTube Data API, we collected comprehensive information on each video, encompassing its ID, title, comments, replies to comments, likes, date, and time. We collected approximately 5,500 hate comments related to these videos. The predominant comments were submitted from January 2024 to January 2025. The extracted comments and replies from each video were stored in distinct CSV files in chronological order. Ultimately, we amalgamated the dataset into a singular CSV file comprising solely the “comments” column. We must focus solely on Manglish comments. The presence of 5500 comments is attributed to the fact that many channels have removed negative comments, specifically those related to bullying, from their videos, which presents a challenge. From a total of 5500, we selected only 3000 for dataset balancing and gathered an additional 3000 non-hateful comments concerning women. In this study, two annotators proficient in both Malayalam and English manually annotated the dataset to identify hate speech. Cohen's Kappa (Miškolci et al., 2020) was employed to evaluate inter-annotator agreement for hate speech annotations across two collections of 10,000 code-mixed texts, resulting in a Kappa score of 0.94, signifying high-quality annotations. In the study, hate speech is designated as “HATE,” whereas non-hateful messages are designated as “NHATE.” We collected an additional 1,200 comments, both hateful and non-hateful, for training purposes, as this is standard practice for all experiments.
Experiment on Machine Translation-Based Synthetic Data
The comprehensive distribution of datasets and sources is presented in Table 5. We started with Hinglish sentences and used ITRANS to change them into Hindi. After that, Google Translate turned these sentences into Malayalam, and then they were transliterated back into Latin script to make Manglish-English code-mixed sentences. As a result, there were 5,892 samples in the final dataset. The goal of the synthetic augmentation was to make the data more diverse, however the results showed that performance dropped since the real and synthetic data were not structured or contextualized in the same way. This experiment shows how hard it is to use MT-generated text to add to complicated code-mixed datasets. The creation of synthetic data through machine translation of Hinglish sentences, combined with authentic data sourced from social media, may result in increased confusion and diminished effectiveness. The writing style, specifically the sentence structure in Manglish and Hinglish, differs, which has also contributed to the issue. The lack of success of this experiment necessitates a third experiment that integrates real data with synthetic data generated through the MLM method.
Experiment on MLM-Based Synthetic Data
In Experiment 3, we combined 3,000 posts from authentic data with 2,860 posts from synthetic data generated through the masked language modelling method discussed in Section 3.2. The comprehensive distribution of datasets and sources is presented in Table 5. Synthetic data augmentation based on Masked Language Modeling (MLM) surpassed that of Machine Translation (MT) owing to its capacity to produce more natural and contextually relevant data. MLM produces synthetic samples by predicting masked words within a given linguistic context, thereby maintaining the original grammatical structure and fluency of code-mixed Malayalam-English text. Conversely, MT-based data frequently results in translation inaccuracies, awkward phrasing, or misinterpretations of colloquial language, thereby diminishing training efficacy. The dataset utilized for Experiment 1 is specifically employed for MLM-based synthetic data generation using XLM-Roberta. Our proposed “
Experiment on Few-Shot Learning-Based Synthetic Data
In Experiment 4, we combined 3,000 posts from authentic data with 2,857 posts from synthetic data generated through the few shot learning using mT5 model discussed in Section 3.3. The comprehensive distribution of datasets and sources is presented in Table 5. The Few-Shot Learning (FSL) approach utilizing synthetic data markedly enhanced the model's efficacy in detecting hate speech in Malayalam-English (Manglish). In contrast to Machine Translation (MT) and Masked Language Modeling (MLM)-based augmentation, FSL-generated samples yielded a more contextually precise and varied array of hate speech variations. Future research may concentrate on refining the FSL model to augment diversity and reduce biases, potentially employing contrastive learning techniques to enhance semantic representation in hate speech classification.
Experiment on Combination of Synthetic Data
In Experiment 5, we amalgamated 3,000 posts from genuine data with 3,000 posts from synthetic data produced via three distinct methods: 1,000 from Machine Translation (MT), 1,000 from Masked Language Modeling (MLM), and 1,000 from Few-Shot Learning (FSL). Table 5 presents the extensive distribution of datasets and sources. The goal was to combine the different patterns and language structures that each augmentation approach added into a single training dataset. MT helped add syntactic differences by translating Hinglish inputs through several language stages. MLM made sure that sentences made sense in their context by predicting masked tokens in Manglish text. FSL added creativity and variety by using prompt-based generation that was tailored to hate speech patterns. The goal of using these several types of augmentation together was to improve model generalization by exposing it to a wider range of expressions and vocabulary that are common in code-mixed and abusive language. This test is very useful for multilingual situations with few resources, like Malayalam-English code-mixed data, where there aren't many high-quality annotated samples.
Experimental Protocol
There were five steps in the experimental workflow: preparing the data, partitioning the dataset, choosing a tokenizer, training the model, and evaluating it. We used the tokenizers and pretrained weights for each transformer model to fine-tune them so they could handle Malayalam-English code-mixed text. Cleaning up the text data involved getting rid of extra characters (such punctuation, emojis, too many spaces, and special characters), changing everything to lowercase, and filtering out everything that wasn't in Manglish. This ensured dataset consistency and reduced noise Tokenizers: Each transformer model was paired with its own tokenizer to make sure that code-mixed input was handled as well as possible: The standard WordPiece tokenizer was used by BERT. The multilingual WordPiece tokenizer was utilized by mBERT. The SentencePiece tokenizer trained on more than 100 languages was utilized by XLM-RoBERTa. IndicBERT uses a SentencePiece tokenizer that was made for Indian languages. MuRIL employed its own multilingual tokenizer that was made just for Indian scripts. Dataset Splits: Each dataset was split up so that 80% was used for training and 20% was used for testing. We didn't use a validation split because we didn't have enough real data. Instead, to avoid overfitting, early halting was adopted. Number of Trials: The final performance measurements were based on the average of three runs of each test, each with a different random seed (42, 56, and 101). We also calculated the standard deviation across runs, which was always less than 1%. This shows that the models were stable and reliable. Hardware Setup: All of the tests were done on an NVIDIA A100 40GB GPU with 512GB of RAM, which was running Ubuntu 20.04 with Python 3.10 and PyTorch 2.0. The Cross-Entropy loss was utilized for binary classification as the loss function and optimizer We used the AdamW optimizer with a learning rate of 2e-5, a weight decay of 0.01, and a dropout chance of 0.1.
This complete setup makes sure that the results are consistent, can be repeated, and show how well the model works in the actual world at detecting code-mixed hate speech.
Evaluation Metrics
We used common classification measures including Precision, Recall, Accuracy, and F1-Score to compare how well different transformer-based models worked at finding hate speech in Malayalam-English code-mixed text. These metrics give a full picture of how well each model works with different types of datasets. Here are the definitions and formulas that were used: True Positive (TP): Hate speech incidences that were correctly predicted False Positive (FP): Non-hate communication that was wrongly thought to be hate speech. True Negative (TN): correctly guessed instances of speech that wasn't hate speech. False Negative (FN): When hate speech is wrongly thought to be non-hate discourse. Precision tells you how many of the projected hate speech examples are actually hate speech
Recall tells you how many real hate speech examples were successfully identified
Accuracy shows the total proportion of samples that were properly classified
F1-Score is the harmonic mean of precision and recall. It is especially helpful when the data is not balanced
To guarantee the reliability of the stated outcomes, each experiment was conducted three times using distinct random seeds. We provide the average Precision, Recall, Accuracy, and F1-score values, as well as their standard deviations. A paired t-test was utilized on F1-scores from model comparisons to verify that the discovered enhancements were statistically significant (p < 0.05). This method is statistically sound and takes into account random differences between trials.
This section goes into great depth on five transformer models—BERT, mBERT, XLM-RoBERTa, IndicBERT, and MuRIL—that can find hate speech in Malayalam-English code-mixed text. We trained the models on both real datasets and synthetic datasets created using three different ways to add data: Machine Translation (MT), Masked Language Modeling (MLM), and Few-Shot Learning (FSL), as well as a combination of these three. The criteria utilized to evaluate include precision, recall, accuracy, and the F1-score. Table 6 shows all of the results.
Performance of various Transformer Models on Different Datasets.
Performance of various Transformer Models on Different Datasets.
The baseline experiment tested all models with just the original Malayalam-English code-mixed dataset, which had 3,000 hateful comments and 3,000 non-hateful comments. The mBERT transformer model has the greatest F1-score of 81.53% out of the five evaluated. This shows that it is very good at understanding multilingual and code-mixed language patterns. This is because it was trained on a lot of different languages, including Indian languages, which makes it better at generalizing across mixed-language inputs. F1-scores of 73.41% and 72.15% followed for XLM-RoBERTa and BERT, respectively. BERT is pretrained on English-only corpora, but its good score suggests that a large part of the code-mixed dataset is in English, which is why it still works well. XLM-RoBERTa has a greater recall (91.67%) and a lower precision, which means that it tends to label hate speech too often but finds more relevant cases. Both IndicBERT and MuRIL were made to work well with Indian languages and got scores of 65.13% and 64.29%, respectively. Their lower scores are probably because they are smaller or newer models that haven't had as much training on code-mixed or informal social media data. These baseline results create an important starting point for figuring out how synthetic data augmentation works and show how hard it is to deal with code-mixed hate speech, which is informal, noisy, and uses a lot of different languages.
Performance with MT-Based Synthetic Data
Adding 3,000 synthetic samples made by Machine Translation (MT) to the real dataset had mixed effects on the models. BERT and XLM-RoBERTa both showed big improvements in performance, with F1-scores of 77.45% and 77.62%, respectively. This means that the translated content added useful language differences that fit well with the training data for these models, especially given both were trained on a lot of English or multilingual data. The boost in performance for BERT, in particular, suggests that the translated synthetic data kept English-dominant structures, which the model was able to work with well. On the other hand, mBERT's performance went down to an F1-score of 72.01%, which is lower than the 81.53% it got on the real dataset. This drop means that the text generated by MT may have added translation artifacts or linguistic patterns that were not natural and did not work with mBERT's multilingual embeddings. mBERT is sensitive to how consistent languages are with each other. If synthetic samples weren't aligned well, it might have misclassified some instances, especially if the syntactic structures were different from how code-mixed use is normally utilized.
IndicBERT and MuRIL's F1-scores went up a little bit from their starting points, to 67.60% and 70.07%, respectively. Both models are made to work with Indian languages, but their performance improvements weren't as clear. This could be because the MT process focused more on translating English material into Malayalam or the other way around, rather than making naturally interwoven code-mixed sentences. In general, MT-based augmentation adds more words, but it could not have the same level of context and casual style as real code-mixed social media content. This could make it less useful for models that are trained on more linguistically sensitive embeddings.
Performance with MLM-Based Synthetic Data Augmentation
Adding 3,000 synthetic samples made using Masked Language Modeling (MLM) to the real dataset made all transformer models work better or stay the same. mBERT's F1-score of 84.48% is a big improvement above its baseline score of 81.53%. MuRIL also made a big jump to 75.02%, and IndicBERT and BERT got F1-scores of 75.93% and 71.90%, respectively. These steady improvements show that using MLM-based augmentation is a good way to improve model performance across different architectures. The main benefit of data generated by MLM is that it keeps the meaning and fluidity of the sentences it creates. Machine Translation can cause structural mismatches or translations that are too literal, but MLM works within the same language distribution as the original training data. The synthetic text stays closer to naturally occurring Malayalam-English code-mixed patterns by masking and predicting tokens in context. This helps make phrases that are both grammatically and contextually correct, which is important for training models that use contextual awareness for classification tasks. MBERT gets a lot out of this addition because its pretraining goal also includes predicting masked tokens. The fact that the model's pretraining and the synthetic data creation technique are in sync may make it better at using MLM-augmented samples in a useful way. IndicBERT and MuRIL, which were also trained on Indian language corpora, seem to work better with MLM-based data. This shows how strong the approach is at keeping the cultural and linguistic differences that are common in informal social media language.In general, synthetic augmentation based on MLM adds to the training set without adding strange patterns or translation noise. This helps with strong generalization, especially in tasks that involve informal and linguistically diverse text, such code-mixed hate speech.
FSL-Based Synthetic Data Augmentation
Using Few-Shot Learning (FSL) to create synthetic data using prompt-based generation significantly improved the performance of all transformer models. mBERT got an F1-score of 85.90%, which was better than the scores it got with both MT and MLM augmentations. BERT and IndicBERT also made big strides, achieving 74.30% and 77.62%, respectively. These improvements show that FSL works well for making high-impact synthetic samples, especially in situations where there aren't many resources or where code is mixed.
FSL's main strength is that it can make samples that are contextually suitable and semantically diverse with very few input examples. MT frequently creates rigid and translation-biased structures, and MLM is limited to masked-token prediction inside existing contexts. FSL, on the other hand, lets the model create whole sentences from scratch based on task-specific suggestions. This lets the synthetic data pick up on more subtle expressions, different sentence structures, and different ways that people talk on social media, all of which are important parts of social media discourse. The prompt-based generating method is also flexible since it can include hate speech types (including sexist, casteist, and communal) and language cues that are specific to a certain field. This makes the data more diverse and helps the model generalize better. This variety is especially good for models like mBERT, which are trained on multilingual corpora and can learn better from samples that are synthetically different but still fit the context. For IndicBERT, FSL-generated samples helped connect pretraining on Indian languages with the informal code-mixed data found on social media, which made it more robust. BERT, on the other hand, is not multilingual, but it did benefit from the semantically rich input space that FSL added, which made it better at telling the difference between languages even when they were mixed together. Overall, FSL is a great way to add synthetic data to hate speech detection in code-mixed text. It strikes a good balance between linguistic inventiveness, task relevance, and lexical diversity, which are all important for modeling language combinations that aren't very common, like Malayalam-English.
Performance with Combined Synthetic Data (MT + MLM + FSL)
The last experiment used 1,000 samples from each of the three methods—Machine Translation (MT), Masked Language Modeling (MLM), and Few-Shot Learning (FSL)—along with 3,000 real examples. This varied augmentation method gave almost all transformer models the best overall performance. mBERT got the best outcome, with an F1-score of 86.42%, which is better than any of its prior settings. IndicBERT and BERT likewise did their best, with F1-scores of 78.52% and 78.02%, respectively. XLM-RoBERTa and MuRIL both showed moderate improvements (77.50% and 73.15%), however their performance stayed rather constant compared to other techniques of augmentation. The significant enhancement seen with integrated synthetic datasets underscores the necessity of diversity in training data. Every method for making synthetic data adds its own linguistic and contextual features: MT introduces structural differences by using bilingual corpora, which often look like real-world user-generated text that has been translated from regional languages to English. MLM adds syntactically correct completions and replacements that keep the context, which helps models learn different but still believable ways to say hateful things. FSL provides samples that are very contextual and semantically rich, and the prompts are designed to fit hate speech categories, idioms, and colloquialisms that are widespread in Malayalam-English code-mixed social media.
When we use all three of these tactics together, the dataset better reflects the linguistic complexity, code-switching behavior, and informal usage patterns that are common in the target domain. This lessens the chance of overfitting on rigid or repeating patterns that are common in single-strategy synthetic datasets and makes it easier to generalize across domains. Also, transformer models, like mBERT and IndicBERT, get better with this multimodal enrichment because they were trained on multilingual and culturally diverse corpora. The synthetically augmented inputs help to better match downstream task representations with pretraining information, which makes categorization more consistent across different types of input. In short, this experiment shows how useful hybrid data augmentation may be when used together. It indicates that no one method can fully capture the diversity of hate speech that mixes several languages. Strategic integration of MT, MLM, and FSL, on the other hand, makes the training dataset more balanced and complete, which leads to better model performance, especially in multilingual, low-resource, and high-noise settings like social media. Figure 6 shows a comparison of F1-scores for five transformer models—BERT, mBERT, XLM-RoBERTa, IndicBERT, and MuRIL—that were trained on Malayalam-English code-mixed hate speech data using different combinations of real and synthetic datasets. We picked the F1-score as the main evaluation metric because it balances precision and recall. This gives us a more complete picture of how well the model works in situations when false positives and false negatives are equally bad, like when detecting hate speech. In this task, strong recall makes sure that dangerous content is not missed, while high precision makes sure that harmless content is not misclassified. The F1-score is a very important measure of how well a model can produce correct and consistent classifications in the real world, where data is often imbalanced and noisy.
F1-Score Comparison of Transformer Models Across Real and Augmented Datasets.
An error analysis was done on the False Positives (FP) and False Negatives (FN) across different transformer models and dataset variants to get a better idea of how well hate speech recognition algorithms work. The stacked bar chart in Figure 7 shows how the errors changed for five models—BERT, mBERT, XLM-RoBERTa, IndicBERT, and MuRIL—trained on five different dataset setups: Real, Real + MT, Real + MLM, Real + FSL, Real + MT + MLM + FSL. False positives are cases where content that isn't hateful is wrongly labeled as hate speech, and false negatives are cases where hateful content isn't found. To make sure that hate speech classification is fair and accurate, a balanced model should reduce both sorts of errors.
Error Analysis Using FP and FN.
When trained on the combined synthetic dataset (Real + MT + MLM + FSL), all models consistently showed lower error rates (both FP and FN). This variation led to the most stable and balanced performance, especially for mBERT, which had the fewest errors overall. This shows how important it is to use a variety of data augmentation methods to improve generalization.
mBERT had the fewest overall errors across all dataset types when compared to other transformer models. Its better performance, especially when combined with MLM, FSL, and their combination, confirms its strength in recognizing multiple languages and code-mixed languages. This makes it perfect for finding hate speech in Malayalam-English.
Most models, including mBERT and IndicBERT, had more false positives (FP) and false negatives (FN) when they were trained on the MT-only synthetic dataset. This makes it possible that machine translation may add synthetic language patterns or noise, which would make it hard to tell what anything is.
The benefit of FSL and MLM is that the synthetic data created by Few-Shot Learning (FSL) and Masked Language Modeling (MLM) always leads to fewer mistakes. These methods add depth to the context and variety to the language, which probably makes the model better at generalizing and finding hate speech trends that aren't obvious.
BERT always had greater error rates, which means it wasn't very good at adapting to code-mixed texts without more multilingual pretraining-RoBERTa and MuRIL did okay, but they were weak against FP on the Real and MT datasets. The combination of synthetic datasets helped IndicBERT a lot, especially when it came to lowering FN.
Computational Efficiency
We also looked at how well the transformer models we used in our study worked in terms of speed and accuracy. The trials were done on an NVIDIA Tesla V100 GPU with 128 GB of RAM and 32 GB of memory. Table 7 shows the average time it took to train and test each model on the combined dataset (Real + MT + MLM + FSL).
Average Computational Time of Transformer Models.
Average Computational Time of Transformer Models.
Lightweight Models: IndicBERT had the fastest training (15.8 min) and inference timings (8.7 s per 1k samples), which is what you would anticipate since it has a lot less parameters.
Balanced Trade-off: mBERT took a moderate amount of time to train (30 min for 10 epochs) but had the best F1-score, which shows that it strikes a good balance between performance and efficiency.
High-Capacity Models: XLM-RoBERTa and MuRIL needed more time to train and provide predictions since they had so many parameters, but they still worked well.
Explainability Analysis
The efficacy of using context-preserving augmentation techniques is confirmed by the performance gains seen during training on synthetic data produced using MLM and FSL techniques. We used the Local Interpretable Model-agnostic Explanations (LIME) (Miškolci et al., 2020) framework to conduct an explainability analysis in order to gain a better understanding of the fundamental elements causing these performance gains. By analysing the role that individual words and subwords play in the classifier's decision-making process, LIME gives us a better understanding of how the model detects hate speech in code-mixed text. Figure 8 shows explainability analysis of a sentence (Hate speech). The explainability analysis also shows how contextual words affect the model's judgment when detecting hate speech. Words like “waste,” “pennu,” and “ariyilla” contribute a lot, indicating that the model is good at spotting offensive language patterns that are frequently used to disparage women in online discourse. Notably, the classifier shows that it can capture implicit hate speech by identifying subtle derogatory expressions in addition to explicit slurs. Furthermore, the contrast between neutral words (blue) and highly contributing words (orange) emphasizes how crucial context is to code-mixed language processing. Functional words like “oru” and “aanu” are given little weight by the classifier, highlighting the importance of gender-specific or demeaning terms in classification. By improving the model's capacity to identify both overt and covert instances of hate speech in Malayalam-English code-mixed text, this analysis demonstrates the value of using MLM and FSL-generated synthetic data.
Explaining why the Post is Labelled as HATE Sentence is: “eee Pennu Total Waste Aanu oru Paniyum Ariyilla”.
The model indicates strong confidence in the offensive nature of the hate speech sentence (“Ee pennu total waste aanu, oru paniyum ariyilla”) by predicting a high probability for the Hate category. Words that convey negativity and demeaning intent toward women, such as “waste,” “paniyum,” and “ariyilla,” contribute significantly to this classification. On the other hand, the non-hate speech sentence (“Ee pennu nannayittundu, aval nalla pani cheyyum”) is highly likely to be predicted as such. Words that reflect appreciation and a neutral or positive tone, such as “nannayittundu” (is good), “nalla” (good), and “cheyyum” (does), positively contribute to this classification (Figure 8).
Despite appearing in both sentences, the word “pennu” (woman) is classified differently depending on the words that surround it. This is an important observation. It coexists with disparaging terms in the hate speech case, which intensifies the negativity. It is linked to affirming words in the non-hate sentence, on the other hand, which results in a positive classification. This demonstrates how contextual meaning, not just word choice, is essential in identifying hate speech.
Furthermore, the model's dependence on overtly offensive language for hate speech detection raises the possibility that it will have trouble identifying more covert hate speech, such as that which uses sarcastic or indirect language. These subtle patterns can be recognized with the aid of techniques such as Few-Shot Learning (FSL) and Masked Language Modeling (MLM).
Particularly in multilingual and code-mixed contexts, the issue of fairness in hate speech detection is of paramount importance (Vargas et al., 2021). Disparities in model performance may arise as a consequence of biases in training data, which can result in the unfair treatment of various groups. We utilize the subsequent metrics to evaluate fairness:
False Positive Rate (FPR): This Indicates the frequency with which non-hate speech is incorrectly classified as hate speech (Parvaresh & Harvey, 2023).
False Negative Rate (FNR): This measure Indicates the frequency with which hate speech is incorrectly classified as non-hate speech (Parvaresh & Harvey, 2023).
The model's performance is compared in the fairness analysis with respect to: Group 1: The actual dataset containing hate speech. Group 2: Hate speech in the synthetic dataset (produced by MT, MLM, and FSL). We evaluate whether the model demonstrates substantial discrepancies between these groups, which could suggest a potential bias in the treatment of real versus synthetic data. It is given in Table 8
Fairness Analysis.
In comparison to synthetic data-enhanced models, the Real Dataset model has a higher FPR (20.94%), suggesting that it is more likely to incorrectly identify non-hate speech as hate speech. The FPR (28.57%) and FNR (25.96%) of the Real + Synthetic (MT) model are the highest, indicating that it experiences difficulties with both false positives and false negatives. MLM and FSL-based synthetic data enhance the model's capacity to distinguish between hate and non-hate speech, thereby reducing both FPR and FNR, thereby improving fairness. The FPR (14.47%) and FNR (12.79%) of the model with the highest performance (MT + MLM + FSL) are the lowest, indicating that the integration of multiple synthetic data techniques enhances fairness.
Demonstrating that the performance of hate speech detection is improved by the combination of multiple synthetic data generation techniques, the model trained on Real + Synthetic (MT + MLM + FSL) data achieves the highest accuracy (86.15%) and F1-score (86.42%).
The LIME explainability analysis demonstrates that contextual meaning is a critical factor in the detection of hate speech, rather than relying solely on individual word occurrences.
The results suggest that the fairness of classification is enhanced by the integration of synthetic data based on MLM and FSL, which reduces both over-flagging (FPR) and under-detection (FNR).
Discussions
This section goes into great detail about the experimental results and critically looks at how well the suggested approaches for creating synthetic data and transformer-based models work for finding hate speech in Malayalam-English code-mixed text. The talk combines the results of several experiments, explains what they mean, and thinks about how strong and better the proposed strategy is compared to baseline procedures.
The findings show clearly that the baseline model, which was trained just on real data, did rather well, with an F1-score of 81.53%. When synthetic data was included, though, performance changed substantially depending on how it was made. Using Machine Translation (MT) to create synthetic data made performance go down (the F1-score dropped to 72.01%). This shows that MT can make data bigger, but it might not keep the linguistic authenticity or naturalness of code-mixed hate speech. This discovery shows how important it is to have good data instead of a lot of it in places with few resources and many languages.
On the other hand, using synthetic data based on Masked Language Modeling (MLM) made a big difference, raising the F1-score to 84.48%. This shows that MLM-generated data can keep the context consistent since its probabilistic token replacement technique is more like real code-mixed patterns. Few-Shot Learning (FSL) showed even more improvements. The mT5 model created synthetic data that resulted in an F1-score of 85.9%, which shows that it can create hate speech examples that are rich in meaning and varied from very few prompts. Experiment 5's combination of all three approaches produced the best F1-score of 86.42%. This shows that using different generation strategies together makes training data more reliable and less likely to overfit to synthetic artifacts.
When we look at several transformer topologies, it's clear that the recommended technique is better. The main focus was on the baseline mBERT model, but other models including BERT, XLM-RoBERTa, IndicBERT, and MuRIL were also tested to see if they could be used in other situations. We used the same real test dataset to test each model under the identical conditions. By using more than one transformer model, we made sure that our results are not limited to only one architecture, which makes the proposed method more reliable. We used grid search to adjust important hyperparameters for each model, such as the learning rate, dropout rate, and batch size. These settings had effects on how quickly and how stable the model converged. For instance, small changes in dropout have a big effect on F1-scores in high-capacity models like XLM-RoBERTa. These results show how important it is to do hyperparameter sensitivity analysis, especially when working with languages with few resources and data that is combined with code. Our method is even more reliable because it works the same way in all of our tests. We did each experiment three times with different random seed values and presented the average stats. This made random initialization less of a problem and made sure that the results were statistically sound. We also showed that the suggested method doesn't rely too much on any one augmentation methodology by adding models trained on synthetic data created using a variety of strategies. This makes the method more useful in various code-mixed situations.
In short, the combination of different approaches for generating synthetic data, careful tweaking of hyperparameters using grid search, and a comparison of multiple transformer models all show how successful and strong our strategy is. These contributions fix problems with previous work that either didn't cover enough languages or used too few ways to add to the data. The suggested method creates a more generic and reliable framework for finding hate speech in multilingual social media settings by focusing on language variety, empirical validation, and parameter tweaking.
5.1 Policy Implications
The suggested solution gives social media sites the ability to find hate speech in under-resourced, code-mixed languages like Malayalam-English, which current moderation algorithms routinely miss. This helps to create content moderation frameworks that include everyone in the region.
Detecting hate speech in code-mixed formats is in keeping with digital safety policies that want to stop online harassment. Policymakers can use these results to make sure that people follow hate speech rules in their area, especially in places like India where several languages are spoken.
The study helps reduce language bias in AI systems by making them work better in languages with few resources. Policymakers can require that AI-driven content regulation technology reflect all languages fairly.
Social media sites can utilize these kinds of models to develop automated filters or flagging systems that work with regional languages. This cuts down on the need for manual moderation and makes it easier to respond to abusive content more quickly.
The methodology focuses on making synthetic data, which means less reliance on collecting a lot of user data. This helps AI development that protects people's privacy and is in line with global data protection laws like the GDPR and India's Digital Personal Data Protection Act.
Governments and NGOs can use what they learn from these models to better understand how people use language when they are being abusive online. This can help them design better instructional programs or counter-speech strategies in schools and communities.
Limitation and Future Work
Our work shows that synthetic data can help find hate speech in Malayalam-English code-mixed text, however there are some problems that need to be addressed. The primary constraints of this study encompass its dependence on a language-specific dataset, the challenges in guaranteeing linguistic quality in synthetic data, the computing demands of transformer models, and the ethical dilemmas that may emerge from text generation. First, our dataset is based on a certain language situation, and it's not certain if the suggested method will work well in other low-resource and code-mixed language situations. Multilingual transformer models like mBERT and XLM-R usually work well, but their performance may change because of differences in language, code-mixing patterns, and cultural differences. To prove that this method can be used more widely, future study should apply it to a wider range of code-mixed languages that aren't well represented.
Also, even while our methods for making synthetic data—Machine Translation (MT), Masked Language Modeling (MLM), and Few-Shot Learning (FSL)—have made the models work better, it is still hard to make sure that the text they make is of high quality and sounds like real language. Currently, diverse synthetic sources are included in a balanced way. However, future research could look at adaptive selection algorithms that change the composition of synthetic data based on feedback from validation or error analysis. This could make categorization systems stronger and better at understanding their surroundings.
Another thing to think about is the possible moral problems that could come up with making synthetic text. We want to stop internet abuse, yet these kinds of technologies could be used to make stuff that is wrong or dangerous. So, future work should include ethical precautions like adversarial filtering, content watermarking, or model auditing tools to stop people from misusing it. Our work can help make hate speech detection algorithms that can work in a lot of different languages that are fair, accountable, and scalable by fixing these technological and moral problems. We understand that future extensions could incorporate more real-world workloads to make sure that this work is generalizable, even though we tested it on our manually gathered dataset and confirmed it through several synthetic augmentation methodologies.
Conclusion
This research examined the efficacy of synthetic data augmentation techniques—Machine Translation (MT), Masked Language Modeling (MLM), and Few-Shot Learning (FSL)—in improving hate speech detection inside Malayalam-English (Manglish) code-mixed social media material. We optimized several transformer models, such as mBERT, BERT, and IndicBERT, by using both real and fake data. Our results demonstrate that the classification performance has improved a lot. For example, mBERT got an F1-score of 86.42%, which is far better than the 81.24% it got with only real data. BERT (72.15% → 78.02%) and IndicBERT (65.13% → 78.52%) also saw similar improvements.
LIME's explainability research showed that contextual clues, not just single objectionable words, are very important for finding hate speech correctly. Additionally, comparing misclassifications between real and augmented datasets shows that synthetic data can make things fairer by lowering the number of false positives and false negatives, and it can also make things broader by showing models a wider range of code-mixed expressions. Even with these changes, there are still some problems. The method might not work for other code-mixed languages or fields, and ethical issues around the creation of synthetic data need to be handled with care. This study shows how integrating fairness-aware, explainable, and synthetic data-augmented models might be useful for multilingual social media platforms. It establishes a foundation for more dependable, low-bias, and contextually informed hate speech identification, paving the way for further investigations in adaptive and responsible AI for digital communication.
Author Agreement Statement
We the undersigned declare that this manuscript is original, has not been published before and is not currently being considered for publication elsewhere. We confirm that the manuscript has been read and approved by all named authors and that there are no other persons who satisfied the criteria for authorship but are not listed. We further confirm that the order of authors listed in the manuscript has been approved by all of us. We understand that the Corresponding Author is the sole contact for the Editorial process.
Footnotes
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
