LogCSS: Log anomaly detection based on BERT-CNN with context-semantics-statistics features

2.1 Unsupervised log anomaly detection

Unsupervised learning algorithms are widely used for feature extraction on log data, which helps to discover the common rules on large amounts of unlabeled data. The training data does not have labels and the target is to classify the data. In the early years, many researchers propose to use unsupervised machine learning algorithms for anomaly detection. Farzad et al. [5] are the first to use autoencoder [6] and isolation forest [7] to detect log anomalies. Lin et al. [8] cluster the system logs based on their similarity. Later, some unsupervised deep learning algorithms are proposed for anomaly detection. Du et al. [2] propose DeepLog, an unsupervised learning framework based on LSTM. They use normal logs for the offline training of the model. For online prediction, they check whether the coming log sequence is consistent with the features learned by the model. If not, they mark the log sequence as an anomaly. Recently, LogAnomaly [9] is proposed to improve DeepLog by considering the influence of synonyms and antonyms on templates. Log2Vec [28] also improves DeepLog. It uses graph embedding to capture the correlation features between logs, and classifies them with a clustering algorithm. A weakness of unsupervised learning algorithms is that they need a large number of labeled normal logs. Another problem is that they cluster the logs simply based on similarity, which fails to capture the time correlation of the logs. As a result, unsupervised log anomaly detection methods generally suffer from low accuracy.

2.2 Supervised log anomaly detection

In the early years, the traditional supervised methods usually use models like Logistic Regression [10], Decision Tree [11], or SVM [12]. These models are built with statistical probability analysis method14s. They lack the analysis of the semantics of the logs, so they have low accuracy. With the development of deep learning, many supervised log anomaly detection methods based on deep models are proposed. Lu et al. [13] propose an embedding technique called logkey2vec and use CNN to detect the anomalies in large-scale system logs. They extract the internal association between logs with different sizes of convolution kernels. Kimura et al. [4] and Zhang et al. [14] extract typical features from log sequences based on expert experience and use traditional machine learning models (such as Random Forest [15]) to learn the anomalous log patterns. Zhang et al. [16] propose LogRobust, which combines the attention mechanism with Bi-LSTM to extract the semantic information of logs for anomaly detection, and it solves the OOT problem by adding noise to the event template data to enhance the model’s generalization ability. However, once a new log appears that differs significantly from all existing log templates, it can lead to a decrease in model accuracy. Le et al. [17] propose NeuralLog, which uses Transformer [18] to detect log anomalies. Xia et al. [19] propose to use GAN and attention mechanism, which solves the problems of lacking anomalous samples and feedback delay in large-scale software systems. In order to solve the problem that labels of Log Anomaly Detection are precious and difficult to obtain, Zhang et al. [27] proposed a PU learning based method PUAD, which can achieve high accuracy by only requiring a few labels. Logclass [21] is a partial label-based framework for automatically and robustly identifying and classifying abnormal logs of networks and services. The mentioned methods can still be improved, as their models may not accurately extract the log semantic information, and they ignore some local relationships between logs. They also lack the solution to the OOT problem, which leads to low accuracy for online prediction. Wang et al. proposed Lightlog [41], which uses word2vec and post processing algorithm (PPA) to create a low dimensional semantic vector space, and performs exception detection based on TCN, which meets the real-time processing performance requirements of edge devices. However, it ignores the statistical characteristics of logs, so it cannot achieve optimal log exception classification performance. Literature [46] used the BERT-CNN model to analyze logs, and the results of the analysis were used to predict the probability of virtual machine operation failures on the server. The method lacks statistical feature extraction for log event templates. It also did not solve the OOT problem. Literature [47] propose LAnoBERT, a parser free system log anomaly detection method that uses the BERT model, exhibiting excellent natural language processing performance. However, it ignores the contextual features between logs, as well as the statistical characteristics of log events, and does not propose effective solutions to OOT problems.

To solve the mentioned problems, we propose a supervised deep-learning-based log anomaly detection framework, LogCSS. It contains the CSSCNN model for training and prediction. In CSSCNN, we first use BERT, a pre-trained model, to extract the semantic features of the logs. Then we use a CNN model to extract the local relational features between the logs, and combine them with the count matrix as statistic features. We also propose a solution to the online OOT problem.

There are three steps for data pre-processing: log cleaning, template extraction, and text segmentation.

Log cleaning aims at removing the characters and strings that do not affect the detection of anomalies. We first filter the strings longer than 20 characters and remove the pure digital information, including the time and IP addresses. Then, we use some templates to remove useless punctuations, such as ‘[]’, ‘()’, ‘{}’, ‘> ’, ‘––––’, etc. Afterward, we use identifiers ‘_’ and ‘–’ to find entity strings, such as ‘epn_volte_sip_zte’. Finally, we remove the duplicated text.

Template extraction aims at structuring the log data. Raw log messages are non-structured data containing timestamps and message contents. Each piece of log records a system behavior with the timestamp, the log level, and the content. It is easy to extract the timestamp and log level, but difficult to extract the content. The content consists of fixed strings and variable values. The fixed parts can be extracted as templates because they are always the same when the system encounters the corresponding code. The variable parts are dynamic information that may change with the physical devices and system runtime environments. We use the Drain algorithm [20] to extract the log templates. The main idea of Drain is to build a parsing tree with a fixed depth based on the log data. The tree contains the rules of template extraction. The output of Drain is the template of the input log, such as ‘worker process < *>exited with code < *> ’.

Text segmentation employs the WordPiece algorithm [22], which is widely used in the research of text encoding [3, 24]. WordPiece has a flexible definition of words. For example, ‘loved’, ‘loving’ and ‘loves’ have the same root. If we define such cases as different words, the vocabulary will be very large, which has a bad effect on the model training speed and accuracy. WordPiece can split the words into ‘lov’, ‘ed’, ‘ing’, and ‘es’, which helps reduce the size of the vocabulary. One problem is that, in our case, the log data usually contains combinations of uncommon words. To prevent WordPiece from generating unreasonable sub-words, we segment the words with additional processing, as shown in Table 1.

As is shown in Table 1, wordpiece may split words like “NameSystem” into ‘names’, ‘##yst’, ‘##em ’, which is obviously not the result we want. We would resplit “NameSystem” into ‘name’ and ‘system’.

LogCSS uses the pre-trained BERT model to extract the semantic features of the logs and encode them into vectors. It also encodes the count matrix as statistic features.To train a Deep Learning model for log anomaly detection, it is necessary to extract the features of the logs, which includes the following aspects:

Semantic features. Logs are text statements composed of words, so they contain semantic features. For example, in Fig. 2, the log statement “081109 204525512 info dfs.datanode$packetresponder: packetresponder 2 for block blk_5724928392872299681 terminating” ends up with the word “terminating”, which contains key semantic features.

OPEN IN VIEWER

Fig. 2

Schematic diagram of three important features of log information.

LogCSS employs a novel CSSCNN model. CSSCNN transfers the semantic features of the logs into a semantic matrix, and encodes it with a CNN model. After a pooling operation for the dimension reduction, the result is concatenated with the statistic features as the input of a fully-connected network. The detection result is outputted by a softmax layer. The detailed process is shown in Section 4.2.

The workflow of LogCSS is separated into an offline training process and an online prediction process. For offline training, we first pre-process the raw log data through log cleaning, template extraction, and text segmentation. Then, we encode the log data with a count matrix and the BERT model. Finally, we use a CNN model to build a classifier for anomaly detection.

For online prediction, we first pre-process the coming log data with a similar procedure. The only difference is that we use DOOT for template matching and updates. Then we use the trained encoder and classifier to predict whether the coming log is an anomaly or not.

In the following subsections, we describe the details of the count matrix, the CSSCNN model, and the DOOT technique.

4.1 Count matrix

The count matrix extracts and encodes the counting information of log events. After pre-processing, the raw logs are parsed into a sequence of log templates. Each template corresponds to a type of log event. We count the number of each type of event in a slice of log sequence and encode the information as the input of the classifier model.

We use sliding windows to generate the log sequence slices. We set the window size and the stride according to the dataset. For each slice, we generate a vector with the number of each type of log event. For example, if the vector is [0, 0, 2, 3, 0, 1, 0], then in the slice, log event 3 appears twice, event 4 appears three times, and event 6 appears once. We encode the vector through normalization. Suppose the vector is C = {C₁, C₂, . . . , C_m}, where C_i is the count of the i-th event in the slice, and n is the number of log templates. We encode C into the count vector V_C = {V_C1, V_C2, . . . , V_Cm} according to Formula (1). V c i = c i - min ( c ) max ( c ) - min ( c )(1)

The count vectors in a log sequence form the count matrix. We concatenate the count vector with the output of the BERT+CNN models, as the input of the classifier.

After pre-processing, each log line is parsed into a word sequence. CSSCNN takes several log lines within a time window as input. It first encodes each line with BERT, then encodes the log sequence with CNN. We concatenate the encoding result with the count vector and input it into a fully connected network as a classifier to predict whether the log sequence is an anomaly. BERT is pre-trained with large-scale unlabeled corpus. It can be used to obtain the representation of the text with rich semantic information, and fine-tuned in specific NLP tasks for the application. In CSSCNN, we use BERT to obtain the semantic representation of the log text, and fine-tune the model during the training process. The output of the BERT model is the log sentence vectors. In recent years, CNN is also used for text classification tasks. In CSSCNN, we use kernels of three sizes to extract the important information between the log sentence vectors from BERT, which extracts the local relational features and improves the accuracy of CSSCNN.

As an example, we take a time window with ws (default ws = 50) log lines LogWindow = {Log₁, Log₂,  . . .  . . .  , Log_WS}. Each log line is segmented into words Log _i  = {Subword₁, Subword ₂,  . . .  . . .  , Subword_mi}, where m_i is the number of words in the i-th log line. To align the sequence lengths for subsequent vector calculations, we implement the following steps: if the number of words in a log is smaller than L, we pad it with 0 to L. If it is larger than L, we remove the words that exceed L. This ensures that the length of each log is L. We consider each log line as a sentence and input it into BERT. The last layer of the BERT encoder generates an embedding vector for each word. After average pooling, we obtain the semantic vector of the i-th log line S_i∈R^es, where es is the dimension of the embedding vector from the BERT encoder. Thus, we obtain the semantic matrix V_Semantic∈R^ws ×es of the ws logs in the time window. As shown in Formula (2): V Semantic ∈ R bs × ws × es ← BertPreTrainModel ( W )(2)

To further extract information from multiple log lines, we input the semantic matrix V_Semantic into a CNN model. The structure and hyper-parameters of the CNN model are shown in Table 2. Similar to TEXT-CNN, we set the size of a convolution kernel as h×es, where es is the same as the dimension of a semantic vector, and h is the number of adjacent log lines that involve in a convolution. v ( i , j ) | ∀ v ( i , j ) ∈ V CNNOutput k ∈ ( 1 , 2 , 3 ) ∈ R hs × ( ws - ( k + 1 ) ) = ( X * W ) ( i , j ) = ∑ m ∑ n x ( i + m , j + n ) w ( m , n )(3)

After computing the convolution and non-linear activation, we obtain (ws-h+1) features, where the i-th feature corresponds to the log lines from the i-th to the (i + h-1)-th. Then we use max-pooling to obtain the maximum feature value, which represents the result of the convolution kernel. In our experiments, we use 3 sizes of convolution kernels, i.e., h = 3, 4, 5. For each size, we have hs = 128 kernels. So, the output of CNN is a vector of (3×hs) dimensions, which contains the semantic and temporal information of the logs in the time window. The output after convolution through each convolution kernel is shown in the following formula 4: V CNNOutput k ∈ ( 1 , 2 , 3 ) ← Pooling ( V CNNOutput k ∈ ( 1 , 2 , 3 ) ) ∈ R hs × 1(4)

Then we concat the three convolution results, as shown in Formula 5 below: V CNNOutput = V CNNOutput k = 1 + V CNNOutput k = 2 + V CNNOutput k = 3(5)

We concatenate the output of CNN with the count vector, As shown in Formula 6 below: V FCInput = V CountMatrix + V CNNOutput(6)

Then we use V_FCInput as the input to the fully connected neural network. The structure of the network is also shown in Table 2, where M is the dimension of the count vector. As shown in Formula (7), we use PReLU as the activation function. σ ( x ) = { x if ( x ) > 0 a i x if ( x ) ⩽ 0(7)

It is a modified version of Leaky ReLU, where a_i∈[0,1) is a learnable parameter determined by the data. It shows good performance on the task of ImageNet classification [25] in computer vision. Finally, we use the softmax function to map the output of the fully connected neural network to the abnormal result space, as shown in the following formula. Anomaly Result = { 1 & ( S ( v i ∈ V Temp ) = e v i ∑ i e v i ) ⩾ 0.5 0 & ( S ( v i ∈ V Temp ) = e v i ∑ i e v i ) < 0.5(8)

We also use dropout to avoid overfitting. Dropout and cross entropy are commonly used regularization methods and loss functions in neural networks. They can work together to improve model performance. By using dropout, the risk of overfitting can be reduced, thereby improving the robustness of the model. Cross entropy can measure the difference between the predicted results of a model and the actual results, thereby improving the accuracy of the model. We used the cross entropy loss function to measure the performance of the model. The loss function is shown in Formula (9). L = - ∑ i = 1 N y ( i ) log ( y ˆ ( i ) ) + ( 1 - y ( i ) ) log ( 1 - y ˆ ( i ) )(9)

In Formula (3), y⁽ⁱ⁾ is the label of the i-th sample in the batch. 0 as normal and 1 as anomalous. y ˆ ( i ) is the estimated probability that the label of the i-th sample is normal.

Table 2 shows the structure and the hyper-parameters of the model. We have hs (hidden size) as the number of the output channels of the CNN kernels, bs (batch size) as the size of a batch for training, ws (window size) as the number of logs in a time window, es (embedding size) as the dimension of the output vector of BERT, L as the number of words in the longest log template, and M as the length of the count matrix. For details of Pseudocode implementation, please refer to algorithm 1, as follows.

From Algorithm 1, it can be seen that our overall processing flow. First, we collect log data from the distributed microservice system, and remove redundant information through data cleaning. Then we select the appropriate log templates through DOOT technology, generate a Count Matrix C based on the statistical information, and encode a V_c matrix. We also generate the word list W from the log templates through the Log Word Segmentation technology, where W = {W₁,W₂, . . .  ,W_L} and L is the number of the words. Then we employ the BERT model to encode the word list into a log semantic vector matrix R^ws×L×es, where ws is the window size referring to the number of logs in an observation time window, and es is the embedding size of the word generated by BERT. The vector matrix R^ws×L×es is further transferred into a feature vector R^ws×es i through Feature Pooling to fit the input of the CNN. Then the feature vector R^ws×es is convolved and pooled by three convolution kernels (hs is the number of each type of convolution kernel in Fig. 3) to generate three one-dimensional vectors R^(hs×1). The three vectors are concat with the statistical matrix V_c to generate a vector V. Finally, the vector V is input into the fully connected neural network and mapped into the result output to the user through sofmax.

OPEN IN VIEWER

Fig. 3

DOOT Log Template Extraction Procedure.

DOOT is designed for log template matching and dynamic updates during online prediction. It continuously updates the log template pool, which increases the precision of log template matching and improves the quality of semantic feature extraction.

We note that OOT (out of template) is almost an inevitable problem for online template matching, where the similarity between the new log and any log template, is larger than the threshold. To solve this problem, DOOT uses Levenshtein Distance [26] for fuzzy matching, which measures the similarity between the strings with their edit distance. DOOT continuously updates the template pool and improves the quality of template extraction.

As shown in Fig. 3, In the offline training process, we extract log templates from training log data with the Drain algorithm, and store them as a template pool. In the online prediction process, we first match the pre-processed log with existing templates. If the similarity is smaller than the threshold, we directly output the matched template. Otherwise, we compute the Levenshtein Distance between the new log and all the existing templates,then output the template with the maximum similarity. We also add the template of the current log into the template pool, and periodically restart the offline training process. DOOT helps increase the rate of successful template matching and improves the accuracy of the anomaly detection model.

5.1 Experiment setup

We conducted all our experiments on a Linux server with four Tesla M40 24GB GPUs, Intel Xeon CPU E5-2680 v4 @ 3.3GHz, and 256GB memory. The operating system is Ubuntu16.04, and the version of the Nvidia driver is 418.126.02, with CUDA 10.1. We implemented LogCSS with Python 3.8.12, torch 1.8.2, numpy 1.21.2, pandas 1.3.4, and transformers 4.12.5.

The best configuration parameters of our LogCSS on the HDFS dataset are as follows: using the BERT-Mini pre-trained model, learning rate as 1e-4, batch size as 128, window size as 50, stride as 1, the ratio of positive and negative samples as 1 : 1. The settings of LogCSS on the BGL dataset are: using BERT-Tiny pre-trained model, learning rate as 1e-4, batch size as 128, window size as 20, stride as 5, the ratio of positive and negative samples as 0.1 : 1.1.

5.2 Evaluation metrics

To measure the effectiveness of LogCSS for anomaly detection, we use precision, recall, and F1-Score as metrics, which are computed as follows.

1) Precision: the ratio of correct predictions in positive predictions. Precision = TP TP + FP

2) Recall: the ratio of correct predictions in anomaly samples. Recall = TP TP + FN

3) F1 Score: the harmonic mean of precision and recall. F 1 - Score = 2 × Precision × Recall Precision + Recall

TP, FP, and FN represent the number of true-positive, false-positive, and false-negative cases.

5.3 Datasets

We evaluate our method with other log anomaly detection methods on the open-source datasets HDFS and BGL. The information of the two datasets is shown in Table 3.

HDFS 1 > is a dataset generated by running Amazon EC2 for over 200 days. It is published by Xu et al. [27]. The size of raw log files is 1.55 GB, containing 11,197,954 log lines. The status of each HDFS block is also recorded, and there are 29 unique log keys. The log data is separated into session windows, and each log line is marked with a block ID and its log key. The original training set contains 4,855 normal sessions and 1,638 anomalous sessions. The original test set contains 553,366 normal sessions and 15,200 anomalous sessions.

BGL (Blue Gene/L) 2 is generated by a supercomputer with 128K processes in LLNL. There are 4,747,963 log lines in the dataset, 348,460 of which are labeled as anomalous. An important property of this dataset is that many log keys only appear in a specific period, so it is impossible to have all the normal log keys and normal operation patterns in the training set.

TSLOG (Test environment system log) was generated in our test environment, which ran a large e-commerce shopping transaction business. We intercepted about two months of data, with a total of 131563 rows of logs, of which 2978 rows were marked as exception logs, with a total size of 40.88 MB.

In the following experiments, we use the 80% data in the front as training data and the rest 20% as test data for each dataset.

LogCSS adds the count matrix of the log templates to the features, and uses DOOT technique to solve the OOT problem. We evaluate the effectiveness of the count matrix and DOOT on the HDFS dataset. The experiment results are shown in Table 4.

There are 29 recommended event templates in HDFS. We number the 29 templates as is shown in Appendix 1. In the first experiment in Table 4, whenever an event template with serial numbers 1–14 is encountered when training the model, the corresponding log event is replaced by the OOV event template when extracting semantic features. The traditional method does not add count matrix features for training, and the final trained model is used to verify the performance of the algorithm on the test set. The experimental results are shown in Table 4 above. The model trained in this way had greatly reduced the precision rate, recall rate and F1 value. We take similar operations on templates from numbers 15 to 29, and the metrics also become lower. The model trained by our method adopts DOOT technology. When we encounter a new template, we use the similarity calculation to find the template with the closest similarity from the template pool to replace it, and continuously update the template pool. When training the model we add the count matrix to the model for training. The experimental results show that in the case of the above two new templates, the accuracy of our algorithm is not greatly affected, and it still has a very good classification effect. This tables shows that the default OOV template for critical logs harms the accuracy of anomaly detection. The DOOT technique can update the template pool by adding new log event templates, which ensures the accuracy of the anomaly detection algorithm.

We evaluate the configurations of LogCSS with the following experiments, including whether to fine-tune BERT, the log window size, and the method of sentence vector generation.

First, we conduct some experiments to evaluate the effectiveness of fine-tuning BERT in LogCSS. The results in Table 5 show that BERT with fine-tune has better performance. Note that fine-tuning BERT increases the computation cost of training. If the machine has enough resources and processing speed, we suggest to fine-tune the BERT model.

Second, we evaluate the influence of the window size on the detection performance. The results are shown in Table 6. The best window size on HDFS is 50. A possible reason is that the HDFS data can be grouped into sessions, and most sessions contain no more than 50 log lines. By setting the window size as 50, the machine learning model can use the session window as input without truncation, which is helpful to the accuracy. The best window size on BGL is 20. The logs in BGL are not grouped into sessions, so it is not easy to find the optimal window size through the dataset.

Third, we compare different methods of sentence vector generation. The sentence vector is the input of the CNN model, which represents the semantic information of a single log line. It can be generated through pooling or the CLS token. The pooling method takes the average-pooling or max-pooling of the output of the BERT encoder, i.e., the embedding vectors of the words in the log line. The CLS token is generated from the hidden state of the last layer of BERT, with a linear transformation and a Tanh activation. According to the results in Table 7, CLS performs better than the pooling methods.

Finally, we compare the training time of LogCSS with LR, SVM, and Lightlog. The results are shown in Table 8. LR and SVM use statistical methods to vectorize the data, without extracting the semantic features of the data, so the training time is relatively short. Lightlog uses word2vec algorithm to represent the data semantics in a fixed vector, and uses PPA algorithm to map the problem to a low dimensional semantic vector space, so it also has short training time. Our LogCSS uses the pre-trained model, BERT, for vector representation, which accounts for most of the long training time. For example, the training time on the BGL dataset is (15633 + 7079) seconds, where BERT takes 15633 seconds for forward propagation and CNN takes 7079 seconds for classification. Although Lightlog takes relatively short training time, it has two drawbacks: 1) It uses the word2vec algorithm to represent each word in a fixed manner, which cannot extract word context information. 2) OOT (out of template) is almost an inevitable issue in online template matching, and Lightlog does not have an effective solution. The LogCSS method proposed by us uses BERT to extract Semantic information of logs, which is more effective, and the DOOT method proposed by us can solve the OOT problem.

We evaluate the precision, recall, and F1 score of LogCSS on the HDFS and BGL datasets, and compare them with 6 baselines: Logistic Regression (LR) [10], Support Vector Machine (SVM) [11], CNN [4], LogRobust [16], Log2Vec [28], Lightlog [41] and NeuralLog [17]. We first analyzed and investigated the current best algorithms in principle, and then conducted experimental comparisons on the same data set and the same experimental environment.

5.6.2 Comparison experimental results with baselines

We compare the performance results of our algorithm with the current state-of-the-art log anomaly detection algorithms under the same experimental environment and conditions. The specific experimental results are shown in Table 10 and Fig. 4 below.

OPEN IN VIEWER

Fig. 4

Comparison of Effectiveness between LogCSS and Baselines on HDFS and BGL.

OPEN IN VIEWER

Table 10

Effectiveness Evaluation on HDFS and BGL

Methods	HDFS Dataset			BGL Dataset
	Precision	Recall	F1 score	Precision	Recall	F1 score
LR	93.5	89.2	91.3	98.2	93.1	95.5
SVM	90.3	78.5	83.9	99.3	91.1	95.1
CNN	97.7	99.3	98.5	98.2	97.7	97.9
LogRobust	98.1	100	99.1	62.1	96.2	75.7
Log2Vec	94.1	94.1	94.1	80.1	98.4	88.3
Lightlog	94.5	99.7	97.1	96.3	98.9	97.5
NeuralLog	96.2	100	98.1	98.1	98.2	98.1
Ours	98.9	99.4	99.2	100	97.7	98.8

Traditional supervised machine learning algorithms LR and SVM take the count vectors of log templates as input, which is extracted through the Drain algorithm. The experiment data of the other baselines comes from Loghub [29], and they all use slide windows for sampling.

As shown in Table 10 and Fig. 4, our method LogCSS has the best F1-Score on both HDFS and BGL. Traditional machine learning algorithms LR and SVM ignore the semantic information of the logs, so they do not perform well. Figure 4(a) shows that SVM has the lowest recall, and LR also has low precision, recall, and F1-Score. CNN extracts the local relational features between the adjacent logs, but still lacks the extraction of the semantic and statistic features, so it performs worse than LogCSS on HDFS. LogRobust only extracts the semantic information and ignores the local relational information, which leads to the worst performance in Fig. 4(b). Log2Vec, as an unsupervised algorithm that improves DeepLog, requires massive normal data and fails to extract the local correlations within the log series. So, it shows bad performance in Fig. 4(b). Lightlog uses word2vec and post processing algorithm (PPA) to create a low dimensional semantic vector space, and performs exception detection based on TCN, which meets the requirements for real-time processing performance of edge devices. However, it ignores the statistical characteristics of logs and does not use Bert pre training model for semantic feature extraction, so its performance is lower than our proposed LogCSS method in F1-Score. NeuralLog uses the Transformer model to detect log anomalies. It performs worse than CNN in extracting the local relational features between the logs. It also ignores the statistic features, so it has lower F1-Scores than LogCSS on both datasets.

As shown in Fig. 4 (a), although our method LogCSS has a lower Recall rate than RerualLog and LogRobust on the HDFS dataset, LogCSS performs best in terms of accuracy and F1 Score, with an Precision improvement of 0.7% compared to LogRobust and 0.1% compared to NeruaLog on F1 Score; As shown in Fig. 4 (b), although our method LogCSS has a lower Recall rate than Log2Vec, LightLog, and NerualLog on the BGL dataset, it performs the best in Precision, with an Precision improvement of over 1.9% compared to Log2Vec, LightLog, and NerualLog, and an F1 Score improvement of over 0.7% compared to Log2Vec, LightLog, and NerualLog; As shown in Fig. 4 (c), our method has a lower Recall rate than LightLog on the TSLog dataset, but an Precision improvement of 4.9% and an F1-score improvement of 3.1% compared to LightLog.

In conclusion, the experiments show that our LogCSS performs better than all the baselines on HDFS and BGL datasets.