ACAM-AD: Autocorrelation and attention mechanism-based anomaly detection in multivariate time series

Abstract

Multivariate time series anomaly detection has made significant progress and has been studied in many fields. One of the difficulties in time-series data analysis is the complex nonlinear dependencies between multiple time steps and multiple variables. Therefore, detecting anomalies in these data is challenging. Although many studies used classical attention mechanisms to model the temporal patterns of data, few have combined multiple attention mechanisms and analyzed the data’s temporal characteristics and feature correlations. Therefore, we propose an autocorrelation and attention mechanism-based anomaly detection (ACAM-AD) framework that combines an autocorrelation model based on the Autoformer model, which is superior to the self-attention mechanism, a multi-head graph attention network, and a dot-product attention mechanism to model the complex dependencies of data considering temporal and feature dimensions. The autoregressive model is parallelized with the neural network, and a sparse autocorrelation mechanism and sparse graph attention network are used to reduce model complexity. Experiments on public datasets show that the model is effective and performs better than the baseline model.

Keywords

Multivariate time series anomaly detection autocorrelation multi-head graph attention network

1 Introduction

Sensor devices are used increasingly in many fields in the Big Data era for monitoring and generating time-series data [19, 29]. Research on multivariate time-series data has been conducted on spacecraft [22], server machines [32], and water treatment systems [18], among others. The use of these data to identify anomalies [7 , 25] is critical because accurate anomaly detection can trigger early warning systems to enable timely troubleshooting to avoid revenue loss [10, 14].

However, detecting anomalies in time-series data is challenging due to complex patterns and data inconsistencies. Complex nonlinear dependencies exist between various variables and time steps. In addition, the dependencies may change dynamically over time, increasing the difficulty of analysis. Therefore, a major challenge for anomaly detection in multivariate time series data is capturing the dynamic dependencies between multiple time steps and variables. This problem has not been adequately addressed in many papers.

Due to the popularity of the transformer model [4], the self-attention mechanism has been increasingly utilized in various fields. Many improved models applicable to the temporal domain have been proposed, such as the Autoformer model [15]. This model is an autocorrelation mechanism that has higher efficiency for modeling the temporal correlation of data than the self-attention mechanism.

In order to model the data information comprehensively, we propose an autocorrelation and attention mechanism-based anomaly detection (ACAM-AD) framework to analyze the features and temporal perspectives of time-series data and improve anomaly detection accuracy. In this model, the data are input ACAM-AD into three parallel branches. The autocorrelation layer is used to model the temporal patterns of the data, the multi-head graph attention layer and gated recurrent unit (GRU) attention layer model the complex features and temporal patterns, and the autoregressive (AR) layer learns the overall trend of the data, predicts the values, and uses extreme value theory to select a threshold to detect anomalies. Each layer has a specific focus, and the framework comprehensively models the complex relationships of the data from multiple perspectives. To the best of our knowledge, this is the first time that an autocorrelation mechanism has been combined with a graph attention network (GAT) and dot-product attention mechanism to model the temporal and feature characteristics of data for time-series anomaly detection.

The main contributions of this paper are as follows:

The autocorrelation mechanism, GAT, and GRU-Attention mechanism are used to model the correlation between the temporal characteristics and features of time-series data.

The graph attention layer is preceded by a graph structure learning layer, which assigns K neighbors to each feature node to reduce the number of operations, unlike the traditional GAT that considers all features as neighbors.

After assessing the vector similarity of the data, only the top K sets of the results with the strongest correlation are retained and assigned weights in the autocorrelation module. This sparse processing reduces model complexity.

A statistical AR model based on the integrated learning theory is established parallel to the neural network to improve model robustness.

The remainder of this paper is organized as follows:

Section 2 describes related work in time-series anomaly detection. Section 3 explains the architecture of the ACAM-AD model. Section 4 presents the experimental setup and results and discusses the results. Section 5 is the conclusion.

2 Related work

Time-series anomaly detection is a complex task and has been extensively studied. Commonly used traditional methods are the 3-sigma criterion method and the local outlier factor (LOF) algorithm [2], which regard outliers in the data as anomalies. In addition to traditional methods, deep learning methods have received significant attention due to their improved ability to model complex relationships between time series. The most widely used methods are recurrent neural networks (RNNs) [20, 23] and their variants, long short-term memory (LSTM) networks [30], and GRUs [17, 21].

Unsupervised anomaly detection methods based on deep learning have developed rapidly in recent years. The LSTM-based variational autoencoder (LSTM-VAE) [8] was proposed in 2017. It combines the LSTM and VAE [9] to project time-series data with multiple variables into the hidden space and has shown excellent performance on multiple datasets. In 2018, the LSTM network with nonparametric dynamic thresholding (LSTM-NDT) [22] was developed for automatic anomaly detection to monitor telemetry data sent back from spacecraft. In the same year, the deep autoencoding Gaussian mixture model (DAGMM) [6] was proposed for anomaly detection in multivariate data without temporal correlation. In 2019, Su et al. [32] proposed a stochastic model for multivariate time-series anomaly detection (OmniAnomaly) to capture the normal patterns in multivariate time-series data using learning feature representations. In 2020, Zhao et al. [16] proposed a multivariate time-series anomaly detection via a graph attention network (MTAD-GAT) algorithm. It uses two juxtaposed GATs to process data separately to achieve higher accuracy. In the same year, a model for multivariate time-series forecasting with graph neural networks (MTGNN) was developed [33]. It learns the one-way relationships between variables using graph neural network (GNN) [24] modules to capture the dependencies in multivariate data better. In 2021, Deng and Hooi [1] proposed a graph deviation network (GDN) that uses a GNN to learn the relationship between variables and enhance the interpretability of the model. The literature review indicates that most recent approaches have focused on analyses from one perspective. Few models can analyze and model the information contained in the data by considering temporal characteristics and feature patterns.

Attention mechanisms have been used increasingly in the analysis of time-series data. Self-attention mechanisms have replaced recurrent models, such as LSTM, for time-series prediction due to their superior performance. The transformer model was proposed in 2017 for processing sequence data for natural language processing [4]. It uses a self-attention mechanism rather than RNNs. The dual self-attention network (DSANet) proposed in 2019 [31] has a two-branch architecture that combines a self-attention mechanism with convolutional neural networks for time-series prediction. It has achieved good results. The Autoformer model [15] was developed in 2022. It is an improvement of the transformer model and uses an autocorrelation mechanism that is more applicable to long time-series data. It performs better than the self-attention mechanism. It has improved transformer models for analyzing time-series data and currently provides optimal performance for time-series forecasting.

The excellent performance of the autocorrelation mechanism and parallel frameworks has inspired our work. In this study, we combine these methods with attention mechanisms that have performed well in time-series prediction and establish a three-branch framework (ACAM-AD) to model data dependencies by considering temporal and feature dimensions to detect anomalies. In addition, many methods have obtained good results at the expense of training speed; therefore, model efficiency is considered in this study. We use sparse attention mechanisms in our model to minimize the training time.

3 Proposed framework

We define the problem addressed in this study in Section 3.1 and outline the proposed framework in Section 3.2. In Section 3.3, we describe our approach to data preprocessing. We elaborate on the components of the prediction part of the model in Section 3.4 and describe the anomaly detection aspect in Section 3.5.

3.1 Problem statement

The goal of this study is anomaly detection in multivariate time-series data. We denote the multivariate time-series data as X ={ x₁, x₂, …, x_{T
_train} }, where x_t ∈ R^k, T_train is the maximum length of the timestamp, and k is the number of input variables. Next, we define X_t, the input time window with length T at a given moment t. The input data matrix is denoted as X_t = { x_t-T, x_t-T+1, …, x_t-1 } ∈ R^k×T, which is used to predict the anomaly at time t, i.e., ${\hat{x}}_{t} \in R^{k}$ , t > T. Finally, threshold selection is performed. The anomaly score is calculated at each moment based on the difference between the predicted ${\hat{x}}_{t}$ and the true value x_t. When the anomaly score exceeds the selected threshold, we consider this moment as an anomaly, i.e., y_t = 1, otherwise, y_t = 0.

3.2 Overview

The process of the ACAM-AD model is summarized in Algorithm 1, and the framework is shown in Fig. 1.

Fig. 1

Structure of the ACAM-AD framework for multivariate time-series anomaly detection.

Algorithm 1 The ACAM-AD training algorithm

Input: raw time-series dataset X : { X₁, ⋯ , X_{T
_train}}, where X_t ={ x_t-T, x_t-T+1, … , x_t-1 }, Number epochs N

Output: Trained algorithm and weights of algorithm.

X ←normalize (X)

Autocorrelation, graph learning, GAT, GRU-Attention, AR ← initialize weights

n ← 1

repeat

for t = 1 to T_train do

$x_{t}^{AC}$ ← Autocorrelation (X_t)

$\tilde{X}$ ← GAT (Graph learning (X_t))

$x_{t}^{GRU}$ ← GRU-Attention ( $\tilde{X}$ )

$x_{t}^{AR}$ ← AR (X_t)

${\hat{x}}_{t} \leftarrow$ FC ( $x_{t}^{AC}$ , $x_{t}^{GRU}$ )+ $x_{t}^{AR}$

L $\leftarrow ∥ X - \hat{X} ∥_{2}$

Autocorrelation, graph learning, GAT, GRU-Attention, AR ← updating weights using L

end for

n ← n + 1

until n = N

The steps are as follows:

The temporal pattern of the data is modeled using the autocorrelation mechanism, and the data are decomposed into seasonal and trend parts. These are predicted separately and combined.

We use the graph structure layer to generate K-nearest neighbors of each feature node and use the multi-head GAT and GRU-Attention for the analysis of the features and temporal characteristics, respectively.

The outputs of the autocorrelation and GAT layers are concatenated and combined with the linear AR model results after the fully connected layer to obtain the model predictions.

The predicted and true values of the model are processed using the peaks over threshold (POT) selection method to obtain the threshold. They are compared with the anomaly scores to determine the outliers.

3.3 Data preprocessing

We use maximum-minimum normalization to normalize the data because the variables have different data ranges. $x_{scale} = \frac{x - \min (X_{train})}{\max (X_{train}) - \min (X_{train})}$ (1) where max(X_train) and min(X_train) are the maximum and minimum values of the training set, respectively.

3.4 Model prediction stage

3.4.1 Auto-correlation layer

The autocorrelation layer is used to model the time-series data from a temporal perspective. This layer consists of the autocorrelation mechanism and sequence decomposition and has a similar progressive decomposition architecture as the Autoformer model. The model performs alternating decomposition and optimization during the prediction process. The trend is extracted gradually using a cumulative approach and is predicted separately from the seasonal term. The result is the sum of the trend and seasonal components.

Autocorrelation Mechanism

Unlike the self-attention mechanism, which aims to determine the similarity between two sequences, the autocorrelation mechanism calculates the correlation between the historical and present sequences. Its coefficient is defined as the similarity between the sequence {X_t,i} and its τ-step delay {X_t-τ,i} of the feature i: $R_{XX} (τ) = X_{t, i} X_{t - τ, i}$ (2)

Due to the sequential aspect of time-series data, we add a mask tensor to the autocorrelation mechanism with values of 1 and 0, representing masked or unmasked positions. The objective of the mask tensor is to prevent the use of future information. The mask structure ensures that the current output value depends only on the current and past data, i.e., the output ${\hat{x}}_{T}$ at time T in the graph is only related to x₁, ⋯ , x_T, preventing future data leakage during predictions.

The autocorrelation coefficients are calculated using fast Fourier transform (FFT) [26] based on the Wiener-Khinchin theorem to improve the computational efficiency and reduce the training time. The data are subsequently inverse-transformed. It has been shown [15] that this method reduces the time complexity from O (L²) to O (LlogL). It is calculated as follows: $S_{XX} (f) = F (X_{t, i}) F^{*} (X_{t - τ, i})$ (3) $R_{XX} (τ) = F^{- 1} (S_{xx} (f))$ (4) where F^* denotes the conjugate complex after the Fourier transform, and F^-1 denotes the inverse Fourier transform.

We perform sparsification of the autocorrelation mechanism because not all two sequences may have the influence and to prevent the effect of irrelevant sequences and reduce the memory requirements. Only the similarity values of the first K₁ sequences with the largest correlation sparsity R_XX are selected, i.e., the attention weights R corresponding to these K₁ sequences are retained, and the remaining attention values are set to 0. $τ_{1}, \dots, τ_{K_{1}} = \underset{τ \in {1, \dots, T}}{arg Top K_{1}} (R_{XX} (τ))$ (5)

$\begin{matrix} R (τ_{1}), \dots, R (τ_{K_{1}}) \\ = softMax (R_{XX} (τ_{1}), \dots, R_{XX} (τ_{K_{1}})) \end{matrix}$ (6)

$Auto - Correlation (X_{t}) = \sum_{K_{1}} X_{t - τ_{j}} R (τ_{j})$ (7) where K₁ = c · LogL, c is the hyperparameter, and X_{t-τ_j,i} denotes the operation to obtain the sequence X with time delay τ.

Sequence decomposition

Since many data have complex temporal patterns, we use the Autoformer model [15] and perform decomposition [28] to divide the series into a more regular seasonal part and a smoother trend part. These two parts reflect the short-term volatility and long-term trend of the time series, respectively. The schematic diagram is shown in Fig. 2.

Fig. 2

Schematic diagram of sequence decomposition.

We use the moving average method for smoothing and perform a padding operation to keep the series length constant. The calculation is as follows.

$X_{tr} = AvgPool (Padding (X))$ (8)

$X_{se} = X - X_{tr}$ (9) where X_se and X_tr denote the seasonal and trend components, respectively, and AvgPool denotes the moving average of the vectors.

We use X_se, X_tr = seriesDecomp (X) to summarize the sequence decomposition process, which is the second internal block of the model in the autocorrelation layer.

Forward process

Incorporating the above modules, such as the autocorrelation mechanism, increases the complexity of the network structure, which may reduce the stability and cause gradient disappearance. Therefore, we incorporate residual connections, which alleviate gradient disappearance and substantially improve model performance.

The flow in the autocorrelation layer is defined as follows:

$\begin{matrix} X_{se}^{1}, X_{tr}^{1} = seriesDecomp \\ (Auto - Correlation (X_{t}) + X_{t}) \end{matrix}$ (10)

$\begin{matrix} X_{se}^{2}, X_{tr}^{2} = seriesDecomp \\ (Feed Forward (X_{se}^{1}) + X_{se}^{1}) \end{matrix}$ (11) where $X_{se}^{i} and X_{tr}^{i}, i \in {1, 2}$ denote the seasonal and trend components, respectively, of the input data after the i-th decomposition. $X_{se}^{2}$ is the final predicted value of the seasonal term.

A cumulative approach is used to extract the trend information step by step as follows:

$X_{tr} = W_{1} \cdot X_{tr}^{1} + W_{2} \cdot X_{tr}^{2}$ (12) where W_i, i∈ { 1, 2 } denotes the weight of the i-th trend term, and X_tr is the final predicted value of the trend term.

The value predicted by the Auto-Correlation layer contains two components: the cumulative result of the trend component and the weighted sum of the seasonal component:

$X^{AC} = W_{se} \cdot X_{se}^{2} + W_{tr} \cdot X_{tr}$ (13) where W_se and W_tr denote the weights of the seasonal and trend components, respectively

3.4.2 Sparse multi-head GAT layer

We treat each feature as a node and use the sparse multi-head GAT layer to capture the similarity between different nodes. First, we use graph structure learning to calculate the most relevant K₂ vector sequences of each sequence, i.e., the K₂ neighbor nodes. Then we use the multi-head GAT mechanism to integrate the information of the K₂ neighbors.

Graph structure learning

We use an embedding vector z for each feature [1]. The embedding vectors are randomly initialized, input into the model, and trained together to update z. They are used to learn the graph structure to determine which features are related. The similarity between the embedding vectors represents the similarity between features.

Our original data have k features. These data are regarded as a graph representing the relationship between features, where the nodes represent features and the edges represent the dependencies between features. An edge from one node to another represents the influence of the first feature on the second feature.

Because the influence between features is generally asymmetric, an undirected graph is not sufficiently accurate. Thus, we use directed graphs. We use the adjacency matrix A to represent this directed graph, where A_ij denotes the presence or absence of directed edges from node i to node j. Each feature i is given a set of candidate relationships with other features C_i, i.e., features that may be related; i has values from 1 to k. In the absence of a priori information, the initial candidate relationship C_i for feature i is all features.

We want to learn the graph structure with K₂ neighbors because the GAT network defaults to an interaction between all neighbors, which may introduce interference information. We first compute e_ij, i.e., the normalized dot product between the embedding vector z_i of feature i and the embedding vector z_j of the candidate correlated feature j. Then, we choose the top K₂ largest normalized dot product values as the relevant features of i. If the values contain j, A_ij is 1, i.e., j is a neighbor of i. The formula is expressed as follows.

$e_{ji} = \frac{{z_{i}}^{T} z_{j}}{∥ z_{i} ∥ \cdot ∥ z_{j} ∥}, j \in C_{i}$ (14)

$A_{ij} = 1 {j \in Top K_{2} ({e_{ij} : j \in C_{i}})}$ (15)

The value of K₂ can be chosen according to the desired sparsity level. The GAT module utilizes this newly learned adjacency matrix A.

Multi-head graph attention network

The GAT mechanism [27] is a GNN model. It performs better for capturing and modeling the correlations between different types of graph nodes than traditional attention mechanisms.

The i-th feature, whose corresponding graph has K₂ nodes, is denoted as {v₁, ⋯ , v_{K
₂} }, representing K₂ features. The input vector of the i-th node is v_i, and the output vector ${\tilde{x}}_{i}$ of its single-headed version is calculated as

${\tilde{x}}_{i} = σ (\sum_{j = 1}^{K_{2}} α_{ij} v_{j})$ (16) where ${\tilde{x}}_{i}$ has the same dimension as v_i and is used to replace the original vector v_i as the output of the GAT layer. σ represents the sigmoid activation function, α_ij is the attention weight representing node j for node i, and K₂ is the total number of neighboring nodes selected by the graph structure layer of node i. The attention weight α_ij can be calculated as follows:

$e_{ij} = LeakyReLU (w^{T} \cdot (v_{i} \oplus v_{j}))$ (17)

$α_{ij} = \frac{\exp (e_{ij})}{\sum_{j = 1}^{K_{2}} \exp (e_{ij})}$ (18)

Here ⊕ denotes the combination of two vectors, LeakyReLU is a nonlinear activation function [5], and w is a learnable weight parameter.

A multi-head GAT layer is used to improve its generalization ability; its structure with 6 nodes and 3 heads is schematically shown in Fig. 3.

Fig. 3

Example of a feature-based multi-head graph attention layer.

Fig. 4

Schematic of the GRU-Attention layer.

Multiple sets of GAT models with different weights are averaged to obtain the multi-head GAT, and multiple v_j are combined to obtain the final ${\tilde{x}}_{i}$ , which is calculated as follows.

${\tilde{x}}_{i} = ∥_{k = 1}^{M} σ (\sum_{j \in N_{i}} α_{ij}^{k} W^{k} v_{j})$ (19) where $α_{ij}^{k}$ is the attention weight, W^k is the weight of each head, N_i denotes the relevant features of feature I, and M is the number of heads.

3.4.3 GRU-Attention layer

We input the output of the GAT layer into the GRU-Attention layer. This layer consists of a GRU, followed by an attention mechanism layer with an inner product kernel based on the temporal dimension. The GRU is an improved version of the LSTM that reduces the internal complexity of the gate, simplifies the model structure, and better captures long-term temporal information. It also alleviates gradient disappearance, a problem of the LSTM model.

A temporal attention mechanism is added after the GRU layer, and the inner product kernel function is used to measure the similarity of the two vectors h_t with h_t-τ is as follows:

$e_{t - τ} (h_{t}, h_{t - τ}) = {h_{t}}^{T} h_{t - τ}$ (20) where ${\tilde{x}}_{t}$ and h_t are the input and output, respectively, of the GRU at time t. The τ is 1, …, T, and the similarity e_t-T, …, e_t-1 is input into the softmax layer to obtain the attention weights α_t-T, …, α_t-1. Specifically, the attention weight α_t-τ is used to denote the similarity of the hidden layer state h_t at the ultimate moment t to the h_t-τ at the moment t - τ, representing the importance of the implied layer state at each timestamp to the output state h_T at the last moment of the implied layer. The GRU-Attention layer calculates the output vector as the weighted sum of all hidden layer states {h_t-T, h_t-τ, ⋯ , h_t-1 }.

$x_{t}^{GRU} = \sum_{τ = 1}^{T} α_{t - τ} h_{t - τ}$ (21)

3.4.4 Autoregressive layer

Due to the nonlinear nature of the attention mechanism component and the recurrent component, the neural network model may be complex and unstable. To reduce model complexity and enhance model robustness, we use the method in Ref. [12] and decompose the final prediction result of the model into the sum of the linear component, which is based on the overall trend, and the neural network component, which is based on specific fluctuations. We use the AR model as the linear component; it is computed as

$x_{t, i}^{AR} = \sum_{k = 0}^{q^{AR} - 1} W_{k}^{AR} x_{t - k, i} + b^{AR}$ (22)

The output of the AR module is $x_{t}^{AR} \in R^{k}$ , which is the combination of the $x_{t, i}^{AR}$ for each feature, and $W_{k}^{AR}$ is the weight matrix of the AR model. We implement this model using a dense layer network to combine the weights and data.

3.4.5 Prediction results

The outputs of the GRU-Attention layer $x_{t}^{GRU}$ and the autocorrelation layer $x_{t}^{AC}$ are concatenated and input into a fully connected layer. We then combine the results with the prediction results of the parallel AR module to obtain the final prediction value ${\hat{x}}_{t}$ .

${\hat{x}}_{t} = FC (x_{t}^{AC}, x_{t}^{GRU}) + x_{t}^{AR}$ (23)

The loss function of the model uses the root mean square error between the predicted value ${\hat{x}}_{t}$ and the true value x_t. The Loss is calculated as follows.

$Loss = \sqrt{\sum_{t = 1}^{T_{train}} {({\hat{x}}_{t} - x_{t})}^{2}}$ (24)

3.5 Model detection phase

The detection phase of the model is described in Algorithm 2.

Algorithm 2 The ACAM-AD detection algorithm

Input: time-series test dataset $\hat{X} : {{\hat{X}}_{1}, \dots,$ ${\hat{X}}_{T_{test}}}$ , where ${\hat{X}}_{t} = {{\hat{x}}_{t - T}, {\hat{x}}_{t - T + 1}, \dots, {\hat{x}}_{t - 1}}$

Output: Labels y : [y₁, ⋯ , y_{T
_test}]

for t = 1 to T_test do

$x_{t}^{AC}$ ← Auto-Correlation ( ${\hat{X}}_{t}$ )

$\tilde{X}$ ← GAT (Graph-learning ( ${\hat{X}}_{t}$ ))

$x_{t}^{GRU}$ ← GRU-Attention ( $\tilde{X}$ )

$x_{t}^{AR}$ ← AR ( ${\hat{X}}_{t}$ )

${\hat{x}}_{t} \leftarrow$ FC ( $x_{t}^{AC}$ , $x_{t}^{GRU}$ )+ $x_{t}^{AR}$

s_t,i ← ${({\hat{x}}_{t, i} - x_{t, i})}^{2}$

$s_{t} \leftarrow max_{i = 1, \dots, k} (s_{t, i})$

if s_t>threshold then

y_t← 1

else

y_t← 0

end if

end for

The predicted value of each feature at time t computed by the model is denoted as ${{\hat{x}}_{t, i}, i = 1, 2, \dots, k}$ , and the anomaly score is defined as:

$s_{t, i} = {({\hat{x}}_{t, i} - x_{t, i})}^{2}, i = 1, \dots, k$ (25)

$s_{t} = max_{i = 1, \dots, k} (s_{t, i})$ (26)

When one of the features has an anomaly, that moment is regarded as an anomalous moment. We use the highest feature outlier score s_t,i of that moment as the anomaly score s_t. Finally, the obtained outlier fraction of the observed points at each moment is denoted as {s₁, s₂, ⋯ , s_{T
_test} }, and T_test is the number of the timestamps in the testing set.

The POT method [32] has been used increasingly for threshold selection in temporal data. It is based on the extreme value theory and maximum likelihood estimation. Unlike the fixed threshold method, the POT method automatically selects the threshold based on certain parameters, which is more effective for subsequent anomaly detection. The root mean square error between the predicted and true values at each timestep in the testing set is denoted as {l₁, l₂, ⋯ , l_{T
_test} } and is used in the POT method to select the threshold automatically.

The calculated anomaly score for each timestamp of the testing set is compared with the threshold. If the anomaly score s_t corresponding to the t-th moment is greater than the threshold th, it is an anomaly, i.e., y_t = 1.

4 Experiments

Sections 4.1 and 4.2 of this section describe the data sets, metrics, and parameters used in the experiments. Section 4.3 compares the proposed model with other state-of-the-art methods to evaluate its performance and effectiveness. Section 4.4 investigates the properties of the model’s modules using an ablation study.

4.1 Datasets

We used five public datasets: a Mars Science Laboratory (MSL) rover dataset, a dataset obtained from the Soil Moisture Active Passive (SMAP) instrument, a server machine dataset (SMD), a water allocation dataset (WADI), and a safe water treatment (SWaT) dataset.

The anomalous values in the dataset obtained from the MSL and the SMAP (https://github.com/ khundman/telemanom) rover have been flagged by NASA experts [22]. The datasets have 25 and 55 indicator features, respectively.

The SMD (https://github.com/NetManAIOps/ OmniAnomaly/tree/master/ServerMachineDataset) is a 5-week server dataset collected by a large Internet company [32]. It contains data from 28 server machines with 33 features. The SMD is divided into two subsets of equal size: the first half is the training set, and the second half is the testing set.

The SWaT dataset (https://itrust.sutd.edu.sg/ testbeds/secure-water-treatment-swat/) consists of data obtained from industrial water treatment systems producing filtered water [13]. It was collected during 11 days of continuous operation: 7 days under normal operation and 4 days under attack scenarios [33].

The WADI dataset (https://itrust.sutd.edu.sg/ testbeds/water-distribution-wadi/) was obtained from the WADI testbed, an extension of the SWaT testbed [3]. It consists of data collected during 16 days of continuous operation: 14 days under normal operation and 2 days under attack scenarios.

The statistics of these five datasets are listed in Table 1.

Table 1
Dataset information

Dataset Training data Test data Dimensions Anomalies (%)

MSL 58317 73729 55 10.72

SMAP 135183 427617 25 13.13

SMD 708405 708420 38 4.16

SWaT 496800 449919 51 11.98

WADI 1048571 172801 123 5.99

Dataset	Training data	Test data	Dimensions	Anomalies (%)
MSL	58317	73729	55	10.72
SMAP	135183	427617	25	13.13
SMD	708405	708420	38	4.16
SWaT	496800	449919	51	11.98
WADI	1048571	172801	123	5.99

4.2 Metrics and parameters

The precision (P), recall (R), and F1 values are used to evaluate the performances of the algorithms. They are calculated as follows.

$P = \frac{TP}{TP + FP}, R = \frac{TP}{TP + FN}, F 1 = 2 \cdot \frac{P \cdot R}{P + R}$ (27) where TP, FP, and FN denote the number of true cases, false positive cases, and false negative cases, respectively. The larger the value of the above three indicators, the more desirable the effect of the model.

We use the point adjustment method [32] to measure model effectiveness, similar to other anomaly detection methods. If the algorithm can detect a point in the anomaly region, we assume that all points in the region are correctly detected. The assumption holds even if only a few points are detected in an anomaly region. We improved the point adjustment method by using a ratio of 50%, i.e., it is assumed that all points in an anomalous region are correctly detected if 50% of the anomalous points are detected.

Parameter selection is a critical step in deep learning. We set the model parameters as follows.

The sliding window size T is 20, the number of GAT heads is 4, the number of GRU hidden layers is 64, q = 10^-4 in the POT threshold selection, K₁=14, K₂=20, and stochastic gradient descent is performed using the Adam optimizer with an initial learning rate of 10^-4 and 10 iterations during model training. All experiments were performed on an NVIDIA GeForce GTX 3090 GPU.

4.3 Comparison with baselines

We compared the proposed model with five other state-of-the-art models: LSTM-NDT [22], DAGMM [6], OmniAnomaly [32], MTAD-GAT [16], and GDN [1]. The results are listed in Table 2.

Table 2
Performance of the proposed ACAM-AD model and four baseline approaches

Methods MSL SMAP SMD

Precision Recall F1 Precision Recall F1 Precision Recall F1

LSTM-NDT 0.5934 0.5374 0.5640 0.8965 0.8846 0.8905 0.5684 0.6438 0.6037

DAGMM 0.5412 0.9934 0.7007 0.8069 0.9891 0.8888 0.5951 0.8782 0.7094

OmniAnomaly 0.8867 0.9117 0.8989 0.7416 0.9776 0.8434 0.8334 0.9449 0.8857

MTAD-GAT 0.8906 0.9123 0.9013 0.8754 0.9440 0.9084 0.9409 0.8524 0.8945

GDN 0.9308 0.9892 0.9591 0.7480 0.9891 0.8518 0.7170 0.9974 0.8342

ACAM-AD 0.8985 0.9599 0.9282 0.8444 0.9969 0.9143 0.9088 0.8987 0.9037

Methods WADI SWaT

Precision Recall F1 Precision Recall F1

LSTM-NDT 0.0138 0.7823 0.0271 0.7778 0.5109 0.6167

DAGMM 0.0760 0.9981 0.1412 0.9933 0.6879 0.8128

OmniAnomaly 0.3158 0.6541 0.4260 0.9782 0.6957 0.8131

MTAD-GAT 0.2818 0.8012 0.4169 0.9718 0.6959 0.8109

GDN 0.2912 0.7931 0.4260 0.9697 0.6957 0.8101

ACAM-AD 0.5479 0.5507 0.5977 0.8463 0.8177 0.8317

Methods	MSL	SMAP	SMD
LSTM-NDT	0.5934	0.5374	0.5640	0.8965	0.8846	0.8905	0.5684	0.6438	0.6037
DAGMM	0.5412	0.9934	0.7007	0.8069	0.9891	0.8888	0.5951	0.8782	0.7094
OmniAnomaly	0.8867	0.9117	0.8989	0.7416	0.9776	0.8434	0.8334	0.9449	0.8857
MTAD-GAT	0.8906	0.9123	0.9013	0.8754	0.9440	0.9084	0.9409	0.8524	0.8945
GDN	0.9308	0.9892	0.9591	0.7480	0.9891	0.8518	0.7170	0.9974	0.8342
ACAM-AD	0.8985	0.9599	0.9282	0.8444	0.9969	0.9143	0.9088	0.8987	0.9037
Methods	WADI	SWaT
	Precision	Recall	F1	Precision	Recall	F1
LSTM-NDT	0.0138	0.7823	0.0271	0.7778	0.5109	0.6167
DAGMM	0.0760	0.9981	0.1412	0.9933	0.6879	0.8128
OmniAnomaly	0.3158	0.6541	0.4260	0.9782	0.6957	0.8131
MTAD-GAT	0.2818	0.8012	0.4169	0.9718	0.6959	0.8109
GDN	0.2912	0.7931	0.4260	0.9697	0.6957	0.8101
ACAM-AD	0.5479	0.5507	0.5977	0.8463	0.8177	0.8317

The ACAM-AD model outperforms the baselines in terms of the F1 values for all datasets except for MSL, where GDN has the highest F1 value (0.9591), demonstrating high accuracy and generalization ability. The results show good consistency between the actual and predicted values and the overall trend on the training set.

All models perform relatively poorly on WADI due to the long sequence lengths and data modality. Specifically, the ACAM-AD model has a 17.17% higher F1 score than the state-of-the-art baseline models. The baseline method LSTM-NDT only scores high on the SMAP dataset but performs poorly on the other datasets, indicating that the model may be sensitive to the information in the datasets. The DAGMM model performs very well for short datasets like SMAP, but its scores are significantly lower for the other datasets with longer sequences. The reason is that this method does not use sequence windows but only a single GRU model. In contrast, the autocorrelation and GRU-Attention mechanisms of the proposed model accurately capture the temporal dependence of the time-series data. Recent models, such as MTAD-GAT and GDN, are the most advanced models. They use graph attention mechanisms to focus on specific modes of the data, and all performed well on the five datasets. MTAD-GAT only utilizes GAT for an attention mechanism, and the GDN does not model the temporal perspective. The proposed multi-module, multi-perspective model is more comprehensive than the other models and achieved the best performance on all five datasets.

Because this model performs anomaly detection by conducting prediction followed by detection, we provide visualizations of the prediction performance of the ACAM-AD model in Fig. 5 using features No. 1 and No. 8 of the SMD 1-1 dataset.

Fig. 5

Actual and predicted values obtained from the ACAM-AD model on the SMD 1-1 training set).

We also use the SMD 1-1 sensor data set as an example and plot the predicted and true value fluctuation of features 1 and 8 on the testing set and the anomaly score (Fig. 6) to show the anomaly detection performance of the ACAM-AD model.

Fig. 6

Anomaly detection performance of the ACAM-AD model on the SMD 1-1 testing set.

The results of plots one and two indicate some fluctuations due to outliers. The third plot shows the anomaly score and threshold. The fourth and fifth plots indicate high consistency between the true and predicted anomalies on the testing set. Figures 5 and 6 demonstrate the effectiveness and high accuracy of the model.

4.4 Training time

We analyzed the training time of the ACAM-AD and compared it to the model without sparse operations and another method using GAT. We measured the average time taken per epoch on the 5 public datasets. Table 3 lists the average training times in seconds for all models on all datasets per epoch, and Fig. 7 also shows some results. The training time of ACAM-AD is lower than those of the baseline methods.

Table 3
Comparison of training times in seconds per epoch

Method MSL SMAP SMD WADI SWaT

ACAM-AD 76.94 162.96 268.32 174.16 54.23

w/o sparse operation 1 81.10 201.15 313.40 212.34 71.11

w/o graph-learning 90.62 220.77 337.55 232.49 80.68

MTAD-GAT 1287.42 1015.03 6564.11 9812.13 103.93

Method	MSL	SMAP	SMD	WADI	SWaT
ACAM-AD	76.94	162.96	268.32	174.16	54.23
w/o sparse operation 1	81.10	201.15	313.40	212.34	71.11
w/o graph-learning	90.62	220.77	337.55	232.49	80.68
MTAD-GAT	1287.42	1015.03	6564.11	9812.13	103.93

Fig. 7

Training time.

4.5 Ablation study

Four sets of ablation experiments were conducted to determine the influence of different modules on model performance. The experiments were conducted by removing the autocorrelation layer, GAT, attention mechanism, and AR layer from the ACAM-AD model and calculating the F1 values. The results are listed in Table 4. All other parameters had the same values to avoid interfering with our validity analysis.

The results show that each module is required in the model, and its deletion has different impacts on model accuracy. Omitting the attention mechanism has the smallest accuracy impact in most cases because the GRU and the autocorrelation mechanism also mode the temporal dimension. Removing the AR model results in a less than 10% reduction in the F1 value because it improves model robustness. Removing the GAT has the largest influence on model accuracy because it is the only module performing correlation analysis between features. Because of the large number of modules, the removal of one module does not degrade the performance significantly. The ablation study results show that each module of the model is necessary and effective.

Table 4
Results of ablation study

Model MSL SMAP SMD WADI SWaT

ACAM-AD 0.9282 0.9143 0.9037 0.5977 0.8317

w/o Auto-Correlation 0.8051 0.8136 0.8120 0.5411 0.7995

w/o GAT 0.7806 0.7731 0.8032 0.5492 0.7847

w/o Attention 0.8665 0.8651 0.8427 0.5588 0.8136

w/o AR 0.8587 0.8354 0.8307 0.5614 0.8058

Model	MSL	SMAP	SMD	WADI	SWaT
ACAM-AD	0.9282	0.9143	0.9037	0.5977	0.8317
w/o Auto-Correlation	0.8051	0.8136	0.8120	0.5411	0.7995
w/o GAT	0.7806	0.7731	0.8032	0.5492	0.7847
w/o Attention	0.8665	0.8651	0.8427	0.5588	0.8136
w/o AR	0.8587	0.8354	0.8307	0.5614	0.8058

5 Conclusion

We proposed the novel ACAM-AD framework for anomaly detection in multivariate time series data. The model consists of an autocorrelation mechanism, GAT network, and a GRU-Attention mechanism. It models the correlation between the temporal characteristics and features of time-series data. Sparse attention mechanisms were used to minimize memory usage and training time. Extensive experiments on spacecraft, server, and water treatment datasets show that the proposed algorithm provides higher accuracy than other state-of-the-art methods. Future research should focus on enhancing the anomaly diagnosis capability and interpretability of the models.

Footnotes

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Nos. 62072024, 41971396), the R&D Program of the Beijing Municipal Education Commission (Nos. KM202210016002, KM202110016001), the Scientific Research Foundation of Beijing University of Civil Engineering and Architecture (Nos. KYJJ2017017, Y19-19), and the Projects of the Beijing Advanced Innovation Center for Future Urban Design, Beijing University of Civil Engineering and Architecture (Nos. UDC2019033324).

References

Deng

, Hooi

Graph Neural Network-Based Anomaly Detection in Multivariate Time Series, 2021.

Bianco

A.M.

, Martinez

E.J.

, Garcia Ben

, Yohai

V.J.

Robust Procedures for Regression Models with ARIMA Errors, Physica-Verlag HD, 1996.

Mathur

A.P.

, Tippenhauer

N.O.

Tippenhauer, SWaT: a water treatment testbed for research and training on ICS security. In 2016 International Workshop on Cyber-physical Systems for Smart Water Networks (CySWater), 2016, 31–36.

Vaswani

, Shazeer

, Parmar

et al. Attention Is All You Need// arXiv, 2017.

, Wang

N.Y.

, Chen

T.Q.

, Li

Empirical Evaluation of Rectified Activations in Convolutional Network, Computer Science, 2015.

Zong

, Song

, Min

M.R.

et al. Deep autoencoding gaussian mixture model for unsupervised anomaly detection// International Conference on Learning Representations, 2018, 781–795.

Hawkins

D.M.

Identification of Outliers, Biometrics 37(4) (1981), 860.

Park

, Hoshi

, Kemp

C.C.

A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-Based Variational Autoencoder, , IEEE Robotics and Automation Letters 3 (2018), 1544–1551.

Kingma

D.P.

, Welling

An Introduction toVariational Autoencoders, 2019.

10.

Box

G.E.P.

, Jenkins

G.M.

Time series analysis: forecasting and control, Journal of Time 31(3) (2010).

11.

Berzal

Outlier analysis, Computing Reviews, 2014, 55–10.

12.

Lai

, Chang

W.C.

, Yang

, Liu

Modeling Long and Short-Term Temporal Patterns with Deep Neural Networks. ACM, 2018.

13.

Jonathan

, Sridhar

, Khurum

N.J.

, Aditya

Dataset to Support Research in the Design of Secure Water Treatment Systems. In Critical Information Infrastructures Security, 2017, 88–99.

14.

Ren

, Xu

, Wang

et al. Time-Series Anomaly Detection Service at Microsoft// Acm Sigkdd International Conference. ACM, 2019.

15.

, Xu

, Wang

et al. Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting, 2021.

16.

Zhao

, Wang

, Duan

, Huang

, Zhang

Multivariate Time-series Anomaly Detection via Graph Attention Network. 2020.

17.

Chung

, Gulcehre

, Cho

, Bengio

Empirical evaluation of gated recurrent neural networks on sequence modeling, 2014.

18.

Goh

, Adepu

, Junejo

K.N.

, Mathur

“A dataset to support research in the design of secure water treatment systems,” in CRITIS, 2016, pp. 88–99.

19.

, Yang

, Guo

, Jensen.

C.S.

Risk-aware path selection with time-varying, uncertain travel costs: a time series approach, VLDBJ 27(2) (2018), 179–200.

20.

Connor

J.T.

, Atlas

L.E.

, Martin

R.D.

Recurrent networks and NARMA modelling, Advances in Neural Information Processing Systems 1992, 01–308.

21.

Cho

, Merrienboer

B.V.

, Gulcehre

et al. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. Computer Science, 2014.

22.

Hundman

, Constantinou

, Laporte

, Colwell

, Soderstrom

Detecting Spacecraft Anomalies UsingLSTMs and Nonparametric Dynamic Thresholding. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery&Data Mining,KDD2018, London, UK, August 19-23, 2018. 387–395.

23.

Deng

, Yu

Deep learning: Methods and applications, Foundations and Trends in Signal Processing 7(3–4) (2014), 197–387.

24.

Defferrard

, Bresson

, Vandergheynst

Convolutional neural networks on graphs with fast localized spectral filtering, Advances in Neural Information Processing Systems, 2016, 3844–3852.

25.

Gupta

, Gao

, Aggarwal

C.C.

, Han

Outlier Detection for Temporal Data: A Survey, IEEE Transactions on Knowledge & Data Engineering 26(9) (2014), 2250–2267.

26.

Wiener

Generalized harmonic analysis, Acta Math, 1930.

27.

Velikovi

, Cucurull

, Casanova

et al. Graph Attention Networks, 2017

28.

Cleveland

R.B.

, Cleveland

W.S.

STL: A seasonal-trend decomposition procedure based on Loess, Journal of Official Statistics 6(1) (1990).

29.

Cirstea

R.G.

, Kieu

, Guo

, Yang

, Pan

S.J.

Enhancenet: Plugin neural networks for enhancing correlated time series forecasting. In ICDE, pages 1739–1750, 2021.

30.

Hochreiter

, Schmidhuber

Long Short-Term Memory, Neural Computation 9(8) (1997), 1735–1780.

31.

Huang

, Wang

, Wu

et al. DSANet: Dual Self-Attention Network for Multivariate Time Series Forecasting, the 28th ACM International Conference. ACM, 2019.

32.

, Zhao

, Niu

, Liu

, Sun

, Pei

Robust anomaly detection for multivariate time series through stochastic recurrent neural network, in SIGKDD, 2019, pp. 2828–2837.

33.

Z.H.

et al. Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks, KDD 2020.

ACAM-AD: Autocorrelation and attention mechanism-based anomaly detection in multivariate time series

Abstract

Keywords

1 Introduction

2 Related work

3 Proposed framework

3.1 Problem statement

3.2 Overview

3.4.1 Auto-correlation layer

4.1 Datasets

Table 1 Dataset information Dataset Training data Test data Dimensions Anomalies (%) MSL 58317 73729 55 10.72 SMAP 135183 427617 25 13.13 SMD 708405 708420 38 4.16 SWaT 496800 449919 51 11.98 WADI 1048571 172801 123 5.99

Table 3 Comparison of training times in seconds per epoch Method MSL SMAP SMD WADI SWaT ACAM-AD 76.94 162.96 268.32 174.16 54.23 w/o sparse operation 1 81.10 201.15 313.40 212.34 71.11 w/o graph-learning 90.62 220.77 337.55 232.49 80.68 MTAD-GAT 1287.42 1015.03 6564.11 9812.13 103.93

Table 4 Results of ablation study Model MSL SMAP SMD WADI SWaT ACAM-AD 0.9282 0.9143 0.9037 0.5977 0.8317 w/o Auto-Correlation 0.8051 0.8136 0.8120 0.5411 0.7995 w/o GAT 0.7806 0.7731 0.8032 0.5492 0.7847 w/o Attention 0.8665 0.8651 0.8427 0.5588 0.8136 w/o AR 0.8587 0.8354 0.8307 0.5614 0.8058

Footnotes

Acknowledgments

References

Table 1
Dataset information

Dataset Training data Test data Dimensions Anomalies (%)

MSL 58317 73729 55 10.72

SMAP 135183 427617 25 13.13

SMD 708405 708420 38 4.16

SWaT 496800 449919 51 11.98

WADI 1048571 172801 123 5.99

Table 3
Comparison of training times in seconds per epoch

Method MSL SMAP SMD WADI SWaT

ACAM-AD 76.94 162.96 268.32 174.16 54.23

w/o sparse operation 1 81.10 201.15 313.40 212.34 71.11

w/o graph-learning 90.62 220.77 337.55 232.49 80.68

MTAD-GAT 1287.42 1015.03 6564.11 9812.13 103.93