Resformer: Combine quadratic linear transformation with efficient sparse Transformer for long-term series forecasting

Abstract

With the continuous development of deep learning, long sequence time-series forecasting (LSTF) has attracted more and more attention in power consumption prediction, traffic prediction and stock prediction. In recent studies, various improved models of Transformer are favored. While these models have made breakthroughs in reducing the time and space complexity of Transformer, there are still some problems, such as the predictive power of the improved model being slightly lower than that of Transformer. And these models ignore the importance of special values in the time series. To solve these problems, we designed a more concise network named Resformer, which has four significant characteristics: (1) The fully sparse self-attention mechanism achieves $O(\textit{LlogL})$ time complexity. (2) The AMS module is used to process the special values of time series and has comparable performance on sequences dependency alignment. (3) Using quadratic linear transformation, a simple LT module is designed to replace the self-attention mechanism. It effectively reduces redundant information. (4) The DistPooling method based on data distribution is proposed to suppress redundant information and noise. A large number of experiments on real data sets show that the Resformer method is superior to the existing improved model and standard Transformer method.

Keywords

Sparse self-attention time-series forecasting quadratic linear transformation

1. Introduction

In daily life, time-series forecasting plays an important role, such as sales analysis, enterprise decisionmaking, and smart grid management, etc. Time-series forecasting can help people have a more clearly understanding of the trend of the next period of time, so as to help people make better decisions. In particular, long sequence time-series forecasting (LSTF) has received increasing attention. As the name suggests, LSTF uses past time data to make predictions about a longer time into the future. In the existing research, people have proposed many methods to solve this problem. However, most studies have considered how to solve the prediction of short-term time series. RNN [1], LSTM [2] and their variants [3, 4] are often used in time series forecasting and have a good effect on solving short-time time series forecasting. As the length of the prediction sequence gradually increases, the requirements for the network model become higher and higher. At this time, the predictive capabilities of RNN and LSTM models are not meet the requirements. In addition, many studies based on temporal convolutional networks (TCN) [5, 6, 7] attempt to model temporal causality using causal convolution.

Figure 1.

(a) Comparison of our model with other models in the case of prediction of multiple variables. (b) Comparison of our model with other models in the case of predicting a single variable.

In recent studies, Transformer models are increasingly used to solve long-term series forecasting. The transformer model shows better performance than the RNN model in capturing remote dependencies. The self-attention mechanism can reduce the maximum length of network signal propagation path to a constant. Compared with the sequential structure of RNN, Transformer has good parallel processing ability, so Transformer has great potential in LSTF problems. However,the transformer model has natural limitations. The running time of the transformer is greatly related to the length of input and output. In the long-term series forecasting, the L-quadratic computation and memory consumption of transformer are unacceptable. As the length of the prediction time series increases, people need to devote a lot of resources to the transformer to obtain results. For many people, these required resources cannot be afforded. Therefore, reducing the time complexity of the transformer is the key to solving the problem. Secondly, Vanilla Transformer’s dynamic decoding caused it to be slow in step-by-step reasoning. The large amount of time and resource consumption of self-attention mechanism becomes the bottleneck of its application in long time series prediction. The long tail information redundancy caused by global attention is also an unavoidable defect.

There have been studies to address some of the problems that the self-attention mechanism cannot avoid. LogSparse Transformer [8] allows each cell to pay attention to only its previous cells and itself in exponential steps. Sparse Transformer [9] divides the self-attention operation into two parts according to specific conditions. Longformer [10] reduce the time complexity of the self-attention mechanism to ${O}(\textit{LlogL})$ by limiting low-quality attentional connections. Reformer [11] completes parallel processing by rearranging the time series with hash values, reducing the computational complexity to ${O}(\textit{LlogL})$ , but it shows good predictive power only for extremely long data. Recently, Linformer [12] reduces the time complexity of the self-attention mechanism to ${O}({L})$ , but it can not handle real world time series well. Informer [13] uses the sparse self-attention mechanism to reduce the time complexity to ${O}(\textit{LlogL})$ . At the same time, it proposes a generative decoder, which can obtain long sequence output with only one forward step. In our research, we also borrowed a lot from the sparse self-attention mechanism. However, compared with the sparsity of attentional links, Informer’s query sparsity is more likely to cause the loss of trend information inside the input sequence. At the same time, Informer’s sparse selfattention mechanism will generate a large amount of mean redundant information of values. Experiments show that directly calculating the mean value of the input data will have a good effect on the results.

At the same time, the above models only make some adjustments to the self-attention mechanism in order to reduce time complexity. As the number of model layers increases, the network still becomes very complex. Therefore, we hope to use a simpler way to replace the self-attention mechanism, rather than modify the self-attention mechanism. In addition, the powerful reasoning ability of transformer has made it attract great attention in the field of NLP recently. This general model is used to solve various NLP and imaging tasks. However, in time series forecasting, the dependence on the self-attention mechanism neglects the importance of some special values such as the mean or maximum value, and this information is easy to learn. Therefore, we hope to design a module that can be used to obtain this information, so as to improve the accuracy of time series forecasting while reducing the complexity of the model.

For this reason, we designed a simpler module based on the existing research to solve these existing problems. Experiments on a large number of real data sets prove that our method is useful for predicting long time series. This paper mainly does the following work:

•

We designed a fully sparse self-attention mechanism. It only retains the header information that is beneficial to the result, and discards all tail information that is not beneficial to the result. It keeps the model complexity to ${O}(\textit{LlogL})$ , while retaining useful information to the greatest extent and suppressing useless information.

•

We designed an information extraction module for the average and maximum value of the time series (AMS). This module can more easily learn the trend information contained in the time series. It can be amicably combined with a fully sparse self-attention mechanism to achieve a significant improvement in predictive ability.

•

We designed a quadratic linear module (LT) with a simple structure to replace the self-attention mechanism. While improving the time series forecasting ability, it simplifies the network structure and reduces the running time.

•

We designed a self-selected pooling mechanism based on the distribution of data, called DistPooling. DistPooling can purify data more friendly, reduce sequence dimension and improve prediction speed.

2. Methodology

In this section, we first introduce the fully sparse self-attention mechanism based on probsparse selfattention mechanism. Then we explain in detail the trend learning module called AMS and the linear transformation used to replace self-attention. Finally, the DistPooling method based on the original distribution of sequences is introduced. In the next section, we will go through our network structure in detail to demonstrate how our proposed module works.

2.1 Fully sparse self-attention mechanism

Self-attention Mechanism

The canonical self-attention in Scaled Dot-Product Attention [14] transforms the time series encoded as feature vectors into the queries and keys of dimension $d_{k}$ and the values of dimension $d_{v}$ . Scaled Dot-Product Attention compute the dot products of the query with all keys, divide the result by $\sqrt{d_{k}}$ , and use the Softmax function to normalize and get the weight. In practice, Scaled dot-Product Attention does parallel calculations on the attention of a set of queries packaged into matrix ${Q}$ . Keys and values are also packaged into matrices ${K}$ and ${V}$ . Note that the calculated output matrix is:

$\displaystyle\textit{Atten}(Q,K,V)=\textit{softmax}\left(\frac{QK^{T}}{\sqrt{d% _{k}}}\right)V$ (1)

In order to make the self-attention mechanism more specific, we further discuss the elements of each row in the query matrix. In the mechanism of attention, let $q_{i}$ , $k_{i}$ , and $v_{i}$ represent the i-th line in Q, K, V. According to the formula in [15], For query $q_{i}$ , the attention result is:

$\displaystyle\textit{Atten}(q_{i},K,V)=\sum_{j}\frac{k(q_{i},k_{j})}{\sum_{l}k% (q_{i},k_{l})}v_{j}=E_{p(k_{j}|q_{i})}[v_{j}]$ (2)

where $p(k_{j}|q_{i})=k(q_{i},k_{j})/\sum_{l}k(q_{i},k_{l})$ and $k(q_{i},k_{j})=\exp(q_{i}k_{j}^{T}/\sqrt{d})$ . The attention between $q_{i}$ and all keys is calculated by the attention mechanism as probability $p(k_{j}|q_{i})$ , the probability $p(k_{j}|q_{i})$ and values are combined to obtain the output. It requires quadratic dot product calculation and memory usage, resulting in significant time and resource consumption, which is the main obstacle to long series forecasting.

Query Sparsity Measurement

According to the Eq. (2), query $q_{i}$ ’s attention can be viewed as a probability $p(k_{j}|q_{i})$ . The probability distribution far from the uniform distribution plays a dominant role in the prediction results. For the nearly uniform distribution of $p(k_{j}|q_{i})$ , the Transformer mechanism produces a large amount of redundant information resulting in the long tail phenomenon. Query Sparsity Measurement measure the likeness through Kullback-Leibler divergence $KL(q||p)=\textup{ln}\sum_{l=1}^{L_{K}}e^{q_{i}k_{i}^{T}/\sqrt{d}}-\frac{1}{L_{% K}}\sum_{j=1}^{L_{K}}q_{i}k_{i}^{T}/\sqrt{d}-\textup{ln}L_{K}$ . Dropping the constant, the sparse metric of the $i$ -th query is

$\displaystyle M(q_{i},K)=\textup{ln}\sum_{j=1}^{L_{K}}e^{\frac{q_{i}k_{j}^{T}}% {\sqrt{d}}}-\frac{1}{L_{K}}\sum_{j=1}^{L_{K}}\frac{q_{i}k_{j}^{T}}{\sqrt{d}}$ (3)

where $\textup{ln}\sum_{j=1}^{L_{K}}e^{\frac{q_{i}k_{j}^{T}}{\sqrt{d}}}$ is the Log-Sum-Exp (LSE) of $q_{i}$ on all the keys, and $\frac{1}{L_{K}}\sum_{j=1}^{L_{K}}\frac{q_{i}k_{j}^{T}}{\sqrt{d}}$ is their arithmetic mean. For queries with a large $M(q_{i},K)$ , its attention has a high probability of containing the dominant header information for sequence differentiation recognition.

ProbSparse Self-attention

ProbSparse self-attention uses the value of the $M(q_{i},K)$ to determine the query to be processed, allowing each key to focus only on queries with a large $M(q_{i},K)$ :

$\displaystyle\textit{Atten}(Q,K,V)=\textit{softmax}\left(\frac{\overline{Q}K^{% T}}{\sqrt{d_{k}}}\right)V$ (4)

where $\overline{Q}$ and $Q$ have the same size, but $\overline{Q}$ only contains queries with a larger value of $M(q_{i},K)$ . In practice, the number of reserved queries is represented by u. ProbSparse Self-attention sets $u=c\cdot\textup{ln}L_{Q}$ .

Fully sparse self-attention mechanism

To ensure that the outputs of the attention mechanism are dimensionally consistent with the inputs, $q_{i}$ that does not belong to $\overline{Q}$ needs to be processed. Attention operation on the tail information increases time consumption and ignores tail information leads to loss of sequence-dependent coherence. In particular, the sparsity of attention leads to the destruction of the inherent trends of time series. We chose to construct a special AMS module to replace the processing of tail information. The AMS module contains the average value and maximum value information of the original input time series, which is easy to mine the trend information of the input time series. In the next chapter we will introduce the AMS module in detail. Therefore, the query that does not belong to Q is not processed here, and the self-attention result is replaced by 0. We call it a fully sparse self-attention mechanism. At the same time, the result of the fully sparse self-attention mechanism is merged with the output of AMS. Experiments show that the fusion of the two kinds of information greatly improves the predictive ability of the model. The processing process of the fully sparse self-attention mechanism is as follows:

$\displaystyle\textit{Atten}(q_{i},K,V)=\left\{\begin{array}[]{ll}\sum_{j}\frac% {k(q_{i},k_{j})}{\sum_{l}k(q_{i},k_{l})}v_{j}&q_{i}\in\overline{Q}\\ 0&q_{i}\notin\overline{Q}\\ \end{array}\right.$ (5)

where $k(q_{i},k_{j})$ selects the asymmetric exponential kernel $\exp(q_{i}k_{j}^{T}/\sqrt{d})$ .

2.2 AMS

Recently, the self-attention mechanism has been widely used to solve the problem of long-term sequence prediction. As the prediction length increases, the high time complexity of the self-attention mechanism becomes an obstacle. The Transformer based models have to use a sparse form of attention mechanism to deal with the quadratic complexity problem, but the trend information is destroyed. Some recent studies [16] have also focused on the use of averages. In addition, with the deepening of research, the lag of the prediction sequence has been alleviated. However, it is still a great challenge to predict the change of sequence trend, especially for the judgment of maximum value, which can not be solved well by the existing prediction methods. To solve these problems, we propose an AMS module for fast learning the mean and maximum information of the input sequences.

For the input sequence $\textbf{X}\in\mathbb{R}^{L_{x}\times d_{\textit{model}}}$ , where $d_{\textit{model}}$ represents the feature number of the input sequence, and $L_{x}$ represents the length of the input sequence. We calculate the average and maximum values $P_{\textit{avg}}\in\mathbb{R}^{1\times d_{\textit{model}}}$ and $P_{{\max}}\in\mathbb{R}^{1\times d_{\textit{model}}}$ in the length dimension. That is, the average value and maximum value are calculated separately for each feature of the input sequence. Then combine $P_{\textit{avg}}$ and $P_{{\max}}$ by concatenation. The computation process is summarized as follow:

$\displaystyle\textit{AMS}(X)=\sigma(\textit{Linear}(P_{\textit{avg}}\circ P_{{% \max}}))$ (6)

where $\sigma$ represents the sigmoid function and $\circ$ represents concatenate operation. Linear represents a fully connected operation. Its purpose is to halve the length of the input sequence.

Figure 2.

Resformer model overview. Right: The whole flow of encoder and decoder. The fully sparse self-attention mechanism and AMS module process the input sequence in parallel. The encoder connects LT module to simplify network structure and improve prediction accuracy. The decoder connects to the standard self-attention mechanism and instantly predicts the output elements in a generated style (orange sequence). Left: The fully sparse self-attention mechanism determines whether q needs to be processed according to the category to which q belongs.

2.3 Linear transformation (LT)

In this section, we will introduce a quadratic linear network structure that combines linear transformation and full connection layers.

In the previous section, we considered using the AMS module to improve prediction accuracy. The stacking of multiple layers of fully sparse self-attention results in repeated learning of information. The redundancy of information makes the network structure become complicated, and causes a large amount of time consumption. Therefore, we hope to replace the self-attention mechanism with a simpler network.

Inspired by MLP-related methods [17, 18, 19] in recent studies. In the prediction task of long time series, we try to use linear transformation instead of self-attention mechanism to reduce the complexity of the network. This method does not use any self-attention, and only relies on the stacking of the linear layer and the full connected layer. This architecture is more stable than self-attention training and does not require a specific batch or cross-channel standardization (such as BatchNorm, GroupNorm, or LayerNorm). At the same time, the idea of residual network is introduced, and the output result of each linear layer is skip-connection with the output result of the previous linear layer. In the experimental part, we also compared the sparse attention mechanism and linear transformation. It can be seen that the stacking of linear layer and full connection layer can obviously improve the accuracy of prediction.

In addition, while discarding the self-attention mechanism, we also discarded Layer Normalization. Relying only on linear transformation, the training results become very stable. The following is the formula for linear transformation:

$\displaystyle\mathrm{Aff}(\mathbf{X})=\textit{Diag}(\mathbf{\alpha})\mathbf{X}% +\mathbf{B}$ (7)

where $\alpha$ is a learnable weight vector. $\textit{Diag}(\mathbf{\alpha})$ is a diagonal matrix generated by $\alpha$ . B is a learnable weight matrix. Thus, the linear transformation layer only enlarges or shifts the input elements. The whole process has almost no computational cost and results can be obtained in a short time. At the same time, the LT layer not only gives consideration to self-attention, but also realizes the standardized operation of input data. Compared with the commonly used Layer Normalization and BatchNorm, the LT layer does not rely on batch information at all, and performs independent operations on the input elements of each layer.

In addition, inspired by the self-attention mechanism, we find that nonlinear multi-level connection is helpful for deep-level information mining. In order to learn the integrity of long-term dependence inside the input sequence, we use the combination of Linear and Aff to complete the construction of quadratic Linear structure. The overall processing is as follows:

$\displaystyle\mathbf{Z}=\mathbf{A}\ast\mathrm{Aff}_{\mathbf{\alpha},\mathbf{% \beta}}(\textit{Linear}(\textbf{X}))+\textbf{X}$ (8)

where A is the weight matrix that can be learned. The Linear function in the formula ensures that the input dimension is the same as the output dimension. Therefore, the entire network structure will not change the size of the input data. This feature allows the network to be flexibly placed in any place in the entire project, and it can also be flexibly combined with other network structures. It can be seen from the above two formulas that we embed the Linear layer in the linear transformation implemented by the first formula. The Linear layer, that is, the full connection operation is also equivalent to a linear transformation. At the same time, the superposition of two linear transformations is similar to the definition of the traditional quadratic curve. Therefore we name this process quadratic linear transformation.

Figure 3.

The figure shows the flow chart of self-attention distillation proposed by us. In Self-attention distillation (DistPooling), the second column of squares represents a window size of 4 in dispooling, where the lighter colored squares represent a value closer to the minimum value in the window, darker squares indicate that the value is closer to the maximum value in the window. The pooling results basically retained the distribution of the original sequence.

2.4 DistPooling

The deep neural network can easily realizes the mapping from low dimension to high dimension of time series and effectively completes the acquisition of local and global information. But the increase of the dimension will bring the increase of noise and information redundancy. Pooling purifies data by reducing the dimension of the input sequence. Average Pooling and Max Pooling are the two most common pooling methods, which are widely used in various NLP and image tasks, but there are still some defects. The average Pooling makes the input sequence to tend to be smooth, which reduces the identification and makes it difficult to obtain the dependent connection inside the time series. Max Pooling can effectively suppress noise in image tasks. For time series, especially long series forecasting, the downsampling strategy relying only on the maximum value destroys the original distribution of the data, resulting in loss of information.

In order to solve these problems, we selectively use the maximum or minimum value as the result of pooling according to the original distribution of data. This approach uses edge values to reduce the dimension of the original sequence, which not only inhibits noise, but also improves the identification of the input sequence and preserves more internal connections between time sequences.

For the sequence $\textbf{X}\in\mathbb{R}^{L_{x}\times d_{\textit{model}}}$ , we analyze it in the $d_{i}$ dimension, where $d_{i}\in d_{\textit{model}}$ and $\mathbf{X}_{di}\in\mathbb{R}^{L_{x}\times 1}$ . When the $X_{di}$ is pooled, we use window size $l$ and step size $s$ . When the pooling result becomes half of the original sequence length, $s=2$ . For any task sequence $X_{t}=[X_{di}^{i},X_{di}^{i+1},\ldots,X_{di}^{i+l-2},X_{di}^{i+l-1}]\in X_{di}$ , we select the maximum value $f_{\max}(X_{t})$ and minimum value $f_{\min}(X_{t})$ of the sequence. Then we observe whether all the other elements in the sequence are closer to $f_{\max}(X_{t})$ or closer to $f_{\min}(X_{t})$ in one-dimensional space. The distribution of the sequence was judged according to the distance, and $f_{\max}(X_{t})$ or $f_{\min}(X_{t})$ is selected as the result of pooling. As shown in Fig. 3, the overall calculation is as follows:

$\displaystyle DP(X_{t})=\left\{\begin{array}[]{ll}f_{\min}(X_{t})&\textit{len}% (f_{\min}(X_{t}))<\textit{len}(f_{\max}(X_{t}))\\ f_{\max}(X_{t})&\textit{len}(f_{\max}(X_{t}))<\textit{len}(f_{\min}(X_{t}))\\ \end{array}\right.$ (9)

where $DP(X_{t})$ stands for $\textit{DistPooling}(X_{t})$ , $\textit{len}(X_{i})$ is the sum of distances between $X_{i}\in X_{t}$ and other elements in $X_{t}$ .

3. The network structure

The existing long series time series prediction methods can be roughly divided into two categories. One is prediction based on traditional methods [20, 21, 22]. The other is deep learning methods based on encoder-decoder structures, such as RNN and Transformer and their variants [23, 24, 25]. Our entire framework is based on the standard encoder-decoder structure.

3.1 Encoder

The function of the encoder is to extract features from the input sequence to reduce redundancy, so as to obtain the correlation between the input sequences. In this section, we will introduce in detail how to combine the fully sparse self-attention mechanism with the AMS and LT to complete the work.

For the encoder, the input $t$ -th sequence $\textbf{X}\in\mathbb{R}^{L_{x}\times d_{\textit{model}}}$ can be regarded as a matrix. We show the schematic diagram of the encoder in Fig. 2. The input long time sequence X first passes through a layer of fully sparse self-attention mechanism. The fully sparse self-attention mechanism greatly reduces the time complexity of the self-attention mechanism, and obtains a greater improvement in computing speed. In the prediction task of long-term series, because the input sequence and the output sequence involve a long time span, the time complexity has a greater impact on the long-term series prediction. The sparse attention mechanism can deal with this problem well. After the above operations, the dimension of the output X becomes half of the input length. At the same time, put input X into the AMS module. It can be seen from 2.2 that the output dimension of the AMS module is half of the input dimension, so the output dimension of the fully sparse self-attention mechanism is aligned with the output dimension of the LT module. Therefore, by connecting the two outputs together, we can integrate trend information and other information useful for prediction. Finally, we send the result of the connection to the linear transformation module to further purify the information and make the prediction result more stable. In addition, in recent studies, the distillation mechanism has been widely used in various tasks. The distillation mechanism can very well purify the input data. While ensuring the accuracy of the prediction, it reduces the dimensions of the input data so as to increase the calculation speed. In the encoder, we use the self-attention distillation mechanism in Resformer, and the overall processing process is as follows:

$\displaystyle\mathbf{X}_{\textit{prob}}^{\textit{out}}=\textrm{DistPooling}(% \textrm{Conv1d}(\alpha\textit{Probspare}(\mathbf{X})+(1-\alpha)\textrm{AMS}(% \mathbf{X})))$ (10) $\displaystyle\mathbf{X}_{\textit{enc}}^{\textit{out}}=\textrm{LT}(\mathbf{X}_{% \textit{prob}}^{\textit{out}})$ (11)

where Probspare represents the fully sparse self-attention mechanism, and Conv1d is a one-dimensional convolution operation, the value of kernel is 3, and the ELU activation function is used. $\alpha$ represents the proportion of sparse self-attention results mixed with AMS results. In the experimental part, we discussed the influence of the value of $\alpha$ on the predictive ability of the model. This is different from the traditional direct addition method, we can manually adjust the size of $\alpha$ to make the model reach the optimal state. We therefore name this process linear mixed. In Fig. 3, we use a plus sign to indicate this process.

3.2 Decoder

As shown in Fig. 2, the decoder consists of a fully sparse self-attention mechanism and a complete self-attention mechanism. At the same time, we designed two input sequences for the decoder. One of the vectors is as follows:

$\displaystyle\mathbf{X}_{de}^{1}=\textit{Concat}(\mathbf{X}_{\textit{token}},% \mathbf{X}_{0})\in\mathbb{R}^{(\mathbf{L}_{\textit{token}}+\mathbf{L}_{y})% \times d_{\textit{model}}}$ (12)

where $\mathbf{X}_{\textit{token}}\in\mathbb{R}^{L_{\textit{token}}\times d_{\textit{% model}}}$ is the start tag, and $\mathbf{X}_{0}\in\mathbb{R}^{L_{y}\times d_{\textit{model}}}$ is a placeholder for the target sequence (set the scalar to 0). Taking the prediction of 168 points as an example (the experimental part is a 7-day temperature prediction), we use the known value of the target sequence in the first 5 days as the “start tag”, and input $\mathbf{X}_{de}^{in}=\{{\mathbf{X}_{5d},\mathbf{X}_{0}}\}$ into the generative inference decoder. $\mathbf{X}_{0}$ contains the time stamp of the target sequence. Masked multi-head attention by setting the masked dot product to $-\infty$ . It prevents each position from paying attention to the future position. Another input vector is as follows:

$\displaystyle\mathbf{X}_{de}^{2}=\textit{Concat}(\mathbf{X}_{\textit{token}},% \mathbf{X}_{\textit{AMS}})\in\mathbb{R}^{(\mathbf{L}_{\textit{token}}+\mathbf{% L}_{y})\times d_{\textit{model}}}$ (13)

We first put $\mathbf{X}_{\textit{token}}$ into the AMS module, and make the length of the output consistent with the length of the target sequence. Then the output of the AMS module and $\mathbf{X}_{\textit{token}}$ are spliced together as the second input of the decoder. After that, the input vector $\mathbf{X}_{de}^{1}$ is passed through the fully sparse self-attention mechanism, and the result is merged with $\mathbf{X}_{de}^{2}$ . We then send the combined results to the full selfattention mechanism. Finally, the full connection layer is added to obtain the prediction results.

4. Experiment

We conducted extensive experiments on four public data sets, including three electricity data ETTs from different counties and cities in China and a public benchmark data sets.

Table 1
Univariate long sequence time-series forecasting result on four cases

Methods		Ours		Informer		LogTrans		Reformer		LSTMa		DeepAR		ARIMA
Metric		MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	24	0.071	0.202	0.097	0.251	0.099	0.254	0.245	0.398	0.105	0.259	0.113	0.298	0.109	0.276
	48	0.091	0.239	0.162	0.324	0.168	0.347	0.279	0.437	0.186	0.357	0.170	0.331	0.174	0.419
	168	0.111	0.257	0.186	0.351	0.211	0.379	1.516	1.164	0.229	0.387	0.234	0.419	0.399	0.514
	336	0.124	0.282	0.232	0.392	0.241	0.394	1.893	1.210	0.611	0.674	0.446	0.572	0.493	0.587
	720	0.223	0.394	0.271	0.464	0.274	0.462	2.123	1.438	0.681	0.752	0.664	0.711	0.657	0.782
ETTh2	24	0.073	0.206	0.095	0.101	0.103	0.264	0.259	0.421	0.179	0.325	0.099	0.271	3.597	0.452
	48	0.129	0.287	0.157	0.322	0.175	0.358	0.449	0.527	0.201	0.372	0.159	0.328	3.127	0.497
	168	0.221	0.384	0.241	0.409	0.247	0.411	1.110	0.894	0.329	0.512	0.254	0.421	2.793	0.738
	336	0.278	0.437	0.270	0.425	0.273	0.439	1.676	1.254	0.593	0.628	0.618	0.691	2.745	0.693
	720	0.222	0.389	0.279	0.425	0.296	0.437	2.129	1.721	0.653	0.649	0.422	0.567	2.869	1.028
ETTm1	24	0.012	0.083	0.042	0.154	0.067	0.193	0.101	0.235	0.120	0.219	0.096	0.209	0.087	0.214
	48	0.030	0.130	0.071	0.221	0.082	0.232	0.251	0.387	0.306	0.425	0.208	0.354	0.175	0.319
	96	0.063	0.195	0.203	0.396	0.206	0.401	0.927	0.752	0.273	0.459	0.368	0.502	0.231	0.406
	288	0.232	0.406	0.406	0.567	0.412	0.576	1.124	1.268	0.533	0.597	0.972	0.785	0.459	0.562
	672	0.327	0.496	0.523	0.658	0.601	0.724	1.768	1.513	1.029	0.854	2.412	1.378	0.634	0.697
Weather	24	0.071	0.211	0.123	0.259	0.125	0.261	0.246	0.410	0.132	0.259	0.124	0.291	0.215	0.342
	48	0.126	0.264	0.186	0.325	0.212	0.374	0.319	0.426	0.189	0.347	0.209	0.361	0.258	0.412
	168	0.150	0.307	0.265	0.383	0.319	0.441	0.648	0.642	0.351	0.476	0.289	0.437	0.511	0.587
	336	0.203	0.329	0.309	0.422	0.367	0.473	1.786	1.028	0.458	0.574	0.593	0.691	0.717	0.726
	720	0.271	0.420	0.362	0.479	0.377	0.495	2.019	1.528	0.854	0.812	0.505	0.631	1.012	0.905

Table 2

Multivariate long sequence time-series forecasting result on four cases

Methods		Ours		Informer		LogTrans		Reformer		LSTMa		LSTnet
Metric		MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	24	0.429	0.457	0.504	0.501	0.691	0.598	0.982	0.763	0.654	0.620	1.331	0.921
	48	0.554	0.552	0.623	0.601	0.773	0.749	1.299	0.915	0.726	0.680	1.432	0.957
	168	0.749	0.687	1.041	0.821	1.097	0.837	1.813	1.254	1.215	0.892	1.821	1.002
	336	0.873	0.742	1.214	0.862	1.352	0.912	2.015	1.364	1.451	1.005	2.783	1.394
	720	1.008	0.815	1.335	0.942	1.423	1.256	2.401	1.506	1.988	1.301	2.136	1.402
ETTh2	24	0.413	0.486	0.466	0.540	0.831	0.733	1.496	1.568	1.164	0.853	2.654	1.357
	48	1.266	0.927	2.295	1.211	1.822	1.031	1.937	1.726	1.625	1.224	3.571	1.720
	168	3.105	1.382	3.616	1.605	4.124	1.759	4.652	1.930	4.139	1.724	3.215	2.506
	336	2.764	1.281	3.153	1.429	3.861	1.723	4.011	1.597	3.568	1.527	3.022	1.329
	720	2.484	1.282	3.710	1.668	4.025	1.723	5.396	1.984	3.975	1.792	4.625	3.704
ETTm1	24	0.248	0.331	0.317	0.384	0.431	0.454	0.716	0.591	0.637	0.644	1.954	1.112
	48	0.381	0.410	0.483	0.472	0.516	0.602	0.983	0.864	1.425	0.971	1.987	1.325
	96	0.510	0.490	0.551	0.496	0.784	0.705	1.425	0.821	1.333	0.902	2.354	1.459
	288	0.682	0.642	0.781	0.693	1.325	1.168	1.931	1.184	1.647	1.097	1.259	1.934
	672	0.832	0.684	1.053	0.802	1.549	1.387	2.064	1.356	2.744	1.397	1.983	1.912
Weather	24	0.274	0.362	0.341	0.395	0.427	0.492	0.684	0.591	0.572	0.570	0.605	0.533
	48	0.298	0.387	0.402	0.473	0.408	0.486	0.731	0.622	0.842	0.704	0.651	0.597
	168	0.505	0.481	0.666	0.591	0.734	0.605	1.306	0.827	1.056	0.809	0.739	0.632
	336	0.590	0.544	0.714	0.625	0.757	0.615	1.894	1.033	1.655	1.027	0.793	0.658
	720	0.701	0.683	0.848	0.744	0.897	0.725	2.654	1.327	1.528	1.085	0.879	0.738

Table 3

Runtime analysis

Prediction length		24		48		96		288		672
Informer	MSE	0.	317	0.	483	0.	551	0.	781	1.	053
	MAE	0.	384	0.	472	0.	496	0.	693	0.	802
	TIME	17.	95	20.	45	27.	31	45.	22	119.	37
Autoformer	MSE	0.	383	0.	454	0.	481	0.	634	0.	606
	MAE	0.	403	0.	453	0.	463	0.	528	0.	542
	TIME	19.	57	21.	79	30.	64	47.	32	131.	27
Ours	MSE	0.	248	0.	381	0.	510	0.	682	0.	832
	MAE	0.	331	0.	410	0.	490	0.	642	0.	684
	TIME	11.	37	16.	66	23.	24	41.	43	108.	23

ETT1
¹
ETT dataset was acquired at https://github.com/zhouhaoyi/ETDataset.

ETT is an important indicator of long-term power configuration. ETT includes three real data sets named ETTh1, ETTh2, and ETTm1. They come from two different counties in China. ETTh1 and ETTh2 record power data every hour, and ETTm1 records power data every 15 minutes. Each data set contains seven characteristics, namely oil temperature and six power loads. Training/validation/test is 12/4/4 months.

Weather2
²
Weather dataset was acquired at http://www.ncei.noaa.gov/data/local-climatological-data/.

The data set contains three years of weather data from nearly 1,600 different locations across the United States. Record data every hour. Each row of data contains 12 features, consisting of wet bulb and other climatic features. The train/val/test is 28/10/10 months.

Our proposed method uses the Adam optimizer for optimization, and its learning rate starts from $1e^{-4}$ , and it decays to half every epoch. The total number of times is 8, and stop appropriately early. We set the comparison method as recommended, and the batch size is 32.

4.1 Result analysis

We chose eight time series prediction methods for comparison, including Informer [13], LogTrans [8], Reformer [11], ARIMA [26], Prophet [27], LSTMa [28], LSTnet [29] and DeepAR [30].

Table 4
Ablation study of the LT module

Prediction length		24		48		168		336		720
Use LT	MSE	0.	429	0.	554	0.	749	0.	873	1.	008
	MAE	0.	457	0.	552	0.	687	0.	742	0.	815
	TIME	11.	32	16.	70	41.	04	54.	23	107.	8
Without LT	MSE	0.	484	0.	613	0.	862	1.	124	1.	247
	MAE	0.	472	0.	582	0.	731	0.	796	0.	874
	TIME	12.	78	19.	75	46.	28	58.	83	113.	3

Without LT uses a fully sparse self-attention mechanism instead of the LT module.

Univariate prediction

We tested the performance of our model for univariate prediction. For example, in the ETT data set, we use the oil temperature in the time series to predict the oil temperature. Table 1 is the comparison result between several time series forecasting methods. We can observe: (1) The proposed Resformer model significantly improves the inference performance of all data sets, and its prediction errors rise steadily and slowly within the ever-increasing prediction range. (2) The prediction ability of Resformer model is superior to Informer, LSTMa and Reformer, and the MSE decreases 29% (at 24), 25% (at 48), 32% (at 168), 41% (at 336), 30% (at 720) in average. The Resformer model shows better results than Reformer, and the MSE decreases 94% in average.

Multivariate prediction

We also tested the power of our model in multivariate prediction. In this setting, we not only predict the target feature, but also predict other features in the time series. This kind of prediction is easy to implement, and it can be achieved only by modifying the output length of the final fully connected layer on the basis of univariate prediction. The experimental results can be seen in Table 2. It can be seen from the results: (1) Our model shows good predictive ability when predicting multiple features. (2) In previous studies, LSTM has been widely used for long series time series prediction. Experimental results show that compared with LSTMa, our method reduced MSE by 56% (at 24), 21% (at 48), 38% (at 168 and 336), and 48% (at 720). (3) compared with other Transformer variants. When metric $=$ 336, our method improved 28% compared to Informer, 35% compared to LogTransformer, and 58% compared to Reformer. When metric $=$ 720, our method improved by 25% compared to Informer, 27% compared to LogTransformer, and 58% compared to Reformer.

Table 5

Ablation study of the AMS module

Prediction length		48		168		336
Encoder’s input		48	96	168	336	336	720
Ours	MSE	0.55	0.55	0.74	0.96	0.87	1.19
	MAE	0.53	0.55	0.68	0.77	0.74	0.89
Ours ${}^{*}$	MSE	0.56	0.68	0.87	1.07	1.13	1.19
	MAE	0.56	0.52	0.72	0.82	0.85	0.87

Ours ${}^{*}$ does not use the AMS module.

Table 6

Ablation study of the fully sparse self-attention mechanism

Prediction length		48			168
Encoder’s input		48	96	168	168	336	720
Ours	MSE	0.55	0.55	0.61	0.74	0.96	0.97
	MAE	0.53	0.55	0.55	0.68	0.77	0.79
Ours ${}^{+}$	MSE	0.59	0.64	0.62	0.92	0.91	1.44
	MAE	0.58	0.59	0.57	0.74	0.74	1.01
Ours ${}^{-}$	MSE	0.61	0.77	0.69	1.19	1.24	1.36
	MAE	0.58	0.78	0.62	0.88	0.91	0.98
Informer	MSE	0.68	0.64	0.67	1.04	1.39	1.39
	MAE	0.63	0.60	0.61	0.82	0.96	0.97

Ours ${}^{+}$ uses canonical self-attention mechanism. Ours ${}^{-}$ uses probsparse self-attention mechanism.

Table 7

Ablation study of the DistPooling

Prediction length		24	48	168	336	720
DistPooling	MSE	0.429	0.554	0.749	0.873	1.008
	MAE	0.457	0.552	0.687	0.742	0.815
MaxPooling	MSE	0.446	0.563	0.775	0.902	1.047
	MAE	0.492	0.597	0.736	0.796	0.834
AvgPooling	MSE	0.501	0.607	0.787	0.933	1.197
	MAE	0.533	0.621	0.746	0.847	0.872

Runtime analysis

We compared the model proposed in this paper with Informer and Autoformer [16] on the ETTm1 dataset in terms of prediction accuracy and running time. The experimental results can be seen in Table 3. It can be seen that when the prediction length is short, our model achieves better results, and when the prediction length increases, the Autoformer gradually shows advantages. In terms of running time, the model proposed in this paper takes the least amount of time.

4.2 Ablation research

The performance of the LT module

In order to verify the function of the linear module, we modify the linear transformation module in the network structure to a fully probsparse self-attention layer. And compared two different network structures. The difference between them is only whether to use linear transformation. Through this comparison, it is verified that the linear transformation module not only has a significant improvement in improving the prediction accuracy, but also can reduce the running time very well. The experimental results can be seen in Table 4. It can be seen from the experimental results: (1) In terms of accuracy improvement, the model using the linear transformation module can improve by 10% to 20% compared to the model without the linear module. And as the prediction length increases, the linear transformation module shows more and more advantages in long-term sequence prediction. (2) In terms of running time, the average consumption time of a single epoch of the two models is given in the Table 4. From the experimental results, it can be seen that the linear transformation module compared with the sparse self-attention mechanism, the average consumption time of a single epoch is reduced by 12% to 15%.

Performance of the fully sparse self-attention mechanism

In order to verify the accuracy of the model, we replace the fully sparse self-attention mechanism in the model with a probsparse self-attention mechanism in Informer. After that, the two models were compared experimentally. At the same time, the fully sparse self-attention mechanism and the standard self-attention mechanism are compared in the same way. The comparison results of the three models are shown in the Table 6. The time complexity of ${O}({L^{2}})$ results in transformer’s inability to predict long time series. Repeated information stacking results in performance degradation of sparse self-attention mechanism.

The performance of the AMS module

In order to verify the function of the AMS module, we set up two different network structures with and without AMS module. A series of experiments were performed on the ETT data set for comparison. The results of the experiment are recorded in Table 5. It shows that the extracted mean information is helpful to the prediction result, especially when the length of the prediction sequence is large. In a series of experiments, we also found that when the trend information is fused with the output of fully sparse self-attention layer, the proportion of the two will have a great influence on the prediction results. In the subsequent parameter adjustment experiments, we will give detailed comparison instructions.

The performance of the DistPooling

In order to verify the effectiveness of DistPooling, Max Pooling and Avg Pooling are used to replace DistPooling in the Resformer model on ETTh1 dataset, and the experimental results are shown in Table 7. By comparing the experimental results of predicting a variety of lengths, the DistPooling method obtained better results than other commonly used pooling methods, which proves that the pooling method we adopted is effective in time series prediction.

Figure 4.

The parameter sensitivity of three components in our network

Figure 5.

Visualization of results using different pooling methods

4.3 Parameter adjustment

We use the data set ETTh1 to analyze the sensitivity of parameters under multivariable conditions. Mixing proportion: The output results of the fully sparse self-attention mechanism and the output results of AMS module are fused with the proportion of $\alpha$ and $1-\alpha$ respectively. We set up five scenarios for this. The values of $\alpha$ from small to large are 0.2, 0.4, 0.5, 0.6 and 0.8. The results of the experiment are recorded in the Fig. 4a. At the same time, in order to better study the influence of $\alpha$ on the residual life of the result, we reduced some variables in the experiment. During the experiment, we fixed the input length of the encoder to 96. Under this setting, it is easier to see the influence of $\alpha$ on the prediction length, and saves time. The variation trend of the curve reflects the loss size and the stability of the model under different proportions. We set $\alpha=$ 0.8 (the purple line) in practice. Factor: The information bandwidth of fully sparse self-attention is controlled by factor. From small factor (c $=$ 3) to large factor, the overall performance improved and finally stabilized in Fig. 4b. Since it is easy to intertwine the forecast length as an indicator curve, and the variable c has only 4 values, we fixed the forecast length and used the input length as a variable to make the result easier to observe. In practice, we set the sample factor c $=$ 5. Input Length: Too long input sequence will lead to repeated acquisition of information and reduce the accuracy of short time prediction. As the predicted length increases, the longer input results in a decrease in MSE. Metric $=$ 336 and metric $=$ 720 were compared experimentally. The experimental results are shown in the Fig. 4c.

4.4 Model analysis

Time complexity of fully sparse self-attention mechanism

The self-attention mechanism includes three parts: query, key and value. Usually the input length of query and key is the same, ie $L_{Q}=L_{K}=L$ . The self-attention module learns feature representations iteratively by associating variables in the query with all variables in the key. This results in a time complexity of $O(L^{2})$ for the self-attention mechanism. In the fully sparse self-attention mechanism, we only select sparse Top-u from query to correlate with the variables in the key. During the experiments, we set u to clnL. This makes the time complexity of the fully sparse self-attention mechanism $O(\textit{LlnL})$ .

Visual Analysis of DistPooling

To explore how DistPooling preserves the information of the original time series during downsampling, we perform a visual analysis on three different pooling methods. The visualization results are shown in Fig. 5. AvgPooling uses the mean of itself and surrounding time points to get a smoother downsampling curve. It can be seen that the AvgPooling ignores the local small fluctuations in the time series, and pays more attention to the acquisition of the overall trend. MaxPooling is more sensitive to fluctuations in time series, but the reflection of MaxPooling on details is still not obvious enough. DistPooling selectively obtains down-sampling results by using the distribution of time series within a certain range. This allows DistPooling to preserve the details of the time series to the greatest extent possible.

5. Conclusions

This paper deals with the prediction of long series time series. On the basis of the sparse self-attention mechanism, we designed the fully sparse self-attention mechanism and AMS module to fuse the multidimensional information of the internal dependence of time series. At the same time, we tested the effectiveness of LT module in replacing the self-attention mechanism, and we will continue to explore the prediction of long series time series using only quadratic linear transformation. Through experiments on a series of real data sets, it is proved that our study is helpful for improving the prediction of long series time series.

Footnotes

Acknowledgments

This work was supported in part by the following: the National Natural Science Foundation of China under Grant Nos. 62272281, 62007017, and 61902220, Youth Innovation Technology Project of Higher School in Shandong Province under Grant No. 2019KJN042, and the project ZR2021QF134 supported by Shandong Provincial Natural Science Foundation.

References

Zaremba

Sutskever

and Vinyals

, Recurrent neural network regularization, arXiv preprint arXiv:1409.2329, 2014.

Xingjian

Chen

Wang

Yeung

D.-Y.

Wong

W.-K.

and Woo

W.-c.

, Convolutional LSTM network: A machine learning approach for precipitation nowcasting, in: Advances in Neural Information Processing Systems, 2015, pp. 802–810.

Shahabi

and Liu

, Diffusion convolutional recurrent neural network: Data-driven traffic forecasting, arXiv preprint arXiv:1707.01926, 2017.

Koltun

and Funkhouser

, Dilated residual networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 472–480.

Oord

Dieleman

Zen

Simonyan

Vinyals

Graves

Kalchbrenner

Senior

and Kavukcuoglu

, A generative model for raw audio, Sound Mach, Learn, 2016.

Borovykh

Bohte

and Oosterlee

C.W.

, Conditional time series forecasting with convolutional neural networks, arXiv preprint arXiv:1703.04691, 2017.

Sen

H.-F.

and Dhillon

, Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting, arXiv preprint arXiv:1905.03806, 2019.

Jin

Xuan

Zhou

Chen

Wang

Y.-X.

and Yan

, Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, Advances in Neural Information Processing Systems 32 (2019), 5243–5253.

Child

Gray

Radford

and Sutskever

, Generating long sequences with sparse transformers, arXiv preprint arXiv:1904.10509, 2019.

10.

Beltagy

Peters

M.E.

and Cohan

, Longformer: The long-document transformer, arXiv preprint arXiv:2004.05150, 2020.

11.

Kitaev

Kaiser

Ł.

and Levskaya

, Reformer: The efficient transformer, arXiv preprint arXiv:2001.04451, 2020.

12.

Wang

B.Z.

Khabsa

Fang

and Ma

, Linformer: Self-attention with linear complexity, arXiv preprint arXiv:2006.04768, 2020.

13.

Zhou

Zhang

Peng

Zhang

Xiong

and Zhang

, Informer: Beyond efficient transformer for long sequence time-series forecasting, in: Proceedings of AAAI, 2021.

14.

Vaswani

Shazeer

Parmar

Uszkoreit

Jones

Gomez

A.N.

Kaiser

Ł.

and Polosukhin

, Attention is all you need, in: Advances in Neural Information Processing Systems, 2017, pp. 5998–6008.

15.

Tsai

Y.-H.H.

Bai

Yamada

Morency

L.-P.

and Salakhutdinov

, Transformer Dissection: A Unified Understanding of Transformer’s Attention via the Lens of Kernel, arXiv preprint arXiv:1908.11775, 2019.

16.

Wang

and Long

, Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting, arXiv preprint arXiv:2106.13008, 2021.

17.

Tolstikhin

Houlsby

Kolesnikov

Beyer

Zhai

Unterthiner

Yung

Keysers

Uszkoreit

Lucic

et al., Mlp-mixer: An all-mlp architecture for vision, arXiv preprint arXiv:2105.01601, 2021.

18.

Touvron

Bojanowski

Caron

Cord

El-Nouby

Grave

Joulin

Synnaeve

Verbeek

and Jégou

, Resmlp: Feedforward networks for image classification with data-efficient training, arXiv preprint arXiv:2105.03404, 2021.

19.

Ding

Zhang

Han

and Ding

, RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition, arXiv preprint arXiv:2105.01883, 2021.

20.

Box

G.E.

Jenkins

G.M.

Reinsel

G.C.

and Ljung

G.M.

, Time series analysis: forecasting and control, John Wiley & Sons, 2015.

21.

Seeger

Salinas

and Flunkert

, Bayesian intermittent demand forecasting for large inventories, in: Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016, pp. 4653–4661.

22.

Seeger

Rangapuram

Wang

Salinas

Gasthaus

Januschowski

and Flunkert

, Approximate Bayesian inference in linear state space models for intermittent demand forecasting at scale, arXiv preprint arXiv:1709.07638, 2017.

23.

Dey

and Salem

F.M.

, Gate-variants of gated recurrent unit (GRU) neural networks, in: 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS), IEEE, 2017, pp. 1597–1600.

24.

Lei

Zhang

Wang

S.I.

Dai

and Artzi

, Simple recurrent units for highly parallelizable recurrence, arXiv preprint arXiv:1709.02755, 2017.

25.

Green

Ben

and O’Banion

, Deep transformer models for time series forecasting: The influenza prevalence case, arXiv preprint arXiv:2001.08317, 2020.

26.

Ariyo

A.A.

Adewumi

A.O.

and Ayo

C.K.

, Stock price prediction using the ARIMA model, in: 2014 UKSim-AMSS 16th International Conference on Computer Modelling and Simulation, IEEE, 2014, pp. 106–112.

27.

Taylor

S.J.

and Letham

, Forecasting at scale, The American Statistician 72(1) (2018), 37–45.

28.

Bahdanau

Cho

and Bengio

, Neural machine translation by jointly learning to align and translate, arXiv preprint arXiv:1409.0473, 2014.

29.

Lai

Chang

W.-C.

Yang

and Liu

, Modeling long-and short-term temporal patterns with deep neural networks, in: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, 2018, pp. 95–104.

30.

Salinas

Flunkert

Gasthaus

and Januschowski

, DeepAR: Probabilistic forecasting with autoregressive recurrent networks, International Journal of Forecasting 36(3) (2020), 1181–1191.

Resformer: Combine quadratic linear transformation with efficient sparse Transformer for long-term series forecasting

Abstract

Keywords

1. Introduction

2.1 Fully sparse self-attention mechanism

Self-attention Mechanism

Query Sparsity Measurement

ProbSparse Self-attention

Fully sparse self-attention mechanism

3.1 Encoder

Table 1 Univariate long sequence time-series forecasting result on four cases

ETT1 1 ETT dataset was acquired at https://github.com/zhouhaoyi/ETDataset.

Weather2 2 Weather dataset was acquired at http://www.ncei.noaa.gov/data/local-climatological-data/.

Table 4 Ablation study of the LT module

Univariate prediction

Multivariate prediction

Runtime analysis

The performance of the LT module

Performance of the fully sparse self-attention mechanism

The performance of the AMS module

The performance of the DistPooling

4.4 Model analysis

Time complexity of fully sparse self-attention mechanism

Visual Analysis of DistPooling

5. Conclusions

Footnotes

Acknowledgments

References

Table 1
Univariate long sequence time-series forecasting result on four cases

ETT1
¹
ETT dataset was acquired at https://github.com/zhouhaoyi/ETDataset.

Weather2
²
Weather dataset was acquired at http://www.ncei.noaa.gov/data/local-climatological-data/.

Table 4
Ablation study of the LT module