VLinear: Enhanced linear complexity time series forecasting model

Abstract

This study presents a novel time series prediction model(VLinear), designed with linear computational complexity, demonstrating superior predictive performance compared to DLiner without increasing parameter count or computational demands. Our model introduces two key innovations: first, a Feature Pyramid Network (FPN) is employed to effectively capture time series data characteristics, bypassing the traditional decomposition into trend and seasonal components. Second, a multi-layer fusion structure is developed to integrate deep and shallow features seamlessly. Empirically, VLinear outperforms DLiner in 31 out of 32 test cases on eight open-source datasets, with an average reduction of 16.8% in mean squared error (MSE) and 11.8% in mean absolute error (MAE). Additionally, compared to the transformer-based PatchTST, VLinear achieves 10 best MSE and 15 best MAE results, using only 8% of PatchTST’s total computational load in the 32 test projects.

Keywords

Long term time series prediction FPN feature fusion MLP

Highlights

This article proposes an enhanced linear complexity time series prediction model, abbreviated as VLinear.

VLinear uses pooling operation to decompose temporal features, and uses fusion operation to fuse different levels of prediction results to obtain the final prediction result.

VLinear achieved state-of-the-art results in multiple out of 32 tests across 8 datasets.

1. Introduction

Time series forecasting, a critical area in deep learning research, finds extensive applications in finance,¹ weather forecasting,² and sensor data analysis,³ among others. The need for accurate long-term predictions in various domains has driven the development of sophisticated algorithms, including deep learning models like ARIMA,⁴ GBRT, and recursive neural networks,⁵ as well as more recent architectures like Causal Time Networks (TCNs)⁶ and the Mixer.⁷

Transformers, initially designed for NLP tasks, have also made significant strides in time series forecasting, with models like PatchTST,⁸ ETSformer,⁹ Autoformer,¹⁰ and FEDformer¹¹ showcasing their prowess. However, these models often come with high computational complexity and a large number of parameters.TSMixer¹² is designed for multivariate forecasting and representation learning on patched time series ,it use Vision MLP-Mixer for time series and introduce empirically validated components to enhance accuracy.In contrast, DLlinear¹³ offers a linear complexity alternative, decomposing data into trend and seasonal components and predicting them separately, achieving strong performance despite its simplicity.

Data decomposition and normalization are the most commonly used methods to improve the predictive ability of models. The earliest origin of data decomposition is the ARMA,⁴ which decomposes time series data into four parts: trend factor (T), cycle factor (C), seasonal factor (S), and random factor (I), which is more in line with human perception of data. Decomposing the data into trend and seasonal terms is a simplification of these four parts. As shown in Figure 1,Traditional machine learning is based on raw data, extracting features from the raw data and predicting future data, such as GBRT, Informar,¹⁴ DLinear,¹³ Logtrans,¹⁵ DeepAR,¹⁶ LSTMnetwork,¹⁷ etc. There is another data decomposition strategy: to decompose time series data into trend and seasonal items. The PatchTST⁸ and DLlinear¹³ decompose time series data into trend and seasonal terms, extract and predict the two separately, and add them up to output the prediction results. Autoformer¹⁰ and FEDformer¹¹ use feature fusion models to fuse trend and seasonal features, and use the fused features to predict the future. After verification by many scientific researchers, binary decomposition is a very suitable method for deep learning. However, the data binary decomposition method has two problems:

The inherent relationship between trend and seasonal characteristics is overlooked in a crude decomposition. This separation can lead to the loss of trend information within seasonal components and vice versa, as the distinct characteristics are intertwined.

Current approaches, which involve feature extraction for trend and seasonal terms separately, followed by simple addition for fusion, do not effectively account for the correlation between these features. This limits the model’s ability to capture and utilize the complex interactions between trend and seasonality in the data.

To address these limitations, this paper contributes a novel FPN-based multi-layer fusion model(VLinear), inspired by the U-Net architecture, for time series forecasting. Our contributions are three-fold:

Feature Pyramid Network (FPN) for time series analysis: We leverage FPN¹⁸ to describe the time series data by applying multi-layer operations through pooling functions. This method replaces the conventional raw data approach and binary decomposition, offering a more efficient way to extract both shallow (seasonal and trend) and deep features. The use of small pooling kernels in FPN reduces computational complexity while enhancing the extraction of temporal features.As shown in subgraph a of Figure 2, the pyramid feature of time-series data becomes more pronounced with increasing depth.

Multi-layer fusion module:Using FPN to extract features from time series data, as the depth increases, the fluctuation features of the time series data are gradually filtered, and the trend features of the time series become more concentrated As shown in subgraph a of Figure 2, the FPN features of time-series data have two important characteristics: (a) as the depth increases, the fluctuation features gradually disappear and the trend features become more concentrated; (b) the trend characteristics between adjacent layers have significant repeatability, and the difference between the two layers lies in the fluctuation characteristics at a specific scale. Simply merging all features will increase the proportion of trend features, so direct prediction cannot effectively improve prediction accuracy. After conducting multi-layer prediction in FPN, the use of multi-layer fusion modules can effectively gradually fuse the fluctuation and trend features of time series data, forming more accurate prediction results. The multi-layer fusion module can gradually reduce the proportion of trend features, and the fusion parameters of each layer effectively combine trend and fluctuation features. The fluctuation features corresponding to a specific scale in that layer will be effective, improving the prediction accuracy.

VLinear model: Our proposed model integrates an FPN for feature extraction with a multi-layer fusion module. FPN captures deep trend information, while low-level features capture seasonal elements. The fusion mechanism gradually combines these features, resulting in improved performance. Empirically, VLinear outperforms DLiner¹³ with 31 optimal mean absolute error (MAE) and mean squared error (MSE) scores, respectively, achieving an average decrease of 16.8% in MSE and 11.8% in MAE across 32 multivariate forecasting tasks. Against PatchTST,⁸ VLinear exhibits 9 optimal MSE and 15 optimal MAE results.

Figure 1.

The pipeline of existing Model TSF solutions.

Figure 2.

(a) Feature Pyramid Network for time series analysis. (b) FPNLinear algorithm structure. (c) FPNMLinear algorithm structure. (d) VLinear algorithm structure.

In Section Proposed Method, we present our methodology in detail and validate its effectiveness through extensive experiments. To analyze the FPN fusion model’s components, we compare it with FPNLinear, FPNMLinear, NLinear, and DLiner models, using 8 publicly available datasets and 32 tests. Figure 2 illustrates the structural diagrams of the FPN feature pyramid, FPNLinear, FPNMLinear, and VLinear.

2. Related work

2.1. Linear Time Series Models

Linear time series models, characterized by a complexity of $O (L)$ , encompass models such as ARMA,⁴ NLinear, and DLlinear.¹³ DLlinear¹³ and NLlinear¹³ are currently fundamental in long-term forecasting tasks. These models linearly normalize input data and employ fully connected networks. NLlinear employs the nearest value, $x_{L}$ , as a base, and constructs a new historical data point, $x^{'}$ , by subtracting $x - x_{L}$ . This is fed into a fully connected layer, generating a prediction, $y^{'}$ , which is then added to $x_{L}$ to yield the forecasted data, $y$ . DLlinear¹³ decomposes time series into seasonal-term, $x_{s e a s o n}$ , and trend-term, $x_{t r e n d}$ , using data decomposition techniques, and predicts each component separately with fully connected layers, summing the results for the final prediction. The overall complexity of these linear models remains at $O (L)$ .

2.2. Transformer-Based Models

The Transformer’s¹⁹ advancements have inspired its application in long-term time series forecasting. Notable examples include LogTrans,¹⁵ which utilizes convolutional self-attention layers and LogSparse design to capture local information and reduce spatial complexity. Informer¹⁴ introduces ProbSparse self-attention for efficient feature extraction. Autoformer¹⁰ combines time series analysis with Transformers, decomposing data into trend and seasonal components, and introduces autocorrelation. FEDformer¹¹ employs Fourier-enhanced structures for linear complexity, while Yformer²⁰ employs a U-Net-inspired Transformer design with downscaling and upsampling for long-range effects. Pyr-aformer²¹ introduces a pyramid attention module with inter- and intra-scale connections, maintaining linear complexity. PatchTST⁸ applies patch techniques to shorten sequences, reducing complexity and enhancing local features.

2.3. Time Series Decomposition Techniques

A time series is composed of trends, seasonal fluctuations, cyclic variations, and irregular components. Long-term trends represent persistent changes over extended periods, seasonal fluctuations are regular patterns due to seasonal variations, cyclic fluctuations are periodic fluctuations without strict rules, and irregular fluctuations result from random influences. Autoformer¹⁰ initiates the use of seasonal trend decomposition to enhance predictability by inputting trend components derived from the sequence. The residual between the original sequence and trend components is considered seasonal. FEDformer¹¹ extends this by proposing an expert strategy for mixed trend extraction, using moving average kernels with varying sizes. PatchTST⁸ and DLlinear¹³ adopt binary seasonal trend decomposition, isolating seasonal and trend components for separate prediction, then combining them for the final forecast.

3. Proposed Method

The task of time series forecasting involves predicting T future data points based on a historical look-back window of fixed length L. Given input data $x = {x_{1}^{c}, x_{2}^{c}, x_{3}^{c}, \dots, x_{L}^{c}}$ , where each $x_{i}^{c}$ represents a vector dimension of channel $c$ at time $t = i$ , and $c$ denotes the number of channels in the dataset, the model aims to output the forecasted data $y = {x_{L + 1}^{c}, x_{L + 2}^{c}, \dots, x_{L + T}^{c}}$ .

3.1. Feature Pyramid Network of time series

In the realm of image analysis, the Feature Pyramid Network¹⁸ effectively addresses the challenge of detecting objects at large scales by capturing features at diverse scales. Similarly, in the context of time series analysis, decomposing data into trend and seasonal components is a prevalent strategy for enhancing prediction accuracy. Inspired by FPN’s success, we employ pooling operations to extract time series features, creating a feature pyramid. Deep layers contain more trend information, while shallow layers hold more seasonal details.

For instance, when setting the data into four layers (with $s t a g e = 4$ ), we utilize average pooling with a kernel size of $3$ , stride of $2$ , and padding of $0$ . Initially, we define the input data as $x$ . After passing through the FPN module, we obtain new input data $X = [x_{1}, x_{2}, x_{3}, x_{4}]$ , where:

\begin{aligned} x_{1} & = x \\ x_{2} & = A v g P o o l (x_{1}, k e r n e l_s i z e = 3, s t r i d e = 2, p a d d i n g = 0) \\ x_{3} & = A v g P o o l (x_{2}, k e r n e l_s i z e = 3, s t r i d e = 2, p a d d i n g = 0) \\ x_{4} & = A v g P o o l (x_{3}, k e r n e l_s i z e = 3, s t r i d e = 2, p a d d i n g = 0) \end{aligned}

The length of each

x_{i}

is calculated as:

len (x_{i}) = ⌊ \frac{x_{i - 1} + 2 \times p a d d i n g - k e r n e l_s i z e}{s t r i d e} + 1 ⌋

As depicted in Figure 2, the FPN architecture for time series effectively isolates trend features. The trend information in the top layer is more concentrated, while the seasonal aspects are more abundant in the lower layers.

3.2. FPNLinear and FPNMLinear

FPNLinear and FPNMLinear are introduced. FPNLinear concatenates FPN feature layers and employs a fully connected output for length prediction. FPNMLinear, on the other hand, applies fully connected layers to each FPN feature layer, generating scale-specific outputs $Y = [y_{1}, y_{2}, y_{3}, y_{4}]$ , where the length of each $y_{i}$ mirrors that of $x$ extraction using average pooling.

3.3. VLinear model

The multi-stage fusion module takes advantage of the multi-scale time series feature $X$ by concatenating $y_{i}$ and $y_{i - 1}$ , followed by a fully connected layer to produce $y_{i}^{'}$ , maintaining the same length as $y_{i}$ .

The proposed VLinear model, illustrated in Figure 2, consists of two main components: time series FPN and a feature fusion module. It mirrors the structure of the U-Net network,²² combining fully connected and pooling layers. The left side processes time series FPN, forming a descriptive feature set $X = {X_{1}, X_{2}, X_{3}, \dots, X_{S}}$ with $S$ layers. The right side houses the fusion module, which integrates upper-layer features with local ones through fully connected layers, preserving feature length.

3.4. Algorithm Engineering and Evaluation Indicators

The algorithm engineering adopts the PatchTST algorithm, which is an open-source framework on https://github.com/yuqinie98/PatchTSTGitHub . The framework currently includes PatchTST,⁸ DLinear,¹³ FED-former,¹¹ Informer,¹⁴ Autoformer,¹⁰ etc. The framework includes data processing and reading, model training, and algorithm evaluation functions. In order to fairly compare the basic algorithms, both PatchTST⁸ and DLinear¹³ algorithms adopt the best results in the author’s paper. This paper proposes that VLinear, FPNLinear, and FPNMLinear are trained and evaluated using the current framework, with evaluation metrics of MSE and MAE.

4. Experiments

In this study, we assess the efficacy of our proposed VLinear model on eight real-world datasets, including ETT (ETTh1, ETTh2, ETTm1, ETTm2), transportation, electricity, weather, and ILI. These datasets, which are commonly employed for benchmarking and publicly available,¹⁰ are chosen for their extensive use. Notably, we focus on larger datasets such as weather, transportation, ILI, and electricity, as they boast a larger number of time series, which aids in obtaining more stable and less prone to overfitting results. For univariate forecasting, ETT is employed, while multivariate forecasting is conducted on the remaining eight datasets. The statistical details of these datasets are summarized in Table 1.

Table 1.
The statistics of the nine popular datasets for benchmark.

Datasets ETTh1 ETTh2 ETTm1 ETTm2 Traffic Electricity Weather ILI

Variates 7 7 7 7 862 321 211 7

Timessteps 17420 17420 69680 69680 17544 26304 52696 966

Granularity 1hour 1hour 5min 5min 1hour 1hour 10min 11week

Datasets	ETTh1	ETTh2	ETTm1	ETTm2	Traffic	Electricity	Weather	ILI
Variates	7	7	7	7	862	321	211	7
Timessteps	17420	17420	69680	69680	17544	26304	52696	966
Granularity	1hour	1hour	5min	5min	1hour	1hour	10min	11week

4.1. Platform

The platform employed for experimentation was a Windows system, equipped with an Intel Xeon Gold 6226R CPU running at 2.90 GHz (dual-core) and a single Nvidia RTX A4000 graphics processing unit, offering 16 GB of memory. The section below presents the results and analysis of univariate long-term forecasting, using the ETT datasets with prediction lengths ranging from 96 to 720 time steps. The best performance for each metric is highlighted in bold for clarity.

4.2. Compared SOTA methods

To compare our model with state-of-the-art (SOTA) methods, we consider Transformer-based models like PatchTST,⁸ FEDformer,¹¹ Autoformer,¹⁰ and Informer.¹⁴ Additionally, we include the linear model DLiner¹³ as a baseline. All models are tested under consistent experimental conditions, with prediction lengths $T$ varying from 24 to 60 for the ILI dataset and from 96 to 720 for the others, and look-back windows $L$ following the original paper specifications. The baseline results for PatchTST⁸ and DLiner¹³ are obtained using the optimal configurations provided in their respective papers, with the PatchTST results taken from the best-performing configurations (patchTST/64 and patchTST/42).

Univarite Time-series Forecasting: Evaluation Summary

Table 2 summarize the univariate evaluation results of all the methods on ETT datasets, which is the univariate series that we are trying to forecast. We cite the baseline results from PatchTST⁸ and DLinear.¹³ Among the 16 tests on the ETT dataset, overall, VLinear achieved 13 optima in terms of mse and 11 optima in terms of mae.

Compared with DLiner, VLinear achieved a maximum reduction of 54% in mse, an average decrease of 11.0%, a maximum reduction of 35.0% in mae, and an average decrease of 5.5%.

Compared with PatchTST, VLinear has 7 leading items and 6 identical items in the mse indicator, with only 3 items worse than PatchTST. In the mae indicator, VLinear has 10 leading items, 1 identical item, and 4 items worse than PatchTST

On the ETTm1 dataset, VLinear achieved all the best results, with an average improvement of 2.1% in MSE and 0.9% in MAE compared to PatchTST, and an average improvement of 14.4% in MSE and 4.2% in MAE compared to DLinear.On the ETTm2 dataset, VLinear also achieved the best results, with an average improvement of 2.5% in MSE and 2.6% in MAE compared to PatchTST, and an average improvement of 2.2% in MSE and 1.2% in MAE compared to DLinear.

On the ETTh1 and ETTh2 datasets, VLinear has a relative advantage in predicting lengths of 96 and 192. When the prediction length on ETTh1 is 720, VLinear shows a significant improvement in performance compared to DLinear.

Table 2.
Univariate long-term forecasting results. ETT datasets are used with prediction lengths $T \in {96, 192, 336, 720}$ . The best results are in bold.

Models VLinear PatchTST DLinear FEDformer Autoformer Informer LogTrans

Metric MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE

ETTh1 96 0.055 0.178 0.055 0.179 0.056 0.180 0.079 0.215 0.071 0.206 0.193 0.377 0.283 0.468

192 0.071 0.205 0.071 0.205 0.071 0.204 0.104 0.245 0.114 0.262 0.217 0.395 0.234 0.409

336 0.078 0.221 0.076 0.220 0.098 0.244 0.119 0.270 0.107 0.258 0.202 0.381 0.386 0.546

720 0.087 0.233 0.087 0.232 0.189 0.359 0.142 0.299 0.126 0.283 0.183 0.355 0.475 0.629

ETTh2 96 0.124 0.271 0.129 0.282 0.131 0.279 0.128 0.271 0.153 0.306 0.213 0.373 0.217 0.379

192 0.165 0.320 0.168 0.328 0.176 0.329 0.185 0.330 0.204 0.351 0.227 0.387 0.281 0.429

336 0.184 0.348 0.171 0.336 0.209 0.367 0.231 0.378 0.246 0.389 0.242 0.401 0.293 0.437

720 0.227 0.382 0.223 0.380 0.276 0.426 0.278 0.420 0.268 0.409 0.291 0.439 0.218 0.387

ETTm1 96 0.026 0.121 0.026 0.121 0.028 0.123 0.033 0.140 0.056 0.183 0.109 0.277 0.049 0.171

192 0.039 0.149 0.039 0.150 0.045 0.156 0.058 0.186 0.081 0.216 0.151 0.310 0.157 0.317

336 0.051 0.171 0.053 0.173 0.061 0.182 0.084 0.231 0.076 0.218 0.427 0.591 0.289 0.459

720 0.071 0.203 0.073 0.206 0.080 0.210 0.102 0.250 0.110 0.267 0.438 0.586 0.430 0.579

ETTm2 96 0.063 0.181 0.065 0.186 0.063 0.183 0.067 0.198 0.065 0.189 0.088 0.225 0.075 0.208

192 0.090 0.224 0.094 0.231 0.092 0.227 0.102 0.245 0.118 0.256 0.132 0.283 0.129 0.275

336 0.116 0.256 0.120 0.265 0.119 0.261 0.130 0.279 0.154 0.305 0.180 0.336 0.154 0.302

720 0.170 0.318 0.171 0.322 0.175 0.320 0.178 0.325 0.182 0.335 0.300 0.435 0.160 0.321

Count 13 11 9 4 2 1 0 0 0 0 0 0 0 0

Models	VLinear	PatchTST	DLinear	FEDformer	Autoformer	Informer	LogTrans
ETTh1	96	0.055	0.178	0.055	0.179	0.056	0.180	0.079	0.215	0.071	0.206	0.193	0.377	0.283	0.468
	192	0.071	0.205	0.071	0.205	0.071	0.204	0.104	0.245	0.114	0.262	0.217	0.395	0.234	0.409
	336	0.078	0.221	0.076	0.220	0.098	0.244	0.119	0.270	0.107	0.258	0.202	0.381	0.386	0.546
	720	0.087	0.233	0.087	0.232	0.189	0.359	0.142	0.299	0.126	0.283	0.183	0.355	0.475	0.629
ETTh2	96	0.124	0.271	0.129	0.282	0.131	0.279	0.128	0.271	0.153	0.306	0.213	0.373	0.217	0.379
	192	0.165	0.320	0.168	0.328	0.176	0.329	0.185	0.330	0.204	0.351	0.227	0.387	0.281	0.429
	336	0.184	0.348	0.171	0.336	0.209	0.367	0.231	0.378	0.246	0.389	0.242	0.401	0.293	0.437
	720	0.227	0.382	0.223	0.380	0.276	0.426	0.278	0.420	0.268	0.409	0.291	0.439	0.218	0.387
ETTm1	96	0.026	0.121	0.026	0.121	0.028	0.123	0.033	0.140	0.056	0.183	0.109	0.277	0.049	0.171
	192	0.039	0.149	0.039	0.150	0.045	0.156	0.058	0.186	0.081	0.216	0.151	0.310	0.157	0.317
	336	0.051	0.171	0.053	0.173	0.061	0.182	0.084	0.231	0.076	0.218	0.427	0.591	0.289	0.459
	720	0.071	0.203	0.073	0.206	0.080	0.210	0.102	0.250	0.110	0.267	0.438	0.586	0.430	0.579
ETTm2	96	0.063	0.181	0.065	0.186	0.063	0.183	0.067	0.198	0.065	0.189	0.088	0.225	0.075	0.208
	192	0.090	0.224	0.094	0.231	0.092	0.227	0.102	0.245	0.118	0.256	0.132	0.283	0.129	0.275
	336	0.116	0.256	0.120	0.265	0.119	0.261	0.130	0.279	0.154	0.305	0.180	0.336	0.154	0.302
	720	0.170	0.318	0.171	0.322	0.175	0.320	0.178	0.325	0.182	0.335	0.300	0.435	0.160	0.321
Count	13	11	9	4	2	1	0	0	0	0	0	0	0	0

Through the detailed analysis above, it has been verified that the feature extraction method of VLinear using FPN has advantages over the DLinear binary method. VLinear has stronger adaptability at different time granularity levels, and at time granularity levels of minutes, FPN’s feature extraction and multi-level fusion method have stronger feature extraction and adaptability compared to the traditional binary method.

Multivariate Time-series Forecasting: Evaluation Summary

Table 3 presents the comprehensive evaluation results of various methods on eight datasets. The VLinear model demonstrated outstanding performance, with a total of 10 best MSE scores and 15 best MAE results across the 32 tests. Compared to DLinear, VLinear exhibited an average improvement of 16.8% in MSE and 11.8% in MAE.

On the weather dataset, which exhibits strong periodicity, VLinear and patchTST achieved superior performance, with a 1.0% decrease in MSE and a 0.4% decrease in MAE compared to DLinear. Notably, these methods outperformed DLinear by 12.6% and 15.1% in terms of MAE.

On traffic and electricity datasets, VLinear demonstrated a consistent improvement of approximately 2% in MSE and 3.3% in MAE compared to DLinear. The ILI dataset showcased VLinear’s prowess, with the best results for a prediction horizon of 60. Compared to PatchTST, VLinear achieved a 7.4% decrease in MSE and a 6.4% decrease in MAE. Against DLinear, the improvements were even more substantial, with a 66.6% decrease in MSE and a 46.7% decrease in MAE.

In the ETT dataset, VLinear exhibited a strong performance, with 11 optimal MAE results and 4 optimal MSE outcomes. Notably, on the test instance with a prediction length of 192 on ETTh1, VLinear exhibited a 3.4% decrease in MSE and a 3.5% decrease in MAE compared to PatchTST.

Table 3.

Multivariate long-term forecasting results. We use prediction lengths $T \in {24, 36, 48, 60}$ for ILI dataset and $T \in {96, 192, 336, 720}$ for the others. The best results are in bold and the second best are underlined.

Models		VLinear		PatchTST		DLinear		FEDformer		Autoformer		Informer		Pyraformer
Metric		MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Weather	96	0.143	0.208	0.147	0.198	0.176	0.237	0.238	0.314	0.249	0.329	0.354	0.405	0.896	0.556
	192	0.187	0.234	0.190	0.240	0.220	0.282	0.275	0.329	0.325	0.370	0.419	0.434	0.622	0.624
	336	0.238	0.274	0.242	0.282	0.265	0.319	0.339	0.377	0.351	0.391	0.583	0.543	0.739	0.753
	720	0.304	0.325	0.304	0.328	0.323	0.362	0.389	0.409	0.415	0.426	0.916	0.705	1.004	0.934
Traffic	96	0.408	0.273	0.360	0.249	0.410	0.282	0.576	0.359	0.597	0.371	0.733	0.410	2.085	0.468
	192	0.419	0.286	0.379	0.256	0.423	0.287	0.610	0.380	0.607	0.382	0.777	0.435	0.867	0.467
	336	0.422	0.285	0.392	0.264	0.436	0.296	0.608	0.375	0.623	0.387	0.776	0.434	0.869	0.469
	720	0.443	0.299	0.432	0.286	0.466	0.315	0.621	0.375	0.639	0.395	0.827	0.466	0.881	0.473
Electricity	96	0.132	0.229	0.129	0.222	0.140	0.237	0.186	0.302	0.196	0.313	0.304	0.393	0.386	0.449
	192	0.146	0.244	0.141	0.241	0.153	0.249	0.197	0.311	0.211	0.324	0.327	0.417	0.386	0.443
	336	0.162	0.262	0.163	0.259	0.169	0.267	0.213	0.328	0.214	0.327	0.333	0.422	0.378	0.443
	720	0.200	0.297	0.197	0.290	0.203	0.301	0.233	0.344	0.236	0.342	0.351	0.427	0.376	0.445
ILI	24	1.696	0.789	1.281	0.704	2.215	1.081	2.624	1.095	2.906	1.182	4.657	1.449	1.420	2.012
	36	1.693	0.811	1.251	0.752	1.963	0.963	2.516	1.021	2.585	1.038	4.650	1.463	7.394	2.031
	48	1.867	0.881	1.673	0.854	2.130	1.024	2.505	1.041	3.024	1.145	5.004	1.542	7.551	2.057
	60	1.421	0.747	1.526	0.795	2.368	1.096	2.742	1.122	2.761	1.114	5.071	1.543	7.662	2.100
ETTh1	96	0.368	0.394	0.370	0.400	0.375	0.399	0.376	0.415	0.435	0.446	0.941	0.769	0.664	0.612
	192	0.406	0.417	0.413	0.431	0.405	0.416	0.423	0.446	0.456	0.457	1.007	0.786	0.790	0.681
	336	0.408	0.425	0.422	0.440	0.439	0.443	0.444	0.462	0.486	0.487	1.038	0.784	0.891	0.738
	720	0.458	0.462	0.447	0.468	0.472	0.490	0.469	0.492	0.515	0.517	1.144	0.857	0.963	0.782
ETTh2	96	0.279	0.333	0.274	0.336	0.289	0.353	0.332	0.374	0.332	0.368	1.549	0.952	0.645	0.597
	192	0.343	0.395	0.339	0.379	0.383	0.418	0.407	0.446	0.426	0.434	3.792	1.542	0.788	0.683
	336	0.379	0.423	0.331	0.380	0.448	0.465	0.400	0.447	0.477	0.479	4.215	1.642	0.907	0.747
	720	0.446	0.464	0.379	0.422	0.605	0.551	0.412	0.469	0.453	0.490	3.656	1.619	0.963	0.783
ETTm1	96	0.286	0.335	0.290	0.342	0.299	0.343	0.326	0.390	0.510	0.492	0.626	0.560	0.543	0.510
	192	0.330	0.359	0.328	0.365	0.335	0.365	0.365	0.415	0.514	0.495	0.725	0.619	0.557	0.537
	336	0.368	0.380	0.361	0.393	0.369	0.386	0.392	0.425	0.510	0.492	1.005	0.741	0.754	0.655
	720	0.425	0.413	0.416	0.419	0.425	0.421	0.446	0.458	0.527	0.493	1.133	0.845	0.908	0.724
ETTm2	96	0.163	0.250	0.162	0.254	0.167	0.260	0.180	0.271	0.205	0.293	0.355	0.462	0.435	0.507
	192	0.216	0.287	0.216	0.293	0.224	0.303	0.252	0.318	0.278	0.336	0.595	0.586	0.730	0.673
	336	0.271	0.324	0.269	0.329	0.281	0.342	0.324	0.364	0.343	0.379	1.270	0.871	1.201	0.845
	720	0.360	0.389	0.350	0.380	0.397	0.421	0.410	0.420	0.414	0.419	3.001	1.267	3.625	1.451
Count		10	15	20	17	1	1	0	0	0	0	0	0	0	0

Through the comparative analysis of multivariate prediction above, in the results of multivariate prediction, VLinear has advantages in some scenarios. Compared with univariate prediction analysis, VLinear’s advantages become weaker, but compared with DLinear, it has a considerable improvement. In complex multivariate scenarios, VLinear has a significant advantage when predicting a length of 60 in the ILI dataset, indicating that VLinear has strong advantages in certain characteristic industrial scenarios and can become one of the basic algorithms for time series prediction.

4.3. Ablation Study

To evaluate the FPN module and fusion mechanism in our VLinear algorithm, we conduct an ablation study by comparing Linear, FPNLinear, FPNMLinear, VLinear, NLinear, and DLiner models. The performance is averaged over five folds for all evaluation metrics.

Table 4 presents the comparative results of these models, with VLinear achieving optimal performance in 32 tests, attaining 25 minimum MSE and 24 minimum MAE instances. By comparing DLinear and FPNLinear, we can conclude that FPN is superior to traditional trend and fluctuation binary methods in feature extraction. By comparing FPNMLinear and FPNLinear algorithms, for shorter prediction lengths, fully connected processing can reduce some feature redundancy and improve MSE and MAE significantly. As the prediction length increases, the effect gradually decreases or even deteriorates. By comparing VLinear, FPNLinear, and FPNMLienar models, highlighting its effectiveness in mitigating duplicate trend features and enhancing the representation of fine-grained information.

Table 4.
Ablation study:multivariate long-term forecasting results. The best results are in bold.

Models VLinear FPNMLinear FPNLinear DLinear NLinear

Metric MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE

Weather 96 0.143 0.208 0.143 0.208 0.145 0.209 0.176 0.237 0.182 0.232

192 0.187 0.234 0.187 0.252 0.190 0.258 0.220 0.282 0.225 0.269

336 0.238 0.274 0.240 0.297 0.242 0.299 0.265 0.319 0.271 0.301

720 0.304 0.325 0.316 0.359 0.316 0.358 0.323 0.362 0.338 0.348

Traffic 96 0.408 0.273 0.411 0.282 0.409 0.282 0.410 0.282 0.410 0.279

192 0.419 0.286 0.420 0.294 0.421 0.295 0.423 0.287 0.423 0.284

336 0.422 0.285 0.436 0.295 0.436 0.292 0.436 0.296 0.435 0.290

720 0.443 0.299 0.458 0.303 0.460 0.312 0.466 0.315 0.464 0.307

Electricity 96 0.132 0.229 0.133 0.230 0.133 0.230 0.140 0.237 0.141 0.237

192 0.146 0.244 0.148 0.244 0.148 0.245 0.153 0.249 0.154 0.248

336 0.162 0.262 0.163 0.262 0.163 0.263 0.169 0.267 0.171 0.265

720 0.200 0.297 0.200 0.297 0.200 0.297 0.203 0.301 0.210 0.297

ILI 24 1.696 0.789 2.097 0.968 2.115 0.951 2.215 1.081 1.683 0.858

36 1.693 0.811 2.108 0.977 2.141 0.982 1.963 0.963 1.703 0.859

48 1.867 0.881 2.246 1.025 2.247 1.036 2.130 1.024 1.719 0.884

60 1.421 0.747 2.198 1.023 2.170 1.013 2.368 1.096 1.819 0.917

ETTh1 96 0.368 0.394 0.373 0.398 0.380 0.392 0.375 0.399 0.374 0.394

192 0.406 0.417 0.406 0.415 0.406 0.421 0.405 0.416 0.408 0.415

336 0.408 0.425 0.427 0.435 0.454 0.445 0.439 0.443 0.429 0.427

720 0.458 0.462 0.471 0.497 0.473 0.483 0.472 0.490 0.440 0.453

ETTh2 96 0.279 0.333 0.290 0.341 0.289 0.345 0.289 0.353 0.277 0.338

192 0.343 0.395 0.385 0.416 0.382 0.419 0.383 0.418 0.344 0.381

336 0.379 0.423 0.453 0.478 0.461 0.483 0.448 0.465 0.357 0.400

720 0.446 0.464 0.412 0.469 0.453 0.490 0.605 0.551 0.394 0.436

ETTm1 96 0.286 0.335 0.287 0.335 0.287 0.335 0.299 0.343 0.306 0.348

192 0.327 0.359 0.328 0.359 0.328 0.359 0.335 0.365 0.349 0.375

336 0.368 0.380 0.369 0.384 0.383 0.398 0.369 0.386 0.375 0.388

720 0.425 0.415 0.425 0.415 0.425 0.416 0.425 0.421 0.433 0.422

ETTm2 96 0.163 0.250 0.168 0.262 0.173 0.264 0.167 0.260 0.167 0.255

192 0.216 0.287 0.236 0.326 0.225 0.308 0.224 0.303 0.221 0.293

336 0.271 0.324 0.283 0.364 0.303 0.357 0.281 0.342 0.274 0.327

720 0.361 0.386 0.392 0.410 0.363 0.399 0.397 0.421 0.368 0.384

Avg 0.480 0.390 0.550 0.426 0.554 0.426 0.562 0.437 0.496 0.403

Count 25 24 4 7 2 3 2 0 6 8

Improvement — 14.6% 9.0% 15.3% 9.2% 16.8% 11.8% 3.1% 3.4%

Models	VLinear	FPNMLinear	FPNLinear	DLinear	NLinear
Weather	96	0.143	0.208	0.143	0.208	0.145	0.209	0.176	0.237	0.182	0.232
	192	0.187	0.234	0.187	0.252	0.190	0.258	0.220	0.282	0.225	0.269
	336	0.238	0.274	0.240	0.297	0.242	0.299	0.265	0.319	0.271	0.301
	720	0.304	0.325	0.316	0.359	0.316	0.358	0.323	0.362	0.338	0.348
Traffic	96	0.408	0.273	0.411	0.282	0.409	0.282	0.410	0.282	0.410	0.279
	192	0.419	0.286	0.420	0.294	0.421	0.295	0.423	0.287	0.423	0.284
	336	0.422	0.285	0.436	0.295	0.436	0.292	0.436	0.296	0.435	0.290
	720	0.443	0.299	0.458	0.303	0.460	0.312	0.466	0.315	0.464	0.307
Electricity	96	0.132	0.229	0.133	0.230	0.133	0.230	0.140	0.237	0.141	0.237
	192	0.146	0.244	0.148	0.244	0.148	0.245	0.153	0.249	0.154	0.248
	336	0.162	0.262	0.163	0.262	0.163	0.263	0.169	0.267	0.171	0.265
	720	0.200	0.297	0.200	0.297	0.200	0.297	0.203	0.301	0.210	0.297
ILI	24	1.696	0.789	2.097	0.968	2.115	0.951	2.215	1.081	1.683	0.858
	36	1.693	0.811	2.108	0.977	2.141	0.982	1.963	0.963	1.703	0.859
	48	1.867	0.881	2.246	1.025	2.247	1.036	2.130	1.024	1.719	0.884
	60	1.421	0.747	2.198	1.023	2.170	1.013	2.368	1.096	1.819	0.917
ETTh1	96	0.368	0.394	0.373	0.398	0.380	0.392	0.375	0.399	0.374	0.394
	192	0.406	0.417	0.406	0.415	0.406	0.421	0.405	0.416	0.408	0.415
	336	0.408	0.425	0.427	0.435	0.454	0.445	0.439	0.443	0.429	0.427
	720	0.458	0.462	0.471	0.497	0.473	0.483	0.472	0.490	0.440	0.453
ETTh2	96	0.279	0.333	0.290	0.341	0.289	0.345	0.289	0.353	0.277	0.338
	192	0.343	0.395	0.385	0.416	0.382	0.419	0.383	0.418	0.344	0.381
	336	0.379	0.423	0.453	0.478	0.461	0.483	0.448	0.465	0.357	0.400
	720	0.446	0.464	0.412	0.469	0.453	0.490	0.605	0.551	0.394	0.436
ETTm1	96	0.286	0.335	0.287	0.335	0.287	0.335	0.299	0.343	0.306	0.348
	192	0.327	0.359	0.328	0.359	0.328	0.359	0.335	0.365	0.349	0.375
	336	0.368	0.380	0.369	0.384	0.383	0.398	0.369	0.386	0.375	0.388
	720	0.425	0.415	0.425	0.415	0.425	0.416	0.425	0.421	0.433	0.422
ETTm2	96	0.163	0.250	0.168	0.262	0.173	0.264	0.167	0.260	0.167	0.255
	192	0.216	0.287	0.236	0.326	0.225	0.308	0.224	0.303	0.221	0.293
	336	0.271	0.324	0.283	0.364	0.303	0.357	0.281	0.342	0.274	0.327
	720	0.361	0.386	0.392	0.410	0.363	0.399	0.397	0.421	0.368	0.384
Avg	0.480	0.390	0.550	0.426	0.554	0.426	0.562	0.437	0.496	0.403
Count	25	24	4	7	2	3	2	0	6	8
Improvement	—	14.6%	9.0%	15.3%	9.2%	16.8%	11.8%	3.1%	3.4%

4.4. Model Efficiency

we analyze the efficiency of VLinear by examining computational complexity, model parameters, and GPU memory usage, comparing it with reference models such as DLiner, PatchTST, Autoformer, and Informer. This evaluation provides insights into the trade-offs between model performance and computational efficiency.

Table 5 presents a detailed comparison of model efficiency, focusing on parameters, computational complexity (measured by Multiply-Accumulate Operations, or MACS), and GPU memory usage. The experiments were conducted with a fixed look-back window of 336, a prediction length of 96, and a batch size of 32 on the ETTh2 dataset. The results are averaged over five runs to ensure fairness in the comparison.

VLinear and DLinear exhibit comparable computational efficiency, with VLinear demonstrating a slight advantage. PatchTST, which relies on patch-based methods, has a significantly higher computational cost, approximately 12.1 times that of VLinear. By employing a smaller pooling kernel, VLinear reduces computational requirements by 6.6%, further enhancing its efficiency.

In terms of model size and memory footprint, VLinear is on par with DLinear and PatchTST. Given its superior performance and resource efficiency, VLinear is a more practical choice for real-world applications, especially in univariate and multivariate time series forecasting tasks where resource constraints are prevalent.

Table 5.
Compare the parameter quantity and resource consumption of the model under the look-back window $L = 336$ , prediction length $T = 96$ and $b a t c h_s i z e = 32$ on the ETTh2. MACS are the number of multiply-accumulate operations.

Method MACs Parameter Memory

VLinear 13.56M 0.42M 20.39M

DLinear 14.52M 0.45M 20.55M

PatchTST 164.97M 0.46M 20.72M

Autoformer 90517.61M 10.54M 119.39M

Informer 79438.08M 11.33M 126.22M

Method	MACs	Parameter	Memory
VLinear	13.56M	0.42M	20.39M
DLinear	14.52M	0.45M	20.55M
PatchTST	164.97M	0.46M	20.72M
Autoformer	90517.61M	10.54M	119.39M
Informer	79438.08M	11.33M	126.22M

The current model effectiveness analysis only focuses on one testing scenario. From the perspective of the spatiotemporal complexity of the model, Autoformer and Informer are both $O (N^{2})$ ,Without pre-processing, $N$ will have the same value as input sequence length $L$ VLinear and DLinear are both $O (L)$ , and the spatiotemporal complexity of PatchTST is $O (L_{P}^{2} \times n)$ ,where $L_{P}$ represents the length of the patch and $n$ represents the number of patches. Combined with performance testing experiments, it can be guaranteed that VLinear has excellent inference performance.

4.5. Results and Discussion

The results of multi-dimensional long-term time series prediction experiments, single dimensional long-term time series prediction experiments, and ablation experiments show that VLlinear outperforms competitive methods such as DLinear in prediction accuracy while maintaining linear computational complexity. The improvement in its performance is attributed to FPN feature extraction and effective multi-layer fusion, which allows for better processing of complex patterns in data, making it one of the new choices for industrial applications. In the future, the structure of FPN for extracting temporal features can be combined with large models or attention mechanisms to further improve the accuracy of temporal prediction.

5. Conclusion and Future Work

In this article, we introduce the VLlinear model, a groundbreaking approach that can create a universal and robust foundation model for various time series tasks in real-world scenarios. By combining FPN with multi-layer fusion, FPN effectively captures the trend and fluctuation characteristics of time series data, and adds fluctuation features to the trend through multi-layer fusion structure. A large number of experiments have demonstrated the powerful performance of VLlinear in 8 standard benchmark tests, providing a new option for the field of time series prediction engineering. However, we recognize some limitations. We mainly study long-term sequences and have not evaluated the performance of VLinears on very long and very short sequences. At the same time, we did not explore the combination of features extracted by FPN with attention mechanisms. For future work, we have envisioned several key directions: first, combining the temporal features extracted by FPN with large models, utilizing the capabilities of large models to improve prediction accuracy; second, combining the design ideas of VLinear models with specific industry algorithms, proposing algorithms for specific industrial scenarios.

Footnotes

ORCID iD

Chu Li

Competing interests

All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript.

Authors’ contribution

Chu Li: Data curation, Formal analysis, Methodology, Software, Validation, Writing – original draft. Bingjia Xiao, Qiping Yuan: Conceptualization, Methodology, Resources, Supervision, Writing – review & editing.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work is supported by the National MCF Energy R&D program of China, Grant No. 2018YFE0302100.

Data availability and access

All datasets are publicly available, You can obtain all the datasets from github-data.

Code availability

The source code of this paper is publicly available via github-code.

References

Zhao

Guo

, et al. Fuzzy hypergraph network for recommending top-k profitable stocks. Inf Sci (Ny) 2022; 613: 239–255.

Liu

, et al. A stock price prediction method based on meta-learning and variational mode decomposition. Knowl Based Syst 2022; 252: 109324.

Zhao

, et al. Mpr-net:multi-scale pattern reproduction guided universality time series interpretable forecasting, 2023.

Helfenstein

. Box-jenkins modelling in medical research. Statistical Methods in Medical Research, 1996; 5: 3.

Rangapuram

Seeger

Gasthaus

, et al. Deep state space models for time series forecasting. Neural Information Processing Systems 2018: 7796–7805.

Bai

Kolter

Koltun

. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018.

Tolstikhin

Houlsby

Kolesnikov

, et al. Mlp-mixer: An all-mlp architecture for vision. Adv Neural Inf Process Syst 2021; 34: 24261–24272.

Nie

Nguyen

Sinthong

, et al. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations, 2023.

Woo

Liu

Sahoo

, et al. Xponential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.01381v, 2022.

10.

Wang

, et al. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Adv Neural Inf Process Syst 2021; 34: 22419–22430.

11.

Zhou

Wen

, et al. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning, pp.27268–27286. PMLR, 2022.

12.

Vijay

Jati

Nguyen

, et al. Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting. arXiv preprint arXiv:2306.09364, 2024.

13.

Zeng

Chen

Zhang

, et al. Are transformers effective for time series forecasting? In: Proceedings of the AAAI conference on artificial intelligence, 2023, volume 37, pp.11121–11128.

14.

Zhou

Zhang

Peng

, et al. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, 2021, volume 35, pp.11106–11115.

15.

Jin

Xuan

, et al. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Proceedings of the 33rd International Conference on Neural Information Processing Systems 2019: 5243–5253.

16.

Salinas

Flunkert

Gasthaus

, et al. Deepar: Probabilistic forecasting with autoregressive recurrent networks. Int J Forecast 2020; 36: 1181–1191.

17.

Zhao

Chen

, et al. Lstm network: a deep learning approach for short-term traffic forecast. IET Intell Trans Syst 2017; 11: 68–75.

18.

Lin

Dollár

Girshick

, et al. Feature pyramid networks for object detection. 2016.

19.

Devlin

Chang

M-W

Lee

, et al. Bert: Pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018.

20.

Madhusudhanan

Burchert

Duong-Trung

, et al. Yformer: U-net inspired transformer architecture for far horizon time series forecasting, 2022.

21.

Liu

Liao

, et al. Pyraformer: low-complexity pyramidal attention for long-range time series modeling and forecasting. In: International conference on learning representations, 2021.

22.

Ronneberger

Fischer

Brox

. U-net: convolutional networks for biomedical image segmentation. Medical image computing and computer-assisted Intervention–MICCAI 2015: 18th international conference, 2015, pp.234–241.

VLinear: Enhanced linear complexity time series forecasting model

Abstract

Keywords

Highlights

1. Introduction

2.1. Linear Time Series Models

2.2. Transformer-Based Models

2.3. Time Series Decomposition Techniques

3. Proposed Method

3.1. Feature Pyramid Network of time series

3.2. FPNLinear and FPNMLinear

3.3. VLinear model

3.4. Algorithm Engineering and Evaluation Indicators

4. Experiments

Table 1. The statistics of the nine popular datasets for benchmark. Datasets ETTh1 ETTh2 ETTm1 ETTm2 Traffic Electricity Weather ILI Variates 7 7 7 7 862 321 211 7 Timessteps 17420 17420 69680 69680 17544 26304 52696 966 Granularity 1hour 1hour 5min 5min 1hour 1hour 10min 11week

4.2. Compared SOTA methods

5. Conclusion and Future Work

Footnotes

ORCID iD

Competing interests

Authors’ contribution

Declaration of conflicting interests

Funding

Data availability and access

Code availability

References

Table 1.
The statistics of the nine popular datasets for benchmark.

Datasets ETTh1 ETTh2 ETTm1 ETTm2 Traffic Electricity Weather ILI

Variates 7 7 7 7 862 321 211 7

Timessteps 17420 17420 69680 69680 17544 26304 52696 966

Granularity 1hour 1hour 5min 5min 1hour 1hour 10min 11week