Multi-task prediction model based on ConvLSTM and encoder-decoder

Abstract

The energy load data in the micro-energy network are a time series with sequential and nonlinear characteristics. This paper proposes a model based on the encode-decode architecture and ConvLSTM for multi-scale prediction of multi-energy loads in the micro-energy network. We apply ConvLSTM, LSTM, attention mechanism and multi-task learning concepts to construct a model specifically for processing the energy load forecasting of the micro-energy network. In this paper, ConvLSTM is used to encode the input time series. The attention mechanism is used to assign different weights to the features, which are subsequently decoded by the decoder LSTM layer. Finally, the fully connected layer interprets the output. This model is applied to forecast the multi-energy load data of the micro-energy network in a certain area of Northwest China. The test results prove that our model is convergent, and the evaluation index value of the model is better than that of the multi-task FC-LSTM and the single-task FC-LSTM. In particular, the application of the attention mechanism makes the model converge faster and with higher precision.

Keywords

Encoder-decoder architecture CNN ConvLSMT LSTM deep learning attention mechanism multi-task learning multi-step prediction load forecasting micro-energy network multi-time scale

1. Introduction

The multi-energy system not only improves the utilization rate of renewable energy and reduces environmental pollution but also improves the flexibility of the energy system. The micro-energy network is a form of multi-energy system application. The micro-energy network integrates multiple energy inputs, outputs of multiple products (including electricity, gas, etc.) and multiple energy conversion units, and thus the uncertainty and time-varying energy in the micro-energy network are notably strong. With deepening of the coupling between various forms of energy such as electricity and gas in the micro-energy network system and access to large-scale renewable energy, the accuracy and real-time performance of energy load forecasting have higher requirements. The micro-energy network is an effective way to use renewable energy. As the proportion of renewable energy in the energy system increases, coordinated control of the energy supply and load within the micro-energy network is expected to become an important research field.

In recent years, deep learning has been widely applied in computer vision (CV) [21] and natural language processing (NLP) [6, 7], e.g., object detection [16], image segmentation [12], speech recognition [23], text classification [25], and machine translation [1, 31]. Energy load forecasting is characterized by time series and non-linearity, and thus two main types of energy load forecasting exist. One type is time series analysis methods, such as multiple linear regression models and autoregressive integrated moving average models (ARIMA). The basic idea is of these models is to extrapolate future data from time series historical data and current data, and their common advantage is that they consider the temporal relationship of data, and thus the model is simple and widely used. The disadvantage of multiple linear regression is an insufficient ability to predict nonlinear relational data. The ARIMA is inherently unable to capture nonlinear relationships and requires a stationary time series for input. The other type of method is shallow learning methods based on machine learning, such as support vector machine (SVM) and shallow neural networks. The main advantage of shallow machine learning models is their strong nonlinear fitting ability, and the neural network method also has self-learning capability. A common shortcoming of shallow learning methods is the lack of correlation considerations for time series data.

The long-short term memory (LSTM) [14] is a recurrent neural networks (RNNs) method with memory characteristics that solves long-term dependence defects due to gradient disappearance. Because the LSTM network can effectively solve the time dependence problem of temporal data [6], it is widely used in NLP [7] and time series prediction [9]. Dr. Shi Xing applied the convolution operation to the LSTM network and proposed the ConvLSTM structure [19]. This neuron structure can solve the problem of the full-connection long-term memory network (FC-LSTM) containing a large amount of redundant spatial data without considering spatial correlation when addressing spatiotemporal sequence problems.

Based on analysis and understanding of the existing results, this paper proposes an encoder-decoder architecture model based on the ConvLSTM, LSTM, attention mechanism and multi-task learning methods. The experimental results show that the solution of each task in the model is beneficial to the solution of each energy load forecasting task in the micro-energy network, and the use of the attention mechanism significantly improves the prediction accuracy and reduces the training time.

Traditional time series analysis methods require stable time series and have an insufficient ability to capture non-linear relationships. A lack of non-linear fitting capabilities exists in shallow machine learning methods, such as SVM regression methods, which also lack consideration of the time correlation of time series.

In previous papers [39, 40], the author used the encoder-decoder structure of CNN-LSTM to encode and decode the input time series, but CNN as an encoder does not have the ability to model time series. This paper uses ConvLSTM as an encoder, which has the ability to describe local features, similar to CNN, and also has the ability to model time series in a manner similar to LSTM. ConvLSTM is a neuron with time and space characteristics. In another paper [41], the author used multi-task regression to solve the problem of power demand forecasting. Although the author considered the power demand problem under the influence of multiple factors, the linear regression model was essentially insufficient in its ability to fit nonlinear problems. The idea proposed in this paper not only considers the energy demand problem under the influence of various factors but also applies the encoder-decoder architecture with ConvLSTM as the encoder. The use of ConvLSTM not only considers the time dependence of the sequence but also considers the interrelationships between the input data and fully explores the potential relationships between the data. Another highlight of this paper is that an attention mechanism is applied between the encoder and decoder. The introduction of the attention mechanism allows the model to give higher weight to the corresponding features and allows the model to learn from the data to a deeper level.

The structure of this paper is described as follows. First, we introduce the basic principles of the LSTM, ConvLSTM, attention mechanism, encoder-decoder architecture and multi-task learning. Second, we establish a model based on the encoder-decoder architecture and give a standard for evaluating the accuracy of the model followed by analysis of the basic model structure. Finally, the feasibility of our model is verified by comparing the multi-tasking model and the single task prediction model using LSTM and CNN as encoders.

Overall, the main contributions of this paper are listed as follows:

•
A hybrid model based on ConvLSTM is proposed for multi-time scale prediction of multi-energy in multi-energy networks;
•
The experimental results show that the model designed in this paper performs better in the multi-energy load prediction experiment on the micro-energy network in Northwest China than the model using CNN and LSTM as the encoder.

2. Related work

2.1 Data standardization

The data used in this paper have a total of ten dimensions: time information, historical data of two variables that must be forecasted, and environmental data. The environmental data include temperature, air pressure, wind speed, wind direction, humidity, cumulative hourly snow volume, and cumulative hourly rainfall volume. In dataset, the missing value and noise information of the data in this paper does not exceed 5%, and the outlier value does not exceed 1%. For missing values, this paper uses the mean of their features to fill the gaps. The wind direction feature is not a digital feature, and thus the wind direction feature is encoded in this paper. The wind direction data are encoded as values from 0 to 7.

Finally, to eliminate the influence of the feature data dimension and accelerate the search for the optimal solution, it is necessary to standardize the data. This paper uses min-max scaling to map the data between 0 and 1. The calculation formula is written as follows:

$\displaystyle x^{\prime}=\frac{1}{x_{\max}-x_{\min}}({x-x_{\min}})$ (1)

The model prediction results are inversely transformed to the normal range, and the transformation is shown below:

$\displaystyle x={x}^{\prime}(x_{\max}-x_{\min})+x_{\min}$ (2)

Figure 1.

Constructed 3D input data.

2.2 Construction data

We define the micro-energy network multi-energy load time series data in the form of $X=\{x_{1},x_{2},\ldots,x_{T}\}$ , where each $x$ is a vector of dimension $1\times m$ ; and $X$ represents a matrix of $\mathbb{R}^{n\times m}$ dimensions, where $n$ represents the number of $x$ in the time series and $m$ represents the number of input features. The data of the previous $t_{1}$ time steps are used to predict the data of the subsequent $t_{2}$ time steps. The sliding window method is used to slide one time step at a time, and the input data are constructed in the form of $n\times t\times m$ , where $n$ is the number of times the sliding window slides, $m$ is the number of feature quantities, and $t$ is the width of the sliding window (Fig. 1).

We treat each $t\times m$ data as a feature map of $1\times t\times m$ , where $t$ is the length of the time series of the input model, and $m$ is the number of feature quantities of the time series and represents the number of channels in the picture. In the feature map of the time series, the data are considered to be a picture with $m$ channels and length and width of $1\times t$ . The characteristic picture is shown in Fig. 2.

Figure 2.

Input feature map.

3. Background

3.1 LSTM

In the deep RNNs, due to the gradient disappearance problem, it is only theoretically possible to establish a long-term dependency between states [29]. The LSTM [14] is an improved RNN structure that introduces a gating mechanism to control the information transfer path. This mechanism can effectively solve the gradient disappearance problem of RNNs. The three doors of the LSTM are the input gate, forgetting gate and output gate. Multiple LSTM neurons can be stacked and connected in time to form a more complex structure. The LSTM structure is shown in Fig. 3.

Figure 3.

LSTM structure.

Where, $x_{t}$ is the input data, $h_{t-1}$ and $c_{t-1}$ are the parameters passed by the previous LSTM, $i_{t},f_{t},o_{t}$ is the input gate, the forgetting gate and the output gate. The calculation formula of each parameter is as follows:

$\displaystyle i_{t}=\sigma(W_{x}^{i}\cdot x_{t}+W_{h}^{i}h_{h-1}+W_{c}^{i}% \circ c_{t-1}+b_{i})$ $\displaystyle f_{t}=\sigma(W_{x}^{f}x_{t}+W_{h}^{f}h_{t-1}+W_{c}^{f}\circ c_{t% -1}+b$ $\displaystyle c_{t}=f_{t}\circ c_{t-1}+i_{t}\circ\text{tanh}(W_{x}^{c}x_{t}+W_% {h}^{c}h_{t-1}+b_{c})$ (3) $\displaystyle o_{t}=\sigma(W_{x}^{o}x_{t}+W_{h}^{o}h_{t-1}+W_{c}^{o}\circ c_{t% }+b_{o})$ $\displaystyle h_{t}=o_{t}\circ\mbox{tanh}(c_{t})$

Where $x_{1},\ldots,x_{t}$ is the input, $c_{1},\ldots,c_{t}$ is the output of the LSTM, $h_{1},\ldots,h_{t}$ is the hidden layer, and ‘ $\circ$ ’ is the multiplication of the elements.

3.2 Convolution LSTM

Dr. Shi Xingjian proposed the use of a convolution operation inside the LSTM, thereby creating the ConvLSTM structure. ConvLSTM not only has LSTM timing modeling capabilities but also portrays local features similar to CNNs [36], and it is a neuron with temporal and spatial characteristics [19] (Fig. 4).

Figure 4.

Inner structure of convolutional LSTM [19].

The main calculation formula of ConvLSTM is given as follows, where $x_{1},\ldots,x_{t}$ is the input; $c_{1},\ldots,c_{t}$ is the output of ConvLSTM; $h_{1},\ldots,h_{t}$ is the hidden layer; the input gate, forget gate and output gate are $i_{t},f_{t},o_{t}$ respectively; ‘ * ’ indicates the convolution operation; and ‘ $\circ$ ’ indicates the Hadamard product.

$\displaystyle i_{t}=\sigma({W_{x}^{i}*x_{t}+W_{h}^{i}*h_{t-1}+W_{c}^{i}\circ c% _{t-1}+b_{i}})$ $\displaystyle f_{t}=\sigma({W_{x}^{f}*x_{t}+W_{h}^{f}*h_{t-1}+W_{c}^{f}\circ c% _{t-1}+b_{f}})$ $\displaystyle c_{t}=f_{t}\circ c_{t-1}+i_{t}\circ\tanh({W_{x}^{c}*x_{t}+W_{h}^% {c}*h_{t-1}+b_{c}})$ (4) $\displaystyle o_{t}=\sigma({W_{x}^{o}*x_{t}+W_{h}^{o}*h_{t-1}+W_{c}^{o}\circ c% _{t}+b_{o}})$ $\displaystyle h_{t}=o_{t}\circ\tanh({c_{t}})$

The convolution product is used in ConvLSTM instead of matrix multiplication. This operation determines the future state of a cell through the input and past states of the surrounding cells. This process can be easily accomplished using convolution operators in state-to-state and input-to-state transitions. In the information construction process, all elements in ConvLSTM are three-dimensional, and all components of the network in LSTM are only one-dimensional arrays. The input to ConvLSTM is a three-dimensional tensor. The first dimension in the tensor is taken as the number of channels in the picture, and the last two dimensions in the tensor represent the time and space information [20].

The above advantages of ConvLSTM allow it to address time and space issues and extract temporal features and extract spatial features, and thus ConvLSTM effectively uses temporal and spatial associations. The input time series has a correlation with time and also has a correlation with each other, and each of the quantities can be considered as a spatial relationship. Therefore, the advantages of the related spatiotemporal characteristics can be extracted using the above ConvLSTM.

3.3 Encoder-decoder

Deep neural networks (DNNs) cannot make the input sequences and output sequences different because DNNs require both the input and output to be fixed dimensions. Cho et al. [1] proposed the RNNs encoder-decoder model. This model contains two sub-models, an encoder model and a decoder model. The role of the encoder is to encode the input data into a vector representation in a continuous feature space. The decoder is used to decode the encoded input sequence and output the target sequence. These two models are jointly trained to maximize the conditional probability of the target sequence.

In brief, the encoder-decoder architecture encodes a variable length sequence into a fixed length vector representation and subsequently decodes the fixed length sequence into a variable length sequence. From the perspective of probability theory, the encoder-decoder model is a general method for learning a conditional distribution on variable-length sequences on the condition of another variable-length sequence, such as $p(y_{1},\cdots,y_{T^{\prime}}|x_{1},\cdots,x_{T})$ , where $T^{\prime}$ and $T$ might not be the same length.

If the encoder is an RNNs, the encoder reads the input sequence $\textbf{x}=(x_{1},\cdots,x_{n})$ in an orderly manner. When the encoder reads all of the sequences, the hidden state $h_{t}$ of the encoder gives an overall representation $c$ of the entire input sequence, which is calculated as follows:

$\displaystyle h_{t}=f(x_{t},h_{t-1})$ (5)

and

$\displaystyle c=q(h_{1},\cdots,h_{n})$ (6)

where $h_{t}\in\mathbb{R}^{n}$ is a hidden state at time $t$ , and $c$ is a vector generated from the sequence of the hidden states. $f$ and $q$ are nonlinear activation function [34].

The decoder is another RNNs that trains the input to generate an output sequence by predicting the next symbol $y_{t}$ of the hidden state $h_{t}$ . Unlike simple RNNs, both $h_{t}$ and $y_{t}$ are conditional on $c$ and $y_{t-1}$ . Therefore, the calculation formula of the hidden state of the decoder is written as follows;

$\displaystyle h_{t}=f({h_{t-1},y_{t-1},c})$ (7)

Where $f$ is a nonlinear activation function.

The architecture of the model is shown in Fig. 5.

Figure 5.

RNN encoder-decoder architecture.

In 2014, Ilya Sutskever proposed replacement of the RNNs in the encoder-decoder model [2] with LSTM and use of a layer of LSTM to read the sentences of the input language, obtain the vector representation of the largest dimension, and use the second layer of LSTM to extract the sentences of the output language from the vector. This paper uses the model structure proposed by Ilya Sutskever to achieve time series prediction.

3.4 Attention mechanism

The attention mechanism is used primarily for information screening, and the relevant information is selected from the input information. The calculation of the attention mechanism can be divided into two steps: one step calculates the attention distribution on all input information, and the other step calculates the weighted average of the input information according to the attention distribution [37].

The attention distribution $\alpha_{i}$ can be interpreted as the degree of attention of the $i$ -th information when the input is $X=[x_{1},x_{2},\cdots,x_{n}]$ and the given related task query is $q$ . We use a ‘soft’ information selection input mechanism to summarize the input information. The soft attention mechanism is shown in Fig. 6, and the attention weight $\alpha_{i}$ calculation is presented as follows:

$\displaystyle\textbf{att}(X,q)=\sum\limits_{i=1}^{N}{\alpha_{i}}x_{i}$ (8) $\displaystyle\alpha_{i}=\frac{\exp({\textit{score}(s_{t-1},h_{i})})}{\sum% \limits_{i=1}^{n}{\exp}({\textit{score}(s_{t-1},h^{\prime}_{i})})}$ (9)

Where, $\textit{score}(\cdot)$ is a scoring function, which can be calculated in the following ways [37]:

$\displaystyle s(x_{i},q)=v^{\text{T}}\textit{tanh}(Wx_{i}+Uq)$ (10) $\displaystyle s(x_{i},q)=x_{i}^{\text{T}}q$ (11) $\displaystyle s(x_{i},q)=\frac{x_{i}^{\text{T}}q}{\sqrt{d}}$ (12) $\displaystyle s(x_{i},q)=x_{i}^{\text{T}}Wq$ (13)

Figure 6.

Soft attention mechanism.

3.5 Multi-tasking learning

This paper must forecast multiple energy loads simultaneously, and thus we decided to use multi-task learning. Multi-task learning [13] is used in many fields [15] and learning is considered to be more fully represented than other single learning. At the same time, this method can capture the intrinsic association features between tasks. Using multiple loss functions to share parameters, the feature representation capability of the neural networks can be improved. Because the tasks are related, the sharing of parameters helps to improve the generalization ability of the network [37].

The reasons for why multi-task learning can improve generalization ability are listed as follows:

•
The multitasking training set is larger than that of single task learning, and the task is related, which is equivalent to the implicit data enhancement, and thus the generalization ability is improved;
•
Multi-task learning takes care of the fit of all tasks, which is equivalent to regularization, thus avoiding overfitting a single task learning.

The mathematical expression of the multitasking model is described as follows. Assume that the model has $M$ related tasks, and the training set of the $m$ -th task is ${\cal D}_{m}$ , containing $N_{m}$ samples [37].

$\displaystyle{\cal D}_{m}=\{{({x^{(m,n)},y^{(m,n)}})}\}_{n=1}^{N_{m}}$ (14)

where $x^{(m,n)}$ and $y^{(m,n)}$ represent the $n$ -th sample in the $m$ -th task and its label.

Assuming that the $M$ task correspondence models are respectively $f_{m}(x,\theta),1\leqslant m\leqslant M$ , the joint objective function of multitasking learning is a linear weighted summation of all loss functions.

$\displaystyle{\cal L}_{m}(\theta)=\sum\limits_{m=1}^{M}{\sum\limits_{n=1}^{N_{% m}}{\eta_{m}}}{\cal L}_{m}({f_{m}({x^{(m,n)},\theta}),y^{(m,n)}})$ (15)

Where ${\cal L}_{m}(\cdot)$ is the loss function of the $m$ -th task, $\eta_{m}$ is the weight of the $m$ -th task, and $\theta$ . represents all the parameters including the private module of the shared module.
4. Proposed multi-task learning architecture

This section describes our model in as much detail as possible such that the reader can better understand the model structure. In our model, we draw on the use of attention mechanisms in target recognition [32], semantic segmentation [33], and machine translation [31]. We use different attention mechanisms in our models to maintain task specificity and establish long-term dependencies for each task.

In essence, micro-energy network multi-energy load forecasting is time series prediction [30]. The encoder-decoder structure is a specific architecture in which RNN solves sequence-to-sequence problems. Instead of directly outputting the predictive vector, the encoder reads and encodes the input sequence and subsequently inputs it into the decoder for decoding. However, both the encoder and the decoder predict the output of the sequence, but the main difference is that in the decoder model, we allow it to know both the predicted value at the previous moment in the sequence and the cumulative internal state in the output sequence. In this paper, the encoded features are repeatedly input multiple times to accumulate internal states in the decoder and to achieve multi-step prediction.

When training the network, we combine two different loss function trainings. After applying the attention mechanism, the network of each task can produce a better fit to the data of the respective tasks in less time, which means that deeper relationships between data can be learned in less time and with fewer model parameters.

The encoder-decoder model is structured as follows Fig. 7, and the model parameters are shown in Table 1 below.

Table 1
Model configuration table

Structure	Layer	Parameters
	Input	0
ConvLSTM	ConvLSTM1	57088
	ConvLSTM2	98560
Attention	Attention1	3
	Attention2	3
LSTM	LSTM1	49800
	LSTM2	468000
Perception	Dense1	930
	Dense2	20100
Output	Dense1	31
	Dense2	101

Figure 7.

Model structure diagram.

Specifically, in the encoder-decoder architecture, the encoder reads the input time series $\textbf{x}=(x_{1},x_{2},\cdots,x_{t})$ mapped to a fixed length vector $c$ . The decoder is trained to predict the value $x_{t+1}$ of the next time series by the fixed length vector $c$ . In other words, the decoder defines the probability that the next time value is $x_{t+1}$ by decomposing the joint probability into an ordered condition, where $\textbf{x}=(x_{1},x_{2},\cdots,x_{t+1})$ :

$\displaystyle p(\textbf{x})=\prod\limits_{t=1}^{N}p({x_{t+1}|\{{x_{1},\cdots,x% _{t}}\},c})$ (16)

In this paper, the attention mechanism is applied between the encoder and the decoder, and our encoder uses the ConvLSTM network. Therefore, we calculate the vector differently from the RNN. The calculation formula for the hidden layer of the encoder calculation in this paper is given in Eq. (3.2).

The scoring function $\textit{score}(\cdot)$ is used to calculate the hidden layer output $s_{t-1}$ at the moment of the decoder and the hidden layer output $h_{j}$ at the moment of the encoder, and the result is converted into probability by softmax. This probability is the weight $\alpha$ that we need, and the process subsequently weighs $\alpha$ as the weight with the input time series to obtain the expression vector $c$ of the input sequence and uses $c$ as the input of the current component of the decoder to obtain the final output, which is computed by:

$\displaystyle c_{i}=\sum\limits_{j=1}^{n}{\alpha_{ij}}h_{j}$ (17)

The formula for calculating the weight $\alpha_{ij}$ of each $h_{j}$ is:

$\displaystyle\alpha_{i}=\frac{\exp({\textit{score}(s_{t-1},h_{i})})}{\sum% \limits_{i=1}^{n}{\exp}({\textit{score}(s_{t-1},h^{\prime}_{i})})}$ (18)

We parameterize the linear model $\textit{score}(\cdot)$ into a feedforward neural network, which is trained in conjunction with all other components of the proposed system, and subsequently derive the parameter values in the scoring function. The scoring function is:

$\displaystyle\textit{score}({s_{t},h_{i}})=v_{a}^{T}\tanh({W_{a}[{s_{t};h_{i}}% ]})$ (19)

The attention mechanism multiplies $h_{i}$ by the corresponding weight $\alpha_{ij}$ to obtain a fixed length vector $c_{i}$ . This paper requires multitasking output, and after we obtain the fixed length vector $c$ , we enter $c$ into the decoder network LSTM of each task. The purpose of the decoder is to establish a conditional probability $p(x_{n+1},\cdots,x_{n+k}|x_{1},\cdots,x_{n})$ between the respective outputs and inputs, where $(x_{1},\cdots,x_{n})$ is the input and $(x_{n+1},\cdots,x_{n+k})$ is the output. LSTM calculates the conditional probability by a fixed dimension vector $c_{i}$ containing all input time series information [2] computed by:

$\displaystyle p({y_{1},\ldots,y_{t}|x_{1},\ldots,x_{t}})=\prod\limits_{t=1}^{N% }p({y_{t}|\{y_{1},\ldots,y_{t-1}\},c})$ (20)

The specific operation process of the attention mechanism is shown in the following Fig. 8.

However, the regression model can use a variety of loss functions such as (mean absolute error) MAE, mean square error (MSE), etc. In our model, the mean absolute error (MAE) is used as the loss function for each task. Because our data are likely to contain outliers and MSE is highly sensitive to outliers, we choose MAE. The loss function for each task in our model is given as follows:

$\displaystyle{\cal L}({\hat{y}_{1},y_{1}})=\frac{1}{m}\sum\limits_{i=1}^{m}{|{% \hat{y}_{1}^{(i)}-y_{1}^{(i)}}|}$ (21) $\displaystyle{\cal L}({\hat{y}_{2},y_{2}})=\frac{1}{m}\sum\limits_{i=1}^{m}{|{% \hat{y}_{2}^{(i)}-y_{2}^{(i)}}|}$

Where $\hat{y}_{1}^{(i)}$ and $\hat{y}_{2}^{(i)}$ are predicted values, and $y_{1}^{(i)}$ and $y_{2}^{(i)}$ are actual values.

Figure 8.

Attention mechanism calculation process.

After obtaining the loss function for each task, we need to minimize the loss function for each task. We combine the linear weighted summation of the individual task loss functions as follows:

$\displaystyle{\cal L}(\theta)=\eta_{1}{\cal L}(\hat{y}_{1},y_{1})+\eta_{2}{% \cal L}(\hat{y}_{,}y_{2})$ (22)

where, $\eta_{1}$ and $\eta_{2}$ are the loss weights of each regression task. The purpose of introducing loss weight is to balance the total loss function ${\cal L}(\cdot)$ such that the primary and secondary tasks can be distinguished.

5. Experiments

In this section, we introduce the data sources, evaluation criteria and experimental results and finally present the performance of our network under various evaluation criteria. We also present a horizontal comparison with other networks. To make the experimental results more convincing, this paper compares the selected network data with the best performance of each network to the network data in this paper.

The data in this paper are the operation data from the micro-energy network of the China Electric Power Research Institute in a certain location in northwestern China. The data are desensitized by professionals. The experimental data are ten-dimensional data including two historical energy data and environmental data. Two types of energy must be forecast in the experiment, namely, electrical load and gas load.

The experimental results show that our model has the advantage of higher prediction accuracy than other excellent models. Specifically, we use the $k_{1}$ step data including the a priori load history data and the environmental variables and the load data of the two energy sources to predict the $k_{2}$ step data of the load of the two energy sources in the future, as follows:

$\displaystyle X_{\text{input}}=[x_{t-k_{1}+1},x_{t-k_{1}+2},\cdots,x_{t}]$ $\displaystyle\hat{Y}^{1}_{\text{output}}=[\hat{y}^{1}_{t+1},\hat{y}^{1}_{t+2},% \cdots,\hat{y}^{1}_{t+k_{2}}]$ (23) $\displaystyle\hat{Y}^{2}_{\text{output}}=[\hat{y}^{2}_{t+1},\hat{y}^{2}_{t+2},% \cdots,\hat{y}^{2}_{t+k_{2}}]$

5.1 Experimental setting

In total, we use 43,000 data sets, and the training set size is 35,000 groups, of which approximately 10% of the data is used as a validation set. Ten feature quantities are present in the input data, and thus the input feature map is a $1\times 10\times 10$ matrix. The training data use a gradient-optimized Adam and backpropagation to train the model [38]. To reduce the training time, we use a small batch gradient to improve the training speed, and the size of the mini-batch is 64.

Experimental details:

•
Data pretreatment. Missing and noise values are filled with the mean. Wind direction data do not have a digital feature. In this paper, wind direction data is are coded so such that the network can be used.
•
Split data set. The experimental data in this paper are time series data. A correlation exists between the time series data before and after, and thus the order of the data cannot be disturbed. Therefore, this paper splits the training set, test set, and validation set in chronological order. We select the last 10% of the data set as the validation set, and use the first 90% of the data as the training and test sets for the experiment. In the first 90% of the data set, the last 5% is selected as the test set. The training set, test set, and validation set are constructed into the data form required by the model according to the method described above. Finally, we split training sets with different proportions to verify the model in this paper.
•
Deep learning framework. This experiment uses the Keras deep learning framework to build a network model.
•
Optimizer selection and fine-tuning of network parameters. The selection of the optimizer and the fine-tuning of the network parameters are based on RMSE, MAPE and $R^{2}$ to select the parameter value with the best performance.
•
Comparative Experiment. In this paper, under the condition that the time scale of the input and output is the same, we choose RMSE, MAPE and $R^{2}$ to show the best result as the comparison data of the network.

5.2 Evaluation criteria

To assess the accuracy of the model, we used RMSE (root mean squared error), MAPE (Mean Absolute Percentage Error), and $R^{2}$ ( $R$ -square) to evaluate the model.

We mainly use RMSE to evaluate the model.

$\displaystyle\textit{RMSE}=\sqrt{\frac{1}{n}\sum\limits_{i=1}^{n}{({\hat{Y}_{i% }-Y_{i}})^{2}}}$ (24)

Since the MAPE index shows the error rate for each information point, the sum of the error rates of these information points can be used for horizontal comparison. All we will use MAPE to compare the models.

$\displaystyle\textit{MAPE}=\frac{100}{n}\sum\limits_{i=1}^{n}{\left|{\frac{A_{% i}-F_{i}}{A_{i}}}\right|}$ (25)

The $R^{2}$ index maps the prediction efficiency of the regression model to 0–1, so that for the same set of data, we can determine the prediction efficiency of the model based on the $R^{2}$ index.

$\displaystyle R^{2}=1-\frac{\sum\limits_{i=1}^{n}{(y_{i}-\hat{y}_{i})^{2}}}{% \sum\limits_{i=1}^{n}{(y_{i}-\bar{y})^{2}}}$ (26)

Figure 9.

Multitasking input and output feature map.

5.3 Multi-step prediction

In this experiment, we set up the first 20 time steps to predict the sequence of the next 5 time steps. The input and output characteristic map is shown in Fig. 9. The two time series of the predicted electrical load and the gas load are used as the tag training neural network of the respective tasks.

The input sequence is repeatedly input to the decoder 5 times after passing through the ConvLSTM encoding. Each unit of the decoder outputs a time series of length 5, which is the number of steps we need to predict, and finally outputs a time series of length 5 after interpretation by the fully connected layer.

5.4 Model comparison and data analysis

In this section, we will compare the models from the three aspects of RMSE, MAPE, $R^{2}$ .

Because the encoder-decoder architecture was proposed, many networks have been used as encoders. However, for multi-energy forecasting of multi-energy load in the micro-energy networks, this paper decided that the ConvLSTM network is best suited for this task. The model designed in this paper uses three evaluation indicators to compare with other multi-task models, single-task models, and models that do not use attention mechanisms. The maximum number of iterations for all methods is 50. The description of each comparison model is given as follows:

•
ConvLSTM+Att+Multi is the model designed in this paper. This model uses ConvLSTM as the encoder and LSTM as the decoder, multi-task learning to improve the generalization ability, and an attention mechanism between the encoder and the decoder to speed up the learning of the model.
•
ConvLSTM+Multi is a model with no attention mechanism in the ConvLSTM+Att+Multi model.
•
CNN+LSTM+Att+Multi is an encoder-decoder architecture model in which CNN is used as the encoder, and LSTM is used as the decoder. This model also uses an attention mechanism and multi-task learning.
•
LSTM+Multi is an encoder-decoder architecture model that uses LSTM as the encoder and decoder and uses a multi-tasking learning model.
•
LSTM+ATT+Multi is similar to the LSTM+Multi model but with an attention mechanism.
•
ConvLSTM+Att+Single is a single-task model with an encoder-decoder architecture that uses ConvLSTM as the encoder and LSTM as the decoder and also uses the attention mechanism.
•
LSTM+Att+Single is a single-task model that uses LSTM as an encoder and decoder and also uses an attention mechanism.

In Table 2, this paper compares the differences between different tasks of several neural networks. As shown from the comparison in Table 2, the ConvLSTM-based model has significant advantages over the LSTM and CNN-based network. Especially for the gas load forecasting, the accuracy of the ConvLSTM model prediction is much higher than that of LSTM and CNN.

From Table 2, we observe that among the six evaluation indicators of the two tasks, the hybrid model designed in this paper (in addition to the RMSE index of the gas energy load) is completely ahead of the other models.

Table 2
RMSE, MAPE and $R^{2}$

Model Task RMSE MAPE $R^{2}$

Multi-task learning Our model Electric 0.0223 0.0156 94.891%

Gas 0.0497 0.0783 93.490%

ConvLSTM+Multi Electric 0.0244 0.0163 93.388%

Gas 0.0509 0.0800 93.159%

LSTM+ATT+Multi Electric 0.0312 0.0200 92.148%

Gas 0.0453 0.0989 67.173%

LSTM+Multi Electric 0.0359 0.0238 89.579%

Gas 0.0446 0.0975 67.233%

CNN+LSTM+ATT+Multi Electric 0.0284 0.0185 87.815%

Gas 0.0535 0.0857 92.444%

Single-task learning ConvLSTM+ATT+Single Electric 0.0279 0.0192 91.389%

Gas 0.0499 0.0817 93.435%

LSTM+ATT+Single Electric 0.0324 0.0176 91.497%

Gas 0.0414 0.0871 72.521%

In Fig. 10. Comparison of evaluation criteria for each step of each model. Comparison of evaluation criteria for each step of each model. below we compare the RMSE, MAPE, $R^{2}$ differences between the different tasks of the ConvLSTM hybrid network and the LSTM multi-tasking network at different time steps. Figure 11 is a comparison of three evaluation criteria for the overall five time steps.

Figure 10.
Comparison of evaluation criteria for each step of each model.

Figure 11.
Comparison of overall evaluation criteria.

Figure 12.
Various time step prediction renderings of electrical load.

Figure 13.
Various time step prediction renderings of gas load.

Figure 14.
Comparison of different training set ratios at each time step.

Figure 15.
Overall comparison of different training set splits.

Figure 16.
Comparison of forecast evaluation criteria of oxynitride and sulfur dioxide.

C+L+At+Mul: CNN+LSTM+Att+Multi; ConL+At+Mul: ConvLSTM+Att+Multi; ConL+Mul: ConvLSTM+Multi; ConL_Sing_Ele: ConvLSTM_single_Ele; L+At+Mul: LSTM+Att+Multi; ConvL_sing_Gas: ConvLSTM_single_Gas; L_Sing_Gas: LSTM_Single_Gas; L_Sing_Ele: LSTM_Single_Ele; L+Mul: LSTM+Multi.

To make the experimental comparison more convincing, the maximum number of iterations for each model is 50, and the mini-batch is 64. The input data are the same processed data. The various parameter settings of the neural network are as consistent as possible.

From the results in Figs 10 and 11, the evaluation indicators of the model proposed in this paper are superior to those of other models. Moreover, as the prediction time scale increases, the prediction accuracy decreases, but the accuracy of the model is slower than other models, especially for the network with LSTM as the encoder. The model’s long-term scale prediction performance is superior to that of other models. It can also be observed that the model performance of the applied attention mechanism is better than the model without the attention mechanism.

It can be observed from Figs 10 and 11 that a large difference exists in the $R^{2}$ value of the gas energy load, and the predicted $R^{2}$ value of the model using the LSTM as the encoder is much smaller than the value using ConvLSTM and CNN as the encoder. The $R^{2}$ index maps the prediction efficiency of the regression model to 0–1, such that for the same set of data, we can determine the prediction efficiency of the model based on the $R^{2}$ index.

In other words, in prediction of the gas energy load, LSTM does not work well as an encoder, and the model using ConvLSTM and CNN as the encoder has better prediction results. Because LSTM does not have feature extraction capabilities, all input features are encoded, resulting in features that are not desired to be learned by the model. Models that use CNN and ConvLSTM as encoders extract features of the input data to reduce the impact of uncorrelated features on the predicted output.

In addition, the electrical energy load is much higher than the predicted accuracy of the gas load. We believe that there are two main reasons for this result:

•
In the micro-energy network, the randomness of the gas energy load is large, which means that the model able is not able to fit the corresponding gas energy load curve effectively;
•
Another reason is that this method is limited by data. The data set contains no gas load-related feature quantity. In other words, that is to say, the change in the amount of features that we use does not have much correlation with the change in the gas load.

Figures 12 and 13 below give a comparison of the predicted and actual values of the electrical energy load and the gas energy load. From the figure, the fitting effect of the model is quite good. Figures 12 and 13 show the top-five fitting maps from top to bottom. It is obvious that the fitting effect of the first time step is the best, and the accuracy begins to decrease.

From Figs 14 and 15 below, we can see the performance of the model in different training sets and test sets. The larger the training set, the better the performance of the three evaluation criteria.

We have found time series data of harmful gas emissions from petrochemical companies. The data includes oxynitride, sulfur dioxide, reaction temperature, reactant concentration, etc. The total number of data sets is 50,000, including two data must to be predicted, and the total number is 13 dimensions. Use the prediction model in this paper to predict the concentration of oxynitride and sulfur dioxide. The comparison with other models is as follows Table 3 and Fig. 16. Overall, our model is superior to other commonly used deep learning models.

Table 3
Evaluation criteria of our multi-task learning model on another data sets against others model

Model Task RMSE MAPE $R^{2}$

Multi-task learning Our model Oxynitride (NO ${}_{\text{X}}$ ) 0.0287 0.0628 92.831%

Sulfur dioxide (SO ${}_{2}$ ) 0.0216 0.1258 91.195%

ConvLSTM+Multi Oxynitride (NO ${}_{\text{X}}$ ) 0.0330 0.0619 89.040%

Sulfur dioxide (SO ${}_{2}$ ) 0.0264 0.1405 89.693%

LSTM+ATT+Multi Oxynitride (NO ${}_{\text{X}}$ ) 0.0439 0.0557 90.520%

Sulfur dioxide (SO ${}_{2}$ ) 0.0254 0.1029 76.902%

LSTM+Multi Oxynitride (NO ${}_{\text{X}}$ ) 0.0354 0.0612 88.412%

Sulfur dioxide (SO ${}_{2}$ ) 0.0282 0.1272 72.239%

CNN+LSTM+ATT+Multi Oxynitride (NO ${}_{\text{X}}$ ) 0.0478 0.0894 81.951%

Sulfur dioxide (SO ${}_{2}$ ) 0.0363 0.1902 82.296%

Single-task learning ConvLSTM+ATT+Single Oxynitride (NO ${}_{\text{X}}$ ) 0.0450 0.0845 85.879%

Sulfur dioxide (SO ${}_{2}$ ) 0.0345 0.1883 87.176%

LSTM+ATT+Single Oxynitride (NO ${}_{\text{X}}$ ) 0.0355 0.0816 86.242%

Sulfur dioxide (SO ${}_{2}$ ) 0.0323 0.1704 72.650%

6. Conclusion

	Model	Task	RMSE	MAPE	$R^{2}$
Multi-task learning	Our model	Electric	0.0223	0.0156	94.891%
		Gas	0.0497	0.0783	93.490%
	ConvLSTM+Multi	Electric	0.0244	0.0163	93.388%
		Gas	0.0509	0.0800	93.159%
	LSTM+ATT+Multi	Electric	0.0312	0.0200	92.148%
		Gas	0.0453	0.0989	67.173%
	LSTM+Multi	Electric	0.0359	0.0238	89.579%
		Gas	0.0446	0.0975	67.233%
	CNN+LSTM+ATT+Multi	Electric	0.0284	0.0185	87.815%
		Gas	0.0535	0.0857	92.444%
Single-task learning	ConvLSTM+ATT+Single	Electric	0.0279	0.0192	91.389%
		Gas	0.0499	0.0817	93.435%
	LSTM+ATT+Single	Electric	0.0324	0.0176	91.497%
		Gas	0.0414	0.0871	72.521%

	Model	Task	RMSE	MAPE	$R^{2}$
Multi-task learning	Our model	Oxynitride (NO ${}_{\text{X}}$ )	0.0287	0.0628	92.831%
		Sulfur dioxide (SO ${}_{2}$ )	0.0216	0.1258	91.195%
	ConvLSTM+Multi	Oxynitride (NO ${}_{\text{X}}$ )	0.0330	0.0619	89.040%
		Sulfur dioxide (SO ${}_{2}$ )	0.0264	0.1405	89.693%
	LSTM+ATT+Multi	Oxynitride (NO ${}_{\text{X}}$ )	0.0439	0.0557	90.520%
		Sulfur dioxide (SO ${}_{2}$ )	0.0254	0.1029	76.902%
	LSTM+Multi	Oxynitride (NO ${}_{\text{X}}$ )	0.0354	0.0612	88.412%
		Sulfur dioxide (SO ${}_{2}$ )	0.0282	0.1272	72.239%
	CNN+LSTM+ATT+Multi	Oxynitride (NO ${}_{\text{X}}$ )	0.0478	0.0894	81.951%
		Sulfur dioxide (SO ${}_{2}$ )	0.0363	0.1902	82.296%
Single-task learning	ConvLSTM+ATT+Single	Oxynitride (NO ${}_{\text{X}}$ )	0.0450	0.0845	85.879%
		Sulfur dioxide (SO ${}_{2}$ )	0.0345	0.1883	87.176%
	LSTM+ATT+Single	Oxynitride (NO ${}_{\text{X}}$ )	0.0355	0.0816	86.242%
		Sulfur dioxide (SO ${}_{2}$ )	0.0323	0.1704	72.650%

In this paper, we design a novel encoder-decoder architecture model to achieve multi-time scale prediction of multi-energy in micro-energy networks. In this paper, the ConvLSTM encoder is used to encode the input data into a fixed-length vector, and LSTM is used to decode the fixed-length vector into the resulting output. ConvLSTM extracts the characteristics of the input data, similar to CNN, and can effectively extract the global characteristics of the input data. The use of multi-task learning improves the generalization ability of the model, and the use of the attention mechanism accelerates the learning speed of the model and improves the robustness of the model.

In future work, we will explore deeper multitasking relationships and design multitasking models that predict more energy at the same time. Additionally, we must also consider the shortcomings of the model in the actual application process and make the corresponding improvements.

References

Cho

Merrienboer

Gulcehre

Bougares

Schwenk

and Bengio

, Learning phrase representations using RNN encoder-decoder for statistical machine translation, in: Arxiv Preprint arXiv:1406.1078, 2014.

Sutskever

Vinyals

and Le

Q.V.

, Sequence to Sequence Learning with Neural Networks [C], in: NIPS. MIT Press, 2014.

Jiaqi

Tao

and Jing

, Multi-task learning based on deep architecture for various types of load forecasting in regional energy system integration [J], Power System Technology 42(3) (2018), 698–707.

Saber

A.Y.

and Alam

A.K.M.R.

, Short term load forecasting using multiple linear regression for big data [C], in: IEEE Symposium Series on Computational Intelligence (SSCI), 2017.

Powell

K.M.

Sriprasad

Cole

W.J.

and Edgar

T.F.

, Heating, cooling, and electrical load forecasting for a large-scale district energy system [J], Energy 74(sep.) (2014), 877–885.

Guangquan

Jiangzhang

Jian

Zhiping

and Xishun

, Multi-task learning using a hybrid representation for text classification, Neural Computing and Applications, 2018.

Pengfei

Xipeng

and Xuanjing

, Recurrent neural network for text classification with multi-task learning, in: International Joint Conference on Artificial Intelligence, 2016, pp. 2873–2879.

Pengfei

Xipeng

and Xuanjing

, Adversarial multi-task learning for text classification, in: Meeting of the Association for Computational Linguistics, Vol. 1, 2017, pp. 1–10.

Yunpeng

Junpeng

et al., Multi-step ahead time series forecasting for different data patterns based on LSTM recurrent neural network [J], in: Proceedings – 2017 14th Web Information Systems and Applications Conference, WISA 2017.

10.

Gers

F.A.

Eck

and Schmidhuber

, Applying LSTM to Time Series Predictable through Time-Window Approaches [C], in: International Conference on Artificial Neural Networks, Springer, Berlin, Heidelberg, 2001.

11.

Girshick

R.B.

Donahue

Darrell

and Malik

, Rich feature hierarchies for accurate object detection and semantic segmentation, in: Computer Vision and Pattern Recognition, 2014, pp. 580–587. arXiv:1311.2524 [cs.CV].

12.

Long

Shelhamer

and Darrell

, Fully convolutional networks for semantic segmentation, in: Computer Vision and Pattern Recognition, 2015, pp. 3431–3440.

13.

Caruana

, Multitask learning [J], Machine Learning 28(11) (1997), 41–75.

14.

Hochreiter

and Schmidhuber

, Long short-term memory [J], Neural Computation 9(8) (1997), 1735–1780.

15.

Balikas

Moura

and Amini

M.R.

, Multitask learning for fine-grained twitter sentiment analysis in: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieva, 2017, pp. 1005–1008.

16.

Collobert

and Weston

, A unified architecture for natural language processing: deep neural networks with multitask learning, in: International Conference on Machine Learning, 2008, pp. 160–167.

17.

Pengfei

Xipeng

and Xuanjing

, Deep multi-task learning with shared memory, in: Conference on Empirical Methods in Natural Language Processing, 2016.

18.

Pengfei

Xipeng

and Xuanjing

, Recurrent neural network for text classification with multi-task learning, in: International Joint Conference on Artificial Intelligence, 2016, pp. 2873–2879.

19.

Xingjian

Zhourong

Hao

et al., Convolutional LSTM network: a machine learning approach for precipitation nowcasting [J], Advances in Neural Information Processing Systems, 2015.

20.

Yong

Kun

Zhanhao

et al., Hybrid forecasting model based on long short term memory network and deep learning neural network for wind signal [J], Applied Energy 236 (2019), 262–272.

21.

Kaiming

Xiangyu

Shaoqing

and Jian

, Deep residual learning for image recognition, in: Computer Vision and Pattern Recognition, 2015, pp. 770–778.

22.

Kaiming

Xiangyu

Shaoqing

and Jian

, Identity mappings in deep residual networks, in: European Conference on Computer Vision, 2016, pp. 630–645.

23.

Bengio

Boulanger-Lewandowski

and Pascanu

, Advances in optimizing recurrent networks, in: International Conference on Acoustics, Speech, and Signal Processing, 2013, pp. 8624–8628.

24.

Deng

Seltzer

Acero

Mohamed

and Hinton

, Binary coding of speech spectrograms using a deep autoencoder, in: Conference of the International Speech Communication Association, 2010, pp. 1692–1695.

25.

Tao

Ruifeng

and Xuan

, Improving sentiment analysis via sentence type classification using BiLSTM-CRF and CNN, Expert Systems with Application 72 (2017), 221–230. [43].

26.

Lin

Ruifeng

and Qin

, Learning representations from heterogeneous network for sentiment classification of product reviews, Knowl Based Syst, 124. doi: 10.1016/j.knosys.2017.02.030.

27.

Jiguang

Ping

Jianlong

and Shuo

, Sentiment classification based on AS-LDA model, Procedia Computer Science 31 (2014), 511–516. doi: 10.1016/j.procs.2014.05.296.

28.

Chong

Heyan

et al., Topic-related Chinese message sentiment analysis, Neurocomputing 210 (2016), 237–246. [48].

29.

Hochreiter

and Bengio

, Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, Wiley-IEEE Press, 2001, pp. 237–243.

30.

Tao

Tian

and Aberer

, Hybrid Neural Networks for Learning the Trend in Time Series [C], Twenty-Sixth International Joint Conference on Artificial Intelligence, AAAI Press, 2017.

31.

Bahdanau

Cho

and Bengio

, Neural Machine Translation by Jointly Learning to Align and Translate [J], Computer ence, 2014.

32.

Jimmy

Volodymyr

and Koray

, Multiple object recognition with visual attention, in: International Conference on Learning Representations, 2015.

33.

Pengfei

Xipeng

and Xuanjing

, Deep multi-task learning with shared memory, in: Conference on Empirical Methods in Natural Language Processing, 2016.

34.

Bahdanau

Cho

and Bengio

, Neural Machine Translation by Jointly Learning to Align and Translate [J], Computer Science, 2014.

35.

JiXiang

and Peiqi

, Short-term load forecasting method based on CNN-LSTM hybrid neural network model, Automation of Electric Power Systems 43(8) (2019), 131–137.

36.

LeCun

Bottou

Bengio

and Haffner

, Gradient-based learning applied to document recognition, Proceedings of the IEEE 86(1) (1998), 2278–2324.

37.

Xipeng

, Neural Network and Deep Learning, 2015.

38.

Rumelhart

D.E.

Hinton

G.E.

and Williams

R.J.

, Learning representations by back-propagating errors, Nature 23 (1986).

39.

Chiou-Jye

and Ping-Huan

, A deep CNN-LSTM model for particulate matter (PM2.5) forecasting in smart cities [J], Sensors 18(7) (2018), 2220.

40.

Cao

Kim

Hwang

and Jung

, CNN-LSTM coupled model for prediction of waterworks operation data, Journal of Information Processing Systems 14 (2018), 1508–1520. doi: 10.3745/JIPS.02.0104.

41.

Fiot

J.B.

and Dinuzzo

, Electricity Demand Forecasting by Multi-Task Learning [J], IEEE Transactions on Smart Grid, 2016, 1–1.

Multi-task prediction model based on ConvLSTM and encoder-decoder

Abstract

Keywords

1. Introduction

2.1 Data standardization

3.1 LSTM

Table 1 Model configuration table

5.4 Model comparison and data analysis

References

Table 1
Model configuration table