Attention mechanism for developing wind speed and solar irradiance forecasting models

Abstract

This article presents the Recurrent Neural Network (RNN) and its Attention mechanism to develop forecasting models for renewable energy applications. In this study, wind speed and solar irradiance forecasting models have been developed as these two factors play a significant role in renewable energy production. The irregular nature of wind poses the challenge of accurate wind speed prediction, while solar irradiance forecasting can aid in the planning and deployment of solar power plants. In this paper, six RNN techniques, namely RNN, GRU, LSTM, Content-based Attention, Luong Attention, and Self-Attention based RNN are considered for forecasting the future values of wind speed and solar irradiance in particular geographical locations. The aim is the identification of the advantages, comparison, and importance of different recurrent neural network methods for forecasting models. All models are developed on the datasets of the National Renewable Energy Laboratory (NREL) and NASA’s Prediction of Worldwide Energy Resource (POWER).

Keywords

Solar irradiance wind speed recurrent neural network gated recurrent unit long short term memory attention model

Introduction

Increasing amounts of environmental pollution and depletion of fossil fuel reserves have led to the search for clean energy sources. Wind and solar power are inexhaustible and clean sources are considered potential energy resources with rapid growth worldwide. The design, installation, and running costs of wind and solar power plants are very high. Several hundred individual wind turbines constitute these wind farms. In order to make a profit from these farms qualitatively, consistent excellent performance is a necessity. Optimization in the placement criteria of turbines and wind farm layout design can maximize the production of operational wind farms. At the design stage, the issues related to placement criteria of turbines and layout of the wind farm are addressed in Chowdhury et al. (2013) and Elkinton et al. (2008), respectively. In research of estimating wind turbine power output, Thapar et al. (2011), Wadhvani et al. (2018), and Dongre et al. (2019) observed that the power produced at wind power plants have a strong relationship with the wind speed of the reported site. Most of the time, the random nature of wind hinders the process of power estimation. Algorithms applied to predict future energy production cannot provide good accuracy, especially in areas with frequent altitude variations. Kushwah et al. (2020) stated that wind speed forecasting plays a significant role in producing cost-effective energy as it is beneficial in the planning and operation process of power generation. In the case of solar energy, the Photovoltaic (PV) system is necessary for converting solar energy to electricity. PV power causes volatility to the grid, which then affects grid stability (Kumar et al., 2020). The power grid operator requires forecasts of production to ensure a cost-effective and secure electricity supply system. Amrouche et al. (2013) stated that there exists a linear correlation between the power output of the PV modules and solar irradiance. Sharma and Kakkar (2017) stated solar irradiance forecasting could be applied for different applications in solar energy systems such as maintenance of stability and regulation, monitoring, and management of scheduling and unit commitment.

Previously several studies were undertaken to develop accurate time series models for wind speed (Mostafaeipour et al., 2019, Kushwah and Wadhvani, 2020) and solar irradiance (Amrouche and Pivert, 2014; Thapar, 2019) forecast applications. Blanchard and Samanta (2019) have applied a nonlinear autoregressive neural network with exogenous inputs for wind speed forecasting, proving that these models perform better than the baseline models. Sheela and Deepa (2013) reviewed wind speed prediction models based on neural networks for the past 15 years concluding that the neural network models performed better than the non-neural network models. Yagli et al. (2019) conducted a study utilizing the irradiance data derived from satellites from many locations on 68 different models based on machine learning. The research concluded that the Multi-Layer Perceptron (MLP) were among the best performing models for assessing the performance of the model. Following this work, the Neural Network (NN) models being utilized here focuses on short-term forecasting based on satellite-derived data. The neural network models do not require data to be stationary and work on nonlinear layers. These are also enormously capable of evaluating complex data structures, capable of reconstructing a data-driven noisy system. This makes the neural networks suitable for variable and complex time series forecasting.

Alzahrani et al. (2017) proposed a solar irradiance prediction approach using deep Recurrent Neural Networks (RNN) to improve the complexity of the model and enable extraction of high-level features. This method performed with better accuracy than conventional support vector machines and feedforward neural networks. The RNN (Rumelhart et al.,1986) architecture accounts for dependencies between data nodes by preserving sequential information in an inner state. However, RNN is prone to exploding and vanishing gradients. As such, variants of RNN were developed, such as Long Short-Term Memory (LSTM) networks (Hochreiter et al., 1997) and Gated Recurrent Units (GRU) by replacement of the conventional perceptron architecture with gating mechanisms that regulate the flow of information across the network. These variants are used widely in forecasting applications.

While designing a cost-effective power system, the first significant issue in the literature is that reliable meteorological data should be available readily to create a techno-economically viable power system. Due to difficulties in installation, maintenance, and the high cost of measuring these data, they are either unavailable or partially available at the installation site. Hence, demand exists for developing alternative ways to predict them. After several studies, it has been observed that the existing methods of wind speed and solar irradiance forecasting suffer from low accuracy of forecasts and less significant data scalability. Further, it has been observed that comparing time series forecasting methods and finding the technique that can provide the future’s best prediction is essential. This paper starts by providing an overview of the RNN models used for forecasting. The RNN variants for time series forecasting, namely, RNN, LSTM, GRU, and Attention-based RNNs have been used to model the data. Attention-based time series models utilize complex and nonlinear interdependencies between time steps as well as time series to predict future data values. Application of the Attention mechanism to deep neural network models allows the network to focus adaptively on input features that are of more importance to the current output and mitigate other features’ interference. An empirical investigation has been conducted on the data provided by NREL and NASA. Finally, the developed models’ performance has been measured using the criteria of mean absolute error (MAE), and root mean squared error (RMSE).

Methodology for forecasting

Deep Neural Networks are a class of artificial neural networks that consist of many layers of perceptrons. It consists of an input, hidden, and an output layer. Recurrent Neural Networks (Unnikrishnan et al., 1994) are different from other deep neural networks due to their capacity for sending information over time-steps. Recurrent Neural Networks can be considered as an optimization over programs (Patterson and Gibson, 2017). This section presents an overview of the deep neural network mechanism of Recurrent Neural Networks, its variants and the Attention-based networks used for forecasting wind speed and solar irradiance.

Recurrent neural network

Recurrent neural networks perform modeling of the time aspect of data by the creation of cycles in the network. RNN is a special type of neural network that accounts for data dependencies between nodes. It preserves the sequential information in an inner state, allowing them to persist the knowledge accrued from subsequent time steps (Goodfellow et al., 2016).

z_{t} = W^{xh} x_{t} + W^{hh} h_{t - 1}

(1)

h_{t} = \tanh (z_{t})

y_{t} = W^{hy} h_{t}

Equation (1) represents the working of an RNN cell with $x_{t}$ as present input, $h_{t - 1}$ the previous state, $W^{xh}$ as the weight between inputs to the hidden unit, $W^{hh}$ being weight between hidden to the hidden unit, i.e., the recurrent weight. $z_{t}$ is the output of the hidden unit before the application of the activation function. Then, $h_{t}$ is the hidden unit output that is sent to the next recurrent units and also used in the computation of the final output of that RNN unit. The final output $y_{t}$ is computed by applying another activation function to the hidden unit output and $W^{hy}$ weight between hidden to the output unit. The selection and application of the activation function depend on the task being performed (Géron, 2017).

Long short term memory network

LSTM (Gers et al., 2000) introduces computational components added to an RNN cell, input gate, output gate, and forget gate. The equation for the hidden vector is changed for LSTM through the use of a longer term of memory. The LSTM operations are specifically for having fine-grained data control. The equations for the forward pass are represented in equation (2):

a_{t} = \tanh (W_{c} x_{t} + R_{c} h_{t - 1})

(2)

i_{t} = σ (W_{i} x_{t} + R_{i} h_{t - 1})

f_{t} = σ (W_{f} x_{t} + R_{f} h_{t - 1})

o_{t} = σ (W_{o} x_{t} + R_{o} h_{t - 1})

c_{t} = i_{t} ⊙ a_{t} + f_{t} ⊙ c_{t - 1}

h_{t} = o_{t} ⊙ \tanh (c_{t})

The present input and previous state are processed by $a_{t}$ . The input gate $i_{t}$ decides which parts are required to be added to the long term state $c_{t}$ . The forget gate $f_{t}$ makes decisions regarding the portions to be erased and erases unnecessary parts while the output gate $o_{t}$ decides on the parts of $c_{t}$ to be read and shown as output. $h_{t}$ is the short term hidden state between the cells and $c_{t}$ is the long term state in which the respective gates add and remove memories. These gates aide in remembering the long term and short term dependencies.

Gated recurrent unit network

The gated recurrent unit (Cho et al., 2014) is a simplification of the LSTM, without the use of cell states explicitly. A single vector is formed from the state vectors. Equation (3) presents the equations of a GRU cell.

z_{t} = σ (W_{xz} x_{t} + W_{hz} h_{t - 1})

(3)

r_{t} = σ (W_{xr} x_{t} + W_{hr} h_{t - 1})

g_{t} = \tanh (W_{xg} x_{t} + W_{hg} (r_{t} \otimes h_{t - 1})

h_{t} = (1 - z_{t}) \otimes \tanh (W_{xg} h_{t - 1} + z_{t} \otimes g_{t})

A gate controller controls the forget gate and input gate. The input gate is opened on output of 1 with the forget gate being closed while an output of 0 causes vice-versa. For the storage of memory, the location is erased for it to be stored.

Attention-based networks for time series forecasting

In a sequence to sequence model, it is the last encoder hidden state which is forwarded to the decoder as a vector representation, providing a numerical summary of an input sequence. As such, in the case of a long input, the decoder receives only one vector representation which results in forgetting. This led to the introduction of Attention which acts as an interface providing the decoder information from each encoder hidden state. This allows the model to focus on useful portions of the input sequence selectively (Brahma and Wadhvani, 2020a).

In general sequence to sequence models, only the first decoder hidden state is used to predict the outputs. However, in an Attention mechanism, a score is calculated between an encoder and decoder hidden state. After the score calculation, these are put into a softmax layer, to represent Attention distribution. Now, these softmax scores are again multiplied with the encoder hidden states to get the alignment vector. The alignment vectors are then summed to get the context vector, which is fed to the decoder for achieving the output (Shih et al., 2019).

score (h_{t - 1}, s_{t - 1}) = ν_{a}^{T} \tanh (W_{a} [h_{t - 1}; s_{t}] + W_{x} x + b_{a})

(4)

α_{t - 1} = \frac{\exp (score (h_{t - 1}, s_{t - 1}))}{\sum_{i = 1}^{t} \exp (score (h_{i - 1}, s_{i - 1}))}

ContextVector = \sum_{i = 1}^{t} α_{i} h_{i}

In equation (4), x is input, $h_{t - 1}$ represents the hidden state while the cell state is denoted by $s_{t - 1}$ and $W_{a}$ , $W_{b}$ and $b_{a}$ are the Attention weights and bias (Brahma and Wadhvani, 2020b). After the input layer, there is an encoder layer processing the input and forwarding it to the next Attention layer. This Attention layer’s output is fed as output to the decoder, which then sends it to the output layer. The encoder and decoder layers are nothing but the neural network nodes and layers that perform learning based on sequential data. The relevant information is further computed by the addition of an Attention layer after its extraction through the scoring function. This leads to further management of long-term temporal dependencies.

Bahdanau and Luong attention

Bahdanau Attention (Bahdanau et al., 2014) was one of the earliest forms of Attention which introduced the concept of Attention. The weights are adjusted, aligned, which is directly responsible for the score while training the model. The encoder and decoder consist of Gated Recurrent Units, while the scoring function is additive.

score (s_{t}, h_{i}) = ν_{a}^{T} \tanh (W_{a} [s_{t}; h_{i}])

(5)

Luong Attention (Luong et al., 2015) was introduced for simplification and generalization of the Bahdanau Attention. Here, the encoder and decoder are both stacked LSTMs. The score function can be additive, dot product, location-based or general.

score (s_{t}, h_{i}) = s_{t}^{T} h_{i}

(6)

Content-based attention

The Attention vectors in cosine scoring or content-based Attention (Graves et al., 2014) are based on key and memory rows similarity. It performs the computation of the cosine similarity, as shown in equation (7), after which the softmax function’s normalization is done.

score (s_{t}, h_{i}) = cosine [s_{t}, h_{i}]

(7)

Self-attention

Self-Attention (Cheng et al., 2016) shares the concepts to Attention fundamentally taking in n inputs and returning n outputs. The self-attention mechanism allows the inputs to interact with self and find out the one to pay more Attention to. The outputs are aggregates of these interactions and Attention scores. It consists of a query, key and value and learns the correlation between the current and past values (Vaswani et al., 2017). The equation below shows the Attention score function with Q, K, and V representing Query, Key, and Value respectively, $d_{k}$ is the dimensionality of key vector and σ denoting the softmax function.

score = σ (\frac{Q . K^{T}}{\sqrt{d_{k}}}) . V

(8)

Proposed methodology

The model architecture followed here is categorized into two types, the memory-based RNN consisting of vanilla RNN, LSTM, and GRU. The Attention-based RNN consists of Luong, content-based, and self-attention. The input is a univariate dataset separated by appropriate window lags. Figure 1 represents the model architecture followed for future prediction of wind speed and solar irradiance data. RNN represents the models since it is the RNN variants that are being evaluated in this work. The layers denote the hidden layers that constitute a deep neural network. Finally, the output is the future value predicted for a particular time horizon.

Figure 1.

Memory-based RNN architecture for wind speed forecasting.

Figure 2 represents the Attention-based architecture for forecasting. The difference between the two architectures is the presence of the Attention layer. The hidden layer before the Attention layer consists of RNN nodes. However, the layer after does not necessarily require to be an RNN layer. The fully connected dense layer can also be used as the last hidden layer. As such, this layer is represented as DNN in the figure shown below.

Figure 2.

Attention-based RNN architecture for wind speed forecasting.

The wind speed forecast modeling in the univariate scenario takes the wind speed data at t-1+△ time, say $W S_{t - 1 + Δ}$ , as target and wind speed data separated by time lags, say $W S_{t - w}, W S_{t - w + 1}, \dots W S_{t - 1}$ , are the input variables. Here, w represents the length of the window for the time-lagged series. w is chosen based on the characteristics of data based on autocorrelation and partial autocorrelation functions. Overall, the future values of wind speed can be predicted by utilizing past and present values of wind speed according to the window length. Similarly, for solar irradiance forecasting, the solar irradiance data is fed into the network to forecast future values.

Empirical investigation with real data

This section represents and discusses the experimental results achieved on the different wind speed and solar irradiance datasets. Firstly, the dataset details used for experimental purposes are discussed. Then, performance evaluation metrics for testing the developed models is described. Finally, result analysis is performed on the performed experiments.

Dataset

The datasets used in the experiments are taken from the NREL (National Renewable Energy Laboratory [NREL], 2012) and NASA’s POWER (Prediction of Worldwide Energy Resources [POWER], 2019) data repository. The task of forecasting wind speed and solar irradiance is performed here considering two applications of renewable energy. The work focuses on univariate forecasting for different time horizons. The proposed methodology is the recurrent neural network forecasting technique, with comparisons between memory-based and Attention-based methods. The wind speed data is taken from wind farms with latitude and longitude (46.95513535,−107.0999908) and (48.80713654,−106.3518372), respectively, while the solar irradiance data was adopted from sites with coordinates (22.71961, 75.85771) and (23.25991, 77.41261) respectively. The characteristics of the wind speed and solar irradiance data are represented in Tables 1 and 2 for the different sites on which the experiments are performed.

Table 1.

Wind speed data characteristics.

Dataset #	Site ID	Year	Mean	Standard deviation	Min	Max
1	826,793	2012	8.965251	4.081776	0.030000	37.990000
2	869,927	2012	8.406120	3.607764	0.050000	25.420000

Table 2.

Solar irradiance data characteristics.

Dataset #	Year	Mean	Standard deviation	Min	Max
1	1983–2019	5.173958	1.383808	0.160000	8.660000
2	1983–2019	5.098533	1.417079	0.200000	8.410000

The wind speed data is an average of 5-min wind speed, while the solar irradiance data consists of daily data. For our experiments, observations are divided into training, validation, and testing pairs.

Performance evaluation metrics

The developed model is required to characterize the underlying patterns of actual data. Model evaluation is required for testing the model’s capacity to generalize. Hence, performance evaluation parameters are essential. For comparison of the models, based on the test performance, the metrics used are Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE). Table 3 state the equations of the performance metrics used for the evaluation of forecast performance where $y_{t}$ denote the actual wind speed data and $\bar{y_{t}}$ denote the predicted values of wind speed.

Table 3.

Performance evaluation metrics.

Metric	Equation
MAE	$\sum_{t = 1}^{n} \frac{\| y_{t} - \bar{y_{t}} \|}{n}$
RMSE	$\sqrt{\frac{1}{n} \sum_{t = 1}^{n} {(y_{t} - \bar{y_{t}})}^{2}}$

MAE measures the vertical distance between the predicted and actual values, while RMSE provides an error insight. Lower MAE and RMSE values denote better performance.

Results analysis

This section presents the results and analysis of the experiments performed on the wind and solar datasets. Tables 4 and 5 show the results for wind forecasting for the different number of hidden layers and a horizon length of 1 and 6 hours for both the datasets, respectively. Similarly, Tables 6 and 7 present the results for solar forecasting datasets with 1 day ahead and 4 days ahead forecasting. Tables 8 and 9 depict the results for the persistence, DNN, and Support Vector Regression (SVR) forecast models in comparison to the RNN based models. The best performing models in each case is represented in bold in Tables 4 –9.

Table 4.

Comparison of prediction errors for different forecast models on first wind speed dataset for different hidden layers and horizon length.

Models	1 hour ahead				6 hours ahead
	Two layers		Three layers		Two layers		Three layers
	MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE
RNN	0.03045	0.04196	0.03058	0.04194	0.06894	0.09066	0.06928	0.09110
GRU	0.03136	0.04276	0.03042	0.04186	0.07008	0.09145	0.07092	0.09172
LSTM	0.03654	0.04705	0.03729	0.04966	0.07160	0.09224	0.06974	0.09111
Cosine	0.03383	0.04537	0.03336	0.04433	0.06879	0.09029	0.07136	0.09222
Luong	0.03315	0.04401	0.03605	0.04655	0.07046	0.09209	0.07059	0.09165
SA	0.03446	0.04510	0.03240	0.04326	0.06841	0.08998	0.06924	0.09096

Table 5.

Comparison of prediction errors for different forecast models on second wind speed dataset for different hidden layers and horizon length.

Models	1 hour ahead				6 hours ahead
	Two layers		Three layers		Two layers		Three layers
	MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE
RNN	0.02967	0.03934	0.03167	0.04151	0.06098	0.08345	0.06142	0.08374
GRU	0.02744	0.03719	0.02778	0.03725	0.06093	0.08327	0.06133	0.08373
LSTM	0.02772	0.03764	0.02813	0.03836	0.06328	0.08504	0.06128	0.08379
Cosine	0.02588	0.03585	0.02750	0.03706	0.06107	0.08328	0.06105	0.08382
Luong	0.02661	0.03652	0.02918	0.03906	0.06210	0.08396	0.06057	0.08319
SA	0.03093	0.04309	0.03155	0.04252	0.06423	0.08728	0.06148	0.08354

Table 6.

Comparison of prediction errors for different forecast models on first solar irradiance dataset for different hidden layers and horizon length.

Models	1 day ahead				4 days ahead
	Two layers		Three layers		Two layers		Three layers
	MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE
RNN	0.05534	0.08806	0.06316	0.08837	0.06817	0.1012	0.06723	0.1029
GRU	0.05507	0.08617	0.05349	0.08561	0.06899	0.1011	0.06629	0.1011
LSTM	0.05599	0.08623	0.05459	0.08714	0.06784	0.1015	0.06940	0.1014
Cosine	0.05664	0.08777	0.05395	0.08734	0.06649	0.1014	0.06736	0.1008
Luong	0.05802	0.08734	0.05396	0.08736	0.06605	0.1019	0.06532	0.1007
SA	0.05764	0.09040	0.05626	0.08879	0.06716	0.1031	0.06989	0.1041

Table 7.

Comparison of prediction errors for different forecast models on second solar irradiance dataset for different hidden layers and horizon length.

Models	1 day ahead				4 days ahead
	Two layers		Three layers		Two layers		Three layers
	MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE
RNN	0.06463	0.09676	0.06429	0.09666	0.07714	0.1137	0.08079	0.1150
GRU	0.06358	0.09631	0.06446	0.09598	0.07589	0.1133	0.08085	0.1138
LSTM	0.06164	0.09592	0.06429	0.09616	0.07649	0.1133	0.07848	0.1138
Cosine	0.06489	0.09632	0.06397	0.09625	0.06649	0.1019	0.07432	0.1131
Luong	0.06553	0.09683	0.06188	0.09663	0.06605	0.1014	0.07538	0.1139
SA	0.06384	0.1029	0.06652	0.1019	0.06716	0.1032	0.07635	0.1141

Table 8.

Performance comparison for forecast models on wind speed datasets.

Models	Dataset 1		Dataset 2
	MAE	RMSE	MAE	RMSE
Persistence	0.03289	0.04455	0.02891	0.03944
SVR	0.03585	0.04618	0.03097	0.04152
DNN	0.03122	0.04238	0.02668	0.03667
RNN	0.03045	0.04196	0.02967	0.03934
GRU	0.03042	0.04186	0.02744	0.03719
LSTM	0.03654	0.04705	0.02772	0.03764
Cosine	0.03336	0.04433	0.02588	0.03585
Luong	0.03315	0.04401	0.02661	0.03652
SA	0.03240	0.04326	0.03093	0.04309

Table 9.

Performance comparison for forecast models on solar irradiance datasets.

Models	Dataset 1		Dataset 2
	MAE	RMSE	MAE	RMSE
Persistence	0.05529	0.09349	0.06359	0.1048
SVR	0.06137	0.09063	0.07035	0.1021
DNN	0.05731	0.08575	0.06310	0.09671
RNN	0.05534	0.08806	0.06463	0.09676
GRU	0.05349	0.08561	0.06358	0.09631
LSTM	0.05459	0.08714	0.06164	0.09592
Cosine	0.05395	0.08734	0.06397	0.09625
Luong	0.05396	0.08736	0.06188	0.09663
SA	0.05626	0.08879	0.06384	0.1029

Table 4 shows the forecast performance for the first wind speed dataset predicting 1 hour ahead and 6 hours ahead values. The table also shows the performance of the models on modifying the number of hidden layers. For 1 hour ahead of forecasting, it can be observed that the GRU model with three hidden layers performs the best. Other models also perform with good performance metrics, however, the GRU outperforms other models in this case. Similarly, for 6 hours ahead forecasting, it is the self-attention based models showing the best results. For the second wind speed dataset, the Attention-based models perform the best in both cases of different horizons. Also, increasing the number of hidden layers doesn’t contribute to better performance for forecasting one step ahead. A hidden layer number of 2 performs better in all the one-hour ahead forecasting cases than three hidden layers. For 1 hour ahead of forecasting, the cosine-based Attention gives the best performance metrics values, while for 6 hours ahead of forecasting, the Luong Attention performs the best. The RNN models, thus, are capable of performing well for the purpose of wind speed forecasting.

In solar irradiance forecasting, the GRU and LSTM perform better for one step ahead forecasting, while the Attention-based models show superior performance in multi-step ahead forecasting. The solar irradiance data consists of strong seasonality and is of a complex nature. But, the RNN models are able to model the data from both locations with good prediction performance. Daily data was utilized to forecast 1 day ahead and 4 days ahead of solar irradiance values. The shorter horizon performs better similar to the case of wind speed forecasting. However, hourly data was present in the case of wind time series. Despite the difference in time intervals and applications, the models prove to perform in a generalized manner in both the case.

The performance of models in case of one step ahead forecast is better than performance for multi-time step ahead. This is so since the recurrent neural networks are sequential models with the capacity of understanding temporal dependencies. All the RNN models can perform exceptionally well for the shorter horizon of time. The different variants were developed to overcome the challenges of vanishing and exploding gradients, such that the models could remember for a longer period of time. It also implies that models performing better in longer horizons are capable of capturing long-term dependencies more efficiently. The results for both wind speed and solar irradiance forecasting cases indicate the superior performance of Attention-based models. The performance achieved by the Attention mechanism is better in all the cases of time series forecasting for the longer horizon of time.

Modifying the number of hidden layers show changes in the performance metrics. In some cases, a lower number of hidden layers perform well, while in other cases, increasing the number of layers also improves the forecast performance. Regularization in the form of dropout is also applied to the models. This allows the model to overcome underfitting and overfitting, generalizing according to the data.

It can also be observed that the models’ performance accuracy is better in case of wind speed datasets compared to solar irradiance. This is so since the wind speed datasets are available in 5-minute intervals while solar irradiance datasets are updated every day. This data precision causes the difference in performance, indicating that data availability at shorter intervals of time can improve forecast performance.

Furthermore, Tables 8 and 9 represent the forecast results of the recurrent neural network models in comparison with the existing time series forecasting models of persistence model, Support Vector Regression (SVR) model, and Deep Neural Network (DNN) models. The forecast performance is shown for one step ahead forecasting in both the datasets. As it can be observed, it is the GRU, Attention-based mechanism, and LSTM that performs the best. The models in comparison are not able to perform as accurately as the RNN based models.

Overall, the RNN models, both memory-based and Attention-based, were used for modeling the wind and solar datasets with nonlinear, complex, and seasonal characteristics. The models could understand the temporal dependencies in the datasets and use them for forecast applications. The Attention-based mechanisms can capture the long term dependencies in multiple time steps and prove to be an enhancement to the traditional memory-based RNN since they outperform the traditional RNN models in many cases. Overall, the RNN models and Attention mechanisms can be utilized for practical wind speed and solar irradiance forecasting in a generalized and cost-effective manner.

Conclusion

Forecasting of wind speed and solar irradiance is required for assessing energy and selection of a site for a wind farm and solar parks. Popular recurrent neural network methods, namely RNN, GRU, LSTM, Content-based Attention, Luong Attention, and Self-Attention based RNN, are applied to model the forecast applications. The models were able to detect the underlying patterns of the datasets and forecast future values. The Attention models also captured the nonlinear and temporal dependencies of the wind speed and solar irradiance datasets. They were able to outperform the traditional RNN models in many cases. Also, the GRU model proved to be an effective forecast model for solar irradiance forecasts. Hence, the recurrent neural network model variants show good forecast performance capturing the long-term temporal dependencies, and the Attention mechanism proves to an enhancement to the memory-based RNNs. This review can be utilized for the planning of power stations and research developments.

Footnotes

Acknowledgements

The data used in the research were obtained from the NASA Langley Research Center (LaRC) POWER Project funded through the NASA Earth Science/Applied Science Program.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The data used in the research were obtained from the NASA Langley Research Center (LaRC) POWER Project funded through the NASA Earth Science/Applied Science Program. needs to be mentioned in the acknowledgment section, since it was mentioned in the official NASA POWER project’s website () to acknowledge through this sentence when using their dataset.

ORCID iD

Rajesh Wadhvani

References

Alzahrani

Shamsi

Ferdowsi

, et al. (2017) Solar irradiance forecasting using deep recurrent neural networks. In: Proceedings of the 6th international conference on renewable energy research and applications (ICRERA), San Diego, CA, USA, 5–8 November 2017, pp.88–994. New York: IEEE.

Amrouche

Pivert

(2014) Artificial neural network based daily local forecasting for global solar radiation. Applied Energy 130: 333–341.

Amrouche

Sicot

Guessoum

, et al. (2013) Experimental analysis of the maximum power point’s properties for four photovoltaic modules from different technologies: Monocrystalline and polycrystallinesilicon, CIS and CdTe. Solar Energy Materials and Solar Cells 118: 124–134.

Bahdanau

Cho

Bengio

(2014) Neural machine translation by jointly learning to align and translate. Preprint. arXiv:1409.0473.

Blanchard

Samanta

(2019) Wind speed forecasting using neural networks. Wind Engineering 44(1): 33–48.

Brahma

Wadhvani

(2020a) Solar irradiance forecasting based on deep learning methodologies and multi-site sata. Symmetry 12(11): 1830.

Brahma

Wadhvani

(2020b) Time series forecasting: A comparison of deep neural network techniques. Solid State Technology 63(6): 1747–1761.

Cheng

Dong

Lapata

(2016) Long short- term memory-networks for machine reading. Preprint. arXiv:1601.06733.

Cho

Bahdanau

Bougare

, et al. (2014) Learning phrase representations using RNN encoder decoder for statistical machine translation. Preprint. arXiv: 1406.1078.

10.

Chowdhury

Zhang

Messac

, et al. (2013) Optimizing the arrangement and the selection of turbines for wind farms subject to varying wind conditions. Renewable Energy 52: 273–282.

11.

Dongre

Pateriya

(2019) Power curve model classification to estimate wind turbine power output. Wind Engineering 43(3): 213–224.

12.

Elkinton

Manwell

McGowan

(2008) Algorithms for offshore wind farm layout optimization. Wind Engineering 32(1): 67–84.

13.

Gers

Schmidhuber

Cummins

(2000) Learning to forget: Continual prediction with LSTM. Neural Computation 12(6): 2451–2471.

14.

Géron

(2017) Hands-On Machine Learning with Scikit-Learn and TensorFlow. Newton, MA: O’Reilly Media, Inc.

15.

Goodfellow

Bengio

Courville

(2016) Deep Learning. Cambridge, MA: MIT Press.

16.

Graves

Greg

Ivo

(2014) Neural turing machines. Preprint. arXiv:1410.5401.

17.

Hochreiter

Schmidhuber

(1997) Long short-term memory. Neural Computation 9: 1735–1780.

18.

Kumar

Yagli

Kashyap

Srinivasan

(2020) Solar irradiance resource and forecasting: a comprehensive review. IET Renewable Power Generation 14(10): 1641–1656.

19.

Kushwah

Wadhvani

(2019) Performance monitoring of wind turbines using advanced statistical methods. Sādhanā 44: 163.

20.

Kushwah

Wadhvani

Kushwah

(2020) Trend-based time series data clustering for wind speed forecasting. Wind Engineering. Epub ahead of print 21 July 2020. DOI: 10.1177/0309524X20941180.

21.

Luong

M T

Pham

Manning

C D

(2015) Effective approaches to attention-based neural machine translation. Preprint. arXiv:1508.04025.

22.

Mostafaeipour

Goli

Rezaei

, et al. (2019) Performance of different hybrid algorithms for prediction of wind speed behavior. Wind Engineering. Epub ahead of print 24 October 2019. DOI: 10.1177/0309524X19882431.

23.

National Renewable Energy Laboratory (NREL) (2012) Eastern dataset. Accessed on: September 6, 2020. [Online]. Available at: https://www.nrel.gov/grid/wind-integration-data.html

24.

Patterson

Gibson

(2017) Deep Learning: A Practitioners Approach. Newton, MA: O’Reilly Media.

25.

Prediction of Worldwide Energy Resources (POWER) (2019) Renewable Energy, NASA Langley Research Center (LaRC) Hampton, VA (USA). Accessed on: September 6, 2020. [Online]. Available at: https://power.larc.nasa.gov/data-accessviewer/

26.

Rumelhart

Hinton

Williams

(1986) Learning representations by back-propagating errors. Nature 323: 533–536.

27.

Sharma

Kakkar

(2018) Forecasting daily global solar irradiance generation using machine learning. Renewable and Sustainable Energy Reviews 82(3): 2254–2269.

28.

Sheela

Deepa

(2013) A review on neural network models for wind speed prediction. Wind Engineering 37(2): 111–124.

29.

Shih

Sun

Lee

(2019) Temporal pattern attention for multivariate time series forecasting. Machine Learning 108: 1421–1441.

30.

Thapar

Agnihotri

Sethi

(2011) Critical analysis of methods for mathematical modelling of wind turbines. Renewable Energy 36(11): 3166–3177.

31.

Thapar

(2019) A revisit to solar radiation estimations using sunshine duration: analysis of impact of these estimations on energy yield of a PV generating system. Energy Sources, Part A: Recovery, Utilization, and Environmental Effects. Epub ahead of print 30 October 2019. DOI: 10.1080/15567036.2019.1677819.

32.

Unnikrishnan

Venugopal

(1994) Alopex: A correlation-based learning algorithm for feedforward and recurrent neural networks. Neural Computation 6(3): 469–490.

33.

Vaswani

Shazeer

Parmar

, et al. (2017) Attention is all you need. Preprint. arXiv:1706.03762.

34.

Wadhvani

Shukla

(2018) Analysis of parametric and non-parametric regression techniques to model the wind turbine power curve. Wind Engineering 43(3): 225–232.

35.

Yagli

Yang

Srinivasan

(2019) Automatic hourly solar forecasting using machine learning models. Renewable and Sustainable Energy Reviews 105: 487–498.