Machine learning based data governance methods for demand response databases

Abstract

With the increasing level of grid intelligence and the related demand response database expanding, it is important to study a compound problem data governance method for demand response, while the traditional data governance methods have problems such as not considering data temporality and ignoring the impact of noise and duplicate data on data repair. As a result, this project will develop an anomaly data extraction and repair model based on two-way long and short memory networks, and repair the anomaly data by respective noise smoothing, missing data filling, and duplicate data cleaning. The paper also provides an adaptive moment estimation approach for optimisation to raise the model’s accuracy. The outcomes demonstrated that the study model’s precision for anomalous data extraction was 100% and its recall rate was 80%, which was a significant improvement over the previous state. In terms of anomalous data repair, the research model had the root mean square error value and lowest mean absolute percentage error value when compared with related models, at 0.0049 MPa and 1.375% respectively. Both the abnormal data extraction and repair performance of the research model are greatly improved over the related models, and have important value in the abnormal data governance of demand response databases.

Keywords

Machine learning data extraction data repair Bi-LSTM Adam’s algorithm

1. Introduction

With the development of the electric power industry, as well as the continuous improvement of the level of grid intelligence, the user’s demand for electricity is also higher and higher, most of the electric power companies began to implement the management mechanism based on the user’s demand response, the demand response data continues to increase, the demand response database data anomalies have also increased, the study of a data governance method for the abnormal data of the demand response database is of great significance. Data governance research includes both abnormal data extraction techniques and abnormal data processing. In terms of abnormal data extraction, traditional machine learning methods have the problem of not considering the temporal characteristics of the data [1]. Most data cleaning techniques only take into account the removal of missing data when processing anomalous data, failing to take into account the effects of duplicate and noise-related data. Time-series data is also not taken into account by the model, which results in low prediction accuracy. Long- and short-term memory networks contain memory cell units and gating mechanisms that can preserve and selectively forget previous input information, capture long-distance dependencies in long-time sequences, and effectively deal with long-term dependencies. Therefore, the research will address the problem that traditional data governance methods do not consider the data time-series, which is not highly accurate, and establish a time-series data extraction and data cleaning model based on Long-Short Time Memory (LSTM) network to effectively deal with the composite abnormal data of demand response. The first part introduces the data extraction techniques based on demand response and the current research status of data cleaning methods; the second part is to build a model for temporal anomalous data based on Bidirectional Long Short-Term Memory Network (Bi-LSTM). In the second part, a model for extracting and repairing temporal anomalous data based on Bi-LSTM is developed, and the Adam algorithm is introduced to optimise the model.

2. Related works

In the area of anomalous data governance, many scholars have conducted related research. An image-based approach for wind turbine anomaly data identification and cleaning was proposed by Long et al. It entails three steps: data pre-cleaning, normal data extraction, and data tagging [2]. A comprehensive method for locating and eliminating outliers from wind power data was proposed by Luo et al. It makes use of the density difference between normal and outlier points to improve cleaning performance and combines boundary extraction and boundary regularization to completely eliminate boundary outliers [3]. The issue of employing raw skeletal data in skeleton-based anomalous gait recognition lowering recognition performance was addressed by Jun et al. The technique, which extracts features from skeletal gait data, increases the performance of skeleton-based anomalous gait recognition and has higher recognition accuracy, according to testing results [4]. The method for detecting network anomalies put out by Wang et al. combined principal component analysis with a single-stage face identification algorithm. It is proposed to use a single-stage face identification algorithm and principal component analysis to detect network anomalies. It has been empirically proven that this method greatly surpasses previous detection methods in terms of speed and accuracy [5]. Dong et al. address the problem that most current anomaly detection models are based on a common dataset for simulation experiments, and the common dataset contains too many kinds of data for traditional machine learning methods to handle [6].

Numerous researchers have also carried out pertinent research in the areas of data cleaning and data pre-processing. Wang et al. proposed a data cleaning method based on mobile edge nodes for data collection, the method obtained model training data through angle-based outlier detection method, and the method built a cleaning model through support vector machine to build a cleaning model. Experiments show that the method greatly improves the cleaning efficiency of data while maintaining data reliability [7]. Zyblewski et al. designed a data processing framework to first train the base classifiers using the hierarchical bagging method, and then integrated the data preprocessing and dynamic ensemble selection methods into the classification of unbalanced data streams. The study designed four preprocessing techniques and two dynamic selection methods for bagging classifiers and base estimators. The experimental results show that the dynamic ensemble selection method combined with data preprocessing outperforms the existing state-of-the-art methods for highly unbalanced data streams [8]. Wang et al. proposed a method for detecting and identifying anomalous data, in response to the problems of identifying and modifying state-based anomalous data, such as non-convergence in iterative computation and difficulty in handling large amounts of data effectively. The method builds a data constraint model based on temporal data similarity, and shows through experiments that the method can effectively detect and repair anomalous data [9]. Diffusion-weighted magnetic resonance imaging is the primary method for noninvasive study of white matter tissue in the human brain. Cieslak et al. constructed a preprocessing platform for diffusion image data that is compatible with all diffusion-weighted magnetic resonance imaging sampling modalities [10].

In conclusion, typical machine learning approaches have the issue of not taking into account the temporal properties of the data when it comes to anomalous data extraction. Most data cleaning techniques merely take into account the filling in of missing data while processing anomalous data, failing to take into account the effects of noise and duplicate data. Moreover, the lack of data temporality can lead to the loss of time-related key information; the model cannot distinguish the sequence of time and thus cannot accurately capture the temporal features of the data, leading to modeling errors and errors in feature importance, and lowering the accuracy of model prediction. To efficiently analyse the compound anomalous demand response data, this research will develop a temporal data extraction and data cleaning model based on Bi-LSTM network.

3. Bi-LSTM based anomaly data pre-processing method research

In the research on data governance, the traditional approach suffers from not considering the time-series characteristics of the data and the model prediction accuracy is not high enough. This research will build a machine learning based data extraction and cleaning model to correctly predict abnormal data and perform cleaning and repair operations on the data based on the demand response database.

3.1 Design of a Bi-LSTM-based data governance framework

The data anomalies of the demand response database mainly include missing data values, duplicate data values, inconsistent data and other anomalies. Only by carrying out data governance operations such as anomaly extraction and data cleaning on the data anomalies of the response database can the data of the demand response database be true and reliable and meet its proper value [11]. This research mainly classifies abnormal data into three categories, one is dead data, i.e. a situation where the data recorded in the database is always the same value without any change. The second is empty data, i.e. a situation where there are records with data that really do not contain data. The third is live data, i.e. data that can change continuously within a certain range and can be used normally. Dead data and empty data are reported as errors, and live data is extracted for further data governance operations. The abnormal data governance framework for the demand response database in this study is shown in Fig. 1.

Figure 1.

Abnormal data governance framework.

As shown in Fig. 1, the data governance solution for this study is broadly divided into 3 modules. The first is the classification and collation of the data, as the demand response database contains a large number of different types of data, so these different types of data need to be collated and classified before the cleaning can be carried out. The next step is to design the abnormal data extraction module of the model. Finally, the data repair module is put in place to carry out data repair operations on the extracted abnormal data. The study has selected three main data repair methods, namely data noise smoothing, missing data filling and duplicate data cleaning methods. The data in the response database has a certain amount of white noise, for the data anomalies caused by noise, the study will introduce the smooth function to carry out data repair, and the Generalized Cross-Validation (GCV) method is chosen to carry out the selection of smooth parameters [12, 13]. Equation (1) displays the smooth function’s mathematical expression.

$\displaystyle f(t)=\min\left(\lambda\int_{a}^{b}{(f^{\prime\prime}(t))^{2}dt}+% \sum\limits_{i=1}^{n}{(-f(t_{i})+y_{i})^{2}}\right)$ (1)

As in Eq. (1), $f(t)$ is the smooth function; $f^{\prime\prime}(t)$ is the second order derivative of the smooth function; $\lambda$ is the smooth parameter; $y_{i}$ and $t_{i}$ denote the $i$ th time series data; $n$ denotes the number of data points, and the specific time series data points are expressed as shown in Eq. (2).

$\displaystyle\{(t_{i},y_{i})\left|{i=1,2,\ldots,n}\right.\}$ (2)

Equation (2) is the expression formula for the temporal data points of the response database, and the GCV method was chosen for the study to carry out the selection of the smooth parameter $\lambda$ , which is calculated as shown in Eq. (3).

$\displaystyle\text{GCV}(\lambda)=\frac{1}{n}\cdot\frac{\sum\limits_{i=1}^{n}{% \left[{(y_{i}-f(t_{i}))}\right]^{2}}}{(1-n^{-1})}$ (3)

As in Eq. (3), $\text{GCV}(\lambda)$ represents the value of $\lambda$ selected using the GCV method. For similar duplicate data, the study chose the LSTM-based sliding window method to perform the duplicate data cleaning operation, and the formula for calculating the similar duplication rate of data is shown in Eq. (4).

$\displaystyle\textit{DRate}=d_{num}/w$ (4)

As in Eq. (4), $D$ represents the sorted dataset; DRate represents the similar repetition rate of the dataset $D$ ; $d_{\textit{num}}$ is the hypothetical window repetition data; $w$ is the window size and $w$ is calculated as shown in Eq. (5).

$\displaystyle w=(w\times(1-\textit{DRate)})+2$ (5)

This study employs a combined data filling strategy that is based on the GPR model and the LSTM neural network model; Eq. (6) gives the method’s mathematical expression.

$\displaystyle y_{T}=\frac{y_{\textit{T,GPR}}/S_{\textit{T,GPR}}+y_{\textit{T,% LSTM}}/S_{\textit{T,LSTM}}}{1/S_{\textit{T,GPR}}+1/S_{\textit{T,LSTM}}}$ (6)

As in Eq. (6), $y_{T}$ represents the mean of the data filling results of the combined model; $y_{\textit{T,GPR}}$ and $y_{\textit{T,LSTM}}$ represent the mean of the filling results of the GPR and LSTM models respectively, and $S_{\textit{T,GPR}}$ and $S_{\textit{T,LSTM}}$ represent the standard deviation of $y_{\textit{T,GPR}}$ and $y_{\textit{T,LSTM}}$ respectively. The LSTM method can effectively overcome the “delay” problem of recurrent neural networks and introduce the “forgetting gate” mechanism for the anomaly management model of demand response database, which enables the LSTM to achieve fast processing of long sequence data [14, 15]. In this study, the LSTM is used as the basis to build a Bi-LSTM model for anomalous data management, i.e. the forward and backward LSTMs are used to obtain the hidden information contained in the past and the future respectively, and the two parts of information together form the final output. Figure 2 depicts the LSTM model’s organisational structure.

Figure 2.

LSTM model structure diagram.

In Fig. 2, the product of vector elements is represented by $\otimes$ and the sum of vector elements is represented by $\oplus$ . The LSTM consists of three main gating units, which are the “forgetting gate,” “input gate” and “output gate.” When the data is processed, the input data will be processed from left to right starting from the “forgetting gate” and when a large amount of information is input, the data will be judged by the controller.

$\displaystyle f(t)=\sigma(\omega_{f}[h_{t-1},x_{t}]+b_{f})$ (7)

As in Eq. (7), $f(t)$ is the control function; $\sigma$ represents the Sigmoid activation function; $h_{t-1}$ represents the short term memory of the previous hidden layer; $x_{t}$ represents the input timing data, and $\omega_{f}$ and $b_{f}$ represent the bias of the weights of the forgetting gate respectively. The output control function of the final output gate is shown in Eq. (8).

$\displaystyle\sigma_{t}=\delta(\omega_{o}[h_{t},x_{t}]+b_{o})$ (8)

As in Eq. (8), $\sigma_{t}$ represents the current output control function value; $\delta$ represents the control coefficient; $\omega_{o}$ and $b_{o}$ represent the output gate weights and bias values; $h_{t}$ is the current output value, and $h_{t}$ is expressed as shown in Eq. (9).

$\displaystyle h_{t}=\sigma_{t}*\tanh^{-1}(c_{t})$ (9)

As in Eq. (9), $c_{t}$ is the current cell status value.

3.2 Design of anomaly data extraction and repair model based on Bi-LSTM with Adam optimization

In this study, the anomaly data governance will be divided into two modules: anomaly data extraction and anomaly data repair, for the building of the Bi-LSTM-based anomaly data governance model depicted in Fig. ‘1. For the anomaly data extraction module, the research will first normalise the input data set and then carry out anomaly data prediction extraction through the Bi-LSTM network model optimised by Adam’s algorithm the training process of the optimised Bi-LSTM network model is shown in Fig. 3.

Figure 3.

Optimized Bi-LSTM network model training process.

As in Fig. 3, the model first preprocesses the dataset and converts it into a 3D vector matching the model input, while setting the time step to 4 and the feature dimension to 5. Dropout is added under the Bi-LSTM layer in order to avoid overfitting during the training process. As data extraction is a two-class problem, the study set the excitation function of the dense layer to be a sigmoid function and the output of the function to be set between (0, 1). Also, the study chose the Adam algorithm to optimise the model training, and the mean square error (MSE) was chosen as the loss function of the model [16]. The normalisation formula for the data set is shown in Eq. (10).

$\displaystyle x^{\prime}_{i}=\frac{x_{i}-x_{\min}}{x_{\max}-x_{\min}}$ (10)

As in Eq. (10), $x^{\prime}_{i}$ is the normalized data; $x_{i}$ is the original data, and $x_{\min}$ and $x_{\max}$ represent the minimum and maximum values in the data set. The final prediction output formula of the Bi-LSTM model is shown in Eq. (11).

$\displaystyle h_{t}=\left({W_{t}[x_{t},h_{t-1},h_{t+1}]+b_{t}}\right)$ (11)

As in Eq. (11), $h_{t}$ denotes the hidden output value of node $t$ ; $h_{t-1}$ and $h_{t+1}$ denote the hidden outputs of the previous node and the next node; $b_{t}$ and $W_{t}$ denote the bias vector and weight value of the Bi-LSTM layer respectively, and $x_{t}$ is the input data of node $t$ . The abnormal data repair module mainly consists of two parts: abnormality identification and abnormality repair. The model will first discriminate the abnormal information extracted by the abnormal data extraction module, and then carry out the abnormal data repair by the data noise smoothing, missing data filling and duplicate data cleaning methods as shown in (1–6), and the specific model abnormality identification and repair process is shown in Fig. 4.

Figure 4.

Abnormality identification and repair program process.

As shown in Fig. 4, the general flow of the module is, firstly, to clean the extracted anomalous data one by one, and secondly, to calculate the residuals of the model, the formula for which is shown in Eq. (12).

$\displaystyle e_{i}=y_{i}-y^{\prime}_{i}$ (12)

As in Eq. (12), $y_{i}$ is the measured value; $y_{i}^{\prime}$ is the predicted value; $e_{i}$ is the residual value predicted by the model. After that, the input data will be discriminated as anomalous and the residual value will be compared with the discriminant threshold, and a residual value greater than the threshold is considered as anomalous data and needs to be repaired. The data will then be repaired according to the anomaly repair algorithm. Finally, the repaired data values are returned to the model, the input data and the model are updated and the operation is repeated until all the anomalous values have been repaired.

In this study, two categorical evaluation metrics, Precision and Recall, will be introduced to assess the effectiveness of abnormal data extraction, and Relative Error, Mean Absolute Percentage Error (MAPE) and Root Mean Squared Error (RMSE) will be introduced to assess the effectiveness of data restoration of the model [17]. The relative error is mainly used to assess the degree of fit between the measured and predicted values, and the relevant formula is shown in Eq. (13).

This study will introduce two classification evaluation indicators, Precision and Recall, to evaluate the extraction effect of abnormal data. Relative error, Mean Absolute Percentage Error (MAPE), and Root Mean Squared Error (RMSE) will be introduced to evaluate the data repair effect of the model [17].

$\displaystyle\delta_{i}=\frac{\varepsilon_{i}}{y_{i}}\times 100\%=\frac{y_{i}^% {\prime}-y_{i}}{y_{i}}\times 100\%$ (13)

As shown in Eq. (13), $\delta_{i}$ is the relative error of the model; $\varepsilon_{i}$ is the residual between $y_{i}$ and $y_{i}^{\prime}$ . The smaller the absolute value of $\delta_{i}$ , the better the prediction. The smaller the absolute value of MAPE [18], the better the prediction. MAPE is used to assess the overall predictive performance of the model, assessing the difference between the predicted value and the corresponding measured value, and is calculated as shown in Eq. (14).

$\displaystyle\text{MAPE}=\frac{1}{n}\sum\limits_{i=1}^{n}{\left|{\frac{y^{% \prime}_{i}-y_{i}}{y_{i}}}\right|}\times 100\%$ (14)

As in Eq. (14), $n$ indicates the number of samples and a smaller value of MAPE indicates a better prediction of the model. RMSE represents the degree to which the actual value comes to the predicted value and assesses how well the model performs overall. The mathematical expression of the model is shown in Eq. (15) [19].

$\displaystyle\text{RMSE}=\sqrt{\frac{1}{n}}\sum\limits_{i=1}^{n}{(y^{\prime}_{% i}-y_{i})^{2}}$ (15)

The smaller the value of RMSE, the less discrete the model is and the better the model prediction.

4. Analysis of the results of anomalous data governance based on Bi-LSTM neural network model

To test the viability of the research model and evaluate how effectively it operates in terms of both anomalous data extraction and anomalous data repair, the study will conduct data governance experiments by actually responding to anomalous information from the database. The hardware and software requirements for this experiment are specified in Table 1, and the model must be trained in a specific experimental environment.

Table 1
Experimental software and hardware environment

Experimental environment	Configuration information
Memory	192 G
Processor	Intel (R) Core (TM) i7-6700
Operating system	Ubuntu 16.04.6 LTS
GPU	GTX080Ti
Programming Language	Python 3.x
Deep learning framework	Keras, TensorFlow

4.1 Analysis of anomalous data extraction results based on Bi-LSTM

The raw data for this experiment came from the real load demand response database of Chaoyang City, Liaoning Province, China, and a total of 2000 demand response data for one month were arbitrarily selected as the experimental raw data for the abnormal data extraction and repair experiments. To confirm the model’s anomalous data repair impact, model training experiments were conducted using the first 1500 of these data as the training group and the final 500 as the test group. The sample situation of the raw data is shown in Fig. 5.

Figure 5.

Abnormality identification and repair program process.

Figure 5 shows the raw data situation of the training and test sets for the experiment, where the training set data will be processed by the Bi-LSTM model optimised by Adam’s algorithm and the test set data will be predicted by the unimproved LSTM model. The anomalies will first be extracted from the raw data by the experiments, and the discriminant rule for the anomalies must be established by the standard deviation of the model’s residuals. Figure 6 displays the cumulative probability plot of the training and test models’ residuals.

Figure 6.

Model residual cumulative probability graph.

As shown in Fig. 6, the training errors of the improved model were more concentrated than before the improvement, and the training errors of the model before and after the improvement were better fitted to the normal reference line, and the standard deviation of the residuals calculated by the model was about 0.005 MPa. The training error of most of the data is within the error interval ( $-$ 1000, 1000), and when the error is outside this interval, the data is judged not to be abnormal, i.e. the discriminatory rule for abnormal data is whether the training error is within this interval. The study will introduce the accuracy precision, recall and loss value to evaluate the abnormal data extraction effect of the model, and the evaluation results are shown in Fig. 7.

Figure 7.

The loss and accuracy of the model on training and testing sets.

Figure 7 demonstrates that the model’s accuracy on the training set was 100%, an improvement of 30.5% over the test set, while the recall was 80%, an improvement of 25.5% over the test set, and the model loss was 7.1%, a reduction of 46.2% over the test set. Secondly, the improved model reached convergence more quickly and the accuracy of the model training reached convergence the fastest. Overall, the model reached convergence in accuracy, recall and loss at approximately 300 iterations, at which point the model reached optimality. The demand response data is time-series data, and the model will test identifying continuous outliers on the original data. The results of this outlier extraction are displayed in Fig. 8.

Figure 8.

Abnormal data extraction results of the model before and after improvement.

As shown in Fig. 8(a), the time series LSTM model before the improvement has obvious cases of missed and false detection, and the first anomalies of a section of continuous anomalous data are all identified accurately, the locations of the actual anomalies and the predicted anomalies do not consistently correspond to each other in Fig. 8(a). Which is due to the fact that the model will be influenced by the time series features and there is a certain lag, resulting in easy false detection and missed detection in the next prediction of the anomalous data. As shown in Fig. 8(b), this research has improved the model through Adam’s algorithm by avoiding the influence of data lagging through bi-directional LSTM improvement, which has greatly improved the recognition accuracy of anomaly data.

4.2 Analysis of anomaly data repair results based on Bi-LSTM

The study will perform data repair process on the anomalous data based on the anomalous data extraction results of the model. On the same dataset as the research model, experiments will introduce Back Propagation (BP) neural network model and Support Vector Machine (SVM) model in order to compare and analyse the data repair effect of the research model. They will also introduce the parameters RMSE and MAPE in order to assess the model’s performance. The first step of anomaly repair is to discriminate the type of anomalies, and the repair operation can only be performed if the anomalous data is live data. The accuracy, precision and recall of the anomaly discrimination of each model are shown in Fig. 9.

Figure 9.

Abnormal point discrimination result graph.

As shown in Fig. 9, firstly, the research improved Bi-LSTM model had the highest accuracy, precision and recall among the experimental models and both reached convergence the fastest, with 100% precision, 95.07% accuracy and 80% recall, respectively. Secondly, the BP and SVM models had higher precision of 90.39% and 94.71% respectively, while the recall was lower, almost 10% less than the study model, while the unimproved LSTM model had the lowest relative performance. In conclusion, overall, the research models improved the performance of the model to some extent, with both accuracy and recall significantly improved over the common models. Next, data repair experiments such as noise smoothing, data filling and duplicate data cleaning will be performed on the anomalous data, and the repair effect will be evaluated by RMSE and MAPE. The data repair evaluation results of the model are shown in Table 2.

Table 2

Abnormal data repair evaluation results

Model	RMSE (MPa)	MAPE (%)
Bi-LSTM	0.0049	1.375
LSTM	0.008	2.463
BP	0.006	1.684
SVM	0.006	1.684

As shown in Table 2, the improved Bi-LSTM model had the best anomaly data repair performance with the lowest RMSE and MAPE values of 0.0049 MPa and 1.375%, respectively. The BP and SVM models both had equal RMSE and MAPE values of 0.006 MPa and 1.684%, respectively. The unimproved LSTM model had the worst repair performance, with RMSE and MAPE values of 0.008 MPa and 2.463% respectively, while the study model reduced the RMSE value by 0.0031 MPa and the MAPE value by 1.088% compared to the pre-improvement model. Overall, the anomalous data repair effectiveness of the research model was improved over both the pre-improvement LSTM model, which is a machine learning method with higher correlation performance. Data repair experiments were conducted on the anomalous data according to the research model, and the results of the experimental data repair compared with the actual data are shown in Fig. 10.

Figure 10.

Abnormal data repair result graph.

As shown in Fig. 10(a), the restoration values of the studied improved model fit the actual values well, and the restoration effect of the model is in line with the real situation. As in Fig. 10(b), the unimproved LSTM model suffers from too high repair values at the crests, which may be due to errors in the model’s preservation of information about the extreme values of the data. At the same time, both models have the problem that the restoration values at the troughs are too flat and some of the fluctuation information is lost. Overall, the overall repair results of the improved model are higher than those of the pre-improved model, and the anomalous data repair is more effective.

5. Conclusion

In order to meet the power industry’s demand for demand response data processing needs, the study is based on LSTM networks to design anomalous data extraction and revision models for effective processing of composite anomalous data for demand response, and relevant algorithms are introduced for comparative experiments. The experimental results show that, firstly, in terms of anomalous data extraction, the standard deviation of the residuals of the research model is around 0.005 MPa, with an error interval ( $-$ 1000, 1000), i.e. when the error is outside this interval, the data is judged to be not anomalous data. The precision of abnormal data extraction of the study model was 100%; the recall rate was 80%, which were 30.5% and 25.5% higher than before the improvement, and the model loss was 7.1%, which was 46.2% lower than before the improvement, i.e. the improved model greatly improved the abnormal data extraction effect. Secondly, in terms of anomalous data repair, the improved Bi-LSTM model had the best anomalous data repair performance, with the lowest RMSE and MAPE values of 0.0049 MPa and 1.375% respectively, which were 0.0031 MPa and 1.088% lower than the RMSE value and MAPE value before the improvement, and lower than the BP and SVM models with RMSE values of 0.002 MPa and 1.088% reduction in MAPE value, i.e. the study model is more effective in anomaly data repair. At the same time, the study mainly conducted a comparative study of prediction-based data restoration methods and did not compare the data restoration methods with other types, such as clustering and filtering, which could be investigated in this direction in the future.

Footnotes

Acknowledgments

The work was financially supported by the 2023 Open Fund Project of the Beijing Key Laboratory of Demand Side Multi-Energy Carriers Optimization and Interaction Technique (Research on Construction and Data Governance Method of Demand Response Surveying Database Based on Machine Learning, YDB51202301442).

References

Dai

Wang

Wan

Imran

. Big data analytics for manufacturing internet of things: opportunities, challenges and enabling technologies. Enterp Inf Syst-UK. 2020; 14(9-10): 1279-1303.

Long

Sang

. Image-based abnormal data detection and cleaning algorithm via wind power curve. IEEE T Sustaine Energ. 2020; 11(2): 938-946.

Luo

Fang

Liu

. Method for cleaning abnormal data of wind turbine power curve based on density clustering and boundary extraction. IEEE T Sustain Energ. 2022; 13(2): 1147-1159.

Jun

Lee

Kim

. Feature extraction using an RNN autoencoder for skeleton-based abnormal gait recognition. IEEE Access. 2020; 8: 19196-19207.

Wang

Han

Liu

Cui

. The abnormal traffic detection scheme based on PCA and SSH. Connect Sci. 2022; 34(1): 1201-1220.

Dong

Xia

Peng

. Network abnormal traffic detection model based on semi-supervised deep reinforcement learning. IEEE Trans Netw Serv. 2021; 18(4): 4197-4212.

Wang

Zheng

Wang

Sangaiah

Liu

. Big data cleaning based on mobile edge computing in industrial sensor-cloud. IEEE T Ind Inform. 2020; 16(2): 1321-1329.

Zyblewski

Sabourin

Woźniak

. Preprocessed dynamic classifier ensemble selection for highly imbalanced drifted data streams. Inform Fusion. 2021; 66(2): 138-154.

Wang

Cao

. A method for cleaning power grid operation data based on spatiotemporal correlation constraints. IEEE Access. 2020; 8: 224741-224749.

10.

Cieslak

Cook

Yeh

Dhollander

Adebimpe

Aguirre

Bassett

Betzel

Bourque

Cabral

Davatzikos

Detre

Earl

Elliott

Fadnavis

Fair

Foran

Fotiadis

Garyfallidis

Giesbrecht

Gur

Kelz

Keshavan

Larsen

Luna

Mackey

Milham

Oathes

Perrone

Pines

Roalf

Richie-Halford

Rokem

Sydnor

Tapera

Tooley

Vettel

Yeatman

Grafton

Satterthwaite

. QSIPrep: an integrative platform for preprocessing and reconstructing diffusion MRI data. Nat Methods. 2021; 18(7): 775-778.

11.

Daryaee

Ahmadi

Peykani

Zayeri

. Prediction of longitudinal and transverse profiles of pressure flushing cones using artificial intelligence and data pre-processing. Water Supply. 2022; 22(2): 1533-1545.

12.

Paneiro

Rafael

. Artificial neural network with a cross-validation approach to blast-induced ground vibration propagation modeling. Undergr Space. 2021; 6(3): 281-289.

13.

Rabinowicz

Rosset

. Cross-validation for correlated data. J Am Stat Assoc. 2022; 117(538): 718-731.

14.

Priyadarshini

Cotton

. A novel LSTM–CNN–grid search-based deep neural network for sentiment analysis. J Supercomput. 2021; 77(11): 13911-13932.

15.

Wang

Celebi

Zhang

Yao

Zhou

Miguel

Tian

Gorriz

Tyukin

. Advances in data preprocessing for biomedical data fusion: an overview of the methods, challenges, and prospects. Inform Fusion. 2021; 76: 376-421.

16.

Zerkouk

Chikhaoui

. Spatio-temporal abnormal behavior prediction in elderly persons using deep learning models. Sensors-Basel. 2020; 20(8): 2359.

17.

Tang

Bao

. Group sparsity-aware convolutional neural network for continuous missing data recovery of structural health monitoring. Struct Health Monit. 2021; 20(4): 1738-1759.

18.

Liu

Zhang

Sun

Cui

Wang

. Stacked bidirectional LSTM RNN to evaluate the remaining useful life of supercapacitor. Int J Energ Res. 2022; 46(3): 3034-3043.

19.

Wang

Cao

. Deep learning for spatio-temporal data mining: A survey. IEEE T Knowl Data En. 2022; 34(8): 3681-3700.

Machine learning based data governance methods for demand response databases

Abstract

Keywords

1. Introduction

2. Related works

3. Bi-LSTM based anomaly data pre-processing method research

3.1 Design of a Bi-LSTM-based data governance framework

Table 1 Experimental software and hardware environment

Footnotes

Acknowledgments

References

Table 1
Experimental software and hardware environment