The multi-dimensional power big data mining based on improved grey clustering algorithm

Abstract

In order to overcome the problems of the traditional power big data mining methods, such as the low integrity of data mining and the long time-consuming of data mining, this paper realizes multi-dimensional power big data mining by improving the grey clustering algorithm. Firstly, a relay multi hop network is established to collect power big data through the collector; Secondly, Lagrange interpolation method is used to fill the missing data of power data mining; Standardized processing of power consumption data; Finally, according to the grey theory and FCM clustering algorithm, the multi-dimensional power big data mining is realized. The experimental results show that the integrity of power big data mining in this method is up to 0.996, the mining time is not more than 3.05 s, and the mining integrity is up to 0.992, which indicates that this method can effectively improve the effect of power big data mining.

Keywords

Improved grey clustering algorithm data mining normalization treatment objective function FCM clustering algorithm

1. Introduction

With the deepening of intelligence into the construction of power grid, four types of data center platforms have been successfully built: massive historical and quasi real-time, unstructured, structured, and grid space; The platform that has accumulated rich big data resources can provide effective support for data analysis and decision-making system construction. Multi dimensional power big data mining is to mine the required power data from the central platform, and use the mined data as reference data to improve the stability of power grid operation [8]. Traditional data management methods only analyze the acquired data simply, and can not help power enterprises obtain valuable and meaningful information from these data, resulting in a great waste of power grid resources [1,14]. Therefore, it is urgent to use data mining technology to deeply mine and analyze these huge power data, and it is extremely important and meaningful to obtain the power consumption information that is instructive and helpful to the operation and management of power enterprises.

Reference [6] proposes a power big data mining method based on GRU-MMD. The feature difference of power data is obtained through the maximum mean difference method. The GRU-MMD method is used to realize power data mining. This method can improve the integrity of data mining, but the efficiency of data mining is poor. Reference [15] proposes a power data mining algorithm based on clustering analysis, which obtains the attribute characteristics of power data through data clustering method, determines the difference of the attribute characteristics of power data according to the dissimilarity matrix, divides the attribute characteristics of power data through analytic hierarchy process, calculates the confidence of the attribute characteristics of power data, and obtains the final power data mining result. This method can effectively improve the integrity of data mining, But the mining algorithm is complex and the mining process is long. Reference [4] proposes an intelligent mining method of power big data based on web crawler technology. It uses web crawler technology to crawl power grid big data, obtains feature clusters of power grid big data according to web crawler clustering, realizes power grid big data sorting according to decision tree method, and realizes mining data transmission control according to mining permission matching. This method can effectively improve the integrity of data mining, but the efficiency of power big data mining is low.

In view of the above problems, this paper proposes a multi-dimensional power big data mining method based on improved grey clustering algorithm. The specific research ideas of this paper are as follows:

First, power big data acquisition. Establish a relay multi hop network, collect the power data read by the meter node through the collector, and send it to the server through 4G/5G, Ethernet and other remote communication networks.

Secondly, multi-dimensional power big data feature processing. Obtain the location of power missing data, fill the multi-dimensional power consumption data missing according to Lagrange interpolation method, normalize the interpolated power consumption data through linear transformation, and complete the multi-dimensional power big data feature missing value processing.

Then, multi-dimensional power big data mining. According to FCM clustering algorithm mining multi-dimensional power big data features, build multi-dimensional power big data mining function, calculate relevant parameters according to the least square method, combined with gray theory to improve the results of multi-dimensional power big data mining, and complete multi-dimensional power big data mining.

Finally, the effect of multi-dimensional power big data mining is verified by the indicators of data mining integrity, data mining time and completeness, and a conclusion is drawn.

2. Multi dimensional power big data processing

2.1. Multi dimensional power big data collection

As the central link of power data collection, the collector node not only needs to establish a relay multi hop network to communicate with all the meter nodes in its network to obtain the power data they read, but also needs to communicate with the server through 4G/5G and other remote communication networks to obtain the instructions issued by the server or forward the power data to the server [7]. Its hardware structure is shown in Fig. 1.

Fig. 1.

Hardware block diagram of collector node.

Transmit the data obtained from Lora communication network to the collector base table for storage and processing. MCU module also selects STM32F103 series chip as the main control chip to support and control the data processing, sending, receiving and other functions of the whole Lora module. The RF module uses SX1278 chip, which mainly supports the sending and receiving function of Lora wireless communication. It is the core part of Lora for wireless packet sending and receiving.

The main process of collector work is: with the support of RF module and MCU module, establish relay multi hop network, collect the power data read by all meter nodes in the network, process and save the data, and send it to the server through 4G/5G, Ethernet and other remote communication networks [5]. It can be seen that the collector is somewhat different from the meter node in terms of workflow. The collector does not only need to communicate with the meter node under it through the Lora communication network. The uplink communication module is pluggable. The shell has communication module components, which can be 4G, WiFi, etc. At the same time, only one can be used. Considering that the working voltage of the module may be different, there is a special power chip on the circuit to supply power to the module, and the level can be adjusted.

2.2. Multi dimensional power big data feature processing

In the process of extracting user electricity data from the power user electricity information collection system, there are missing values in the collected data due to human or natural factors. If these missing data are deleted directly, it will lead to incomplete power consumption information of users. This will directly affect the abnormal detection model to judge the user’s power consumption behavior, and even lead to missed judgment, misjudgment and so on. Therefore, it is necessary to deal with the missing data in the collected data [10]. Due to the compact structure of Lagrange interpolation method, it is convenient to calculate. Therefore, this paper uses Lagrange interpolation method to interpolate and fill the missing user electricity data. The specific steps are as follows:

Firstly, the missing data location of the collected multi-dimensional power big data is extracted, and the five data before and after the location are divided to obtain a multi-dimensional power big data group with 10 data. The missing data of power big data is interpolated and filled according to the Lagrange interpolation method. The specific formula is shown in formula (1) and formula (2): $\begin{array}{l} (1) & L_{n} (x) = \sum_{i = 0}^{n} l_{i} (x) y_{i} \\ (2) & l_{i} (x) = \frac{(x - x_{0}) \dots (x - x_{i - 1}) (x - x_{i + 1}) \dots (x - x_{n})}{(x_{i} - x_{0}) \dots (x_{i} - x_{i - 1}) (x_{i} - x_{i + 1}) \dots (x_{i} - x_{n})} \end{array}$

Among them, x represents the serial number of missing load data of multi-dimensional power data, $L_{n} (x)$ represents the interpolation function of missing load data of multi-dimensional power big data, $x_{i}$ represents missing load data of multi-dimensional power data, and $y_{i}$ represents non missing load data of multi-dimensional power data [2].

There are usually different dimensions and orders of magnitude for different power consumption indicators. When the level difference between indicators is great, if the original data is directly used for power consumption anomaly detection and analysis [9,11,16].

There are three methods for data standardization: Z-score standardization, log function standardization, and max min standardization. In this paper, the minimum maximum standardization method is used to normalize the collected power consumption data, that is, the power consumption data after missing data interpolation is linearly transformed, so that the processed power consumption data is between 0 and 1. The specific calculation formula is as follows: $\begin{matrix} (3) & v_{new} = \frac{v - v_{min}}{v_{max} - v_{min}} \end{matrix}$

Among them, $v_{new}$ is the standardized power consumption data, v is the user’s original power consumption data, $v_{min}$ is the minimum value in the original power consumption data, and $v_{max}$ is the maximum value in the original power consumption data, so as to realize multi-dimensional power big data feature processing.

In order to obtain a large number of multi-dimensional power big data, the collection, storage and transmission of power data are completed through data collectors and communication networks. In order to improve the mining accuracy of multi-dimensional power big data, the Lagrange interpolation method is used to fill the missing power data, and the multi-dimensional power data feature processing is completed through linear changes.

3. Multi dimensional power big data improved grey clustering mining

3.1. Multi dimensional power big data clustering mining

FCM clustering algorithm is a flexible fuzzy division method. Through hard division of C-means (HCM), multi-dimensional power big data clustering mining objective function is: $\begin{matrix} (4) & J (U, C) = \sum_{i = 1}^{n} \sum_{j = 1}^{c} u_{i j}^{m} ‖ x_{i} - c_{j} ‖^{2} \end{matrix}$

Where, $‖ x_{i} - c_{j} ‖$ is the Euclidean distance between the data object and the cluster center $c_{j}$ [3]. In order to find the best value of $(U, C)$ , that is, the smallest value of $J (U, C)$ , it is necessary to calculate the membership degree and clustering center of the data set.

Membership $u_{i j}$ calculation formula: $\begin{matrix} (5) & u_{i j} = \{\begin{matrix} 1 & d_{i j} = {min}_{1 ⩽ r ⩽ c} d_{i r} \\ 0 & else \end{matrix} \end{matrix}$

Calculation formula of cluster center $c_{j}$ : $\begin{matrix} (6) & c_{j} = \frac{\sum_{i = 1}^{n} u_{i j} x_{i}}{\sum_{i = 1}^{n} u_{i j}}, j = 1, 2, \dots, c \end{matrix}$

The steps of FCM clustering algorithm are:

input the multi-dimensional power big data clustering number c and the iteration termination condition;

initialize cluster center $c_{j}^{0}$ ;

calculate membership matrix $u_{i j}$ and cluster center matrix $c_{j}^{1}$ ;

if $‖ c_{j}^{1} - c_{j}^{0} ‖ ⩽ ε$ , the algorithm ends and outputs $(U, C)$ , otherwise $c_{j}^{0} - c_{j}^{1}$ , and jumps to step 3 to continue iterative calculation to obtain the multi-dimensional power big data clustering mining results, so as to realize multi-dimensional power big data clustering mining.

3.2. Grey theory improvement of multi-dimensional power big data mining

Grey system refers to a system with grey nature, that is, the indicators of system data can not fully reflect the data information or have uncertainty. By combining grey theory, FCM clustering algorithm is improved to effectively mine power data [13]. The improvement process of power data mining function is as follows:

The power equipment collects $X^{(0)} = {X^{(0)} (1), X^{(0)} (2), \dots, X^{(0)} (n)}$ as the original data and accumulates the data sequence $X^{(0)}$ to obtain a new sequence: $\begin{matrix} (7) & \begin{matrix} X^{(1)} = {X^{(1)} (1), X^{(1)} (2), \dots, X^{(1)} (n)} \\ X^{(1)} (1) = X^{(0)} (1) \\ X^{(1)} (2) = X^{(0)} (1) + X^{(0)} (2) \\ \dots \\ X^{(1)} (n) = X^{(0)} (n - 1) + X^{(0)} (n) \end{matrix} \end{matrix}$

In the above formula, $X^{(1)}$ represents the accumulated value of multi-dimensional power big data mining. Through the verification of level ratio and smooth ratio, judge whether the gray mining function can be constructed. If the level ratio and smooth ratio meet formula (7), the gray mining function $ρ (k)$ can be constructed for the data sequence: $\begin{array}{l} (8) & ρ (k) = \frac{X^{(0)} (k - 1)}{X^{(0)} (k)} \in (e^{- \frac{2}{n + 1}}, e^{\frac{2}{n + 1}}), k = 2, 3, \dots, n \\ (9) & X^{(0)} (k) / X^{(1)} (k - 1) \in (0, 0.5), k ⩾ 3 \end{array}$

The original form of $GM (1, 1)$ is: $X^{(0)} (k) + a X^{(1)} (k) = b$ . The least square method is used to calculate parameter a, b. $\begin{matrix} (10) & {(a, b)}^{T} = {(B B^{T})}^{- 1} B^{T} Y \end{matrix}$

Where: $\begin{matrix} (11) & B = (\begin{matrix} - 0.5 [X^{(1)} (2) + X^{(2)} (1)] & 1 \\ - 0.5 [X^{(2)} (2) + X^{(3)} (1)] & 1 \\ \dots & \dots \\ - 0.5 [X^{(n)} (2) + X^{(n +)} (1)] & 1 \end{matrix}), Y = (\begin{matrix} X^{(0)} (2) \\ X^{(0)} (3) \\ \dots \\ X^{(0)} (n) \end{matrix}) \end{matrix}$

Build an approximate differential equation to solve the original form of multi-dimensional power big data gray mining function: $\begin{matrix} (12) & d X^{(1)} / d t + a X^{(1)} (t) = b \end{matrix}$

In the above formula, a represents the development coefficient of grey mining function [12], that is, the sequence development trend of multi-dimensional power big data. b represents the grey action of the grey mining function, that is, the relationship between multi-dimensional power big data changes; t represents the time utilization factor

According to equations (11) and (12), the time response function $X^{(1)} (t)$ is: $\begin{matrix} (13) & X^{(1)} (t) = (X^{(1)} (1) - b / a) e^{- a t} + b / a \end{matrix}$

The time response sequence of $GM (1, 1)$ model $X^{(0)} + a z^{(1)} (k) = b$ is: $\begin{array}{l} (14) & X^{(1)} (1) = X^{(0)} \\ (15) & X^{(1)} (k + 1) = (X^{(0)} (1) - b / a) e^{- a k} + b / a, k = 1, 2, \dots, n \end{array}$

At this time, the final mining result $X^{(0)} (k + 1)$ of the multi-dimensional power big data improved by the grey clustering algorithm is: $\begin{matrix} (16) & X^{(0)} (k + 1) = X^{(1)} (k + 1) - X^{(1)} (k) = (1 - e^{a}) (X^{(0)} (1) - b / a) e^{- a k}, k = 1, 2, \dots, n \end{matrix}$

The solution of equation (15) is the final mining result of multi-dimensional power big data. Therefore, the multi-dimensional power big data mining research based on improved grey clustering algorithm is realized.

4. Experiment

The above process completes the theoretical research of multi-dimensional power big data mining. In order to verify the practical application performance of this method, the performance test of the method is carried out.

4.1. Experimental data

In order to meet the experimental requirements of multi-dimensional power big data mining, the multi-dimensional power big data used in this experiment comes from the publicly available data of the city’s power grid, including power generation data, power consumption data, power loss data, electricity price data and many other power big data. data, the overall data volume reaches 250 GB.

This section is based on the MATLAB platform to verify the feasibility and effectiveness of the multi-dimensional power big data mining method described above. For the multi-dimensional power big data mining problem, the Yamip toolbox is used to solve the multi-dimensional power big data mining. This paper improves the gray clustering algorithm under the Tensorflower2.0 framework in the Python3.7 environment.

4.2. Experimental scheme

In order to fully verify the mining performance of the proposed mining method, the experimental scheme is strictly set before the experiment starts. Comparing the indicators, the method in this paper is compared and verified with the methods in reference [6] and reference [15].

4.3. Experimental result

4.3.1. Integrity of power big data mining

In order to verify the mining effect of this method on power big data, GRU-MMD method (reference [6] method), cluster analysis method and this method are used to obtain the integrity of power big data mining. The results are shown in Fig. 2.

Fig. 2.

Integrity of power big data mining.

According to the results in Fig. 2, when the amount of power data to be mined is 50 GB, the integrity of power big data mining of reference [6] method is 0.265, the integrity of power big data mining of reference [15] method is 0.200, and the integrity of power big data mining of this method is 0.950; When the amount of power data to be mined is 100 GB, the integrity of power big data mining of reference [6] method is 0.432, the integrity of power big data mining of reference [15] method is 0.351, and the integrity of power big data mining of this method is 0.953; When the amount of power data to be mined is 300gb, the integrity of power big data mining of reference [6] method is 0.682, the integrity of power big data mining of reference [15] method is 0.478, and the integrity of power big data mining of this method is 0.982; This method can effectively improve the integrity of power big data mining and the effect of power big data mining.

4.3.2. Power big data mining takes time

In order to verify the mining efficiency of this method for power big data, the reference [6] method, reference [15] method and this method are used to obtain the time-consuming of power big data mining. The results are shown in Table 1.

Table 1
Time consumption of power big data mining with different methods (s)

Large amount of power data / GB Power big data mining with different methods takes time/s

Proposed method Reference [6] method Reference [15] method

50 0.62 16.65 8.92

100 1.28 28.69 16.28

150 1.68 58.91 38.96

200 2.32 88.25 69.25

250 3.05 129.92 99.63

Large amount of power data / GB	Power big data mining with different methods takes time/s
50	0.62	16.65	8.92
100	1.28	28.69	16.28
150	1.68	58.91	38.96
200	2.32	88.25	69.25
250	3.05	129.92	99.63

According to Table 1, when the amount of power data to be mined is 50 GB, the power big data mining time of reference [6] method is 16.65 s, the power big data mining time of reference [15] method is 8.92 s, and the power big data mining time of this method is 0.62 s; When the amount of power data to be mined increases to 200 GB, the power big data mining time of reference [6] method is 88.25 s, the power big data mining time of reference [15] method is 69.25 s, and the power big data mining time of this method is 2.32 s; When the amount of power data to be mined is 250 GB, the power big data mining time of reference [6] method is 129.92 s, the power big data mining time of reference [15] method is 99.63 s, and the power big data mining time of this method is 3.05 s; This method always has a shorter time-consuming of power big data mining with different methods, which shows that the efficiency of power big data mining has been effectively improved after the improvement of grey clustering algorithm.

4.3.3. Power big data mining completeness

Compare the completeness of power big data mining of this method, reference [6] method and reference [15] method, and the comparison results are shown in Table 2.

Table 2
Comparison of completeness of power big data mining with different methods

Large amount of power data / GB Power big data mining completeness

Reference [6] method Reference [15] method Proposed method

50 0.763 0.692 0.992

100 0.826 0.688 0.985

150 0.802 0.705 0.969

200 0.776 0.732 0.975

250 0.701 0.806 0.986

Large amount of power data / GB	Power big data mining completeness
50	0.763	0.692	0.992
100	0.826	0.688	0.985
150	0.802	0.705	0.969
200	0.776	0.732	0.975
250	0.701	0.806	0.986

By analyzing the results shown in Table 2, we can see that the power big data mining completeness of reference [6] method is between 0.701–0.826, that of reference [15] method is between 0.688–0.806, and that of this method is between 0.969–0.992. Compared with the two traditional evaluation methods, the result of this method is closer to 1, which shows that its power big data mining integrity has stronger ability to analyze the collected power big data. This is because this method uses Lagrange interpolation to interpolate and fill the missing power consumption data, which effectively avoids the problem of missing information in power big data mining and improves the completeness of power big data mining.

5. Conclusion

This paper presents a multi-dimensional power big data mining method based on improved grey clustering algorithm. Collect power big data through the collector, fill in the missing power consumption data for interpolation, mine the characteristics of multi-dimensional power big data according to FCM clustering algorithm, construct the objective function of power big data mining, and obtain the final mining results of multi-dimensional power big data according to the improved grey clustering algorithm. The following conclusions are drawn through experiments:

The integrity of power big data mining of this method is up to 0.996, which shows that this method can effectively improve the effect of power big data mining.

The power big data mining time of this method is always less than 3.05 s, which shows that the efficiency of power big data mining has been effectively improved after the improved grey clustering algorithm.

The completeness of power big data mining in this method can reach 0.992, which shows that this method effectively improves and avoids the lack of information in power big data mining, and improves the effect of power big data mining.

References

Cai,

Ma,

Sun and

Huang, Power equipment operation and maintenance and decision analysis method based on data mining, Journal of South China University of Technology (Natural Science) 47(06) (2019), 57–64+71.

Chen,

Cao and

Si, Benchmark value determination of energy efficiency indexes for coal-fired power units based on data mining methods, Advanced Engineering Informatics 43(26) (2020), 101029. doi:10.1016/j.aei.2019.101029.

Dong,

Wei,

An,

Mei and

Xu, Distribution network project cost control method based on data mining, Computer and Network 45(17) (2019), 61–64.

Feng,

Lao,

Ye,

Sun and

Kang, Research on power grid big data intelligent mining technology based on web crawler, Electronic Design Engineering 27(16) (2019), 161–164+169.

Hoogeveen,

Tomczyk and

T.C.

van der Zanden, Flower power: Finding optimal plant cutting strategies through a combination of optimization and data mining, Computers & Industrial Engineering 127(06) (2019), 39–44. doi:10.1016/j.cie.2018.11.047.

Huang,

Chen,

Li,

Zheng and

Wu, Power data mining method based on GRU-MMD, Industrial Heating 50(10) (2021), 61–65.

Konstantinos and

Georgios, Efficient power theft detection for residential consumers using mean shift data mining knowledge discovery process, SSRN Electronic Journal 10(1) (2019), 69–85.

Li,

Yu and

Sang, Research on automatic fault diagnosis method of power system based on data mining, Machine Building & Automation 51(02) (2022), 228–231.

Liu,

Zhang and

Peng, Distribution network load prediction based on k-adaboost data mining, Zhejiang Electric Power 38 (2019), 104–110.

10.

H.M.

Wang, Research on data mining algorithm in power marketing analysis, Journal of Physics: Conference Series 1635(01) (2020), 012002.

11.

Wang,

Shan,

E.-Q.

Chen,

Dai,

Zhao,

Q.-C.

Zheng and

Li, Research and application of data mining and visualization technology for power grid incident correlation cause tracing, Automation Applications 33(02) (2019), 79–83.

12.

Wen,

Qian,

Peng,

Ma and

Shi, Modeling and optimization of SCR denitrification system based on GA-GRNN data mining, Journal of Shanghai Dianli University 36(02) (2020), 161–167.

13.

Xie and

Wen Research on sensitive data mining technology based on differential privacy, Journal of Shanghai Electric Power University 36(04) (2020), 401–407.

14.

Ye,

Dai,

Chen and

Zheng, Short-term power load forecasting method based on data mining technology, Electronics World 32(01) (2019), 30–31.

15.

Zang,

Xie,

Zhang,

Wu and

Bai, Research on data mining algorithm based on clustering analysis of electric power marketing, Information Technology 44(04) (2020), 56–59+64.

16.

Zhou,

Shen,

Gao,

Wang and

Lu, Microgrid fault location method based on data mining, Science and Technology Bulletin 35(11) (2019), 53–56+62.

Large amount of power data / GB	Power big data mining with different methods takes time/s

	Proposed method	Reference [6] method	Reference [15] method
50	0.62	16.65	8.92
100	1.28	28.69	16.28
150	1.68	58.91	38.96
200	2.32	88.25	69.25
250	3.05	129.92	99.63