A combined model based on feature selection and support vector machine for PM2.5 prediction

Abstract

With the increasing attention to the environment and air quality, PM2.5 has been paid more and more attention. It is expected to excavate useful information in meteorological data to predict air pollution, however, the air quality is greatly affected by meteorological factors, and how to establish an effective air quality prediction model has always been a problem that people urgently need to solve. This paper proposed a combined model based on feature selection and Support Vector Machine (SVM) for PM2.5 prediction. Firstly, aiming at the influence of meteorological factors on PM2.5, a feature selection method based on linear causality is proposed to find out the causality between features and select the features with strong causality, so as to remove the redundant features in air pollution data and reduce the workload of data analysis. Then, a method based on SVM is proposed to analyze and solve the nonlinear problems in the data, for reducing the prediction error, a method of particle swarm optimization is also used to optimize SVM parameters. Finally, the above methods are combined into a prediction model, which is suitable for the current air pollution control. 12 representative data sets on the UCI (University of California, Irvine) website are used to verify the combined model, and the experimental results show that the model is feasible and effective.

Keywords

Feature selection linear regression support vector machine combined forecasting model PM2.5 prediction

1 Introduction

Air pollution control is a research hotspot in today’s society. In various practical application scenarios, real-time air quality monitoring information obtained through existing air quality monitoring instruments, air quality monitoring stations and satellite meteorological data cannot fully meet the requirements of various applications, and the prediction of future air quality needs to be carried out [13]. Considering the correlation between PM2.5 concentration and temperature, relative humidity, precipitation, wind, visibility and other meteorological factors, and inevitably incomplete and repeated data in the meteorological data, so it is necessary to select valuable data to improve the accuracy of the forecast. In recent years, artificial intelligence technology has been used to analyze meteorological data and predict air pollution [11 , 31]. Feature selection method can remove the data that is not associated with the target, and then improve the accuracy of data prediction and the efficiency [28]. A machine learning model is used to analyze and predict unknown data according to the known data characteristics is a hot topic in scientific research. SVM is a kind of machine learning technology based on statistical learning theory. It takes the principle of minimizing system structural risk as the prediction idea. Since it was proposed last century, it has been deeply studied by scholars for decades. In order to effectively solve the “over-learning” problem of other forecasting methods, SVM introduces concepts such as kernel function, slack variables, and minimum criteria based on structural risk, which can better solve the problem of nonlinear data classification. At present, the SVM method has been widely used in linear inseparable problems in the fields of finance, biological information recognition, and construction science [16 , 32].

Based on the above technology, this paper proposed a combined model based on feature selection and SVM for PM2.5 prediction, and the main contributions of this paper are as follows: (1) A feature selection method based on causality between features, namely Causality Based Linear (CBL), is proposed. By using the CBL method to delete redundant features, subsequent data analysis can be effectively reduced. (2) SVM is used in model learning, and Particle Swarm Optimization (PSO) is used to optimize SVM parameters to further reduce the model prediction error. (3) A combinatorial model of the CBL method and SVM, namely CBL-PSO-SVM, is proposed to predict PM2.5, and the CBL method is used to reduce the data, and then input into SVM to establish the learning model. The combination model combines the advantages of feature selection method and SVM and has better fault tolerance and anti-interference ability. (4) 12 sets of representative data sets from UCI website are used to verify the combined model. The results show that the combined model is more feasible and accurate than the single SVM model.

The remainder of this paper is organized as follows. Section 2 of this paper introduces the research status of feature selection and SVM. Section 3 introduces the design and implementation of the CBL method. Section 4 introduces the design and implementation of the CBL-PSO-SVM combination model. Section 5 gives the analysis of the experimental results and makes a comparative analysis between the proposed method and existing methods. The last section gives conclusions and future work.

2 Related work

2.1 Research status of feature selection

Feature selection is currently a mainstream algorithm that is highly respected and favored by the industry. It effectively guarantees the accuracy and timeliness of classification by removing noise and reducing dimensions. Simply put, in the process of data processing, it first removes redundant features that are not related to the target, and then maximize the generalization advantage of the classifier, thus it can minimize the impact of high-dimensional data, properly resolve the contradiction between classification accuracy and high-dimensional training [1].

At present, some feature selection methods based on causal analysis and correlation between features are favored by researchers. Williams et al. [17] studied the accuracy of classification selection methods of various statistical features, and evaluated and analyzed the reliability and accuracy of classification selection algorithms by using PSO and machine learning. Yang et al. [29] extracted 12 representative features by using the mainstream classification selection method, and calculated and analyzed each feature one by one, and found that there was an inevitable correlation between classification accuracy and statistical feature selection algorithm. Chersan et al. [8] used linear regression analysis (single and multiple) and analysis of variance to test whether there is a circular causal relationship between audit fees and the financial performance, so as determining the determinants of audit fees. Li et al. [30] had carried out systematic research on the feature selection problem by using the classic genetic algorithm. The results show that the algorithm has good adaptability and retrieval ability, and also shows significant advantages in dealing with complex problems, but its biggest defect is that the convergence speed is too fast. Pereira et al. [21] constructed a feature selection algorithm with feature relevance. The results show that the algorithm has a relatively wide range of applications and is mainly calculated based on the relevant characteristics of the curve fitting trend. However, it also has many problems in practice, such as long investment time, low calculation efficiency, and large calculation amount. Doğan et al. [9] tested the relationship among CO₂ emissions, energy consumption and economic growth variables in Turkey through parametric and non-parametric causality models, and the tested results show that there is a bidirectional causal relationship among them. To solve the problem of missing environmental data, Izonin et al. [10] proposed a prediction method based on two series-connected General Regression Neural Networks (GRNN) and the extended-input Successive Geometric Transformations Model (SGTM) neural-like structure to improve the recovery of partially lost or completely lost data. The effectiveness of this method is verified by experiments. Mohammadi et al. [23] proposed a Cyber intrusion detection system by a combined feature selection algorithm, which based on feature selection and clustering algorithm.

Meteorological data features are the main factors affecting PM2.5 prediction, and monitoring the relationship between these features and PM2.5 concentration is a necessary condition for accurate prediction of PM2.5 concentration. However, the existing filtering methods such as correlation analysis and causal analysis ignore the correlation between features. Based on the previous studies of feature selection, we need to study more effective methods to select the features that are highly correlated with PM2.5 concentration.

2.2 Research status of SVM

In recent years, many domestic scholars have done in-depth research on the SVM algorithm. Gao et al. [15] proposed and designed an air quality prediction method based on prediction characteristics by using LightGBM (Light Gradient Boosting Machine), which can effectively predict PM2.5 concentration in Beijing in the next 24 hours. Chen et al. [24] established a PSO-SVM hybrid model for effective short-term air pollutant concentration prediction and used a clustering algorithm to enhance the regularity of the data before inputting data onto the mixed model. Aiming at the problems such as low learning efficiency, poor generalization performance, and noise sensitivity when SVM is applied to online learning tasks, Guo et al. [5] proposed an online SVM learning accelerated model based on window technology and Karush-Kuhn-Tucker conditions. The experimental results show that the model can accelerate the online learning process effectively and has good robustness and generalization performance. Zheng et al. [14] proposed a quantitative assessment algorithm of ship collision risk based on SVM, which overcomes the defects of traditional risk assessment methods, and has certain advantages for ship collision risk assessment. Dai et al. [12] put forward a kind of Kalman filter based on SVM for the vibration caused by the physiological tremor of the hand. The results show that the method can effectively filter out the tremor signal to improve the accuracy of surgery.

Foreign scholars have also done lots of research in the field. Izonin et al. [7] proposed two compatible methods based on Wiener Polynomial and SVM to solve the classification task of medical implant materials. Experimental results show that the methods have high accuracy. Jamil et al. [3] proposed the application of kernel Fisher Discriminant Analysis (KFDA) and Kernel SVM (KSVM) to the fault detection and classification of Pakistani nuclear reactors. By comparing the fault detection and classification results of KFDA and KSVM, it was found that the two kernel technologies had similar classification performance and could be applied to the fault detection and classification of nuclear reactors. Cuingnet et al. [18] proposed a new method of detecting group level differences in brain images by using SVM based on spatial regularization. The spatial regularization of SVM was performed by using the graph Laplacian. The results show that the accuracy of the method is more effective than that of the traditional machine detection method. Pouteau et al. [19] proposed to use SVM to integrate multisource-derived biophysical descriptors (overstory plant species, physiography and climate) for the indirect detection of the small invasive tree Miconia calveses in tropical rainforests on the island of Tahiti (South Pacific). A range of accuracy metrics was calculated to assess the SVM-based model which widely outperforms the commonly used GARP (Genetic algorithm for rule-set production) model. Ravindra et al. [2] discussed the use of the SVM neural network to classify chronic kidney disease (CKD) and non-chronic kidney disease (NCKD). The experimental results found that the SVM-based method is a potential candidate for CKD and NCKD classification.

To sum up, the research in this area is still in the exploratory stage, and the methods and models are oriented to specific applications and are not universal. Thus, we need to carry out more in-depth discussion, and propose more feasible and effective methods for PM2.5 prediction.

3 CBL method

In this section, we proposed CBL, which is a feature selection method based on causality. CBL method can be expressed as follows: supposing there is a continuous response variable b, whose conditional characteristics are α₁ and α₂, then its expected value based on linear regression prediction is shown in Equation (1): $b = β_{0} + β_{1} α_{1} + β_{2} α_{2} + β_{12} α_{1} α_{2}$ (1)

Among them, β_s (s ∈ N⁺) is the regression coefficient [20]. By t-test on the regression coefficient β₁₂, the interactive relationship between conditional features α₁ and α₂ is analyzed. For n conditional features, if the interactive relationship between all features is considered, the total number of interactions is 2ⁿ, which is the case of the largest exponent. Therefore, we only need to calculate the feature subset with a significant interaction.

The CBL method calculates the interaction between pairwise features according to the results of the causal relationship analysis and then uses linear regression to calculate the relative importance of each conditional feature. It can be seen from Fig. 1 that CBL method mainly involves the operation of the following four links:

Fig. 1

Main steps of CBL method.

Use Convergent Cross Mapping (CCM) to analyze the causal relationship between the conditional features and the target features in the original data, and extract the conditional features which have a causal relationship of the target features. CCM is used to test the causal relationship between two time series variables. It is often used for nonlinear, dynamic and non-random events in reality.

Considering the relationship between conditional features which have a causal relationship of target features, linear regression is performed to obtain the mutual coefficient t value. The larger the value of t, the greater the interaction.

According to the value of mutual coefficient t, it can be judged whether the interaction between two conditional features is significant. For pairwise conditional features of a significant relationship, the relative importance of each feature is calculated by using the linear regression method.

According to the order for relative importance, the conditional features which have a great influence on target feature is obtained.

In CBL method, CCM is the core technology for causal analysis of various characteristics of data. Compared with the Granger causality method, CCM can better identify the causality of nonlinear variables. At present, the non-linear causality of multiple time series is a hot issue that is urgently discussed and studied by scholars [6]. Sugihara et al. [4] innovatively proposed CCM with extensive significance by using the classical embedding theory and state space reconstruction (SSR) technology. In the time series X and Y with period length L, they can be directly described by the following Equations (2) and (3): ${X} = {X (1), X (2), X (3), \dots, X (L)}$ (2) ${Y} = {Y (1), Y (2), Y (3), \dots, Y (L)}$ (3)

For sequence variables X and Y, if X is used to describe the triggering process and Y is used to describe the response process, CCM uses the historical value of Y to evaluate the state of X, and then determines the relationship between the two. The shadow manifolds M_X and M_Y reconstructed on the original basis can be directly described by the following Equations (4) and (5): $\begin{matrix} M_{X} : \underline{x} (t) = \\ 〈 X (t), X (t - τ), X (t - 2 τ), \dots, X (t - (E - 1) τ) 〉 \end{matrix}$ (4) $\begin{matrix} M_{Y} : \underline{y} (t) = \\ 〈 Y (t), Y (t - τ), Y (t - 2 τ), \dots, Y (t - (E - 1) τ) 〉 \end{matrix}$ (5)

In a word, when using CBL method to reduce data, we should pay special attention to the correlation between features. The core idea of this method is to add the conditional features which are highly related to the target features of the reduction set as much as possible. The interaction between the conditional features can be obtained according to the results of causal analysis, and then the relative importance of each conditional feature can be calculated by linear regression, then we can reduce data according to the size of the feature correlation. The specific steps are as follows:

Input: Original data table S, reduced threshold τ

Output: feature reduction set B

Step 1: Standardize all the sample data in the original data table and initialize it to B = {∅}.

Step 2: For each feature c_i (i = 1, 2, ⋯ , m), calculate the correlation γ (c_i, d) between it and the target feature.

Step 3: Arrange the correlations between features in an orderly manner (from large to small). For all conditional features, if γ (c_i, d) ⩾ τ, add c_i to B to obtain a reduced set B.

Step 4: Output B.

4 CBL-PSO-SVM combination model

4.1 Construction of CBL-PSO-SVM

Combining the characteristics and advantages of SVM and feature selection of raw data processing, we can form a new combined learning model “CBL-PSO-SVM”. Using this combined model, we can design and improve the learner by reducing the prediction error and learning efficiency. In addition, the PSO algorithm is used to optimize the kernel parameter selection of SVM and further improve the combination model. The main steps of CBL-PSO-SVM combined learning model when processing data are:

Initializes the training data set and assigns an initial value to the data object.

Normalize the original data to unify the dimensions.

A feature selection algorithm is used to remove the redundant features and the features that have little impact on the results from the normalized data table to reduce the amount of data. In this paper, the CBL method is used to preprocess the SVM data, and the characteristics of the selection method are studied.

PSO and SVM parameters are optimized to obtain the global optimal parameters.

Then a prediction model is established based on the SVM training reduced data, and the kernel parameters required by SVM are optimized by using PSO, which, on the one hand, reduces the computational burden of the learner, and on the other hand, prevents the over-fitting caused by useless features.

Finally, the generalization ability of the composite model is tested with the test samples.

4.2 Data standardization processing

In real applications, because of the obvious differences in the units or dimensions between features, making the data comparison loses its meaning, and the incommensurability is significant. Therefore, in order to make the features uniformly processed, it is necessary to normalize all features data onto dimensionless intervals according to a certain mapping function. Through the use of standardized data into complex features of dimensionless processing, the aim is to reduce the adverse effects on the analysis results from the source. In fact, many algorithms are implemented on the premise of data standardization processing. Before data analysis by feature selection method, feature data should be standardized to map to the space of (0,1), and the linear features of data should be protected from destruction to the maximum extent. The transformation function of this paper is shown in Equation (6): $y = \frac{x - MinValue}{MaxValue - MinValue}$ (6)

Where, x and y are the values before and after the transformation, and MaxValue and MinValue are the maximum and minimum values of the sample respectively.

4.3 Optimization of SVM parameters

The selection of SVM kernel functions and kernel parameters is the important problem that needs to be improved with the development of the SVM. Its essence is an optimized search process, which directly affects the promotion ability of the model. Due to the lack of theoretical guidance, the traditional selection of nuclear parameters is mostly through repeated experiments. At present, there are three kinds of kernel functions widely used, namely S kernel function, polynomial kernel function and radial basis function. Among them, the first two types of functions contain two parameters, while the last one only involves one parameter. Generally speaking, the more parameters, the more difficult the optimization problem will be. In view of this, this paper decides to adopt a radial basis function, which has good applicability and can be used for various distributed data [25]. Then, a rigorous and reasonable learning model should be created through the parameters C and σ, where C is the penalty coefficient, and the larger the value, the smaller the deviation range; σ is the parameter of the radial basis function.

Generalization performance estimation is another important problem in machine learning and also the basis of parameter selection, the common methods include PSO, Cross Validation (CV) and Genetic Algorithm (GA). PSO has been successfully applied to many optimization problems due to it is easy to understand, easy to implement, and has few adjustable parameters. When the PSO algorithm is used, its parameters to be determined include inertia weight w, learning factors c₁ and c₂, maximum iteration number M, searching space dimension D, initializing group number N, etc.

In this paper, PSO is applied to the parameter optimization of SVM, and the improved model, namely PSO-SVM, is put forward for solving the problem of parameter selection of SVM. The framework of PSO-SVM is shown in Fig. 2:

A set of penalty coefficient C and parameter σ of radial basis function are generated randomly by an initial operation of particle swarm optimization.

SVM is used to train all particles, and the model accuracy is defined as the fitness value of particles.

When the iteration times meet the requirements, the optimal parameters are determined.

If the termination conditions are not met, the velocity and orientation of particles are readjusted to build a new particle population, and the search operation is performed again to determine the optimal value through formula calculation.

The most suitable parameters are selected as the input parameters of SVM to construct the model.

Fig. 2

The framework of PSO-SVM.

Figure 3 shows the POS optimization of parameters w, c₁, c₂ and SVM kernel function in the detailed process, the specific implementation steps are as follows:

Fig. 3

Specific process of PSO-SVM.

Initialize the PSO and SVM parameters. Before the code is run, the parameters of w, c₁ and c₂ in PSO are initialized based on experience, while the kernel function in SVM is selected by a random function.

The SVM model was constructed. Input the training data to train the SVM model, and then verify the SVM model through the verification data.

Take the model evaluation indicator as the fitness value of PSO, and sort the fitness value.

Judge whether it is the optimal value? If yes, output the parameters of PSO and SVM. Otherwise, perform evolution operation, jump to (1).

Output the parameters of PSO and SVM to complete parameter optimization.

Since the CBL method is used to calculate the causal relationship between PM2.5 and other features to reduce the data dimension, the time complexity of the method depends on the number of features. The reduced-dimensional data set is trained by SVM. Assuming that there are n experimental data sets, the time complexity of training by SVM is O (n²). Because the number of features is much smaller than the experimental data, the time complexity of the CBL-PSO-SVM model is O (n²).

5 Evaluation

In this section, experiments have been performed to evaluate the performance of our methods and models. In the experiment, Python 3.7.3 language is used, and all the experimental environments are completed on Windows 7 64bit operating system, 8GB memory, Intel(R) Core (TM) i7-6700 CPU@3.40GHz.

5.1 Experimental data

This experiment uses 12 data sets from UCI machine learning database (https://archive.ics.uci.edu/ml/machine-learning-databases). The data sets contain 420 thousand surface meteorological data and air quality data from March 1, 2013 to February 28, 2017 in 12 stations of Beijing, among which air quality data include PM2.5, PM10, SO₂, NO₂, CO and O₃, and surface meteorological data include air temperature, air pressure, dew point temperature, rainfall and wind speed. The 12 stations are Aotizhongxin, Changping, Dingling, Dongsi, Guanyuan, Gucheng, Huairou, Nongzhanguan, Shunyi, Tiantan, Wanliu and Wanshouxigong. All data were recorded once an hour except that those of rainfall and wind speed were daily records. Table 1 shows the characteristics of the experimental data, and Table 2 shows part of the original hourly meteorological data in Aotizhongxin.

Table 1
Characteristics of experimental data

Number Feature Unit

1 station –

2 year year

3 month month

4 day day

5 hour hour

6 PM2.5 mg/m³

7 PM10 mg/m³

8 SO₂ μg/m³

9 NO₂ μg/m³

10 CO mg/m³

11 O₃ μg/m³

12 TEMP °C

13 PRES Pa

14 DEWP °C

15 RAIN mm

16 WSPM m/s

Number	Feature	Unit
1	station	–
2	year	year
3	month	month
4	day	day
5	hour	hour
6	PM2.5	mg/m³
7	PM10	mg/m³
8	SO₂	μg/m³
9	NO₂	μg/m³
10	CO	mg/m³
11	O₃	μg/m³
12	TEMP	°C
13	PRES	Pa
14	DEWP	°C
15	RAIN	mm
16	WSPM	m/s

Table 2

Partial experimental data sets

Station	Years	Month	Day	Hour	PM2.5	SO₂	NO₂	CO	PRES	TEMP	WSPM
Aotizhongxin	2013	3	1	0	4	4	7	300	1023.0	–0.7	4.4
Aotizhongxin	2013	3	1	1	8	4	7	300	1023.2	–1.1	4.7
Aotizhongxin	2013	3	1	2	7	5	10	300	1023.5	–1.1	5.6
Aotizhongxin	2013	3	1	3	6	11	11	300	1024.5	–1.4	3.1
Aotizhongxin	2013	3	1	4	3	12	12	300	1025.2	–2	2
Aotizhongxin	2013	3	1	5	5	18	18	400	1025.6	–2.2	3.7
Aotizhongxin	2013	3	1	6	3	18	32	500	1026.5	–2.6	2.5
Aotizhongxin	2013	3	1	7	3	19	41	500	1027.4	–1.6	3.8
Aotizhongxin	2013	3	1	8	3	16	43	500	1028.3	0.1	4.1
Aotizhongxin	2013	3	1	9	3	12	28	400	1028.5	1.2	2.6

5.2 Evaluation indexes

In order to evaluate the reliability and accuracy of the combined model, we use four representative indexes: Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Coefficient of Determination R², Mean Absolute Error (MAE), and the calculation equations are as shown in (7), (8), (9), (10): $MSE = \frac{1}{m} \sum_{i = 1}^{m} (y_{i} - \hat{y_{i}})^{2}$ (7) $RMSE = \sqrt{\frac{1}{m} \sum_{i = 1}^{m} (y_{i} - \hat{y_{i}})^{2}}$ (8) $R^{2} = \frac{TSS - RSS}{TSS} = 1 - \frac{RSS}{TSS}$ (9) $MAE = \frac{1}{m} \sum_{i = 1}^{m} | (y_{i} - \hat{y_{i}}) |$ (10)

Among them, y_i refers to the true value, ${\hat{y}}_{i}$ refers to the predicted value, m refers to the amount of data, $(Total Sum of Squares) TSS = \sum_{i = 1}^{m} (y_{i} - \bar{y})^{2}$ , $\bar{y}$ is the mean value of true value, and (Residual Sum of Squares) $RSS = \sum_{i = 1}^{m} (y_{i} - {\hat{y}}_{i})^{2}$ .

MSE is the average of the square of the difference between the real target y and the predicted value ${\hat{y}}_{i}$ , which is more sensitive to outliers than MAE. RMSE describes the varying levels of the difference between the target value and the test value. If the difference is large, the RMSE value will be large. On the contrary, the smaller the RMSE, the smaller the difference between most predicted values and actual values. The determination coefficient R² represents the proportion of target changes caused by features. Theoretically, R² will always increase as long as the number of features is increased, regardless of whether the feature x is related to the target y. Therefore, R² is usually used for feature selection. If a feature is added, the R² value of the model will increase a lot, which indicates that the feature is related to the target. MAE represents the average value of the difference between the real target y and the predicted value ${\hat{y}}_{i}$ . In this paper, we calculate the MSE, RMSE, R² of the data. We not only calculate the prediction error, but also evaluate the learning efficiency comprehensively.

5.3 Performance of CBL model

To evaluate the performance of CBL model, this paper uses CBL model combined with a variety of machines learning algorithms to predict PM2.5. In the experiment, the original data set was divided between a ratio of 7:3, that is, 70% was used as the training model to learn the data model, and the remaining 30% was used as the prediction model. In the original data table, there are 11 meteorological features, among which PM2.5 is taken as the target feature and PM10, SO₂, NO₂, CO, O₃, the air temperature, air pressure, dew point temperature, rainfall and wind speed are the conditional features. CBL model in section 3 was used to analyze the original data table and the causal relationship of the given data set. The results showed that there was a causal relationship between air temperature, NO₂, CO, wind speed, SO₂ and PM2.5. Therefore, CBL method considers the interactive relationship between air temperature, NO₂, CO, wind speed, and SO₂ concentration and performs linear regression. The results of the linear regression coefficients corresponding to the t value and its related probability are shown in Table 3.

Table 3
Coefficient estimation of linear regression

Parameter Estimate Std. Error t value Pr(> |t|)

TEMP –1.9491 0.85646 –2.276 0.0231*

DEWP 0.14009 0.03139 4.462 0.0913

WSPM –1.4192 0.56194 –2.525 0.0117*

CO 78.5856 12.3454 6.366 0.000000000309

NO₂ 1.98317 0.31737 6.249 0.000000000637

SO₂ 0.03459 0.04232 0.817 0.00000414*

O₃ 0.08748 0.11595 0.754 0.4508

WSPM: CO –0.8195 0.14746 –5.557 0.0000000361

WSPM: NO₂ –0.1838 0.03965 –4.635 0.0000041

TEMP: WSPM 0.10103 0.01889 5.35 0.000000112**

CO: NO₂ –0.1501 0.05863 –2.559 0.0106*

TEMP: CO –1.3366 0.33888 –3.944 0.0000862**

TEMP: NO₂ –0.0025 0.00955 –0.258 0.7968

Parameter	Estimate	Std. Error	t value	Pr(> \|t\|)
TEMP	–1.9491	0.85646	–2.276	0.0231*
DEWP	0.14009	0.03139	4.462	0.0913
WSPM	–1.4192	0.56194	–2.525	0.0117*
CO	78.5856	12.3454	6.366	0.000000000309**
NO₂	1.98317	0.31737	6.249	0.000000000637**
SO₂	0.03459	0.04232	0.817	0.00000414*
O₃	0.08748	0.11595	0.754	0.4508
WSPM: CO	–0.8195	0.14746	–5.557	0.0000000361**
WSPM: NO₂	–0.1838	0.03965	–4.635	0.0000041**
TEMP: WSPM	0.10103	0.01889	5.35	0.000000112**
CO: NO₂	–0.1501	0.05863	–2.559	0.0106*
TEMP: CO	–1.3366	0.33888	–3.944	0.0000862**
TEMP: NO₂	–0.0025	0.00955	–0.258	0.7968

*: Said 0.01< P<0.05, is significant difference; **: said P<0.01, is the difference is very significant.

It is further understood from the data analysis in Table 3 that only the interaction between CO and wind speed, NO₂ and wind speed, wind speed and temperature, NO₂ and CO, and CO and air temperature are significant. Then, for the features with a significant interaction relationship, the relative importance percentage value of each feature is obtained by using the linear regression method, as shown in Table 4.

Table 4

Relative importance of features

Parameter	Relative importance (%)
NO₂	41.2255154
CO	21.2922478
WSPM	18.3452669
TEMP	5.3834217
DEWP	3.0737116
SO₂	2.8377518
TEMP: WSPM	2.5199141
WSPM: CO	1.7881085
WSPM: NO₂	1.3162923
TEMP: CO	0.9862647
CO: NO₂	0.5675425

It can be seen from the Table 4 that the most relevant features selected by CBL method are NO₂ concentration, CO concentration and wind speed, indicating that these three features have a greater impact on PM2.5.

Machine learning algorithms such as the SVM algorithm, Random Forest (RF), K-nearest neighbor (KNN), Back Propagation (BP), and Multiple Linear Regression (MLR) were used to conduct an experimental analysis of air quality data in 12 areas of Beijing. In the experiment, we mark 12 stations as the corresponding dataset number. CBL model combined with these machines learning algorithms is namely CBL-SVM, CBL-RF, CBL-KNN, CBL-BP and CBL-MLR model. Four indexes of MSE, RMSE, R² and MAE are calculated for each model, as shown in Fig. 4.

Fig. 4

Changes of four indexes of each learning model.

It can be seen from Fig. 4 that the experimental results of SVM are better than the other machine learning algorithms. The effect of machine learning combined with CBL method is better than that of a single model. Further observation shows that the CBL-SVM model is better than other models in the four indicators. The effectiveness of CBL proposed in this paper is verified by the experimental results.

5.4 Performance of SVM optimization

In order to evaluate the performance of SVM optimization, PSO, CV and GA three random search algorithms were used to carry out comparative experiments on SVM parameter optimization. The four indexes of PSO-SVM, CV-SVM and GA-SVM learning models are shown in Fig. 5.

Fig. 5

Four indexes of PSO-SVM, CV-SVM and GA-SVM.

According to the experimental results in Fig. 5, we can see that no matter which experimental index is used as the evaluation standard, the predicted effect of the PSO-SVM model is higher than that of the other two models, the GA-SVM has the worst result, while the CV-SVM is among the first two models. At the same time, compared with the experimental results of SVM, we can also find that the PSO-SVM model has improved the predictive effectiveness compared with the SVM model before optimization. Therefore, we can conclude that PSO-SVM has the good effect of the parameter optimization of SVM, and it helps to find the optimal parameters for SVM training model construction.

5.5 Performance of CBL-PSO-SVM model

Through the experimental results from 5.4 above, it can be seen that PSO-SVM has the best effect on parameter optimization of SVM. Therefore, this paper combines the CBL method and PSO-SVM to form a new combined model CBL-PSO-SVM. The prediction results and learning time of the models are obtained, and the four characteristic indexes of the predicted results are shown in Fig. 6.

Fig. 6

Four indexes of PSO-SVM and CBL-PSO-SVM.

By analyzing the test results of 12 groups of data in Fig. 6, it can be seen that the advantages of the CBL-PSO-SVM model are much greater than that of the PSO-SVM model. No matter which of MSE, RMSE, R² and MAE is taken as the evaluation criteria, CBL-PSO-SVM has a high predictive ability. The reason is that CBL-PSO-SVM can eliminate the feature that has no or little influence on the target object, and can also reduce the influence of redundant complexity to the minimum, so as to avoid the problem of overfitting.

Tables 5 and 6 respectively list the running time of the two combination models PSO-SVM and CBL-PSO-SVM on 12 data sets. From these tables, it can be seen that the training time is in direct proportion to the number of features reduced by the CBL method, and CBL shortens the time in data preprocessing. The reason is that CBL can quickly and efficiently select features, and the overall time of the combination model mainly depends on the learning time of the SVM, so it can reasonably control the execution time of the combination model. In short, CBL-PSO-SVM, which using CBL method to do data preprocessing for learners, has been improved both in predicting results and learning efficiency.

Table 5

Learning time of PSO-SVM (s)

Data\Index	MSE	RMSE	R²	MAE
1	1411.6120	1972.0370	2151.1346	181.7170
2	620.6430	2102.0210	2243.8297	1984.8010
3	293.8350	1730.1740	2220.6954	2110.6140
4	1830.6190	1876.2770	2360.0242	1962.0390
5	956.8740	1898.9020	2353.4034	889.2750
6	1787.1930	1764.9820	2489.2529	1013.5100
7	457.8190	1670.9850	2090.8818	1950.7460
8	1875.500	1372.9190	2193.2320	134.9730
9	1834.0510	1236.7110	1204.9857	2088.1261
10	1041.1190	2115.3740	2303.2566	1450.4125
11	982.5590	1688.5630	2300.0412	1284.2790
12	1106.6580	1251.8860	2526.9704	2022.8244

Table 6

Learning time of CBL-PSO-SVM (s)

Data\Index	MSE	RMSE	R²	MAE
1	139.7294	176.6391	513.7368	73.8193
2	162.9423	124.1294	779.1344	80.8237
3	203.6116	108.98718	370.8984	122.8814
4	94.8326	208.3072	598.9262	104.9414
5	104.0990	341.9682	410.1846	71.2297
6	189.1659	145.8135	576.4476	123.8642
7	88.5614	116.5634	330.0241	161.7411
8	309.1301	182.4579	507.4190	134.6906
9	137.0774	104.9882	567.8746	167.8095
10	94.1618	113.0222	426.8799	215.1071
11	122.0238	152.6463	302.6266	141.3986
12	105.5498	180.7419	387.0950	100.7606

Before initializing the population, the scope of each dimension should be determined according to the effectiveness of the network and the capacity of hardware resources. If the number of particle swarm layers is too small, the index change will not be significant; if the number of layers is too large, the capacity of the hardware required to run the experiment will be very high. After comprehensive consideration, the experiment in this paper uses 20×20 particle swarms with iterative search. Figures 7 and 8 show the changes of MSE, RMSE and Mae of the CBL-PSO-SVM model when the evolution algebra is 5, 10, 15 and 20 respectively. In order to make the graph beautiful, 12 data sets are divided into two large graphs for presentation. Figures 7 and 8 show the index changes of datasets 1 to 6 and 7 to 12, respectively.

Fig. 7

Evolution algebra and changes of each index in datasets 1 to 6.

Fig. 8

Evolution algebra and changes of each index in datasets 7 to 12.

From Figs. 7 and 8, it can be found that MSE, RMSE and Mae of the 12 data sets show a steady downward trend from the first generation, which indicates that the predicted effect of CBL-PSO-SVM model is constantly improving and tends to be stable after the 15th generation, indicating that CBL-PSO-SVM model has begun to converge.

Figure 9 shows the changes of evaluation index R² of CBL-PSO-SVM model. It can be seen that the evaluation index R² increased from the 1st generation to the 6th generation and approached 1 continuously after the 6th generation, indicating that the generalization ability of the model was constantly improved. Moreover, after the 7th generation, R² tends to be stable and does not rise, indicating that the model has begun to converge. So, this study only shows the top 10 generations of R² change information.

Fig. 9

Evolution algebra and changes of R² in datasets 1 to 12.

To sum up, CBL-PSO-SVM has been improved both in predicting results and learning efficiency, and the generalization ability is also improved.

6 Conclusions and future work

In this paper, we propose the CBL method to delete redundant features. At the same time, the PSO-SVM prediction model is constructed to optimize SVM parameters by PSO. Then, a combined model CBL-PSO-SVM is designed to predict air pollution. The results show that our work is more feasible and accurate. However, there are still some defects and deficiencies, which are shown as follows: (1) CBL method is used to deal with numerical data, but not symbolic data. (2) An effective combination of SVM and CBL method is realized through the filtering mode, although the operational flexibility is effectively enhanced, the accuracy of prediction cannot be guaranteed. (3) It still needs to invest a lot of time when CBL-PSO-SVM is applied to a large and complex data set. Therefore, in the following work, this paper will focus on the above imperfections, and obtain the most ideal results through extensive and in-depth discussion and research.

Footnotes

Acknowledgments

This research is supported by the National Natural Science Foundation of China under Grant No. 61862010; “BAGUI Scholar” Program of Guangxi Zhuang Autonomous Region of China; Guangxi Collaborative Innovation Center of Multi-source Information Integration and Intelligent Processing.

References

Kulkarni

and Metta

, A New Code Obfuscation Scheme for Software Protection, IEEE International Symposium on Service Oriented System Engineering, IEEE Computer Society 2014, pp. 409–414.

Ravindra

B.V.

, Sriraam

and Geetha

, Classification of non-chronic and chronic kidney disease using SVM neural networks, International Journal of Engineering and Technology 7 (2018), 191–194.

Jamil

, Abid

, Adil

, et al., Kernel approaches for fault detection and classification in PARR-2, Journal of Process Control 64 (2018), 1–6.

Sugihara

, May

, Ye

, et al., Detecting causality in complex ecosystems, Science 338 (2012), 496–500.

Guo

, Zhang

and Wang

, An accelerator for online SVM based on the fixed-size KKT window, Engineering Applications of Artificial Intelligence 92 (2020), 36–54.

, Leng

and Chen

, Data-based prediction and causality inference of nonlinear dynamics, Science China Mathematics 61 (2018), 403–420.

Izonin

, Trostianchyn

, Duriagina

, et al., The Combined Use of the Wiener Polynomial and SVM for Material Classification Task in Medical Implants Production, International Journal of Intelligent Systems and Applications 10 (2018), 40–47.

Chersan

I.C.

, Robu

I.B.

, Carp

, et al., A Circular Causality Analysis on the Determinants of the Audit Fees within the NYSE-Quoted Companies, Communications of The IBIMA 1 (2012), 1–17.

Doğan

and Topalli

, Income, Carbon Emission and Energy Consumption: The Analysis of Linear and Non-Linear Causality Relationship for Turkey, Business and Economics Research Journal 7 (2016), 107–121.

10.

Izonin

, Tkachenko

, Verhun

, et al., An approach towards missing data management using improved GRNN-SGTM ensemble method, Engineering Science and Technology an International Journal, 2020, pp. 1–11.

11.

, Shao

and Sun

, A DBN-Based Deep Neural Network Model with Multitask Learning for Online Air Quality Prediciton, Journal of Control Science and Engineering 1 (2019), 1–9.

12.

Dai

, Du

, Yan

, et al., Least Squares Support Vector Machine Kalman Filter for Physiological Tremor Suppression in Minimally Invasive Surgical Robot, Journal of Harbin Institute of Technology 27 (2020), 22–28.

13.

Huang

, Duan

, Ji

, et al., A Crowdsource-Based Sensing System for Monitoring Fine-Grained Air Quality in Urban Environments, IEEE Internet of Things Journal 6 (2019), 3240–3247.

14.

Zheng

, Chen

, Jiang

, et al., A SVM based ship collision risk assessment algorithm, Ocean Engineering 202 (2020), 1–11.

15.

Gao

, Zhang

, et al., Air quality prediction approach based on integrating forecasting dataset, Journal of Shandong University 50 (2020), 91–99.

16.

Poongodi

, Sharma

, Vijayakumar

, et al., Prediction of the price of Ethereum blockchain cryptocurrency in an industrial finance system, Computers & Electrical Engineering 81 (2020), 1–12.

17.

Williams

, Zander

and Armitage

, A preliminary performance comparison of five machine learning algorithms for practical IP traffic flow classification, ACM Special Interest Group on Data Communication 36 (2006), 5–16.

18.

Cuingnet

, Rosso

, Chupin

, et al., Spatial regularization of SVM for the detection of diffusion alterations associated with stroke outcome, Medical Image Analysis 15 (2011), 729–737.

19.

Pouteau

, Meyer

J.Y.

and Stoll

, A SVM-based model for predicting distribution of the invasive tree Miconia calvescens in tropical rainforests, Ecological Modelling 222 (2011), 2631–2641.

20.

Piedmont

R.L.

, Regression Coefficients. Springer Netherlands, 2014.

21.

Pereira

R.B.

, Plastino

, Zadrozny

, et al., A.A. Freitas, Lazy attribute selection: Choosing attributes at classification time, Intelligent Data Analysis 15 (2011), 715–732.

22.

Chang

S.W.

, Chang

C.L.

, Li

L.T.

, et al., Reinforcement Learning for Improving the Accuracy of PM2.5 Pollution Forecast Under the Neural Network Framework, IEEE Access 8 (2019), 9864–9874.

23.

Mohammadi

, Mirvaziri

, Ghazizadeh-Ahsaee

, et al., Cyber intrusion detection by combined feature selection algorithm, Journal of Information Security and Applications 44 (2019), 80–88.

24.

Chen

, Wang

and Zhang

H.Y.

, A hybrid PSO-SVM model based on clustering algorithm for short-term atmospheric pollutant concentration forecasting, Technological Forecasting and Social Change 146 (2019), 41–54.

25.

Milovanovic

and Sydow

, Radial Basis Function generated Finite Differences for option pricing problems, Computers & Mathematics with Applications 75 (2017), 1462–1481.

26.

Krishna

, Lakra

A.D.

, Shukla

, et al., Identification of potential histone deacetylase1 (HDAC1) inhibitors using multistep virtual screening approach including SVM model, pharmacophore modeling, molecular docking and biological evaluation, Journal of Biomolecular Structure and Dynamics 38 (2020), 3280–3295.

27.

Chen

, An

J.M.

and Yanhan , A novel prediction model of PM2.5 mass concentration based on back propagation neural network algorithm, Journal of Intelligent and Fuzzy Systems 37 (2019), 1–9.

28.

Liu

, Cao

J.J.

, Diao

X.C.

, et al., Survey on Stability of Feature Selection, Journal of Software 29 (2018), 2559–2579.

29.

Yang

Y.X.

, Wang

, Liu

, et al., Solving P2P Traffic Identification Problems Via Optimized Support Vector Machines, 2007 IEEE/ACS International Conference on Computer Systems and Applications, 2007, pp. 165–171.

30.

, Zhang

and Zeng

, Research of multi-population agent genetic algorithm for feature selection, Expert Systems with Applications 36 (2009), 11570–11581.

31.

and Tao

, Daily PM10 concentration forecasting based on multiscale fusion support vector regression, Journal of Intelligent and Fuzzy Systems 34 (2018), 1–12.

32.

Zhang

, Zhang

, Chongzhi

, et al., State-of-the-art review of soft computing applications in underground excavations, Geoscience Frontiers 11 (2020), 1095–1106.

A combined model based on feature selection and support vector machine for PM2.5 prediction

Abstract

Keywords

1 Introduction

2 Related work

2.1 Research status of feature selection

2.2 Research status of SVM

3 CBL method

4.1 Construction of CBL-PSO-SVM

4.2 Data standardization processing

5.1 Experimental data

Table 1 Characteristics of experimental data Number Feature Unit 1 station – 2 year year 3 month month 4 day day 5 hour hour 6 PM2.5 mg/m3 7 PM10 mg/m3 8 SO2 μg/m3 9 NO2 μg/m3 10 CO mg/m3 11 O3 μg/m3 12 TEMP °C 13 PRES Pa 14 DEWP °C 15 RAIN mm 16 WSPM m/s

Footnotes

Acknowledgments

References

Table 1
Characteristics of experimental data

Number Feature Unit

1 station –

2 year year

3 month month

4 day day

5 hour hour

6 PM2.5 mg/m³

7 PM10 mg/m³

8 SO₂ μg/m³

9 NO₂ μg/m³

10 CO mg/m³

11 O₃ μg/m³

12 TEMP °C

13 PRES Pa

14 DEWP °C

15 RAIN mm

16 WSPM m/s