Intelligent sports feature recognition system based on texture feature extraction and SVM parameter selection

SVM research begins with the analysis of the optimal classification plane in the case of linear separability.

The given training set is: { ( x 1 , y 1 ) , ( x 2 , y 2 ) , ⋯ , ( x i , y i ) } , x ∈ R d , y ∈ { + 1 , - 1 }

In the formula, x is the input vector, i is the number of samples, d is the dimension of the input vector, and y is the category to which the input vector belongs.

The hyperplane equation in the case of linear separability is: w · x + b = 0(1)

In the formula, w is the normal vector of the hyperplane, and b is the distance of the hyperplane from the origin. y i [ ( w · x i ) + b ] - 1 ⩾ 0 i = 1 , 2 , ⋯ , n(2)

The optimal classification line requires that the classification line not only correctly separate the two types, but also maximize the classification interval, as shown in Fig. 3. After being promoted to a high-dimensional space, the classification line is the classification surface. In the figure, solid points and hollow points represent two types of samples, and H is the optimal classification line. H₁ and H₂ are the straight lines of all kinds of samples that are closest to the classification line and parallel to the classification line. The distance d between them is called the classification interval (margin). d = min { x i : y i = 1 } | w · x i + b | ∥ w ∥ + min { x i : y i = - 1 } | w · x j + b | ∥ w ∥(3)

OPEN IN VIEWER

Fig. 3

Schematic diagram of the optimal classification surface.

From Equation (3), it can be obtained that the classification interval is 2/  -∥ w ∥. Therefore, the maximum classification interval is equivalent to minimizing ∥w∥. Therefore, the classification surface that satisfies the Equation (3) and has the largest classification interval is called the optimal classification surface. The training sample points on H₁ and H₂ are called support vector machines. Maximizing the classification interval refers to the control of the promotion ability, which is one of the core ideas of the support vector machine.

For the linear separable case, it is necessary to minimize the functional solution to solve the optimal hyperplane under constraint (2). Φ ( w ) = 1 2 ∥ w ∥ 2(4)

This optimization problem can be solved by introducing a Lagrangian function: L ( w , b , α ) = 1 2 ∥ w ∥ 2 - ∑ i = 1 n α i [ y i ( w · x i + b ) - 1 ](5)

In the formula, α _i is a Lagrangian multiplier. Through the optimal solution condition (Karush-Kuhn-Tuchke condition), the above optimization problem can be transformed into the dual problem of Equation (2). The maximum functional is: W ( α ) = ∑ i = 1 n α i - 1 2 ∑ i , j = 1 n α i α j y i y j ( x i · x j )(6)

The constraints are: ∑ i = 1 n y i α i = 0 , α i ⩾ 0 , i = 1 , 2 , ⋯ , n(7)

This is a quadratic programming problem with inequality constraints, so there is a unique solution. Only the support vector SV can have a non-zero coefficient α _i . The final decision function is: f ( x ) = sgn ( ∑ SV y i α i * ( x i · x ) + b * )(8)

For the linear indivisible case, in order to construct the optimal hyperplane, the inequality constraints must be relaxed to accommodate linear indivisible data. A slack variable ξ _i  (ξ _i  ⩾ 0) can be introduced in constraint formula (2). Then, the constraint becomes: y i [ ( w · x i ) + b ] ⩾ 1 - ξ i(9)

Obviously, ξ _i is greater than zero when there is an error in the partition. The corresponding objective function is: Φ ( w , ξ ) = 1 2 ∥ w ∥ 2 + C ∑ i = 1 n ξ i(10) ∑ i = 1 n y i α i = 0 , 0 ⩽ α i ⩽ C , i = 1 , 2 , ⋯ , n(11)

In the case of linear inseparability, the dual problem of the generalized optimal classification surface is almost identical to the linear separable case (6).

When the sample is linearly inseparable, the training set sample can be mapped to the high-dimensional feature space by a nonlinear function, and the nonlinear problem of the input space can be transformed into a linear problem of a high-dimensional feature space, so that the optimal classification surface is sought in the transformation space, and the decision function of the classifier is obtained, as shown in Fig. 4.

OPEN IN VIEWER

Fig. 4

Example of transforming input space nonlinear problem into high dimensional spatial linear problem.

The mapped high-dimensional space may be finite or infinite, and this mapping can be expressed as

x → Φ ( x ) = ( a 1 Φ 1 ( x ) , a 2 Φ 2 ( x ) , ⋯ , a n Φ n ( x ) ) a n ∈ R , Φ n ∈ R(12)

In support vector machines, the specific implementation of this mapping is achieved through a kernel function. K ( x i , x j ) = Φ ( x i ) · Φ ( x j )(13)

This article uses the Mercer condition. Mercer condition: For any symmetric function, it is a necessary and sufficient condition for an inner product in a feature space to have the following formula for any φ (x) = 0 and ∫φ² (x) dx< ∞. ∫ ∫ K ( x , x ′ ) φ ( x ) φ ( x ′ ) dxdx ′ > 0

It can be seen that the introduction of an appropriate kernel function K (x _i , x _j ) = φ (x _i ) · φ (x _j ) in the optimal hyperplane can replace the dot product operation of the high-dimensional feature space with a simple kernel operation in the low-dimensional input space, and it does not increase the computational complexity. At this time, the dual problem shown in Equation (6) is transformed into the following maximum functional: W ( α ) = ∑ i = 1 n α i - 1 2 ∑ i , j = 1 n α i α j y i y j ( x i · x j )(14)

The corresponding classification decision function is: f ( x ) = sgn ( ∑ SV y i α i * K ( x i · x ) + b * )(15)

Different inner product functions in the SVM algorithm will form different algorithms. The commonly used kernel functions are:

(1) Linear kernel function K ( x · x i ) = ( x · x i )(16)

(2) Polynomial kernel function (Gaussian kernel function) K ( x · x i ) = [ ( x · x i ) + 1 ] q(17)

In the formula, q is the order of the polynomial.

(3) Radial basis kernel function (RBF kernel function) K ( x · x i ) = exp { - | x - x i | 2 σ 2 }(18)

The resulting classifier is similar in form to the neural network RBF. However, the difference is that the center, network structure and network weight of the radial basis kernel function SVM classifier are automatically determined by the quadratic optimization algorithm, while in addition to the weight value, the neural network RBF is determined by heuristic method.

(4) Sigmoid kernel function K ( x · x i ) = tanh ( ( x · x i ) + c )(19)

In addition, there are other kernel functions such as spline kernel function, Fourier series kernel function, wavelet kernel function, and tensor product kernel function.

Image classification is essentially a complex multi-category problem, and SVM itself is a way to deal with two types of problems. Therefore, when dealing with multiple types of problems, the construction of the multi-classifier can be realized by combining a plurality of two-class classifiers. Considering the complexity of the algorithm and the accuracy of the classification, the multi-classification methods commonly used in practical applications are mainly one-against-rest and one-against-one methods. One-against-rest discriminant strategy, that is, a classifier separates each type of sample from other categories. One-against-one discriminant strategy, that is, a classifier is only used to classify two types of problems, and multiple types of classifiers are combined to complete multi-class identification.

We set the multi-class training sample set to { (x₁, y₁) , (x₂, y₂) , ⋯ , (x _l , y _l ) }, where l is the training sample set size, x _i ∈ R ^d , y _i  ∈ { 1, 2, ⋯ , k } and k is the number of categories. The two commonly used combination methods of classifiers are described below.

The one-against-rest method is the first method to use support vector machines for multi-classification problems. If there are k categories of sample sets, k SVM sub-classifiers are constructed. When training the i^th support vector machine, the samples belonging to the i^th class in the training sample use the positive class label, and the remaining samples not belonging to the i^th class use the negative class label. The corresponding optimization problems are as follows: min w i , b i ξ i 1 2 ∥ w i ∥ 2 + C i ∑ j = 1 l ξ j i s . t . w i · φ ( x j ) + b i - 1 + ξ j i ⩾ 0 , if y i = i w i · φ ( x j ) + b i + 1 - ξ j i ⩽ 0 , if y i ≠ i ξ j i ⩾ 0 , i = 1 , ⋯ l(20)

By solving k such problems, k decision functions are obtained. { f 1 ( x ) = sgn ( w 1 · φ ( x ) + b 1 ) ⋯ f k ( x ) = sgn ( w k · φ ( x ) + b k )(21)

When testing the sample x of the unknown category attribute, the sample x is substituted into the k decision functions of Equation (18), and the category corresponding to the largest discriminant function value is selected as the category of the sample x: Class arg max i = 1 , ⋯ , k ( w i · φ ( x ) + b i )(22)

The method is simple in testing and fast in decision-making and can be used for large-scale data. However, when the number of training samples is large, the training speed is slow, the classification efficiency is not high, and there is an inseparable region.

The one-against-one method is to construct an SVM sub-classifier between every two types of problems, and a total of k (k - 1)/2 SVM sub-classifiers need to be constructed. The test samples are sequentially input into the constructed k (k - 1)/2 SVM sub-classifiers to be classified, and then the samples is cumulatively superimposed, and the sample belongs to the class with the most number of superpositions. For the i and j classes, we solve the following two classification problems: min w ij , b ij ξ ij 1 2 ∥ w ij ∥ 2 + C ij ∑ j = 1 l ξ j ij s . t . w ij · φ ( x l ) + b ij - 1 + ξ l ij ⩾ 0 , if y l = i w ij · φ ( x l ) + b ij + 1 - ξ j i ⩽ 0 , if y l ≠ i ξ l ij ⩾ 0 , i = 1 , ⋯ n(23)

By solving these k (k - 1)/2 optimization problems, we can get k (k - 1)/2 decision functions: f ij ( x ) = sgn ( w ij · φ ( x ) + b ij ) i . j = 1 , 2 , ⋯ , k , i ≠ j(24)

The advantage of this method is that it is simple to train and has high classification accuracy. However, the disadvantage is that there are a large number of support vector machines that need to be constructed. Moreover, for classification problems with a large number of categories, this method is slow to train, and as if the one-to-rest method, has an inseparable region.

5.1 Selection of kernel functions

The kernel function K (x · x _i ) can construct a learning machine that implements different types of nonlinear decisions in the input space, thereby generating different support vector machine algorithms. Therefore, the kernel function plays a crucial role in SVM and directly affects the performance of SVM recognition. However, there is still no guiding principle for the choice of kernel function.

When classifying, it is generally selected 0-1 loss function and k-fold cross-validation. The first is to randomly divide l training points into k disjoint subsets: S₁, S₂, ⋯ , S _k and make each subset roughly equal in size. Then, k training and testing were performed. i = 1, 2, ⋯ , k is iterated k times. After the iteration is complete, we obtain l₁, ⋯ , l _k . The ratio of the number of error classifications ∑ i = 1 k l i for all k iterations to the total number of training points l is: ∑ i = 1 k l i(25)

It is called the k-fold cross validation error.

In general, there are two ways to take k: when taking k = 10, we can get 10-fold cross- validation and 10-fold cross- validation error. when taking k = l, S₁ = (x₁, y₁) , S₂ = (x₂, y₂) , ⋯ , S _l  = (x _l , y _l ). Moreover, for each iteration, one training point is left as a test point, and the remaining training points constitute the training set used in the algorithm, which is also called leave-one-out (LOO). At this time, the l-folding cross- validation error is called the LOO error.

When the kernel function is selected, the smaller of the 10-fold cross-validation error and the LOO error of any two kernel function algorithms is superior. Based on the theoretical analysis, this paper examines the classification ability of several types of kernel functions through simulation experiments. In this paper, three types of images, particle images, wood grain images, and stripe images in the Brodatz texture library are used for experiments, as shown in Fig. 5.

OPEN IN VIEWER

Fig. 5

Three types of images used to determine the kernel function.

First, the selected three images are equally divided into 32 sub-pictures that do not overlap each other, so that a new image library composed of 96 images is obtained. Select 20 images from each type of image, a total of 60 as the training sample set T1. Afterwards, the above four kernel functions are used as the kernel functions of the SVM classifier, and the SVM classifier is built to train the training sample set T1 to obtain four different classification models. The remaining 12 sheets in each type of image were selected, and a total of 36 sheets were used as the test sample set T2. After that, the T2 is tested by using the four classification models that have been established, and the test results are shown in Table 1.

It can be seen from the above table that the classification ability of RBF kernel function is higher than that of the other three types of kernel functions in terms of recognition accuracy and recognition time. Therefore, this paper uses the RBF kernel function as the kernel function of the support vector machine model.

For an SVM based on the RBF kernel function, its performance is determined by the parameter (C, σ). When different penalty factors C and σ are selected, the classification performance of the support vector machine will be very different. Therefore, our goal is to find the best combination of parameters to make the performance of the SVM the best, that is, the promotion error rate is the lowest.

C is a positive constant (penalty parameter) that can tend to infinity. After the kernel function is determined, the mapping function and feature space are determined.

If the value of σ is not appropriate, the SVM will not achieve the expected learning effect. At present, how to choose the appropriate parameters is one of the difficulties in the research of support vector machines. For different problems, a specific analysis is needed. This paper uses the violent test tool grid to test the penalty factor C and the kernel parameter σ of the RBF kernel function. The result is that the classification accuracy is the highest when C = 147, σ = 1.

5.3 Selection of training samples

The support vector machine is utilized to solve the superiority of the small sample problem, and the training sample number is optimized to improve the recognition efficiency. The mean and variance of the extracted low frequency sub-band image and the energy of the high frequency sub-band image are grouped into a feature vector group [f₂, f₄] and input to a support vector machine. The support vector machine kernel function selects the RBF kernel function, the penalty factor is C = 147, the kernel parameter is σ = 1, and a one-to-one classifier is used. Under the premise of the constant number of test samples, the number of training samples is 100, 150, 200, 250, 270, 300, 320, respectively. The experimental results are shown in Table 2.

Through experimental simulation, it can be seen that when the number of training samples is 270, the recognition rate no longer changes. After that, increasing the number of test samples will only increase the complexity and time of the calculation. Therefore, the recognition efficiency can be improved by appropriately reducing the number of training samples.