Applying SVR-PPSO,SVR-SSO,and SVR-BBO to estimate california bearing capacity of stabilized pond ash using admixtures

2.1 Dataset explanation

To train the proposed models for predicting the CBR value, a dataset comprising 109 experimental measurements was collected from published documents (refer to Table 1) [38]. Eight different influencing variables to the value of output were taken into account as input parameters. The statistical explanation of the dependent and independent parameters utilized for proposing the integrated models is supplied in Table 2. Fig. 1 presents a multi-Y scatter plot illustrating the connection between the input and output variables, visually depicting their interrelation.

OPEN IN VIEWER

Fig. 1

The scatter of input and output variables.

Support vector machines are regression and classification models derived from statistical learning theory. The Support vector machines-based classification methods are based on the generality of the best separation of classes. If the classes are separable –this method selects, from among the unlimited number of linear classifiers, the one that minimizes the generalization error, or at least an upper bound on this error, derived from structural risk minimization. SVR has additive advantages when compared to other regression methods [39, 40]. Considering a training dataset { (x _i ,  y _i ) ,  I = 1,  …  n }, where n is the size of the training dataset, (x _i ) is the input vector (y _i ) is the output vector, respectively.

f ( x ) = 〈 m , x 〉 + d(1) where mɛR ^N and d ɛ R . 〈m, x〉 represents the dot product in space R ^N Vector m in Equation (1) specifies the orientation of a discriminant plane, while scalar d determines the distance between the discriminant plane and the origin.

Minimizing the Euclidean norm leads to a smaller value of m, indicating the flatter nature of Equation (1) defined by ∥m ∥ ². Thus, an optimization problem for regression can be written as:

minimize 1 2 ∥ m ∥ 2(2)

subject to { y i - 〈 m , x i 〉 - d ≤ ɛ 〈 m , x i 〉 + d - y i ≤ ɛ(3)

Equation (2) assumes that there exists an expansion that can generate an error smaller than ɛ for all training pairs. However, in real-world scenarios, there may be situations, such as those encountered in classification problems, that do not conform to this assumption.so that’s why, to allow some more error, slack variables ξ, ξ can be optimized and to handle the infeasible constraints of the optimization problem defined in Equation (3), the following approach can be employed [41].

minimise 1 2 ∥ m ∥ 2 + G ∑ i = 1 n ( ξ i + ξ i ′ ) subject to y i - 〈 m , x i 〉 - d ≤ ɛ + ξ i 〈 m , x i 〉 + d - y i ≤ ≤ ɛ + ξ i ′ So ξ i , ξ i ′ ≥ 0 for all i = 1 , 2 , 3 … . n .(4)

The constant G > 0 is a parameter of user-defined which determines the trade-off between function flatness. Equation (3) defines the objective primal function in the minimization problem [42]. By transforming the problem into a dual space representation using Lagrangian multipliers, the inequalities can be replaced with a simpler form, allowing the optimization problem to be solved more easily. Equation (5) the lagrangian can be formed by introducing positive multipliers λ i , λ i ′ , η i , η i ′ , i = 1 , 2 , 3 , 4 , … , n and constraint equations multiplying by these multipliers, and finally subtracting the results from the objective function (i . e, ∥ m ∥ ²) in Equation (5). The lagrangian can be written as:

P = 1 2 ∥ m ∥ 2 + G ∑ i = 1 n ( ξ i + ξ i ′ ) - ∑ i = 1 n λ i ( ɛ + ξ i - y i + 〈 m , x i 〉 + d )(5)

- ∑ i = i n λ i ′ ( ɛ + ξ i ′ + y i - 〈 m , x i 〉 - d ) - ∑ i = 1 n ( η i ξ i + η i ′ ξ i ′ )(6)

In Equation (6), the dual variables have to suffice λ i , λ i ′ , η i , η i ′ ≥ 0 . The formalin of the optimization problem involved in the specification of SVR can be sourced in Equation (6) by locating the saddle point of the Lagrange function defined. Equation (6) saddle points can be obtained by partial equate derivative of P concerning ξ _i and ξ i ′ , m , d , to 0 and get:

∂ m P = m - ∑ i = 1 n ( λ i ′ - λ i ) . x i = 0(7)

∂ d P = ∑ i = 1 n ( λ i ′ - λ i ) = 0(8)

∂ ξ i P = G - λ i - η i = 0(9)

∂ ξ i ′ P = G - η i ′ - λ i ′ = 0(10)

Results of substituting Equations (7)–(10) in Equation (6) in the optimization problem of maximizing:

- 1 2 ∑ i = 1 n ∑ j = 1 n ( λ i ′ - λ i ) ( λ j ′ - λ j ) ( x i . x j ) - ɛ ∑ i = 1 n ( λ i ′ + λ i ) + ∑ i = 1 n y i ( λ i ′ - λ i )(11)

subject to ∑ i = 1 n ( λ i ′ - λ i ) = 0 and λ i , λ i ′ ∈ [ 0 , G ](12)

In Equations (9) and (10) dual variables η i , η i ′ are eliminated by using conditions and can now be written as λ i ′ = G - η i ′ and λ i = G - η i , in Equation (13), the prediction problem can now be written as:

f ( X ) = ∑ i = 1 n ( λ i ′ - λ i ) 〈 x i , x 〉 + d(13)

In Equation (7), the regression problem in feature space can be written by replacing x_j . x _i with Φ  (x _j ) .   Φ  (x _i ). Thus, in Equation (11), the optimization problem can be written as:

maximize - 1 2 ∑ i = 1 n ∑ j = 1 n ( λ j ′ - λ j ) ( λ i ′ - λ i ) N ( x j . x i ) - ɛ ∑ i = 1 n ( λ i + λ i ′ ) + ∑ i = 1 n y i ( λ i - λ i ′ )(14)

subject to ∑ i = 1 n ( λ i - λ i ′ ) = 0 and λ i ′ , λ i ∈ [ 0 , G ](15)

N ( x j , x i ) ≡ Φ ( x j ) . Φ ( x i )(16)

In Equation (8), the function of SVR can now be written as:

f ( x ) = ∑ i = 1 n ( λ i - λ i ′ ) N ( x , x i ) + d(17)

Finally, the kernel function. In this optimization problem, it is computed rather than Φ  (x) to reduce the computational cost of dealing with the high-dimension feature space [43].

This subsection presents an overview of the biogeography-based optimization algorithm. Imagine there is an optimization problem along with a set of candidate solutions. In this context, a favorable solution can be compared to a habitat with a high habitat suitability index (HSI), analogous to a geographical area well-suited for biological species in biogeography. Within the optimization problem, the HSI serves as a measure of the solution’s quality, often referred to as fitness or goodness of the habitat [46]. Conversely, a subpar solution resembles a habitat with a low HSI. High Habitat Suitability Index (HSI) in solutions indicates habitats that support a diverse array of species, whereas low HSI solutions signify habitats with fewer species. In essence, the number of species represented by a solution depends on its corresponding HSI. Solutions with high HSI are more likely to share their characteristics with other solutions, while solutions with low HSI are more inclined to accept shared attributes from other solutions. This novel approach to address general optimization problems is biogeography-based optimization (BBO). Like other evolutionary algorithms, BBO relies on two crucial steps: information sharing, achieved through migration in BBO, and mutation. Migration, as a probabilistic operator, enhances a habitat, denoted as H _i . The migration rates of each habitat are utilized to probabilistically share features among habitats. For every habitat H _i , its immigration rate, K _i , is employed to determine probabilistically whether it should undergo immigration [47]. If immigration is chosen, then the emigrating habitat, H _j , is also selected probabilistically based on the emigration rate, l _j . In this context, migration is defined as the process by which features are exchanged between habitats to improve their overall optimization performance.

H i ( SIV ) ← H j ( SIV )(25)

In the field of biogeography, an SIV, known as a Suitability Index Variable, plays a crucial role in assessing an island’s livability. Within the context of BBO an SIV represents a characteristic of a solution, analogous to a gene in GAs. Mutation serves as a probabilistic operator responsible for randomly altering an island’s SIV, relying on the pre-existing probability of species count in that habitat [48, 49]. Mutation serves the primary objective of promoting diversity within the population. In the case of low HSI solutions, mutation offers an opportunity to enhance the quality of these solutions. Conversely, for high HSI solutions, mutation can further improve them beyond their current state. The pseudocode for BBO′s migration operator is depicted in Algorithm 1.

SSO [50] is a novel optimization technique that addresses diverse optimization problems. This method emulates the movement of salps, barrel-shaped plankton belonging to the Salp family, found in the wild. Salps share similarities with jellyfish in terms of their tissue structure, and their movement is primarily driven by the expulsion and contraction of water within their gel-like bodies, which have a high water content. By imitating this natural behavior, the optimization method finds solutions efficiently for various optimization challenges [51, 52]. The marine salp exhibits a herding behavior known as the salp chain, which aids them in inefficient movement and feeding by adapting swiftly to changes in their environment. Drawing inspiration from this behavior, the algorithm has mathematically modeled and applied the Salp chain to test its effectiveness in solving an optimization problem.

SSO initiates by segregating the population into two categories: leaders and followers. The rest of the salps are referred to as followers, while the salp preceding them in the sequence assumes the role of the leader. The salp’s position is determined in an n-dimensional space, representing the search space of the problem, with n denoting the number of variables under consideration. These salps diligently search for food sources, which indicate the direction of the entire herd’s destination. To frequently update the location of the Salp leader, the following formula is employed for this purpose:

y j 1 = { P j + b 1 ( ( u d j - l d j ) b 2 + l d j ) b 3 ≤ 0 P j + b 1 ( ( u d j - l d j ) b 2 - l d j ) b 3 > 0(26)

Here y j 1 is the leader’s position in the j ^th dimension, the food source in that dimension is P _j , and the lower and upper bounds are ld and ud _j , respectively. b₂ and b₃ are randomly generated in the range [0, 1] to maintain explore space. The parameter b₁ plays a crucial role in this algorithm as it acts as a key factor in balancing the search stage and the utilization stage. Its calculation is carried out as follows:

n 1 = 2 e - ( 4 l l max ) 2(27)

Here l _max and l show the maximum number of iterations and the current repetition, respectively. Once the leader’s position has been updated, SSA proceeds to update the position of the followers using the following formula:

y j i = 1 2 ( y j i + y j i - 1 )(28)

y j i is the ith position of follower in the jth dimension, and (i) is higher than 1.

The pseudocode for the SSA algorithm is illustrated in Algorithm 2.

In machine learning models, hybridization involves amalgamating various algorithms, techniques, or approaches to form a cohesive, unified model. The purpose of hybridization is to capitalize on the unique strengths of each component, resulting in improved overall performance, accuracy, or efficiency compared to using any single method in isolation. Hybridizing Support Vector Regression (SVR) with different optimization algorithms can potentially lead to improved performance and model robustness. In all three cases, the primary objective is to find the best hyperparameters for the SVR model to achieve better generalization and performance on the given dataset. By leveraging the strengths of each optimization algorithm, the hybridized approach can potentially enhance the model’s ability to handle complex relationships and improve its overall accuracy. However, it is essential to conduct thorough experimentation and fine-tuning to ensure the hybridized model’s optimal configuration. To enhance comprehension of the procedure utilized in this study, Fig. 3 has been provided.

OPEN IN VIEWER

Fig. 3

The process of Hybridization.

Various appraisals were utilized to evaluate the workability of models for forecasting the CBR value. Coefficient of determination (R²), root mean squared error (RMSE), mean absolute percentage error (MAPE), mean absolute error (MAE), performance index (PI), and OBJ (Equations (29)–(33)):

R 2 = ( ∑ p = 1 P ( t P - t ¯ ) ( y P - y ¯ ) [ ∑ p = 1 P ( t P - t ¯ ) 2 ] [ ∑ p = 1 P ( y P - y ¯ ) 2 ] ) 2(29)

RMSE = 1 P ∑ p = 1 P ( y p - t p ) 2(30)

MAE = 1 P ∑ p = 1 P | y p - t p |(31)

PI = 1 | t ¯ | RMSE R 2 + 1(32)

OBJ = n train P * ( RMS E train + MA E train R train 2 + 1 ) + n test P * ( RMS E test + MA E test R test 2 + 1 )(33) where, y _P , t _P , t ¯ , and y ¯ represent the predicted values of the Pth pattern, the target values of the Pth pattern, the averages of the target values, and the averages of the predicted values, respectively.

3.1 Convergence curve analysis

Figure 4 illustrates the convergence curve of the proposed machine learning models, also known as a learning curve, indicating how model performance evolves or stabilizes with increasing training data. The y-axis represents the RMSE, while the x-axis displays the number of iterations. Analyzing this curve provides valuable insights into the model’s learning behavior and facilitates data-driven decisions concerning the need for additional training data or whether the model has reached its optimal performance. The initial iterations demonstrate the steepness of the models’ learning, and some overlap exists among the models. However, beyond the 90th iteration, all models attain their optimal results. Remarkably, SVR-SSO stands out with its superior performance relative to the other models, as evident from the results.

OPEN IN VIEWER

Fig. 4

The convergence curve for the presented hybrid models.

This section presents the results of training the admixture inputs of the CBR using machine learning methods. The classical SVR model achieved acceptable results but with a lack of accuracy. Hybridization techniques were employed to address this, utilizing PPSO, BBO, and SSO to enhance the SVR model’s performance. The proposed hybrid models are denoted as SVR-PPSO, SVR-BBO, and SVR-SSO. The evaluation of these models is carried out in two stages: training and testing. The dataset consists of a total of 109 CBR samples, which are utilized in both stages. More specifically, 70% of the data is allocated for training, while the remaining 30% is used for testing. To ensure unbiased and robust model performance comparisons, various performance evaluators, including R², RMSE, MAE, PI, and OBJ, were utilized. These evaluators provide a comprehensive assessment of the models and enable a thorough evaluation of their effectiveness. In the context of the evaluators, a high R2 value approaching 1 indicates an optimal result, while low values for RMSE, MAR, PI, and OBJ, approaching 0, indicate an optimal outcome. Table 3 presents the mentioned models’ results and their corresponding metrics. A rank score ranging from 1 to 4 has been provided to facilitate a more effective comparison of the models. The highest accuracy result for each metric is assigned rank 4, while the least accurate results are assigned rank 1.

As shown in Table 3, the results clearly indicate the SVR single form’s lack of accuracy compared to the other models, as evidenced by its RMSE value of 2.172 in the training phase and 2.604 in the testing phase. However, when applying optimizers to SVR, a significant improvement of at least 20% in its lowest RMSE is observed, demonstrating the effectiveness of the applied optimizers. The SVR-SSO model achieved the highest accuracy among all the other models, demonstrated by the highest R² value and the lowest error indicators. Notably, the SVR-SSO model’s accuracy further improved during the testing phase compared to its performance during the training phase. These results underscore the model’s effectiveness in predicting CBR values with precision. However, despite the improvement observed in the other two hybrid models compared to the single form of SVR, their accuracy still falls short of the SVR-SSO model. The SVR-SSO model outperforms the SVR-PPSO model by 45% and the SVR-BBO model by 58% in terms of the lowest RMSE.

Figure 5 presents a scatter plot illustrating the correlation between the measured and predicted values of the proposed hybrid models, allowing for an assessment of their accuracy. The plot differentiates the values from the train and test phases using distinct colored squares scattered around a center line that represents an R² value of 1. The y-axis represents the measured values, while the x-axis represents the predicted values. In an ideal scenario, the majority of train and test values are closely aligned with the center line, indicating the accuracy of the models. The proximity of the scattered points to the center line suggests that the predictions are in good agreement with the actual measurements, and the models’ performance is noteworthy. Upon observing the SVR-SSO scatter plot, it becomes evident that the test phase values are almost perfectly aligned with the center line, indicating an excellent correlation between the predicted and measured values. Furthermore, the training phase values show a maximum level of correlation with the center line, signifying the model’s strong performance in accurately predicting both the training and test data points. The proximity of the scattered points to the center line in both phases demonstrates the remarkable accuracy and effectiveness of the SVR-SSO model in this analysis. However, both the SVR-PPSO and SVR-BBO models exhibit similar behavior in their performance, with their data showing an acceptable dispersion around the center line. While they do not achieve the same level of perfect alignment as the SVR-SSO model, the scattered points for both SVR-PPSO and SVR-BBO remain close to the center line. This indicates a reasonable correlation between the predicted and measured values for both training and test phases. Though not as accurate as the SVR-SSO model, the SVR-PPSO and SVR-BBO models still demonstrate commendable performance in their predictions, making them valuable alternatives in this analysis.

OPEN IN VIEWER

Fig. 5

The scatter plot for the correlation between predicted and measured CBR.

The line-symbol plot in Fig. 6 offers a visually detailed representation of the correlation between the measured and predicted values of the hybrid models. The measured values are depicted by a black line, and the alignment of the train and test values with this line indicates the accuracy of the models. By using this plot, it becomes evident how closely the train and test values align with the measured value line, allowing for a quick assessment of each model’s accuracy. The clarity of the plot aids in visualizing the performance of the hybrid models and provides valuable insights into the quality of their predictions.

OPEN IN VIEWER

Fig. 6

The line-symbol plot for the comparison between predicted and measured CBR.

OPEN IN VIEWER

Fig. 7

The error percentage for the column plot of developed models.

To better visualize the models’ performance, Figs. 7 and 8 have been provided to indicate the errors of the models. As evident from the figures, the majority of SVR-SSO data values are below 5 percent error, with a maximum error of 7.74% occurring between sample numbers 30 to 40. Figure 7 reinforces the low error percentage, with the majority of data points having errors close to 0 and distributed below 5% error.

OPEN IN VIEWER

Fig. 8

The scatter interval plot for the error percentage of presented models.

However, SVR-PPSO and SVR-BBO exhibit higher error rates of 10.5% and 11.79%, respectively, both occurring in the training phase. This is evident when observing their error balance, ranging between -10 to 10, and the distribution of errors is more pronounced in Fig. 7. In contrast, the error distribution of SVR-BBO exhibits a notably flatter normal distribution in comparison to alternative models, as discernible from Fig. 8. The distribution of errors in SVR-BBO, as visually represented in Fig. 8, is distinguishably flatter when compared to other models. Conversely, SVR-SSO demonstrates a higher concentration of data points near zero percent, indicative of heightened model precision and a distinctively sharper normal distribution. This observation underscores the superiority of SVR-SSO in terms of predictive accuracy and model performance, as evidenced by the distribution characteristics displayed in Fig. 8. Such nuanced distinctions within the error distributions, as portrayed in Fig. 8, shed light on the efficacy of different optimization strategies when applied to support vector regression.

Figure 9 illustrates a Taylor diagram comparing CBR’s predicted and measured values. The Taylor diagram serves as a valuable tool for evaluating the performance of multiple models in relation to a reference dataset, which, in this case, is the CBR value. In Fig. 9, the correlation coefficient is represented on a scale from -1 to 1, indicating the extent of agreement or disagreement between the model predictions and the observed CBR values. A correlation coefficient of 1 denotes a perfect match, while -1 indicates a perfect inverse relationship. Additionally, the standard deviation is depicted on a scale from 0 to 15, signifying the variability of the model predictions in comparison to the measured CBR values. A lower standard deviation indicates greater consistency and agreement between the models and the reference dataset. In Fig. 9, the green, pink, and blue dots represent the SVR-SSO, SVR-PPSO, and SVR-BBO models, respectively. Upon analyzing the diagram, it becomes evident that the SVR-SSO model is the closest to the reference point, indicating a suitable match with the target. Both SVR-BBO and SVR-PPSO demonstrate a close performance, but upon closer inspection, it can be observed that SVR-PPSO slightly outperforms SVR-BBO.

OPEN IN VIEWER

Fig. 9

The Taylor diagram for the predicted and measured CBR.