A new parallel galactic swarm optimization algorithm for training artificial neural networks

Abstract

Metaheuristic algorithms are a family of algorithms that help solve NP-hard problems by providing near-optimal solutions in a reasonable amount of time. Galactic Swarm Optimization (GSO) is the state-of-the-art metaheuristic algorithm that takes inspiration from the motion of stars and galaxies under the influence of gravity. In this paper, a new scalable algorithm is proposed to help overcome the inherent sequential nature of GSO and helps the modified version of the GSO algorithm to utilize the full computing capacity of the hardware efficiently. The modified algorithm includes new features to tackle the problem of training an Artificial Neural Network. The proposed algorithm is compared with Stochastic Gradient Descent based on performance and accuracy. The algorithm’s performance was evaluated based on per-CPU utilization on multiple platforms. Experimental results have shown that PGSO outperforms GSO and other competitors like PSO in a variety of challenging settings.

Keywords

nature inspired metaheuristic parallel computation galactic swarm optimization artificial neural networks

1 Introduction

"Look deep into nature and then you will understand everything better" - Albert Einstein.

Historically man has looked up to nature for seeking clues and inspiration for inventions. From taking inspiration from birds for constructing airplanes to observing swarm movements to inspire the creation of the family of meta-heuristic algorithms. For example - Genetic Algorithm is inspired by Darwin’s theory of evolution [8]. The Particle Swarm Algorithm is designed based on the flocking behavior of birds [7]. Each algorithm is suited for a certain type of problem. The choice boils down to what the application requires. Each algorithm tinkers the balance between exploration and exploitation phase. An algorithm which correctly balances the exploration and exploitation phase with multiple iterations of each cycle,in general, becomes a strong performer. These characteristics are shown by the GSO algorithm.

Our Universe has a certain balance and symmetry in the way objects interact with each other, everything revolves around something. The electrons around the nucleus, the planets around the sun, the stars around a galaxy center, the galaxies act like dots in the universe and revolve around what we today call superclusters [13]. GSO algorithm takes inspiration from this hierarchical motion. The point masses in the universe can be taken as solutions in a search space. These solutions are improved by complex communications among themselves. The solutions are then improved further by combining them. The universe is our search space and each point mass is an explorer. Notice that, each explorer has the responsibility of revolving around locally and also being part of a bigger revolving entity that gets to visit other parts of the universe. In heuristic terms, thereby providing the correct balance in terms of exploration and exploitation phases.

In the GSO algorithm [13], the stars revolve around galaxy centers. The galaxy revolves around the supercluster center. Each galaxy is represented by their global best solution which is also the galaxy center.

GSO can have any swarm algorithm mimicking the notion of a search for a solution starting from some random location in our bounds. In this paper, the focus is on using PSO as the algorithm playing the role of a galaxy with stars as the particles. This choice is made because of its faster converging property which will be useful when describing how to train a Neural Network with PGSO in Section 4. The GSO algorithm works in 2 stages. Stage 1 has multiple PSOs running independently. Having multiple independent swarms explore search space helps independent exploration. This guarantees robustness in solution and deals with multimodalities. Stage 2 involves using individual best solutions as initialization points to start another PSO. This step enforces the exploitation phase which guarantees faster convergence since the initialization points are themselves sub-optimal local minima. Modifications were made to this architecture to make it scalable. The performance was evaluated based on the amount of work done by 2 implementations (sequential and parallel) in the same amount of time rather than time comparisons of what portion of the code is parallel and what is not. The architecture modifications are highlighted in Section 3.

The rest of the paper is organized as follows. Literature Review is presented in Section 2, the algorithm’s architecture is presented in Section 3, the methodology that was followed to design the architecture is presented in Section 4. The Artificial Neural Network training procedure is presented in Section 5. Detailed performance analysis in terms of latency, average CPU utilization and evaluation against benchmark suite is presented in Section 6. The discussion and conclusion is presented in Section 7.

2 Related work

Parallel metaheuristic algorithms have found their use in various fields of science, business and industry. Some applications being - Vehicle Routing [11], Scheduling, flood management [2], designing strong car chassis [15], training artificial neural networks [10] and more. Our focus in this paper, is on exploring how the PGSO algorithm performs on optimizing an Artificial Neural Network. Let us first look at the classical approaches that are used to parallelize a population-based metaheuritic algorithm -

Population parallelization

Computational parallelization

Population parallelization involves letting each explorer of your swarm to work independently and simultaneously along with other members, whereas, the computational parallelization involves splitting the computational intensive region to be computed in parallel. It may also be the case that each swarm is considered as a single unit and that multiple swarms run simultaneously and explore the same search space. The authors have adopted the latter approach.

Popular algorithms like Ant Colony Optimization have been parallelized using SIMD(Single Instruction Multiple Data) format by [5]. In this implementation, each ant’s walk over the graph is treated as an independent task, where each ant’s pheromone trail is recorded in independent matrices τ_j. The only drawback of this implementation is that all the independent matrices need to be merged at the end of a cycle which is a highly computationally expensive O(n²) operation. For the problem of parallelizing the Artificial Bee Colony algorithm, after employing multiple parallelizing strategies [16] found that giving a coarser level (swarm level) task to the CPU helps improve the accuracy and keep CPU cores busy for a longer time. [14] have parallelized the Genetic Algorithm [8] to train an Artificial Neural Network. The Genetic Algorithm intrinsically allows flexible distribution of tasks. The master directs sub-population of candidate solutions to slave nodes using a message passing interface. They got significant speedups from the parallel and distributed implementation.

2.1 PSO - particle swarm optimization

The author in [6] proposed the Particle Swarm Optimization algorithm inspired by the flocking behaviour of birds. By far, it is one of the most popular algorithms to have successful applications and performance in various fields, as given by [1].

The velocity and position update equations 1,2 for the sequential algorithm are given below. Here v_j is the velocity of particle j, w represents the weight/importance to be given to the inertia of the particle, c1 (cognitive factor), c2 (social factor) weight represent the importance to be given to the best position found by the particle and the best position found among all particles respectively, r1, r2 are randomly initialized numbers, p_j, g_j, x_j represent the best-recorded position by the particle, best-recorded position globally and particle’s current position respectively.

$\begin{matrix} v_{j} = & w * v_{j} + c 1 * r 1 * (p_{j} - x_{j}) \\ + c 2 * r 2 * (g_{j} - x_{j}) \end{matrix}$ (1) $x_{j} = x_{j} + v_{j}$ (2)

2.2 PPSO - parallel particle swarm optimization

In [3] a novel algorithm is proposed called Parallel Particle Swarm Optimization(PPSO). The algorithm was applied to infinite impulse response image filters for designing image enhancement filters along with arbitrary frequency results to the multidimensional optimization problem. The experiment mainly consisted of two phases: The first phase was used for global exploration to explore multiple local minima, in which the PSO algorithm was divided into sub-populations and later on, best solutions from each sub-population are being interchanged between the cores. In the second phase, a local search is done to refine the solution computed by PSO, using the Nelder-Mead method. The author concluded that PPSO is giving better results as compared to other global optimization algorithms in terms of the mean square error between CPU usage and the ideal and designed filter frequency response. They used Computational Parallelization in their approach.

2.3 GSO - galactic swarm optimization

Muthiah-Nakarajan and Noel in [13] proposed Galactic Swarm Optimization(GSO) which is a new algorithm, inspired by the motion of stars, galaxies and supercluster of galaxies. This paper used a 2 phase approach similar to 2.1. In this paper, two phases were being discussed namely - exploration and exploration. In the explorations phase, sub-population explores the search space independently whereas, in the exploitation phase, the best solutions of sub-populations clustered together are called as supercluster, which moves towards best solutions. They proposed a methodology in which both sub-populations, as well as superwarm, were being updated and their statistical results showed GSO algorithms converges faster to more reasonable and accurate solutions compared to state-of-the-art PSO algorithms on a wide variety of high dimensional and multi-modal benchmark optimization problems.

The GSO algorithm is given in Algorithm 1.

[4] proposed a variant of GSO named Fuzzy Galactic swarm optimization (FGSO) for dynamic adjustment of parameters for the optimization of the benchmark. The term c1,c2 refers to social and cognitive factors in level 1, c3 and c4 refer to the social and cognitive factors in level 2. The fuzzy systems were of the Mamdani type in which the input was defined as the number of iterations and with two outputs. the first was with c3 in increase and c4 in decrease and the second variant was with c3 in decrease and c4 in the increase. To measure the performance of the algorithm with 17 benchmark functions with the different number of dimensions, they suggested different fuzzy systems for the dynamic adaptation of the c3 and c4 parameters. The experiments after comparing FGSO and GSO showed that their proposal achieved an improvement against the original algorithm in some cases where the original algorithm fails to reach zero. FGSO looked very intuitive in its approach to this problem. In essence, at a given time we have more explorers traversing and exploring the search space simultaneously. So when we move to higher dimensions we do not face exploration related problems as the FGSO and GSO faced. The validation for this effect has been provided in the results section.

3 Architecture

Coming up with a parallel version for a sequential algorithm is non-trivial. It involves the need to think parallel in a bottom-up manner to maximize gains. If the need arises even modifying the algorithm’s logic could be the right recourse given that there is no significant trade-off with accuracy. The model was evaluated based on how much work it can do in the same amount of time as the serial algorithm. In essence, it must be able to explore the search space more exhaustively in the same amount of time. The Gustafson-Barsis’s Law is based on evaluating performance based on work is given by 3 $S_{scaled} (s) = (1 - p) + sp$ (3) where S_scaled (s) is the speedup with respect to the latency of the execution of the whole task, s is the speedup in latency of execution of the parallelizable portion of the algorithm and p is the execution workload of the parallelizable code before the code was parallelized. The Parallel Galactic Swarm Optimization’s algorithm is given in Algorithm 2

4 Methodology

We have applied SIMD (Single Instruction Multiple Data) based parallelism to the outermost loop which run’s upto M times in 2. This means we are effectively forking out M threads. In terms of initial architecture design, the authors refrained from using any shared data structure for exchanging information in this fork procedure. Doing so would have introduced latency due to synchronization. But in practice we found that exchanging information about the best solution globally after the galaxies have sufficiently explored the search space (about M/2 iterations) was found to give better results. Specifically, the global_best variable which stores the information about the best solution encountered among all explorers is present in the shared_list. Making it visible to all threads. In this paper, a lock region is introduced which gets activated after M/2 iterations, where each PSO thread updates itself on the location of the best solution globally. This trade-off is done to encourage exploitation. The explorers get a global picture of the search space which ultimately helps to converge quickly. This feature is used to obtain the solutions. Users can toggle this option in the implementation as per their convenience, should their use case require faster processing time. Toggling it off would then trade off a little bit of performance for the gain of speed. The authors could have gone a step further and could have sought ways to distribute finer-grained jobs to each processor instead of a PSO job. But doing so would have reduced the overall throughput and frequent collection of results from multiple fine-grained jobs would increase overall compute time.

5 Applications

5.1 Training a neural network with PGSO

An Artificial Neural Network (ANN) was trained with PGSO for tackling the Churn Prediction problem and the NBA player prediction problem. The former problem required the ANN to detect/classify whether a customer is going to cancel a subscription. The data collection procedure involved collecting the data on how frequently the customers use the subscribed product. The latter problem involves predicting based on a player’s statistics whether the player would last for the next 5 years. A 3-layer Neural Network with 6 nodes in the hidden layer was used for this purpose. The binary cross-entropy loss was used as the target function to optimize as given in equation 4 where $\hat{y}$ are the outputs obtained from forward pass on a neural network, N represents the number of samples fed as a batch to the neural network, y _ train are the corresponding ground truth values.

$L = \frac{\sum_{0}^{N} y_train * \log (\hat{y}) + (1 - y_train) * \log (1 - \hat{y})}{N}$ (4)

The PGSO algorithm was modified to adapt to the problem of training a neural network. Our problem was to estimate the best set of weights given a neural network architecture. The modified Neural Network architecture is given in Fig. 1. Imagine a situation where a baby is shown a million images of cats and dogs over and over again to make him understand how to recognize cats and dogs. This is what modern day algorithms for training a Neural Networks. In particular, Stochastic Gradient Descent has following drawbacks -

Wasteful of data - Too much recycling

Cannot approximate Non-differentiable functions

PGSO is a promising alternative towards training a neural network because it overcomes the above difficulties. In practice, PGSO could get similar amount of gains as the Adam Optimizer got within same on lesser amount of epochs for the datasets we have tested both of them on. Table 1 shows some of the datasets we tested the PGSO on.

Fig. 1

Training a Neural Network with PGSO.

Table 1

F1 scores for different data-sets

Optimizer	Churn Prediction		NBA Player Prediction
	Epochs	F1 score	Epochs	F1 score
Adam	100	0.84	100	0.68
PGSO	100	0.804	20	0.6791

6 Experiments and results

The performance of the proposed PGSO is evaluated based on the following metrics -

Performance on single thread (Latency)

Average CPU utilization

Evaluation against benchmark suite

6.1 Performance on a single thread

The algorithm was prototyped using Numba python library which allowed us to gain C like performance in python. Even though our code should have staggered against a single-thread implementation of the original GSO our implementation dominates the single-thread implementation of GSO and PSO even on a single thread. On the Rosenbrock function the PSO algorithm took (13.3 +/-309ms), the GSO algorithm took (13.8 +/-128ms) while the PGSO took (12.9 +/-70.1ms) using identical parameters for evaluation averaged over 50 runs.

Another observation taken using the Intel VTune Amplifier tool was that it took almost twice the time (25.828sec + -40ms) on a single thread without the use of Numba.

6.2 Average CPU utilization

When it comes to scaling, the emphasis was given on how efficiently was the algorithm able to utilize all the CPU resources over time and not let any core remain idle. This intuitively translated to that exhaustive exploration of the search space of possible solutions by employing more number of workers at the same time. The Intel VTune Amplifier and other libraries were used to measure average CPU utilization over time. Figure 2 illustrates that the algorithm is in the ideal range of average CPU utilization and most of the time the algorithm was able to utilize 6 to 7 out of 8 logical CPU cores.

Fig. 2

Average CPU utilization (Intel VTune Readings).

Fig. 3

Convergence Characteristics of Non Continuous Rastrigin.

Fig. 4

Convergence Characteristics of Ackley.

Fig. 5

Convergence Characteristics of Rastrigin.

Fig. 6

Convergence Characteristics of Rosenbrock.

Fig. 7

Convergence Characteristics of Rotated Rastrigin.

Fig. 8

Convergence Characteristics of Rotated Griewank.

Fig. 9

Convergence Characteristics of Rotated Ackley.

Fig. 10

Convergence Characteristics of Shifted Rotated Rastrigin.

6.3 Evaluation against benchmark suite

PGSO was evaluated against PSO, GSO over multiple benchmark functions which extend the DeJong Test Suite [12]. The benchmark functions are considered as a gold standard in the meta-heuristics community. The benchmark functions serve to mimic real-world problem’s optimization functions. These algorithm have a varying degree of complexity. with the sphere function being the simplest and the shifted rotated versions being the hardest. The functions chosen are multimodal, which means we can have multiple good solutions which may or may not be the best solution. It also implies that there are numerous opportunities for an algorithm to get stuck at local minimas. Besides, a common trend is recorded among metaheuristic algorithms that they tend to perform exponentially bad when the dimensions of search space for each of these functions is increased considerably. These properties make the benchmark suite an ideal framework for comparing our algorithm with others.

Tables 4 –6 shows how easily PGSO outclasses PSO and GSO over multiple benchmark functions. Notice how low the error values are in the case of PGSO and how big they are in the case of GSO and PSO. Moreover, when we try finding solutions for the benchmark functions in higher dimensions the performance gap becomes exponentially significant between PGSO and others. All instructions to reproduce the results, benchmarks along with the application of PGSO with neural network is properly documented along with code implementation.

Table 2
Unimodal benchmark functions

Function Definition Search Range Global minimum location x^*

Exponential $f_{1} (x) = \exp (- 0.5 \sum_{i = 1}^{D} x_{i}^{2})$ [-1, 1] ^D [0, 0, . . . , 0]

Powell Sum $f_{2} (x) = \sum_{i = 1}^{D} | x_{i} |^{i + 1}$ [-1, 1] ^D [0, 0, . . . , 0]

Sum of Squares $f_{3} (x) = \sum_{i = 1}^{D} {ix}_{i}^{2}$ [-10, 10] ^D [0, 0, . . . , 0]

Schwefel 2.20 $f_{4} (x) = \sum_{i = 1}^{D} | x_{i} |$ [-100, 100] ^D [0, 0, . . . , 0]

Schwefel 2.22 $f_{5} (x) = \sum_{i = 1}^{D} | x_{i} | + \prod_{i = 1}^{D} | x_{i} |$ [-100, 100] ^D [0, 0, . . . , 0]

Griewank $f_{6} (x) = 1 + \sum_{i = 1}^{D} \frac{x_{i}^{2}}{4000} - \prod_{i = 1}^{D} \cos (\frac{x_{i}}{\sqrt{i}})$ [-600, 600] ^D [0, 0, . . . , 0]

Zakharov $f_{7} (x) = \sum_{i = 1}^{D} x_{i}^{2} + (\sum_{i = 1}^{D} (i / 2) * x_{i}^{2})^{2} + (\sum_{i = 1}^{D} (i / 2) * x_{i}^{2})^{4}$ [-10, 10] ^D [0, 0, . . . , 0]

Sphere $f_{8} (x) = \sum_{i = 1}^{D} x_{i}^{2}$ [-100, 100] ^D [0, 0, . . . , 0]

Function	Definition	Search Range	Global minimum location x^*
Exponential	$f_{1} (x) = \exp (- 0.5 \sum_{i = 1}^{D} x_{i}^{2})$	[-1, 1] ^D	[0, 0, . . . , 0]
Powell Sum	$f_{2} (x) = \sum_{i = 1}^{D} \| x_{i} \|^{i + 1}$	[-1, 1] ^D	[0, 0, . . . , 0]
Sum of Squares	$f_{3} (x) = \sum_{i = 1}^{D} {ix}_{i}^{2}$	[-10, 10] ^D	[0, 0, . . . , 0]
Schwefel 2.20	$f_{4} (x) = \sum_{i = 1}^{D} \| x_{i} \|$	[-100, 100] ^D	[0, 0, . . . , 0]
Schwefel 2.22	$f_{5} (x) = \sum_{i = 1}^{D} \| x_{i} \| + \prod_{i = 1}^{D} \| x_{i} \|$	[-100, 100] ^D	[0, 0, . . . , 0]
Griewank	$f_{6} (x) = 1 + \sum_{i = 1}^{D} \frac{x_{i}^{2}}{4000} - \prod_{i = 1}^{D} \cos (\frac{x_{i}}{\sqrt{i}})$	[-600, 600] ^D	[0, 0, . . . , 0]
Zakharov	$f_{7} (x) = \sum_{i = 1}^{D} x_{i}^{2} + (\sum_{i = 1}^{D} (i / 2) * x_{i}^{2})^{2} + (\sum_{i = 1}^{D} (i / 2) * x_{i}^{2})^{4}$	[-10, 10] ^D	[0, 0, . . . , 0]
Sphere	$f_{8} (x) = \sum_{i = 1}^{D} x_{i}^{2}$	[-100, 100] ^D	[0, 0, . . . , 0]

Table 3

Multimodal benchmark functions

Function	Definition	Search Range	Global minimum location x^*
Non Continuous Rastrigin	$f_{9} (x) = 10 D + \sum_{i = 1}^{D} [y_{i}^{2} - 10 cos 2 π y_{i}]$ $y_{i} = {\begin{matrix} \| x_{i} \| & \| x_{i} \| \leq 0.5 \\ ⌈ 2 x - i + 0.5 ⌉ / 2 & \| x_{i} \| > 0.5 \end{matrix}$	[-10, 10] ^D	[0, 0, . . . , 0]
Ackley	$f_{10} (x) = - 20 \exp (- 0.2 \sqrt{\frac{1}{D} \sum_{i = 1}^{D} x_{i}^{2}}) - \exp (\frac{1}{D} \sum_{i = 1}^{D} \cos (2 π x_{i})) + 20$	[-100, 100] ^D	[0, 0, . . . , 0]
Rastrigin	$f_{11} (x) = 10 D + \sum_{i = 1}^{D} (x_{i}^{2} - 10 \cos (2 π x_{i}))$	[-100, 100] ^D	[0, 0, . . . , 0]
Rosenbrock	$f_{12} (x) = \sum_{i = 1}^{D}$	[-30, 30] ^D	[1, 1, . . . , 1]
Rotated Rastrigin	$f_{13} (x) = 10 D + \sum_{i = 1}^{D} (x_{i}^{2} - 10 \cos (2 π x_{i}))$ y = Mx	[-100, 100] ^D	[0, 0, . . . , 0]
Rotated Griewangk	$f_{14} (x) = 1 + \sum_{i = 1}^{D} \frac{x_{i}^{2}}{4000} - \prod_{i = 1}^{D} \cos (\frac{x_{i}}{\sqrt{i}})$ y = Mx	[-600, 600] ^D	[0, 0, . . . , 0]
Rotated Ackley	$f_{15} (x) = - 20 \exp (- 0.2 \sqrt{\frac{1}{D} \sum_{i = 1}^{D} x_{i}^{2}})$ $- \exp (\frac{1}{D} \sum_{i = 1}^{D} \cos (2 π x_{i})) + 20$ y = Mx	[-600, 600] ^D	[0, 0, . . . , 0]
Shifted Rotated Rastrigin	$f_{16} (x) = 10 D + \sum_{i = 1}^{D} (x_{i}^{2} - 10 \cos (2 π x_{i}))$ y = M (x - o)	[-5.12, 5.12] ^D	[o₁, o₂, . . . , o_D]
Shifted Rotated Ackley	$f_{17} (x) = - 20 \exp (- 0.2 \sqrt{\frac{1}{D} \sum_{i = 1}^{D} x_{i}^{2}})$ $- \exp (\frac{1}{D} \sum_{i = 1}^{D} \cos (2 π x_{i})) + 20$ y = M (x - o)	[-10, 10] ^D	[o₁, o₂, . . . , o_D]

Table 4

Results on 10-dimensions

Function	Cost of best solution
	PSO	GSO	PGSO
Exponential	–0.921306	–0.999999	0.0
Powell Sum	2.093603e-16	1.455717e-09	0.0
Sum of Squares	10.0	1.065358e-10	0.0
Schwefel 2.20	6.892269e-05	0.027480	0.0
Schwefel 2.22	202.255357	94.163522	0.0
Griewank	0.167509	0.084378	0.0
Zakharov	37.958089	7.010134	0.0
Sphere	1.374336e-89	6.73424e-13	0.0
Non Continuous Rastrigin	51.936414	34.654575	0.0
Ackley	7.565546	1.531584	0.0
Rastrigin	31.142114	27.062807	0.0
Rosenbrock	9641.613969	6.035564	0.198991
Rotated Rastrigin	72.929831	40.001291	0.099495
Rotated Griewank	44.357084	0.644639	0.0
Rotated Ackley	8.351079	2.115643	0.0
Shifted Rotated Rastrigin	84.450279	79.683572	0.099495
Shifted Rotated Ackley	5.088526	4.730487	0.198991

Table 5

Results on 30-dimensions

Function	Cost of best solution
	PSO	GSO	PGSO
Exponential	–0.728176	–0.924616	0.0
Powell Sum	4.648054e-06	5.611626e-05	0.0
Sum of Squares	1214.458714	337.720880	0.0
Schwefel 2.20	332.564225	263.270496	0.0
Schwefel 2.22	1141.990471	996.182188	0.0
Griewank	118.802937	2.692985	0.0
Zakharov	343.598332	314.160213	0.0
Sphere	15278.228377	326.863319	0.0
Non Continuous Rastrigin	15273.871741	1493.889869	0.0
Ackley	19.811653	19.044585	0.0
Rastrigin	10105.271870	1504.148681	0.0
Rosenbrock	23993629.675192	40405.022809	0.298487
Rotated Rastrigin	19767.202009	16871.833221	0.0
Rotated Griewank	462.059950	270.317716	0.0
Rotated Ackley	19.930969	19.458463	0.0
Shifted Rotated Rastrigin	518.206237	489.743710	0.0
Shifted Rotated Ackley	11.384165	10.363873	0.0

Table 6

Results on 50-dimensions

Function	Cost of Best Solution
	PSO	GSO	PGSO
Exponential	–0.293084	–0.491401	0.0
Powell Sum	7.924315e-05	0.000544	0.0
Sum of Squares	7828.761277	3119.232471	0.0
Schwefel 2.20	919.804781	793.400554	0.0
Schwefel 2.22	2130.738971	1885.202317	0.0
Griewank	327.295449	121.112666	0.198991
Zakharov	1144.403120	1140.669149	0.0
Sphere	32395.366641	15434.805716	0.0
Non Continuous Rastrigin	43956.727194	19474.525504	0.099495
Ackley	19.981123	19.906923	0.0
Rastrigin	39168.115819	15126.372899	0.0
Rosenbrock	44058513.785068	2030183.081471	0.198991
Rotated Rastrigin	95974.803194	48207.000964	0.0
Rotated Griewank	1507.186058	921.008503	0.198991
Rotated Ackley	19.981462	19.938538	0.0
Shifted Rotated Rastrigin	942.944276	988.692668	0.0
Shifted Rotated Ackley	12.793345	12.606802	0.0

Figures 3–10 depict the convergence characteristics of PSO, GSO, and PGSO over multiple benchmark functions. PGSO returns loss values below machine precision for most of the functions and does so in around 200 iterations. Its competitors get stuck at some minima and are not able to find a better position after a certain number of iterations. Also, to encourage fairness to record these results the authors had not executed the supercluster step which combines all outputs from the various explorers. Including that step would result in faster convergence in a shorter number of iterations.

7 Discussion and conclusion

In this paper a new parallel galactic swarm optimization algorithm is proposed and applied to the training of a neural network. Parallel implementation of the algorithm exhibited superior performance against state-of-the-art algorithms on all the benchmark optimization problems in terms of convergence and accuracy of the best solution. The proposed algorithm was also used to train a neural network to solve the problem of Churn Prediction and the problem of predicting whether an NBA player would last for the next 5 years. PGSO algorithm can be applied to Design, Finance, Combinatorial Optimization, Controllers, Clustering and Classification, Communication Networks, Faults, Images and Videos as mentioned by [9] on application domains of meta-heuristic algorithms. Future work will explore implementing the PGSO algorithm on a GPU to utilize the vertical scaling potential of the proposed algorithm’s architecture.

References

Abdel-Basset

, Hessin

A.-N.

, Abdel-Fatah

, A comprehensive study of cuckoo-inspired algorithms, Neural Computing and Applications29(2) (2018), 345–361. bibtex*[publisher=Springer].

Ahmad

, El-Shafie

, Razali

S.F.M.

, Mohamad

Z.S.

, Reservoir optimization in water resources: a review, Water Resources Management28(11) (2014), 3391–3405. bibtex*[publisher=Springer].

Amali

G.B.

, Bhuyan

, and Aju, Design of image enhancement filters using a novel parallel particle swarm optimisation algorithm, International Journal of Advanced Intelligence Paradigms9(5–6) (2017), 576–588. bibtex*[publisher=Inderscience Publishers (IEL)].

Bernal

, Castillo

, Soria

, Valdez

, Galactic Swarm Optimization with Adaptation of Parameters Using Fuzzy Logic for the Optimization of Mathematical Functions. In Fuzzy Logic Augmentation of Neural and Optimization Algorithms: Theoretical Aspects and Real Applications, pages 131–140. Springer, 2018.

Delisle

, Krajecki

, Gravel

, Gagné

, Parallel implementation of an ant colony optimization metaheuristic with openmp. In Proceedings of the 3rd EuropeanWorkshop on OpenMP (EWOMP’01), Barcelona, Spain, pages 1–7, 2001.

Eberhart

, Kennedy

, A new optimizer using particle swarm theory. In Micro Machine and Human Science, 1995. MHS’95., Proceedings of the Sixth International Symposium on, pages 39–43, 1995. bibtex*[organization=IEEE].

Eberhart

, Kennedy

, Particle swarm optimization. In Proceedings of the IEEE International Conference on Neural Networks4 (1995), 1942–1948. bibtex*[ organization=Citeseer].

Goldberg

D.E.

, Holland

J.H.

, Genetic algorithms and machine learning, Machine Learning3(2) (1988), 95–99. bibtex*[publisher=Springer].

Hussain

, Salleh

M.N.M.

, Cheng

, Shi

, Metaheuristic research: a comprehensive survey, Artificial Intelligence Review (2018), 1–43. bibtex*[publisher=Springer].

10.

Kawam

A.A.

, Mansour

, Metaheuristic optimization algorithms for training artificial neural networks, Int. J. Comput. Inf. Technol1(2) (2012)156–161.

11.

Marinakis

, Metaheuristic algorithms for the vehicle routing problem, Encyclopedia of Optimization (2009), 2055–2061. bibtex*[publisher=Springer].

12.

Molga

, Smutnicki

, Test functions for optimization needs, Test Functions for Optimization Needs101 (2005).

13.

Muthiah-Nakarajan

, Noel

M.M.

, Galactic swarm optimization: A new global optimization metaheuristic inspired by galactic motion, Applied Soft Computing38 (2016), 771–787. bibtex*[publisher=Elsevier].

14.

Nugroho

E.D.

, Wibowo

M.E.

, Pulungan

, Parallel implementation of genetic algorithm for searching optimal parameters of artificial neural networks. In 2017 3rd International Conference on Science and Technology-Computer (ICST), pages 136–141. IEEE, 2017.

15.

Qin

, Guo

, Liu

, Zhong

, Shape optimization of automotive body frame using an improved genetic algorithm optimizer, Advances in Engineering Software121 (2018), 235–249. bibtex*[publisher=Elsevier].

16.

Subotic

, Tuba

, Stanarevic

, Parallelization of the artificial bee colony (abc) algorithm. In Proceedings of the 11th WSEAS international conference on nural networks and 11th WSEAS international conference on evolutionary computing and 11th WSEAS international conference on Fuzzy systems, World Scientific and Engineering Academy and Society (WSEAS). Stevens Point, Wisconsin, USA, NN10 (2010), 191–196.