Ant colony inspired method for reducing load imbalance in multiprocessor systems

Abstract

Parallel processing is crucial for accelerating computation in many high-performance applications and modern technologies including computational modeling, optimization and simulation, Web and DNS servers, peer-to-peer systems, grid computing and cloud computing. Due to the heterogeneity nature of various processing nodes and the differences of workloads of various tasks, some processors can be idle while others are overloaded. In this paper, we present a simple, yet efficient, solution inspired by the intelligence of ant colonies to adequately mitigate the load imbalance and communication overhead problems in multiprocessor environments. The proposed approach is based on defining and maintaining data structures to dynamically track the load of each processor. We implemented the proposed algorithm and evaluated its performance under different scenarios against the baseline round-robin algorithm. The results showed that the proposed algorithm has more effective properties than the round-robin algorithm.

Keywords

Parallel processing cloud computing grid computing load balancing artificial ant colony optimization

1 Introduction

In early stages of computing, computers were executing instructions sequentially causing a huge delay for computationally- and data-intensive programs. In order to overcome this problem, many models have been proposed in the past for parallel computing and multiprogramming systems, symmetric multiprocessing and massively parallel processing [1]. The concept of parallel processing is based on processing program instructions by dividing them over multiple processors aiming to execute the program in less time with high throughput and better utilization of resources. Parallel processing has become an essential paradigm for many real-world applications such computer graphics and visualization, image and video processing, computational modeling, optimization and simulation, weather forecasting, environmental modeling, material and geophysics sciences, etc. However, several challenges are associated with such systems [2, 3]:

Algorithmic overhead: presented when programmers are required to spend more efforts in parallelizing sequential programs by considering and adding extra parts in their code such as the related parallel prefix.

Communication overhead: presented when the time spent in communication between nodes proportionally increases.

Critical paths problem: presented when dependencies exist between operations spread over multiple processors.

Bottlenecks: presented when one processor is holding a required resource while the others are idle waiting for this processor to release this resource.

Memory performance: presented when the time taken to access data from memory is much higher than the time required for execution.

Load imbalance: presented when the load is inefficiently distributed over multiple nodes. This issue becomes extremely pivotal in system performance as nodes heterogeneity increases.

The problem considered here is the load imbalance in multiprocessor environments, where the load is distributed unequally between the processing units. In such scenarios, the result is that one or more processors are having high load while others are having much less. This inefficient load distribution between processors leads to inefficient utilization of the available processing resources. Moreover, it increases the overall time required for the execution of tasks. In this paper, we briefly review the state-of-the-art techniques and present a novel approximate approach based on the collective behavior of ants to overcome the load imbalance problem while eliminating the communication overhead.

The rest of the paper is organized as follows. Section 2 describes the problem under the study. Section 3 reviews related work. Section 4 describes the proposed methodology. Section 5 evaluates and compares the effectiveness of the proposed method under various scenarios. Section 6 concludes the paper and highlights future work.

2 Problem formulation

The problem considered in this study is to efficiently distribute N tasks over P processors. Each task has a workload on each processor which may differ from the other tasks. Tasks arrive over time and are allocated to processors by the load balancer. To achieve the highest efficiency in load balancing between the processors, all processors should have almost the same load. The highest possible efficiency is when the load difference between processors equals zero, which means all processors have the same load as presented by the below formulas.

We use the notation I_i (P_j) =1 to mean the i-th task is allocated to j-th processor; otherwise I_i (P_j) =0. We also use L_i (P_j) to refer to the workload of the i-th task on the j-th processor. For a specific distribution of tasks, the total load on the j-th processor is given by: $L (P_{j}) = \sum_{i = 1}^{N} L_{i} (P_{j}) . I_{i} (P_{j})$ (1)

For the ideal distribution, each processor must have an equal share of the total workload which is given by: $μ = \frac{1}{P} \sum_{i, j} L_{i} (P_{j}) I_{i} (P_{j})$ (2)

The imbalance factor α (P_j) refers to the deviation of the load on the j-th processor from the perfect average load μ and is given by: $α (P_{j}) = | L (P_{j}) - μ |; j = 1, . . ., P$ (3)

It is required to distribute arrived tasks over existing processors efficiently in order to minimize the maximum imbalance factor between processors while reducing the communication overhead.

3 Related work

Load balancing refers to spreading the computation load evenly across the processors while minimizing the required communication among them. The proper load balancing technique for regular or irregular algorithms relies on certain properties concerned with the behavior of data and tasks, e.g. [4]: (a) Task-oriented or data-oriented algorithmic structure, (b) Regular or irregular data input, and (c) Static or dynamic data nature. Another very important issue is the communication pattern resulting due to the distribution of data and tasks. The communication behavior depends on the characteristics of the algorithm and the implementation strategies. The locality of data dependencies, the locality of data patterns and the locality of communication patterns are all very important factors as well. Moreover, issues related to load balancing should be considered such data redistribution or tasks migration between processors [5].

A recent literature survey is presented in [6]. The authors classified load balancing techniques into seven categories: Hadoop MapReduce, natural phenomena-based, agent-based, general load balancing, application-oriented, network-aware, and workflow-specific.

Several algorithms or models based on swarm systems have been proposed and applied in real life [7]: (a) Ant Colony Optimization (ACO), (b) Particle Swarm Optimization (PSO), (c) Glowworm Swarm Optimization, and (d) Artificial Bee Colony (ABC). ACO was inspired from the behavior of biological ants [8]. Mainly these ants deposit a chemical substance, known as ‘pheromone’, on the ground while searching for food, in order to mark the best paths that should be followed by other members of the colony. Pheromone also evaporates as time passes. The net result is that more pheromone is accumulated on shorter paths. ACO uses a similar mechanism in order to solve optimization problems such as the traveling salesman problem [9]. An ant which is currently in the i-th city has to choose the next city to visit via a stochastic mechanism similar to the behavior of natural ants.

Willebeek-LeMair and Reeves [10] presented five strategies for dynamic load balancing on highly parallel computers. These methods are built on the tradeoff between the accuracy of balancing decisions and the communication and processing overheads. In [11], Sinclair proposed another dynamic algorithm which provides quick and accurate assignment for a number of interacting tasks over multiple processing nodes based on Global Scheduling Table (GST). This algorithm combines the information contained in the program graph and the network graph of the system with the current state of the system for the assignment of tasks. Ludwig and Moallem [12] compared multiple swarm intelligence approaches for load balancing in grid computing.

Recently the concept of cloud computing has emerged based on the idea of distributed and grid computing. Many research studies have contributed to realizing this powerful technology as a scalable service over the Internet. One of the core tasks in cloud computing is load balancing to avoid having some nodes overloaded while others are idle. In [3], the authors presented a systematic comprehensive review of existing challenges, techniques, and implementations of load balancing in cloud environments. In [14], a task scheduling policy is proposed for cloud computing based on ant colony optimization. Cloudsim is used for setting up the testing environment and the number of virtual machines in each datacenter is set to 50. In [15], another method is introduced based on ant colony optimization to overcome the load imbalance in cloud computing. The introduced algorithm is based on a distributed pheromone level update mechanism for load balancing. In [16], the advantages and disadvantages of multiple load balancing schemes for different cloud environment are discussed. The paper concludes that static load balancing schemes are easier in terms of simulation and monitoring of the environment. However, static methods cannot model the heterogeneous nature of cloud computing. On the other hand, dynamic load balancing schemes are difficult to simulate, but they are considered as the best solutions for the heterogeneous environment of cloud computing.

In [17], the authors discussed load balancing problems on grid systems and proposed a solution based on ant colony optimization. The experimental results show that while implementing the proposed algorithm to the grid environment, increasing the number of jobs and their length has an insignificant impact on the system response time. Another related work has proposed a grid balancing algorithm using ant colony optimization [18]. This algorithm takes into consideration the available resources, the critical path and the processing speed of the resources.

Although there have been a number of related research studies, the majority of the existing solutions are either increasing the time spent for communication between nodes or do not achieve load balance between nodes in high efficiency. In this paper, the proposed method is focusing on mitigating the load imbalance while eliminating the time spent in communication between the processing nodes for the purpose of getting the current load status on each processor.

4 Proposed methodology

There are multiple processors (P) and multiple tasks (N), which may be different in terms of nature and behavior. The proposed method is based on the behavior of ants. For each task, there is one ant. Each processor is represented by an ant colony, thus the total number of ant colonies equals the total number of processors. We use pheromone to represent the estimated time required for execution of a task. The proposed method assumes that ants are directed towards the highest pheromone level, which refers to the lowest required execution time. The proposed load balancing is based on defining two data structures (e.g. tables):

Ants Table: This table represents the tasks and includes for each task its assigned ant colony and estimated execution time (aka task’s pheromone level). An example is shown in Fig. 1(a).

Ants Colony Table: This table represents the processors and includes the scheduled tasks for each processor as well as the total pheromone of assigned tasks. For each ant colony, the assigned ants (tasks) are stored as a linked list of Tasks IDs. An example is shown in Fig. 1(b).

Fig.1

Illustrative example of Ants and Ant Colonies Tables.

Moreover, the proposed method defines two procedures: (a) Add task: triggered when a task arrives (pseudocode is shown in Algorithm 1), and (b) Remove task: triggered when a task is completed (pseudocode is shown in Algorithm 2).

Algorithm 1 Add a newly arrived task

Require: Task ID, Task workload, Ants Table, Ant Colonies Table

1: Estimate the pheromone level for this task (i.e. required estimation time on each processor)

2: Select the ant colony with the minimum pheromone

3: Add an ant to the Ants Table with its assigned Ant Colony and Pheromone level

4: Update the Ant Colonies Table by adding the new Ant to the list

5: Update the Pheromone level of this specific colony by adding the estimated pheromone of that Ant to its total pheromone level

Algorithm 2 Remove a completed task

Require: Task ID, Ants Table, Ant Colonies Table

1: Access the Ants Table and get the associated Pheromone level and the associated Ant Colony

2: Access the Ant Colonies Table and remove the task

3: Update the Pheromone level of this specific colony by deducting the Pheromone Level of the departed ant

4: Perform shift left for the tasks assigned to the associated Ant Colony

5: Remove the Ant’s record from the Ants Table

5 Evaluation and results

In this section, we evaluate the performance of the proposed algorithm taking into consideration two critical factors: (a) number of tasks, and (b) size of each task.

For the evaluation purpose, we use the following criteria: (a) maximum imbalance α_max, (b) average imbalance α_avg, (c) maximum percentage imbalance α_max (%), and (d) average percentage imbalance α_avg (%): $α_{\max} = \max_{j} {α (P_{j})}, α_{\max} (%) = 100 \times α_{\max} / μ$ (4) $α_{avg} = \frac{1}{P} \sum_{j = 1}^{P} α (P_{j}), α_{avg} (%) = 100 \times α_{avg} / μ$ (5) where μ is defined in Equation 2.

5.1 Testing the algorithm with unified-size tasks

The initial assumption was having multiple same size tasks or programs that need to be run in parallel, e.g. SPMD. To study the behavior and the accuracy of the proposed algorithm, the algorithm was tested on several scenarios. Initially, the size of tasks was maintained but the number of tasks at each time was changed. Then, the number of tasks was maintained but the size of tasks was changed.

5.1.1 Effect of number of tasks

Here, the size of tasks was maintained fixed at 10, and three scenarios were tested when the number of tasks was 10, 20, and 30. The results of maximum and average percentage imbalance are shown in Table 1. It is apparent from these results that imbalance could be predicted if the size of tasks is fixed by the following rule:

Table 1
Maximum and average percentage imbalance of the proposed algorithm for 10, 20 and 30 unified-size tasks

P 10 Tasks 20 Tasks 30 Tasks

Max Avg Max Avg Max Avg

1 0 0 0 0 0 0

2 0 0 0 0 0 0

3 20 13.333 10 6.667 0 0

4 20 20 0 0 6.667 6.667

5 0 0 0 0 0 0

6 40 26.667 20 13.333 0 0

7 40 34.286 30 8.571 16.667 9.524

8 60 30 20 20 20 10

P	10 Tasks	20 Tasks	30 Tasks
1	0	0	0	0	0	0
2	0	0	0	0	0	0
3	20	13.333	10	6.667	0	0
4	20	20	0	0	6.667	6.667
5	0	0	0	0	0	0
6	40	26.667	20	13.333	0	0
7	40	34.286	30	8.571	16.667	9.524
8	60	30	20	20	20	10

Ifmod (N, P) =0 then

α (P_j) ←0

else

$α_{\max} \leftarrow \max (| \max_{i, j} {T_{i} (P_{j})} - μ |, | μ - \min_{i, j} {T_{i} (P_{j})} |)$

end if

Obviously, both the number of tasks, N, and number of processors, P, directly affect the accuracy of the proposed algorithm.

5.1.2 Effect of tasks sizes

Here, three sizes of tasks were considered (10, 20 and 30) but the number of tasks was maintained in each test to be 100. The algorithm was run multiple times and at each time the number of processors was changed. From the results of the three test cases, it is obvious that the imbalance factor increases as the sizes of the tasks increase, as shown in Fig. 2. However, the percentage imbalance remains the same if the number of processors and the number of tasks remain the same, as shown in Fig. 3.

Fig.2

Imbalance factor when the number of tasks is kept fixed and sizes of tasks are changed.

Fig.3

Percentage imbalance when the number of tasks is kept fixed and sizes of tasks are changed.

5.2 Random-size tasks

Additionally, we considered the case of having multiple tasks of random sizes that need to run in parallel, e.g. MPMD. To study the behavior and the accuracy of the algorithm, the algorithm was run for three scenarios; each having 100 tasks with randomly generated sizes. First, the performance was evaluated for unordered tasks. Then, the effect of changing the sequence of arrival was examined. Therefore, the tasks were sorted according to their sizes from the smallest to the largest (ascending order) and from the largest task to the smallest (descending order). The same tasks that were used in the previous part were used again here, but the order of task assignment to processors was sorted. The results were illustrated in the Figs. 4 to 6. Here, it is obvious that having tasks ascendingly sorted (with random arrival) does not affect the accuracy of the algorithm that much. However, the significant change was observed when the tasks were descendingly sorted. This fits well with the theory of ‘The Big Rocks of Life’ by Stephen Covey [19]. When the assignment of tasks starts with the largest task, the degree of imbalance was reduced in all three test cases.

Fig.4

Performance for the first test case (unordered tasks).

Fig.5

Performance for the second test case (ascendingly-ordered tasks).

Fig.6

Performance for the third test case (descendingly-ordered tasks).

5.3 Discussion and performance comparison

The proposed algorithm could be considered as hybrid due to its both static and dynamic approaches. The dynamic approach is observed in the dynamic process of updating the tables (Ant Colonies table and Ants table). The main purpose of maintaining these tables updated is to eliminate the requirement for each node to communicate with the other nodes to get the total pheromone level on the other nodes. The only time required to be spent here by each node is the time to access the Pheromone Table and the time required to update the tables. Therefore, it is possible to say that the time spent in communication between nodes to assign a new task is reduced. The static approach of the algorithm is observed as there is no reassignment of tasks once assigned to the destination node. The advantage here is that the time spent in re-balancing the load is omitted. However, the task size is an estimated number and tasks with big sizes may finish execution before smaller sizes tasks due multiple reasons (e.g. file does not exist). Therefore, due to the hybrid nature of the algorithm, it is obvious that the algorithm may help to reduce the overall time spent in communication between nodes. Comparing the results generated by the algorithm with the findings in [12], the average load balancing while using ant colony is 65± 0.7 % and it comes in the fourth place in terms of accuracy after PSO, State Broadcast Algorithm and Random Space Shared. However, from the results provided here, it is obvious that the accuracy level of the proposed algorithm is much more accurate than the results presented in the paper due to the consideration of multiple directly affecting variables. These variables are: (a) Programming model (SPMD or MPMD), (b) Task size, (c) Sequence of the arrival of tasks, and (d) Number of processing nodes. Considering these factors leads to realizing that it is impossible to stick to a static average imbalance or average load balancing level, as the imbalance value and the way to calculate the imbalance vary if one or more of the variables above was changed. Moreover, the results which were presented previously and generated based on the proposed algorithm are more accurate than the results presented in the paper which are based on the algorithms used by the researchers. On the other hand, there are also multiple variables ignored which may affect the accuracy such as Computer Architecture, Technology, etc. The main reason to ignore these factors is to maintain the generalization of the algorithm rather than the dependency on one technology.

In addition to that, the results of the proposed algorithm were compared with the results of the baseline Round Robin approach. For this comparison, both approaches were tested under similar conditions and using the same data. For the first set of comparisons, the same data sets used previously without ordering were used. The results are shown in Table 2. From these results, it is clear that the the proposed algorithm is more accurate compared to the Round Robin approach.

Table 2
Comparing the Proposed algorithm with Round Robin in terms of percentage maximum and average errors – for unordered tasks

P Proposed Round Robin

First Second Third First Second Third

Max Avg Max Avg Max Avg Max Avg Max Avg Max Avg

1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

2 1.85 1.85 0.78 0.78 0.54 0.54 2.92 2.92 4.65 4.65 7.30 7.30

3 1.97 1.32 1.33 0.89 2.65 1.77 4.82 3.21 8.51 5.67 19.44 12.96

4 4.51 2.26 1.42 0.71 3.45 1.82 12.59 7.41 8.88 4.65 17.85 9.21

5 4.45 1.97 2.29 1.46 3.98 1.63 21.30 13.01 12.36 6.93 7.34 3.19

6 6.30 3.26 2.14 1.16 6.28 2.21 11.60 6.61 19.92 7.17 27.45 15.44

7 7.16 4.11 4.03 2.11 7.31 3.06 24.29 16.18 11.28 5.49 17.67 7.31

8 9.15 4.54 3.80 2.41 9.14 3.99 36.25 13.65 15.94 7.22 37.61 14.37

P	Proposed	Round Robin
1	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
2	1.85	1.85	0.78	0.78	0.54	0.54	2.92	2.92	4.65	4.65	7.30	7.30
3	1.97	1.32	1.33	0.89	2.65	1.77	4.82	3.21	8.51	5.67	19.44	12.96
4	4.51	2.26	1.42	0.71	3.45	1.82	12.59	7.41	8.88	4.65	17.85	9.21
5	4.45	1.97	2.29	1.46	3.98	1.63	21.30	13.01	12.36	6.93	7.34	3.19
6	6.30	3.26	2.14	1.16	6.28	2.21	11.60	6.61	19.92	7.17	27.45	15.44
7	7.16	4.11	4.03	2.11	7.31	3.06	24.29	16.18	11.28	5.49	17.67	7.31
8	9.15	4.54	3.80	2.41	9.14	3.99	36.25	13.65	15.94	7.22	37.61	14.37

In the second set of comparisons, the same ascendingly-sorted data sets used previously were used again. The results in Table 3 show that both algorithms have very close performance. Finally, Table 4 shows the results for the descendingly-sorted data sets. From these results, it is obvious that the proposed algorithm is more accurate than the baseline Round Robin approach.

Table 3

Comparing the Proposed algorithm with Round Robin in terms of percentage maximum and average imbalances – ascendingly-sorted tasks

P	Proposed						Round Robin
	First		Second		Third		First		Second		Third
	Max	Avg	Max	Avg	Max	Avg	Max	Avg	Max	Avg	Max	Avg
1	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
2	1.05	1.05	0.99	0.99	0.96	0.96	1.05	1.05	0.99	0.99	0.96	0.96
3	2.36	1.57	1.82	1.21	1.85	1.23	2.36	1.57	1.82	1.21	1.85	1.23
4	3.09	1.98	3.12	2.09	3.34	2.20	3.09	1.98	3.12	2.09	3.34	2.20
5	4.46	2.68	4.04	2.48	4.53	2.59	4.46	2.68	4.04	2.48	4.53	2.59
6	5.61	3.30	5.01	3.03	4.71	2.80	5.61	3.30	5.01	3.03	4.71	2.80
7	6.70	3.97	5.99	3.17	7.14	3.81	6.70	3.97	5.99	3.17	7.14	3.81
8	7.57	4.55	6.98	3.68	8.15	4.48	7.57	4.55	6.98	3.68	8.15	4.48

Table 4

Comparing the Proposed algorithm with Round Robin in terms of percentage maximum and average imbalances – descendingly-sorted tasks

P	Proposed						Round Robin
	First		Second		Third		First		Second		Third
	Max	Avg	Max	Avg	Max	Avg	Max	Avg	Max	Avg	Max	Avg
1	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
2	0.01	0.01	0.00	0.00	0.02	0.02	1.05	1.05	0.99	0.99	0.96	0.96
3	0.03	0.02	0.03	0.02	0.05	0.04	2.36	1.57	1.82	1.21	1.85	1.23
4	0.08	0.06	0.05	0.02	0.02	0.01	3.09	1.98	3.12	2.09	3.34	2.20
5	0.11	0.08	0.11	0.07	0.10	0.04	4.46	2.68	4.04	2.48	4.53	2.59
6	0.24	0.13	0.21	0.08	0.20	0.09	5.61	3.30	5.01	3.03	4.71	2.80
7	0.15	0.08	0.20	0.12	0.19	0.10	6.70	3.97	5.99	3.17	7.14	3.81
8	0.22	0.13	0.64	0.22	0.25	0.15	7.57	4.55	6.98	3.68	8.15	4.48

6 Conclusions and future work

A simple approach based on the behavior of ant colonies was suggested for solving the load imbalance problem in parallel and distributed processing while reducing the communication time between nodes. Using global tables, newly arrived tasks are assigned to the processing node with the least load. The effectiveness of the presented approach is evaluated under different workload scenarios. The results of its implementation show that the suggested algorithm has improved performance compared to the baseline round-robin load balancing algorithm. Moreover, the maximum imbalance could be predicted for the programming model used. As future work, we recommend taking into consideration the task priority for the revision of tables during the execution of tasks, which can further improve the efficiency through dynamic load redistribution. Therefore, we recommend more research on this issue. We also recommend exploration and comparison with other swarm intelligence algorithms.

Footnotes

Acknowledgments

The authors would like to thank Ahlia University (Bahrain) and King Fahd University of Petroleum and Minerals (Saudi Arabia) for support during this work.

References

Grama

, Gupts

, Karypis

and Kumar

, Introduction to parallel computing. Addison Wesley, 2nd Edition; 2003.

Kirk

D.B.

and Hwu

W.-M.W.

, Programming massively parallel processors: a hands-on approach. Morgan Kaufmann; 2016.

Eddy

W.M.

and Allman

, Advantages of parallel processing and the effects of communications time; 2000.

Rünger

, Parallel programming models for irregular algorithms. In Parallel Algorithms and Cluster Computing: Implementations, Algorithms and Applications, pp. 3–23. Springer, 2006.

Hoffmann

K.H.

and Meyer

, Parallel Algorithms and Cluster Computing: Implementations, Algorithms and Applications. Springer; 2006.

Ghomi

E.J.

, Rahmani

A.M.

and Qader

N.N.

, Load-balancing algorithms in cloud computing: A survey, Journal of Network and Computer Applications88(C) (2017), 50–71.

Ahmed

and Glasgow

, Swarm intelligence: concepts, models and applications. Technical report, School of Computing, Queens University; 2012.

Blum

, Ant colony optimization: Introduction and recent trends, Physics of Life reviews2(4) (2005), 353–373.

Dorigo

, Birattari

and Stutzle

, Ant colony optimization, IEEE computational intelligence magazine1(4) (2006), 28–39.

10.

Willebeek-LeMair

M.H.

and Reeves

A.P.

, Strategies for dynamic load balancing on highly parallel computers, IEEE Transactions on Parallel and Distributed Systems4(9) (1993), 979–993.

11.

Sinclair

, The GST load balancing algorithm for parallel and distributed systems, International Journal of Approximate Reasoning19(1-2) (1998), 39–56.

12.

Ludwig

S.A.

and Moallem

, Swarm intelligence approaches for grid load balancing, Journal of Grid Computing9(3) (2011), 279–301.

13.

Milani

A.S.

and Navimipour

N.J.

, Load balancing mechanisms and techniques in the cloud environments: systematic literature review and future trends, Journal of Network and Computer Applications71 (2016), 86–98.

14.

Tawfeek

M.A.

, El-Sisi

, Keshk

and Torkey

F.A.

, Cloud task scheduling based on ant colony optimization, International Arab Journal of Information Technology12(2) (2015), 129–137.

15.

Mishra

and Jaiswal

, Ant colony optimization: A solution of load balancing in cloud, International Journal of Web & Semantic Technology3(2) (2012), 33.

16.

Katyal

and Mishra

, A comparative study of load balancing algorithms in cloud computing environment, arXiv preprint arXiv:1403.6918, 2014.

17.

Karimpour

, Khayyambashi

M.R.

and Movahhedinia

, Applying ant colony optimization for load balancing on grid, Journal of the Chinese Institute of Engineers39(1) (2016), 49–56.

18.

Nadimi-Shahraki

, Fard

E.S.

and Safi

, Efficient load balancing using ant colony optimization, Journal of Theoretical and Applied Information Technology77(2) (2015), 253–258.

19.

Covey

S.R.

, Merill

A.R.

and Merill

R.R.

, First things first; 2015.