Anatomy of machine learning algorithm implementations in MPI,Spark,and Flink

Abstract

With the ever-increasing need to analyze large amounts of data to get useful insights, it is essential to develop complex parallel machine learning algorithms that can scale with data and number of parallel processes. These algorithms need to run on large data sets as well as they need to be executed with minimal time in order to extract useful information in a time-constrained environment. Message passing interface (MPI) is a widely used model for developing such algorithms in high-performance computing paradigm, while Apache Spark and Apache Flink are emerging as big data platforms for large-scale parallel machine learning. Even though these big data frameworks are designed differently, they follow the data flow model for execution and user APIs. Data flow model offers fundamentally different capabilities than the MPI execution model, but the same type of parallelism can be used in applications developed in both models. This article presents three distinct machine learning algorithms implemented in MPI, Spark, and Flink and compares their performance and identifies strengths and weaknesses in each platform.

Keywords

Machine learning Big Data HPC MDS MPI Spark Flink K-means Terasort

Get full access to this article

View all access options for this article.

References

Abadi

Barham

Chen

. (2016) TensorFlow: a system for large-scale machine learning. In: ‘12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)’, pp. 265–283. Berkeley, CA: USENIX Association.

Akidau

Bradshaw

Chambers

. (2015) The dataflow model: a practical approach to balancing correctness, latency, and cost in massive scale, unbounded, out-of-order data processing. Proceedings of the VLDB Endowment 8(12): 1792–1803.

Apache Giraph (n.d.) Available at: http://giraph.apache.org/ (accessed 14 December 2016).

Bae

Qiu

Fox

(2010) Multidimensional scaling by deterministic annealing with iterative majorization algorithm. In: ’2010 IEEE sixth international conference on e-Science’, pp. 222–229. Brisbane, Australia: IEEE.

Carbone

Katsifodimos

Ewen

. (2015) Apache flink: stream and batch processing in a single engine, Bulletin of the IEEE Computer Society Technical Committee on Data Engineering 36(4).

Doekemeijer

Varbanescu

(2014) A survey of parallel graph processing frameworks, Delft, Netherlands: Delft University of Technology. Available at: http://www.ds.ewi.tudelft.nl/fileadmin/pds/reports/2014/PDS-2014-003.pdf (accessed January 2016).

Ekanayake

Zhang

. (2010) Twister: a runtime for iterative map reduce. In: Proceedings of the 19th ACM international symposium on high performance distributed computing’, HPDC’ 10, pp. 810–818. New York, NY, USA: ACM. Available at: http://doi.acm.org/10.1145/1851476.1851593

Ekanayake

Kamburugamuve

Fox

(2016) Spidal java: high performance data analytics with java and mpi on large multicore hpc clusters. In: Proceedings of the 24th high performance computing symposium’, HPC ’16, San Diego, CA, USA, pp. 3:1–3:8. Available at: https://dx-doi-org.web.bisu.edu.cn/10.22360/SpringSim.2016.HPC.031

Ekanayake

Kamburugamuve

Wickramasinghe

. (2016) Java thread and process performance for parallel machine learning on multicore hpc clusters. In: 2016 IEEE international conference on big data (Big Data), Washington, DC, USA: IEEE, pp. 347–354.

10.

Fox

Qiu

Jha

. (2016) Big data, simulations and HPC convergence. In: Rabl

Nambiar

Baru

. (eds.) Big data benchmarking: 6th international workshop, WBDB 2015, Toronto, ON, Canada, 16–17 June 2015 and 7th international workshop, WBDB 2015, New Delhi, India, 14–15 December 2015, Revised Selected Papers’, pp. 3–17. Cham: Springer International Publishing. Available at: https://dx-doi-org.web.bisu.edu.cn/10.1007/978-3-319-49748-81

11.

Fox

Qiu

Kamburugamuve

. (2015) Hpc-abds high performance computing enhanced apache big data stack. In: 2015 15th IEEE/ACM international cluster, cloud and grid computing (CCGrid).

12.

Gonzalez

Xin

Dave

. (2014) Graphx: graph processing in a distributed dataflow framework. In: 11th USENIX symposium on operating systems design and implementation (OSDI 14), Berkeley, CA, USA: USENIX Association, pp. 599–613.

13.

H2O (2017) Available at: http://www.h2o.ai/h2o/ (accessed 02 January 2017).

14.

Intel Data Analytics Acceleration Library (n.d.) Available at: https://software.intel.com/en-us/intel-daal (accessed 02 January 2017).

15.

Kruskal

(1964) Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika 29(1): 1–27.

16.

Liang

Feng

. (2014) Performance benefits of datampi: a case study with bigdatabench. In: Zhan

Han

Weng

(eds.) Big data benchmarks, performance optimization, and emerging hardware: 4th and 5th workshops, BPOE 2014, Salt Lake City, USA, 1 March 2014 and Hangzhou, China, 5 September 2014, Revised Selected Papers’, Cham: Springer International Publishing, pp. 111–123. Available at: https://dx-doi-org.web.bisu.edu.cn/10.1007/978-3-319-13021-79

17.

Islam

Wasi-Ur-Rahman

. (2013) High-performance design of hadoop rpc with rdma over infiniband. In: 2013 42nd international conference on parallel processing’, New York, NY, USA: IEEE, pp. 641–650.

18.

Liang

Wang

. (2014) DataMPI: extending MPI to hadoop-like big data computing. In: 2014 IEEE 28th international parallel and distributed processing symposium, New York, USA: IEEE, pp. 829–838.

19.

Malewicz

Austern

Bik

. (2010) Pregel: a system for large-scale graph processing. In: Proceedings of the 2010 ACM SIGMOD international conference on management of data’, SIGMOD’10, New York, NY, USA: ACM, pp. 135–146. Available at: http://doi.acm.org/10.1145/1807167.1807184

20.

Marcu

O-C

Costan

Antoniu

. (2016) Spark versus flink: understanding performance in big data analytics frameworks. In: 2016 IEEE international conference on cluster computing (CLUSTER), New York, USA: IEEE, pp. 433–442.

21.

Meng

Bradley

Yuvaz

. (2016) ‘Mllib: machine learning in apache spark. Journal of Machine Learning Research 17(34): 1–7.

22.

Owen

Anil

Dunning

. (2011) Mahout in action, 1st edn. Shelter Island, NY, USA: Manning.

23.

Project Tungsten: Bringing Apache Spark Closer to Bare Metal (2015) https://databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html

24.

Reyes-Ortiz

Oneto

Anguita

(2015) Big data analytics in the cloud: Spark on Hadoop vs MPI/OpenMP on Beowulf. Procedia Computer Science 53: 121–130.

25.

Ruan

Fox

(2013) A robust and scalable solution for interpolative multidimensional scaling with weighting. In: Proceedings of the 2013 IEEE 9th international conference on e-Science’, ESCIENCE ’13, Washington, DC, USA: IEEE Computer Society, pp. 61–69. Available at: https://dx-doi-org.web.bisu.edu.cn/10.1109/eScience.2013.30

26.

Saha

Shah

Seth

. (2015) Apache tez: a unifying framework for modeling and building data processing applications, In: Proceedings of the 2015 ACM SIGMOD international conference on management of data’, SIGMOD’15, New York, NY, USA: ACM, pp. 1357–1369. Available at: http://doi.acm.org/10.1145/2723372.2742790

27.

Seo

Yoon

Kim

. (2010) Hama: an efficient matrix computation with the mapreduce framework. In: 2010 IEEE second international conference on cloud computing technology and science (CloudCom)’, New York, NY, USA: IEEE, pp. 721–726.

28.

Wang

Zhan

Luo

. (2014) BigDataBench: a big data benchmark suite from internet services. In: 2014 IEEE 20th international symposium on high performance computer architecture (HPCA)’, New York, USA: IEEE, pp. 488–499.

29.

White

(2009) Hadoop: the definitive guide, 1st edn. Sebastopol, CA, USA: O’Reilly Media, Inc.

30.

Zaharia

Chowdhury

Das

. (2012) Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of the 9th USENIX conference on networked systems design and implementation’, NSDI’12, Berkeley, CA, USA: USENIX Association, pp. 2–2. Available at: http://dl.acm.org/citation.cfm?id=2228298.2228301

31.

Zaharia

Chowdhury

Franklin

. (2010) Spark: cluster computing with working sets. In: Proceedings of the 2Nd USENIX conference on hot topics in cloud computing, HotCloud’10, USENIX Association, Berkeley, CA, USA, pp. 10–10. Available at: http://dl.acm.org/citation.cfm?id=1863103.1863113.