In this article, we present a new hybrid algorithm to enable and scale the high-performance conjugate gradients (HPCG) benchmark on large-scale heterogeneous systems such as the Tianhe-2. Based on an inner–outer subdomain partitioning strategy, the data distribution between host and device can be balanced adaptively. The overhead of data movement from both the MPI communication and the PCI-E transfer can be significantly reduced by carefully rearranging and fusing operations. A variety of parallelization and optimization techniques for performance-critical kernels are exploited and analyzed to maximize the performance gain on both host and device. We carry out experiments on both a small heterogeneous computer and the world’s largest one, the Tianhe-2. On the small system, a thorough comparison and analysis has been presented to select from different optimization choices. On Tianhe-2, the optimized implementation scales to the full-system level of 3.12 million heterogeneous cores, with an aggregated performance of 623 Tflop/s and a parallel efficiency of 81.2%.
CourtecuisseHAllardJ (2009) Parallel dense Gauss–Seidel algorithm on many-core processors. In: Proceeding 11th International Conference on High Performance Computing and Communications (HPCC’09), Seoul, Korea, 25–27 June 2009, pp. 139–147.
2.
DattaKMurphyMVolkovV. (2008) Stencil computation optimization and auto-tuning on state-of-the-art multicore architectures. In: Proceedings of ACM/IEEE Conference on Supercomputing (SC’08), Austin, TX, 15–21 November 2008, pp. 4:1–4:12. Piscataway, NJ: IEEE.
3.
DongarraJHerouxMA (2013) Toward a new metric for ranking high performance computing systems. Sandia Report SAND2013-4744. Albuquerque, NM: Sandia National Laboratories.
GraunkeGThakkarS (1990) Synchronization algorithms for shared-memory multiprocessors. Computer23(6): 60–69.
6.
IwashitaTShimasakiM (2003) Block red-black ordering: a new ordering strategy for parallelization of ICCG method. International Journal of Parallel Programming31(1): 55–75.
7.
KowarschikMRüdeUWeissC. (2000) Cache-aware multigrid methods for solving Poisson’s equation in two dimensions. Computing64(4): 381–399.
8.
KumahataKMinamiKMaruyamaN (2014) HPCG on the K computer. ASCR HPCG Workshop, Bethesda, MD, 25 March 2014.
9.
LiuXSmelyanskiyMChowE. (2013) Efficient sparse matrix-vector multiplication on x86-based many-core processors. In: Proceedings of 27th International Conference on Supercomputing (ICS’13), Eugene, OR, 10–14 June 2013, pp. 273–282. New York: ACM.
10.
LiuYZhangXYangC. (2014) Accelerating HPCG on Tianhe-2: A hybrid CPU-MIC algorithm. In: Proceedings of 24th IEEE International Conference on Parallel and Distributed System ICPADS’14, Hsinchu, Taiwan, 16–19 December 2014, pp. 542–551.
11.
Mellor-CrummeyJMScottML (1991) Algorithms for scalable synchronization on shared-memory multiprocessors. ACM Transactions on Computer Systems9(1): 21–65.
12.
MonakovALokhmotovAAvetisyanA (2010) Automatically tuning sparse matrix-vector multiplication for GPU architectures. In: PattYNFogliaPDuesterwaldEFaraboschiPMartorel (eds) High Performance Embedded Architectures and Compilers. Heidelberg: Springer, pp. 111–125.
13.
ParkJSmelyanskiyMSundaramN. (2014a) Sparsifying synchronization for high-performance shared-memory sparse triangular solver. In: Proceedings of 28th International Conference on Supercomputing (ICS’14), Leipzig, Germany, 22–26 June 2014, pp. 124–140. New York: Springer-Verlag.
14.
ParkJSmelyanskiyMVaidyanathanK. (2014b) Efficient shared-memory implementation of high-performance conjugate gradient benchmark and its application to unstructured matrices. In: Proceedings of International Conference for High Performance Computing, Networking, Storage and Analysis (SC’14), New Orleans, Louisiana, 16–21 November 2014, pp. 945–955.
15.
PhillipsEFaticaM (2014) A CUDA implementation of the High Performance Conjugate Gradient benchmark. In: Proceedings of the 5th International Workshop, PMBS 2014, New Orleans, LA, 16 November 2014, pp. 68–84.
16.
Sridevi (2014) Intel optimized technology preview for High Performance Conjugate Gradient benchmark. Technical report. Santa Clara: Intel.
17.
VuducRDemmelJWYelickKA (2005) OSKI: a library of automatically tuned sparse matrix kernels. Journal of Physics: Conference Series16(1): 521.
18.
WallinDLofHHagerstenE. (2006) Multigrid and Gauss–Seidel smoothers revisited: parallelization on chip multiprocessors. In: Proceedings of 20th International Conference on Supercomputing (ICS’06), Queensland, Australia, 28 June–1 July 2006, pp. 145–155. New York: ACM.
19.
WelleinGHagerGZeiserT. (2009) Efficient temporal blocking for stencil computations by multicore-aware wavefront parallelization. In: Proceedings of Computer Software and Applications Conference (COMPSAC’09), Vol 1, Seattle, WA, 20–24 July 2009, pp. 579–586.
20.
WilliamsSKalamkarDDSinghA. (2012) Optimization of geometric multigrid for emerging multi-and manycore processors. In: Proceedings of International Conference for High Performance Computing, Networking, Storage and Analysis (SC’12), Salt Lake City, Utah, 10–16 November 2012, pp. 96:1–96:11.
21.
WilliamsSOlikerLVuducR. (2009) Optimization of sparse matrix–vector multiplication on emerging multicore platforms. Parallel Computing35(3): 178–194.
22.
WonnacottD (2000) Using time skewing to eliminate idle time due to memory bandwidth and network limitations. In: Proceedings of 14th IEEE International Parallel and Distributed Processing Symposium (IPDPS’00), Cancun, Mexico, 1–5 May 2000, pp. 171–180.
23.
XueWYangCFuH. (2014) Enabling and scaling a global shallow-water atmospheric model on Tianhe-2. In: Proceedings of 28th IEEE International Parallel and Distributed Proceeding Symposium (IPDPS’14), Phoenix, AZ, 19–23 May 2014, pp. 745–754.
24.
ZhangXYangCLiuF. (2014) Optimizing and scaling HPCG on Tianhe-2: early experience. In: Proceedings of 14th International Conference on Algorithms and Architectures for Parallel Processing (ICA3PP’14), Dalian, China, 24–27 August 2014, pp. 28–41. Switzerland: Springer.
