An efficient MPI/OpenMP parallelization of the Hartree–Fock–Roothaan method for the first generation of Intel® Xeon Phi™ processor architecture

Abstract

The Hartree–Fock method in the General Atomic and Molecular Structure System (GAMESS) quantum chemistry package represents one of the most irregular algorithms in computation today. Major steps in the calculation are the irregular computation of electron repulsion integrals and the building of the Fock matrix. These are the central components of the main self consistent field (SCF) loop, the key hot spot in electronic structure codes. By threading the Message Passing Interface (MPI) ranks in the official release of the GAMESS code, we not only speed up the main SCF loop (4× to 6× for large systems) but also achieve a significant ( $> 2$ ×) reduction in the overall memory footprint. These improvements are a direct consequence of memory access optimizations within the MPI ranks. We benchmark our implementation against the official release of the GAMESS code on the Intel® Xeon Phi™ supercomputer. Scaling numbers are reported on up to 7680 cores on Intel Xeon Phi coprocessors.

Keywords

Parallel Hartree–Fock–Roothaan OpenMP MPI quantum chemistry GAMESS Intel Xeon Phi irregular computation integral computation

Get full access to this article

View all access options for this article.

References

ALCF (2017) Argonne National Laboratory Leadership Computing Facility. Available at: http://www.alcf.anl.gov/ (accessed 11 April 2016).

Alexeev

Kendall

Gordon

(2002) The distributed data SCF. Computer Physics Communications 143(1): 69–82.

Alexeev

Mickelson

Leyffer

. (2014) The heuristic static load-balancing algorithm applied to the community earth system model. In: 2014 IEEE international parallel & distributed processing symposium workshops (IPDPSW), Salt Lake City, UT, USA, 10–16 November 2012, pp. 1581–1590. Washington, DC, USA: IEEE Computer Society.

Alexeev

Mickelson

Leyffer

. (2014) The heuristic static load-balancing algorithm applied to the community earth system model. In: 2014 IEEE international symposium on parallel and distributed processing symposium workshops (IPDPSW), Phoenix, AZ, USA, 19–23 May 2014, pp. 1581–1590. Washington, DC, USA: IEEE Computer Society.

Alexeev

Schmidt

Windus

. (2007) A parallel distributed data CPHF algorithm for analytic Hessians. Journal of Computational Chemistry 28(10): 1685–1694.

Asadchev

Gordon

(2012) New multithreaded hybrid CPU/GPU approach to Hartree–Fock. Journal of Chemical Theory and Computation 8(11): 4166–4176.

Berman

Deshmukh

Sankaranarayanan

. (2015) Macroscale superlubricity enabled by graphene nanoscroll formation. Science 348(6239): 1118–1122.

Chow

Liu

Misra

. (2015) Scaling up Hartree-Fock calculations on Tianhe-2. International Journal of High Performance Computing Applications 30(1): 1094–3420.

Dupuis

Rys

King

(1976) Evaluation of molecular integrals over Gaussian basis functions. The Journal of Chemical Physics 65(1): 111–116.

10.

Fedorov

Kitaura

(2007) Extending the power of quantum chemistry to large systems with the fragment molecular orbital method. The Journal of Physical Chemistry A 111(30): 6904–6914.

11.

Fletcher

Schmidt

Bode

. (2000) The distributed data interface in GAMESS. Computer Physics Communications 128(1): 190–200.

12.

Flocke

Lotrich

(2008) Efficient electronic integrals and their generalized derivatives for object oriented implementations of electronic structure calculations. Journal of Computational Chemistry 29(16): 2722–2736.

13.

Foster

Tilson

Wagner

. (1996) Toward high-performance computational chemistry: I. Scalable Fock matrix construction algorithms. Journal of Computational Chemistry 17(1): 109–123.

14.

Furlani

Kong

Gill

(2000) Parallelization of SCF calculations within Q-Chem. Computer Physics Communications 128(1): 170–177.

15.

Harrison

Guest

Kendall

. (1996) Toward high-performance computational chemistry: II A scalable self consitent field program. Journal of Computational Chemistry 17(1): 124–132.

16.

INCITE (2017) Department of Energy INCITE program. Available at: http://www.doeleadershipcomputing.org/ (accessed 11 April 2016).

17.

Ishimura

Kuramoto

Ikuta

. (2010) MPI/OpenMP hybrid parallel algorithm for Hartree–Fock calculations. Journal of Chemical Theory and Computation 6(4): 1075–1080.

18.

Liu

Patel

Chow

(2014) A new scalable parallel algorithm for Fock matrix construction. In: 2014 IEEE 28th international parallel and distributed processing symposium, Phoenix, AZ, USA, 19–23 May 2014, pp. 902–914. Washington, DC, USA: IEEE Computer Society.

19.

McMurchie

Davidson

(1978) One and two-electron integrals over Cartesian Gaussian functions. Journal of Computational Physics 26(2): 218–231.

20.

Mironov

Khrenova

Moskovsky

(2015) On quantum chemistry code adaptation for RSC PetaStream architecture. In: High performance computing: 30th international conference, ISC high performance 2015, Frankfurt, Germany, July 12–16, 2015, pp. 113–121. Cham, Switzerland: Springer International Publishing.

21.

Obara

Saika

(1986) Efficient recursive computation of molecular integrals over Cartesian Gaussian functions. The Journal of Chemical Physics 84(7): 3963–3974.

22.

Pritchard

Chow

(2016) Horizontal vectorization of electron repulsion integrals. Journal of Computational Chemistry 37(28): 2537–2546.

23.

Reinders

(2012) An overview of programming for Intel Xeon processors and Intel Xeon Phi coprocessors. Available at: https://software.intel.com/en-us/blogs/2012/11/14/an-overview-of-programming-for-intel-xeon-processors-and-intel-xeon-phi (accessed 11 April 2016).

24.

Schmidt

Baldridge

Boatz

. (1993) General atomic and molecular electronic structure system. Journal of Computational Chemistry 14(11): 1347–1363.

25.

Semin

Druzhinin

Mironov

. (2014) The performance characterization of the RSC Petastream module. In: Supercomputing: 29th international conference, ISC 2014, Leipzig, Germany, 22–26 June 2014, pp. 420–429. Cham, Switzerland: Springer International Publishing.

26.

Shan

Austin

de Jong

. (2014) Performance tuning of Fock matrix and two-electron integral calculations for NWChem on leading HPC platforms. In: Jarvis

Wright

Hammond

(eds) High performance computing systems. Performance modeling, benchmarking and simulation: 4th international workshop, PMBS 2013, Denver, CO, USA, 18 November 2013, pp. 261–280. Cham, Switzerland: Springer International Publishing.

27.

Shan

Williams

de Jong

. (2015) Thread-level parallelization and optimization of NWChem for the Intel MIC architecture. In: Proceedings of the sixth international workshop on programming models and applications for multicores and manycores, PMAM ‘15, San Francisco, CA, USA, 7–11 February, 2015, pp. 58–67. New York, NY, USA: ACM.

28.

Sun

(2015) Libcint: an efficient general integral library for Gaussian basis functions. Journal of Computational Chemistry 36(22): 1664–1671.

29.

Ufimtsev

Martinez

(2008) Quantum chemistry on graphical processing units. 1. Strategies for two-electron integral evaluation. Journal of Chemical Theory and Computation 4(2): 222–231.

30.

Ufimtsev

Martinez

(2009) Quantum chemistry on graphical processing units. 2. Direct self-consistent-field implementation. Journal of Chemical Theory and Computation 5(4): 1004–1015.

31.

Umeda

Inadomi

Watanabe

. (2010) Parallel Fock matrix construction with distributed shared memory model for the FMO-MO method. Journal of Computational Chemistry 31(13): 2381–2388.

32.

Valiev

Bylaska

Govind

. (2010) NWChem: a comprehensive and scalable open-source solution for large scale molecular simulations. Computer Physics Communications 181(9): 1477–1489.

33.

Wilkinson

Sherwood

Guest

. (2011) Acceleration of the GAMESS-UK electronic structure package on graphical processing units. Journal of Computational Chemistry 32(10): 2313–2318.