The Exascale Computing Project (ECP) is invested in co-design to assure that key applications are ready for exascale computing. Within ECP, the Co-design Center for Particle Applications (CoPA) is addressing challenges faced by particle-based applications across four “sub-motifs”: short-range particle–particle interactions (e.g., those which often dominate molecular dynamics (MD) and smoothed particle hydrodynamics (SPH) methods), long-range particle–particle interactions (e.g., electrostatic MD and gravitational N-body), particle-in-cell (PIC) methods, and linear-scaling electronic structure and quantum molecular dynamics (QMD) algorithms. Our crosscutting co-designed technologies fall into two categories: proxy applications (or “apps”) and libraries. Proxy apps are vehicles used to evaluate the viability of incorporating various types of algorithms, data structures, and architecture-specific optimizations and the associated trade-offs; examples include ExaMiniMD, CabanaMD, CabanaPIC, and ExaSP2. Libraries are modular instantiations that multiple applications can utilize or be built upon; CoPA has developed the Cabana particle library, PROGRESS/BML libraries for QMD, and the SWFFT and fftMPI parallel FFT libraries. Success is measured by identifiable “lessons learned” that are translated either directly into parent production application codes or into libraries, with demonstrated performance and/or productivity improvement. The libraries and their use in CoPA’s ECP application partner codes are also addressed.
AdedoyinAANegreCFABockN, et al. (2019) Performance optimizations of recursive electronic structure solvers targeting multi-core architectures (LA-UR, 20–26665). arXiv:2102.08505.
2.
AlexanderFAlmgrenABellJ, et al. (2020) Exascale applications: skin in the game. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences378(2166): 20190056. DOI: 10.1098/rsta.2019.0056.
3.
AlmgrenASBellJBLijewskiMJ, et al. (2013) Nyx: a massively parallel AMR code for computational cosmology. The Astrophysical Journal765(1): 39. DOI: 10.1088/0004-637X/765/1/39.
4.
AyalaATomovSLuoX, et al. (2019) Impacts of multi-GPU MPI collective communications on large FFT computation. In: Workshop on Exascale MPI (ExaMPI) at SC19, Denver, CO, 17 November 2019. New York: IEEE. DOI: 10.1109/ExaMPI49596.2019.00007.
BockNNegreCFAMniszewskiSM, et al. (2018) The basic matrix library (BML) for quantum chemistry. The Journal of Supercomputing74(11): 6201–6219.
8.
BowersKJAlbrightBJYinL, et al. (2009) Advances in petascale kinetic plasma simulation with VPIC and Roadrunner. Journal of Physics: Conference Series180: 012055.
9.
BrackbillJUForslundDW (1985) Simulation of low-frequency, electromagnetic phenomena in plasmas. In: Brackbill JU and Cohen BI (eds) Multiple Time Scales. Cambridge Mass: Academic Press.
10.
ChenGChacónL (2015) A multi-dimensional, energy- and charge-conserving, nonlinearly implicit, electromagnetic Vlasov-Darwin particle-in-cell algorithm. Computer Physics Communications197: 73–87.
11.
ChenGChacónLBarnesDC (2011) An energy- and charge-conserving, implicit, electrostatic particle-in-cell algorithm. Journal of Computational Physics230(18): 7018–7036.
12.
ChenGChacónLBarnesDC (2012) An efficient mixed-precision, hybrid CPU-GPU implementation of a nonlinearly implicit one-dimensional particle-in-cell algorithm. Journal of Computational Physics231(16): 5374–5388.
13.
ChenGChacónLYinL, et al. (2020) A semi-implicit, energy- and charge-conserving particle-in-cell algorithm for the relativistic Vlasov-Maxwell equations. Journal of Computational Physics407: 109228.
14.
CuiQElstnerM (2014) Density functional tight binding: values of semi-empirical methods in an ab initio era. Physical Chemistry Chemical Physics16(28): 14368–14377. DOI: 10.1039/c4cp00908h.
15.
DavidsonRC (2001) Physics of Nonneutral Plasmas. London, England: Imperial College Press London.
16.
DawsonWNakajimaT (2018) Massively parallel sparse matrix function calculations with NTPoly. Computer Physics Communications225: 154–165. DOI: 10.1016/j.cpc.2017.12.010.
17.
DesaiSReeveSTBelakJF (2020) Implementing a neural network interatomic model with performance portability for emerging exascale architectures. arXiv:2002.00054.
18.
DongarraJGatesMHaidarA, et al. (2014) Accelerating numerical dense linear algebra calculations with GPUs. In: Kindratenko V (ed) Numerical Computations with GPUs. New York: Springer.
19.
EdwardsHCTrottCRSunderlandD (2014) Kokkos: enabling manycore performance portability through polymorphic memory access patterns. Journal of Parallel and Distributed Computing74(12): 3202–3216. DOI: 10.1016/j.jpdc.2014.07.003.
20.
EssmannUPereraLBerkowitzML, et al. (1995) A smooth particle mesh Ewald method. The Journal of Chemical Physics103(19): 8577–8593. DOI: 10.1063/1.470117.
21.
FalgoutRDYangUM (2002) hypre: a library of high performance preconditioners. In: SlootPMAHoekstraAGTanCJK, et al. (eds) Computational Science—ICCS 2002, Lecture Notes in Computer Science. Berlin, Heidelberg: Springer, 632–641. ISBN 978-3-540-47789-1. DOI: 10.1007/3-540-47789-6_66.
22.
FattebertJLOsei-KuffuorDDraegerEW, et al. (2016) Modeling dilute solutions using first-principles molecular dynamics: computing more than a million atoms with over a million cores. In: SC’16: proceedings of the international conference for high performance computing, networking, storage and analysis, Salt Lake City, UT, 13–18 November 2016, pp. 12–22. New York: IEEE.
23.
FranchettiFSpampinatoDGKulkarniA, et al. (2020) FFT and solver libraries for exascale: FFTx and spectralpack. In: Exascale computing project (ECP) annual meeting poster, Houston, 3–7February 2020.
GayatriRMooreSWeinbergE, et al. (2020) Rapid exploration of optimization strategies on advanced architectures using TestSNAP and LAMMPS. arXiv e-prints: arXiv:2011.12875.
26.
GenoniTClarkRWelchD (2010) A fast implicit algorithm for highly magnetized charged particle motion. The Open Plasma Physics Journal3(1): 36–41.
27.
GermannTCMcPhersonALBelakJF, et al. (2013) Exascale co-design center for Materials in Extreme environments (ExMatEx) annual report—year 2. Lawrence Livermore National Laboratory, Technical Report LLNL-SR-647437.
28.
GriebelMSchneiderMZengerC (1992) A combination technique for the solution of sparse grid problems. In: De Groen P and Beauwens R (eds) Iterative Methods in Linear Algebra. North Holland,Amsterdam.
29.
HabibSMorozovVFinkelH, et al. (2012) The universe at extreme scale: multi-petaflop sky simulation on the BG/Q. arXiv e-prints: arXiv:1211.4864.
30.
HabibSPopeAFinkelH, et al. (2016) HACC: simulating sky surveys on state-of-the-art supercomputing architectures. New Astronomy42: 49–65. DOI: 10.1016/j.newast.2015.06.003.
31.
HockneyRWEastwoodJW (1989) Computer Simulation Using Particles. 1st edition. Bristol England, PA: CRC Press. ISBN 978-0-85274-392-8.
32.
HourahineBAradiBBlumV, et al. (2020) DFTB+, a software package for efficient approximate density functional theory based atomistic simulations. The Journal of Chemical Physics152(12): 124101.
33.
KuSChangCSHagerR, et al. (2018) A fast low-to-high confinement mode bifurcation dynamics in the boundary-plasma gyrokinetic code XGC1. Physics of Plasmas25: 056107. DOI: 10.1063/1.5020792.
34.
LauY-TFinnJM (1990) Three-dimensional kinematic reconnection in the presence of field nulls and closed field lines. The Astrophysical Journal350: 672–691.
35.
Lebrun-GrandiéDProkopenkoATurcksinB, et al. (2019) ArborX: a performance portable search library. arXiv:1908.11807.
36.
LiuGRLiuMB (2003) Smoothed Particle Hydrodynamics: A Meshfree Particle Method. Singapore: World Scientific.
37.
MarxDHutterJ (2009) Ab Initio Molecular Dynamics: Basic Theory and Advanced Methods. Cambridge, England: Cambridge University Press.
38.
MeloMCRBernardiRCRudackT, et al. (2018) NAMD goes quantum: an integrative suite for hybrid simulations. Nature Methods15(5): 351–354. DOI: 10.1038/nmeth.4638.
39.
MniszewskiSMCawkwellMJWallME, et al. (2015) Efficient parallel linear scaling construction of the density matrix for Born-Oppenheimer molecular dynamics. Journal of Chemical Theory and Computation11(10): 4644–4654.
40.
MniszewskiSMPerriotRRubenssonEH, et al. (2019) Linear scaling pseudo Fermi-operator expansion for fractional occupation. Journal of Chemical Theory and Computation15(1): 190–200.
41.
Mohd-YusofJSakharnykhNMniszewskiSM, et al. (2015) Fast sparse matrix multiplication for QMD using parallel merge. In: GPU Technology Conference, San Jose, CA, 17–20March 2015: NVDIA.
42.
NegreCFAMniszewskiSMCawkwellMJ, et al. (2016) Recursive factorization of the inverse overlap matrix in linear-scaling quantum molecular dynamics simulations. Journal of Chemical Theory and Computation12(7): 3063–3073.
43.
NiklassonAMNMniszewskiSMNegreCFA, et al. (2016) Graph-based linear scaling electronic structure theory. The Journal of Chemical Physics144: 234101.
44.
NiklassonAMNTymczakCJChallacombeM (2003) Trace resetting density matrix purification in O(N) self-consistent-field theory. The Journal of Chemical Physics118(19): 8611–8620. DOI: 10.1063/1.1559913.
45.
ParkerSEBirdsallCK (1991) Numerical error in electron orbits with large ωceΔt. Journal of Computational Physics97(1): 91–102.
46.
PeeblesPJE (1980) The Large-Scale Structure of the Universe. Princeton, NJ: Princeton Univ Press.
47.
PerezD.CubukE. D.WaterlandA., et al. (2016) Long-time dynamics through parallel trajectory splicing. Journal of Chemical Theory and Computation12(1): 18–28.
48.
PlimptonS (1995) Fast parallel algorithms for short-range molecular dynamics. Journal of Computational Physics117(1): 1–19. Available at: http://lammps.sandia.gov.
49.
PlimptonSKohlmeyerACoffmanP, et al. (2018) fftMPI, a library for performing 2d and 3d FFTs in parallel, version 00. Available at: https://www.osti.gov//servlets/purl/1457552.
50.
PopeADanielDFrontiereN (2017) SWFFT: a stand-alone version of HACC’s distributed-memory, pencil-decomposed, parallel 3D FFT. Available at: https://xgitlab.cels.anl.gov/hacc/SWFFT.
51.
RicketsonLFCerfonAJ (2016) Sparse grid techniques for particle-in-cell schemes. Plasma Physics and Controlled Fusion59(2): 024002.
52.
RicketsonLFCerfonAJ (2018) Sparse grid particle-in-cell scheme for noise reduction in beam simulations. In: Proceeding of the 13th international computational accelerator physics conference, Key West, FL, 20–24 October 2018.
53.
RicketsonLFChacónL (2020) An energy-conserving and asymptotic-preserving charged-particle orbit implicit time integrator for arbitrary electromagnetic fields. Journal of Computational Physics418: 109639
54.
SaadY (2003) Iterative Methods for Sparse Linear Systems. 2nd edition. Philadelphia: Society for Industrial and Applied Mathematics. DOI: 10.1137/1.9780898718003.
55.
ScheinbergAChenGEthierS, et al. (2019) Kokkos and Fortran in the exascale computing project plasma physics code XGC. In: Proceedings of SC19 conference, Denver, CO, 17–22 November 2019. New York: IEEE.
56.
StanierAChacónLChenG (2019) A fully implicit, conservative, non-linear, electromagnetic hybrid particle-ion/fluid-electron algorithm. Journal of Computational Physics376: 597–616.
57.
The CP2K Developers Group (2020) DBCSR: distributed block compressed sparse row matrix library. Available at: https://github.com/cp2k/dbcsr.
58.
ThompsonAPSwilerLPTrottCR, et al. (2015) Spectral neighbor analysis method for automated generation of quantum-accurate interatomic potentials. Journal of Computational Physics285: 316–330.
59.
TuszewskiM (1988) Field reversed configurations. Nuclear Fusion28(11): 2033–2092.
60.
VayJ-LAlmgrenABellJ, et al. (2018) Warp-X: a new exascale computing platform for beam-plasma simulations. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment909: 476–479.
61.
VayJ-LHaberIGodfreyBB (2013) A domain decomposition method for pseudo-spectral electromagnetic simulations of plasmas. Journal of Computational Physics243: 260–268.
62.
VuHXBrackbillJU (1995) Accurate numerical solution of charged particle motion in a magnetic field. Journal of Computational Physics116(2): 384–387.
