Rapid growth in data, computational methods, and computing power is driving a remarkable revolution in what variously is termed machine learning (ML), statistical learning, computational learning, and artificial intelligence. In addition to highly visible successes in machine-based natural language translation, playing the game Go, and self-driving cars, these new technologies also have profound implications for computational and experimental science and engineering, as well as for the exascale computing systems that the Department of Energy (DOE) is developing to support those disciplines. Not only do these learning technologies open up exciting opportunities for scientific discovery on exascale systems, they also appear poised to have important implications for the design and use of exascale computers themselves, including high-performance computing (HPC) for ML and ML for HPC. The overarching goal of the ExaLearn co-design project is to provide exascale ML software for use by Exascale Computing Project (ECP) applications, other ECP co-design centers, and DOE experimental facilities and leadership class computing facilities.
AlexanderFAlmgrenABellJ, et al. (2020) Exascale applications: skin in the game. Philosophical Transactions of the Royal Society A378(2166): 20190056.
2.
AllcockWEAllenBSAnanthakrishnanR, et al. (2019) Petrel: A programmatically accessible research data service. In: Practice and Experience in Advanced Research Computing, pp. 1–7.
BabujiYWoodardALiZ, et al. (2019) Parsl: pervasive parallel programming in python. In: 28th international symposium on high-performance parallel and distributed computing, Phoenix AZ USA, June 22–29 2019. ACM. DOI: 10.1145/3307681.3325400.
5.
BilbreyJAHeindelJPSchramM, et al. (2020) A look inside the black box: using graph-theoretical descriptors to interpret a continuous-filter convolutional neural network (CF-CNN) trained on the global and local minimum energy structures of neutral water clusters. The Journal of Chemical Physics153(2): 024302. DOI: 10.1063/5.0009933.
6.
BlaiszikBWardLSchwartingM, et al. (2019) A data ecosystem to support machine learning in materials science. MRS Communications9(4): 1125–1133. DOI: 10.1557/mrc.2019.118.
7.
BrockmanGCheungVPetterssonL, et al. (2016) Openai Gym. arXiv preprint arXiv:1606.01540.
8.
ChardKTueckeSFosterI (2016) Globus: recent enhancements and future plans. In: XSEDE16 conference on diversity, big data, and science at scale, Miami USA, July 17–21 2016, pp. 1–8.
9.
ChardRBabujiYLiZ, et al. (2020) funcX: a federated function serving fabric for science. In: 29th ACM international symposium on high-performance parallel and distributed computing, Virtual (slated to be Stockholm, Sweden), 23–26 June 2020. DOI: 10.1145/3369583.3392683.
10.
ChardRLiZChardK, et al. (2019) DLHub: Model and data serving for science. In: International parallel and distributed processing symposium, Rio de Janeiro, Brazil, 20–24 May 2019. IEEE, pp. 283–292.
11.
ChengLAssaryRSQuX, et al. (2015) Accelerating electrolyte discovery for energy storage with high-throughput screening. The Journal of Physical Chemistry Letters6(2): 283–291. DOI: 10.1021/jz502319n.
12.
CreswellAWhiteTDumoulinV, et al. (2018) Generative adversarial networks: an overview. IEEE Signal Processing Magazine35: 53–65. DOI: 10.1109/MSP.2017.2765202.
13.
DanduNWardLAssaryRS, et al. (2020) Quantum-chemically informed machine learning: prediction of energies of organic molecules with 10 to 14 non-hydrogen atoms. The Journal of Physical Chemistry A124(28): 5804–5811. DOI: 10.1021/acs.jpca.0c01777.
14.
DehghannasiriRXueDBalachandranPV, et al. (2017) Optimal experimental design for materials discovery. Computational Materials Science129: 311–322.
15.
DrydenNMaruyamaNBensonT, et al. (2019) Improving strong-scaling of CNN training by exploiting finer-grained parallelism. In: International parallel and distributed processing symposium, Rio de Janeiro, Brazil, 20–24 May 2019.
16.
DunnABrenneckJJainA (2019) Rocketsled: a software library for optimizing high-throughput computational searches. Journal of Physics: Materials2(3): 034002. DOI: 10.1088/2515-7639/ab0c3d.
17.
EchekkiTMirgolbabaeiH (2015) Principal component transport in turbulent combustion: a posteriori analysis. Combustion and Flame162(5): 1919–1933.
EspeholtLMarinierRStanczykP, et al. (2019) SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference. arXiv preprint arXiv:1910.06591.
20.
EspeholtLSoyerHMunosR, et al. (2018) Impala: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures. arXiv preprint arXiv:1802.01561.
21.
FosterIAinsworthMAllenB, et al. (2017) Computing just what you need: Online data analysis and reduction at extreme scales. In: European conference on parallel processing, Jaipur, India, 18–21 December 2017, pp. 3–19.
22.
FosterIAinsworthMBessacJ, et al. (2020) Online data analysis and reduction: An important co-design motif for extreme-scale computers. The International Journal of High Performance Computing Applications. DOI: 10.1177/10943420211023549.
23.
FrazierPIPowellWBDayanikS (2008) A knowledge-gradient policy for sequential information collection. SIAM Journal on Control and Optimization47(5): 2410–2439.
24.
GalYGhahramaniZ (2016) Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In: International conference on machine learning, New York, NY, 19–24 June 2016, pp. 1050–1059.
25.
Garcia-CardonaCKannanRJohnstonT, et al. (2019) Learning to Predict Material Structure from Neutron Scattering Data. In: IEEE international conference on big data, Los Angeles, CA,9–12 December 2019, pp. 4490–4497. DOI: 10.1109/BigData47090.2019.9005968.
GoodfellowIPouget-AbadieJMirzaM, et al. (2014) Generative adversarial nets. In: GhahramaniZWellingMCortesC, et al. (eds), Advances in Neural Information Processing Systems, 27, pp. 2672–2680. URLhttp://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf.
28.
HahnOAbelT (2011) Multi-scale initial conditions for cosmological simulations. Monthly Notices of the Royal Astronomical Society415(3): 2101–2121. DOI: 10.1111/j.1365-2966.2011.18820.x.
29.
HerouxMADoerflerDWCrozierPS, et al. (2009) Improving Performance via Mini-Applications. Technical Report SAND2009-5574, Sandia National Laboratories.
30.
JiangSMalkomesGAbbottM, et al. (2018) Efficient nonmyopic batch active search. In: BengioSWallachHLarochelleH, et al. (eds), Advances in Neural Information Processing Systems, 31, pp. 1099–1109.
31.
JinWBarzilayRJaakkolaTS (2018) Junction Tree Variational Autoencoder for Molecular Graph generation. CoRR Abs/1802 04364. URLhttp://arxiv.org/abs/1802.04364.
32.
LiZChardRWardL, et al. (2020) DLHub: simplifying publication, discovery, and use of machine learning models in science. Journal of Parallel and Distributed Processing147: 64–76. in press.
33.
LookmanTEidenbenzSAlexanderF, et al. (2018) Materials Discovery and Design: By Means of Data Science and Optimal Learning. Switzerland AG: Springer International Publishing, Springer Nature, Vol. 280.
MathuriyaABardDMendygralP, et al. (2018) CosmoFlow: using deep learning to learn the universe at scale. In: SC ‘18: proceedings of the international conference for high performance computing, networking, storage, and analysis, November 2018, pp. 1–11. IEEE Press. DOI: 10.1109/SC.2018.00068.
36.
MoritzPNishiharaRWangS, et al. (2017) Ray: a distributed framework for emerging ai applications.
37.
MustafaMBardDBhimjiW, et al. (2017) CosmoGAN: creating high-fidelity weak lensing convergence maps using generative adversarial networks. Computational Astrophysics and Cosmology6: 1–13.
38.
MustafaMBardDBhimjiW, et al. (2019) CosmoGAN: creating high-fidelity weak lensing convergence maps using generative adversarial networks. Computational Astrophysics and Cosmology6(1): 1. DOI: 10.1186/s40668-019-0029-9.
39.
NoackMMDoerkGSLiR, et al. (2020) Advances in kriging-based autonomous x-ray scattering experiments. Scientific Reports10(1): 1325. DOI: 10.1038/s41598-020-57887-x.
40.
NoackMMYagerKGFukutoM, et al. (2019) A Kriging-based approach to autonomous experimentation with applications to x-ray scattering. Scientific Reports9(1): 11809. DOI: 10.1038/s41598-019-48114-3.
OyamaYMaruyamaNDrydenN, et al. (2020) The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelism. arXiv e-prints : arXiv:2007.12856.
43.
PaszkeAGrossSChintalaS, et al. (2017). Automatic differentiation in pytorch. In: NIPS-W. New York, NY: ACM Digital Library.
44.
PronzatoL (2008) Optimal experimental design and some related control problems. Automatica44(2): 303–325.
45.
RadfordAMetzLChintalaS (2016) Unsupervised representation learning with deep convolutional generative adversarial networks. In: ICLR. New York, NY: Springer US, pp. 1–16. DOI: 10.1007/s11280-018-0565-2.
46.
RichardsDAazizOCookJ, et al. (2020) Quantitative Performance Assessment of Proxy Apps and Parents. USA: LLNL-TR-809403, Lawrence Livermore National Laboratory.
47.
SergeevABalsoMD (2018) Horovod: Fast and Easy Distributed Deep Learning in TensorFlow. arXiv preprint arXiv:1802.05799.
48.
SettlesB (2009) Active Learning Literature Survey. Technical report. University of Wisconsin-Madison Department of Computer Sciences.
49.
SmithDGAAltarawyDBurnsLA, et al. (2020) The MolSSI QCArchive project: an open-source platform to compute, organize, and share quantum chemistry data. WIREs Computational Molecular Science11: e1491. DOI: 10.1002/wcms.1491.
50.
SpragueMAAnanthanSVijayakumarG, et al. (2020) ExaWind: a multifidelity modeling and simulation environment for wind energy. Journal of Physics: Conference Series1452: 012071. DOI: 10.1088/1742-6596/1452/1/012071.
51.
SuttonRSBartoAG (1998) Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, Vol. 1.
52.
TassevSEisensteinDJ (2015) Wand Elt BD and Zaldarriaga MsCOLA: The N-Body COLA Method Extended to the Spatial Domain. arXiv e-prints : arXiv:1502.07751.
53.
TassevSZaldarriagaMEisensteinDJ (2013) Solving large scale structure in ten easy steps with COLA. Journal of Cosmology and Astroparticle Physics2013(6): 036. DOI: 10.1088/1475-7516/2013/06/036.
54.
TobyBHVon DreeleRB (2013) GSAS-II: the genesis of a modern open-source all purpose crystallography software package. Journal of Applied Crystallography46(2): 544–549.
55.
Van EssenBKimHPearceR, et al. (2015) LBANN: livermore big artificial neural network HPC toolkit. In: Workshop on Machine Learning in High-Performance Computing Environments. New York, NY, USA: MLHPC ’15Association for Computing Machinery. ISBN 9781450340069. DOI: 10.1145/2834892.2834897.
56.
WaltherMOñorbeJHennawiJF, et al. (2019) New constraints on IGM thermal evolution from the Lyα forest power spectrum. The Astrophysical Journal872(1): 13. DOI: 10.3847/1538-4357/aafad1.
57.
WardLBlaiszikBFosterI, et al. (2019) Machine learning prediction of accurate atomization energies of organic molecules from low-fidelity quantum chemical calculations. MRS Communications9(3): 891–899. DOI: 10.1557/mrc.2019.107.
58.
WozniakJMJainRBalaprakashP, et al. (2018) CANDLE/supervisor: a workflow framework for machine learning applied to cancer research. BMC Bioinformatics19(S18): 491. DOI: 10.1186/s12859-018-2508-4.
59.
YoonB-JQianXDoughertyER (2013) Quantifying the objective cost of uncertainty in complex dynamical systems. IEEE Transactions on Signal Processing61(9): 2256–2266.
60.
ZhouZKearnesSLiL, et al. (2019) Optimization of molecules via deep reinforcement learning. Scientific Reports9(1): 10752. DOI: 10.1038/s41598-019-47148-x.
