As high performance computing architecture evolves to deliver ever-increasing performance, the middleware tools also need to adapt in order for applications to better use these higher-performance features. The Adaptable Input Output System (ADIOS), which provides scalable IO performance for exascale HPC applications is one such middleware. During the Exascale Computing Project (ECP), key portions of the ADIOS environment were adapted to respond to ongoing developments in exascale computing and the stresses and opportunities inherent in those changes. This paper examines those changes and where appropriate compares them to pre-exascale implementations.
AgarwalaSEisenhauerGSchwanK (2005) Lightweight morphing support for evolving middleware data exchanges in distributed applications. In: 25th IEEE International Conference on Distributed Computing Systems (ICDCS’05). Piscataway: IEEE, 697–706. https://api.semanticscholar.org/CorpusID:2190657
2.
BozdağEPeterDLefebvreM, et al. (2016) Global adjoint tomography: first-generation model. Geophysical Journal International207(3): 1739–1766. DOI: 10.1093/gji/ggw356.
ChengJDominskiJChenY, et al. (2020) Spatial core-edge coupling of the particle-in-cell gyrokinetic codes GEM and XGC. Physics of Plasmas27(12): 122510. DOI: 10.1063/5.0026043.
5.
DiffenderferJFoxALHittingerJA, et al. (2019) Error analysis of zfp compression for floating-point data. SIAM Journal on Scientific Computing41(3): A1867–A1898. DOI: 10.1137/18M1168832.
6.
EisenhauerGWolfMAbbasiH, et al. (2009) Event-based systems: opportunities and challenges at exascale. In: Proceedings of the Third ACM International Conference on Distributed Event-Based Systems, DEBS ’09. New York, NY, USA: Association for Computing Machinery, 1–10. DOI: 10.1145/1619258.1619261.
7.
EisenhauerGWolfMAbbasiH, et al. (2011a) A type system for high performance communication and computation. In: 2011 IEEE Seventh International Conference on E-Science Workshops. Piscataway: IEEE, 183–190. https://api.semanticscholar.org/CorpusID:7893343
8.
EisenhauerGWolfMAbbasiH, et al. (2011b) A type system for high performance communication and computation. In: 2011 IEEE Seventh International Conference on E-Science Workshops. Piscataway: IEEE, 183–190. DOI: 10.1109/eScienceW.2011.16.
9.
FedeliLHueblABoillod-CerneuxF, et al. (2022) Pushing the frontier in the design of laser-based electron accelerators with groundbreaking mesh-refined particle-in-cell simulations on exascale-class supercomputers. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC ’22. Piscataway: IEEE Press, 1–12.
10.
FolkMChengAYatesK (1999) Hdf5: a file format and i/o library for high performance computing applications. Proceedings of Supercomputing99: 5–33.
11.
GodoyWFPodhorszkiNWangR, et al. (2020) ADIOS 2: the adaptable input output system. A framework for high-performance data management. SoftwareX12: 100561. DOI: 10.1016/j.softx.2020.100561.
HollmanDSLelbachBAEdwardsHC, et al. (2019) Mdspan in c++: a case study in the integration of performance portable features into international language standards. In: 2019 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC). Piscataway: IEEE Press, 60–70. DOI: 10.1109/P3HPC49587.2019.00011.
LathamRZingaleMThakurR, et al. (2003) Parallel netcdf: a high-performance scientific i/o interface. In: SC Conference. Los Alamitos, CA, USA: IEEE Computer Society, 39. DOI: 10.1109/SC.2003.10053.
18.
LiangZLombardiJChaarawiM, et al. (2020) Daos: a scale-out high performance storage stack for storage class memory. In: Supercomputing Frontiers: 6th Asian Conference, SCFA 2020, Singapore, February 24–27, 2020, Proceedings. Berlin, Heidelberg: Springer-Verlag, 40–54. DOI: 10.1007/978-3-030-48842-0_3.
19.
LiuQLoganJTianY, et al. (2014) Hello adios: the challenges and lessons of developing leadership class i/o frameworks. Concurrency and Computation: Practice and Experience26(7): 1453–1473. DOI: 10.1002/cpe.3125.
20.
MerloGJanhunenSJenkoF, et al. (2021) First coupled GENE–XGC microturbulence simulations. Physics of Plasmas28(1): 012303. DOI: 10.1063/5.0026661.
21.
MoodyABronevetskyGMohrorK, et al. (2010) Design, modeling, and evaluation of a scalable multi-level checkpointing system. In: SC’10: Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis. Piscataway: IEEE Press, 1–11.
22.
MooreJKunisS (2023) Zarr: a cloud-optimized storage for interactive access of large arrays. Proceedings of the Conference on Research Data Infrastructure1: 285. DOI: 10.52825/cordi.v1i.285.
23.
NicolaeBMoodyAGonsiorowskiE, et al. (2019) Veloc: towards high performance adaptive asynchronous checkpointing at large scale. In: 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS). Piscataway: IEEE, 911–920. DOI: 10.1109/IPDPS.2019.00099.
24.
PolteMLofsteadJBentJGibsonGKlaskySALiuQParasharMPodhorszkiNSchwanKWingateMWolfM (2009) …and eat it too: high read performance in write-optimized hpc i/o middleware file formats. In: Proceedings of the 4th Annual Workshop on Petascale Data Storage, PDSW ’09. New York, NY, USA: Association for Computing Machinery, 21–25. DOI: 10.1145/1713072.1713079.
25.
Sarpangala VenkateshREisenhauerGKlaskyS, et al. (2023) Enhancing metadata transfer efficiency: unlocking the potential of daos in the adios context. In: Proceedings of the SC ’23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W ’23. New York, NY, USA: Association for Computing Machinery, 1223–1228. DOI: 10.1145/3624062.3624193.
26.
SchmuckFHaskinR (2002) Gpfs: a shared-disk file system for large computing clusters. In: Proceedings of the 1st USENIX Conference on File and Storage Technologies, FAST ’02. USA: USENIX Association, 19–es.
27.
TaylorMCaldwellPMBertagnaL, et al. (2023) The simple cloud-resolving e3sm atmosphere model running on the frontier exascale system. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’23. New York, NY, USA: Association for Computing Machinery, 1–11. DOI: 10.1145/3581784.3627044.
28.
TianYKlaskySAbbasiH, et al. (2011) Edo: improving read performance for scientific applications through elastic data organization. In: 2011 IEEE International Conference on Cluster Computing. Piscataway: IEEE, 93–102. https://api.semanticscholar.org/CorpusID:18030868
29.
TrottCRLebrun-GrandieDArndtD, et al. (2022) Kokkos 3: programming model extensions for the exascale era. IEEE Transactions on Parallel and Distributed Systems33(4): 805–817. DOI: 10.1109/TPDS.2021.3097283.
30.
WanLHueblAGuJ, et al. (2022) Improving i/o performance for exascale applications through online data layout reorganization. IEEE Transactions on Parallel and Distributed Systems33(4): 878–890. DOI: 10.1109/TPDS.2021.3100784.
