Exascale studies project reliability challenges for future HPC systems. We present the Global View Resilience (GVR) system, a library for portable resilience. GVR begins with a subset of the Global Arrays interface, and adds new capabilities to create versions, name versions, and compute on version data. Applications can focus versioning where and when it is most productive, and customize for each application structure independently. This control is portable, and its embedding in application source makes it natural to express and easy to maintain. The ability to name multiple versions and “partially materialize” them efficiently makes ambitious forward-recovery based on “data slices” across versions or data structures both easy to express and efficient. Using several large applications (OpenMC, preconditioned conjugate gradient (PCG) solver, ddcMD, and Chombo), we evaluate the programming effort to add resilience. The required changes are small (< 2% lines of code (LOC)), localized and machine-independent, and perhaps most important, require no software architecture changes. We also measure the overhead of adding GVR versioning and show that overheads < 2% are generally achieved. This overhead suggests that GVR can be implemented in large-scale codes and support portable error recovery with modest investment and runtime impact. Our results are drawn from both IBM BG/Q and Cray XC30 experiments, demonstrating portability. We also present two case studies of flexible error recovery, illustrating how GVR can be used for multi-version rollback recovery, and several different forward-recovery schemes. GVR’s multi-version enables applications to survive latent errors (silent data corruption) with significant detection latency, and forward recovery can make that recovery extremely efficient. Our results suggest that GVR is scalable, portable, and efficient. GVR interfaces are flexible, supporting a variety of recovery schemes, and altogether GVR embodies a gentle-slope path to tolerate growing error rates in future extreme-scale systems.
AntypasKWrightNCardoNPAndrewsACorderyM (2014) Cori: A Cray XC pre-exascale system for NERSC. In: Cray User Group Proceedings. Cray Research, Inc.
2.
AupyGBenoitAHeraultTRobertYVivienFZaidouniD (2013) On the combination of silent error detection and checkpointing. In: 2013 IEEE 19th Pacific Rim International Symposium on Dependable Computing (PRDC), pp. 11–20. DOI:10.1109/PRDC.2013.10.
3.
BariusoRKniesA (1994) Shmem user’s guide. Cray Research, Inc.
4.
Bautista-GomezLTsuboiSKomatitschDCappelloFMaruyamaNMatsuokaS (2011) FTI: High performance fault tolerance interface for hybrid systems. In: SC ’11. DOI:10.1145/2063384.2063427.
5.
BergerMColellaP (1989) Local adaptive mesh refinement for shock hydrodynamics. Journal of Computational Physics82(1): 64–84.
6.
BergerMOligerJ (1984) Adaptive mesh refinement for hyperbolic partial differential equations. Journal of Computational Physics53(3): 484–512.
BorkarSChienAA (2011) The future of microprocessors. Communications of the ACM54: 67–77.
9.
BridgesPGHoemmenMFerreiraKBHerouxMASolteroPBrightwellR (2012) Cooperative application/OS DRAM fault recovery. In: Resilience’11, pp. 241–250. DOI:10.1007/978-3-642-29740-3_28.
10.
BronevetskyGde SupinskiB (2008) Soft error vulnerability of iterative linear algebra methods. In: ICS.
11.
CappelloFCasanovaHRobertY (2011) Preventive migration vs. preventive checkpointing for extreme scale supercomputers. Parallel Processing Letters21(02): 111–132.
12.
CappelloFGeistAGroppWKaleLKramerWSnirM (2009) Towards exascale resilience. International Journal of High Performance Computing Applications23(4): 374–388.
13.
CarbinMMisailovicSRinardMC (2013) Verifying quantitative reliability for programs that execute on unreliable hardware. In: OOPSLA ’13, pp. 33–52. DOI:10.1145/2509136.2509546.
14.
CarlsonWDraperJCullerDYelickKBrooksEWarrenK (1999) Introduction to UPC and language specification. Technical Report CCS-TR-99-157, IDA Center for Computing Sciences.
15.
ChamberlainBLCallahanDZimaHP (2007) Parallel programmability and the chapel language. The International Journal of High Performance Computing Applications21(3): 291–312.
16.
ChangFDeanJGhemawatSHsiehWCWallachDABurrowsMChandraTFikesAGruberRE (2006) Bigtable: A distributed storage system for structured data. In: OSDI ’06, p. 15.
17.
CharlesPGrothoffCSaraswatVDonawaCKielstraAEbciogluKvon PraunCSarkarV (2005) X10: An object-oriented approach to non-uniform cluster computing. In: OOPSLA ’05, pp. 519–538. DOI:10.1145/1094811.1094852.
18.
ChenZ (2013) Online-ABFT: an online algorithm based fault tolerance scheme for soft error detection in iterative methods. In: PPoPP ’13, pp. 167–176. DOI:10.1145/2442516.2442533.
19.
ColellaP (1990) Multidimensional upwind methods for hyperbolic conservation laws. Journal of Computational Physics87(1): 171–200.
20.
ColellaPGravesDKeenNLigockiTMartinDMcCorquodalePModianoDSchwartzPSternbergTVan StraalenB (2009) Chombo software package for AMR applications design document. Technical report, LBNL, Applied Numerical Algorithms Group, Computational Research Division.
21.
DalyJT (2006) A higher order estimate of the optimum checkpoint interval for restart dumps. Future Generation Computer Systems22(3). DOI:10.1016/j.future.2004.11.016.
22.
DavisTAHuY (2011) The university of florida sparse matrix collection. ACM Transaction on Mathematical Software38(1): 1:1–1:25.
23.
DeHonACarterNQuinnH (2011) Final report for CCC cross-layer reliability visioning study. Available at:http://www.cra.org/ccc/xlayer.php.
24.
DennardRHGaensslenFHRideoutVLBassousELeBlancAR (1974) Design of ion-implanted MOSFET’s with very small physical dimensions. IEEE Journal of Solid-State Circuits9(5): 256–268.
25.
Di MartinoCKalbarczykZIyerRBaccanicoFFullopJKramerW (2014) Lessons learned from the analysis of system failures at petascale: The case of Blue Waters. In: DSN 2014, pp. 610–621. DOI:10.1109/DSN.2014.62.
26.
DubeyAMohapatraPWeideK (2013) Fault tolerance using lower fidelity data in adaptive mesh applications. In: Proceedings of the 3rd Workshop on Fault-tolerance for HPC at Extreme Scale. New York: ACM Press, pp. 3–10. DOI:10.1145/2465813.2465817.
27.
DunNFujitaHTrammJChienAASiegelAR (2014) Data decomposition in Monte Carlo particle transport simulations using global view arrays. Technical Report TR-2014-09, Department of Computer Science, University of Chicago.
28.
ElliottJMuellerFStoyanovMWebsterC (2013) Quantifying the impact of single bit flips on floating point arithmetic. Technical Report TR 2013-2, Department of Computer Science, North Carolina State University.
29.
ElnozahyM (2009) System resilience at extreme scale: A white paper. DARPA Resilience Report for ITO, William Harrod.
30.
EsmaeilzadehHBlemESt AmantRSankaralingamKBurgerD (2011) Dark silicon and the end of multicore scaling. In: 2011 38th Annual International Symposium on Computer Architecture (ISCA). IEEE, pp. 365–376.
31.
FangAChienAA (2014) Applying GVR to molecular dynamics: Enabling resilience for scientific computations. Technical Report TR-2014-04, University of Chicago.
32.
FangAChienAA (2015) How much SSD is useful for resilience in supercomputers. Manuscript submitted for publication.
33.
FerreiraKStearleyJLarosJHIIIOldfieldRPedrettiKBrightwellRRiesenRBridgesPGArnoldD (2011) Evaluating the viability of process replication reliability for exascale systems. In: SC ’11.
34.
FialaDMuellerFEngelmannCRiesenRFerreiraKBrightwellR (2012) Detection and correction of silent data corruption for large-scale high-performance computing. In: Proceedings of Supercomputing, p. 78.
35.
GlosliJNRichardsDFCaspersenKJRuddREGunnelsJAStreitzFH (2007) Extending stability beyond CPU millennium: a micron-scale atomistic simulation of Kelvin–Helmholtz instability. In: Proceedings of SC ’07, pp. 1–11. DOI:10.1145/1362622.1362700.
GuptaRBeckmanPParkBHLuskEHargrovePGeistAPandaDLumsdaineADongarraJ (2009) CIFTS: A coordinated infrastructure for fault-tolerant systems. In: Proceedings of ICPP ’09, pp. 237–245. DOI:10.1109/ICPP.2009.20.
39.
GVR Team (2014a) Global View Resilience (GVR) API documentation, version 1.0. Technical report, University of Chicago, Department of Computer Science.
40.
GVR Team (2014b) Global View Resilience (gvr) documentation, release 1.0. Technical Report TR-2014-13, University of Chicago, Department of Computer Science.
41.
HargrovePHDuellJC (2006) Berkeley Lab checkpoint/restart (BLCR) for Linux clusters. Journal of Physics: Conference Series46: 494.
42.
HariSKSAdveSVNaeimiH (2012) Low-cost program-level detectors for reducing silent data corruptions. In: IPDPS.
43.
HerouxMA (2014) Toward resilient algorithms and applications. CoRRPreprint abs/1402.3809.
44.
HerouxMABartlettRAHowleVEHoekstraRJHuJJKoldaTGLehoucqRBLongKRPawlowskiRPPhippsET. (2005) An overview of the trilinos project. ACM Transactions on Mathematical Software31(3): 397–423.
45.
HerouxMADongarraJLuszczekP (2013) HPCG technical specification. Technical report, Sandia National Laboratories.
46.
HoganSHammondJRChienAA (2012) An evaluation of difference and threshold techniques for efficient checkpoints. In: 2012 IEEE/IFIP 42nd International Conference on Dependable Systems and Networks Workshops (DSN-W). IEEE, pp. 1–6.
47.
HoogenboomJEMartinWRPetrovicB (2011) The Monte Carlo performance benchmark test - aims, specifications and first results. In: ANS M&C.
48.
HorelikNForgetBSmithKSiegelA (2014) Domain decomposition and terabyte tallies with the OpenMC Monte Carlo neutron transport code. In: PHYSOR 2014 – Advances in Reactor Physics – The Role of Reactor Physics toward a Sustainable Future.
49.
HuangKHAbrahamJA (1984) Algorithm-based fault tolerance for matrix operations. IEEE Transactions on Computers33(6): 518–528.
50.
KershawDS (1978) The incomplete Cholesky–conjugate gradient method for the iterative solution of systems of linear equations. Journal of Computational Physics26(1): 43–65.
51.
KonishiRAmagaiYSatoKHifumiHKiharaSMoriaiS (2006) The Linux implementation of a log-structured file system. SIGOPS Operating Systems Review40(3): 102–107.
52.
LakshminarasimhanSShahNEthierSKuSHChangCSKlaskySLathamRRossRSamatovaNF (2013) Isabela for effective in situ compression of scientific data. Concurrency and Computation: Practice and Experience25(4): 524–540.
LuCdReedDA (2004) Assessing fault sensitivity in MPI applications. In: Proceedings of Supercomputing.
55.
LuGZhengZChienAA (2013a) When is multi-version checkpointing needed? In: FTXS ’13. New York: ACM Press. DOI:10.1145/2465813.2465821.
56.
LuGZhengZChienAA (2013b) When is multi-version checkpointing needed? In:
Proceedings of the 3rd Workshop on Fault-tolerance for HPC at Extreme Scale (FTXS ’13). New York: ACM Press. DOI:10.1145/2465813.2465821.
57.
LyonsREVanderkulkW (1962) The use of triple-modular redundancy to improve computer reliability. IBM Journal of Research and Development6(2): 200–209.
58.
MartinWR (2012) Challenges and prospects for whole-core Monte Carlo analysis. Journal of Nuclear Engineering Technology44(2): 151–160.
59.
MisailovicSCarbinMAchourSQiZRinardMC (2014) Chisel: Reliability- and accuracy-aware optimization of approximate computational kernels. In: OOPSLA ’14, pp. 309–328. DOI:10.1145/2660193.2660231.
60.
MoodyABronevetskyGMohrorKSupinskiBRd (2010) Design, modeling, and evaluation of a scalable multi-level checkpointing system. In: SC ’10. IEEE Computer Society. DOI:10.1109/SC.2010.18.
61.
NieplochaJPalmerBTipparajuVKrishnanMTreaseHApràE (2006) Advances, applications and performance of the Global Arrays shared memory programming toolkit. The International Journal of High Performance Computing Applications20(2): 203–231.
RandellBLeePTreleavenPC (1978) Reliability issues in computing system design. ACM Computer Surveys10(2): 123–165.
65.
RodehOBacikJMasonC (2013) Btrfs: The Linux B-tree filesystem. Transactions on Storage9(3): 9:1–9:32.
66.
RomanoPKForgetB (2013) The OpenMC Monte Carlo particle transport code. Annals of Nuclear Energy51: 274–281.
67.
RomanoPKSiegelARForgetBSmithK (2013) Data decomposition of Monte Carlo particle transport simulations via tally servers. Journal of Computational Physics252: 20–36.
68.
RubensteinZFujitaHZhengZChienA (2013) Error checking and snapshot-based recovery in a preconditioned conjugate gradient solver. Technical Report TR-2013-11, Department of Computer Science, University of Chicago.
69.
SaadY (1993) A flexible inner-outer preconditioned GMRES algorithm. SIAM Journal on Scientific Computing14(2): 461–469.
70.
SampsonADietlWFortunaEGnanapragasamDCezeLGrossmanD (2011) EnerJ: Approximate data types for safe and general low-power computation. In: PLDI ’11, pp. 164–174. DOI:10.1145/1993498.1993518.
71.
SchlichtingRDSchneiderFB (1983) Fail-stop processors: An approach to designing fault-tolerant computing systems. ACM Transactions on Computing Systems1(3): 222–238.
72.
SchroederBGibsonGA (2006) A large-scale study of failures in high-performance computing systems. In: DSN.
73.
ShantharamMSrinivasmurthySRaghavanP (2011) Characterizing the impact of soft errors on iterative methods in scientific computing. In: Proceedings of Supercomputing.
74.
SiegelARSmithKRomanoPKForgetBFelkerK (2013) The effect of load imbalances on the performance of Monte Carlo codes in LWR analysis. Journal of Computational Physics235: 901–911.
75.
SnirMWisniewskiRWAbrahamJAAdveSVBagchiSBalajiPBelakJBosePCappelloFCarlsonBChienAACoteusPDebardelebenNADinizPEngelmannCErezMFazzariSGeistAGuptaRJohnsonFKrishnamoorthySLeyfferSLibertyDMitraSMunsonTSSchreiberRStearleyJHensbergenEV (2014) Addressing failures in exascale computing. International Journal of High Performance Computing28(2): 129–173.
StreitzFHGlosliJNPatelMVChanBYatesRKde SupinskiBRSextonJGunnelsJA (2006) Simulating solidification in metals at high pressure: The drive to petascale computing. Journal of Physics: Conference Series46(1): 254.
78.
SuttonTMDonovanTJTrumbullTHDobreffPSCaroEGriesheimerDPTyburskiLJCarpenterDCJooH (2007) The MC21 Monte Carlo transport code. In: Joint International Topical Meeting on Mathematics and Computation and Supercomputing in Nuclear Applications.
79.
YoungJW (1974) A first order approximation to the optimum checkpoint interval. Communications of the ACM17(9).
80.
ZhengZYuLTangWLanZGuptaRDesaiNCoghlanSBuettnerD (2011) Co-analysis of RAS log and job log on Blue Gene/P. In: Proceedings of IPDPS.
