Challenges in implementing AI/ML training job recovery from GPU/Accelerator data poisoning events

Anil Agrawal (Meta Platform Corp - Hardware Systems Engineer) David Xiao (Meta - Engineering Manager) AI/ML training job interruptions due to hardware faults such as GPU/Accelerator memory uncorrected errors is a growing issue- especially as we build very large clusters (100K+). In this presentation- we would like to share the challenges we have experienced as a result of uncorrected errors during the development of Meta's large training clusters built using the Grand Teton Training Platform- how such errors affected the training job 'interruption rate'- and how RAS technology is used in contain the impact. At the end- we would like to share a call to action for the OCP community on how to reduce the interruptions due to such hardware uncorrected errors by using various 'recovery' techniques.