PCIe Express corrected errors handling (RAS) solution implementation considerations

Anil Agrawal, Hardware Systems Engineer, Meta Carlos Fernandez, Presenter, Meta Meta's AI/ML Training Clusters are built using a large number of PCIe devices including, GPUs, NICs, NVMe drives, and PCIe switches. It is important to implement a robust fault handling (RAS) solution within this PCIe device hierarchy to ensure target uptime, availability, and serviceability objectives. A high rate of PCIe correctable errors is expected. In this presentation, we would like to share our learnings and an innovative solution we developed to manage such large scale PCIe correctable errors within Meta AI/ML training clusters.