Lessons From Scaling BPF To Detect RDMA Device Drivers Bugs In Real Time

Abstract Training large models requires significant resources, and failure of any GPU or host can significantly prolong training times. At Meta, we observed that 17% of our jobs fail due to RDMA-related syscall errors, which arise due to bugs in the RDMA driver code. Unlike other parts of the kernel, RDMA-related syscalls are opaque, and the errors create a mismatched application/kernel view of hardware resources. As a result of this opacity and mismatch, existing observability tools provided limited visibility and DevOps found it challenging to triage – we required a new scalable framework to analyze kernel state and identify the cause of this mismatch. Direct approaches like tracing the kernel calls and capturing metadata involved in the systems turned out to be prohibitively expensive. In this talk, we will describe the set of optimizations used to scale tracking kernel state and the map-based systems designed to efficiently export relevant state without impacting production workloads. Prankur Gupta: Prankur Gupta is a Staff Software Engineer at Meta with over 12 years of experience driving reliability and observability at scale. His expertise spans the entire technology stack, from developing AI transport protocols in NIC firmware and kernel drivers to pioneering network optimizations in user space using eBPF. At Meta, Prankur has played a key role in building advanced ecosystems for transport tuning and congestion control, delivering major performance gains. He currently leads the productionization of Meta’s in-house hardware initiatives—including NICs and MTIA—while enabling the next generation of AI transport protocols for Meta’s infrastructure. https://nanog.org/events/nanog-97/con...