AI-Powered Root Cause Analysis at Scale: From Theory To Production... Letícia Mota & Yevgeny Gladun

Don't miss out! Join us at our next KubeCon + CloudNativeCon events in Mumbai, India (18-19 June, 2026), Yokohama, Japan (29-30 July, 2026), and Shanghai, China (8-9 September, 2026). Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at https://kubecon.io AI-Powered Root Cause Analysis at Scale: From Theory To Production Lessons From Nubank's 120M+ Cus - Letícia Mota & Yevgeny Gladun, Nubank This session presents an AI-powered SRE Agent designed to autonomously orchestrates complex, multi-source investigations by querying internal observability providers and knowledge bases. A primary focus is the "Data Volume Problem." Modern observability systems generate terabytes of metrics and logs daily; at Nubank’s scale, the Prometheus MCP alone has more than 23,000 metrics available, while log queries can span billions of rows. The team overcame LLM context limits through on-premises data filtering, intelligent summarization, and selective context assembly. This architecture utilizes "Expert Guides" to reduce 23,000 raw metrics to approximately 14 relevant data points before LLM processing. The talk covers multi-source orchestration using the Model Context Protocol (MCP) for pluggable tool discovery, allowing the AI to progressively load and correlate only the observability sources. The platform enables the delivery of expert instructions for any specific scenario through targeted, versioned prompts. This transformation allows the platform to scale across the enterprise, performing virtually any investigative task beyond its original root cause analysis mission.