The Ironies of AI in Incidents: What 99% of Orgs Get WRONG

When your software stack is on fire, can you actually trust AI to fix it? In this InfoQ video, J. Paul Reed exposes how unbridled AI enthusiasm is quietly creating more severe, longer-lasting system outages. As engineering leaders, we are racing to automate everything - but we are ignoring a 40-year-old warning. Drawing from Dr. Erik Hollnagel’s Efficiency-Thoroughness Trade-Off (ETTO) and Mica Endsley’s foundational research on situational awareness, J. Paul Reed breaks down the "Ironies of AI and Automation." Through real-world case studies (including an AI code-porting disaster that extended an incident by 300%), Reed explains why relying on AI recommendations during high-tempo, high-consequence incidents erodes critical human mental models, camouflages system states, and actively degrades human performance. Learn how to design robust human-AI joint cognitive systems that protect your system's resilience when things inevitably break. ⏱️ Video Timestamps (For Navigation) 0:00 - The Ironies of AI Squared ($AI^2$) 01:45 - 1983 Warnings: Bainbridge’s Ironies of Automation 03:15 - How Automation Camouflages System Failures (The Autopilot Problem) 05:02 - Connectivity vs. Coordination in Joint Cognitive Systems 06:54 - The Ironies of AI: Why ML Fails in Novel Situations 09:12 - Incident Story Time: When Claude Agents Go Rogue 11:05 - Case Study: How AI Code Generation Multiplied an Incident by 3x 13:10 - The ETTO Principle: Why You Already Lost the Efficiency Bet 14:45 - Data Breakdown: Does AI Actually Degrade Human Performance? 17:30 - Actionable Strategies for Incident Commanders & Architects 🔗 Transcript available on InfoQ: https://bit.ly/4e6ani8 #SRE #SoftwareArchitecture #ArtificialIntelligence #IncidentResponse