CrowdStrike Outage Explained: The 78 Minutes That Blue-Screened 8.5 Million PCs

04:09 UTC, Friday 19 July 2024. A file of a few kilobytes of configuration data leaves a server and fans out across the planet. Within minutes, airport departure boards go black, hospital systems die mid-shift, and around 8.5 million Windows machines crash and refuse to come back. No hacker. No virus. The largest IT outage in history was an accident. Episode 1 of When Everything Went Down. CrowdStrike Falcon runs partly inside the Windows kernel - the engine room, where code is trusted absolutely because there is nothing above it left to do the checking. Channel File 291 asked the sensor for a twenty-first field of data in a world built for twenty. The sensor reached for a value that had never been filled in, used it as an address in memory, and went there. Windows did the only safe thing it could: it killed the machine. Then the machine rebooted, read the same file, and died again. In this episode: what endpoint protection is and why it replaced list-based antivirus; user mode versus kernel mode, and why security vendors shipped kernel drivers at all; the deal every customer had signed, with files landing on their machines all day without anyone approving them one by one; sensor 7.11, released on 28 February 2024 with a new template for detecting attacks that abuse Windows named pipes; the twenty versus twenty-one mismatch, and the wildcard that meant nobody ever asked the twenty-first field a real question; the 78 minutes between release and rollback, and exactly which machines were hit; the day itself - ground stops at American, United and Delta, more than 5,000 flights cancelled, a later study finding 759 of 2,232 US hospitals with a detectable disruption to digital services, 911 lines down in several states, and the carriers whose older systems never received the file at all; the repair, done by hand, in Safe Mode, with BitLocker recovery keys sitting on servers that were also blue-screened; the Parametrix estimate of about 5.4 billion dollars in direct losses for the US Fortune 500 alone, and what the insurers expected to cover; the August root cause analysis, where everything anybody tested passed honestly while the one condition that mattered was never created; the September hearing before the US Congress; the lawsuits between Delta and CrowdStrike, which remain claims by each side; and Microsoft Windows Resiliency Initiative, the attempt to get security software out of the kernel for good. The lesson underneath all of it is monoculture. Thousands of critical organisations, one vendor, one kernel driver, one update pipeline, one instant. Efficiency and safety pulled in opposite directions, and efficiency had been winning for years. And the answer is not clever, it is boring: roll changes out in stages to small groups first, let the people who depend on you choose when to take them, check the edge of the array, and keep the most dangerous code out of the most dangerous room. Full episode: https://empire-ops.com/niels - Next Friday on When Everything Went Down: October 2021, the six hours when Facebook, Instagram and WhatsApp vanished from the internet. Follow the channel so you do not miss it. More from the channel: https://empire-ops.com/niels Sources: the CrowdStrike External Technical Root Cause Analysis of Channel File 291, the company Preliminary Post Incident Report and blog; the CISA alert of 19 July 2024, including its warning about fake CrowdStrike fix domains; the Microsoft estimate of about 8.5 million affected devices, the Microsoft recovery tool of 20 July and the Windows Resiliency Initiative announcement of November 2024; the Parametrix and CyberCube loss estimates; the written testimony of Adam Meyers to the US House Subcommittee on Cybersecurity and Infrastructure Protection in September 2024; Delta own public account and court filing; reporting by TechTarget, CIO Dive, Cybersecurity Dive, SecurityWeek and The Register. Every damage figure is an estimate and is attributed to the firm that produced it, and the Delta and CrowdStrike claims are described as allegations by each side. #CrowdStrike #BlueScreen #CyberSecurity #ITOutage #TechHistory