Production Engineering When Trading Billions of Dollars a Day

What's it like to monitor and operate software that trades billions of dollars a day on stock markets around the world? When your software has near-unlimited access to your bank account, every single message counts. Speedy alerting and incident response have a direct and measurable impact on the P&L. In this talk, Mark Doss, a production engineer at Jane Street, explores the day-to-day technical operations of a trading firm, with a heavy focus on what happens when things go wrong. He covers the unique features of the trading environment that make production engineering especially high-stakes, how Jane Street approaches monitoring and alerting (and why traditional SLO-based approaches often fall short), the role of defense in depth and cross-team communication, and walks through a realistic sample incident to make it all concrete. 00:00 Video starts 00:05 About Mark Doss 00:27 Intro and Talk Outline 05:29 Features of the production environment 17:07 What it means for Production Engineering at Jane Street 37:22 Sample Incident 49:26 Summary of Production Engineering at Jane Street 51:17 Q+A