Autoformalization Agents: AI Referee Catches a Gap in a STOC Proof

An off-the-shelf coding agent on a $200-a-month subscription read a proof that already cleared peer review at one of theory's most selective conferences, tried to make it machine-checkable, and got stuck on one line that turns out not to follow — handing back a counterexample you can check by hand. The trick is treating a math proof like a software project, with data types, unit tests, and a manager who can order a rewrite. You'll come away seeing why the real bottleneck in math is no longer writing proofs but trusting them. Full episode page: https://paperdive.ai/episodes/188-bey... Paper: Beyond the Library: An Agentic Framework for Autoformalizing Research Mathematics Authors: Moakhar, Gholami, Springer et al. Read the paper: https://arxiv.org/abs/2606.31134 What you'll take away: Why general-purpose coding models have quietly overtaken specialist Lean-tuned models at formalizing math The reframe at the heart of the paper: treat a proof like software, with types, unit-test lemmas, and an orchestrator that can backtrack and refactor How the system caught a genuine gap in a 2025 STOC proof — and returned a hand-checkable counterexample of one triangle and ten dots Why the '$5 per problem' and '91%' headline numbers are softer than they sound — subscription pricing arbitrage and a statistical floor from just 32 problems Where the guarantee actually lives: the Lean kernel proves the proof, but only AI judgment guarantees the formal statement means what the paper said The reframing of AI-for-math from theorem discovery to tireless, literal-minded refereeing Chapters: 0:00 Why trusting a proof is the new bottleneck 1:41 The escape hatch that can't be argued with 2:33 Two walls that block autoformalization 4:16 What if a proof were a software project? 6:31 Proving the parent before the children 10:14 The line the machine couldn't force 12:32 Green nodes, orange nodes, and honesty 13:49 The numbers that oversell themselves 15:58 The compiler proves; only AI judges meaning 18:03 Would you trust the machine referee? This episode is AI-generated. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. The on-screen illustrations were generated by OpenAI GPT Image.