IsoSci Benchmark: 63 of 69 LLM Reasoning Gains Never Transfer

Same two models, a 44-point swing OpenAI's reasoning model beats its ordinary sibling by nineteen points on one science benchmark — and loses by twenty-five on another covering the same sciences. A new paper from Texas A&M explains the reversal with a simple counting trick: build twin problems with identical logic but zero shared facts, and watch whether reasoning gains travel between them. More than nine in ten don't — suggesting the industry's expensive 'reasoning premium' may mostly be buying a longer sweep of the model's memory, not better logic. ▶ Watch next: How Teaching an AI to Predict, Not Act, Made It a Better Actor    • Language World Models: predicting environm...   Full episode page: https://paperdive.ai/episodes/197-iso... Paper: IsoSci: A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs Authors: Abdaljalil, Serpedin, Kurban Read the paper: https://arxiv.org/abs/2607.01431 What you'll take away: Why every science benchmark fuses two separate skills — knowing facts and executing procedure — making a gain in fact-fishing indistinguishable from a gain in logic The twin-problem trick: 144 problem pairs with identical solution steps but zero shared knowledge, letting you test whether an improvement travels with the logic or stays with the facts Across five model pairs, 63 of 69 reasoning-mode gains were one-sided — over nine in ten stayed with the facts, though the authors flag this as a ceiling, not an exact figure The cleanest experiment in the paper: toggling reasoning on the same model was a statistical wash (helped 8 items, hurt 9 on Gemini 2.0 Flash), suggesting visible extended thinking bought nothing on short procedural problems Where the paper is soft: a contamination asymmetry between seen and fresh twins, 23% of API calls excluded due to token-cap truncation, and a mostly multiple-choice format all make the dramatic 25-point reversal attackable What this doesn't cover — twenty-step derivations and open-ended problems, where the GPQA result hints extended reasoning may still earn its keep Chapters: 0:00 One model, two tests, opposite verdicts 2:03 Why benchmarks can't see what improved 2:53 The chicken-and-mushroom trick behind IsoSci 6:08 How to catch a cheat sheet 9:07 Sixty-three to six 9:54 Same model, reasoning on: coin flip 11:03 Sprinter versus marathoner: the paradox dissolves 12:31 The loudest number is the softest This episode is AI-generated. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. The on-screen illustrations were generated by OpenAI GPT Image and Google Gemini. #AI #MachineLearning #ChainOfThought