86% of Biased AI Analyses Passed Review — Same Data, Opposite Findings

One belief paragraph flips scientific conclusions One paragraph stating a political belief was enough to make AI analysts reach opposite conclusions from identical data — and 86% of those biased analyses passed hostile expert review, because nothing in any single report was actually wrong. A Stanford team's fix is a new statistic, a sibling of the p-value, that measures whether a finding was fished from the extreme edge of everything the data could have said. On its first deployment, the instrument built to catch AI bias caught the humans instead. ▶ Watch next: An AI Designed Its Own Psychology Studies, Then Confirmed What It Found    • AutoCog AI rediscovered prospect theory — ...   Full episode page: https://paperdive.ai/episodes/196-the... Paper: The Agentic Garden of Forking Paths Authors: Miao, Pritchard, Zou Read the paper: https://arxiv.org/abs/2607.01507 What you'll take away: How a single persona paragraph made coding agents reproduce 72% of the ideological gap found across 42 human research teams — with a claimed-significance gap nearly 9x the human one Why peer review structurally can't catch this bias: 86% of analyses passed a cross-model AI audit and 78% passed blinded human PhD statisticians, with skeptics' work exactly as clean as believers' The two mechanisms visible for the first time in agent logs — exploration bias and selection bias — including two agents reading the same negative estimate as 'evidence' vs 'a flaw to fix' The m-value: the p-value's mirror statistic, measuring how often re-running the analysis (not re-collecting the data) would produce a result that extreme — and why analyst choice moved answers 2.8x more than noise What happened when the instrument was pointed at the human teams: 40% of their statistically significant results sat in the most extreme 5% of the analysis space The steelman that survives the episode: extreme is not the same as wrong — the m-value measures typicality, not quality, and can't distinguish a grandmaster's move from a fished result in any single study Chapters: 0:00 Forty accountants, forty answers, all legal 1:43 Can one paragraph bend a rigorous analysis? 4:21 Why hostile review couldn't find the bias 6:14 Watching bias enter, decision by decision 9:04 4,400 defensible answers to one question 10:13 The p-value's missing sibling 13:10 The instrument turns on the humans 14:46 But extreme isn't the same as wrong This episode is AI-generated. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. The on-screen illustrations were generated by OpenAI GPT Image and Google Gemini. #AI #MachineLearning #PersonaPrompting