Everything you need to know about LLM benchmarks. (and why they're flawed), OpenAI's Healthbench

Whenever there was AI, there were benchmarks- from the turing test, to society-changing benchmarks like MNIST and ImageNet to modern problems like the ARC prize, benchmarks served a vital purpose to measure the performance of AI models. But something has shifted in modern times, in the LLM era have benchmarks lost their utility, becoming mere advertisement for big tech? Even seemingly more sophisticated benchmarks like LM Arena can be gamed by tech giants. We also deep dive into healthcare benchmarks like OpenAI's Healthbench (deeply problematic) and microsofts AI DXO orchestrator agent for diagnosis. Where is this all going? How do we make the perfect benchmark? Or is the real work to be done afterwards in the real world? 👋 Hey! If you are enjoying our conversations, reach out, share your thoughts and journey with us. Don't forget to subscribe whilst you're here :) Timestamps 00:00 intro - The OG benchmarks - Turing test, MNIST , ImageNET 06:40 are large language models benchmarks similar to humans taking tests? 10:05 Are we testing model capability vs production ready? 12:00 LLM era - data contamination 15:30 LM arena - The leaderboard illusion paper- how big tech games benchmarks 28:35 Goodhart's law- When a measure becomes a target, it ceases to be a good measure 32:05 some good benchmarks - games- Pokemon , ARC prize, minecraft 34:35 Medical benchmarks - OpenAI's healthbench has some big problems 46:50 microsoft AI-DXO orchestrator for case reports 👨🏻‍⚕️Doc - Dr. Joshua Au Yeung -   / dr-joshua-auyeung   🤖Dev - Zeljko Kraljevic   / zeljkokr   References Rethinking benchmarks , data contamination paper - https://arxiv.org/pdf/2311.04850 leaderboard illusion - https://arxiv.org/pdf/2504.20879 openAI's healthbench subanalysis   / a-closer-look-at-openais-new-healthbench-e...   microsoft - towards sequential ddx https://arxiv.org/pdf/2506.22405 YT -    / @devanddoc   Spotify - https://podcasters.spotify.com/pod/sh... Apple- https://podcasts.apple.com/gb/podcast... Substack- https://aiforhealthcare.substack.com/ For enquiries - 📧[email protected] 🎞️ Editor- Dragan Kraljević   / dragan_kraljevic   🎨Brand design and art direction - Ana Grigorovici https://www.behance.net/anagrigorovic...