Масштабирование LLM упёрлось в предел: исследование MIT

In this video, I analyze a groundbreaking study—awarded by a team from MIT at NeurlIPS (the world's top conference on neural networks and AI)—that reveals a fundamental limitation of modern LLMs: the problem isn't with the architecture or the data, but with the low dimensionality of the embeddings. When the vector space isn't large enough, the semantic representations of different words begin to overlap, creating collisions that distort the model's understanding. It's this limitation, not a lack of parameters or data, that sets a hard limit on scaling: simply adding more parameters is pointless—the model will "confuse" concepts at the representation level. I show why the industry is running into a mathematical barrier and what research suggests as a way around it. Article: https://arxiv.org/pdf/2505.10465 Article from Anthropic: https://arxiv.org/pdf/2209.10652 🤓Interactive database of interview questions - https://botayinterview.site 💻More useful information on my Telegram channel - https://t.me/Ai_bolno_ml 💐Consultation, interview preparation - https://filonov.site TIMECODES: 00:00 - Start 00:41 - How much does it cost to launch LLM yourself 01:50 - How much does ChatGPT-5 cost 02:14 - Costs of large AI companies 02:42 - Scaling laws 03:04 - OpenAI formula 03:58 - OpenAI and Chinchilla error 04:46 - MIT article - prize-winning NeurlIPS 05:38 - New Problem: Vector Dimensions 06:01 - What Are Text Embeddings 07:05 - The Dimensionality Problem 07:34 - Words Just Overlap 08:05 - The Main Point of the Research 10:07 - Bonus