Kimi K3’s 2.8 Trillion Parameter Trick — Only 16 Experts Wake Up

Kimi K3 contains almost 2.8 trillion parameters—but for each token, only 16 out of 896 experts are activated. That apparent contradiction is the key to understanding its architecture. In this video, we explore how Kimi K3 combines enormous model capacity with selective computation, efficient communication, long-context memory, and improved information flow. You’ll learn how four important architectural ideas work together: • Mixture of Experts — activating only a small group of experts for each token • Stable Latent MoE — reducing communication between GPUs and stabilizing expert routing • Kimi Delta Attention — processing extremely long contexts more efficiently • Attention Residuals — allowing deeper layers to retrieve useful earlier representations This is not simply a story about building a larger AI model. It is a story about selective scale: giving a model access to enormous capacity while teaching it when not to use it. CHAPTERS 00:00 — The 2.8-trillion-parameter paradox 01:00 — Why full activation is too expensive 01:16 — Mixture of Experts explained 02:17 — Why experts must be distributed across GPUs 02:52 — Stable Latent MoE 03:25 — Expert routing and quantile balancing 04:33 — Why attention becomes expensive 05:18 — Kimi Delta Attention 05:48 — A more efficient kind of memory 07:18 — Why deep models can lose information 07:58 — Attention Residuals 08:56 — The bigger picture 10:02 — Why selective scale matters If you enjoy clear, visual explanations of AI systems, model architectures, and software engineering, subscribe for more. What architecture should I explain next? Let me know in the comments. #kimik3 #artificialintelligence #machinelearning #mixtureofexperts #aiengineering #deeplearning #llm