Mixture of Experts Part 2: Router Math, Top-K Gating & Switch Transformer

In Part 2 of this Mixture of Experts series, we go deeper into how MoE models actually work under the hood: router math, noisy top-k gating, expert load balancing, GShard, expert capacity, Switch Transformer, routing stability, and expert specialization. We cover: A short history of MoE models Router math: how the gate chooses experts Noisy Top-k gating and sparse activation Why routers collapse and how load balancing helps GShard and MoE inside Transformers Expert capacity and capacity factor Switch Transformer and top-1 routing Stability tricks like router z-loss What experts learn inside MoE models Why adding more experts helps, then flattens Based on the Hugging Face article: “Mixture of Experts Explained”. link: https://huggingface.co/blog/moe #MixtureOfExperts #MoE #Transformers