Upcycling LLMs into MoE with Nvidia Researcher, Ethan He

Nvidia researcher, Ethan He, joins the Oxen Herd to give a deep dive into his co-authored paper, Upcycling Large Language Models into Mixture of Experts. -- Use Oxen AI 🐂 https://oxen.ai/ Oxen AI makes versioning your datasets as easy as versioning your code! Even is millions of unstructured images, the tool quickly handles any type of data so you can build cutting-edge AI. -- Paper 📜 https://arxiv.org/abs/2410.07524 Links + Notes 📝 https://www.oxen.ai/blog/how-upcyclin... Join Arxiv Dives 🤿 https://oxen.ai/community Discord 🗿   / discord   -- Chapters 0:00 Who is Ethan He 2:16 Ethan He Presents Upcycling LLMs 2:30 What is MoE 5:27 How Does the MoE Layer Work 8:52 How the Router Works 13:32 The auxiliary loss: Switch Transformers 14:55 Mixtral vs. Switch Transformer 18:50 The Takeaway 23:38 Plain Upcycling 28:44 Weight Scaling 32:55 Fine-Grained MoE 37:28 Fine-Grained MoE Upcycling 43:01 Experiments 43:50 The Importance of Learning Rate 45:48 Analysis of the Wave Similarity 49:43 Number of Experts 50:14 Large Scale Upcycling 55:10 Questions