Deep dive into Mixture of Experts (MOE) with the Mixtral 8x7B paper

Arxiv Dives is a group from Oxen.ai of engineers, researchers, and practitioners that gets together every Friday to dig into state of the art research that relates to Machine Learning and Artificial Intelligence. If you would like to join the live discussion we would love to have you! Join here: https://lu.ma/oxenbookclub Each week we dive deep into a topic in ML/AI. Whether it is a research paper, a blog post, a book, or a YouTube video, we break down the content into a digestible format and have an open discussion with the Oxen.ai team, and anyone else who wants to join. We try to cover the content as high level so that anyone can understand it, and will dive into deeper technical details to get a clearer understanding. This week we cover the Mixtral paper from the team at Mistral.ai. This paper goes over how Mistral used Mixture of Experts (MOE) in their latest Mistral-8x7B-instruct-1.0 paper to achieve better performance than larger models as well as competitive performance with GPT-3.5. All the notes and previous dives can all be found on the Oxen.ai blog: https://blog.oxen.ai/tag/arxiv-dives/