Have you ever wondered how state of the art large language models work under the hood? This talk will dive deep on the mixture of experts (MoE) architecture used by the recently released Mixtral 8x7b model (and many of the other top performing closed source models). We will cover large language model (LLM) fundamentals, how this architecture differs from typical LLMs, and even how you can fine-tune MoE models for your own tasks!
Bio: Isaac Privitera is a Senior Data Scientist in Amazon Web Services’ Generative AI Innovation Center. As part of the GenAIIC he develops bespoke generative AI based solutions to address customers’ business problems. He works primarily on building responsible AI systems using fine-tuning, retrieval augmented generation (RAG), and chain of thought reasoning with large language models. In his spare time he enjoys golf, football, and walking with his dog Barry.
