Activation Functions: ReLU, GELU | Build Your Own LLM Workshop #4

Activation Functions for LLMs (in Excel): ReLU, ReLU², GELU, Leaky ReLU, and Why Nonlinearity Matters. Part of a Build your own LLM workshop. =========== LINKS Justin's twitter: https://x.com/JustinAngel Workshop overview: https://go.justinangel.ai/substack Deck: https://go.justinangel.ai/deck Google Drive: https://go.justinangel.ai/drive Excel exercise: https://go.justinangel.ai/excel-4 ============== CHAPTERS 00:00 Workshop Intro 01:18 Why Nonlinearity Matters 03:26 ReLU Basics Explained 06:26 ReLU Demo Walkthrough 09:29 Modern Activations Beyond ReLU 11:27 Activation Function Buffet 12:27 Excel ReLU Visualizations 15:54 ReLU Squared In Practice 17:06 Leaky ReLU And GELU 20:17 Comparing Many Functions 21:54 Why Choose ReLU Squared 24:07 Lego Builder And Wrap Up =============== ABOUT THIS TALK In this Build Your Own Large Language Model workshop, Justin Angel introduces activation functions as the source of nonlinearity needed for perceptrons to model real-world “edges” and stop/start thresholds, including examples like edge detection and absolute value (and sentiment intensity in text). Using only Excel (no code yet), he explains ReLU (max(0, x)), why logits often fall between about −4 and 4 (standard deviations), and demonstrates how ReLU blocks negative outputs while bias/weight changes shift when it activates. He notes modern LLMs rarely use ReLU now, with GLU/SwiGLU and GELU more common, and shows ReLU², Leaky ReLU, and GELU behaviors via charts and 2D/3D “landscapes,” emphasizing that activation choice is empirical. He adopts ReLU² for a small GPT-2-style model based on Karpathy’s results and previews a later section on GPU-oriented code implementations and a “lego” tool for generating pretraining scripts.