Reverse Engineering Large Language Models | Build Your Own LLM Workshop #2 [Refreshed]

Reverse Engineering LLMs: print(), summary(), and a Roadmap to Building GPT-2-Style Transformers. Part of a Build your own LLM workshop. =========== LINKS Justin's twitter: https://x.com/JustinAngel Workshop overview: https://go.justinangel.ai/substack Deck: https://go.justinangel.ai/deck Google Drive: https://go.justinangel.ai/drive Code exercise: https://go.justinangel.ai/code-2 =========== CHAPTERS 00:00 Workshop Intro 01:14 Load GPT2 in Colab 01:28 Model Size and Params 02:14 Print and Summary 03:49 Module Tree Breakdown 04:39 Roadmap from Components 05:34 Function Call Graph 06:43 Hugging Face Exercise 08:48 Ablation Tests Explained 10:10 LLMs Grown Not Built 11:28 Full Roadmap and Deliverables 13:13 Interactive Demos and Models 14:52 Next Up Perceptrons =============== ABOUT THIS TALK In this episode of the Build Your Own Large Language Model workshop, the presenter begins Section 2, “Reverse Engineering LLMs,” using GPT-2 in a Colab notebook to extract an inference-time roadmap by calling print(model) and torchinfo summary. They note GPT2-small has about 124M parameters and show how the architecture reveals repeated GPT2 blocks (12 layers) and components like embeddings, dropout, LayerNorm, Conv1D/linear layers, attention, MLPs, and Gelu, plus function-level elements like initialization and softmax. They demonstrate repeating the process on a different Hugging Face model (TinyLlama/LlamaModel), highlighting shared parts and differences like RMSNorm and rotary embeddings, and encourage viewers to try other models. The episode introduces ablation tests as “taking away” components to see what works, shares quotes about LLMs being “grown,” and previews upcoming sections starting with perceptrons, activations, MLPs, and training fundamentals like loss and backprop.