How to Reduce LLM Latency

Join the AI Evals September 2026 cohort: https://maven.com/parlance-labs/evals... Most people assume a slow LLM just means too many tokens. But the same model, on the same GPU, running the same code can be 12x slower because of inference physics. Abi teaches LLM inference and inference engineering, and leads research in distributed systems for high-performance computing. In this session, she builds a mental model for inference time from scratch, the four variables that define every request, why prefill and decode behave nothing alike, and why decode (not your retrieval pipeline) is where your latency and cost actually live. Timestamps: 00:00:00 Introduction 00:00:15 Same model, same GPU — 12x slower 00:01:50 The four variables of an inference "shape" 00:02:34 Four workload patterns: chat, RAG, creative, agents 00:03:45 Prefill vs decode explained 00:07:14 Live notebook: where the time actually goes 00:15:49 Why decode dominates latency 00:17:05 Writing a token costs ~300x more than reading one 00:18:07 Why small requests starve the GPU 00:19:43 KV cache: smaller than you'd think 00:20:49 Agents: re-reading history every step 00:25:03 Why a 5-step agent costs 12x more 00:29:16 Summary: five rules for latency and cost 00:30:51 Homework to try yourself 00:33:34 Q&A: mistakes, hosting, speculative decoding Connect with Hamel: Website ► https://hamel.dev LinkedIn ►   / hamelhusain   Twitter/X ► https://x.com/hamelhusain Instagram ►   / hamelsmu   Tik Tok ►   / hamel_husain   Connect with Abi: Website ► https://topmate.io/goabiaryan LinkedIn ►   / goabiaryan