LLM Inference Optimisation Explained — GQA, RoPE, FlashAttention & MoE Intermediate Level

The clean Transformer works but it's slow and memory-hungry at scale. This intermediate tier covers the tricks that make real LLMs practical: grouped-query attention, rotary positions (RoPE), FlashAttention, Mixture-of-Experts, the KV cache memory math, PagedAttention, and sliding-window long context. The unifying theme: at inference, attention is memory-bound, not compute-bound. Every technique here either shrinks the bytes you move or moves them less. *What you'll learn* Why decoding is memory-bandwidth bound (the roofline view) MHA vs MQA vs GQA and how each sizes the KV cache RoPE: encoding position by rotation FlashAttention: exact attention, IO-aware tiling Mixture-of-Experts and load balancing KV cache math and PagedAttention (vLLM) Sliding-window / local attention