KV Caching: Speeding up LLM Inference [Lecture]

This is a single lecture from a course. If you you like the material and want more context (e.g., the lectures that came before), check out the whole course: https://users.umiacs.umd.edu/~jbg/tea... (Including homeworks and reading.) I often refer to LLMs / Foundation Models / Frontier Models as "Muppet Models". Here's why:    • What general term should you use for model...   I got a free EdCafe subscription for adding it into these slides: https://www.edcafe.ai/ Music:   / review-and-rest  

Join Today
Deep Dive: Optimizing LLM inference
▶︎

Deep Dive: Optimizing LLM inference

KV Cache in 15 min
▶︎

KV Cache in 15 min

How to make LLMs fast: KV Caching, Speculative Decoding, and Multi-Query Attention | Cursor Team
▶︎

How to make LLMs fast: KV Caching, Speculative Decoding, and Multi-Query Attention | Cursor Team

Optimizing Models: Finetuning, Distillation, LoRA, and QLoRA[Lecture]
▶︎

Optimizing Models: Finetuning, Distillation, LoRA, and QLoRA[Lecture]

Attention in transformers, step-by-step | Deep Learning Chapter 6
▶︎

Attention in transformers, step-by-step | Deep Learning Chapter 6

Using DSPy for Prompt Optimization in Python: Example of Calibrating Quiz Bowl Questions [Lecture]
▶︎

Using DSPy for Prompt Optimization in Python: Example of Calibrating Quiz Bowl Questions [Lecture]

MIT 6.S191: Secrets of Massively Parallel Training
▶︎

MIT 6.S191: Secrets of Massively Parallel Training

The KV Cache: Memory Usage in Transformers
▶︎

The KV Cache: Memory Usage in Transformers

What is Prompt Caching? Optimize LLM Latency with AI Transformers
▶︎

What is Prompt Caching? Optimize LLM Latency with AI Transformers

LLM inference optimization: Architecture, KV cache and Flash attention
▶︎

LLM inference optimization: Architecture, KV cache and Flash attention

Key Value Cache from Scratch: The good side and the bad side
▶︎

Key Value Cache from Scratch: The good side and the bad side

KV Cache in LLMs Explained Visually | How LLMs Generate Tokens Faster
▶︎

KV Cache in LLMs Explained Visually | How LLMs Generate Tokens Faster

Mathe-News 🚨 KI löst das Erdős-Einheitsabstand-Problem!
▶︎

Mathe-News 🚨 KI löst das Erdős-Einheitsabstand-Problem!

Mastering LLM Inference Optimization From Theory to Cost Effective Deployment: Mark Moyou
▶︎

Mastering LLM Inference Optimization From Theory to Cost Effective Deployment: Mark Moyou

Andrej Karpathy: From Vibe Coding to Agentic Engineering w/ Stephanie Zhan
▶︎

Andrej Karpathy: From Vibe Coding to Agentic Engineering w/ Stephanie Zhan

🇩🇪 German industry JUST died (it’s WORSE than you think)
▶︎

🇩🇪 German industry JUST died (it’s WORSE than you think)

What is vLLM? Efficient AI Inference for Large Language Models
▶︎

What is vLLM? Efficient AI Inference for Large Language Models

KV Cache Crash Course
▶︎

KV Cache Crash Course

Faster LLMs: Accelerate Inference with Speculative Decoding
▶︎

Faster LLMs: Accelerate Inference with Speculative Decoding

How LLMs survive in low precision | Quantization Fundamentals
▶︎

How LLMs survive in low precision | Quantization Fundamentals