Groq Head of Evals: How to Actually Make RAG & Agents Fast

If you're building AI agents that take forever to respond, this talk is for you. Aarush Sah, who leads evals at Groq, breaks down practical strategies to dramatically reduce agent latency without sacrificing quality, and not just by switching to faster models. You'll learn how to measure what matters (TTFT, tokens per second, step latency, and end-to-end latency), implement parallelism for multi-step workflows, stream intermediate steps to improve user experience, and reinvest speed gains into better reasoning. Aarush demonstrates these concepts with real examples showing how a 45-second agent workflow can be cut to seconds. if you want to learn more about improving rag applications check out https://improvingrag.com/ TIME STAMPS 00:00 Introduction and Key Takeaway 02:14 Understanding Latency Metrics 05:05 Demo: Compound Beta Mini vs. Perplexity Sonar 08:02 Complex Agent Example: Travel Planning 10:15 Optimization Strategies for Reducing Latency 17:36 Q&A: Real-World Applications and Tools 26:49 Optimizing UI for Faster Responses 28:55 Outcome-Based Pricing Models 30:30 Evaluating Subjective Criteria 37:21 Optimizing Models for Specific Use Cases 41:41 Impact of Fast Inference on Evaluations 44:23 Real-Time Intelligent Co-Pilots 46:35 Latency in Production Systems