Eval-Driven Development: The Fine Line Between Agentic Success and Failure | Urs Peter

Recording brought to you by American Express. https://americanexpress.io/kotlin-jobs Agentic systems unlock capabilities that traditional “deterministic” applications simply can’t deliver. But there’s a catch: their probabilistic nature introduces real and often unexpected risks—hallucinations, context drift, prompt degradation, and multi-step workflows that evolve in surprising ways. And you definitely don’t want to discover those in production. The good news: there is a remedy. We can borrow the most reliable practice from deterministic software—test-driven development—and adapt it to the agentic world. The result is Eval-Driven Development (EDD): a systematic, engineering-first approach to bringing reliability into inherently probabilistic systems. In this talk, we explore Eval-Driven Agentic Development and how it can transform your agents from unpredictable to reliable. We’ll dive into the techniques, tools, and patterns needed to make evaluation a first-class citizen of your development process—demonstrating all of these in a real-world application built with the powerful AI framework: Koog. Along the way, you will learn how to: Test agents at multiple layers—schema validation, tool correctness, decision flows, and end-to-end goal completion Gather metrics that serve as the backbone of meaningful evaluations Turn complex agent traces into actionable insight rather than noise Use LLMs to generate test cases, assertions, synthetic data, and act as “judges” in your eval pipeline Detect regressions when prompts, models, or data change Build a continuous evaluation loop with real-world data—so your agents improve over time instead of quietly degrading Evaluation-Driven Development is the only path forward for reliable, smart next-generation applications. Let's learn how to walk it - together!