MLflow for AI Agents: The Open Source Tool Every Builder Should Know

🚀 In this video I walk you through MLflow for LLMs and AI Agents from absolute zero. By the end you will have your own observability stack running locally, tracing every agent call, scoring quality with LLM-as-a-judge, and versioning prompts like code. Free, open source, works with Claude, OpenAI, Gemini, and any other provider. ───────────────────────────────────── ⏱ CHAPTERS ───────────────────────────────────── 00:00 Intro and Goal 01:09 Local vs Databricks Plan 02:51 Prereqs and Setup 03:24 Install MLflow with UV 05:20 Start Local MLflow UI 06:19 Enable Claude Autologging 08:59 Fix Stop Hook Error 10:02 Traces Not Showing Debug 14:11 Restart Server See Traces 15:07 Explore Trace Details 16:24 Switch to Databricks Backend 17:24 Configure Databricks Autolog 20:35 Databricks Hook Profile Fix 22:09 Set Workspace Experiment Path 22:55 Fix Shared Experiment Path 23:31 Debug Hook Error Logs 24:07 Reauth Databricks CLI Profile 25:24 Claude Works and Traces Appear 26:44 Web Search Tool Trace 28:41 Analyze Traces Insights 30:11 Sync Traces to Unity Catalog 34:00 Verify Table and Sample Data 38:59 Why Unity Catalog and Genie 41:47 Recap and Wrap Up ───────────────────────────────────── 🔗 RESOURCES ───────────────────────────────────── ↗ MLflow website → https://mlflow.org/ ↗ MLflow GenAI docs → https://mlflow.org/docs/latest/genai/ ↗ Tracing quickstart → https://mlflow.org/docs/latest/genai/... ↗ Evaluation quickstart → https://mlflow.org/docs/latest/genai/... ↗ Prompt Registry docs → https://mlflow.org/docs/latest/genai/... ↗ AI Gateway docs → https://mlflow.org/docs/latest/genai/... ↗ Live demo → https://demo.mlflow.org/ ↗ MLflow GitHub → https://github.com/mlflow/mlflow ↗ MLflow integrations → https://mlflow.org/docs/latest/genai/... ↗ uv installer → https://docs.astral.sh/uv/getting-sta... ↗ Anthropic Claude → https://claude.ai ↗ Claude 101 Course → https://anthropic.skilljar.com/claude... --------------------------------------------------------------------- ☕ Buy me a Coffee: https://ko-fi.com/datasciencebasics ✌️ Patreon:   / datasciencebasics   ------------------------------------------------------------------------------------------ 🤝 Connect with me: 📺 Youtube:    / @datasciencebasics   👔 LinkedIn:   / sudarshan-koirala   🐦 Twitter:   / mesudarshan   🔉 Medium:   / sudarshan-koirala   💼 Consulting: https://topmate.io/sudarshan_koirala #mlflow #llmops #aiagents #datasciencebasics #aiobservability #llmtracing #promptengineering #claudeai #opensource #generativeai #datasciencebasics