Concurrent AI Evaluation: Scaling Model Performance Monitoring with OTP- Ramiro Matteoda | Code BEAM

✨ This talk was recorded at Code BEAM Europe in November 2025. If you're curious about our upcoming event, check https://codebeameurope.com/✨ --- Evaluating model performance remains a significant challenge in the rapidly evolving AI landscape. Traditional evaluation approaches often struggle with scale, consistency, and real-time feedback integration—precisely the problems that Elixir and the BEAM were designed to solve. We'll explore: Live Benchmarking Pipelines: Implementing resilient GenServers and dynamic supervision trees that continuously process evaluation data at scale. Concurrent Prompt Evaluation: Building distributed worker pools that can evaluate thousands of prompt variations across multiple LLM providers. - Systematic Human-in-the-Loop Automation: Designing resilient feedback processing pipelines using GenStage and Broadway that validate human annotations, detect inconsistent labelers, and automatically route corrections into training loops. LangChain Integration: Implementing "LLM-as-a-judge" evaluation patterns using LangChain to create sophisticated, criteria-based evaluations of AI outputs with minimal code overhead. This talk demonstrates why Elixir's unique strengths make it the ideal platform for building AI evaluation systems that scale from prototype to production. --- Let's keep in touch! Follow us on: 💥 Bluesky: / codebeam.bsky.social 💥 Twitter: / codebeamio 💥 LinkedIn: / code-sync