Ray Agent Engine: Deploying AI Agents with Ray Serve | Ray Summit 2025

At Ray Summit 2025, Juncheng Ma, Deepak Chandramouli, and Bhumik Thakkar from Apple share how Ray Serve can be used as a powerful, framework-agnostic foundation for deploying the next generation of AI agents at scale. They begin by outlining the growing complexity of modern AI agents—spanning multi-step reasoning, tool use, memory, planning, and dynamic interaction patterns—and the corresponding challenges in deploying these systems reliably in production. Traditional serving frameworks often assume static inference graphs, making them ill-suited for agentic workloads that require flexible orchestration and adaptive execution. The speakers then walk through how Apple leverages Ray as an Agent Engine, combining Ray Serve’s distributed execution model with its built-in autoscaling, request routing, and traffic management capabilities. This approach enables: Framework-agnostic agent deployment, allowing teams to integrate agents built using any architecture or library Dynamic, scalable execution for agents that require multi-step reasoning and complex control flows High robustness and resiliency, even under unpredictable load patterns Simplified operational workflows, reducing the burden of managing complex agent pipelines They share practical lessons learned from deploying agentic systems internally at Apple, highlighting patterns that generalize to a wide variety of agent frameworks and application domains. Attendees will leave with actionable insights on how to build scalable, resilient, and production-ready agent applications using Ray Serve—regardless of the underlying agent design or development workflow. Liked this video? Check out other Ray Summit breakout session recordings    • Ray Summit 2025 - Breakout Sessions   Subscribe to our YouTube channel to stay up-to-date on the future of AI!    / anyscale   🔗 Connect with us: LinkedIn:   / joinanyscale   X: https://x.com/anyscalecompute Website: https://www.anyscale.com/