From Pull To Predict: Accelerating AI Model Deployment on Kubernetes - Lucas Duarte & Tiago Reichert

Don't miss out! Join us at our next Flagship Conference: KubeCon + CloudNativeCon events in Amsterdam, The Netherlands (23-26 March, 2026). Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at https://kubecon.io From Pull To Predict: Accelerating AI Model Deployment on Kubernetes - Lucas Duarte & Tiago Reichert, AWS In the era of large AI models, deployment latency and resource utilization present significant challenges for Kubernetes operators. This session demonstrates techniques to reduce model startup times and optimize cluster resources. We'll deploy a 7B parameter LLM using Ray and vLLM for scaling and serving, implementing three key optimizations: SOCI (Seekable OCI) for lazy loading of container images, enabling containers to start without downloading the entire image first; an optimized storage layer that keeps models pre-downloaded and ready for quick access; and intelligent node provisioning using Karpenter for dynamic resource allocation. We'll compare a standard deployment against one using these optimizations, showing the differences in startup times, resource usage, and operational costs. Attendees will learn implementation steps for these techniques, which they can apply to their own Kubernetes environments to improve AI model deployment efficiency.