ArgoCon '21 - Dynamic, Event-Driven Machine Learning Pipelines with Argo Workflows (Ian McGraw)

Most people would say Argo is a YAML-based orchestration framework. But really, it's a highly expressive YAML-based API. Recently at Arthur, we've harnessed Argo to build highly dynamic, event-driven workflows using the Golang Argo Client and the Kubernetes API. Using this setup, we can create 1000's of variations of the same workflow in seconds to configure multiple machine learning pipelines unique to each of our customers. These pipelines stand up dynamic infrastructure, build and deploy containers, and set up autoscaling resources to implement flexible, scalable, and cost-effective big data processing. In this talk, we will describe the journey that lead us to Argo and some of the road bumps we had with competitors along the way. In addition, we will describe how we use GO to build and run event-driven, dynamic workflows that power our machine learning as a service capability. By the end of this talk, listeners will learn how Argo not only stands above the competition for static, scheduled workloads but continues to power the most cutting-edge, event-driven machine learning capabilities.