ElixirConf 2023 - Anthony Accomazzo - Building recursive workflows with Broadway

Leave comments at: https://elixirforum.com/t/elixirconf-... Broadway makes processing streams of data easy. A producer pulls work from a queue like SQS, fans the work out to workers, and handles lifecycle events like acknowledgments. But what about situations where the work to be done can’t be modeled or built on top of a message queue? At Sequin, we needed a way to process high volumes of requests to crawl APIs. We wanted to model after the producer/consumer architecture of Broadway. But in operations like crawling, work is created iteratively as work is completed—unlike a classic, queue-centric data pipeline. In this talk, we’ll discuss Broadway, building custom producers, how to model concurrency in your pipelines, and tips to keep your pipeline fast.