Ins and Outs of the Outbox Pattern, Gunnar Morling

The outbox pattern is a common solution for implementing data flows between microservices. By channeling messages through an outbox table, it enables services to update their own local datastore and at the same time send out notifications to other services via data streaming platforms such as Apache Kafka, in a reliable and consistent way. However, as with everything in IT, there’s no free lunch. How to handle backfills of outbox events, how to ensure idempotency for event consumers? Doesn’t the pattern cause the database to become a bottleneck? And what about alternatives such as “Listen-to-Yourself”, or the upcoming Kafka support for 2-phase commit transactions (KIP-939)? It’s time to take another look at the outbox pattern! In this session I’ll start by bringing you up to speed on what the outbox pattern *is*, and then go on to discuss more details such as: - Implementing the pattern safely and efficiently - Its semantics, pros and cons - Dealing with backfills - Potential alternatives to the outbox pattern and the trade-offs they make