"Distributed Commit Log: Application Techniques for Transaction Processing" by David McNeil

Messaging systems built on the idea of a distributed commit log such as Apache Kafka and Amazon Kinesis offer a powerful approach for building robust, real-time systems. However, because of their unique nature there are particular techniques required to use them effectively. This talk will catalog approaches used in practice to address topics such as: dealing with duplicate event delivery, merging & splitting of shards, poison events, exception handling, detecting failed workers, performing rapid failover, and providing facilities for debugging. These approaches provide a foundation for using distributed commit logs not just for applications such as click-stream analysis but for transaction processing.