Speedtest: Benchmark your Apache Kafka

In this video, Mark Teehan, a Principal Solution Engineer at Confluent, takes us through a demo on how to benchmark Apache Kafka for your particular use case, defining what success looks like and how to configure clients. He says that before benchmarking, durability, availability, and ordering are common requirements when you're building out a new Kafka system. The throughput and latency requirements depend on your business needs and requires tuning of the producer and consumer. He also touches upon the importance of choosing the right kind of compression algorithm for different data types, and how it can make a huge difference to performance. He concludes by saying that when benchmarking, the test environment should be representative of the actual running environment to get accurate results. 0:00 Introduction 4:30 Service goals and tradeoffs 7:13 Producers 10:52 Start of demo 15:05 Kafka performance test tools 18:36 Producer and consumer perf test 21:38 Consumers 27:00 Results 32:36 Recommendations and conclusions 33: Q & A