Master Kafka Log Aggregation with ELK Stack

In this video we cover the log aggregation use case of kafka where we collect the logs from 6 different microservices where we got on an average of 300 logs from each service per minute and send it kafka and from there we have a consumer called logstash which consumes these messages and pass it down to elastic search and we use Kibana for the log analysis and monitoring. Titles: kafka log aggregation kafka with elasticsearch kafka elasticsearch kibana kafka with elk stack Hastags: #kafka #apachekafka #systemdesign #microservices Series: Part1:    • Kafka fundamentals every developer must know!   Part2:    • Kafka fundamentals every developer must kn...   Part3:    • Kafka fundamentals | Brokers   Part4:    • Kafka Without Zookeeper? Setting Up KRaft ...   Part5:    • kafka CLI fundamentals every developer sho...   Playlist:    • Zero to Hero Kafka   Chapters: 00:00 intro 00:33 Architecture 03:40 Microservices base 07:03 Microservice logs demo 08:36 Multiple microservices trick 09:33 Spinning services step by step 10:57 Log rotation 14:13 Verifying logs 15:22 Filebeat 17:08 Kafka 19:51 Topic setup with 10 partitions 20:49 Filebeat config 24:01 Logstash & Elasticsearch 30:27 Kibana 32:38 Logs Analysis 34:34 Do we need all these services? 37:19 Kibana dashboard setup 40:25 Conclusion About Me: ----------------- LinkedIn:   / vivek-bharatha   Discord:   / discord   Github: https://github.com/vivekbharatha Medium:   / vivek.bharatha   YouTube:    / @swe-with-vivekbharatha