This story was originally published on HackerNoon at:
https://hackernoon.com/scaling-a-kafka-consumer-from-4k-to-25k-events-per-second-while-preserving-ordering.
A production redesign using batching, safe offset commits, and record-level fallback raised Kafka consumer throughput from 4K to 25K without adding partitions.
Check more stories related to data-science at:
https://hackernoon.com/c/data-science.
You can also check exclusive content about
#apache-kafka,
#software-architecture,
#distributed-systems,
#software-development,
#scalability,
#performance-optimization,
#kafka-consumer,
#event-streaming, and more.
This story was written by:
@manjushaguntur. Learn more about this writer by checking
@manjushaguntur's about page,
and for more stories, please visit
hackernoon.com.
A Kafka consumer processing 4,000 events per second had to meet an 8,000 event SLO. Rather than add partitions, we redesigned the consumer around batching, safe offset commits, and record-level fallback. The solution was validated in production at approximately 25,000 events per second while preserving ordering and at-least-once delivery for recoverable events. This article explains how the design handled successful batches, transient failures, and terminal failures without silently losing events or blocking a partition.