Kafka & events · Capacity design
How many partitions, how many consumers?
Size a Kafka topic and its consumer group for a known peak rate. How many partitions and consumers do you need?
Capacity design · order-eventsIllustrative scenario, not a real incident
A new order-events topic goes live before next month's sale, and someone has to choose its numbers
Checkout will publish every order event to one topic. A group of consumers reads it and writes each event to the fulfilment database. The same event must be handled by one consumer of the group.
The team has measured one consumer instance under load, and has last year's traffic for the sale. Today's test topic has 12 partitions, chosen without much thought.
What the dashboards and logs show
- Peak rate
- 12,000 events a second, for about two hours
- Average rate
- 5,000 events a second
- One consumer
- 800 events a second, measured, database write included
- Event size
- about 1 KB
Estimate · scored under "Numbers"
Do the arithmetic first
Work each figure out from what is given, on paper or in your head. Round freely: an estimate counts when it is close, and the working is shown once you check.
- Peak rate
- 12,000 events a second
- Average rate
- 5,000 events a second
- One consumer handles
- 800 events a second
- Event size
- 1 KB
- Retention
- 7 days (the Kafka default)
- Replication factor
- 3
- Headroom wanted
- twice the peak, to catch up after a pause
- consumers
- partitions
- TB
- minutes
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.