Path 8 · lesson 4 of 5
Hot partitions and backpressure
Why one partition falls behind while the rest are idle, how to find and fix a skewed key, and how a pull-based consumer protects a slow database instead of flooding it.
What you will learn
- Recognise a hot partition from per-partition lag and log growth, and tell it apart from a group that is too small
- Explain why adding partitions or consumers does not help a single hot key, with the arithmetic
- Choose between a better key, salting, and a dedicated topic, and state what each does to ordering
- Describe backpressure in a pull-based system: what buffers the work and what the limit of that buffer is
- Pause and resume consumption in Spring to protect a slow or unavailable database without triggering a rebalance
- Use client quotas to stop one producer or consumer from taking the cluster's capacity
Before you start
The concept
Partition sizing assumes traffic spreads evenly: 1,200 records per second over 12 partitions is 100 each. That holds only if the keys are many and no key is much busier than the others. A keyed record goes to the partition chosen by a hash of its key, so every record with the same key goes to the same partition. If one key carries a large share of the traffic, its partition receives that whole share on top of its normal load. That is a hot partition.
A hot partition has one reader, like any other. The consumer that owns it falls behind while its neighbours sit almost idle, and the group's spare capacity cannot be lent to it, because a partition cannot be shared within a group.
Hot partition key
Simulation, not a real clusterThe animated lesson shows one key taking over a partition while the other partitions stay quiet.
Step 1 · The pattern
The key picks the partition
Every message has a key. Kafka hashes it to choose a partition, so messages with the same key always share a partition and stay in order.
- message
- consumed
Backpressure is the second topic of this lesson, and the two are related: both are about a consumer that cannot keep up with what arrives. Backpressure is the way a slow stage makes the stages before it slow down, instead of being overwhelmed. Kafka consumers pull: a broker never sends a record that a consumer has not asked for. A slow consumer therefore is not flooded. It simply asks less often, and the unread records wait in the log. That waiting work is lag.
Hot partition
- Lag grows on one partition (or a few), the rest stay near zero
- Cause: uneven keys, so uneven partitions
- More consumers or partitions do not help one hot key
- Fix: change how records are keyed or routed, or make that key's processing faster
Whole group too slow
- Lag grows on every partition at a similar rate
- Cause: produce rate above the group's ceiling, or a slow dependency
- More consumers help up to the partition count
- Fix: capacity, faster processing, or controlled backpressure while the dependency recovers
Why it exists
Keys become skewed for ordinary reasons. Each of these is a sensible-looking choice that turns out uneven in production:
- One large customer. A multi-tenant system keys by tenant id so that each tenant's events stay in order. One tenant is a marketplace that produces 40% of all orders.
- A low-cardinality key. Keying by country, event type or status gives a handful of distinct values. With 5 keys and 12 partitions, at most 5 partitions ever receive data, and their load follows the popularity of each value.
- A default value. Records for anonymous users all carry the key
guestor an empty string, which is one key like any other. - Records without a key, on an old client. A null key lets the producer choose. The original sticky partitioner (KIP-480) could send more records to partitions on slower brokers, a problem described and replaced in KIP-794. The built-in partitioner of current clients switches partition after about
batch.sizebytes and spreads null-key records evenly. If you see skew with null keys, check the client version and whether a custom partitioner is configured.
A key with few values uses few partitions
Simulation · not a real clusterRecords are keyed by country. Send several for each key and count how many of the eight partitions ever receive anything.
Backpressure matters for a different reason. Before a log sat between services, a slow receiver had two choices under a burst: refuse requests, which pushes errors to callers, or accept them into memory and risk running out. Kafka gives a third: the burst is written to disk on the brokers and the consumer works through it at its own pace. This is safe as long as you remember two limits. The buffer is finite, because retention deletes old records whether or not they were read. And the consumer is usually not the last stage: behind it is a database or an API that has its own limit, and the consumer must not push that over.
Consumer lag & backpressure
Simulation, not a real clusterThe animated lesson shows a burst being absorbed by the log while the consumer keeps its own pace.
Step 1 · The pattern
A log between two speeds
The producer writes messages to a topic. A consumer reads each one and writes it to the database. The topic sits between them so neither has to wait for the other.
- message
- processed
How it works inside
Why more partitions do not fix a hot key
Assume orders.events receives 1,000 records per second into 12 partitions, keyed by tenant id, and one tenant produces 40% of them. Each consumer thread handles 100 records per second (10 ms per record). All figures are assumptions for this example.
| Design | Load on the busiest partition | Against 100/s per thread | Ordering kept |
|---|---|---|---|
| Key = tenant id, 12 partitions | 400 + 600 / 12 = 450/s | Lag grows by 350/s, 21,000 per minute | Per tenant |
| Key = tenant id, 24 partitions | 400 + 600 / 24 = 425/s | Lag grows by 325/s | Per tenant (and moved keys during the change) |
| Key = tenant id + bucket, 8 buckets | 400 / 8 + 50 = 100/s if no two buckets share a partition | 100% busy at best; 150/s where two buckets collide | Per bucket only |
| Key = order id, 12 partitions | 1,000 / 12 = 83/s | 83% busy, keeps up | Per order |
| Big tenant on its own topic, 8 partitions, key = order id | 400 / 8 = 50/s there; 600 / 12 = 50/s on the main topic | 50% busy on both | Per order |
The three fixes
- A better key. Use the entity whose events must stay in order: order id instead of tenant id, device id instead of site id. This is the cleanest fix when the wider ordering was never required. It changes where existing keys go, so it is a migration like any change of partitioning.
- Salting. Append a bucket number to the hot key (
tenant-42#3), so its records spread over several partitions. Derive the bucket from something stable, such as a hash of the order id, so that events of one order still share a key. The cost: ordering now holds per bucket, not per tenant, and a consumer that needs everything for the tenant must read several partitions. Buckets can also collide on a partition. - A dedicated topic. Route the large tenant to its own topic with its own partition count and its own consumer group or concurrency. Its load can no longer delay other tenants, and it can be sized and throttled separately. The cost is routing logic in the producer and one more topic to operate.
- Faster processing for that key. If the ordering really must be per tenant, the only remaining lever is the time per record: batch the database writes, remove a remote call, or use key-ordered lanes inside the partition on a finer key.
Detecting a hot partition
- Lag per partition.
kafka-consumer-groups.sh --describelists every partition of the group. One partition with a large and growingLAGwhoseCURRENT-OFFSETis still advancing is overloaded. (If the offset is not moving at all, the consumer is stuck on a record, which is a different problem.) - Growth per partition. The broker's
BytesInPerSecandMessagesInPerSecare per topic, not per partition. For one partition, sample the end offset twice withkafka-get-offsets.shand subtract, or graph the rate of change of the broker metrickafka.log:type=Log,name=LogEndOffset,topic=…,partition=….name=Sizegives its bytes on disk. - Which key. Read a sample of the hot partition with
kafka-console-consumer.sh --partition Nprinting keys, and count them. One key dominating confirms skew.
Backpressure in a pull-based system
| Where pressure builds | What absorbs it | The limit | What you control |
|---|---|---|---|
| Consumer slower than the produce rate | The log: unread records wait as lag | Retention (retention.ms, 7 days by default) and the delay the business accepts | Capacity, headroom, alerts on lag growth |
| Database slower than the consumer | Nothing by default: each thread waits on its call, so the consumer slows down with it | max.poll.interval.ms (5 minutes by default) per batch | max.poll.records, pausing the container, pool size, timeouts |
| Producer faster than the brokers accept | The producer's buffer.memory | max.block.ms, then send() fails | What the application does when it cannot send |
| One client taking too much of a shared cluster | A quota: the broker delays that client's responses | The configured byte rate or request percentage | producer_byte_rate, consumer_byte_rate, request_percentage |
A consumer that processes records one at a time on its listener thread is naturally bounded: it cannot have more calls in flight than it has threads. With 12 threads, the database sees at most 12 concurrent writes from this group, however large the lag is. This property is easy to destroy. Handing records to an unbounded executor, or raising concurrency far above what the connection pool allows, turns a backlog in Kafka into a flood on the database at the moment it is least able to take it.
When the dependency is down or far too slow, continuing to poll and fail is wasteful: every record goes through retries and may end in a dead letter topic although nothing is wrong with it. The better response is to stop taking records and leave them in the log. KafkaConsumer.pause(partitions) does this: later calls to poll() return no records for the paused partitions, while the consumer keeps polling and so stays in the group. resume(partitions) reverses it. Spring wraps this as pause() and resume() on the listener container. The Spring reference states that a paused container continues to poll() the consumer, avoiding a rebalance, but does not retrieve records, and that a pause takes effect before the next poll, after the records from the previous poll have been processed.
Quotas
On a shared cluster, a backfill job or a misbehaving producer can saturate brokers for everyone. Client quotas put a ceiling on a user or client id: a byte rate for producing, a byte rate for fetching, and a share of broker request-handling time. They are enforced per broker. When a client exceeds its quota the broker computes the delay needed to bring it back under, returns the response with that delay, and stops reading from the client's connection until the delay is over. The client sees slower responses, not errors, and its produce-throttle-time-avg or fetch-throttle-time-avg metric rises. By default clients have no quota.
Architecture
Drawing diagram…
Flowchart: a large tenant's records all carry one key and land on partition 0 together with a share of other traffic, giving it 450 records per second against a consumer thread that handles 100. Other partitions and threads are half idle. All threads share a database pool, and when the database is slow the listener container is paused so records wait in the log.
flowchart LR
subgraph PRODUCERS["Producers"]
BIG["Large tenant 400 per second"]
REST["All other tenants 600 per second"]
end
BIG -->|"key tenant-42"| P0["Partition 0 at 450 per second"]
REST -->|"many keys"| P0
REST -->|"many keys"| PN["Partitions 1 to 11 at 50 per second each"]
P0 --> C0["Consumer thread 0 at 100 per second"]
PN --> CN["Consumer threads 1 to 11 half idle"]
C0 --> LAG["Lag on partition 0 grows 350 per second"]
C0 --> DB["Shared database pool"]
CN --> DB
DB --> SLOW{"Database slow or down"}
SLOW -->|"yes"| PAUSE["Pause the listener container, records stay in the log"]
SLOW -->|"no"| RUN["Process and commit"]- The skew starts at the producer: one key, one partition. Nothing downstream can spread it out again.
- Threads 1 to 11 have spare capacity that thread 0 cannot use, because a partition has one reader per group.
- The database pool is shared by all threads. It is the real last stage, and its capacity bounds the group regardless of partitions.
- The pause decision sits at the consumer. Kafka does not need to be told anything: an unpolled record simply remains in the log.
Message flow
Drawing diagram…
Sequence diagram: the producer keeps writing, the listener's database write times out, a health probe pauses the container, the container keeps polling without receiving records while lag grows, and when the probe succeeds the container resumes and works through the backlog.
sequenceDiagram
participant P as Order producer
participant K as Kafka partition
participant C as Listener container
participant H as Health probe
participant D as Database
P->>K: order events at 1000 per second
C->>K: poll
K-->>C: batch of records
C->>D: write
D-->>C: timeout
H->>D: probe
D-->>H: failing
H->>C: pause
loop while paused
C->>K: poll keeps group membership
K-->>C: no records for paused partitions
P->>K: events keep arriving, lag grows
end
H->>D: probe
D-->>H: healthy
H->>C: resume
C->>K: poll
K-->>C: backlog from committed position
C->>D: write at bounded concurrency- The producer is never told to slow down. Decoupling means it keeps writing, and the cost appears as lag.
- While paused the container still calls
poll(). That keeps it insidemax.poll.interval.ms, so the group does not rebalance and the partitions are not handed to another instance that would hit the same database. - The record whose write timed out was not committed. After resume it is delivered again, so the write must be idempotent.
- After resume the backlog is drained at the group's normal bounded concurrency. The database is not hit with the whole backlog at once.
In a real system
A typical multi-tenant order platform publishes to orders.events with 12 partitions. It was first keyed by tenant id. The Inventory Service reads it in the group inventory-service with 12 threads and writes each event to a relational database through a pool of 20 connections. One tenant, a marketplace, grows to 40% of all traffic.
The first sign is a lag alert on a single partition during the marketplace's busy hours, while group-level dashboards look fine. Scaling the service from 12 to 24 threads changes nothing: 12 partitions, 12 working threads. The team checks which ordering the Inventory Service needs. Stock reservations must be applied in order per order, and nothing depends on the order between two different orders of the same tenant. They move to keying by order id on a new topic and cut over, which brings the busiest partition from 450 to 83 records per second.
Worked example: what a database slowdown costs
| Quantity | Arithmetic | Result |
|---|---|---|
| Normal ceiling | 12 threads × 1,000 / 10 ms | 1,200 records/s |
| Produce rate | Assumed, steady | 1,000 records/s |
| Database slows to 40 ms per write | 12 × 1,000 / 40 | 300 records/s |
| Lag growth during the slowdown | 1,000 − 300 | 700 records/s |
| Backlog after 30 minutes | 700 × 1,800 s | 1,260,000 records |
| Spare capacity after recovery | 1,200 − 1,000 | 200 records/s |
| Time to clear the backlog | 1,260,000 / 200 | 6,300 s = 105 minutes |
| Same backlog if traffic falls to 400/s overnight | 1,260,000 / (1,200 − 400) | 1,575 s = 26 minutes |
| One poll batch during the slowdown | 500 records × 40 ms | 20 s, inside the 300 s limit |
| One poll batch if writes take 700 ms | 500 × 700 ms | 350 s: over the limit, the group rebalances |
max.poll.records must be sized for the slow case, not the normal one.The marketplace tenant on one partition
Simulation · not a real cluster40% of 1,000 records per second share one key. Read the limit the model names, then try to fix it with partitions and consumers before changing the hot key share.
Java and Spring Boot
Three pieces: a publisher that routes a large tenant to its own topic and keys by order id, a component that pauses the inventory listener while its database is failing, and the quota command for a backfill client.
package com.example.order;import java.util.Set;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.stereotype.Component;@Componentpublic class OrderEventPublisher { static final String SHARED_TOPIC = "orders.events"; static final String DEDICATED_PREFIX = "orders.events.tenant."; private final KafkaTemplate<String, OrderEvent> kafka; private final Set<String> dedicatedTenants; public OrderEventPublisher(KafkaTemplate<String, OrderEvent> kafka, OrderTopicProperties properties) { this.kafka = kafka; this.dedicatedTenants = Set.copyOf(properties.dedicatedTenants()); } public void publish(OrderEvent event) { String topic = dedicatedTenants.contains(event.tenantId()) ? DEDICATED_PREFIX + event.tenantId() : SHARED_TOPIC; kafka.send(topic, event.orderId(), event); } /** Alternative when per-tenant consumers need fewer streams: spread one tenant over a fixed number of buckets. */ static String saltedKey(String tenantId, String orderId, int buckets) { int bucket = Math.floorMod(orderId.hashCode(), buckets); return tenantId + "#" + bucket; }}Why it is written this way
- The key is the order id. All events of one order share a partition and stay in order; orders of the same tenant spread over all partitions.
- Tenants listed in configuration go to their own topic, which has its own partition count and its own listener. Their volume can no longer delay anyone else, and they can be given a quota of their own.
saltedKeyderives the bucket from the order id, not from a random number or a counter. A random salt would scatter the events of one order over partitions and lose their order.Math.floorModkeeps the bucket non-negative whenhashCode()is negative.- The bucket count is part of the key. Changing it later moves keys, just like changing the partition count.
- The send result is ignored here to keep the example short. A real publisher handles the returned future or writes through an outbox.
package com.example.order;import java.util.List;import org.springframework.boot.context.properties.ConfigurationProperties;@ConfigurationProperties(prefix = "orders.topics")public record OrderTopicProperties(List<String> dedicatedTenants) { public OrderTopicProperties { dedicatedTenants = dedicatedTenants == null ? List.of() : List.copyOf(dedicatedTenants); }}Why it is written this way
- Bound from
orders.topics.dedicated-tenantsand registered with@ConfigurationPropertiesScanor@EnableConfigurationPropertieson the application class. - Moving a tenant between topics is a routing change with ordering consequences for its in-flight orders. Do it at a quiet time and let the old topic drain first.
package com.example.inventory;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.dao.DataAccessException;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.config.KafkaListenerEndpointRegistry;import org.springframework.kafka.listener.MessageListenerContainer;import org.springframework.scheduling.annotation.Scheduled;import org.springframework.stereotype.Component;@Componentpublic class DatabaseBackpressure { private static final Logger log = LoggerFactory.getLogger(DatabaseBackpressure.class); private static final String LISTENER_ID = "inventory-service"; private final KafkaListenerEndpointRegistry registry; private final JdbcTemplate jdbc; public DatabaseBackpressure(KafkaListenerEndpointRegistry registry, JdbcTemplate jdbc) { this.registry = registry; this.jdbc = jdbc; } @Scheduled(fixedDelay = 5000) public void adjust() { MessageListenerContainer container = registry.getListenerContainer(LISTENER_ID); if (container == null || !container.isRunning()) { return; } boolean healthy = databaseResponds(); if (!healthy && !container.isPauseRequested()) { log.warn("Database probe failed: pausing listener {}", LISTENER_ID); container.pause(); } else if (healthy && container.isPauseRequested()) { log.info("Database probe succeeded: resuming listener {}", LISTENER_ID); container.resume(); } } private boolean databaseResponds() { try { jdbc.queryForObject("SELECT 1", Integer.class); return true; } catch (DataAccessException e) { return false; } }}Why it is written this way
LISTENER_IDis theidof the@KafkaListener. The registry returns its container, which for a concurrency above 1 pauses all of its consumers.pause()does not stop the consumer threads. They keep callingpoll()and receive nothing, so the members stay in the group and no rebalance happens. A stopped container would leave the group and hand its partitions to another instance facing the same database.- By default the pause takes effect after the records of the current poll have been processed. Those records still fail against the broken database and go through the container's error handler, so keep
max.poll.recordssmall for this listener. The container propertypauseImmediatemakes the pause apply after the current record. @Scheduledneeds@EnableSchedulingon a configuration class. Give the probe a short query timeout (for example through theJdbcTemplatebean'squeryTimeout) so that a hanging database does not block the scheduler thread.- The probe checks reachability only. To react to a database that is slow but answering, drive the same pause and resume from a circuit breaker or from the processing timer instead.
- Each application instance runs this independently and pauses only its own consumers.
# Limit the backfill job (client.id=order-backfill) to 2 MB/s produce and 4 MB/s fetch PER BROKER.bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter \ --add-config 'producer_byte_rate=2097152,consumer_byte_rate=4194304' \ --entity-type clients --entity-name order-backfill# Show what is configured for that client id.bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe \ --entity-type clients --entity-name order-backfillWhy it is written this way
- Quotas apply per broker. A client whose partitions have leaders on 3 brokers can reach up to three times the configured rate in total.
- A quota on a client id works without authentication. On a secured cluster, prefer quotas on the user (
--entity-type users), since a client id is chosen by the client. - A throttled client is delayed, not failed. Watch
produce-throttle-time-avgandfetch-throttle-time-avgon the client and the broker's per-client throttle metrics. request_percentagelimits the share of broker network and I/O thread time instead of bytes.
Configuration
Settings that shape how a consumer behaves under pressure and how the cluster limits a client. Defaults are from the Kafka 4.4 references; the quota keys are set with kafka-configs.sh and have no default, which means unlimited.
max.poll.records (consumer)Default 500
- What it does
- Maximum records per
poll(): the size of the unit of work between two polls. - When to change it
- Size it for the slow case: records × worst processing time must stay well below
max.poll.interval.ms. Smaller batches also let a pause take effect sooner.
max.poll.interval.ms (consumer)Default 300000
- What it does
- Longest time between polls before the consumer is considered failed and the group rebalances.
- When to change it
- Do not raise it to hide slow processing. A consumer that exceeds it loses its partitions, and the next owner repeats the same slow batch.
retention.ms (topic)Default 604800000
- What it does
- How long records are kept under the delete policy. It is the size, in time, of the buffer that lag can use.
- When to change it
- Must comfortably exceed the longest outage plus recovery time you plan for. Records older than this are deleted even if a group has not read them.
producer_byte_rate (quota)Default none (unlimited)
- What it does
- Bytes per second a user or client id may produce to each broker before it is throttled.
- When to change it
- Set it for batch jobs, backfills and tenants on shared clusters, so one producer cannot take the brokers' capacity.
consumer_byte_rate (quota)Default none (unlimited)
- What it does
- Bytes per second a user or client id may fetch from each broker before it is throttled.
- When to change it
- Set it for replay and analytics consumers that would otherwise read history at full speed and evict the page cache used by live consumers.
request_percentage (quota)Default none (unlimited)
- What it does
- Share of broker request handler and network thread time a client group may use, as a percentage of one thread.
- When to change it
- Use it against clients that send very many small requests, which a byte-rate quota does not catch.
partitioner.ignore.keys (producer)Default false
- What it does
- When true, the built-in partitioner does not use the record key to choose a partition and distributes records as if they had no key.
- When to change it
- An option when the key is needed in the record (for example for compaction elsewhere or for consumers) and per-key ordering is not. It gives up ordering by key completely.
spring.kafka.listener.concurrencyDefault none (one consumer per listener)
- What it does
- Consumer threads per listener in one instance, which is also the maximum number of concurrent calls this instance makes to its database when records are processed on the listener thread.
- When to change it
- Keep instances × concurrency within what the database and its connection pool can serve. The pool must be at least as large as the concurrency.
Hands-on exercise
Fix a hot partition, then protect a slow database
See why a single hot key defeats scaling, find the changes that do help, and then make the database the limit to see what backpressure has to deal with.
Steps · 0 of 6 done
One hot key against twelve consumers
Simulation · not a real clusterA simple queueing model with illustrative numbers. The group has 1,200 records per second of capacity for 1,000 arriving, and still falls behind.
What you should see
With 40% on one key, partition 0 receives 450 records per second against 100 handled, and the model names the hot partition as the limit although total capacity exceeds the produce rate. At 24 partitions it still receives 425. Even at a 10% share partition 0 receives 175 per second and falls behind; with no hot key every partition receives 83 and the group keeps up. With the 40% share back, partition 0 keeps up only when one record takes about 2 ms. With a pool of 5 the ceiling is 500 records per second and adding consumers changes nothing: they queue for connections.
What can go wrong
Adding partitions to cure a hot key
- Symptom
- The topic went from 12 to 48 partitions. The same partition, or one new one, is still far behind, and some keys now have events in two partitions.
- Cause
- One key maps to one partition at any partition count. The change only thinned out the other traffic and moved keys.
- Fix
- Find the hot key first. Change the key, salt it or route it to its own topic. Keep the partition count for capacity planning.
A random salt
- Symptom
- After salting the tenant key, load is even, and orders occasionally show a cancellation processed before the placement.
- Cause
- The salt was random per record, so events of one order went to different partitions and lost their relative order.
- Fix
- Derive the salt from the entity that must stay ordered, such as a hash of the order id, so its events always share a key.
Alerting on the group's total lag
- Symptom
- One tenant's events are an hour late while the lag dashboard shows a total well under the alert threshold.
- Cause
- Eleven healthy partitions average out one partition that is far behind.
- Fix
- Alert on the worst partition's lag and on its growth over several minutes. Show lag per partition on the dashboard.
Draining a backlog into a recovering database
- Symptom
- The database comes back after an outage and is immediately overloaded again by the consumers catching up.
- Cause
- Concurrency was raised, or records were handed to an executor, to clear the backlog faster. The database received far more concurrent writes than in normal operation.
- Fix
- Keep catch-up at the normal bounded concurrency, or raise it in steps while watching the database. The time to drain is known from backlog / spare capacity.
Failing every record while the database is down
- Symptom
- During a database outage thousands of healthy records end up in the dead letter topic and must be replayed by hand.
- Cause
- The listener kept consuming. Each record exhausted its retries against a dependency that was down for all of them.
- Fix
- Treat a failing dependency as a reason to stop consuming: pause the container (or use a circuit breaker that does), and resume when a probe succeeds. Reserve the dead letter topic for records that are themselves bad.
Stopping instead of pausing
- Symptom
- Each time the listener is stopped to protect the database, the group rebalances, and it rebalances again on restart.
- Cause
- A stopped container closes its consumers, which leave the group. Their partitions go to other instances, which then hit the same database.
- Fix
- Use
pause()andresume(). A paused consumer keeps polling and keeps its assignment.
How to troubleshoot
Lag is growing and you need to know whether it is one partition or all of them
Check
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group inventory-service
- How to read it
- One row per partition. A single large
LAGwith an advancingCURRENT-OFFSETis a hot partition. Similar lag on all rows is a capacity or dependency problem. A row whose offset does not move is a stuck record.
You suspect one partition receives much more than the others
Check
bin/kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic orders.events --time -1 (run twice, a minute apart)
- How to read it
- Prints
topic:partition:end offset. Subtract the two samples per partition to get records per minute. A partition growing several times faster than the rest is hot. The broker metrickafka.log:type=Log,name=LogEndOffsetgives the same continuously.
You need to find the key behind a hot partition
Check
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders.events --partition 0 --offset latest --max-messages 2000 --formatter-property print.key=true --formatter-property print.value=false | sort | uniq -c | sort -rn | head
- How to read it
- Counts keys in a sample of the partition. One key with most of the records confirms skew. (
--formatter-propertyis the 4.x option name; older releases use--property.)
The listener stopped consuming and nobody knows whether it is paused or broken
Check
The application log for the pause and resume lines, Spring's `ConsumerPausedEvent` / `ConsumerResumedEvent`, and `kafka-consumer-groups.sh --describe --members`
- How to read it
- A paused consumer is still a group member with partitions assigned and its lag grows steadily. A consumer that left the group shows no member for those partitions.
A client is slow for no visible reason on a shared cluster
Check
Client metrics `produce-throttle-time-avg` / `fetch-throttle-time-avg`; broker metrics kafka.server:type={Produce|Fetch},user=…,client-id=… (attributes throttle-time, byte-rate)- How to read it
- A throttle time above zero means a quota is delaying this client. Compare
byte-ratewith the configured quota fromkafka-configs.sh --describe.
The group rebalances repeatedly while the database is slow
Check
Consumer metrics `time-between-poll-max` and the log line saying the consumer left the group because the poll interval was exceeded
- How to read it
- A batch took longer than
max.poll.interval.ms. Lowermax.poll.recordsso the slow case fits, and pause the container when the dependency is failing.
Key takeaways
- All records with one key go to one partition, and one consumer of the group reads it. A busy key makes a hot partition that spare capacity elsewhere cannot help.
- More partitions or consumers do not fix a single hot key. Change the key, salt it from a stable value, or give the large tenant its own topic.
- The key is also the ordering scope. Use the narrowest key that gives the ordering consumers really need.
- Look at lag and log growth per partition. Group totals hide the one partition that matters.
- Kafka consumers pull, so the log is the buffer and lag is its fill level. Its limits are retention and the delay the business accepts.
- When a dependency is failing, pause the listener container instead of failing every record. A paused consumer keeps polling and keeps its partitions.
- Recovery time is backlog / spare capacity. Headroom is what makes it short.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A telemetry topic has 16 partitions, keyed by site id. One site, a large factory, sends 900 of the 2,000 records per second. Each consumer thread handles 200 records per second and the group has 16 threads. Lag grows on one partition by about 770 records per second while the other fifteen are near zero. Readings must stay in order per device, and each site has hundreds of devices.
Which change removes the hot partition while keeping the ordering that is needed?
Scenario 2 of 3
A consumer group of 10 threads writes each record to a database and normally handles 2,000 records per second against 1,500 produced. The database fails over and is unavailable for 10 minutes. During that time the listener keeps running: every record is retried three times and then sent to the dead letter topic. Afterwards about 900,000 healthy records are in the dead letter topic.
What should the service do differently during the next failover?
Scenario 3 of 3
A shared cluster carries the order topics and is also used by a new analytics job that re-reads 30 days of history every night with client id history-scan. While it runs, live consumers of the order topics see their fetch latency rise from 5 ms to 400 ms and broker disks show heavy read activity that is absent during the day.
Which measure protects the live consumers without removing the job?
References
References: Apache Kafka 4.4: Design (the producer and partitioning by key; quotas and their enforcement) ↗ · Apache Kafka 4.4: Basic Kafka operations (checking consumer position, setting quotas) ↗ · Apache Kafka 4.4: Monitoring (per-partition log metrics, quota and throttle metrics, consumer lag metrics) ↗ · Apache Kafka 4.4: KafkaConsumer Javadoc (consumption flow control: pause and resume) ↗ · Apache Kafka 4.4: Producer configs (partitioner.ignore.keys, partitioner.adaptive.partitioning.enable) ↗ · Apache Kafka 4.4: Multi-tenancy (isolating tenants with quotas) ↗ · Spring for Apache Kafka: Pausing and resuming listener containers ↗ · KIP-794: Strictly Uniform Sticky Partitioner ↗ · Report a problem