Path 1 · lesson 6 of 6
When not to use Kafka
The limits of a partitioned log: seven situations where Kafka is the wrong tool or needs an adaptation, and what to use or do instead.
What you will learn
- Recognise seven requirements that a partitioned log does not meet out of the box
- Explain each limit from how partitions, offsets and retention work, not from a list of rules
- Choose an alternative (a direct call, a queue broker, a database table, object storage) or an adaptation that keeps Kafka
- Estimate what running Kafka costs a small team before adopting it
- Make and defend a fit or no-fit decision for a concrete system
Before you start
The concept
The previous lesson showed how much one log can do. This one is about where it stops. Every strength of Kafka comes from the same design: records are appended to a partition, stay there for a retention period, and each consumer group moves a single position forward through them. Every limit comes from that design too. If a requirement needs something the log does not have (a per-message decision by the broker, an immediate answer, one order across everything), you either adapt the design around it or pick a different tool.
| Requirement | Why the log struggles | Use instead, or adapt |
|---|---|---|
| A simple task queue, with per-message routing or priority | A partition is read in offset order by one consumer of a group. A slow task holds back everything behind it, and the broker cannot pick "the urgent one" or route by content. | A queue broker such as RabbitMQ or Amazon SQS, or a jobs table in your database. In Kafka: share groups (4.2+) for per-record acknowledgement, separate topics per priority. |
| Request/response where a caller waits for the answer | Two asynchronous hops, a reply topic, correlation ids and a timeout, to imitate what one HTTP or gRPC call does directly. | A synchronous call for the question. Publish an event afterwards for everyone else. |
| A small team with low volume | A production cluster is several servers to patch, monitor, secure and upgrade, whatever the traffic. | A managed queue, a database table with an outbox, or a managed Kafka service if you need the log semantics. |
| Large payloads | A record batch is limited to about 1 MB by default, and big records hurt batching, memory and replication for everyone on the cluster. | Store the file in object storage and send a small event with its location (claim check). |
| Strict global ordering | Order is guaranteed only within a partition. One total order means one partition, which means one active consumer per group. | Order per key, which is usually what the business needs. If truly global: a single partition with the throughput limit accepted, or a database sequence. |
| Per-message TTL or delayed delivery | Retention is set per topic and removes whole segments. A consumer reads what is next in the log; nothing can be held back until later. | An expiry time in the event that the consumer checks; retry topics with back-off; a scheduler or a broker with native delay. |
| Exactly-once side effects outside Kafka | Kafka transactions cover writes to Kafka and consumer offsets. A database row, an HTTP call or an email is outside them. | Idempotent consumers, a transactional outbox, or storing the offset in the same database transaction as the result. |
Why it exists
Kafka was built for high-volume streams read by many independent consumers. To get there, it moved work out of the broker. The broker does not track which messages each consumer has finished, does not filter or route by content, and does not delete a message when it is consumed. It appends, replicates and serves byte ranges. That is why it is fast and why consumers can replay. It is also why features that depend on the broker knowing about individual messages are missing.
Queue broker (RabbitMQ, SQS)
- The broker hands each message to one worker and tracks its acknowledgement
- A slow or failed message does not block the others; it is redelivered or dead-lettered on its own
- Routing by key or header, priorities, per-message TTL and delay are broker features
- A message is gone once acknowledged: no replay, no second independent reader without another queue
Kafka with consumer groups
- A consumer group tracks one position per partition; a commit means "everything before this offset is done"
- Records in a partition are processed in order, so one slow record delays those behind it
- No routing, priority, TTL or delay per message in the broker
- Records stay for the retention period: replay and any number of independent groups
The second reason this lesson exists is cost. Kafka is rarely the cheapest thing that works. Teams adopt it because a conference talk or a larger company uses it, and then spend months operating a cluster that carries a few messages per second. A relational database that the team already runs, already backs up and already understands can do the same job for a long time. Choosing Kafka should follow from a requirement the simpler option cannot meet: several independent consumers, replay, ordered streams per key at high volume, or integration through Connect.
How it works inside
1. Task queues: why one slow record blocks a partition
In a consumer group, each partition is assigned to exactly one consumer, and that consumer's progress is one number: the committed offset, which is the position of the next record to read. There is no way to say "I finished offset 12 but not offset 11". So a consumer works through a partition in order. If the record at offset 11 is a video that takes 90 seconds to transcode, the thumbnails at offsets 12 and 13 wait, even if other consumers in the group are idle. Adding consumers does not help beyond the partition count, because extra consumers get no partition.
It gets worse when a task is slower than max.poll.interval.ms (default 5 minutes). A consumer that does not call poll() again within that time is considered failed, the group rebalances, the partition goes to another consumer, and the same slow record is delivered again from the last committed offset. A queue broker handles this case per message: the others are delivered to other workers and the failed one is retried or dead-lettered alone. Kafka applications rebuild that with retry topics and dead letter topics, covered in the reliability path.
2. Request/response: the wrong shape, not only the wrong speed
A caller that needs an answer must publish a request, wait on a reply topic, match the reply by a correlation id and give up after a timeout. Spring provides ReplyingKafkaTemplate for this, so it is possible. But each direction passes through producer batching (linger.ms is 5 ms by default since 4.0), replication when acks=all, and the consumer's fetch. Worse, the failure behaviour is wrong for a waiting user: if the responder is down, requests queue up and are answered minutes later, long after the caller timed out. A direct call fails fast and tells the caller so. Use Kafka for the fact that follows ("order placed"), not for the question that precedes it ("is this item in stock?").
3. Operational cost: the fixed part does not shrink with traffic
A cluster that survives one server failure without losing acknowledged data needs replication factor 3, so at least three brokers, and a controller quorum of three so that a majority survives one loss. The documentation advises against combining broker and controller roles in critical deployments. Add monitoring of lag, under-replicated partitions and disk, security (TLS, authentication, ACLs), capacity planning, and rolling upgrades about three times a year if you follow the project's release rhythm. A managed service removes the servers but not the design work: keys, partition counts, consumer error handling and idempotency remain yours.
4. Large payloads: the limit is on the batch, and it is about 1 MB
The broker setting message.max.bytes defaults to 1048588 bytes and is, in the documentation's words, the largest record batch size allowed, measured after compression. The producer has its own cap, max.request.size, default 1048576 bytes. Both can be raised, per topic with max.message.bytes. The reason not to is what large records do to the rest of the system: each one occupies producer buffer memory, travels to every replica, sits in the page cache, and is fetched whole by every consumer group. A 20 MB PDF read by four groups from three replicas is moved many times for a file that most readers only need a link to.
5. Global ordering: one partition, one consumer
Kafka guarantees order only within a partition, for records as the broker appended them. A topic with six partitions has six independent orders. If you truly need a single order across all records (a strictly sequential ledger, for example), the topic must have one partition. Then one consumer per group does all the work, throughput is what one consumer can process, and a single broker leads all writes. Before accepting that, ask whether order per account, per order or per device would do. It nearly always does, and it scales.
6. TTL and delay: retention is not expiry
retention.ms is a topic setting. The broker deletes whole log segments once they are old enough, so a record can live noticeably longer than the configured time, and every record in a topic shares the same setting. A consumer that is behind will still receive a promotion that ended an hour ago. And because a consumer always reads the next offset, there is no "deliver this at 09:00". Adaptations: put expiresAt in the event and let the consumer drop stale ones; for delays, Spring's non-blocking retry topics forward a record to a retry topic with a due timestamp and pause that partition until it is due, which works for a few fixed delays but not for arbitrary schedules. Amazon SQS, by comparison, supports a delivery delay per queue or per message of up to 15 minutes, and RabbitMQ supports a TTL per message.
7. Side effects outside Kafka
This last point is not a reason to avoid Kafka, because every asynchronous system has it. It is a reason not to choose Kafka *because* of the phrase "exactly-once", and to budget for idempotent consumers from the first design.
Architecture
Drawing diagram…
Decision flowchart. If a caller is waiting for the answer, use a direct call. Otherwise, if several independent consumers need the data or it must be replayable, Kafka is a candidate. If not, background work needing priority, delay or per-task routing goes to a queue broker or jobs table, and low volume with nobody to run a cluster goes to the existing database or a managed queue. For a Kafka candidate, three further questions lead to adaptations: large payloads are stored elsewhere with a reference, a need for total order is reduced to order per key or one partition, and effects outside Kafka need an idempotent consumer or an outbox.
flowchart TD
START["New interaction between two parts of the system"]
Q1{"Is a user or caller waiting for the answer?"}
R1["Use a direct call - HTTP or gRPC"]
Q2{"Do several independent consumers need it, or must it be replayable?"}
Q3{"Is it background work with priority, delay or per-task routing?"}
R2["Use a queue broker or a jobs table"]
Q4{"Low volume and nobody to run a cluster?"}
R3["Use the database you have - outbox table or a managed queue"]
K["Kafka is a candidate - now check the adaptations"]
Q5{"Payloads above about 1 MB?"}
A1["Store the file elsewhere and send a reference"]
Q6{"Need one order across all records?"}
A2["Prefer order per key - else one partition and one consumer"]
Q7{"Must an effect outside Kafka happen only once?"}
A3["Idempotent consumer or outbox"]
FIT["Kafka fits - design keys, partitions and consumer groups"]
START --> Q1
Q1 -->|"yes"| R1
Q1 -->|"no"| Q2
Q2 -->|"no"| Q3
Q3 -->|"yes"| R2
Q3 -->|"no"| Q4
Q4 -->|"yes"| R3
Q4 -->|"no"| K
Q2 -->|"yes"| K
K --> Q5
Q5 -->|"yes"| A1
Q5 -->|"no"| Q6
A1 --> Q6
Q6 -->|"yes"| A2
Q6 -->|"no"| Q7
A2 --> Q7
Q7 -->|"yes"| A3
Q7 -->|"no"| FIT
A3 --> FIT- Apply the flowchart to one interaction at a time, for example "checkout asks for the stock level" or "payment tells others the charge succeeded". A system has many interactions and they will not all take the same exit.
- The first question removes the most common misuse. If someone is waiting for the answer, call the service. You can still publish an event about what happened.
- The second question is the positive test for Kafka: several independent readers, or the need to read the same data again later. If neither applies, a queue is simpler.
- The questions below the Kafka box do not reject it. They add a design element: a claim check for large payloads, a keying decision for order, idempotency for outside effects.
- Volume alone is not a question in the chart. High volume makes Kafka attractive, but low volume does not forbid it if you need replay or fan-out and use a managed service.
Message flow
Drawing diagram…
Sequence diagram. An upload API writes a 90 second video job and two 1 second thumbnail jobs to partition 0 of a media-jobs topic. Worker 1 owns partition 0 and receives all three. Worker 2 owns partition 1, which is empty, and stays idle. The thumbnails wait behind the video and all three are committed after 92 seconds. In the second half, a queue broker delivers the video to worker 1 and the thumbnails to worker 2, which finishes them in 2 seconds.
sequenceDiagram participant A as Upload API participant P as media-jobs partition 0 participant W1 as Worker 1 owns partition 0 participant W2 as Worker 2 owns partition 1 A->>P: Offset 11 transcode video - 90 seconds A->>P: Offset 12 make thumbnail - 1 second A->>P: Offset 13 make thumbnail - 1 second P-->>W1: Poll returns offsets 11 12 13 Note over W1: Busy with offset 11 Note over W2: Idle - partition 1 is empty Note over P: Offsets 12 and 13 wait behind 11 W1->>P: After 92 seconds commit offset 14 Note over A,W2: Same jobs through a queue broker A->>W1: Queue delivers video job A->>W2: Queue delivers both thumbnail jobs W2-->>A: Thumbnails acknowledged after 2 seconds W1-->>A: Video acknowledged after 90 seconds
- All three jobs landed in partition 0, either by key or by chance. Worker 1 owns that partition and receives them in offset order.
- Worker 2 is free but cannot take offsets 12 and 13: in a consumer group a partition belongs to one consumer at a time.
- Worker 1 commits offset 14 after finishing all three. If it crashed during the video, another worker would restart from the last committed offset and repeat the work.
- A queue broker tracks each message separately, so the thumbnails go to the idle worker. Kafka share groups (4.2+) behave like the lower half for records that need no ordering; classic consumer groups behave like the upper half.
- If the 90 seconds were 6 minutes, worker 1 would exceed the default
max.poll.interval.ms, be removed from the group, and the video would be delivered again to the next owner of the partition.
In a real system
Take the academy's order system at a typical online shop and go through its interactions one by one. Some belong on Kafka and some do not.
| Interaction | Decision | Why |
|---|---|---|
| Checkout page asks whether an item is in stock | Direct call to the Inventory Service | A user is waiting and needs an answer or a fast failure. |
| Order Service announces "order placed" | Kafka: topic orders, key = order id | Payment, Inventory, Notification and the warehouse each need it independently, in order per order, and must catch up after downtime. |
| Payment Service charges the card | Kafka consumer plus an idempotency key at the payment provider | The record can be delivered twice. Kafka cannot make the charge happen once; the key can. |
| Invoice PDF, about 2 MB, sent to the customer | PDF in object storage; invoices event carries its location | Above the default batch limit, and most consumers only need to know that it exists. |
| "Send a reminder if the basket is still unpaid after 24 hours" | A scheduler or a database table with a due time | Kafka has no delayed delivery. A consumer would have to hold the record or re-publish it for a day. |
| Resize product images uploaded by staff, urgent ones first | A queue broker with priorities, or a jobs table | Independent tasks, uneven duration, a priority, no need for replay or a second reader. |
| Voucher valid for 10 minutes | Kafka event with expiresAt; consumers ignore it after that | Retention cannot expire one record. The check belongs in the consumer. |
Now make the shop smaller: two developers, one PostgreSQL database, a few hundred orders a day. The honest design has no Kafka in it yet. The Order Service writes the order and an outbox row in one transaction; a worker reads the outbox with SELECT ... FOR UPDATE SKIP LOCKED and calls the payment and email code. If the outbox rows are already shaped as events (type, key, payload), moving to Kafka later means replacing the worker with a connector or a publisher, not redesigning the services.
Java and Spring Boot
Two adaptations in code: the claim check for a large payload (store the file, publish a reference) and an expiry check in the consumer for events that are only valid for a while.
package com.example.shop.invoice;import java.net.URI;import java.security.MessageDigest;import java.security.NoSuchAlgorithmException;import java.util.HexFormat;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.stereotype.Service;@Servicepublic class InvoicePublisher { /** The event is a few hundred bytes whatever the size of the PDF. */ public record InvoiceRendered(String orderId, URI pdfLocation, long sizeBytes, String sha256) {} /** Thin wrapper over the object storage the company already uses. */ public interface BlobStore { URI put(String objectKey, byte[] content, String contentType); byte[] get(URI location); } private static final String TOPIC = "invoices"; private final BlobStore blobStore; private final KafkaTemplate<String, InvoiceRendered> kafkaTemplate; public InvoicePublisher(BlobStore blobStore, KafkaTemplate<String, InvoiceRendered> kafkaTemplate) { this.blobStore = blobStore; this.kafkaTemplate = kafkaTemplate; } public void publish(String orderId, byte[] pdf) { // 1. Store the file first, under a name derived from the order id: // a retry overwrites the same object instead of creating a second one. URI location = blobStore.put("invoices/" + orderId + ".pdf", pdf, "application/pdf"); // 2. Publish the small event. Key = order id, like every other event of this order. InvoiceRendered event = new InvoiceRendered(orderId, location, pdf.length, sha256(pdf)); kafkaTemplate.send(TOPIC, orderId, event).join(); } private static String sha256(byte[] content) { try { return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(content)); } catch (NoSuchAlgorithmException e) { throw new IllegalStateException("SHA-256 is required on every Java platform", e); } }}Why it is written this way
- Upload first, publish second. If the publish fails, an unreferenced file remains, which a storage lifecycle rule can clean up. The other order would publish a reference to a file that does not exist yet.
join()waits for the broker's acknowledgement and throws if the send finally fails, so the caller knows the event was not published. For a high-volume path you would handle the future asynchronously instead.- The checksum lets a consumer detect a file that was replaced or truncated after the event was written.
- The file must outlive the event: object retention has to be at least the topic's retention, or a replay will find dead links.
package com.example.shop.invoice;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;import com.example.shop.invoice.InvoicePublisher.BlobStore;import com.example.shop.invoice.InvoicePublisher.InvoiceRendered;@Componentpublic class InvoiceEmailListener { /** Remembers which invoices were mailed, in the service's own database. */ public interface SentInvoices { /** Returns false if this order's invoice was already recorded. */ boolean markSentIfNew(String orderId); } public interface InvoiceMailer { void send(String orderId, byte[] pdf); } private final BlobStore blobStore; private final SentInvoices sentInvoices; private final InvoiceMailer mailer; public InvoiceEmailListener(BlobStore blobStore, SentInvoices sentInvoices, InvoiceMailer mailer) { this.blobStore = blobStore; this.sentInvoices = sentInvoices; this.mailer = mailer; } @KafkaListener(id = "invoice-email", groupId = "invoice-email", topics = "invoices") public void onInvoiceRendered(InvoiceRendered event) { // Kafka may deliver this record again after a rebalance or restart. // The unique order id in our own table is what prevents a second email. if (!sentInvoices.markSentIfNew(event.orderId())) { return; } byte[] pdf = blobStore.get(event.pdfLocation()); mailer.send(event.orderId(), pdf); }}Why it is written this way
- Only the consumer that needs the bytes fetches them. An accounting consumer on the same topic can work from the event alone.
markSentIfNewis consumer-side idempotency, typically an insert with a unique constraint on the order id. It is application logic; no Kafka setting provides it.- This version marks before sending, so a crash between the two lines loses the email (at-most-once for the side effect). Marking after sending risks a duplicate instead. Email cannot join a transaction, so one of the two must be chosen deliberately.
package com.example.shop.voucher;import java.time.Clock;import java.time.Instant;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;@Componentpublic class VoucherListener { public record VoucherIssued(String voucherId, String customerId, int percentOff, Instant expiresAt) {} public interface VoucherWallet { void add(VoucherIssued voucher); } private static final Logger log = LoggerFactory.getLogger(VoucherListener.class); private final VoucherWallet wallet; private final Clock clock; public VoucherListener(VoucherWallet wallet, Clock clock) { this.wallet = wallet; this.clock = clock; } @KafkaListener(id = "voucher-wallet", groupId = "voucher-wallet", topics = "vouchers") public void onVoucherIssued(VoucherIssued voucher) { // The broker will deliver this record as long as it is retained, // also to a consumer that is hours behind. Expiry is our job. if (voucher.expiresAt().isBefore(clock.instant())) { log.info("dropping expired voucher {}", voucher.voucherId()); return; } wallet.add(voucher); }}Why it is written this way
- The expiry time travels inside the event as an absolute instant. A relative "valid for 10 minutes" would be meaningless to a consumer that reads the record late.
- Returning normally lets the listener container advance past the record. Dropping an expired event is a successful outcome, not an error to retry.
- Injecting
Clock(declare aClockbean, for exampleClock.systemUTC()) makes the expiry rule testable with a fixed time.
Configuration
The settings behind the limits in this lesson. Defaults are from the Apache Kafka 4.4 configuration reference. Knowing them tells you when you have hit a design limit and not a bug.
message.max.bytes (broker) / max.message.bytes (topic)Default 1048588
- What it does
- The largest record batch the broker accepts, after compression if compression is enabled.
- When to change it
- Raise it per topic only for a measured, bounded need (a few MB at most), together with the producer's
max.request.size. For files, use a claim check instead.
max.request.size (producer)Default 1048576
- What it does
- The largest request the producer will send, which is also effectively the cap on one uncompressed record batch. Checked in the client before anything is sent.
- When to change it
- A record above it fails in the producer with
RecordTooLargeException. Treat that as a design signal before raising the number.
max.partition.fetch.bytes (consumer)Default 1048576 (1 mebibyte)
- What it does
- How much data per partition a fetch returns. Not an absolute maximum: if the first batch is larger, it is still returned so the consumer can make progress.
- When to change it
- You do not have to raise it for a consumer to read a larger record, but each such fetch then carries one batch per partition, which lowers throughput. The same holds for
replica.fetch.max.byteson brokers.
linger.ms (producer)Default 5
- What it does
- How long the producer waits for more records to fill a batch before sending. It was 0 before Kafka 4.0.
- When to change it
- One of several small delays on every hop. Lowering it to 0 on a latency-critical producer trades throughput for a few milliseconds; it does not make Kafka a request/response system.
max.poll.interval.ms (consumer)Default 300000 (5 minutes)
- What it does
- The longest allowed gap between two
poll()calls before the consumer is considered failed and its partitions are reassigned. - When to change it
- If single tasks approach this, the work is a job queue, not a stream. Raising the value hides the problem and delays the detection of consumers that are really stuck.
retention.ms (topic)Default 604800000 (7 days)
- What it does
- Age after which log segments become eligible for deletion. Applies to the whole topic and acts on whole segments.
- When to change it
- Do not use it as a business expiry. Put expiry times in the event and check them in the consumer.
group.share.delivery.count.limit (broker)Default 5
- What it does
- For share groups: how many times a record is delivered before the broker stops redelivering it.
- When to change it
- Relevant only if you use share groups for queue-style work on Kafka 4.2 or later. Together with
group.share.record.lock.duration.ms(default 30 seconds) it bounds how long a failing record is retried.
Hands-on exercise
Feel the parallelism cap, then sort seven requirements
See in a simulated consumer group why more workers do not help a queue-shaped workload, then make fit or no-fit decisions for a small product.
Steps · 0 of 6 done
Six workers, three partitions
Simulation · not a real clusterAdd consumers beyond the partition count and see what the extra ones do. Then crash one and add partitions.
What you should see
With 3 partitions, consumers 4 to 6 receive no partition and the lag does not fall faster: a consumer group's parallelism is capped by the partition count. After the crash an idle consumer takes over partition 0. For the startup: (a) direct call; (b) Kafka, key = order id, one group per reader; (c) object storage plus a reference event; (d) a scheduler or a timestamped row checked by a job, with the outcome published as an event; (e) an idempotency key in the refund service's database, whatever carries the request.
What can go wrong
A job queue built on a consumer group
- Symptom
- Lag grows on one or two partitions while most workers are idle, and long tasks are processed more than once.
- Cause
- Tasks of very different duration share partitions. A slow task blocks those behind it, and when it exceeds
max.poll.interval.msthe consumer is removed from the group and the task is redelivered to the next owner. - Fix
- Move the work to a queue broker or a jobs table, or to a share group on Kafka 4.2+. If it must stay on a consumer group, separate slow and fast tasks into different topics and use retry and dead letter topics.
Synchronous request/response through topics
- Symptom
- Under load or after a responder restart, users see timeouts, then the responder spends minutes answering requests whose callers have already given up.
- Cause
- Requests are retained and processed in order even when nobody is waiting any more. Nothing tells the caller that the responder is down.
- Fix
- Use HTTP or gRPC with a timeout and a circuit breaker for the question. If a reply over Kafka is unavoidable, add an expiry to each request and drop stale ones in the responder.
Raising the size limit until the files fit
- Symptom
- After
message.max.bytesandmax.request.sizewere raised to tens of MB, producers report buffer exhaustion, consumers use far more memory, and unrelated topics on the same brokers show higher latency. - Cause
- Every large record is buffered in the producer, replicated to each follower, and fetched whole by each consumer group. Batching, which most of Kafka's efficiency comes from, no longer applies.
- Fix
- Store the payload in object storage and publish its location, size and checksum. Keep records small enough that many fit in a batch.
Treating retention as a TTL
- Symptom
- A consumer that was behind applies a discount that ended hours ago, although the topic's retention is one hour.
- Cause
- Retention removes whole segments some time after they pass the age limit, and the newest segment is still being written. A record remains readable until its segment is deleted.
- Fix
- Carry
expiresAtin the event and check it in every consumer. Use retention only for disk management and replay windows.
Trusting "exactly-once" for a database or an API
- Symptom
- With transactions enabled and
read_committedconsumers, some customers are still charged or emailed twice after a consumer restart. - Cause
- The Kafka transaction covered the output topic and the offsets. The call to the payment provider happened outside it and was repeated when the record was redelivered.
- Fix
- Make the side effect idempotent: a unique business key checked in the database, or an idempotency key sent to the external API. See effectively-once processing.
One partition for safety
- Symptom
- A topic created with a single partition "to keep everything in order" cannot keep up, and adding consumers changes nothing.
- Cause
- One partition has one active consumer per group. The requirement was order per entity, which a key would have provided across many partitions.
- Fix
- Find the entity whose events must be ordered and use its id as the key. Keep a single partition only for a real total-order requirement, and size the consumer for it.
How to troubleshoot
Producer fails with RecordTooLargeException
Check
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name invoices --describe
- How to read it
- If the send failed immediately, before any request left the application, the record exceeded the producer's
max.request.size. If the broker rejected it, compare the size with the topic'smax.message.bytes(shown here if overridden, otherwise the broker'smessage.max.bytes, default 1048588). Either way, decide whether this payload belongs in a record at all.
Lag grows on one partition while other consumers are idle
Check
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group media-workers
- How to read it
- A large LAG on one partition whose CURRENT-OFFSET barely moves, next to partitions with lag 0, is head-of-line blocking: one slow or failing record. It is a sign of a queue-shaped workload or of a missing dead letter path, not of too few consumers.
The same long task runs repeatedly on different instances
Check
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group media-workers --state
- How to read it
- A group that keeps leaving the Stable state while the consumer log reports that the time between
poll()calls exceededmax.poll.interval.msmeans tasks outlast the poll interval. The partition is reassigned and the uncommitted task starts again elsewhere.
Adding consumers does not increase throughput
Check
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group media-workers --members
- How to read it
- The output lists each member with the number of partitions it holds. Members with 0 partitions are idle because the group already has one consumer per partition.
Callers of a request/reply flow over Kafka time out although every service is up
Check
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group stock-responder
- How to read it
- Lag on the request topic is waiting time added to every request. Any backlog here means callers are waiting on a queue, which a direct call would have turned into an immediate error or an immediate answer.
Key takeaways
- Kafka's limits come from its design: ordered partitions, one position per group and partition, retention per topic. They are not missing features that a setting turns on.
- If someone is waiting for the answer, make a direct call. Publish the resulting fact to Kafka for everyone else.
- Independent background jobs with priority, delay or uneven duration are queue work. Use a queue broker or a jobs table; share groups (Kafka 4.2+) cover per-record acknowledgement but not priority, routing or delay.
- A record batch is limited to about 1 MB by default. Send a reference to large payloads, not the payload.
- Order exists per partition. Ask for order per key before accepting a single-partition topic.
- Retention is not a TTL and there is no delayed delivery: expiry and scheduling are application logic.
- Kafka transactions and producer idempotence end at Kafka's edge. Effects in databases and external APIs need idempotent consumers or an outbox.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A startup with two backend developers runs one Spring Boot application on PostgreSQL. It needs to generate PDF reports in the background: about 300 jobs a day, each taking 5 to 120 seconds. Paying customers' jobs should run before free-tier jobs, and a failed job should be retried after 10 minutes. Nothing else needs to know about these jobs.
What do you recommend?
Scenario 2 of 3
An insurance company wants claim events (submitted, assessed, approved, paid) to reach four systems: payouts, fraud scoring, customer notifications and a reporting warehouse. Volume is about 50 events per second. Events of one claim must be processed in order, and the fraud team wants to re-run new models over the last 90 days. Some events come with scanned documents of 5 to 15 MB.
Does Kafka fit, and in what form?
Scenario 3 of 3
In a ride-hailing app, the passenger taps "Get price" and must see a fare within 300 ms. The pricing team proposes: the API gateway publishes a price-requests record, the pricing service consumes it and publishes to price-replies, and the gateway waits for the matching reply. They also want every quoted price available to analytics and to the fraud team.
Which design do you choose?
References
References: Apache Kafka documentation: Design (consumer position, message delivery semantics, the share consumer) ↗ · Apache Kafka documentation: Broker configs (message.max.bytes, replica.fetch.max.bytes, share group settings) ↗ · Apache Kafka documentation: Producer configs (max.request.size, linger.ms, enable.idempotence) ↗ · Apache Kafka documentation: Consumer configs (max.partition.fetch.bytes, max.poll.interval.ms) ↗ · Apache Kafka documentation: Upgrading (share groups production-ready in 4.2, linger.ms default changed in 4.0) ↗ · Apache Kafka documentation: KRaft (controller quorum size, combined mode) ↗ · KIP-932: Queues for Kafka ↗ · Spring for Apache Kafka reference: Sending messages (ReplyingKafkaTemplate) ↗ · Spring for Apache Kafka reference: Non-blocking retries, how the pattern works ↗ · RabbitMQ documentation: Priority queues ↗ · RabbitMQ documentation: Time-to-live and expiration ↗ · Amazon SQS developer guide: Delay queues ↗ · PostgreSQL documentation: SELECT, the locking clause (SKIP LOCKED) ↗ · Enterprise Integration Patterns: Claim Check ↗ · Report a problem