Skip to content

Path 1 · lesson 6 of 6

Foundation17 minApache Kafka 4.4 documentation (KRaft only; share groups production-ready since 4.2) · Java 21 · Spring Boot 3.4 with Spring for Apache Kafka 3.3 (kafka-clients 3.8 to 3.9)

When not to use Kafka

The limits of a partitioned log: seven situations where Kafka is the wrong tool or needs an adaptation, and what to use or do instead.

What you will learn

  • Recognise seven requirements that a partitioned log does not meet out of the box
  • Explain each limit from how partitions, offsets and retention work, not from a list of rules
  • Choose an alternative (a direct call, a queue broker, a database table, object storage) or an adaptation that keeps Kafka
  • Estimate what running Kafka costs a small team before adopting it
  • Make and defend a fit or no-fit decision for a concrete system

Before you start

The concept

The previous lesson showed how much one log can do. This one is about where it stops. Every strength of Kafka comes from the same design: records are appended to a partition, stay there for a retention period, and each consumer group moves a single position forward through them. Every limit comes from that design too. If a requirement needs something the log does not have (a per-message decision by the broker, an immediate answer, one order across everything), you either adapt the design around it or pick a different tool.

RequirementWhy the log strugglesUse instead, or adapt
A simple task queue, with per-message routing or priorityA partition is read in offset order by one consumer of a group. A slow task holds back everything behind it, and the broker cannot pick "the urgent one" or route by content.A queue broker such as RabbitMQ or Amazon SQS, or a jobs table in your database. In Kafka: share groups (4.2+) for per-record acknowledgement, separate topics per priority.
Request/response where a caller waits for the answerTwo asynchronous hops, a reply topic, correlation ids and a timeout, to imitate what one HTTP or gRPC call does directly.A synchronous call for the question. Publish an event afterwards for everyone else.
A small team with low volumeA production cluster is several servers to patch, monitor, secure and upgrade, whatever the traffic.A managed queue, a database table with an outbox, or a managed Kafka service if you need the log semantics.
Large payloadsA record batch is limited to about 1 MB by default, and big records hurt batching, memory and replication for everyone on the cluster.Store the file in object storage and send a small event with its location (claim check).
Strict global orderingOrder is guaranteed only within a partition. One total order means one partition, which means one active consumer per group.Order per key, which is usually what the business needs. If truly global: a single partition with the throughput limit accepted, or a database sequence.
Per-message TTL or delayed deliveryRetention is set per topic and removes whole segments. A consumer reads what is next in the log; nothing can be held back until later.An expiry time in the event that the consumer checks; retry topics with back-off; a scheduler or a broker with native delay.
Exactly-once side effects outside KafkaKafka transactions cover writes to Kafka and consumer offsets. A database row, an HTTP call or an email is outside them.Idempotent consumers, a transactional outbox, or storing the offset in the same database transaction as the result.
Seven requirements to check before choosing Kafka. Only some of them rule Kafka out; the others change the design.

Why it exists

Kafka was built for high-volume streams read by many independent consumers. To get there, it moved work out of the broker. The broker does not track which messages each consumer has finished, does not filter or route by content, and does not delete a message when it is consumed. It appends, replicates and serves byte ranges. That is why it is fast and why consumers can replay. It is also why features that depend on the broker knowing about individual messages are missing.

Queue broker (RabbitMQ, SQS)

  • The broker hands each message to one worker and tracks its acknowledgement
  • A slow or failed message does not block the others; it is redelivered or dead-lettered on its own
  • Routing by key or header, priorities, per-message TTL and delay are broker features
  • A message is gone once acknowledged: no replay, no second independent reader without another queue

Kafka with consumer groups

  • A consumer group tracks one position per partition; a commit means "everything before this offset is done"
  • Records in a partition are processed in order, so one slow record delays those behind it
  • No routing, priority, TTL or delay per message in the broker
  • Records stay for the retention period: replay and any number of independent groups

The second reason this lesson exists is cost. Kafka is rarely the cheapest thing that works. Teams adopt it because a conference talk or a larger company uses it, and then spend months operating a cluster that carries a few messages per second. A relational database that the team already runs, already backs up and already understands can do the same job for a long time. Choosing Kafka should follow from a requirement the simpler option cannot meet: several independent consumers, replay, ordered streams per key at high volume, or integration through Connect.

How it works inside

1. Task queues: why one slow record blocks a partition

In a consumer group, each partition is assigned to exactly one consumer, and that consumer's progress is one number: the committed offset, which is the position of the next record to read. There is no way to say "I finished offset 12 but not offset 11". So a consumer works through a partition in order. If the record at offset 11 is a video that takes 90 seconds to transcode, the thumbnails at offsets 12 and 13 wait, even if other consumers in the group are idle. Adding consumers does not help beyond the partition count, because extra consumers get no partition.

It gets worse when a task is slower than max.poll.interval.ms (default 5 minutes). A consumer that does not call poll() again within that time is considered failed, the group rebalances, the partition goes to another consumer, and the same slow record is delivered again from the last committed offset. A queue broker handles this case per message: the others are delivered to other workers and the failed one is retried or dead-lettered alone. Kafka applications rebuild that with retry topics and dead letter topics, covered in the reliability path.

2. Request/response: the wrong shape, not only the wrong speed

A caller that needs an answer must publish a request, wait on a reply topic, match the reply by a correlation id and give up after a timeout. Spring provides ReplyingKafkaTemplate for this, so it is possible. But each direction passes through producer batching (linger.ms is 5 ms by default since 4.0), replication when acks=all, and the consumer's fetch. Worse, the failure behaviour is wrong for a waiting user: if the responder is down, requests queue up and are answered minutes later, long after the caller timed out. A direct call fails fast and tells the caller so. Use Kafka for the fact that follows ("order placed"), not for the question that precedes it ("is this item in stock?").

3. Operational cost: the fixed part does not shrink with traffic

A cluster that survives one server failure without losing acknowledged data needs replication factor 3, so at least three brokers, and a controller quorum of three so that a majority survives one loss. The documentation advises against combining broker and controller roles in critical deployments. Add monitoring of lag, under-replicated partitions and disk, security (TLS, authentication, ACLs), capacity planning, and rolling upgrades about three times a year if you follow the project's release rhythm. A managed service removes the servers but not the design work: keys, partition counts, consumer error handling and idempotency remain yours.

4. Large payloads: the limit is on the batch, and it is about 1 MB

The broker setting message.max.bytes defaults to 1048588 bytes and is, in the documentation's words, the largest record batch size allowed, measured after compression. The producer has its own cap, max.request.size, default 1048576 bytes. Both can be raised, per topic with max.message.bytes. The reason not to is what large records do to the rest of the system: each one occupies producer buffer memory, travels to every replica, sits in the page cache, and is fetched whole by every consumer group. A 20 MB PDF read by four groups from three replicas is moved many times for a file that most readers only need a link to.

5. Global ordering: one partition, one consumer

Kafka guarantees order only within a partition, for records as the broker appended them. A topic with six partitions has six independent orders. If you truly need a single order across all records (a strictly sequential ledger, for example), the topic must have one partition. Then one consumer per group does all the work, throughput is what one consumer can process, and a single broker leads all writes. Before accepting that, ask whether order per account, per order or per device would do. It nearly always does, and it scales.

6. TTL and delay: retention is not expiry

retention.ms is a topic setting. The broker deletes whole log segments once they are old enough, so a record can live noticeably longer than the configured time, and every record in a topic shares the same setting. A consumer that is behind will still receive a promotion that ended an hour ago. And because a consumer always reads the next offset, there is no "deliver this at 09:00". Adaptations: put expiresAt in the event and let the consumer drop stale ones; for delays, Spring's non-blocking retry topics forward a record to a retry topic with a due timestamp and pause that partition until it is due, which works for a few fixed delays but not for arbitrary schedules. Amazon SQS, by comparison, supports a delivery delay per queue or per message of up to 15 minutes, and RabbitMQ supports a TTL per message.

7. Side effects outside Kafka

This last point is not a reason to avoid Kafka, because every asynchronous system has it. It is a reason not to choose Kafka *because* of the phrase "exactly-once", and to budget for idempotent consumers from the first design.

Architecture

Drawing diagram…

Decision flowchart. If a caller is waiting for the answer, use a direct call. Otherwise, if several independent consumers need the data or it must be replayable, Kafka is a candidate. If not, background work needing priority, delay or per-task routing goes to a queue broker or jobs table, and low volume with nobody to run a cluster goes to the existing database or a managed queue. For a Kafka candidate, three further questions lead to adaptations: large payloads are stored elsewhere with a reference, a need for total order is reduced to order per key or one partition, and effects outside Kafka need an idempotent consumer or an outbox.

flowchart TD
  START["New interaction between two parts of the system"]
  Q1{"Is a user or caller waiting for the answer?"}
  R1["Use a direct call - HTTP or gRPC"]
  Q2{"Do several independent consumers need it, or must it be replayable?"}
  Q3{"Is it background work with priority, delay or per-task routing?"}
  R2["Use a queue broker or a jobs table"]
  Q4{"Low volume and nobody to run a cluster?"}
  R3["Use the database you have - outbox table or a managed queue"]
  K["Kafka is a candidate - now check the adaptations"]
  Q5{"Payloads above about 1 MB?"}
  A1["Store the file elsewhere and send a reference"]
  Q6{"Need one order across all records?"}
  A2["Prefer order per key - else one partition and one consumer"]
  Q7{"Must an effect outside Kafka happen only once?"}
  A3["Idempotent consumer or outbox"]
  FIT["Kafka fits - design keys, partitions and consumer groups"]
  START --> Q1
  Q1 -->|"yes"| R1
  Q1 -->|"no"| Q2
  Q2 -->|"no"| Q3
  Q3 -->|"yes"| R2
  Q3 -->|"no"| Q4
  Q4 -->|"yes"| R3
  Q4 -->|"no"| K
  Q2 -->|"yes"| K
  K --> Q5
  Q5 -->|"yes"| A1
  Q5 -->|"no"| Q6
  A1 --> Q6
  Q6 -->|"yes"| A2
  Q6 -->|"no"| Q7
  A2 --> Q7
  Q7 -->|"yes"| A3
  Q7 -->|"no"| FIT
  A3 --> FIT
A decision path for one interaction. The first three questions can rule Kafka out; the last three only change the design.
  • Apply the flowchart to one interaction at a time, for example "checkout asks for the stock level" or "payment tells others the charge succeeded". A system has many interactions and they will not all take the same exit.
  • The first question removes the most common misuse. If someone is waiting for the answer, call the service. You can still publish an event about what happened.
  • The second question is the positive test for Kafka: several independent readers, or the need to read the same data again later. If neither applies, a queue is simpler.
  • The questions below the Kafka box do not reject it. They add a design element: a claim check for large payloads, a keying decision for order, idempotency for outside effects.
  • Volume alone is not a question in the chart. High volume makes Kafka attractive, but low volume does not forbid it if you need replay or fan-out and use a managed service.

Message flow

Drawing diagram…

Sequence diagram. An upload API writes a 90 second video job and two 1 second thumbnail jobs to partition 0 of a media-jobs topic. Worker 1 owns partition 0 and receives all three. Worker 2 owns partition 1, which is empty, and stays idle. The thumbnails wait behind the video and all three are committed after 92 seconds. In the second half, a queue broker delivers the video to worker 1 and the thumbnails to worker 2, which finishes them in 2 seconds.

sequenceDiagram
  participant A as Upload API
  participant P as media-jobs partition 0
  participant W1 as Worker 1 owns partition 0
  participant W2 as Worker 2 owns partition 1
  A->>P: Offset 11 transcode video - 90 seconds
  A->>P: Offset 12 make thumbnail - 1 second
  A->>P: Offset 13 make thumbnail - 1 second
  P-->>W1: Poll returns offsets 11 12 13
  Note over W1: Busy with offset 11
  Note over W2: Idle - partition 1 is empty
  Note over P: Offsets 12 and 13 wait behind 11
  W1->>P: After 92 seconds commit offset 14
  Note over A,W2: Same jobs through a queue broker
  A->>W1: Queue delivers video job
  A->>W2: Queue delivers both thumbnail jobs
  W2-->>A: Thumbnails acknowledged after 2 seconds
  W1-->>A: Video acknowledged after 90 seconds
Head-of-line blocking in a partition, and the same three jobs through a queue that hands each message to any free worker.
  1. All three jobs landed in partition 0, either by key or by chance. Worker 1 owns that partition and receives them in offset order.
  2. Worker 2 is free but cannot take offsets 12 and 13: in a consumer group a partition belongs to one consumer at a time.
  3. Worker 1 commits offset 14 after finishing all three. If it crashed during the video, another worker would restart from the last committed offset and repeat the work.
  4. A queue broker tracks each message separately, so the thumbnails go to the idle worker. Kafka share groups (4.2+) behave like the lower half for records that need no ordering; classic consumer groups behave like the upper half.
  5. If the 90 seconds were 6 minutes, worker 1 would exceed the default max.poll.interval.ms, be removed from the group, and the video would be delivered again to the next owner of the partition.

In a real system

Take the academy's order system at a typical online shop and go through its interactions one by one. Some belong on Kafka and some do not.

InteractionDecisionWhy
Checkout page asks whether an item is in stockDirect call to the Inventory ServiceA user is waiting and needs an answer or a fast failure.
Order Service announces "order placed"Kafka: topic orders, key = order idPayment, Inventory, Notification and the warehouse each need it independently, in order per order, and must catch up after downtime.
Payment Service charges the cardKafka consumer plus an idempotency key at the payment providerThe record can be delivered twice. Kafka cannot make the charge happen once; the key can.
Invoice PDF, about 2 MB, sent to the customerPDF in object storage; invoices event carries its locationAbove the default batch limit, and most consumers only need to know that it exists.
"Send a reminder if the basket is still unpaid after 24 hours"A scheduler or a database table with a due timeKafka has no delayed delivery. A consumer would have to hold the record or re-publish it for a day.
Resize product images uploaded by staff, urgent ones firstA queue broker with priorities, or a jobs tableIndependent tasks, uneven duration, a priority, no need for replay or a second reader.
Voucher valid for 10 minutesKafka event with expiresAt; consumers ignore it after thatRetention cannot expire one record. The check belongs in the consumer.

Now make the shop smaller: two developers, one PostgreSQL database, a few hundred orders a day. The honest design has no Kafka in it yet. The Order Service writes the order and an outbox row in one transaction; a worker reads the outbox with SELECT ... FOR UPDATE SKIP LOCKED and calls the payment and email code. If the outbox rows are already shaped as events (type, key, payload), moving to Kafka later means replacing the worker with a connector or a publisher, not redesigning the services.

Java and Spring Boot

Two adaptations in code: the claim check for a large payload (store the file, publish a reference) and an expiry check in the consumer for events that are only valid for a while.

InvoicePublisher.javaJava
package com.example.shop.invoice;import java.net.URI;import java.security.MessageDigest;import java.security.NoSuchAlgorithmException;import java.util.HexFormat;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.stereotype.Service;@Servicepublic class InvoicePublisher {    /** The event is a few hundred bytes whatever the size of the PDF. */    public record InvoiceRendered(String orderId, URI pdfLocation, long sizeBytes, String sha256) {}    /** Thin wrapper over the object storage the company already uses. */    public interface BlobStore {        URI put(String objectKey, byte[] content, String contentType);        byte[] get(URI location);    }    private static final String TOPIC = "invoices";    private final BlobStore blobStore;    private final KafkaTemplate<String, InvoiceRendered> kafkaTemplate;    public InvoicePublisher(BlobStore blobStore, KafkaTemplate<String, InvoiceRendered> kafkaTemplate) {        this.blobStore = blobStore;        this.kafkaTemplate = kafkaTemplate;    }    public void publish(String orderId, byte[] pdf) {        // 1. Store the file first, under a name derived from the order id:        //    a retry overwrites the same object instead of creating a second one.        URI location = blobStore.put("invoices/" + orderId + ".pdf", pdf, "application/pdf");        // 2. Publish the small event. Key = order id, like every other event of this order.        InvoiceRendered event = new InvoiceRendered(orderId, location, pdf.length, sha256(pdf));        kafkaTemplate.send(TOPIC, orderId, event).join();    }    private static String sha256(byte[] content) {        try {            return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(content));        } catch (NoSuchAlgorithmException e) {            throw new IllegalStateException("SHA-256 is required on every Java platform", e);        }    }}

Why it is written this way

  • Upload first, publish second. If the publish fails, an unreferenced file remains, which a storage lifecycle rule can clean up. The other order would publish a reference to a file that does not exist yet.
  • join() waits for the broker's acknowledgement and throws if the send finally fails, so the caller knows the event was not published. For a high-volume path you would handle the future asynchronously instead.
  • The checksum lets a consumer detect a file that was replaced or truncated after the event was written.
  • The file must outlive the event: object retention has to be at least the topic's retention, or a replay will find dead links.

Configuration

The settings behind the limits in this lesson. Defaults are from the Apache Kafka 4.4 configuration reference. Knowing them tells you when you have hit a design limit and not a bug.

message.max.bytes (broker) / max.message.bytes (topic)Default 1048588
What it does
The largest record batch the broker accepts, after compression if compression is enabled.
When to change it
Raise it per topic only for a measured, bounded need (a few MB at most), together with the producer's max.request.size. For files, use a claim check instead.
max.request.size (producer)Default 1048576
What it does
The largest request the producer will send, which is also effectively the cap on one uncompressed record batch. Checked in the client before anything is sent.
When to change it
A record above it fails in the producer with RecordTooLargeException. Treat that as a design signal before raising the number.
max.partition.fetch.bytes (consumer)Default 1048576 (1 mebibyte)
What it does
How much data per partition a fetch returns. Not an absolute maximum: if the first batch is larger, it is still returned so the consumer can make progress.
When to change it
You do not have to raise it for a consumer to read a larger record, but each such fetch then carries one batch per partition, which lowers throughput. The same holds for replica.fetch.max.bytes on brokers.
linger.ms (producer)Default 5
What it does
How long the producer waits for more records to fill a batch before sending. It was 0 before Kafka 4.0.
When to change it
One of several small delays on every hop. Lowering it to 0 on a latency-critical producer trades throughput for a few milliseconds; it does not make Kafka a request/response system.
max.poll.interval.ms (consumer)Default 300000 (5 minutes)
What it does
The longest allowed gap between two poll() calls before the consumer is considered failed and its partitions are reassigned.
When to change it
If single tasks approach this, the work is a job queue, not a stream. Raising the value hides the problem and delays the detection of consumers that are really stuck.
retention.ms (topic)Default 604800000 (7 days)
What it does
Age after which log segments become eligible for deletion. Applies to the whole topic and acts on whole segments.
When to change it
Do not use it as a business expiry. Put expiry times in the event and check them in the consumer.
group.share.delivery.count.limit (broker)Default 5
What it does
For share groups: how many times a record is delivered before the broker stops redelivering it.
When to change it
Relevant only if you use share groups for queue-style work on Kafka 4.2 or later. Together with group.share.record.lock.duration.ms (default 30 seconds) it bounds how long a failing record is retried.

Hands-on exercise

Feel the parallelism cap, then sort seven requirements

See in a simulated consumer group why more workers do not help a queue-shaped workload, then make fit or no-fit decisions for a small product.

Steps · 0 of 6 done

Six workers, three partitions

Simulation · not a real cluster

Add consumers beyond the partition count and see what the extra ones do. Then crash one and add partitions.

Take it further in the lab: Adding and removing consumers

What can go wrong

A job queue built on a consumer group
Symptom
Lag grows on one or two partitions while most workers are idle, and long tasks are processed more than once.
Cause
Tasks of very different duration share partitions. A slow task blocks those behind it, and when it exceeds max.poll.interval.ms the consumer is removed from the group and the task is redelivered to the next owner.
Fix
Move the work to a queue broker or a jobs table, or to a share group on Kafka 4.2+. If it must stay on a consumer group, separate slow and fast tasks into different topics and use retry and dead letter topics.
Synchronous request/response through topics
Symptom
Under load or after a responder restart, users see timeouts, then the responder spends minutes answering requests whose callers have already given up.
Cause
Requests are retained and processed in order even when nobody is waiting any more. Nothing tells the caller that the responder is down.
Fix
Use HTTP or gRPC with a timeout and a circuit breaker for the question. If a reply over Kafka is unavoidable, add an expiry to each request and drop stale ones in the responder.
Raising the size limit until the files fit
Symptom
After message.max.bytes and max.request.size were raised to tens of MB, producers report buffer exhaustion, consumers use far more memory, and unrelated topics on the same brokers show higher latency.
Cause
Every large record is buffered in the producer, replicated to each follower, and fetched whole by each consumer group. Batching, which most of Kafka's efficiency comes from, no longer applies.
Fix
Store the payload in object storage and publish its location, size and checksum. Keep records small enough that many fit in a batch.
Treating retention as a TTL
Symptom
A consumer that was behind applies a discount that ended hours ago, although the topic's retention is one hour.
Cause
Retention removes whole segments some time after they pass the age limit, and the newest segment is still being written. A record remains readable until its segment is deleted.
Fix
Carry expiresAt in the event and check it in every consumer. Use retention only for disk management and replay windows.
Trusting "exactly-once" for a database or an API
Symptom
With transactions enabled and read_committed consumers, some customers are still charged or emailed twice after a consumer restart.
Cause
The Kafka transaction covered the output topic and the offsets. The call to the payment provider happened outside it and was repeated when the record was redelivered.
Fix
Make the side effect idempotent: a unique business key checked in the database, or an idempotency key sent to the external API. See effectively-once processing.
One partition for safety
Symptom
A topic created with a single partition "to keep everything in order" cannot keep up, and adding consumers changes nothing.
Cause
One partition has one active consumer per group. The requirement was order per entity, which a key would have provided across many partitions.
Fix
Find the entity whose events must be ordered and use its id as the key. Keep a single partition only for a real total-order requirement, and size the consumer for it.

How to troubleshoot

Producer fails with RecordTooLargeException

Check

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name invoices --describe
How to read it
If the send failed immediately, before any request left the application, the record exceeded the producer's max.request.size. If the broker rejected it, compare the size with the topic's max.message.bytes (shown here if overridden, otherwise the broker's message.max.bytes, default 1048588). Either way, decide whether this payload belongs in a record at all.
Lag grows on one partition while other consumers are idle

Check

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group media-workers
How to read it
A large LAG on one partition whose CURRENT-OFFSET barely moves, next to partitions with lag 0, is head-of-line blocking: one slow or failing record. It is a sign of a queue-shaped workload or of a missing dead letter path, not of too few consumers.
The same long task runs repeatedly on different instances

Check

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group media-workers --state
How to read it
A group that keeps leaving the Stable state while the consumer log reports that the time between poll() calls exceeded max.poll.interval.ms means tasks outlast the poll interval. The partition is reassigned and the uncommitted task starts again elsewhere.
Adding consumers does not increase throughput

Check

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group media-workers --members
How to read it
The output lists each member with the number of partitions it holds. Members with 0 partitions are idle because the group already has one consumer per partition.
Callers of a request/reply flow over Kafka time out although every service is up

Check

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group stock-responder
How to read it
Lag on the request topic is waiting time added to every request. Any backlog here means callers are waiting on a queue, which a direct call would have turned into an immediate error or an immediate answer.

Key takeaways

  • Kafka's limits come from its design: ordered partitions, one position per group and partition, retention per topic. They are not missing features that a setting turns on.
  • If someone is waiting for the answer, make a direct call. Publish the resulting fact to Kafka for everyone else.
  • Independent background jobs with priority, delay or uneven duration are queue work. Use a queue broker or a jobs table; share groups (Kafka 4.2+) cover per-record acknowledgement but not priority, routing or delay.
  • A record batch is limited to about 1 MB by default. Send a reference to large payloads, not the payload.
  • Order exists per partition. Ask for order per key before accepting a single-partition topic.
  • Retention is not a TTL and there is no delayed delivery: expiry and scheduling are application logic.
  • Kafka transactions and producer idempotence end at Kafka's edge. Effects in databases and external APIs need idempotent consumers or an outbox.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A startup with two backend developers runs one Spring Boot application on PostgreSQL. It needs to generate PDF reports in the background: about 300 jobs a day, each taking 5 to 120 seconds. Paying customers' jobs should run before free-tier jobs, and a failed job should be retried after 10 minutes. Nothing else needs to know about these jobs.

What do you recommend?

Scenario 2 of 3

An insurance company wants claim events (submitted, assessed, approved, paid) to reach four systems: payouts, fraud scoring, customer notifications and a reporting warehouse. Volume is about 50 events per second. Events of one claim must be processed in order, and the fraud team wants to re-run new models over the last 90 days. Some events come with scanned documents of 5 to 15 MB.

Does Kafka fit, and in what form?

Scenario 3 of 3

In a ride-hailing app, the passenger taps "Get price" and must see a fare within 300 ms. The pricing team proposes: the API gateway publishes a price-requests record, the pricing service consumes it and publishes to price-replies, and the gateway waits for the matching reply. They also want every quoted price available to analytics and to the fraud team.

Which design do you choose?

References

References: Apache Kafka documentation: Design (consumer position, message delivery semantics, the share consumer) ↗ · Apache Kafka documentation: Broker configs (message.max.bytes, replica.fetch.max.bytes, share group settings) ↗ · Apache Kafka documentation: Producer configs (max.request.size, linger.ms, enable.idempotence) ↗ · Apache Kafka documentation: Consumer configs (max.partition.fetch.bytes, max.poll.interval.ms) ↗ · Apache Kafka documentation: Upgrading (share groups production-ready in 4.2, linger.ms default changed in 4.0) ↗ · Apache Kafka documentation: KRaft (controller quorum size, combined mode) ↗ · KIP-932: Queues for Kafka ↗ · Spring for Apache Kafka reference: Sending messages (ReplyingKafkaTemplate) ↗ · Spring for Apache Kafka reference: Non-blocking retries, how the pattern works ↗ · RabbitMQ documentation: Priority queues ↗ · RabbitMQ documentation: Time-to-live and expiration ↗ · Amazon SQS developer guide: Delay queues ↗ · PostgreSQL documentation: SELECT, the locking clause (SKIP LOCKED) ↗ · Enterprise Integration Patterns: Claim Check ↗ · Report a problem