Example project
Order flow: four Spring Boot services choreographed over Kafka
How this code was verified
NotificationRetryIT (EmbeddedKafka) passes. The four services were started together against a real Apache Kafka 3.9.2 broker (single node, KRaft, topic auto-creation off) and PostgreSQL 17, both run as local processes, and scripts/demo.sh produced the four expected order statuses. Republishing the outbox, a gateway outage ending in the dead letter topic, a non-JSON record, and the bounce-1 retry topic path were each run and behaved as chapter 10 describes. What was not done: Docker was not available. docker-compose.yml has never been started. The two Testcontainers tests, OrderOutboxIT and PaymentIdempotencyIT, compile but were not executed as written; their test bodies passed against the local broker and database with the container declarations removed, so the @Container and @ServiceConnection wiring itself is unverified. The other experiments in chapter 10 (stopping Kafka, kill -9, a second instance, the six mouse orders, removing the acknowledgement) describe expected behaviour and were not run. Nothing was load tested. What to do first: with Docker running, run mvn verify in the project root. If it fails, the Testcontainers setup or the Compose file is the most likely place. Treat this as teaching code: issues may exist, and the last chapter lists what a production system would still need.Versions
| Component | Version | Note |
|---|---|---|
| Java | 21 | Built and run with Eclipse Temurin 21.0.13. The code uses records, sealed interfaces and pattern matching for switch. |
| Spring Boot | 3.5.16 | The newest 3.x release on Maven Central when this was written. It manages every library version below. |
| Spring for Apache Kafka | 3.3.16 | Managed by Spring Boot 3.5.16; no version in any pom.xml. |
| Apache Kafka clients (kafka-clients) | 3.9.2 | Managed by Spring Boot 3.5.16. |
| Apache Kafka broker | 3.9.2 | Docker image apache/kafka:3.9.2, one node in KRaft combined mode, no ZooKeeper. Kafka 4.x is KRaft-only and works with these clients too. |
| Testcontainers | 1.21.4 | Managed by Spring Boot. org.testcontainers.kafka.KafkaContainer with the apache/kafka image, and PostgreSQLContainer. |
| PostgreSQL | 17 | Docker image postgres:17-alpine; JDBC driver 42.7.11 and Flyway 11.7.2 managed by Spring Boot. |
| Test libraries | JUnit Jupiter 5.12.2, Mockito 5.17.0, AssertJ 3.27.7, Awaitility 4.2.2 | All managed by Spring Boot. |
| Apache Maven | 3.9.9 | Spring Boot 3.5 needs Maven 3.6.3 or newer. |
Architecture
Drawing diagram…
Flowchart. A client posts an order to the Order Service, which stores it in its own database and publishes to the order-placed topic through an outbox relay. The Payment Service reads order-placed and writes payment-completed or payment-failed. The Inventory Service reads payment-completed and writes stock-reserved or stock-rejected. The Payment Service also reads stock-rejected to refund. The Order Service reads all four result topics to update the order status. The Notification Service reads payment-failed, stock-reserved and stock-rejected. Each service has its own PostgreSQL database.
flowchart LR
client["Client"] -->|"POST /api/orders"| order["Order Service"]
order --- ordersdb[("orders DB: orders, outbox_events")]
payment["Payment Service"] --- paymentsdb[("payments DB: payments, processed_events")]
inventory["Inventory Service"] --- inventorydb[("inventory DB: stock_items, reservations")]
notification["Notification Service"] --- notificationsdb[("notifications DB: sent_notifications")]
subgraph kafka["Kafka: one topic per event type, record key is the order id"]
tplaced["orders.order-placed.v1"]
tcompleted["payments.payment-completed.v1"]
tfailed["payments.payment-failed.v1"]
treserved["inventory.stock-reserved.v1"]
trejected["inventory.stock-rejected.v1"]
end
order -->|"outbox relay"| tplaced
tplaced --> payment
payment --> tcompleted
payment --> tfailed
tcompleted --> inventory
inventory --> treserved
inventory --> trejected
trejected -->|"refund"| payment
tcompleted --> order
tfailed --> order
treserved --> order
trejected --> order
tfailed --> notification
treserved --> notification
trejected --> notificationThe order flow
Drawing diagram…
Sequence diagram of a successful order. The client posts an order; the Order Service inserts the order and an outbox row in one transaction and answers 201 with status PLACED. The outbox relay publishes OrderPlaced to Kafka, waits for the acknowledgement and marks the row as published. The Payment Service consumes it, stores the payment, publishes PaymentCompleted and commits its offset. The Inventory Service consumes PaymentCompleted, reserves stock, publishes StockReserved and acknowledges. The Order Service consumes both results and ends at CONFIRMED. The Notification Service sends the confirmation email. The client reads the final status with a GET request.
sequenceDiagram autonumber participant C as Client participant O as Order Service participant ODB as orders DB participant K as Kafka participant P as Payment Service participant I as Inventory Service participant N as Notification Service C->>O: POST /api/orders O->>ODB: one transaction, insert order row and outbox row O-->>C: 201 Created with status PLACED Note over O,K: Outbox relay, runs every 500 ms O->>ODB: lock unpublished outbox rows O->>K: OrderPlaced to orders.order-placed.v1 K-->>O: acknowledged by all in-sync replicas O->>ODB: set published_at and commit K->>P: OrderPlaced P->>P: one transaction, claim event id, charge, store payment P->>K: PaymentCompleted to payments.payment-completed.v1 P->>K: commit offset of OrderPlaced K->>O: PaymentCompleted, order becomes PAID K->>I: PaymentCompleted I->>I: one transaction, conditional stock update, store reservation I->>K: StockReserved to inventory.stock-reserved.v1 I->>K: acknowledge, offset of PaymentCompleted is committed K->>O: StockReserved, order becomes CONFIRMED K->>N: StockReserved N->>N: claim event id and send the confirmation email C->>O: GET /api/orders/id O-->>C: 200 OK with status CONFIRMED
Chapter 1
The system and its event flow
A customer places an order. The order has to be paid, the stock has to be reserved, and the customer has to be told how it ended. Four services do this, and none of them calls another one. Each service reacts to an event on a Kafka topic, changes its own database, and publishes what it decided as a new event. This style is called choreography: there is no central coordinator, and the flow is the sum of the reactions.
| Service | Reacts to | Decides | Publishes |
|---|---|---|---|
| Order (port 8081) | POST /api/orders, and all four result topics | The order's status | orders.order-placed.v1 |
| Payment (8082) | orders.order-placed.v1, inventory.stock-rejected.v1 | Charge or decline; refund | payments.payment-completed.v1, payments.payment-failed.v1 |
| Inventory (8083) | payments.payment-completed.v1 | Reserve stock or reject | inventory.stock-reserved.v1, inventory.stock-rejected.v1 |
| Notification (8084) | payments.payment-failed.v1, inventory.stock-reserved.v1, inventory.stock-rejected.v1 | Which email to send | Nothing (only its own retry topics) |
The three ways an order can end
- Confirmed. Payment completed, stock reserved. The order goes
PLACED,PAID,CONFIRMED. - Payment failed. The gateway declined. The order goes
PLACED,PAYMENT_FAILED. Inventory never hears about it. - Rejected. Payment completed, then there was no stock. The order goes
PLACED,PAID,REJECTED, and the payment service refunds because it also listens toinventory.stock-rejected.v1. That refund is a compensating action: there is no transaction across services to roll back, so a step that already succeeded is undone by another step.
One rule, applied four times
No service can write to its database and to Kafka atomically. They are two systems, and Kafka transactions cover Kafka writes and offset commits only, never a PostgreSQL commit. So every service in this project follows the same order of work: commit the database first, publish second, commit the consumer offset last. A crash at any point then leads to a repeat, never to a loss. Repeats are made harmless by making every consumer idempotent. The chapters that follow show four different ways to do that.
| Service | How it publishes | How it stays idempotent | How it retries |
|---|---|---|---|
| Order | Transactional outbox and a relay | State machine that only moves forward | In place, short back-off, then dead letter topic |
| Payment | Directly, after the database commit | processed_events table and unique order_id | In place, exponential back-off, then dead letter topic |
| Inventory | Directly, then manual acknowledgement | Reservation row keyed by order id | In place, exponential back-off, then dead letter topic |
| Notification | Does not publish | sent_notifications claim per event id | Non-blocking retry topics, then dead letter handler |
Files in this chapter
Chapter 2
The event contract
The common-events module is the only code the four services share. It holds five Java records, the topic names, and the logical type ids. It has no dependencies: no Spring, no Kafka client, no Jackson.
What is on the wire
- Key: the order id as a string.
- Value: the event as JSON. Instants are ISO-8601 strings, amounts are JSON numbers read into
BigDecimal. - Header
__TypeId__: a logical name such asorderPlaced. The consumer maps it to a class withspring.json.type.mapping.
__TypeId__:paymentCompletedkey: 3045dd5e-a323-4386-8c01-7ad80c6bf591value: {"eventId":"95e26fef-1e95-4fe4-87ab-8d246575d5ef","orderId":"3045dd5e-a323-4386-8c01-7ad80c6bf591", "customerId":"customer-42","paymentId":"3f333e51-ae50-4283-af0d-321dd7971c1c","amount":179.80, "currency":"EUR","sku":"SKU-KEYBOARD","quantity":2,"occurredAt":"2026-11-15T13:31:17.118229545Z"}Why it is written this way
- Captured from a run of this project (line breaks added). The type header holds the logical id, not
com.example.events.PaymentCompleted.
Events carry the data consumers need
PaymentCompleted contains the SKU and the quantity, although the payment service has no use for them. It copies them from OrderPlaced so that the inventory service can do its work from the event alone. The alternative, an event with only ids and a REST call back to the order service, would make inventory fail whenever the order service is down. The price is a larger event and a field the payment service must pass along faithfully.
Shared library or schema registry
Shared contract library (this project)
- One Maven module with Java records. Every service compiles against it.
- Simple: no extra infrastructure, and the compiler checks producers and consumers.
- Java only. A Python or Go consumer gets nothing from it.
- Compatibility is a matter of discipline. Nothing stops someone from renaming a field and releasing.
- Services are coupled at build time. If upgrading the library is mandatory for everyone at once, the services are no longer independently deployable.
Schemas in a registry
- Avro, Protobuf or JSON Schema files, stored in a schema registry. Each service generates its own classes.
- Works for any language.
- The registry rejects an incompatible schema when the producer registers it, before a single bad record is written.
- Records carry a schema id instead of field names, so they are smaller.
- One more highly available system to run, and a build step for code generation.
A shared library is a reasonable start for a handful of Java services owned by one team. The rules that make it safe are the same rules a registry enforces: only add optional fields, never rename or remove a field, never change a field's meaning, and put a breaking change on a new topic (.v2). EventJsonContractTest pins the field names and checks that a consumer ignores a field it does not know.
Files in this chapter
Chapter 3
Order service and the transactional outbox
Placing an order has two effects: a row in the orders table and an OrderPlaced event on a topic. The whole system depends on these two never disagreeing. An order without an event is never paid. An event without an order charges a customer for nothing.
Why not save, then send
| Approach | What goes wrong |
|---|---|
Save the order, commit, then kafkaTemplate.send | The process dies after the commit and before the send. The order exists, the event is never published, and nothing will ever retry it. |
| Send inside the transaction, then commit | The send succeeds and the commit fails (constraint, connection loss, deadlock). Payment charges for an order that does not exist. |
| Send first, save afterwards | The same as the previous row, with a longer gap. |
| A Kafka transaction around both | A Kafka transaction makes several Kafka writes and offset commits atomic. PostgreSQL is not a participant. Spring can start both transactions and commit one after the other, which narrows the gap and does not remove it. |
The outbox
The fix is to stop writing to two systems. OrderService.placeOrder writes two rows in one PostgreSQL transaction: the order, and a row in outbox_events that holds the event as JSON. Either both rows are committed or neither is. Publishing becomes a separate, repeatable job.
OutboxRelay.publishPendingruns every 500 ms. It selects the oldest unpublished rows withFOR UPDATE SKIP LOCKED.- It sends each row with
KafkaTemplateand waits until the broker has acknowledged all of them. - It sets
published_atand commits. - If anything fails between steps 2 and 3, the transaction rolls back, the rows stay unpublished, and the next run sends them again.
Decisions inside the relay
- Wait for the acknowledgement.
KafkaTemplate.sendreturns aCompletableFuture. Marking rows before the futures complete would mark records that are still in the producer's buffer. SKIP LOCKED. Two instances of the service can both run the relay. Each locks a different batch, so neither waits and neither double-publishes.- Stored bytes. The payload is serialised when the order is placed. The relay sends a
byte[]and adds the__TypeId__header itself.KafkaProducerConfigmakes the producer passbyte[]values through and serialise everything else as JSON. - Timeouts. The producer's
delivery.timeout.ms(8 s) is shorter than the relay's wait (10 s), so the relay never gives up on a record that Kafka is still trying to deliver.
Consuming the results
OrderResultListener subscribes to the four result topics with one consumer group and maps each event to a target status. OrderStatus gives each status a rank, and Order.transitionTo only moves to a higher rank and never leaves a terminal state. Two consequences follow. A duplicate result changes nothing, so no deduplication table is needed. And a PaymentCompleted that is consumed after StockReserved is ignored instead of moving a confirmed order back to PAID.
Results for one order can be handled by two consumer threads at once, because they come from different topics. The @Version column on Order makes the second writer fail with an optimistic locking error. The error handler in KafkaErrorHandlingConfig redelivers the record after 0.5 seconds, and the second attempt reads the state the first writer committed.
Files in this chapter
Chapter 4
Payment service and the idempotent consumer
The previous chapter ended with a promise: OrderPlaced is published at least once. So the payment service will, sooner or later, receive the same event twice. It also receives events twice for its own reasons: the consumer crashes after processing and before committing the offset, or a rebalance moves the partition while a record is in progress. Charging a customer twice is not acceptable, so the consumer has to be idempotent: processing an event a second time must have no further effect.
The mechanism
PaymentProcessor.processopens a database transaction.- It inserts the event id into
processed_eventswithON CONFLICT DO NOTHING. One row inserted means this is the first delivery. - It looks for a payment with this order id. If one exists, it returns it and does nothing else.
- Otherwise it calls the gateway, stores a
COMPLETEDorFAILEDpayment, and commits. The claim and the payment commit together.
The claim is only useful because it is in the same transaction as the payment. If the gateway call throws, the transaction rolls back and the claim disappears with it, so the redelivery is treated as a first delivery. If the claim were written to Redis, or in a separate transaction, a crash between the two writes would leave an event marked as processed with no payment, and the order would hang forever.
Publishing the result: the approach and its failure window
The payment service does not use an outbox. OrderPlacedListener does three things in a fixed order: commit the payment to PostgreSQL, publish the result and wait for the broker's acknowledgement, and return, after which the listener container commits the offset (ack-mode: record).
| Crash or failure | What happens next | Outcome |
|---|---|---|
| Before the database commit | The event is redelivered and processed as new | Correct |
| After the database commit, before the result is acknowledged | The event is redelivered. The processor finds the stored payment and returns it. The listener publishes the result. | Correct, but the result is late: it waits for the redelivery |
| After the result is acknowledged, before the offset commit | The same, and the result is published a second time with the same event id | A duplicate result, which every consumer of the payment topics tolerates |
| The gateway charged, then the database transaction rolled back | The event is redelivered and the gateway is called again with the same idempotency key | Correct only if the real gateway honours idempotency keys |
For the redelivery to produce a useful duplicate, the result must be the same event both times. Payment stores resultEventId when the row is created, and toResultEvent uses the stored id and the stored timestamp. The notification service can then recognise the second copy by its event id.
Error handling
A listener that throws hands the record to the container's error handler. KafkaErrorHandlingConfig declares a DefaultErrorHandler bean, which Spring Boot applies to the auto-configured listener container factory.
- Back-off.
ExponentialBackOffWithMaxRetries(4)with a 1 second initial interval and multiplier 2: redeliveries after 1, 2, 4 and 8 seconds. The consumer thread waits, so the partition is blocked and order is kept. Fifteen seconds is far belowmax.poll.interval.ms. - Dead letter topic. After the last attempt,
DeadLetterPublishingRecovererwrites the record toorders.order-placed.v1.payment-service-dlt, in the same partition number, with headers that hold the original topic, partition, offset and the exception. The offset is then committed and the partition continues. - Not retryable.
InvalidEventExceptionskips the back-off. Spring Kafka already treats deserialization and conversion exceptions this way. - Poison records.
ErrorHandlingDeserializercatches a record that is not valid JSON and passes the failure to the error handler with the original bytes. Without it, the failure would happen insidepoll()and the consumer would be stuck on that offset.
Files in this chapter
Chapter 5
Inventory service and manual acknowledgement
The inventory service consumes PaymentCompleted and decides whether the stock can be reserved. It has the same shape as the payment service, with two differences worth studying: how it prevents overselling, and who decides when an offset is committed.
Reserving stock without overselling
Three consumer threads may reserve the same SKU for three different orders at the same moment. StockRepository.tryReserve runs a single statement:
UPDATE stock_items SET available = available - :quantity, reserved = reserved + :quantity WHERE sku = :sku AND available >= :quantityWhy it is written this way
- One row updated means the stock was taken. Zero rows means there was not enough, and the order is rejected.
- PostgreSQL locks the row for the update and re-checks the
WHEREclause against the committed value, so two concurrent statements cannot both take the last unit.
Conditional update (used here)
- The check and the change are one statement.
- Never needs a retry: it either takes the stock or reports that there is none.
- Only works when the rule can be written in SQL.
Optimistic locking with @Version
- Read the row, decide in Java, write with
WHERE version = ?. - A concurrent change makes the write fail, and the operation is retried from the read.
- Right when the decision needs logic that does not fit in a
WHEREclause. The order service uses it for status changes.
Idempotency comes from the reservations table, whose primary key is the order id. InventoryService.reserve first looks for a reservation and returns it if there is one, so a redelivered event does not take stock twice. The stock update and the reservation insert are one transaction.
Manual acknowledgement
With spring.kafka.listener.ack-mode: manual the listener receives an Acknowledgment and the container commits an offset only after acknowledge() was called for that record. PaymentCompletedListener calls it on its last line, after the database commit and after the broker confirmed the result event.
| Ack mode | The offset is committed | Use it when |
|---|---|---|
BATCH (Spring's default) | After all records returned by one poll() were handled | Redelivering a whole batch after a crash is acceptable |
RECORD | After each record whose listener call returned normally | Processing is synchronous and one redelivered record is the most you want. Order, payment and notification use it. |
MANUAL | For records the listener acknowledged, together, once the records of the current poll are processed | The code should state explicitly when a record is done |
MANUAL_IMMEDIATE | Synchronously inside acknowledge() | The commit must happen before the next line runs, at the cost of a broker round trip per record |
For a listener that does everything synchronously, RECORD and MANUAL commit the same offsets. The reason to prefer manual acknowledgement here is that the commit is tied to a visible line of code that comes after the last side effect. With RECORD, returning from the method is the acknowledgement. If someone later makes the publish asynchronous to gain throughput, the method returns early and offsets are committed for results that may never be sent. With MANUAL, that change does not compile into a silent data loss: the acknowledge() call has to be moved deliberately, for example into the send's completion callback.
Files in this chapter
Chapter 6
Notification service and retry topics
The notification service turns three kinds of result into an email. The email sender is a stub that writes to the log. The service is the last link of the chain and the one place where a different retry strategy is the better choice.
Blocking and non-blocking retries
Retry in place (DefaultErrorHandler)
- The failed record is redelivered from the same partition after a pause.
- Nothing else in that partition is processed meanwhile. Order is preserved.
- Right when records depend on each other, as payments and stock changes of one order do.
- A dependency that is down for minutes stops the partition for minutes.
Retry topics (@RetryableTopic)
- The failed record is published to a retry topic and the original offset is committed.
- The partition continues with the next record at once.
- Right when records are independent, as emails to different customers are.
- A retried record is overtaken by later ones. Order within the key is lost.
A mail provider that rejects one address should not delay everybody else's confirmation. And if a customer's "order confirmed" email arrives a few seconds after it would have, nobody is harmed. So NotificationListener uses @RetryableTopic.
What the annotation sets up
| Topic (for inventory.stock-reserved.v1) | Read after | Attempt |
|---|---|---|
inventory.stock-reserved.v1 | immediately | 1 |
inventory.stock-reserved.v1.notification-service-retry-0 | 2 s | 2 |
inventory.stock-reserved.v1.notification-service-retry-1 | 4 s more | 3 |
inventory.stock-reserved.v1.notification-service-retry-2 | 8 s more | 4 |
inventory.stock-reserved.v1.notification-service-dlt | after attempt 4 failed | @DltHandler |
- Each retry topic gets its own listener container. It does not sleep: it pauses the partition until the record's due time and keeps polling, so the consumer stays in the group.
- The record is forwarded with the application's
KafkaTemplate. That is why a service that publishes no events still has producer settings and aKafkaProducerConfig. exclude = IllegalArgumentException.classsends an event with no customer message straight to the dead letter topic.dltStrategy = FAIL_ON_ERRORstops the dead letter handler from being retried if it throws.- The suffixes are set explicitly and contain the service name. With the default suffixes (
-retry,-dlt), two services that use retry topics on the same source topic would share them. - Spring creates the retry and dead letter topics through
KafkaAdmin, with the partition count and replication factor given in the annotation.
Duplicates
Upstream services publish at least once, and the retry machinery forwards at least once. Without protection, a customer could get the same email several times. NotificationService.notify claims the event id in sent_notifications with ON CONFLICT DO NOTHING and sends only if the claim succeeded. The claim and the send are in one database transaction, with the send last: a failed send rolls the claim back so that the retry can send.
Files in this chapter
Chapter 7
Configuration, topics and Docker Compose
Almost everything Kafka-related is configured in application.yml under spring.kafka. Spring Boot builds the producer factory, the KafkaTemplate, the consumer factory, the listener container factory and a KafkaAdmin from these properties. The Java configuration in each service adds only what properties cannot express: the value serializer, the error handler and the topics.
Producer settings
| Setting | Value | Why |
|---|---|---|
acks | all | The leader answers only when all in-sync replicas have the record. It is meaningful together with the topic's min.insync.replicas: with replication factor 3 and min.insync.replicas=2, an acknowledged record survives the loss of one broker. |
enable.idempotence | true | The broker discards a batch it already has when the producer retries. No duplicates and no reordering from producer retries. It is the default in current clients; it is written down because the system relies on it. |
delivery.timeout.ms | 8000 | The total time a send may take, retries included. Kept below the 10 second wait in the relay and the publishers. |
request.timeout.ms | 5000 | The producer requires delivery.timeout.ms >= linger.ms + request.timeout.ms and refuses to start otherwise. |
linger.ms | 5 | Wait up to 5 ms for more records to fill a batch. The outbox relay sends up to 50 records at once and benefits from it. |
key-serializer | StringSerializer | Keys are order ids as strings. |
Consumer and listener settings
| Setting | Value | Why |
|---|---|---|
group-id | the service name | Every service has its own group, so each one gets every record of the topics it reads. Instances of the same service share the group and split the partitions. |
enable-auto-commit | false | The Spring listener container commits offsets according to the ack mode. With auto commit the client would commit on a timer, possibly before the record was processed. |
auto-offset-reset | earliest | Used only when the group has no committed offset. A new service then starts from the oldest retained record instead of skipping everything published before it started. |
value-deserializer | ErrorHandlingDeserializer | Delegates to JsonDeserializer and turns a failure into something the error handler can deal with. |
spring.json.trusted.packages | com.example.events | The deserializer creates classes named by a header. It must not be possible to name an arbitrary class. |
spring.json.type.mapping | orderPlaced:com.example.events.OrderPlaced,... | Maps the logical ids in __TypeId__ to classes. |
listener.ack-mode | record (inventory: manual) | Commit after each handled record, so a crash repeats at most one. |
listener.concurrency | 3 | Three consumer threads, one per partition. More threads than partitions would be idle: a partition is read by one consumer of a group. |
isolation.level | not set (read_uncommitted) | No producer here uses Kafka transactions. A consumer of transactional producers sets read_committed to skip aborted records; consumers never read past the high watermark either way. |
Topics
Each service declares the topics it owns as NewTopic beans (KafkaAdmin.NewTopics): the ones it writes to, and its own dead letter topics. KafkaAdmin creates missing topics at startup. The broker in docker-compose.yml has topic auto-creation switched off, so a topic exists only because some service declared it.
- Partitions: 3. Enough to show parallel consumers. In production the number follows from the throughput and the consumer parallelism you need. It can be increased later, but then the same key maps to a different partition and per-key ordering breaks across the change.
- Replication factor: 1 in
application.yml, because there is one local broker. Production: 3, withmin.insync.replicas=2set on the topic (TopicBuilder.config(...)) or as a broker default. - Dead letter topics have the same partition count as their source topic, because the recoverer writes a dead letter to the same partition number.
- Who creates topics is a policy question. Many teams provision topics with infrastructure code and give applications no permission to create them. The beans are then still useful as a statement of what the service expects.
Docker Compose
- Kafka is the official
apache/kafkaimage, version 3.9.2, as a single node in KRaft combined mode: the same process is broker and controller, and there is no ZooKeeper. - Two listeners.
PLAINTEXT_HOST://localhost:9092for the services, which run on the host, andPLAINTEXT://kafka:19092for other containers such as Kafka UI. A broker tells clients which address to use after the first connection, and that address has to be reachable from where the client runs. - Internal topics (
__consumer_offsets, the transaction log) are set to replication factor 1. Their default is 3, which one broker cannot provide. - PostgreSQL is one container with four databases and four logins, created by
init-databases.sqlon the first start. - Kafka UI is optional (
--profile ui) and shows topics, records and consumer group lag.
Files in this chapter
Chapter 8
Observability and consumer lag
In an event-driven system a failure is rarely an error page. It is an order that stays PLACED. To see why, each service exposes metrics through Spring Boot Actuator and Micrometer, with a Prometheus endpoint at /actuator/prometheus.
| Metric | Where it comes from | What it tells you |
|---|---|---|
spring.kafka.listener | spring.kafka.listener.observation-enabled: true | A timer per listener: how many records were handled, how long each took, and how many ended in an error. |
spring.kafka.template | spring.kafka.template.observation-enabled: true | A timer per send, tagged with the destination topic. Sends to a -dlt topic show up here, which makes dead-lettering visible. |
kafka.consumer.fetch.manager.records.lag.max | The Kafka client's own metrics, bound by Spring Boot | The largest lag this consumer instance sees on its assigned partitions. |
kafka.producer.* | The Kafka client's own metrics | Record send rate, error rate, retry rate and request latency of the producer. |
notifications.dead.lettered | A counter in NotificationListener | Emails that were given up. Should stay at zero; alert when it does not. |
The two observation-enabled settings do more than timers. They create Micrometer observations, so when a tracing bridge is on the classpath (for example micrometer-tracing-bridge-otel), the template writes the trace context into record headers and the listener continues the trace. One trace then covers the HTTP request, the relay, and all four services.
Consumer lag
Lag is the distance between the newest offset in a partition (the log end offset) and the offset the consumer group has committed. It is the number of records waiting. A lag that grows means the consumers are slower than the producers, or stopped.
docker compose exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \ --bootstrap-server localhost:9092 \ --describe --group payment-serviceWhy it is written this way
- The output has one line per partition with
CURRENT-OFFSET(committed),LOG-END-OFFSETandLAG. - The committed offset is the next offset the group will read, so a fully caught-up group shows
LAG 0.
- Measure lag from outside the consumer.
records.lag.maxis reported by the consumer itself. A consumer that is dead reports nothing, and the metric disappears exactly when it matters. Kafka UI,kafka-consumer-groups.shor an exporter that reads committed offsets from the broker keep working when the consumer does not. - Blocking retries look like lag. While the payment service backs off on one record, the partition does not move and lag grows behind it. That is the cost of keeping order.
- Watch the outbox as well. Rows in
outbox_eventswithpublished_at IS NULLthat are older than a few seconds mean the relay cannot reach Kafka. That backlog is invisible to any Kafka metric, because the records are not in Kafka yet. - Watch the dead letter topics. A record there is an order that will not finish by itself.
Files in this chapter
Chapter 9
Testing
The code is arranged so that the interesting decisions are in plain classes: Order, PaymentProcessor, InventoryService, NotificationService. Listeners and configuration stay thin. That decides what kind of test covers what.
| Level | Tools | Covers | Runs in |
|---|---|---|---|
Unit (*Test) | JUnit 5, Mockito, AssertJ | State machine, idempotency decisions, order of calls in a listener, contract JSON | mvn test, milliseconds each, no Docker |
Wiring (NotificationRetryIT) | @SpringBootTest, @EmbeddedKafka, @MockitoBean, Awaitility | That @RetryableTopic really redelivers and the dead letter handler really runs | mvn verify, no Docker |
Integration (OrderOutboxIT, PaymentIdempotencyIT) | @SpringBootTest, Testcontainers (Kafka and PostgreSQL), @ServiceConnection, Awaitility | The real configuration against a real broker and database: Flyway, JPA mappings, serializers, type headers, the relay, deduplication | mvn verify, needs Docker |
Unit tests
OrderTestchecks the state machine, including a result that arrives out of order and every terminal state.PaymentProcessorTestchecks that a redelivery does not call the gateway, that a second event for a paid order does not charge, and that a gateway outage leaves the method as an exception.OrderPlacedListenerTestandPaymentCompletedListenerTestuse Mockito'sInOrderto pin the sequence: database, publish, acknowledge. They also assert that nothing is acknowledged when a step fails.EventJsonContractTestfails when a field of the contract is renamed.
Integration tests with Testcontainers
Both tests declare a PostgreSQLContainer and an org.testcontainers.kafka.KafkaContainer (the apache/kafka image) as static fields with @Container and @ServiceConnection. Spring Boot reads the containers' addresses and uses them instead of spring.datasource.* and spring.kafka.bootstrap-servers. Everything else comes from the service's real application.yml.
OrderOutboxITposts an order over HTTP and then readsorders.order-placed.v1with a plainKafkaConsumer. It asserts the key, the__TypeId__header and the JSON, and then that no unpublished outbox row is left.PaymentIdempotencyITsends the sameOrderPlacedtwice. It waits until twoPaymentCompletedrecords for that order are on the result topic, which proves both deliveries were handled, and then asserts that there is one payment row and that both results carry the same event id.
What these tests do not cover: killing a service between the database commit and the publish. That failure window is covered by reasoning in chapter 4 and by the unit tests of the redelivery path. The last chapter shows how to provoke it by hand.
Files in this chapter
Chapter 10
Running it, and what to try to break
You need JDK 21, Maven 3.6.3 or newer, and Docker. Copy the files into a directory using the paths shown, then:
# 1. Build everything and run all tests (the *IT tests start containers, so Docker must be running)mvn verify# 2. Infrastructure: Kafka and PostgreSQL (add "--profile ui" for Kafka UI on http://localhost:8080)docker compose up -d# 3. The four services, each in its own terminaljava -jar order-service/target/order-service-1.0.0-SNAPSHOT.jarjava -jar payment-service/target/payment-service-1.0.0-SNAPSHOT.jarjava -jar inventory-service/target/inventory-service-1.0.0-SNAPSHOT.jarjava -jar notification-service/target/notification-service-1.0.0-SNAPSHOT.jar# 4. One order for each path through the system./scripts/demo.sh# 5. Read a topic, with keys and headersdocker compose exec kafka /opt/kafka/bin/kafka-console-consumer.sh \ --bootstrap-server localhost:9092 --topic payments.payment-completed.v1 \ --from-beginning --property print.key=true --property print.headers=trueWhy it is written this way
mvn verify -DskipITsbuilds and runs the unit tests without Docker.- The demo script prints the expected and the actual final status of four orders:
CONFIRMED,PAYMENT_FAILED,REJECTED,CONFIRMED. docker compose down -vremoves the containers and the data, including the databases.
Experiments
Each experiment names what to do, what you should see, and which mechanism is responsible. They follow the academy's sequence: build it, break it, watch it, understand the fix.
| Break it | What you should see | Why |
|---|---|---|
Stop Kafka (docker compose stop kafka), place an order, start Kafka again | The POST still answers 201. The order stays PLACED, and the order service logs that an outbox batch was not acknowledged. After Kafka is back, the order completes. | The outbox. Accepting an order does not depend on the broker. |
Republish the outbox: UPDATE outbox_events SET published_at = NULL; in the orders database | Every OrderPlaced is sent again. The logs say "already has payment", "was already decided", "was already notified". No second payment, no change in stock, no second email. | Idempotent consumers, three different implementations. |
| Stop the payment service, place three orders, start it again | The orders wait as PLACED. kafka-consumer-groups.sh --describe --group payment-service shows lag 3. After the restart the lag returns to 0 and the orders complete. | Offsets are stored per group in Kafka. A stopped consumer resumes where it committed. |
Kill the payment service with kill -9 while orders are flowing | After the restart, some results may be published a second time. Count the rows in payments: one per order. | At-least-once delivery plus the processed_events and order_id checks. |
Place an order with "currency":"XXX" | The payment service logs five failed delivery attempts over about 15 seconds. Then the record appears in orders.order-placed.v1.payment-service-dlt with kafka_dlt-exception-* headers. Orders with keys in the same partition wait during those 15 seconds. | Blocking retries with exponential back-off, then the dead letter topic. |
Produce a line of text that is not JSON to orders.order-placed.v1 with kafka-console-producer.sh | The record goes to the dead letter topic at once, without retries, with its original bytes. The consumer continues with the next record. | ErrorHandlingDeserializer, and deserialization failures being not retryable. |
Place an order for customer bounce-1 | The order is CONFIRMED. The email fails four times: at once, then after 2, 4 and 8 seconds. Then the notification service logs "Giving up on the notification". Emails for other customers are not delayed. | Non-blocking retry topics and the @DltHandler. |
Order SKU-MOUSE (stock 5) six times with quantity 1 | Five orders are CONFIRMED and one is REJECTED, with its payment REFUNDED. stock_items.available is 0 and never negative. | The conditional update, and the compensating refund. |
Start a second instance of the payment service: java -jar ... --server.port=8092 | Both instances log a rebalance, and the three partitions are split between them (six consumer threads, three of them idle). | A group's parallelism is capped by the partition count. |
In PaymentCompletedListener, comment out acknowledgment.acknowledge(), rebuild, place orders, restart the inventory service | Everything seems to work until the restart. Then every PaymentCompleted since the last commit is delivered again, and the log fills with "was already decided". | With manual acknowledgement nothing is committed without the call. The idempotent consumer hides the mistake from customers, not from the logs. |
Files in this chapter
All files
root · pom.xml
<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <!-- The Spring Boot parent manages every version below it: Spring Kafka 3.3.16, kafka-clients 3.9.2, Testcontainers 1.21.4, Flyway, the PostgreSQL driver, JUnit, Awaitility. --> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>3.5.16</version> <relativePath/> </parent> <groupId>com.example.shop</groupId> <artifactId>order-flow</artifactId> <version>1.0.0-SNAPSHOT</version> <packaging>pom</packaging> <name>order-flow</name> <description>Order, Payment, Inventory and Notification services choreographed over Apache Kafka</description> <modules> <module>common-events</module> <module>order-service</module> <module>payment-service</module> <module>inventory-service</module> <module>notification-service</module> </modules> <properties> <java.version>21</java.version> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> </properties> <dependencyManagement> <dependencies> <dependency> <groupId>com.example.shop</groupId> <artifactId>common-events</artifactId> <version>${project.version}</version> </dependency> </dependencies> </dependencyManagement> <build> <pluginManagement> <plugins> <!-- Unit tests (*Test) run in "mvn test" through Surefire. Integration tests (*IT) need Docker and run in "mvn verify" through Failsafe. The Boot parent already binds the integration-test and verify goals; a module only has to declare the plugin. --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-failsafe-plugin</artifactId> </plugin> </plugins> </pluginManagement> </build></project>Why it is written this way
spring-boot-starter-parent3.5.16 is the only version number in the build that matters. It manages Spring for Apache Kafka 3.3.16,kafka-clients3.9.2, Testcontainers 1.21.4, Flyway, Hibernate and JUnit, so no module declares a version for them and they cannot drift apart.java.version21 is what makes the code's pattern matching forswitchover sealed types compile. Spring Boot 3.5 needs Java 17 or newer.common-eventsis listed independencyManagementso the four services depend on it without repeating the version.- Failsafe is declared once under
pluginManagement. Classes named*ITrun inmvn verify; classes named*Testrun inmvn test. A developer without Docker can still run the unit tests.
References: Spring for Apache Kafka reference (3.3): sending messages with KafkaTemplate ↗ · Spring for Apache Kafka reference (3.3): @KafkaListener annotation ↗ · Spring for Apache Kafka reference (3.3): message listener containers, committing offsets and ack modes ↗ · Spring for Apache Kafka reference (3.3): serialization, JsonSerializer, type mapping, ErrorHandlingDeserializer, DelegatingByTypeSerializer ↗ · Spring for Apache Kafka reference (3.3): handling exceptions, DefaultErrorHandler, DeadLetterPublishingRecoverer ↗ · Spring for Apache Kafka reference (3.3): non-blocking retries, @RetryableTopic configuration ↗ · Spring for Apache Kafka reference (3.3): transactions ↗ · Spring for Apache Kafka reference (3.3): testing applications, @EmbeddedKafka ↗ · Spring for Apache Kafka reference (3.3): monitoring, Micrometer observation ↗ · Spring Boot reference (3.5): Apache Kafka support and spring.kafka.* properties ↗ · Spring Boot reference (3.5): Testcontainers and @ServiceConnection ↗ · Spring Boot 3.5.16 managed dependency versions (spring-boot-dependencies POM) ↗ · Apache Kafka official Docker image (apache/kafka): tags and KRaft environment variables ↗ · Apache Kafka documentation: producer, consumer and topic configuration, delivery semantics ↗ · Testcontainers: Kafka module (org.testcontainers.kafka.KafkaContainer) ↗ · PostgreSQL documentation: SELECT ... FOR UPDATE SKIP LOCKED ↗ · Pattern: Transactional outbox (microservices.io) ↗ · Report a problem