Skip to content

Example project

Order flow: four Spring Boot services choreographed over Kafka

A complete, small order system in Java 21 and Spring Boot 3.5. An order is placed over REST, paid, reserved in stock and announced by email, and the four services involved talk only through Kafka topics. It shows a transactional outbox, three idempotent consumers, blocking retries with dead letter topics, non-blocking retry topics, manual acknowledgement, and tests against a real broker.

How this code was verified

This code was written against the referenced Spring and Kafka documentation and then built and run, with limits you should know. What was done: the project was compiled with JDK 21 (Temurin 21.0.13) and Maven 3.9.9. All 41 unit tests pass. NotificationRetryIT (EmbeddedKafka) passes. The four services were started together against a real Apache Kafka 3.9.2 broker (single node, KRaft, topic auto-creation off) and PostgreSQL 17, both run as local processes, and scripts/demo.sh produced the four expected order statuses. Republishing the outbox, a gateway outage ending in the dead letter topic, a non-JSON record, and the bounce-1 retry topic path were each run and behaved as chapter 10 describes. What was not done: Docker was not available. docker-compose.yml has never been started. The two Testcontainers tests, OrderOutboxIT and PaymentIdempotencyIT, compile but were not executed as written; their test bodies passed against the local broker and database with the container declarations removed, so the @Container and @ServiceConnection wiring itself is unverified. The other experiments in chapter 10 (stopping Kafka, kill -9, a second instance, the six mouse orders, removing the acknowledgement) describe expected behaviour and were not run. Nothing was load tested. What to do first: with Docker running, run mvn verify in the project root. If it fails, the Testcontainers setup or the Compose file is the most likely place. Treat this as teaching code: issues may exist, and the last chapter lists what a production system would still need.

Versions

ComponentVersionNote
Java21Built and run with Eclipse Temurin 21.0.13. The code uses records, sealed interfaces and pattern matching for switch.
Spring Boot3.5.16The newest 3.x release on Maven Central when this was written. It manages every library version below.
Spring for Apache Kafka3.3.16Managed by Spring Boot 3.5.16; no version in any pom.xml.
Apache Kafka clients (kafka-clients)3.9.2Managed by Spring Boot 3.5.16.
Apache Kafka broker3.9.2Docker image apache/kafka:3.9.2, one node in KRaft combined mode, no ZooKeeper. Kafka 4.x is KRaft-only and works with these clients too.
Testcontainers1.21.4Managed by Spring Boot. org.testcontainers.kafka.KafkaContainer with the apache/kafka image, and PostgreSQLContainer.
PostgreSQL17Docker image postgres:17-alpine; JDBC driver 42.7.11 and Flyway 11.7.2 managed by Spring Boot.
Test librariesJUnit Jupiter 5.12.2, Mockito 5.17.0, AssertJ 3.27.7, Awaitility 4.2.2All managed by Spring Boot.
Apache Maven3.9.9Spring Boot 3.5 needs Maven 3.6.3 or newer.

Architecture

Drawing diagram…

Flowchart. A client posts an order to the Order Service, which stores it in its own database and publishes to the order-placed topic through an outbox relay. The Payment Service reads order-placed and writes payment-completed or payment-failed. The Inventory Service reads payment-completed and writes stock-reserved or stock-rejected. The Payment Service also reads stock-rejected to refund. The Order Service reads all four result topics to update the order status. The Notification Service reads payment-failed, stock-reserved and stock-rejected. Each service has its own PostgreSQL database.

flowchart LR
  client["Client"] -->|"POST /api/orders"| order["Order Service"]

  order --- ordersdb[("orders DB: orders, outbox_events")]
  payment["Payment Service"] --- paymentsdb[("payments DB: payments, processed_events")]
  inventory["Inventory Service"] --- inventorydb[("inventory DB: stock_items, reservations")]
  notification["Notification Service"] --- notificationsdb[("notifications DB: sent_notifications")]

  subgraph kafka["Kafka: one topic per event type, record key is the order id"]
    tplaced["orders.order-placed.v1"]
    tcompleted["payments.payment-completed.v1"]
    tfailed["payments.payment-failed.v1"]
    treserved["inventory.stock-reserved.v1"]
    trejected["inventory.stock-rejected.v1"]
  end

  order -->|"outbox relay"| tplaced
  tplaced --> payment
  payment --> tcompleted
  payment --> tfailed
  tcompleted --> inventory
  inventory --> treserved
  inventory --> trejected
  trejected -->|"refund"| payment

  tcompleted --> order
  tfailed --> order
  treserved --> order
  trejected --> order

  tfailed --> notification
  treserved --> notification
  trejected --> notification
Each service owns one database and the topics it writes to. Services never call each other; they react to events.

The order flow

Drawing diagram…

Sequence diagram of a successful order. The client posts an order; the Order Service inserts the order and an outbox row in one transaction and answers 201 with status PLACED. The outbox relay publishes OrderPlaced to Kafka, waits for the acknowledgement and marks the row as published. The Payment Service consumes it, stores the payment, publishes PaymentCompleted and commits its offset. The Inventory Service consumes PaymentCompleted, reserves stock, publishes StockReserved and acknowledges. The Order Service consumes both results and ends at CONFIRMED. The Notification Service sends the confirmation email. The client reads the final status with a GET request.

sequenceDiagram
  autonumber
  participant C as Client
  participant O as Order Service
  participant ODB as orders DB
  participant K as Kafka
  participant P as Payment Service
  participant I as Inventory Service
  participant N as Notification Service

  C->>O: POST /api/orders
  O->>ODB: one transaction, insert order row and outbox row
  O-->>C: 201 Created with status PLACED

  Note over O,K: Outbox relay, runs every 500 ms
  O->>ODB: lock unpublished outbox rows
  O->>K: OrderPlaced to orders.order-placed.v1
  K-->>O: acknowledged by all in-sync replicas
  O->>ODB: set published_at and commit

  K->>P: OrderPlaced
  P->>P: one transaction, claim event id, charge, store payment
  P->>K: PaymentCompleted to payments.payment-completed.v1
  P->>K: commit offset of OrderPlaced

  K->>O: PaymentCompleted, order becomes PAID
  K->>I: PaymentCompleted
  I->>I: one transaction, conditional stock update, store reservation
  I->>K: StockReserved to inventory.stock-reserved.v1
  I->>K: acknowledge, offset of PaymentCompleted is committed

  K->>O: StockReserved, order becomes CONFIRMED
  K->>N: StockReserved
  N->>N: claim event id and send the confirmation email

  C->>O: GET /api/orders/id
  O-->>C: 200 OK with status CONFIRMED
The happy path. Every service commits its own database first, publishes second, and commits its Kafka offset last.

Chapter 1

The system and its event flow

A customer places an order. The order has to be paid, the stock has to be reserved, and the customer has to be told how it ended. Four services do this, and none of them calls another one. Each service reacts to an event on a Kafka topic, changes its own database, and publishes what it decided as a new event. This style is called choreography: there is no central coordinator, and the flow is the sum of the reactions.

ServiceReacts toDecidesPublishes
Order (port 8081)POST /api/orders, and all four result topicsThe order's statusorders.order-placed.v1
Payment (8082)orders.order-placed.v1, inventory.stock-rejected.v1Charge or decline; refundpayments.payment-completed.v1, payments.payment-failed.v1
Inventory (8083)payments.payment-completed.v1Reserve stock or rejectinventory.stock-reserved.v1, inventory.stock-rejected.v1
Notification (8084)payments.payment-failed.v1, inventory.stock-reserved.v1, inventory.stock-rejected.v1Which email to sendNothing (only its own retry topics)
Each topic has exactly one writer. A topic may have any number of readers, each with its own consumer group.

The three ways an order can end

  • Confirmed. Payment completed, stock reserved. The order goes PLACED, PAID, CONFIRMED.
  • Payment failed. The gateway declined. The order goes PLACED, PAYMENT_FAILED. Inventory never hears about it.
  • Rejected. Payment completed, then there was no stock. The order goes PLACED, PAID, REJECTED, and the payment service refunds because it also listens to inventory.stock-rejected.v1. That refund is a compensating action: there is no transaction across services to roll back, so a step that already succeeded is undone by another step.

One rule, applied four times

No service can write to its database and to Kafka atomically. They are two systems, and Kafka transactions cover Kafka writes and offset commits only, never a PostgreSQL commit. So every service in this project follows the same order of work: commit the database first, publish second, commit the consumer offset last. A crash at any point then leads to a repeat, never to a loss. Repeats are made harmless by making every consumer idempotent. The chapters that follow show four different ways to do that.

ServiceHow it publishesHow it stays idempotentHow it retries
OrderTransactional outbox and a relayState machine that only moves forwardIn place, short back-off, then dead letter topic
PaymentDirectly, after the database commitprocessed_events table and unique order_idIn place, exponential back-off, then dead letter topic
InventoryDirectly, then manual acknowledgementReservation row keyed by order idIn place, exponential back-off, then dead letter topic
NotificationDoes not publishsent_notifications claim per event idNon-blocking retry topics, then dead letter handler

Files in this chapter

Chapter 2

The event contract

The common-events module is the only code the four services share. It holds five Java records, the topic names, and the logical type ids. It has no dependencies: no Spring, no Kafka client, no Jackson.

What is on the wire

  • Key: the order id as a string.
  • Value: the event as JSON. Instants are ISO-8601 strings, amounts are JSON numbers read into BigDecimal.
  • Header __TypeId__: a logical name such as orderPlaced. The consumer maps it to a class with spring.json.type.mapping.
A PaymentCompleted record as kafka-console-consumer prints itText
__TypeId__:paymentCompletedkey:   3045dd5e-a323-4386-8c01-7ad80c6bf591value: {"eventId":"95e26fef-1e95-4fe4-87ab-8d246575d5ef","orderId":"3045dd5e-a323-4386-8c01-7ad80c6bf591",        "customerId":"customer-42","paymentId":"3f333e51-ae50-4283-af0d-321dd7971c1c","amount":179.80,        "currency":"EUR","sku":"SKU-KEYBOARD","quantity":2,"occurredAt":"2026-11-15T13:31:17.118229545Z"}

Why it is written this way

  • Captured from a run of this project (line breaks added). The type header holds the logical id, not com.example.events.PaymentCompleted.

Events carry the data consumers need

PaymentCompleted contains the SKU and the quantity, although the payment service has no use for them. It copies them from OrderPlaced so that the inventory service can do its work from the event alone. The alternative, an event with only ids and a REST call back to the order service, would make inventory fail whenever the order service is down. The price is a larger event and a field the payment service must pass along faithfully.

Shared library or schema registry

Shared contract library (this project)

  • One Maven module with Java records. Every service compiles against it.
  • Simple: no extra infrastructure, and the compiler checks producers and consumers.
  • Java only. A Python or Go consumer gets nothing from it.
  • Compatibility is a matter of discipline. Nothing stops someone from renaming a field and releasing.
  • Services are coupled at build time. If upgrading the library is mandatory for everyone at once, the services are no longer independently deployable.

Schemas in a registry

  • Avro, Protobuf or JSON Schema files, stored in a schema registry. Each service generates its own classes.
  • Works for any language.
  • The registry rejects an incompatible schema when the producer registers it, before a single bad record is written.
  • Records carry a schema id instead of field names, so they are smaller.
  • One more highly available system to run, and a build step for code generation.

A shared library is a reasonable start for a handful of Java services owned by one team. The rules that make it safe are the same rules a registry enforces: only add optional fields, never rename or remove a field, never change a field's meaning, and put a breaking change on a new topic (.v2). EventJsonContractTest pins the field names and checks that a consumer ignores a field it does not know.

Files in this chapter

Chapter 3

Order service and the transactional outbox

Placing an order has two effects: a row in the orders table and an OrderPlaced event on a topic. The whole system depends on these two never disagreeing. An order without an event is never paid. An event without an order charges a customer for nothing.

Why not save, then send

ApproachWhat goes wrong
Save the order, commit, then kafkaTemplate.sendThe process dies after the commit and before the send. The order exists, the event is never published, and nothing will ever retry it.
Send inside the transaction, then commitThe send succeeds and the commit fails (constraint, connection loss, deadlock). Payment charges for an order that does not exist.
Send first, save afterwardsThe same as the previous row, with a longer gap.
A Kafka transaction around bothA Kafka transaction makes several Kafka writes and offset commits atomic. PostgreSQL is not a participant. Spring can start both transactions and commit one after the other, which narrows the gap and does not remove it.
This is the dual-write problem: two systems, no common transaction.

The outbox

The fix is to stop writing to two systems. OrderService.placeOrder writes two rows in one PostgreSQL transaction: the order, and a row in outbox_events that holds the event as JSON. Either both rows are committed or neither is. Publishing becomes a separate, repeatable job.

  1. OutboxRelay.publishPending runs every 500 ms. It selects the oldest unpublished rows with FOR UPDATE SKIP LOCKED.
  2. It sends each row with KafkaTemplate and waits until the broker has acknowledged all of them.
  3. It sets published_at and commits.
  4. If anything fails between steps 2 and 3, the transaction rolls back, the rows stay unpublished, and the next run sends them again.

Decisions inside the relay

  • Wait for the acknowledgement. KafkaTemplate.send returns a CompletableFuture. Marking rows before the futures complete would mark records that are still in the producer's buffer.
  • SKIP LOCKED. Two instances of the service can both run the relay. Each locks a different batch, so neither waits and neither double-publishes.
  • Stored bytes. The payload is serialised when the order is placed. The relay sends a byte[] and adds the __TypeId__ header itself. KafkaProducerConfig makes the producer pass byte[] values through and serialise everything else as JSON.
  • Timeouts. The producer's delivery.timeout.ms (8 s) is shorter than the relay's wait (10 s), so the relay never gives up on a record that Kafka is still trying to deliver.

Consuming the results

OrderResultListener subscribes to the four result topics with one consumer group and maps each event to a target status. OrderStatus gives each status a rank, and Order.transitionTo only moves to a higher rank and never leaves a terminal state. Two consequences follow. A duplicate result changes nothing, so no deduplication table is needed. And a PaymentCompleted that is consumed after StockReserved is ignored instead of moving a confirmed order back to PAID.

Results for one order can be handled by two consumer threads at once, because they come from different topics. The @Version column on Order makes the second writer fail with an optimistic locking error. The error handler in KafkaErrorHandlingConfig redelivers the record after 0.5 seconds, and the second attempt reads the state the first writer committed.

Files in this chapter

Chapter 4

Payment service and the idempotent consumer

The previous chapter ended with a promise: OrderPlaced is published at least once. So the payment service will, sooner or later, receive the same event twice. It also receives events twice for its own reasons: the consumer crashes after processing and before committing the offset, or a rebalance moves the partition while a record is in progress. Charging a customer twice is not acceptable, so the consumer has to be idempotent: processing an event a second time must have no further effect.

The mechanism

  1. PaymentProcessor.process opens a database transaction.
  2. It inserts the event id into processed_events with ON CONFLICT DO NOTHING. One row inserted means this is the first delivery.
  3. It looks for a payment with this order id. If one exists, it returns it and does nothing else.
  4. Otherwise it calls the gateway, stores a COMPLETED or FAILED payment, and commits. The claim and the payment commit together.

The claim is only useful because it is in the same transaction as the payment. If the gateway call throws, the transaction rolls back and the claim disappears with it, so the redelivery is treated as a first delivery. If the claim were written to Redis, or in a separate transaction, a crash between the two writes would leave an event marked as processed with no payment, and the order would hang forever.

Publishing the result: the approach and its failure window

The payment service does not use an outbox. OrderPlacedListener does three things in a fixed order: commit the payment to PostgreSQL, publish the result and wait for the broker's acknowledgement, and return, after which the listener container commits the offset (ack-mode: record).

Crash or failureWhat happens nextOutcome
Before the database commitThe event is redelivered and processed as newCorrect
After the database commit, before the result is acknowledgedThe event is redelivered. The processor finds the stored payment and returns it. The listener publishes the result.Correct, but the result is late: it waits for the redelivery
After the result is acknowledged, before the offset commitThe same, and the result is published a second time with the same event idA duplicate result, which every consumer of the payment topics tolerates
The gateway charged, then the database transaction rolled backThe event is redelivered and the gateway is called again with the same idempotency keyCorrect only if the real gateway honours idempotency keys
The window is the second row. An outbox would publish the result without waiting for a redelivery. This design accepts the delay in exchange for less code.

For the redelivery to produce a useful duplicate, the result must be the same event both times. Payment stores resultEventId when the row is created, and toResultEvent uses the stored id and the stored timestamp. The notification service can then recognise the second copy by its event id.

Error handling

A listener that throws hands the record to the container's error handler. KafkaErrorHandlingConfig declares a DefaultErrorHandler bean, which Spring Boot applies to the auto-configured listener container factory.

  • Back-off. ExponentialBackOffWithMaxRetries(4) with a 1 second initial interval and multiplier 2: redeliveries after 1, 2, 4 and 8 seconds. The consumer thread waits, so the partition is blocked and order is kept. Fifteen seconds is far below max.poll.interval.ms.
  • Dead letter topic. After the last attempt, DeadLetterPublishingRecoverer writes the record to orders.order-placed.v1.payment-service-dlt, in the same partition number, with headers that hold the original topic, partition, offset and the exception. The offset is then committed and the partition continues.
  • Not retryable. InvalidEventException skips the back-off. Spring Kafka already treats deserialization and conversion exceptions this way.
  • Poison records. ErrorHandlingDeserializer catches a record that is not valid JSON and passes the failure to the error handler with the original bytes. Without it, the failure would happen inside poll() and the consumer would be stuck on that offset.

Files in this chapter

Chapter 5

Inventory service and manual acknowledgement

The inventory service consumes PaymentCompleted and decides whether the stock can be reserved. It has the same shape as the payment service, with two differences worth studying: how it prevents overselling, and who decides when an offset is committed.

Reserving stock without overselling

Three consumer threads may reserve the same SKU for three different orders at the same moment. StockRepository.tryReserve runs a single statement:

StockRepository.tryReserveSQL
UPDATE stock_items   SET available = available - :quantity,       reserved  = reserved + :quantity WHERE sku = :sku   AND available >= :quantity

Why it is written this way

  • One row updated means the stock was taken. Zero rows means there was not enough, and the order is rejected.
  • PostgreSQL locks the row for the update and re-checks the WHERE clause against the committed value, so two concurrent statements cannot both take the last unit.

Conditional update (used here)

  • The check and the change are one statement.
  • Never needs a retry: it either takes the stock or reports that there is none.
  • Only works when the rule can be written in SQL.

Optimistic locking with @Version

  • Read the row, decide in Java, write with WHERE version = ?.
  • A concurrent change makes the write fail, and the operation is retried from the read.
  • Right when the decision needs logic that does not fit in a WHERE clause. The order service uses it for status changes.

Idempotency comes from the reservations table, whose primary key is the order id. InventoryService.reserve first looks for a reservation and returns it if there is one, so a redelivered event does not take stock twice. The stock update and the reservation insert are one transaction.

Manual acknowledgement

With spring.kafka.listener.ack-mode: manual the listener receives an Acknowledgment and the container commits an offset only after acknowledge() was called for that record. PaymentCompletedListener calls it on its last line, after the database commit and after the broker confirmed the result event.

Ack modeThe offset is committedUse it when
BATCH (Spring's default)After all records returned by one poll() were handledRedelivering a whole batch after a crash is acceptable
RECORDAfter each record whose listener call returned normallyProcessing is synchronous and one redelivered record is the most you want. Order, payment and notification use it.
MANUALFor records the listener acknowledged, together, once the records of the current poll are processedThe code should state explicitly when a record is done
MANUAL_IMMEDIATESynchronously inside acknowledge()The commit must happen before the next line runs, at the cost of a broker round trip per record
All of these require enable-auto-commit: false. With auto commit on, the Kafka client commits on a timer regardless of the listener.

For a listener that does everything synchronously, RECORD and MANUAL commit the same offsets. The reason to prefer manual acknowledgement here is that the commit is tied to a visible line of code that comes after the last side effect. With RECORD, returning from the method is the acknowledgement. If someone later makes the publish asynchronous to gain throughput, the method returns early and offsets are committed for results that may never be sent. With MANUAL, that change does not compile into a silent data loss: the acknowledge() call has to be moved deliberately, for example into the send's completion callback.

Files in this chapter

Chapter 6

Notification service and retry topics

The notification service turns three kinds of result into an email. The email sender is a stub that writes to the log. The service is the last link of the chain and the one place where a different retry strategy is the better choice.

Blocking and non-blocking retries

Retry in place (DefaultErrorHandler)

  • The failed record is redelivered from the same partition after a pause.
  • Nothing else in that partition is processed meanwhile. Order is preserved.
  • Right when records depend on each other, as payments and stock changes of one order do.
  • A dependency that is down for minutes stops the partition for minutes.

Retry topics (@RetryableTopic)

  • The failed record is published to a retry topic and the original offset is committed.
  • The partition continues with the next record at once.
  • Right when records are independent, as emails to different customers are.
  • A retried record is overtaken by later ones. Order within the key is lost.

A mail provider that rejects one address should not delay everybody else's confirmation. And if a customer's "order confirmed" email arrives a few seconds after it would have, nobody is harmed. So NotificationListener uses @RetryableTopic.

What the annotation sets up

Topic (for inventory.stock-reserved.v1)Read afterAttempt
inventory.stock-reserved.v1immediately1
inventory.stock-reserved.v1.notification-service-retry-02 s2
inventory.stock-reserved.v1.notification-service-retry-14 s more3
inventory.stock-reserved.v1.notification-service-retry-28 s more4
inventory.stock-reserved.v1.notification-service-dltafter attempt 4 failed@DltHandler
attempts = "4", back-off 2 s with multiplier 2. The same five topics exist for each of the three source topics.
  • Each retry topic gets its own listener container. It does not sleep: it pauses the partition until the record's due time and keeps polling, so the consumer stays in the group.
  • The record is forwarded with the application's KafkaTemplate. That is why a service that publishes no events still has producer settings and a KafkaProducerConfig.
  • exclude = IllegalArgumentException.class sends an event with no customer message straight to the dead letter topic.
  • dltStrategy = FAIL_ON_ERROR stops the dead letter handler from being retried if it throws.
  • The suffixes are set explicitly and contain the service name. With the default suffixes (-retry, -dlt), two services that use retry topics on the same source topic would share them.
  • Spring creates the retry and dead letter topics through KafkaAdmin, with the partition count and replication factor given in the annotation.

Duplicates

Upstream services publish at least once, and the retry machinery forwards at least once. Without protection, a customer could get the same email several times. NotificationService.notify claims the event id in sent_notifications with ON CONFLICT DO NOTHING and sends only if the claim succeeded. The claim and the send are in one database transaction, with the send last: a failed send rolls the claim back so that the retry can send.

Files in this chapter

Chapter 7

Configuration, topics and Docker Compose

Almost everything Kafka-related is configured in application.yml under spring.kafka. Spring Boot builds the producer factory, the KafkaTemplate, the consumer factory, the listener container factory and a KafkaAdmin from these properties. The Java configuration in each service adds only what properties cannot express: the value serializer, the error handler and the topics.

Producer settings

SettingValueWhy
acksallThe leader answers only when all in-sync replicas have the record. It is meaningful together with the topic's min.insync.replicas: with replication factor 3 and min.insync.replicas=2, an acknowledged record survives the loss of one broker.
enable.idempotencetrueThe broker discards a batch it already has when the producer retries. No duplicates and no reordering from producer retries. It is the default in current clients; it is written down because the system relies on it.
delivery.timeout.ms8000The total time a send may take, retries included. Kept below the 10 second wait in the relay and the publishers.
request.timeout.ms5000The producer requires delivery.timeout.ms >= linger.ms + request.timeout.ms and refuses to start otherwise.
linger.ms5Wait up to 5 ms for more records to fill a batch. The outbox relay sends up to 50 records at once and benefits from it.
key-serializerStringSerializerKeys are order ids as strings.

Consumer and listener settings

SettingValueWhy
group-idthe service nameEvery service has its own group, so each one gets every record of the topics it reads. Instances of the same service share the group and split the partitions.
enable-auto-commitfalseThe Spring listener container commits offsets according to the ack mode. With auto commit the client would commit on a timer, possibly before the record was processed.
auto-offset-resetearliestUsed only when the group has no committed offset. A new service then starts from the oldest retained record instead of skipping everything published before it started.
value-deserializerErrorHandlingDeserializerDelegates to JsonDeserializer and turns a failure into something the error handler can deal with.
spring.json.trusted.packagescom.example.eventsThe deserializer creates classes named by a header. It must not be possible to name an arbitrary class.
spring.json.type.mappingorderPlaced:com.example.events.OrderPlaced,...Maps the logical ids in __TypeId__ to classes.
listener.ack-moderecord (inventory: manual)Commit after each handled record, so a crash repeats at most one.
listener.concurrency3Three consumer threads, one per partition. More threads than partitions would be idle: a partition is read by one consumer of a group.
isolation.levelnot set (read_uncommitted)No producer here uses Kafka transactions. A consumer of transactional producers sets read_committed to skip aborted records; consumers never read past the high watermark either way.

Topics

Each service declares the topics it owns as NewTopic beans (KafkaAdmin.NewTopics): the ones it writes to, and its own dead letter topics. KafkaAdmin creates missing topics at startup. The broker in docker-compose.yml has topic auto-creation switched off, so a topic exists only because some service declared it.

  • Partitions: 3. Enough to show parallel consumers. In production the number follows from the throughput and the consumer parallelism you need. It can be increased later, but then the same key maps to a different partition and per-key ordering breaks across the change.
  • Replication factor: 1 in application.yml, because there is one local broker. Production: 3, with min.insync.replicas=2 set on the topic (TopicBuilder.config(...)) or as a broker default.
  • Dead letter topics have the same partition count as their source topic, because the recoverer writes a dead letter to the same partition number.
  • Who creates topics is a policy question. Many teams provision topics with infrastructure code and give applications no permission to create them. The beans are then still useful as a statement of what the service expects.

Docker Compose

  • Kafka is the official apache/kafka image, version 3.9.2, as a single node in KRaft combined mode: the same process is broker and controller, and there is no ZooKeeper.
  • Two listeners. PLAINTEXT_HOST://localhost:9092 for the services, which run on the host, and PLAINTEXT://kafka:19092 for other containers such as Kafka UI. A broker tells clients which address to use after the first connection, and that address has to be reachable from where the client runs.
  • Internal topics (__consumer_offsets, the transaction log) are set to replication factor 1. Their default is 3, which one broker cannot provide.
  • PostgreSQL is one container with four databases and four logins, created by init-databases.sql on the first start.
  • Kafka UI is optional (--profile ui) and shows topics, records and consumer group lag.

Files in this chapter

Chapter 8

Observability and consumer lag

In an event-driven system a failure is rarely an error page. It is an order that stays PLACED. To see why, each service exposes metrics through Spring Boot Actuator and Micrometer, with a Prometheus endpoint at /actuator/prometheus.

MetricWhere it comes fromWhat it tells you
spring.kafka.listenerspring.kafka.listener.observation-enabled: trueA timer per listener: how many records were handled, how long each took, and how many ended in an error.
spring.kafka.templatespring.kafka.template.observation-enabled: trueA timer per send, tagged with the destination topic. Sends to a -dlt topic show up here, which makes dead-lettering visible.
kafka.consumer.fetch.manager.records.lag.maxThe Kafka client's own metrics, bound by Spring BootThe largest lag this consumer instance sees on its assigned partitions.
kafka.producer.*The Kafka client's own metricsRecord send rate, error rate, retry rate and request latency of the producer.
notifications.dead.letteredA counter in NotificationListenerEmails that were given up. Should stay at zero; alert when it does not.
Metric names use dots in Micrometer (/actuator/metrics) and underscores in the Prometheus output (/actuator/prometheus).

The two observation-enabled settings do more than timers. They create Micrometer observations, so when a tracing bridge is on the classpath (for example micrometer-tracing-bridge-otel), the template writes the trace context into record headers and the listener continues the trace. One trace then covers the HTTP request, the relay, and all four services.

Consumer lag

Lag is the distance between the newest offset in a partition (the log end offset) and the offset the consumer group has committed. It is the number of records waiting. A lag that grows means the consumers are slower than the producers, or stopped.

Lag of the payment service's groupShell
docker compose exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \  --bootstrap-server localhost:9092 \  --describe --group payment-service

Why it is written this way

  • The output has one line per partition with CURRENT-OFFSET (committed), LOG-END-OFFSET and LAG.
  • The committed offset is the next offset the group will read, so a fully caught-up group shows LAG 0.
  • Measure lag from outside the consumer. records.lag.max is reported by the consumer itself. A consumer that is dead reports nothing, and the metric disappears exactly when it matters. Kafka UI, kafka-consumer-groups.sh or an exporter that reads committed offsets from the broker keep working when the consumer does not.
  • Blocking retries look like lag. While the payment service backs off on one record, the partition does not move and lag grows behind it. That is the cost of keeping order.
  • Watch the outbox as well. Rows in outbox_events with published_at IS NULL that are older than a few seconds mean the relay cannot reach Kafka. That backlog is invisible to any Kafka metric, because the records are not in Kafka yet.
  • Watch the dead letter topics. A record there is an order that will not finish by itself.

Files in this chapter

Chapter 9

Testing

The code is arranged so that the interesting decisions are in plain classes: Order, PaymentProcessor, InventoryService, NotificationService. Listeners and configuration stay thin. That decides what kind of test covers what.

LevelToolsCoversRuns in
Unit (*Test)JUnit 5, Mockito, AssertJState machine, idempotency decisions, order of calls in a listener, contract JSONmvn test, milliseconds each, no Docker
Wiring (NotificationRetryIT)@SpringBootTest, @EmbeddedKafka, @MockitoBean, AwaitilityThat @RetryableTopic really redelivers and the dead letter handler really runsmvn verify, no Docker
Integration (OrderOutboxIT, PaymentIdempotencyIT)@SpringBootTest, Testcontainers (Kafka and PostgreSQL), @ServiceConnection, AwaitilityThe real configuration against a real broker and database: Flyway, JPA mappings, serializers, type headers, the relay, deduplicationmvn verify, needs Docker

Unit tests

  • OrderTest checks the state machine, including a result that arrives out of order and every terminal state.
  • PaymentProcessorTest checks that a redelivery does not call the gateway, that a second event for a paid order does not charge, and that a gateway outage leaves the method as an exception.
  • OrderPlacedListenerTest and PaymentCompletedListenerTest use Mockito's InOrder to pin the sequence: database, publish, acknowledge. They also assert that nothing is acknowledged when a step fails.
  • EventJsonContractTest fails when a field of the contract is renamed.

Integration tests with Testcontainers

Both tests declare a PostgreSQLContainer and an org.testcontainers.kafka.KafkaContainer (the apache/kafka image) as static fields with @Container and @ServiceConnection. Spring Boot reads the containers' addresses and uses them instead of spring.datasource.* and spring.kafka.bootstrap-servers. Everything else comes from the service's real application.yml.

  • OrderOutboxIT posts an order over HTTP and then reads orders.order-placed.v1 with a plain KafkaConsumer. It asserts the key, the __TypeId__ header and the JSON, and then that no unpublished outbox row is left.
  • PaymentIdempotencyIT sends the same OrderPlaced twice. It waits until two PaymentCompleted records for that order are on the result topic, which proves both deliveries were handled, and then asserts that there is one payment row and that both results carry the same event id.

What these tests do not cover: killing a service between the database commit and the publish. That failure window is covered by reasoning in chapter 4 and by the unit tests of the redelivery path. The last chapter shows how to provoke it by hand.

Files in this chapter

Chapter 10

Running it, and what to try to break

You need JDK 21, Maven 3.6.3 or newer, and Docker. Copy the files into a directory using the paths shown, then:

Build, start, tryShell
# 1. Build everything and run all tests (the *IT tests start containers, so Docker must be running)mvn verify# 2. Infrastructure: Kafka and PostgreSQL (add "--profile ui" for Kafka UI on http://localhost:8080)docker compose up -d# 3. The four services, each in its own terminaljava -jar order-service/target/order-service-1.0.0-SNAPSHOT.jarjava -jar payment-service/target/payment-service-1.0.0-SNAPSHOT.jarjava -jar inventory-service/target/inventory-service-1.0.0-SNAPSHOT.jarjava -jar notification-service/target/notification-service-1.0.0-SNAPSHOT.jar# 4. One order for each path through the system./scripts/demo.sh# 5. Read a topic, with keys and headersdocker compose exec kafka /opt/kafka/bin/kafka-console-consumer.sh \  --bootstrap-server localhost:9092 --topic payments.payment-completed.v1 \  --from-beginning --property print.key=true --property print.headers=true

Why it is written this way

  • mvn verify -DskipITs builds and runs the unit tests without Docker.
  • The demo script prints the expected and the actual final status of four orders: CONFIRMED, PAYMENT_FAILED, REJECTED, CONFIRMED.
  • docker compose down -v removes the containers and the data, including the databases.

Experiments

Each experiment names what to do, what you should see, and which mechanism is responsible. They follow the academy's sequence: build it, break it, watch it, understand the fix.

Break itWhat you should seeWhy
Stop Kafka (docker compose stop kafka), place an order, start Kafka againThe POST still answers 201. The order stays PLACED, and the order service logs that an outbox batch was not acknowledged. After Kafka is back, the order completes.The outbox. Accepting an order does not depend on the broker.
Republish the outbox: UPDATE outbox_events SET published_at = NULL; in the orders databaseEvery OrderPlaced is sent again. The logs say "already has payment", "was already decided", "was already notified". No second payment, no change in stock, no second email.Idempotent consumers, three different implementations.
Stop the payment service, place three orders, start it againThe orders wait as PLACED. kafka-consumer-groups.sh --describe --group payment-service shows lag 3. After the restart the lag returns to 0 and the orders complete.Offsets are stored per group in Kafka. A stopped consumer resumes where it committed.
Kill the payment service with kill -9 while orders are flowingAfter the restart, some results may be published a second time. Count the rows in payments: one per order.At-least-once delivery plus the processed_events and order_id checks.
Place an order with "currency":"XXX"The payment service logs five failed delivery attempts over about 15 seconds. Then the record appears in orders.order-placed.v1.payment-service-dlt with kafka_dlt-exception-* headers. Orders with keys in the same partition wait during those 15 seconds.Blocking retries with exponential back-off, then the dead letter topic.
Produce a line of text that is not JSON to orders.order-placed.v1 with kafka-console-producer.shThe record goes to the dead letter topic at once, without retries, with its original bytes. The consumer continues with the next record.ErrorHandlingDeserializer, and deserialization failures being not retryable.
Place an order for customer bounce-1The order is CONFIRMED. The email fails four times: at once, then after 2, 4 and 8 seconds. Then the notification service logs "Giving up on the notification". Emails for other customers are not delayed.Non-blocking retry topics and the @DltHandler.
Order SKU-MOUSE (stock 5) six times with quantity 1Five orders are CONFIRMED and one is REJECTED, with its payment REFUNDED. stock_items.available is 0 and never negative.The conditional update, and the compensating refund.
Start a second instance of the payment service: java -jar ... --server.port=8092Both instances log a rebalance, and the three partitions are split between them (six consumer threads, three of them idle).A group's parallelism is capped by the partition count.
In PaymentCompletedListener, comment out acknowledgment.acknowledge(), rebuild, place orders, restart the inventory serviceEverything seems to work until the restart. Then every PaymentCompleted since the last commit is delivered again, and the log fills with "was already decided".With manual acknowledgement nothing is committed without the call. The idempotent consumer hides the mistake from customers, not from the logs.

Files in this chapter

All files

root · pom.xml

pom.xmlXML
<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0"         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">    <modelVersion>4.0.0</modelVersion>    <!-- The Spring Boot parent manages every version below it: Spring Kafka 3.3.16,         kafka-clients 3.9.2, Testcontainers 1.21.4, Flyway, the PostgreSQL driver, JUnit, Awaitility. -->    <parent>        <groupId>org.springframework.boot</groupId>        <artifactId>spring-boot-starter-parent</artifactId>        <version>3.5.16</version>        <relativePath/>    </parent>    <groupId>com.example.shop</groupId>    <artifactId>order-flow</artifactId>    <version>1.0.0-SNAPSHOT</version>    <packaging>pom</packaging>    <name>order-flow</name>    <description>Order, Payment, Inventory and Notification services choreographed over Apache Kafka</description>    <modules>        <module>common-events</module>        <module>order-service</module>        <module>payment-service</module>        <module>inventory-service</module>        <module>notification-service</module>    </modules>    <properties>        <java.version>21</java.version>        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>    </properties>    <dependencyManagement>        <dependencies>            <dependency>                <groupId>com.example.shop</groupId>                <artifactId>common-events</artifactId>                <version>${project.version}</version>            </dependency>        </dependencies>    </dependencyManagement>    <build>        <pluginManagement>            <plugins>                <!-- Unit tests (*Test) run in "mvn test" through Surefire. Integration tests (*IT) need Docker                     and run in "mvn verify" through Failsafe. The Boot parent already binds the                     integration-test and verify goals; a module only has to declare the plugin. -->                <plugin>                    <groupId>org.apache.maven.plugins</groupId>                    <artifactId>maven-failsafe-plugin</artifactId>                </plugin>            </plugins>        </pluginManagement>    </build></project>

Why it is written this way

  • spring-boot-starter-parent 3.5.16 is the only version number in the build that matters. It manages Spring for Apache Kafka 3.3.16, kafka-clients 3.9.2, Testcontainers 1.21.4, Flyway, Hibernate and JUnit, so no module declares a version for them and they cannot drift apart.
  • java.version 21 is what makes the code's pattern matching for switch over sealed types compile. Spring Boot 3.5 needs Java 17 or newer.
  • common-events is listed in dependencyManagement so the four services depend on it without repeating the version.
  • Failsafe is declared once under pluginManagement. Classes named *IT run in mvn verify; classes named *Test run in mvn test. A developer without Docker can still run the unit tests.

References: Spring for Apache Kafka reference (3.3): sending messages with KafkaTemplate ↗ · Spring for Apache Kafka reference (3.3): @KafkaListener annotation ↗ · Spring for Apache Kafka reference (3.3): message listener containers, committing offsets and ack modes ↗ · Spring for Apache Kafka reference (3.3): serialization, JsonSerializer, type mapping, ErrorHandlingDeserializer, DelegatingByTypeSerializer ↗ · Spring for Apache Kafka reference (3.3): handling exceptions, DefaultErrorHandler, DeadLetterPublishingRecoverer ↗ · Spring for Apache Kafka reference (3.3): non-blocking retries, @RetryableTopic configuration ↗ · Spring for Apache Kafka reference (3.3): transactions ↗ · Spring for Apache Kafka reference (3.3): testing applications, @EmbeddedKafka ↗ · Spring for Apache Kafka reference (3.3): monitoring, Micrometer observation ↗ · Spring Boot reference (3.5): Apache Kafka support and spring.kafka.* properties ↗ · Spring Boot reference (3.5): Testcontainers and @ServiceConnection ↗ · Spring Boot 3.5.16 managed dependency versions (spring-boot-dependencies POM) ↗ · Apache Kafka official Docker image (apache/kafka): tags and KRaft environment variables ↗ · Apache Kafka documentation: producer, consumer and topic configuration, delivery semantics ↗ · Testcontainers: Kafka module (org.testcontainers.kafka.KafkaContainer) ↗ · PostgreSQL documentation: SELECT ... FOR UPDATE SKIP LOCKED ↗ · Pattern: Transactional outbox (microservices.io) ↗ · Report a problem