Path 7 · lesson 3 of 8
The saga pattern
A business transaction across services as a chain of local transactions, each with a compensating action. Choreography vs orchestration, idempotency, timeouts and living without isolation.
What you will learn
- Break a cross-service business transaction into local transactions and define the compensating action for each
- Compare choreography and orchestration and choose between them for a given workflow
- Implement a choreographed order saga in Spring Boot in which a failed payment cancels the order and rejected stock refunds the payment
- Make every saga step idempotent and state-guarded so duplicates, late events and repeated compensations are harmless
- Add a timeout so a saga cannot stay unfinished, and handle events that arrive after it fired
- Explain what the missing isolation means for readers of intermediate state, and apply countermeasures
Before you start
The concept
Placing an order changes three databases: the Order Service records the order, the Payment Service takes the money, the Inventory Service reserves the stock. Each service owns its database, so no single transaction can cover all three. A saga achieves the business outcome without one: it is a sequence of local transactions, one per service, linked by messages. Each step commits on its own and then triggers the next.
Because earlier steps are already committed when a later one fails, there is nothing to roll back. Instead, every step that can need undoing has a compensating action: another local transaction that semantically reverses it. If stock cannot be reserved after the payment was authorised, the saga runs the payment's compensation, a refund, and then cancels the order. The saga ends in one of two consistent outcomes: every step done, or every completed step compensated.
| Step | Service | Local transaction | Compensating action |
|---|---|---|---|
| 1 | Order | Create order as PENDING | Mark order CANCELLED |
| 2 | Payment | Authorise payment | Refund (or void) the payment |
| 3 | Inventory | Reserve stock | Release the reservation |
| 4 | Order | Mark order CONFIRMED | None: nothing follows that can fail |
| 5 | Notification | Send confirmation or cancellation | None: an email cannot be unsent, so it goes last |
Two ways to coordinate
Choreography: services react to events
- Each service subscribes to the others' events and publishes its own. There is no central component.
- Adding a reader such as Notification needs no change to existing services
- No single point of failure and no extra service to run
- The workflow is not written down anywhere: it emerges from who listens to what
- Hard to see where one order is stuck without tracing events across services
- Cyclic dependencies appear as the flow grows: services must know each other's events
Orchestration: a coordinator sends commands
- An orchestrator holds the saga's state and tells each participant what to do, then waits for the reply
- The whole flow, including compensations and timeouts, is in one place and one state table
- Participants stay simple: they execute commands and report results
- Easy to answer 'what state is order 1001 in?' and to add steps in the middle
- The orchestrator is one more service to build, run and keep available
- Risk of pulling business logic out of the services into the coordinator
A rule of thumb that holds up: choreography suits a short, stable flow of three or four steps with few branches. Once a flow has many steps, several failure branches, or timeouts between steps, the implicit version becomes difficult to reason about and an orchestrator pays for itself. Both use the same building blocks from this path: an outbox to publish, idempotent consumers to receive.
Advantages of a saga
- Keeps data consistent across services without a distributed transaction
- Each service stays available independently: a slow participant delays the saga without blocking the others' databases
- No locks held across services or across network calls
- Every step is an ordinary local transaction that the owning team can test alone
Disadvantages of a saga
- No isolation: intermediate states are visible to other requests and other sagas
- No automatic rollback: every compensation is code you design, write and test
- Eventually consistent: the caller gets 'pending' and learns the outcome later
- Needs idempotency, timeouts and monitoring for stuck sagas to be correct
- Debugging means following one business transaction through several logs
Why it exists
In a single database the order flow is one transaction: insert the order, debit the account, decrement the stock, commit. If the stock check fails, everything rolls back and no other session ever saw the half-finished state. Splitting the system into services with their own databases removes that option, and the obvious replacements both fail.
- Two-phase commit across the services' databases. It needs every participant to support it and to be available at the same moment. While a participant has voted yes and is waiting for the decision, it holds locks. If the coordinator disappears at that point, the locks stay. Kafka cannot take part in such a transaction, and neither can an external payment provider.
- A chain of synchronous calls with try/catch. The Order Service calls Payment, then Inventory, and on an exception calls Payment again to refund. If the Order Service crashes after the payment call returned, nothing remembers that a refund is owed. If the refund call times out, was it applied? The undo logic lives in the memory of a process that can die.
The saga idea comes from a 1987 paper by Hector Garcia-Molina and Kenneth Salem on long-lived database transactions: split the long transaction into a sequence of shorter ones that each commit, and pair each with a compensating transaction, so the system either completes the whole sequence or runs the compensations of the completed part. Applied to services, the sequence is driven by durable messages instead of one process. That is the difference from the try/catch chain: the fact that a step happened is stored in a database and announced on a durable log, so the next step, or the compensation, happens even if every service restarts in between.
Kafka fits this role well. Events are retained, so a consumer that was down picks up where it stopped. Keying every saga event by order id keeps the events of one saga on one topic in order. Consumer groups let each service process its share independently. What Kafka does not provide is atomicity between a service's database and its events. Each saga step must update local state and publish its result atomically, which is exactly what the transactional outbox is for.
How it works inside
The shape of one step
Every participant does the same four things in one local database transaction: record the incoming event id (skip if already seen), check that its own state allows the transition, change its state, and write the outgoing event to its outbox. The offset is committed after the transaction. A crash anywhere leaves either nothing done (the event is redelivered and processed) or everything done (the redelivery is skipped).
State guards make steps and compensations idempotent
Deduplication by event id handles the same event delivered twice. It does not handle two different events that ask for the same thing: a refund can be triggered by StockRejected and also by OrderCancelled from the timeout. The second defence is a guarded transition: UPDATE payment SET status = 'REFUNDED' WHERE order_id = ? AND status = 'AUTHORIZED'. Only one statement can ever match. Whoever gets zero rows does nothing and publishes nothing.
Drawing diagram…
State diagram of an order: it starts PENDING, becomes CONFIRMED on StockReserved, or CANCELLED on PaymentFailed, PaymentRefunded or a timeout. CONFIRMED and CANCELLED are final.
stateDiagram-v2 [*] --> PENDING: order placed PENDING --> CONFIRMED: StockReserved PENDING --> CANCELLED: PaymentFailed PENDING --> CANCELLED: PaymentRefunded PENDING --> CANCELLED: timeout CONFIRMED --> [*] CANCELLED --> [*]
No isolation, and how to live with it
Between step 1 and step 4 the order exists, the money may be held, and the stock may or may not be reserved. Anyone can observe that. A second saga can read stock that a first saga is about to give back. An ACID transaction would hide all of this, and a saga cannot. The countermeasures are design choices:
- Semantic lock. The
PENDINGstatus is a flag that says 'a saga is working on this'. The UI shows 'processing', other operations (edit, cancel by the customer, ship) check the status and refuse or wait. - Reserve, do not consume. Authorise the payment and capture later. Reserve stock and deduct on shipment. Reservations are cheap to compensate and honest about being provisional.
- Guard on current state, not on what you read earlier.
available = available - ? WHERE available >= ?decides at write time, so two sagas competing for the last item cannot both win. - Order the steps by how hard they are to undo. Put steps that can fail for business reasons early and steps that cannot be compensated (sending an email, shipping a parcel) last, after the point where the saga can no longer fail.
- Make compensations commutative with late events. A compensation may run before the step it compensates has even been processed by a slow service. Record the cancellation so the late step is ignored.
Timeouts
A choreographed saga has no component waiting for the next event. If the Payment Service is down for a day, the order simply stays PENDING. The Order Service therefore runs a sweeper: orders still PENDING after a deadline are cancelled and OrderCancelled is published. Every participant treats that event as a request to compensate whatever it has done. The timeout does not stop the other services, so an event can arrive after it: the payment is authorised a minute after the order was cancelled. Because OrderCreated and OrderCancelled are on the same topic with the same key, the Payment Service sees them in order and its guarded refund handles it.
Failure scenarios
| Failure point | Resulting state | Recovery |
|---|---|---|
| Payment Service crashes after authorising at the provider, before its commit | Provider holds the money. No payment row update, no event, offset not committed | OrderCreated is redelivered. The authorise call is repeated with the same idempotency key (the order id), the provider returns the original result, the step commits. |
| Payment Service crashes after its commit, before the offset commit | Payment AUTHORIZED, PaymentAuthorized in its outbox | The redelivered OrderCreated is skipped by event id. The relay publishes the outbox row. |
| Payment is declined | Payment FAILED, PaymentFailed published | Order Service cancels the order. Nothing else was done, so nothing else is compensated. |
| Stock is not available | Payment AUTHORIZED, stock request REJECTED, order still PENDING | Payment Service refunds on StockRejected and publishes PaymentRefunded. Order Service then cancels. |
| The refund call to the provider fails | Payment still AUTHORIZED: the transaction rolled back | The error handler redelivers StockRejected with backoff. If retries are exhausted the record goes to a dead letter topic and a person is alerted: money is owed. |
Inventory receives PaymentAuthorized before it has processed OrderCreated | No stock request row to act on | The listener throws, the record is retried after a backoff, by which time the other partition has usually caught up. |
| A participant is down for longer than the saga deadline | Order PENDING past the deadline | The sweeper cancels and publishes OrderCancelled. When the participant returns it processes its backlog in order and compensates. |
| Order Service crashes between any two events | Order PENDING, events waiting on the topics | On restart it resumes from its committed offsets. Saga state is in the database and the log, never in memory. |
Architecture
Drawing diagram…
Flowchart: Order, Payment and Inventory services each publish to their own topic, orders.events, payments.events and inventory.events. Payment and Inventory and Notification consume orders.events. Inventory and Order consume payments.events. Payment and Order consume inventory.events. The Payment Service also calls an external payment provider with an idempotency key.
flowchart LR OS["Order Service"] PS["Payment Service"] IS["Inventory Service"] NS["Notification Service"] OE["Topic orders.events"] PE["Topic payments.events"] IE["Topic inventory.events"] OS -->|"OrderCreated, OrderConfirmed, OrderCancelled"| OE PS -->|"PaymentAuthorized, PaymentFailed, PaymentRefunded"| PE IS -->|"StockReserved, StockRejected"| IE OE --> PS OE --> IS OE --> NS PE --> IS PE --> OS IE --> PS IE --> OS PSP["Payment provider"] PS -.->|"authorise, refund with idempotency key"| PSP
- Each service is the only writer of its topic, and writes through its outbox, so a state change and its event cannot diverge.
- The arrows into each service are its subscriptions. The workflow is the sum of those subscriptions, which is why choreography is hard to read from any single codebase.
- The order id is the key on all three topics. Within one topic the events of one order are ordered. Across topics they are not: Inventory can see
PaymentAuthorizedbeforeOrderCreated. - The payment provider is outside every transaction. The only protection for that call is an idempotency key that makes repeating it safe.
Message flow
Drawing diagram…
Sequence diagram: the Order Service publishes OrderCreated. Inventory stores the request and Payment authorises and publishes PaymentAuthorized. Inventory finds no stock and publishes StockRejected. Payment refunds and publishes PaymentRefunded. The Order Service cancels the order and publishes OrderCancelled. Notification emails the customer, and Payment receives OrderCancelled and does nothing because the payment is already refunded.
sequenceDiagram participant O as Order Service participant K as Kafka participant P as Payment Service participant I as Inventory Service participant N as Notification Service O->>K: OrderCreated o-1001 K->>P: OrderCreated K->>I: OrderCreated I->>I: store stock request RECEIVED P->>P: authorise, payment AUTHORIZED P->>K: PaymentAuthorized K->>I: PaymentAuthorized I->>I: no stock left, request REJECTED I->>K: StockRejected K->>P: StockRejected P->>P: refund, payment REFUNDED P->>K: PaymentRefunded K->>O: PaymentRefunded O->>O: order PENDING to CANCELLED O->>K: OrderCancelled K->>N: OrderCancelled N->>N: email order cancelled, money returned K->>P: OrderCancelled P->>P: already REFUNDED, nothing to do
- Compensations run in reverse order of the steps: stock was never reserved, so only the payment and the order are undone.
- The order is cancelled on
PaymentRefunded, not onStockRejected. When the customer is told 'cancelled', the refund has already been issued. - The last message shows why guards matter:
OrderCancelledalso asks Payment to compensate, and the guarded update finds the payment already refunded. - In the declined-payment path the sequence is shorter:
PaymentFailedgoes straight to the Order Service, which cancels. Inventory never acts.
In a real system
A typical shop runs this saga on three topics with 12 partitions each, replication factor 3, all keyed by order id: orders.events, payments.events, inventory.events. Consumer groups are named after the services: order-service reads payments and inventory events, payment-service reads orders and inventory events, inventory-service reads orders and payments events, notification-service reads orders events. The checkout API returns 202 Accepted with the order id while the order is PENDING. The client polls GET /orders/{id} or receives a push when the status changes.
| Question a user or operator asks | Where the answer lives |
|---|---|
| Did my order go through? | The order's status in the Order Service: PENDING, CONFIRMED or CANCELLED with a reason |
| Why is order 1001 still pending? | The last event for key o-1001 on each topic, and the lag of the group that should have reacted |
| Was the customer refunded? | The payment row's status and the PaymentRefunded event |
| How many sagas are stuck? | Count of orders PENDING for longer than the deadline: this should be zero and is the saga's main alert |
When to use it
- A business transaction changes data owned by several services, and the outcome must be all-or-compensated.
- Every step that can fail has a meaningful compensation the business accepts (refund, release, cancel).
- The caller can accept an asynchronous result: 'pending' now, the outcome shortly after.
- Choreography when the flow is short and stable. Orchestration when it has many steps, branches or waiting periods.
When not to use it
- All the data sits in one database. Use a local transaction: it is simpler and gives real isolation.
- A step cannot be compensated and cannot be moved to the end, and the business cannot tolerate it being wrong.
- The caller needs a consistent answer before the request returns and cannot work with a pending state.
- The services are split so that almost every request needs a saga. That usually means the service boundaries are in the wrong place.
Java and Spring Boot
A choreographed order saga on Spring Boot 3 with one PostgreSQL database per service. Every listener runs in a local transaction that deduplicates the incoming event, applies a guarded state change and writes the outgoing event to the service's outbox (published by the relay or CDC connector from the previous two lessons). All records carry the order id as key and the headers id and eventType.
-- Every service database has these two tables (see the transactional outbox lesson).CREATE TABLE outbox_event ( seq bigserial PRIMARY KEY, id uuid NOT NULL UNIQUE, aggregatetype varchar(255) NOT NULL, -- 'orders' | 'payments' | 'inventory' -> <type>.events aggregateid varchar(255) NOT NULL, -- always the order id: the saga's correlation key type varchar(255) NOT NULL, payload jsonb NOT NULL, published_at timestamptz);CREATE TABLE processed_event ( consumer varchar(64) NOT NULL, event_id uuid NOT NULL, PRIMARY KEY (consumer, event_id));-- Order ServiceCREATE TABLE orders ( id varchar(36) PRIMARY KEY, customer_id varchar(36) NOT NULL, sku varchar(64) NOT NULL, quantity integer NOT NULL, total_cents bigint NOT NULL, status varchar(16) NOT NULL, -- PENDING | CONFIRMED | CANCELLED cancel_reason varchar(64), created_at timestamptz NOT NULL DEFAULT now());-- Guarded transition, used for every status change (1 row = this caller won):-- UPDATE orders SET status = 'CANCELLED', cancel_reason = ? WHERE id = ? AND status = 'PENDING';-- Timeout sweep:-- SELECT id FROM orders WHERE status = 'PENDING' AND created_at < now() - interval '15 minutes';-- Payment ServiceCREATE TABLE payment ( order_id varchar(36) PRIMARY KEY, -- one payment per order amount_cents bigint NOT NULL, status varchar(16) NOT NULL -- PENDING | AUTHORIZED | FAILED | REFUNDED);-- Compensation, safe to request any number of times:-- UPDATE payment SET status = 'REFUNDED' WHERE order_id = ? AND status = 'AUTHORIZED';-- Inventory ServiceCREATE TABLE stock_level ( sku varchar(64) PRIMARY KEY, available integer NOT NULL CHECK (available >= 0));CREATE TABLE stock_request ( order_id varchar(36) PRIMARY KEY, sku varchar(64) NOT NULL, quantity integer NOT NULL, status varchar(16) NOT NULL -- RECEIVED | RESERVED | REJECTED | RELEASED | CANCELLED);-- Reserve without overselling (decided at write time, not from an earlier read):-- UPDATE stock_level SET available = available - ? WHERE sku = ? AND available >= ?;-- Compensation:-- UPDATE stock_level SET available = available + ? WHERE sku = ?;Why it is written this way
- Each step's transaction touches only that service's tables:
processed_event, one business table,outbox_event. No statement crosses a service boundary. aggregateidis the order id in all three outboxes, including payments and inventory. It becomes the record key, so all events of one saga on one topic stay in order.- The status columns are the saga's state. There is no separate saga table in a choreographed design, which is part of why the overall flow is hard to see.
- The
available >= ?condition is the isolation countermeasure for stock: two concurrent sagas cannot both take the last unit, whatever each of them read earlier.
package com.example.saga;import com.fasterxml.jackson.core.JsonProcessingException;import com.fasterxml.jackson.databind.ObjectMapper;import java.nio.charset.StandardCharsets;import java.util.UUID;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.common.header.Header;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.stereotype.Component;/** The two things every saga step needs: "have I seen this event?" and "publish with my state change". */@Componentpublic class SagaMessaging { private final JdbcTemplate jdbc; private final ObjectMapper json; public SagaMessaging(JdbcTemplate jdbc, ObjectMapper json) { this.jdbc = jdbc; this.json = json; } /** Records the event id. Returns false if this consumer has already processed it. */ public boolean firstTime(String consumer, ConsumerRecord<String, String> record) { return jdbc.update(""" INSERT INTO processed_event (consumer, event_id) VALUES (?, ?::uuid) ON CONFLICT DO NOTHING""", consumer, header(record, "id")) == 1; } /** Writes an outbox row in the caller's transaction. The relay publishes it to <aggregateType>.events. */ public void publish(String aggregateType, String orderId, String eventType, Object payload) { try { jdbc.update(""" INSERT INTO outbox_event (id, aggregatetype, aggregateid, type, payload) VALUES (?, ?, ?, ?, ?::jsonb)""", UUID.randomUUID(), aggregateType, orderId, eventType, json.writeValueAsString(payload)); } catch (JsonProcessingException e) { throw new IllegalStateException("Event cannot be serialised", e); } } public <T> T read(ConsumerRecord<String, String> record, Class<T> type) { try { return json.readValue(record.value(), type); } catch (JsonProcessingException e) { throw new IllegalArgumentException("Unreadable " + type.getSimpleName(), e); } } public static String header(ConsumerRecord<String, String> record, String name) { Header h = record.headers().lastHeader(name); if (h == null) { throw new IllegalArgumentException("Missing header " + name); } return new String(h.value(), StandardCharsets.UTF_8); }}Why it is written this way
- Both methods use the
JdbcTemplateof the calling service and join the listener's transaction. They do nothing atomic by themselves: the@Transactionallistener method is the unit. publishnever touches Kafka. An event exists if and only if the state change that caused it committed.- An unreadable payload throws
IllegalArgumentException, which the error handler below treats as not retryable and sends to the dead letter topic.
package com.example.payments;import com.example.saga.SagaMessaging;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;import org.springframework.transaction.annotation.Transactional;@Componentpublic class PaymentSagaListener { /** The external provider. Both calls take the order id as idempotency key. */ public interface PaymentGateway { boolean authorize(String idempotencyKey, long amountCents); void refund(String idempotencyKey); } public record OrderCreated(String orderId, String customerId, String sku, int quantity, long totalCents) {} public record PaymentAuthorized(String orderId, long amountCents) {} public record PaymentFailed(String orderId, String reason) {} public record PaymentRefunded(String orderId) {} private final JdbcTemplate jdbc; private final SagaMessaging messaging; private final PaymentGateway gateway; public PaymentSagaListener(JdbcTemplate jdbc, SagaMessaging messaging, PaymentGateway gateway) { this.jdbc = jdbc; this.messaging = messaging; this.gateway = gateway; } @KafkaListener(id = "payment-saga", topics = {"orders.events", "inventory.events"}, groupId = "payment-service") @Transactional public void onEvent(ConsumerRecord<String, String> record) { if (!messaging.firstTime("payment-service", record)) { return; } String orderId = record.key(); switch (SagaMessaging.header(record, "eventType")) { case "OrderCreated" -> authorize(messaging.read(record, OrderCreated.class)); // Two different events can ask for the same compensation. case "StockRejected", "OrderCancelled" -> refundIfAuthorized(orderId); default -> { /* not part of this service's saga steps */ } } } private void authorize(OrderCreated order) { jdbc.update(""" INSERT INTO payment (order_id, amount_cents, status) VALUES (?, ?, 'PENDING') ON CONFLICT (order_id) DO NOTHING""", order.orderId(), order.totalCents()); boolean approved = gateway.authorize(order.orderId(), order.totalCents()); int changed = jdbc.update( "UPDATE payment SET status = ? WHERE order_id = ? AND status = 'PENDING'", approved ? "AUTHORIZED" : "FAILED", order.orderId()); if (changed == 0) { return; } if (approved) { messaging.publish("payments", order.orderId(), "PaymentAuthorized", new PaymentAuthorized(order.orderId(), order.totalCents())); } else { messaging.publish("payments", order.orderId(), "PaymentFailed", new PaymentFailed(order.orderId(), "DECLINED")); } } private void refundIfAuthorized(String orderId) { int changed = jdbc.update( "UPDATE payment SET status = 'REFUNDED' WHERE order_id = ? AND status = 'AUTHORIZED'", orderId); if (changed == 0) { return; // never authorised, declined, or already refunded: nothing to compensate } gateway.refund(orderId); // throws on failure -> rollback -> the record is retried messaging.publish("payments", orderId, "PaymentRefunded", new PaymentRefunded(orderId)); }}Why it is written this way
- The provider call is outside the database transaction's protection. If the service crashes after
authorizereturns and before the commit, the redelivered event callsauthorizeagain with the same idempotency key, and a provider that supports such keys returns the first result instead of charging twice. Without that feature this step is not safe. - A declined payment is a business outcome, not an exception: it commits
FAILEDand publishesPaymentFailed. Exceptions are reserved for technical failures that should be retried. refundIfAuthorizedchanges the status first, under a guard, and calls the provider only if the guard matched. If the refund call throws, the transaction rolls back, the status returns toAUTHORIZED, and the retry tries again.- A network call inside a database transaction holds a connection while it waits. Give the gateway client a short timeout, or split the step into 'record intent' and 'call provider' transactions if that becomes a bottleneck.
OrderCreatedandOrderCancelledshare a topic and a key, so a timeout cancellation is always seen after the creation. The refund then finds the paymentAUTHORIZEDand compensates it.
package com.example.inventory;import com.example.saga.SagaMessaging;import java.util.List;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;import org.springframework.transaction.annotation.Transactional;@Componentpublic class InventorySagaListener { public record OrderCreated(String orderId, String customerId, String sku, int quantity, long totalCents) {} public record StockReserved(String orderId, String sku, int quantity) {} public record StockRejected(String orderId, String sku, String reason) {} private record StockRequest(String sku, int quantity, String status) {} private final JdbcTemplate jdbc; private final SagaMessaging messaging; public InventorySagaListener(JdbcTemplate jdbc, SagaMessaging messaging) { this.jdbc = jdbc; this.messaging = messaging; } @KafkaListener(id = "inventory-saga", topics = {"orders.events", "payments.events"}, groupId = "inventory-service") @Transactional public void onEvent(ConsumerRecord<String, String> record) { if (!messaging.firstTime("inventory-service", record)) { return; } String orderId = record.key(); switch (SagaMessaging.header(record, "eventType")) { case "OrderCreated" -> remember(messaging.read(record, OrderCreated.class)); case "PaymentAuthorized" -> reserve(orderId); case "OrderCancelled" -> release(orderId); default -> { /* not part of this service's saga steps */ } } } private void remember(OrderCreated order) { jdbc.update(""" INSERT INTO stock_request (order_id, sku, quantity, status) VALUES (?, ?, ?, 'RECEIVED') ON CONFLICT (order_id) DO NOTHING""", order.orderId(), order.sku(), order.quantity()); } private void reserve(String orderId) { StockRequest request = find(orderId); if (request == null) { // PaymentAuthorized (payments.events) overtook OrderCreated (orders.events). // Throwing rolls back, including the processed_event row, and the record is retried. throw new IllegalStateException("Order " + orderId + " not seen yet"); } if (!"RECEIVED".equals(request.status())) { return; // already handled, or cancelled by a timeout before the payment arrived } int reserved = jdbc.update( "UPDATE stock_level SET available = available - ? WHERE sku = ? AND available >= ?", request.quantity(), request.sku(), request.quantity()); if (reserved == 1) { jdbc.update("UPDATE stock_request SET status = 'RESERVED' WHERE order_id = ?", orderId); messaging.publish("inventory", orderId, "StockReserved", new StockReserved(orderId, request.sku(), request.quantity())); } else { jdbc.update("UPDATE stock_request SET status = 'REJECTED' WHERE order_id = ?", orderId); messaging.publish("inventory", orderId, "StockRejected", new StockRejected(orderId, request.sku(), "OUT_OF_STOCK")); } } private void release(String orderId) { StockRequest request = find(orderId); if (request == null) { return; } if ("RESERVED".equals(request.status())) { jdbc.update("UPDATE stock_level SET available = available + ? WHERE sku = ?", request.quantity(), request.sku()); jdbc.update("UPDATE stock_request SET status = 'RELEASED' WHERE order_id = ?", orderId); } else if ("RECEIVED".equals(request.status())) { // Cancelled before any payment event: remember it so a late PaymentAuthorized is ignored. jdbc.update("UPDATE stock_request SET status = 'CANCELLED' WHERE order_id = ?", orderId); } } private StockRequest find(String orderId) { List<StockRequest> rows = jdbc.query( "SELECT sku, quantity, status FROM stock_request WHERE order_id = ? FOR UPDATE", (rs, n) -> new StockRequest(rs.getString("sku"), rs.getInt("quantity"), rs.getString("status")), orderId); return rows.isEmpty() ? null : rows.get(0); }}Why it is written this way
- Inventory learns the order's lines from
OrderCreatedand the go-ahead fromPaymentAuthorized. They are on different topics, so their relative order is not guaranteed. The listener handles the wrong order explicitly by failing and being retried. FOR UPDATElocks the request row for the transaction, so a concurrentOrderCancelledon another partition's thread waits and then sees the final status.- Running out of stock is a business outcome: it commits
REJECTEDand publishesStockRejected, which starts the compensation in the Payment Service. releaserecordsCANCELLEDeven when nothing was reserved. That is the commutative compensation: whichever of the cancellation and the late payment event arrives first, the result is the same.
package com.example.orders;import com.example.saga.SagaMessaging;import java.util.List;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.scheduling.annotation.Scheduled;import org.springframework.stereotype.Component;import org.springframework.transaction.annotation.Transactional;import org.springframework.transaction.support.TransactionTemplate;@Componentpublic class OrderSagaListener { public record OrderConfirmed(String orderId) {} public record OrderCancelled(String orderId, String reason) {} private final JdbcTemplate jdbc; private final TransactionTemplate tx; private final SagaMessaging messaging; public OrderSagaListener(JdbcTemplate jdbc, TransactionTemplate tx, SagaMessaging messaging) { this.jdbc = jdbc; this.tx = tx; this.messaging = messaging; } @KafkaListener(id = "order-saga", topics = {"payments.events", "inventory.events"}, groupId = "order-service") @Transactional public void onEvent(ConsumerRecord<String, String> record) { if (!messaging.firstTime("order-service", record)) { return; } String orderId = record.key(); switch (SagaMessaging.header(record, "eventType")) { case "StockReserved" -> confirm(orderId); case "PaymentFailed" -> cancel(orderId, "PAYMENT_FAILED"); case "PaymentRefunded" -> cancel(orderId, "OUT_OF_STOCK"); default -> { /* PaymentAuthorized and StockRejected need no action here */ } } } /** Saga timeout: nothing else in a choreographed saga notices that a participant never answered. */ @Scheduled(fixedDelay = 60_000) public void cancelExpired() { List<String> expired = jdbc.queryForList(""" SELECT id FROM orders WHERE status = 'PENDING' AND created_at < now() - interval '15 minutes' LIMIT 200""", String.class); for (String orderId : expired) { tx.executeWithoutResult(status -> cancel(orderId, "TIMEOUT")); } } private void confirm(String orderId) { int changed = jdbc.update( "UPDATE orders SET status = 'CONFIRMED' WHERE id = ? AND status = 'PENDING'", orderId); if (changed == 1) { messaging.publish("orders", orderId, "OrderConfirmed", new OrderConfirmed(orderId)); } // 0 rows: the order was already cancelled by the timeout. OrderCancelled has been published, // and Inventory releases the reservation when it reads it. } private void cancel(String orderId, String reason) { int changed = jdbc.update(""" UPDATE orders SET status = 'CANCELLED', cancel_reason = ? WHERE id = ? AND status = 'PENDING'""", reason, orderId); if (changed == 1) { messaging.publish("orders", orderId, "OrderCancelled", new OrderCancelled(orderId, reason)); } }}Why it is written this way
- Every transition is
... AND status = 'PENDING'. Exactly one of confirm, cancel-by-event and cancel-by-timeout can win for an order, and only the winner publishes. - The order is cancelled on
PaymentRefunded, the end of the compensation chain, not onStockRejected. The customer-facing status changes only when the money is on its way back. - The sweeper cancels each expired order in its own short transaction, so one failure does not block the others. It uses the same guarded
cancelas the listener. - If the timeout wins against a late
StockReserved, the confirm matches zero rows. The already publishedOrderCancelledreaches Inventory after it has reserved, andreleasereturns the stock. Choose the deadline well above normal processing time so this stays rare.
package com.example.saga;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;import org.springframework.kafka.listener.DefaultErrorHandler;import org.springframework.kafka.support.ExponentialBackOffWithMaxRetries;@Configurationpublic class SagaErrorHandling { @Bean public DefaultErrorHandler sagaErrorHandler(KafkaTemplate<String, String> template) { var backOff = new ExponentialBackOffWithMaxRetries(6); backOff.setInitialInterval(500L); backOff.setMultiplier(2.0); backOff.setMaxInterval(10_000L); var handler = new DefaultErrorHandler(new DeadLetterPublishingRecoverer(template), backOff); handler.addNotRetryableExceptions(IllegalArgumentException.class); return handler; }}Why it is written this way
- Spring Boot associates a single
CommonErrorHandlerbean with the auto-configured listener container factory, so every@KafkaListenerabove uses it. - The default error handler retries immediately, nine times (
FixedBackOff(0L, 9)). That is useless for 'the other event has not arrived yet' and for a provider outage, which both need time. An exponential backoff gives them time. - After the retries the record goes to the dead letter topic, named after the original topic with the suffix
-dltby default in this version. A saga event in a dead letter topic is a saga that will not finish without a person: alert on it. - Retries happen in the consumer thread and delay the records behind them in that partition. The total backoff must stay well under
max.poll.interval.ms, or the consumer is removed from the group.
Configuration
A saga's correctness comes from the code above, but these settings decide how it behaves under failure. Kafka defaults are from the 4.1 consumer configuration reference, Spring behaviour from the Spring for Apache Kafka 3.3 reference. The last two are application settings.
enable.auto.commitDefault true in the Kafka consumer. Spring for Apache Kafka sets it to false unless you configure it
- What it does
- Whether the consumer commits offsets periodically in the background, regardless of whether processing finished.
- When to change it
- Leave Spring's
false. The listener container then commits after the listener returns, which is what makes a saga step at-least-once.
spring.kafka.listener.ack-modeDefault BATCH
- What it does
- When the container commits:
BATCHcommits after all records returned by one poll have been processed. - When to change it
- The default is fine with idempotent steps.
RECORDcommits after each record and reduces the number of redeliveries after a crash, at the cost of more commits.
max.poll.interval.msDefault 300000 (5 minutes)
- What it does
- Maximum time between two calls to
poll()before the consumer is considered failed and its partitions are reassigned. - When to change it
- Processing of one poll's records, including in-thread retries and provider calls, must fit within it. Lower
max.poll.records(default 500) or shorten timeouts before raising this.
auto.offset.resetDefault latest
- What it does
- Where a group with no committed offset starts reading.
- When to change it
- Set
earliestfor saga participants. Withlatest, a new service or a renamed group silently skips events that were published before it first connected.
DefaultErrorHandler back-offDefault FixedBackOff(0L, 9): nine immediate retries
- What it does
- How often and how quickly a failed record is redelivered before the recoverer is called.
- When to change it
- Replace it with an exponential backoff, and declare exceptions that can never succeed as not retryable so they go to the dead letter topic at once.
saga deadlineDefault none (application setting, 15 minutes in this lesson)
- What it does
- How long an order may stay
PENDINGbefore the sweeper cancels it. - When to change it
- Well above the normal completion time plus the retry budget of every step, and below what a customer will tolerate. Too short and healthy sagas are cancelled while still in flight.
provider idempotency keyDefault none (application setting, the order id in this lesson)
- What it does
- The key sent with every authorise and refund call so the provider can recognise a repeat.
- When to change it
- Derive it from the saga, never generate it per attempt. A fresh key on each retry turns a redelivery into a second charge.
Hands-on exercise
Find the double refund
See why a saga step must be idempotent by crashing a payment step between its effect and its offset commit, then decide which defence each failure needs.
Steps · 0 of 5 done
A payment step that crashes before its offset is committed
Simulation · not a real clusterRun the crash with the idempotent consumer off, then on, and watch how many times the payment effect is applied.
What you should see
Without the idempotent consumer the effect is applied twice: the record is redelivered because its offset was never committed. With it, the record is still delivered twice but applied once. In the saga, the event-id check does not stop the two different events. The guarded UPDATE ... AND status = 'AUTHORIZED' does: the second one matches zero rows. For the crash after gateway.refund, the transaction rolled back, so the status is AUTHORIZED again and the redelivery calls the provider a second time. Only the idempotency key makes that a no-op.
What can go wrong
Compensation that is not idempotent
- Symptom
- A customer receives two refunds for one cancelled order, or stock for one order is added back twice and the count drifts upward.
- Cause
- The compensation is triggered by more than one event (rejection and timeout) or by a redelivery, and it applies its effect unconditionally.
- Fix
- Guard every compensation on the current state (
WHERE status = 'AUTHORIZED'), deduplicate by event id, and pass a saga-derived idempotency key to external systems.
Sagas that never finish
- Symptom
- Orders sit in PENDING for hours. Money is held on customers' cards for orders that will never ship.
- Cause
- A participant is down or lagging, or its event went to a dead letter topic, and nothing in a choreographed saga is waiting for it.
- Fix
- A deadline with a sweeper that cancels and publishes the cancellation, an alert on the count of orders pending past the deadline, and an alert on every saga dead letter topic.
Assuming order across topics
- Symptom
- Inventory occasionally fails with 'order not found' when handling a payment event, mostly under load or after a restart.
- Cause
OrderCreatedandPaymentAuthorizedare on different topics. Kafka orders records within one partition only, and the two partitions are consumed independently.- Fix
- Handle the missing predecessor explicitly: fail and retry with backoff, or park the event. Or carry the needed data in the later event so the step does not depend on the earlier one.
Publishing the event outside the step's transaction
- Symptom
- A payment is authorised but the order never moves on, or a
StockReservedevent exists for a reservation that was rolled back. - Cause
- The listener updates its database and calls
KafkaTemplate.senddirectly. This is the dual write from the outbox lesson, now in the middle of a saga where a lost event stalls it and a phantom event corrupts it. - Fix
- Write the outgoing event to the outbox in the same transaction as the state change, in every participant.
Acting on intermediate state
- Symptom
- The warehouse starts picking an order that is cancelled a minute later. A report counts revenue for orders that were refunded.
- Cause
- Another process read state written by a saga step that was later compensated. A saga has no isolation, so that state was really there.
- Fix
- Expose the pending status and make downstream processes wait for the final event (
OrderConfirmed). Use reservations instead of final effects until the saga completes.
An irreversible step in the middle
- Symptom
- Customers get an 'order confirmed' email followed by a cancellation, or a parcel is dispatched for a payment that then fails.
- Cause
- A step that cannot be compensated runs before steps that can still fail.
- Fix
- Reorder: every step that can fail for business reasons comes first, irreversible steps come after the last one that can fail, and they retry until they succeed.
How to troubleshoot
One order is stuck in PENDING
Check
kafka-console-consumer.sh --bootstrap-server broker:9092 --topic payments.events --from-beginning --property print.key=true --property print.headers=true | grep o-1001
- How to read it
- Repeat for each of the three topics to find the last event published for that order id. The saga stopped at the service that should have reacted to it. On large topics start from a timestamp instead of the beginning. Recent 4.x releases deprecate
--propertyhere in favour of--formatter-property.
Many orders are stuck at the same step
Check
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group inventory-service
- How to read it
- Growing lag on
payments.eventsfor the group means Inventory is not keeping up or is stopped. Lag stuck on one partition with the same CURRENT-OFFSET suggests a record that is being retried over and over.
A step failed permanently
Check
kafka-console-consumer.sh --bootstrap-server broker:9092 --topic inventory.events-dlt --from-beginning --property print.headers=true
- How to read it
- Each record in the dead letter topic is one saga that cannot continue. The exception headers added by the recoverer say why. Fix the cause and republish, or complete the compensation by hand.
Events seem not to be published at all
Check
SELECT aggregatetype, count(*), min(seq) FROM outbox_event WHERE published_at IS NULL GROUP BY aggregatetype;
- How to read it
- Rows waiting in a participant's outbox mean its relay is down. The step committed and the saga will continue when the relay publishes. An empty outbox with a missing event means the step itself never ran.
How many sagas are unfinished right now
Check
SELECT count(*), min(created_at) FROM orders WHERE status = 'PENDING';
- How to read it
- A small count of recent orders is normal work in progress. Any order older than the deadline means the sweeper is not running or its cancellation is failing.
Key takeaways
- A saga replaces one distributed transaction with a sequence of local transactions and a compensating action for each step that may need undoing.
- Compensations are business operations with visible effects. They must be idempotent and retried until they succeed.
- Choreography spreads the flow across subscriptions and suits short flows. Orchestration puts the flow in one coordinator and suits long or branching ones.
- Every step: deduplicate the event, check and change state under a guard, write the next event to the outbox, all in one local transaction.
- Sagas have no isolation. Use pending states, reservations and write-time guards, and put irreversible steps last.
- Without a timeout a choreographed saga can stay unfinished forever. Count the pending-too-long orders and alert on them.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
Inventory was down for 40 minutes. Afterwards finance finds 23 orders that were refunded twice. For each, the Payment Service log shows a refund on OrderCancelled (reason TIMEOUT) and another on StockRejected a few minutes later. The refund handler deduplicates by event id and then calls the provider with a newly generated request id.
Which change prevents the second refund?
Scenario 2 of 3
A checkout flow has grown to nine steps across six services, with a fraud review that can take up to two days and three different cancellation paths. It is choreographed. On-call engineers regularly need an hour to find out why a given order is not progressing, and adding a step last month required changes in four services.
What is the most appropriate direction?
Scenario 3 of 3
A warehouse service subscribes to payments.events and starts picking as soon as it sees PaymentAuthorized, to save time. About 2% of picked orders are later cancelled because stock was rejected for another line or the fraud check failed, and staff have to put items back.
What is wrong with the design?
References
References: Apache Kafka 4.1: consumer configuration reference (enable.auto.commit, max.poll.interval.ms, auto.offset.reset) ↗ · Spring for Apache Kafka 3.3 reference: Handling exceptions (DefaultErrorHandler, back-off, dead letter publishing) ↗ · Spring for Apache Kafka 3.3 reference: Message listener containers (AckMode, enable.auto.commit) ↗ · Spring for Apache Kafka 3.3 reference: Transactions (a database transaction on a listener method) ↗ · Hector Garcia-Molina and Kenneth Salem, 'Sagas', ACM SIGMOD 1987 (the original paper) ↗ · Chris Richardson, microservices.io: Saga pattern (choreography and orchestration) ↗ · Report a problem